Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes to apply MAML-style meta-learning to few-shot semantic segmentation in images. It argues that this type of algorithm may be more computationally-efficient than existing methods and may offer better performance with a higher number of examples. They further propose to perform hyper-parameter search to choose a new learning rate for the inner learning process after optimizing for the network parameters.

As far as I know, this is the first paper to apply gradient-based (i.e. MAML-style) meta-learning to this specific problem. Existing approaches to few-shot semantic segmentation have mostly used multi-branch conv-networks to condition the output on the training examples. This paper shows that (FO)MAML achieves similar accuracy to the FSS-1000 baseline. This is an empirical contribution in itself. The paper also demonstrates that the EfficientNet architecture can be applied to segmentation.

Major concerns:

(1.1) The improvement obtained by the hyper-parameter optimization seems quite marginal (79.0 - 81.4 and 73.3 - 73.9) and there is no study of the variance of the results. The fact that better performance is obtained by tuning the learning rate on the *training* set suggests to me that the improvement might not be significant. You could repeat the experiment by sampling multiple different training and testing sets (with different classes) to estimate (some of) the variance.

(1.2) The formalization in Section 4 is mathematically appealing but seems unnecessary. In the end, the paper is essentially arguing that it's better to use different learning rates (for the inner loop) during meta-training and meta-testing. This seems obvious, since this includes equal learning rates as a special case. The paper proposes to optimize the latter learning rate using a gradient-free method. This argument can be made without considering generalization bounds.

(1.3) One of the central claims of the paper is that "meta-learned representations smoothly transition as more data becomes available." It's not entirely clear what this means. I suppose it means that, with few shots, it should perform as well as existing few-shot methods, but with many shots, it should perform as well as a standard learning algorithm. The paper failed to present any evidence that existing algorithms for few-shot segmentation do not satisfy this property. It would strengthen the argument to include an existing few-shot segmentation algorithm in Figure 2.

(1.4) The details of the experiment in Figure 2 are not clear. Was a different number of iterations used when there are hundreds of examples? Were different hyper-parameters used when optimizing from a pre-trained checkpoint? It would be unfair to use the same hyper-parameters which had been optimized specifically for the meta-learned initialization.

Other issues:

(2.1) It's not clear what it means to achieve human-level performance in the few-shot task and the many-shot task. What is human-level performance at few-shot segmentation? It seems to me that humans are capable of segmenting novel objects (i.e. zero-shot) with almost perfect accuracy. Does this mean that your method should achieve the same accuracy with few- and many-shots?

(2.2) The use of early stopping was unclear. Do you use a fixed number of SGD iterations during training and a variable number of iterations (determined by a stopping criterion) during testing? However, this seems to be contradicted by the statement that the UHO algorithm determined an optimal number of iterations (8) for testing? On the other hand, this seems like too few iterations with hundreds of shots. Maybe the automatic stopping criterion was only used with many shots? Or maybe training proceeds until either the stopping criterion is satisfied or the maximum number of iterations is reached? Furthermore, the early stopping criterion was not specified.

(2.3) Missing reference: Meta-SGD (arxiv 2017) considers a different learning rate for every parameter and updates the learning rates during meta-training .

(2.4) There was no discussion of the running time of different methods. This would be particularly interesting in the many-shot regime. How slow are the RelationNet approaches?

(2.5) It is claimed that Figure 1 demonstrates that  "the estimated optimal hyperparameters for the update routine ... are not the same as those specified a priori during meta-training". However, it seems that the optimal learning rate is awfully close to the dotted blue line (within the variance of the results).

(2.6) For the IOU loss (equation 10), what are the predicted y values? Are they arbitrary real numbers? Do you use a sigmoid to constrain them to real numbers in [0, 1]?

Minor:

(3.1) It is not worth stating the optimal learning rate to more than 3 or 4 significant figures.
(3.2) Use 1 \times 1 instead of 1 x 1.
(3.3) Is there a reference for the Dice score? Where does the name come from?
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary - The paper first makes the observation that training algorithms and architectures for meta-learning have become increasingly specific to the few-shot set of tasks. Following this, the authors first investigate if it’s possible to learn good initializations for dense structured prediction tasks such as segmentation (across arbitrary amounts of available input data). Concretely, the claimed contributions of the paper include -- (1) extension and analysis of first order MAML like approaches to image segmentation; (2) using a formalized notion of the generalization error of episodic meta-learning approaches to decrease error on unseen tasks; (3) doing this via a novel neural network parametrically efficient segmentation architecture and (4) empirically comparing meta-learned initializations with ImageNet pre-trained initializations with increasing training set sizes.

Strengths

- Apart from the flaws mentioned under weaknesses, the paper is generally easy to follow. While it’s somewhat hard to understand the motivations and the concrete contributions made by the paper, sections are more-or-less well-written.
Using the proposed hyper-parameter search scheme over first-order MAML approaches demonstrates improvements over not baselines which do not use the same and baselines which do not involve meta-learning.

Weaknesses

The paper has some major weaknesses that affect the clarity of the points being conveyed in several sections. These weaknesses form the basis of my rating and addressing these would not only help in adjusting the same but would also help in improving the current version of the paper significantly. Highlighting these below:

- The paper claims to make several contributions but it’s hard to concretely understand them in several sections. For instance, the abstract mentions -- ‘’A natural question that ….. human level performance in both.” The statement is slightly unclear to me -- is the intended sentiment the fact that the goal should be to develop a single algorithm that works well for both few-shot and many-shot settings? If so, why should that be the case? Essentially, what is the limiting factor being identified that restricts few-shot approaches from performing well in many-shot settings? Maybe the statement could be framed better but in it’s current form it’s unclear what is being conveyed. When this is mentioned again in the introductory section, it is followed by a statement indicating that meta-learning an initialization is one solution. Why is this surprising? Maybe I’m mis-understanding the motivation behind the claim. Could the authors clarify this? 

- Similarly, it’s unclear what question (3) in the introduction is trying to address. Which “data” (training / testing set of tasks) is the fixed update policy not conditioned on? Could the authors clarify this?

- The description of the single update hyper-parameter optimization (UHO) is hard to understand in Sec. 4. -- specifically the text surrounding eqns (5) and (6). The transition from Eqn (5) -> Eqn (6) is unclear. Could the authors clarify this clearly? This section is further referred to in subsequent sections as a supporting basis for some of the obtained results (specifically, the last para on page 6)

Reasons for rating

I found certain sections of the paper particularly hard to understand and interpret. I would encourage the authors to address these more clearly in the responses. The highlighted strengths and weaknesses of my rating and addressing those clearly would help in improving my current rating of the paper.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper examines the performance of MAML, FOMAML, and Reptile gradient based meta-learning algorithms on the task of semantic image segmentation. The paper proposes to do black box optimization (successive halving) on the hyperparameters of the inner loop of the gradient meta-learners for improved performance. The paper proposes some modification on the segmentation model architecture (no ablation study presented). Finally, it is shown that pre-training using meta-learning on similar segmentation tasks works better then just using ImageNet based model pre-training.

In its current form I suggest to reject the paper and urge the authors to improve it according to the following points:

1. In parts (specifically the intro and some other earlier parts) the paper is very well written, but the later parts, the description of the model modifications (did you consider to add an architecture diagram?), the details of the experiments, the punch-line of the theory development that has been attempted, etc are not very clear and hard to follow. I suggest the authors to improve the readability of these parts, add some helpful / motivating diagrams and examples (perhaps some qualitative results too?), state more clearly what is used for meta-training? how it is made sure that meta-testing is done on a separate set of categories? (I did not see this split in the Appendix) and etc

2. My main concern is novelty. As it stands, the current novelty proposition is: black-box optimization of LR and number of iterations in MAML style meta learning (hardly novel), architecture modifications (no ablation study if these help or not), small improvement on FSS-1000 5-shot test (what about other shots? still not sure about the splits), and showing meta-training on similar tasks is better then not doing it (that is using ImageNet pre-trained backbone for init) - again hardly a novel insight. For the last point, saying that meta-learned model was initialized from scratch does not cut it, as it was meta-trained on massive data that is more related to the test tasks then the ImageNet.

I suggest the authors to mainly focus on 2, although making the writing clearer and better is also very important for a high quality paper.
review_title: Official Blind Review #3

Meta-review: The reviewers reached a consensus that the paper was not ready to be accepted in its current form. The main concerns were in regard to clarity, relatively limited novelty, and a relatively unsatisfying experimental evaluation. Although some of the clarity concerns were addressed during the response period, the other issues still remained, and the reviewers generally agreed that the paper should be rejected.
Decision: Reject