Meta-Learning Initializations for Image Segmentation
Authors: Sean M. Hendryx
Andrew B. Leach
Paul D. Hein
Clayton T. Morrison
Venue:  (2020)
Keywords: meta-learning
image segmentation

Abstract: While meta-learning approaches that utilize neural network representations have made progress in few-shot image classification, reinforcement learning, and, more recently, image semantic segmentation, the training algorithms and model architectures have become increasingly specialized to the few-shot domain. A natural question that arises is how to develop learning systems that scale from few-shot to many-shot settings while yielding human level performance in both. One scalable potential approach that does not require ensembling many models nor the computational costs of relation networks, is to meta-learn an initialization. In this work, we study first-order meta-learning of initializations for deep neural networks that must produce dense, structured predictions given an arbitrary amount of train- ing data for a new task. Our primary contributions include (1), an extension and experimental analysis of first-order model agnostic meta-learning algorithms (including FOMAML and Reptile) to image segmentation, (2) a formalization of the generalization error of episodic meta-learning algorithms, which we leverage to decrease error on unseen tasks, (3) a novel neural network architecture built for parameter efficiency which we call EfficientLab, and (4) an empirical study of how meta-learned initializations compare to ImageNet initializations as the training set size increases. We show that meta-learned initializations for image segmentation smoothly transition from canonical few-shot learning problems to larger datasets, outperforming random and ImageNet-trained initializations. Finally, we show both theoretically and empirically that a key limitation of MAML-type algorithms is that when adapting to new tasks, a single update procedure is used that is not conditioned on the data. We find that our network, with an empirically estimated optimal update procedure yields state of the art results on the FSS-1000 dataset, while only requiring one forward pass through a single model at evaluation time.

Under review as a conference paper at ICLR 2020
META-LEARNING INITIALIZATIONS FOR IMAGE SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
While meta-learning approaches that utilize neural network representations have made progress in few-shot image classification, reinforcement learning, and, more recently, image semantic segmentation, the training algorithms and model archi- tectures have become increasingly specialized to the few-shot domain. A natural question that arises how to develop learning systems that scale from few-shot to many-shot settings while yielding competitive performance in both. One scalable potential approach that does not require ensembling many models nor the compu- tational costs of relation networks, is to meta-lear an initialization. In this work, we study first-order meta-lcarning of initializations for deep neural networks that must produce dense, structured predictions given an arbitrary amount of training data for a new task. Our primary contributions include (1), an extension and exper- imental analysis of first-order model agnostic meta-learning algorithms (including FOMAML and Reptile) to image segmentation, (2) a novel neural network archi- tecture built for parameter efficiency and fast learning which we call EfficientLab. (3) a formalization of the generalization eπror of meta-learning algorithms,, which we leverage decrease error on unseen tasks, and (4) a small benchmark dataset, FP-k, for the empirical study of how meta-learning systems perform in both few- and many-shot settings. We show that meta- heaa-leerned initializations for image seg- mentation provide value for both canonical few-shot learning problems and larger datasets, outperforming random and ImageNet-trained initializations for up to 400 densely labeled examples. Finally, show both theoretically and empirically that a key limitation of MAML-type algorithms is that when adapting to new tasks, a single update routine is used that not conditioned on the available data for new task. We find that our network, with an empirically estimated optimal update pro- cedure yields state of the art results the FSS- 1000 dataset, while only requiring one forward pass through a single model at evaluation time.
INTRODUCTION
Humans have a remarkable capability to not only learn new concepts from a small number of labeled examples but also to gain expertise more data becomes available. In recent years, there has been substantial progress in high accuracy image segmentation the high data regime (see [iu al. (2019) and their references). While meta-learinng approaches that utilize neural network representations have made progress in few-shot image classification, reinforcement learning, and, more recently, image semantic segmentation, the training algorithms and model architectures have become increasingly specialized to the low data regime. A desirable property of learning system one that effectively applies knowledge gained from few or many examples, while reducing the generalization gap when trained on little data and not being encumbered by its own learning routines when there are many examples. This property desirable because training and maintaining multiple models is more cumbersome than training and maintaining is how to develop learning systems that scale from few-shot to many-shot settings while yielding competitive accuracy in both. One scalable potential approach that does not require ensembling many models nor the computational costs of relation networks, is to meta-learn an initialization.
model. A natural question that arises
In this work, we specifically address the problem of meta- meaanng initializations for deep neural net- works that must produce dense, structured output, such as for the semantic segmentation of images. We ask the following questions:

---

Under review as a conference paper at ICLR 2020
1. Do first-order MAML-type algorithms extend to the higher dimensional parameter spaces, dense prediction, and skewed distributions required of semantic segmentation?
2. How robust are the representations that the model has meta-learned to perturbations in the hyperparameters of the update routine?
3. Are MAML-ttpe algorithms hindered by having a fixed update policy for training and testing tasks that is not conditioned on the available labeled examples for new task?
4. What is the number of labeled examples beyond which a conventional approach to training deep neural networks outperforms the meta-learned initializations?
To address the fourth question, we put together small benchmark dataset, which we call FP-K, that contains 400 training examples for 5 tasks each. In recent works eL al. 2017 Shaban et al. 2017
Rusu et al. 2018 Zhang et al. 2019 Lee al. 2019), few-shot learning approaches have become increasingly y complex and appear to be specialized to the few-shot domain. This raises many open questions, such as: What is the accuracy of a few-shot learning system when more labeled examples become available? After certain number of labeled examples, will the few-shot learning system have the same accuracy as a simpler training approach such as conventional training via SGD? so, what is the number of labeled examples beyond which conventional approach to training deep neural networks outperforms a meta-learning system? We address these questions in 5544 providing empirical justification for our meta-leariing approach for up to 400 densely labeled examples.
Through a series of theoretical and empirical analyses, we shed new light on the representations that model agnostic meta-leane meta-learning algorithms learn and how they adapt to unseen tasks. In summary, address the above research questions as follows: We show that MAML--tppe algorithms do extend to few shot image segmentation, yielding state of the art results when their update routine is optimized after meta-training and when the model is regularized. Addressing question 2, we find that the meta-learned initialization's 's performance when being evaluated on a task is particularly sensitive to changes in the update routine's hyperparameters (see Figure 2 We show theoretically in 3.2 and empirically in our results (see Table 2 that a fixed update routine does hinder generalization performance. Finally, we address question& 4 by showing that our meta-learnet initializations are competitive with ImageNet (Deng et al. (009) trained initializations for up to 400 labeled examples. We will release our code and the FP-k dataset upon acceptance.
RELATED WORK
Learning useful models from a small number of labeled examples of a new concept has been studied for decades (Thrun, 1996) yet remains a challenging problem with no semblance of unified solu- tion. The advent of larger labeled datasets containing examples from many distinct concepts (Vinyals) eet al. 2016) has enabled progress in the field in particular by enabling approaches that leverage the representations of nonlinear neural networks. Image segmentation is a well-suited domain for ad- vances in few-shot learning given that the labels are particularly costly to generate (Wei et al. 2019).
Recent work in few-shot learning for image segmentation has utilized three key components: (I) model ensembling (Shaban et al.) (017), (2) relation networks (Santoro et al.] (017)), and (3) late fusion of representations (Rakelly et al. 2018 Wei et al. 2019). The inference procedure of ensem- bling models with a separately trained model for each example has been shown produce better predictions than single shot approaches but will scale linearly in time and/or space complexity (de- pending on the implementation) in the number of training examples, as implemented in Shaban al.) 2017). An exciting recent approach is that of using relation networks to learn modules that can rea- son about the relationships of examples in a domain (Santoro et al. 2017). Relation networks have seen increased adoption in meta-learning systems (Rusu et al. 2018 and were recently employed in few-shot segmentation in [Zhang et al. (2019) and Wei et al. (201)). While relation networks yield impressive results in the few-shot domain, they typically come with O(n)) training costs where n the number of support and query examples. The use of multiple passes through subnetworks via iterative optimization modules used by Zhang et al. (2019) further exacerbate these costs. The authors in Wei et al. (2019) take an elegant approach to reducing the computational complexity by element- wise summing all feature maps of the support examples over the channel dimension, such that the encoded feature map tensors have the same depth regardless of the size of the support set, forming type of embedded memory of the class and then relating that to the query examples. While
2

---

Under review as a conference paper at ICLR 2020
this approach reduces computational complexity of traditional applications of relation networks, its efficacy in scaling as more training data becomes available remains unclear.
Model Agnosite Meta- Meta-Learning (MAML) is gradient- based meta-learning approach introduced in Finn et al. (007)) First Order MAML (FOMAML) reduces the computational cost by not requiring backpropogating the meta-gradient through the inner-loop gradient and has been shown work sim- ilarly well on classification tasks (Finn et al.) 2017] Nichol & Schulman, 2018). Though learning an initialization has the potential unity few-shot and many-shot domains, initializations learned from MAML-type algorithms have been seen to overfit in the low-shot domain when adapting suffi- ciently expressive models such deep residual networks that may be more than a small number of convolutional layers | (Mishra et al. 2018 Rusu et al. 2018). The Meta-SCC learning framework added additional capacity to the same network architecture used in MAML. with improved gener- alization by meta-learning learning rate for each parameter in the network (Li et al., 201 7), but lacks a first order approximation. In addition to possessing potential to unify few- and many-shot domains, MAML-type algorithms are intriguing in that they impose constraints model archi- tecture, given that the output of the meta-learning process is simply an initialization. Futhermore, the meta-learning dynamics, which learn temporary memory of sampled task, related to the older idea of fast weights (Hinton & Plaut 1987 Ba et al., (016). Despite being dataset size and model architecture agnostic, MAML-ttyp algorithms are unproven for high dimensionality of the hypothesis spaces and the skewed distributions of image segmentation problems data (Rakelly et al.) 2018). In this work, we show, for the first time, that model agnostic meta-learning algorithms do in fact naturally extend to image segmentation, yielding state of the art results when their update procedure is optimized.
PRELIMINARIES
We assume access distribution over tasks ((T) that sampled from generating distribution P(T). In the context of image segmentation, an example from task T; is comprised of an image and its corresponding binary masks y. which assign each pixel mutually exclusive membership to the target (for example, black bear) or background class. Following the notation put forth in Rusu al. 2018). the tasks in p(T) are divided into disjoint meta-training S validation Sval and test Stest sets. The objective is to find set of optimal model parameters 6 that minimize the loss L of predictions 9 with respect to the distribution of examples qT, (z, y) for a new task T. To estimate 6 in practice, apply update procedure U with hyperparameters to train model on the labeled examples == (x((), (x) } from eT, (Ir, v).
3.1 MODEL AGNOSTIC META-LEARNING
The meta-learning algorithms introduced in Finn et al. (2017) and Nichol & Schulman (2018), work by sampling few-shot dataset D, from a task T. adapting model's internal representations, 0, for small number of gradient updates, j. to produce 8g after steps of gradient descent. We define this update routine as the function U. The objective is find an optimal initialization 6; which minimizes loss after training on new task:
0 arg arg min Ep [Bar, (U D,, []]] Ap
(1)
Taking the derivative with respect to 6,
U(())
(2)
ae,
results in the term U" which is the derivative of gradient- based update procedure and, hence, contains second order derivatives.
In first-order renditions, finite difference is used to approximate the gradient of the meta-update between 8 and 0,- FOMAML differs from Reptile in that D, is split into mini-training, Der. and
'The original MAML and Reptile convolutional neural networks (CNNs) use four convolutional layers with 32 filters each for Minilmagenet (Finn al. 2017 Nichol & Schulman 2018
3

---

Under review as a conference paper at ICLR 2020
validation, Dval, sets. The last batch of the inner-loop is used to "correct" for overfitting to Drr by taking a gradient step in the direction of descent pointed to by the loss, L. on Deal- The difference between the two approximations can be summarized by how they make use of two update steps an initial update using the training examples, and second using the validation examples:
θ - U(6:: ; Der ,Wrr)
(3)
0' + U(O Deals Weat)
(4)
Reptile only makes use of the initial update in its first order difference, where FOMAMI. uses both Reptile: VO, x 0,
(5)
FOMAML: VO, 6'
(6)
This gradient approximation can then be used optimize the initialization via stochastic gradient descent:
0. - 0. t vO.
or any other gradient- based update procedure.
3.2 GENERALIZATION ERROR OF LEAA-ING ALGORIHTMS
The generalization gap between the error on the generating distribution and the empirical data in meta-learning is twofold: from the domain of all tasks T to the sample Tir, and within that, from all examples in T to Dg. Taking P as p(Ti]T: E Ter), and similarly, qT, as qT. (z, ul(z. y) E D.), we approximate the empirically optimal initialization
0- = arg min Ep [Eär, (U :., w]]]
(8)
%g
The generalization error can then be expressed
Ep Eurr C (V(())
EniT c (U(O))
Ep
(9)
To reduce the generalization gap in practice, model agnostic meta-learning approaches must address both generalization across tasks, and within tasks from the limited examples available. Because the loss is defined in terms of the update routine U and given that U in practice is parameterized by both a sampled dataset D, and w, there will also be an optimal set of hyperparameters, w for every O.. which may or may not be equal w. Thus, the expectation of the loss with respect to w given fixed 6, can also be minimized:
W = arg min Eg [Ber, (U :0, D.)).]
(10)
Thus, on many problems, it may useful to estimate an optimal update procedure on the available data given the meta-learned initialization. We apply this insight to developing a simple update hyperparameter optimization (UHO) algorithm which we find significantly improves performance
on the test set of tasks Stest We discuss hyperparameter tuning further in 5.3 The algorithm is shown in full pseudocode the appendix in
4 EFFICIENTLAB ARCHITECTURE FOR IMAGE SEGMENTATION
To extend first-order MAML-type algorithms to more expressive models, with larger hypothesis spaces, while yielding state of the art few-shot learning results, we developed a novel neural network architecture, which we term EfficientLab. The top level hierarchy of the network's 's organization of computational layers is similar Chen et al. (018)), with 4 convolutional blocks that successively halve the features in spatial resolution while increasing the number of feature maps. This is followed by a 4x bilinear upsampling which is concatenated with features from a long skip connection from the second downsampling block in the encoding part of the network. The concatenated low and high resolution features are then fed through an atrous spatial pyramid pooling (ASPP) module and finally bilinearly upsampled original image size.
4

---

Under review as a conference paper at ICLR 2020
14
5x5
MBConvv
bilinear upsample
28
3x3
MBCCovv6
skip
connection
concarenate
28
5x5
MBCCrvV
1xl conv
3x3 atrous CODV
average pool
reckina skip decoder
56
3x.3 MBConv66
concatenare
3x3
3x3
MBCarvi
bilinear upsample
234
conv
!
Figure 1: Diagram of the computations performed by the Efficientlaai Lab neural network. Nodes rep- resent functions and edges represent output tensors. Output spatial resolutions are written next to the output edge. The high level architecture shows the EfficientNet feature extractor the left with mobile inverted bottleneck convolutional blocks (see Tan Le (2019); [andler et al. (2018) for more details). On the right is the residual skip decoder module that we utilize in the upsampling branch of EfficientLab.
The differences between our model and the DeepLab model are in (1) the encoder network used and (2) how the low resolution embedded features are upsampled full resolution predictions. For the encoding subnetwork, we utilize the recently proposed EfficientNet (Tan & Le 2019). After encoding the images, instead of feeding them directly into an atrous spatial pyramid pooling module (ASPP), we first immediately bilinearly upsample the features by 4x. The upsampled features are then concatenated with features from the second downsampling block. Moving the ASPP module to the usampled resolution provides two advantages. First, it allows us use convolutional module in place of two. Due to the high dimensionality of the features along the channel axis at the lower resolution feature maps, the convolutional kernels are especially expensive in terms of number of parameters. Second, the ASPP module is designed to learn multiscale context which could be useful in refining the boundaries of semantic features in mid-resolution feature maps. Our ASPP module utilizes three parallel branches of a 1 X convolution, 3 x convolution with dilation rate 2, and simple veeaae-ppoolnng across spatial dimensions of the feature maps. The output of the three branches is concatenated and fed into a final 3 x 3 convolutional layer with 112 filters. A residual connection wraps around the convolutional layers ease gradient flow P We call this structure residual skip decoder (RSD) and its computational graph of operations is shown in Figure Before the final 1 convolution that produces the unnormalized heatmap of class scores, we use single layer of dropout with a drop rate probability 0.2 We use the standard softmax to produce the normalized predicted probabilities.
We use batch normalization layers following convolutional layers (loffe & Szegedy 2015). We meta-learn the 3 and 7 parameters, adapt them at test time to test tasks, and use running averages as estimates for the population mean and variance, E(ar] and Var Var(z), at inference time as suggested in
Residdul connections have been suggested to make the loss landscape of deep neural networks (Li al. 2018). If this the case, it could be especially helpful in finding low-error minima via gradient- based update routines such as those used by MAMI. FOMAMI.
Reptile.
As described in Li al. 2019) and used in Tan & Le 2019) the dropout layer is applied after all batch layers.
5

---

Under review as a conference paper at ICLR 2020
Antoniou et al. (2018). All parameters at the end of evaluation call are reset their pre-adaptation
values to stop information leakage between the training and validation sets. The network is trained with the binary cross entropy minus the log of the dice score (Dice) 1945), which we adapt from the loss function of (Iglovikov et al. 2017), plus an L2 regularization on the weights:
C - Iog(T) + A 11012
(11)
where H is binary cross entropy loss:
H
(ys log ii + (1 y/ log (1 ū())
(12)
is the modified Dice score:
J= 210U
(13)
IoU +1
and ToU is the intersection over union metric:
Σ n1
yid: +
IoU
(14)
U/s + 01 - y:ui +
1=1
EXPERIMENTS
We evaluate the FOMAML and Reptile meta-learning algorithms the FSS-1000 dataset. Model topology development f and hyperparameter search was done on a held out set of validation tasks Sual and not the final test tasks. For the final evaluations, the training and validation tasks, Str U using a meta-batch size of 5, an inner loop batch size of 8. and 5 inner loop iterations. During training. we use stochastic gradient descent (SGD) in the inner loop with a fixed learning rate of 0.005. During training and evaluation, we apply simple augmen- tations to the few-shot examples including random translation, horizontal flips, additive Gaussian noise. brightness, and random eraser (Zhong et al. 2017). We use Lg regularization on all weights with a coefficient 5e-4.
meta-train for ~00 epochs through
5.1 DATASETS
The first few-shot image segmentation dataset was the PASCAL-5 presented in Shaban et al. (017)) which reimagines the PASCAL dataset (Everingham et al. 2010) as a few-shot binary segmentation problem for each of the classes in the original dataset. Unfortunately, the dataset contains relatively few distinct tasks (20 excluding background and unlabeled). The idea of a meta-learning dataset for image segmentation was further developed with the recently introduced FSS-1000 dataset, which contains 1000 classes, 240 of which are dedicated to the meta-test set Stest, with 10 image-mask pairs for each class Wei et al. 2019). For each of the rows in the results table 2 we evaluate the network on the 240 test tasks, sampling two random splits into training and testing sets for each task, yielding 480 data points per meta-learming approach for which the mean intersection over union (IoU) (eq. [14) and 95% confidence interval are reported. The FSS-1000 dataset is the focus of the empirical comparisons of network ablations and meta-learning approaches that we experiment with in this paper.
5.2 FP-K DATASET
For investigating how the meta-learned representations integrate new information as more data be- comes available, we put together small benchmark dataset that call FP-k. FP-k takes 5 tasks from FSS-1000 and 5 tasks from PASCAL-5' for the same concepf" Using this dataset, we train over range of "k"-training shots from ImageNet-trained initialization and our meta-learned ini- tializations. We report the performance of our EfficientLab network meta-trained with FOMAML
'Though extensive model experimentation was done on Swal, we report ablations on the meta-test set Stest in the results section for comparison to our best methods.
$ See the Appendix for more details on dataset construction.
'The encoder is trained on ImageNet, while the residual skip decoder and final layer weights are initialized using the Glorot uniform initialization (Glorot & Hengio 2010)

---

Under review as a conference paper at ICLR 2020
range of k examples benchmark which hope will inspire future empirical research into studying how meta-learning approaches scale accuracy and computational complexity as more labeled data become available. These results are show in Figure and discussed in 5.4
5.3 HYPERPARAMETER SELECTION
Generalization meta-leaman meta-learning requires both the ability to learn representations for new tasks ef- ficiently (Trr to Ttest) from few training examples, and select representations that able to capture unseen test examples effectively (Dir to Dtest). The approximation scheme of FOMAML addresses the latter by taking the finite difference between updates using the train and validation sets in 6 favoring initializations that differ less between splits of D.. To address sample efficiency for learning new tasks, we leverage the flexibility to choose hyperparameters Whr and Wtest separately.
Empirical estimations of the optimal initialization have an implicit dependence on Tú, and w (eq- 8). and the optimal hyperparameters w" depend on the initialization 6, in turn (eq. [TP. For this reason, after learning 6% we then fix 0z and tune Wtrain to find Wteat- Selecting a small learning rate in training Ckty leads to initializations 0: close to parameters with low loss 07 for tasks in Ter (see Nichol & Schulman (2018) for further discussion). But, because the hyperparameters wer that govern the dynamics of the update routine that is used to adapt to new tasks are not learned nor conditioned on D,, wer is not guaranteed to minimize the empirical risk even over the training tasks in S
In our experiments, after we have meta-learned an initialization 0,, we then applied the UHO algo- rithm to tune the update routine's 's hyperparameters on 100 randomly sampled tasks from the training set. We specifically search over the learning rate and the number of gradient updates that are applied when adapting to new task T.
5.4 RESULTS
We show the results of experimenting with different decoder architectures for EfficientLab in Table Each network topology is meta-trained with FOMAML and the same meta-training hyperparameters defined in
Network Architecture
loU
EfficientNet w/o decoder
75.66 + 1.01%
EfficientNet Auto-DeepLab decoder
73.054 + 01.09%
EfficientNet + RSD at Stage w/o residual
78.17 + 1.02%
EfficientNet RSD Stages & 6 EfficientNet + RSD at Stage 3
80. + 0.94%
80.60 + 0.93%
Table 1: EfficientL.ab architecture ablations. Each network is meta-trained in the way follow- ing and tested on the set of test tasks from FSS-1000 (Wei et al.) 2019). The 3rd row contains results of removing the short-range residual connection from our proposed RSD module. The final row is the best network we find for few-shot performance via model agnostic meta-learning. We call this network EfficientLab in reference the encoder of EfficientNet (Tan Le 2019) and the decoder of Auto-DeepLab (Chen et al. 2018), which it is inspired by.
The results of our model with an initialization meta-leamed using Reptile and FOMAML are shown in Table 2 We find that our model trained with FOMAML and importantly with regularization and improved use of batch normalization yields state of the art results. Given that previous works have used regularization minimally or not all during meta-training, we also conducted an abla- tion of removing regularization the model. We find, unsurprisingly. that the combination of an L2 loss on the weights, with simple augmentations, and a final layer of dropout does significantly increase generalization performance. After optimizing the update hyperparameters, sets the new state of the art for the FSS-1000 dataset. We have included a visualization of example predictions for small set of randomly sampled test tasks in A
approach
To address question in section we also searched through a range of update routine learning rates, o, that were 10x less to 10x greater than the learning rate used during meta-training. As

---

Under review as a conference paper at ICLR 2020
Method
ToU
FSS-I0D0 Baseline
80.12%
Method
ToU
Reptile
62.36 + 2.12%
FSS-1000 Baseline
73.47%
FOMAMI.
77.89 1.03%
FOMAML
75.19 = 01.28%
FOMAMI. regularization
80.60 = 0.93%
(a) FSS-1000 1-shot
FOMAML regularization + UHO
82.19 + 0.91%
(b) FSS- 1000 5-shot
Table 2: Mean IoU scores of the EfficientLab network evaluated on FSS- 1000 test set of tasks for I-shot and 5-shot learning. We model combined FOMAML, EfficientLab, regularization, and the UHO algorithm.
We report the FSS-1000 baseline from (Wei et al. 2019). Our best found
ES iterations
0.80
ElooUeee
0.75
0.70
10
0.65
0.01
0.02
0.04
0.05
Figure 2: Each point represents the mean IoU for the validation tasks with a sampled learning rate a. The blue dashed line indicates the learning rate used by SGD the inner loop during meta vetannrainnng. Points are colored by how many iterations they were trained before stopped by early stopping (ES) with patience of 3 iterations.
clearly shown Figure 2 the learned representations are not robust to such large variations in the hyperparameter.
In this work we posit that a fixed update procedure that is used at test time and not conditioned on task D,, T; is one of the major hinderances of apply MAML-type
the labeled examples for algorithms to unseen tasks. In section 3.2 we show that MAML-type algorithms do not natively optimal. We find this analysis be sup- ported empirically as well. We find that: (1) the estimated optimal hyperparameters for the update routine even on the training tasks are not the same as those specified priori during meta tecannrainnng, as illustrated in Figure 2 One may expect that MAML-type algorithms would converge to a point parameter space from which optimal minima for each of the training tasks are reachable. We find that even after 200 epochs through the training set, this was not the case. The best learning rate and number iterations we found via the random search UHO algorithm were 0.007475 and 8, respec- tively, compared to learning rate of 0.005 and 5 iterations used during training. (2) Optimizing the hyperparameters (even on the set of training tasks 5*") after meta-training improves test-time results unseen tasks. Furthermore, the UHO-selected hyperparameters learning rate 0.007475 and inner-iterations 8 yields nearly identical results to meta-training with the initial hyper parameters learning rate 0.005 and inner-
guarantee that the update routine's hyperparameters
find that meta-training from scratch and evaluating with
8

---

Under review as a conference paper at ICLR 2020
Initialization
k
IoU
ImageNet
16.76
ImageNet
5
5.009
08
ImageNet
10
27.08 5.58%
ImageNet
50
31.37
0.5
ImageNet
100
3.699
ImageNet
200
47.50
mou
04
ImageNet
400
7.15%
Meta-learned
1
38.30 9.42%
03
Meta-learned
5
4
Meta-learned
10
43.94+9.86%
Meta-learned
50
8..12+
infielization
meta- leamned
Meta-learned
100
5..07
imagenet
Meta-learned
200
55.55
7.73%
0.1
Meta-leamed
400
58.68
7.48%
K
Figure 3: Mean IoU results as function of the training set size of our EfficientLab model adapted to tasks of the FP-k dataset. Error bars represent 95% confidence intervals. The meta-learned ini- tialization utilizes the top performing learning algorithm in Table The meta-learned initialization outperformed Efficientl .ab initialized with an ImageNet-trained encoder and a randomly initialized decoder for all numbers of labeled training examples that we evaluated. For additional experimental details see Clnn the appendix.
iterations = 5. This further suggests that it may useful to tune the hyperparameters w to improve the generalization performance of the gradient-bused adaptation routine U.
By training our model on the FP-k dataset, we also found that our meta-learned initializations out- performed ImageNet-trained encoder and randomly initialized decoder for up 400 training examples
DISCUSSION
In this work, we showed that gradient-based first order model agnostic meta-learning algorithms do in fact extend the high dimensionality of the hypothesis spaces and the skewed distributions of few-shot image segmentation problems, but are sensitive to the hyperparameters, routine, U. Furthermore, we find that the representations that are meta-learned are valuable more data becomes available, unifying few- and many-shot regimes. We have also presented novel neural network architecture, Efficientl.ab, for semantic image segmentation.
of the update
Future work should investigate more critically, both empirically and theoretically, the efficacy of few-shot learning systems as more labeled data becomes available. To this end, we have reported baseline results small meta-test benchmark dataset, FP-k, which contains 5 tasks with 400 train- ing and test examples per task. Additionally, future work could investigate more deeply learned update procedures, forms of meta-regularrizttion, and second order methods for image segmentation. would also be useful future work to take a more critical look at the interplay between batch nor- malization and meta-learning. While single task deep neural networks in large data regimes apply batch normalization with a consistent pattern, different groups working in few-shot meta-learning have incorporated batch norm in completely different ways such as by: (1) not using it at all for the meta-lcarning components (Rusu et al. 2018)) (2) not using learned 3 and 7 parameters at all while still using estimated population means and variances during inference ZZhang al. (2019). or (3) meta-learning 3 and ry while only using batch statistics for the normalization (Finn et al. 2017
'The examples in the PASCAL dataset known be more challenging than the FSS-1000 dataset Wei al. 2019). From visual inspection of the two datasets, also clear that the PASCAL dataset contains more label noise than the FSS-1000 dataset. For these reasons, the mean IoU values shown in Figure 3 which contain examples from both datasets, are not directly comparable to the results shown in Table which contain examples only from FSS- 1000. Furthermore, discussed in the appendix in C did not tune w in these experiments.

---

Under review as a conference paper at ICLR 2020
Nichol & Schulman 2018), (4) meta-learning 3 and y and also using population estimates of the mean and variance, as done conventionally when training deep neural networks in the large data regime, which is the approach that adopt and find to be most useful. Finally, another interesting question to address would be to evaluate how Efficientl .ab performs on more standard many-shot multi-class image segmentation problems such as the CityScapes dataset (Cordts et al.. 2016).
In conclusion, we have shown in this work that the optimal hyperparameter configuration for the update routine may not be the same configuration used during meta-learning. These findings are supported by our theoretical analyses which show that MAML-type algorithms minimize the em- pirical risk on the training set of a fixed update routine and the initialization 6., but do not natively guarantee that the update routine's 's hyperparameters are optimal. We suspect that improvements realized by relation networks (Wei al.) [019 Zhang et al. 2019 Rusu et af.. 2018), models that learn to generate parameters conditioned on the training data (Rusu et al. 2018 S-Saaan et al. 2017), and models with learned learning rates (Li al. 2017 Antoniou et al. 2018) directly leverage in-
formation on how to adapt given few-shot sample of labeled examples. We also suspect that the previous work in Mishra et al. (2018) may have found MAML-type algorithms to overfit when ap- plied to high dimensional parameter spaces due to lack of regularization and lack of an empirical risk minimization of the update routine's 's hyperparamters. It is our hope that our empirical analyses and formalization of the generalization error of meta-learning systems lead better explanations of why some meta-learning systems work better than others in different problem spaces. Lastly, we hope that this work draws, what we argue is necessary, attention to the open problem of building learning systems that can unify small and large data regimes by gaining expertise and integrating new information as more data becomes available, much as people do.
REFERENCES
Antreas Antoniou. Harrison Edwards, and Amos Storkey. How train your maml. arXiv preprint arXiv:1810.09500 2018.
Jimmy Ba, Geoffrey E. Hinton, Volodymyr Mnih, Joel Z. Leibo, and Catalin lonescu. Using fast weights to attend to the recent past. In Advances in Neural Information Processing Systems, pp. 4331 1339,4433,,44 2016.
Liang- Chichh Chen, Yukun Zhu, George Papandreou. Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248-255. IEEE, 2009.
Lee R Dice. Measures of the amount of ecologic association between species. Ecology, 26(3): 297-302, 1945.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journaa of computer vision, 88(2): 303-338, 2010.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv: 700.00000000 2017.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256, 2010.
Geoffrey E Hinton and David C Plaut. Using fast weights to deblur old memories. In Proceedings of the ninth annual conference of the Cognitive Science Society, pp. 177-186, 1987.
10

---

Under review as a conference paper at ICLR 2020
Vladimir Iglovikov, Sergey Mushinskiy, and Vladimir Osin. Satellite imagery feature detection ing deep convolutional neural network: A kaggle competition. arXiv preprint orXiv:1706.0616, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Artificial Intelligence and Statistics, pp. 240-248, 2016.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In Advances in Neural Information Processing Systems, pp. 6389-6399, 2018.
Xiang Li, Shuo Chen, Xiaolin Hu, and Jian Yang. Understanding the disharmony between dropout and batch normalization by variance shift. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2682-2690, 2019.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few- shot learning. arXiv preprint arXiv: 170.00983, 2017.
Chenxi Liu, Liang-Chie Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 82-92, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. simple neural attentive leta-learner. In International Conference on Learning Representations, 2018. URL https:
//openreview.....m nwt/forruit weammid=Blliai 11122
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999, 2018.
Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alyosha Efros, and Sergey Levine. Conditional networks for few-shot semantic segmentation, 2018. URL httpps::/ppeereevve net.
[oormm1d=skMjj
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960 05960, 2018.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chin Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition, pp. 4510-4520, 2018
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski. Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pp. 4967-4976, 2017.
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.0344 2017.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 1946, 2019.
Sebastian Thrun. Is learning the n-th thing any easier than learning the first? In Advances in neural information processing systems, pp. 640-646, 1996.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.

---

Under review as a conference paper at ICLR 2020
Tianhan Wei, Xiang Li, Yau Pun Chen, Yu- Wing Tai, and Chi-Keung Tang. Fss-1000: A 1000-class dataset for few-shot segmentation. arXiv preprint arXiv:1907113447 2019.
Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Class-agnostic segmen- tation networks with iterative refinement and attentive few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5217-5226, 2019.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. arXiv preprint arXiv: 04896, 2017.
A EXAMPLE PREDICTIONS
We have included here visualization of small random sample of predictions test examples Drest from test tasks Stest that were never seen during meta-training.
iPhone
Figure 4:
Randomly sampled example 5-shot predictions on the test images from test tasks. Positive class prediction is overlaid in red. From left to right, top to bottom, the classes are apple icon, austral wtw.iinnneeiiirre terrier, church, motorbike, flying frog, flying_snaken hover board, porcupine
B
FP-K DATASET
Table contains the five tasks in PASCAL-5' that have direct analogs in FSS-1000. Each row contains the name of task in FSS-1000 and PASCAL-5', respectively. We combine all examples for synonymous tasks. During evaluation, we simply randomly sample 20 test examples, and sample training set of & examples the range: [1, 5, 10, 50, 100, 200, 400). For more details our training and evaluation procedures see
PASCAL-5 Task
FSS-1000 Task
aeroplane bus motorbike potted_plant
airliner
bus motorbike potted_pland television
tvmonitor
Table 3: PASCAL-5 tasks with FSS-1000 analog.
12

---

Under review as a conference paper at ICLR 2020
C FP-K EXPERIMENTAL DETAILS
In this section, we describe our testing protocol and simple hyperparameter choices when training the ImageNet and randomly initialized and the FOMAML FOMAMLinitialized EflicientLab network. For each tuple of (initialization, k-training shots) we randomly sample 20 examples for a test set, Drest for the task and train on labeled examples Day. We repeat this random sampling and training process times for each of the 5 tasks, yielding 20 evaluation samples per (initialization, k- k-training shots) tu- ple. For training both networks on the available labeled examples, we simply use the number of itera- tions [1, 5, 10, 25, 50, 100, 200] for each k in the set of k-shot tasks [1, 5, 10, 50, 100, 200, 400), respectively. We acknowledge that it is likely possible that both networks could achieve better per- formance if their update routine's hyperparameters are empirically optimized or conditioned on the k examples in Dtr.
D HYPERPARAMETER TUNING METHODOLOOGY
We apply the insight discussed in section 3.2 that the loss of a meta-learning algorithm with a fixed initialization 9 is function of the update routine U hyperparameters w to develop simple update hyperparameter optimization (UHO) algorithm. This algorithm is outlined pseudocode in The routine samples n. hyperparameter values within predefined, broad range, evaluates each of the hyperparameters on a sample of tasks, Toat- After evaluating all Ti in Teat, UHO defines a range around the 2% best hyperparameter configurations and samples from that space, repeating until predefined computational budget is exhausted or the expectation of the loss is no longer reduced. Finally, the best configuration of the hyperparameters that was seen returned. This algorithm can be a viewed as variant of Sequential Halving (Jamieson & Talwalkar 2016).
Because the effects of the learning rate are intertwined with the number of gradient updates, we also leverage early stopping (ES) to decrease runtime and to estimate the optimal number of gradient steps, j when adapting to new task. The use of early stopping in this way is purely an implemen- tation optimization that reduces the search space that is explored when tuning the hyperparameters w. We could have simply randomly sampled the number of iterations, but this would have lead to many wasted tests of combinations learning rates and gradient steps such as when both the learn- ing rate and the number of gradient updates would be high. We use patience of 3 steps when using ES, meaning that the mean IoU on Drest did not improve for 3 gradient updates, we stop running SGD and, in actual code, also return the number of updates with the best performance Diest on line of the UHO pseudocode.
E FSS-1000 TEST TASKS
The set of test tasks that the meta-learning systems are evaluated on in this paper are available the FSS-1000 github repository https///ghhi.cccm////SSScssssss/
13

---

Under review as a conference paper at ICLR 2020
Algorithm Update hyperparameter optimization (UHO) via random search and successive de- crease in search space. Returns the estimated optimal configuration of hyperparameters w", such as the learning rate a' and number of gradient steps j", that minimize the empirical loss p(T)
Require: 0,: an initialization
Require: f: model parameterized by 0
Require: p(T)): distribution over tasks
Require: U: an update routine with hyperparameters
Require: S: set of ranges for hyperparameters in w to search through
Require: æ: floating point value in (0, 1) decrease the search ranges by
Require: b: the number of iterations to refine the search space, which defines a computational
budget
Require: the number of configurations to sample per refinement
Initialize list of metries, metrics
2: Initialize list of sampled hyperparameter configurations, parameters
3: for = 0 to do
4: for ( to do
5: WP f sample(S)
6: metric, j + evaluate(U W.
Evaluatethe expectation of the loss over p. returning loss and best seen number gradient steps from ES. netricss.appen(metrii.
7:
8:
W. + parameteers.append(u.
> Add number of steps returned by ES to the set
9:
10: end for
11: St bt=bbetxx.confiii so-betttt.oo metrics, parameters)
b Get the x% of the best
hyperpurameters configurations identified by metrics.
12: end for
13: return argmar(metrics)
14


Reviews:
[{"review_id": "HJgcpxe1qH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to apply MAML-style meta-learning to few-shot semantic segmentation in images. It argues that this type of algorithm may be more computationally-efficient than existing methods and may offer better performance with a higher number of examples. They further propose to perform hyper-parameter search to choose a new learning rate for the inner learning process after optimizing for the network parameters.\n\nAs far as I know, this is the first paper to apply gradient-based (i.e. MAML-style) meta-learning to this specific problem. Existing approaches to few-shot semantic segmentation have mostly used multi-branch conv-networks to condition the output on the training examples. This paper shows that (FO)MAML achieves similar accuracy to the FSS-1000 baseline. This is an empirical contribution in itself. The paper also demonstrates that the EfficientNet architecture can be applied to segmentation.\n\nMajor concerns:\n\n(1.1) The improvement obtained by the hyper-parameter optimization seems quite marginal (79.0 - 81.4 and 73.3 - 73.9) and there is no study of the variance of the results. The fact that better performance is obtained by tuning the learning rate on the *training* set suggests to me that the improvement might not be significant. You could repeat the experiment by sampling multiple different training and testing sets (with different classes) to estimate (some of) the variance.\n\n(1.2) The formalization in Section 4 is mathematically appealing but seems unnecessary. In the end, the paper is essentially arguing that it's better to use different learning rates (for the inner loop) during meta-training and meta-testing. This seems obvious, since this includes equal learning rates as a special case. The paper proposes to optimize the latter learning rate using a gradient-free method. This argument can be made without considering generalization bounds.\n\n(1.3) One of the central claims of the paper is that \"meta-learned representations smoothly transition as more data becomes available.\" It's not entirely clear what this means. I suppose it means that, with few shots, it should perform as well as existing few-shot methods, but with many shots, it should perform as well as a standard learning algorithm. The paper failed to present any evidence that existing algorithms for few-shot segmentation do not satisfy this property. It would strengthen the argument to include an existing few-shot segmentation algorithm in Figure 2.\n\n(1.4) The details of the experiment in Figure 2 are not clear. Was a different number of iterations used when there are hundreds of examples? Were different hyper-parameters used when optimizing from a pre-trained checkpoint? It would be unfair to use the same hyper-parameters which had been optimized specifically for the meta-learned initialization.\n\nOther issues:\n\n(2.1) It's not clear what it means to achieve human-level performance in the few-shot task and the many-shot task. What is human-level performance at few-shot segmentation? It seems to me that humans are capable of segmenting novel objects (i.e. zero-shot) with almost perfect accuracy. Does this mean that your method should achieve the same accuracy with few- and many-shots?\n\n(2.2) The use of early stopping was unclear. Do you use a fixed number of SGD iterations during training and a variable number of iterations (determined by a stopping criterion) during testing? However, this seems to be contradicted by the statement that the UHO algorithm determined an optimal number of iterations (8) for testing? On the other hand, this seems like too few iterations with hundreds of shots. Maybe the automatic stopping criterion was only used with many shots? Or maybe training proceeds until either the stopping criterion is satisfied or the maximum number of iterations is reached? Furthermore, the early stopping criterion was not specified.\n\n(2.3) Missing reference: Meta-SGD (arxiv 2017) considers a different learning rate for every parameter and updates the learning rates during meta-training .\n\n(2.4) There was no discussion of the running time of different methods. This would be particularly interesting in the many-shot regime. How slow are the RelationNet approaches?\n\n(2.5) It is claimed that Figure 1 demonstrates that  \"the estimated optimal hyperparameters for the update routine ... are not the same as those specified a priori during meta-training\". However, it seems that the optimal learning rate is awfully close to the dotted blue line (within the variance of the results).\n\n(2.6) For the IOU loss (equation 10), what are the predicted y values? Are they arbitrary real numbers? Do you use a sigmoid to constrain them to real numbers in [0, 1]?\n\nMinor:\n\n(3.1) It is not worth stating the optimal learning rate to more than 3 or 4 significant figures.\n(3.2) Use 1 \\times 1 instead of 1 x 1.\n(3.3) Is there a reference for the Dice score? Where does the name come from?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1x8O66AFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary - The paper first makes the observation that training algorithms and architectures for meta-learning have become increasingly specific to the few-shot set of tasks. Following this, the authors first investigate if it’s possible to learn good initializations for dense structured prediction tasks such as segmentation (across arbitrary amounts of available input data). Concretely, the claimed contributions of the paper include -- (1) extension and analysis of first order MAML like approaches to image segmentation; (2) using a formalized notion of the generalization error of episodic meta-learning approaches to decrease error on unseen tasks; (3) doing this via a novel neural network parametrically efficient segmentation architecture and (4) empirically comparing meta-learned initializations with ImageNet pre-trained initializations with increasing training set sizes.\n\nStrengths\n\n- Apart from the flaws mentioned under weaknesses, the paper is generally easy to follow. While it’s somewhat hard to understand the motivations and the concrete contributions made by the paper, sections are more-or-less well-written.\nUsing the proposed hyper-parameter search scheme over first-order MAML approaches demonstrates improvements over not baselines which do not use the same and baselines which do not involve meta-learning.\n\nWeaknesses\n\nThe paper has some major weaknesses that affect the clarity of the points being conveyed in several sections. These weaknesses form the basis of my rating and addressing these would not only help in adjusting the same but would also help in improving the current version of the paper significantly. Highlighting these below:\n\n- The paper claims to make several contributions but it’s hard to concretely understand them in several sections. For instance, the abstract mentions -- ‘’A natural question that ….. human level performance in both.” The statement is slightly unclear to me -- is the intended sentiment the fact that the goal should be to develop a single algorithm that works well for both few-shot and many-shot settings? If so, why should that be the case? Essentially, what is the limiting factor being identified that restricts few-shot approaches from performing well in many-shot settings? Maybe the statement could be framed better but in it’s current form it’s unclear what is being conveyed. When this is mentioned again in the introductory section, it is followed by a statement indicating that meta-learning an initialization is one solution. Why is this surprising? Maybe I’m mis-understanding the motivation behind the claim. Could the authors clarify this? \n\n- Similarly, it’s unclear what question (3) in the introduction is trying to address. Which “data” (training / testing set of tasks) is the fixed update policy not conditioned on? Could the authors clarify this?\n\n- The description of the single update hyper-parameter optimization (UHO) is hard to understand in Sec. 4. -- specifically the text surrounding eqns (5) and (6). The transition from Eqn (5) -> Eqn (6) is unclear. Could the authors clarify this clearly? This section is further referred to in subsequent sections as a supporting basis for some of the obtained results (specifically, the last para on page 6)\n\nReasons for rating\n\nI found certain sections of the paper particularly hard to understand and interpret. I would encourage the authors to address these more clearly in the responses. The highlighted strengths and weaknesses of my rating and addressing those clearly would help in improving my current rating of the paper.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgryYKpFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper examines the performance of MAML, FOMAML, and Reptile gradient based meta-learning algorithms on the task of semantic image segmentation. The paper proposes to do black box optimization (successive halving) on the hyperparameters of the inner loop of the gradient meta-learners for improved performance. The paper proposes some modification on the segmentation model architecture (no ablation study presented). Finally, it is shown that pre-training using meta-learning on similar segmentation tasks works better then just using ImageNet based model pre-training.\n\nIn its current form I suggest to reject the paper and urge the authors to improve it according to the following points:\n\n1. In parts (specifically the intro and some other earlier parts) the paper is very well written, but the later parts, the description of the model modifications (did you consider to add an architecture diagram?), the details of the experiments, the punch-line of the theory development that has been attempted, etc are not very clear and hard to follow. I suggest the authors to improve the readability of these parts, add some helpful / motivating diagrams and examples (perhaps some qualitative results too?), state more clearly what is used for meta-training? how it is made sure that meta-testing is done on a separate set of categories? (I did not see this split in the Appendix) and etc\n\n2. My main concern is novelty. As it stands, the current novelty proposition is: black-box optimization of LR and number of iterations in MAML style meta learning (hardly novel), architecture modifications (no ablation study if these help or not), small improvement on FSS-1000 5-shot test (what about other shots? still not sure about the splits), and showing meta-training on similar tasks is better then not doing it (that is using ImageNet pre-trained backbone for init) - again hardly a novel insight. For the last point, saying that meta-learned model was initialized from scratch does not cut it, as it was meta-trained on massive data that is more related to the test tasks then the ImageNet.\n\nI suggest the authors to mainly focus on 2, although making the writing clearer and better is also very important for a high quality paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers reached a consensus that the paper was not ready to be accepted in its current form. The main concerns were in regard to clarity, relatively limited novelty, and a relatively unsatisfying experimental evaluation. Although some of the clarity concerns were addressed during the response period, the other issues still remained, and the reviewers generally agreed that the paper should be rejected.
Decision: Reject