Dense Gaussian Processes for Few-Shot Segmentation
Authors: Joakim Johnander
Johan Edstedt
Michael Felsberg
Fahad Khan
Martin Danelljan
Venue: ICLR 2022 Submitted (2022)
Keywords: few-shot learning
few-shot segmentation
segmentation
gaussian processes

Abstract: Few-shot segmentation is a challenging dense prediction task, which entails segmenting a novel query image given only a small annotated support set. The key problem is thus to design a method that aggregates detailed information from the support set, while being robust to large variations in appearance and context. To this end, we propose a few-shot segmentation method based on dense Gaussian process (GP) regression. Given the support set, our dense GP learns the mapping from local deep image features to mask values, capable of capturing complex appearance distributions. Furthermore, it provides a principled means of capturing uncertainty, which serves as another powerful cue for the final segmentation, obtained by a CNN decoder. Instead of a one-dimensional mask output, we further exploit the end-to-end learning capabilities of our approach to learn a high-dimensional output space for the GP. Our approach sets a new state-of-the-art for both 1-shot and 5-shot FSS on the PASCAL-5$^i$ and COCO-20$^i$ benchmarks, achieving an absolute gain of $+14.9$ mIoU in the COCO-20$^i$ 5-shot setting. Furthermore, the segmentation quality of our approach scales gracefully when increasing the support set size, while achieving robust cross-dataset transfer. 

Under review as a conference paper at ICLR 2022
DENSE GAUSSIAN PROCESSES FOR FEW-SHOT SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Few-shot segmentation is challenging dense prediction task, which entails seg- menting novel query image given only small annotated support set. The key problem is thus to design method that aggregates detailed information from the support set, while being robust to large variations in appearance and context. To this end, we propose a few-shot segmentation method based on dense Gaussian process (GP) regression. Given the support set, our dense GP learns the map- ping from local deep image features to mask values, capable of capturing com- plex appearance distributions. Furthermore, it provides a principled means capturing uncertainty, which serves another powerful cue for the final segmen- tation, obtained by CNN decoder. Instead of a one-dimensional mask output, we further exploit the end-to-end learning capabilities of our approach to learn high-ding nigh-dimensional output space for the GP. Our approach sets new state-of-the arr for both 1-shot and 5-shot FSS on the PASCAL-5' and COCO-20' benchmarks. achieving an absolute gain of +14.9 mloU in the COCO-20' 5-shot setting. Fur- thermore, the segmentation quality of our approach scales gracefully when in- creasing the support set size, while achieving robust cross-dataset transfer.
INTRODUCTION
Image few-shot segmentation (FSS) of semantic classes (haban et al. 2017) has received increased attention in recent years. The aim is to segment novel query images based on only a handful anno- tated training samples, usually referred to as the support set. The FSS method thus needs to extract information from the support set in order to accurately segment a given query image. The problem is highly challenging, since the query image may present radically different views, contexts, scenes, and objects than what is represented in the support set.
The core component in any FSS framework is the mechanism that extracts information from the support set to guide the segmentation of the query image. However, the design of this module presents several challenges. First, it needs to aggregate detailed yet generalizable information from the support set, which requires a flexible representation. Second, the FSS method should effectively leverage larger support sets, achieving scalable segmentation performance when increasing its size. While perhaps trivial at first glance, this has proved to be major obstacle for many state-of-the-aan methods, visualized in Fig. Third, the method is bound be queried with appearances not included in the support set. To achieve robust predictions even in such common cases, the method needs to assess the relevance of the information in the support images in order to gracefully revert to e.g. learned segmentation priors when necessary.
We address the aforementioned challenges by densely aggregating information in the support set using Gaussian Processes (GPs). Specifically, we use a GP to learn mapping between dense local deep feature vectors and their corresponding mask values. The mask values are assumed to have jointly Gaussian distribution with covariance based on the similarity between the corresponding feature vectors. This permits us to extract detailed relations from the support set, with the capability of modeling complex, non-linear mappings. As non con-parametric model, the GP further effectively benefits from additional support samples, since all given data is retained. As shown in Fig. D the segmentation accuracy of our approach improves consistently with the number of support samples. Lastly, the predictive covariance from the GP provides a principled measure of the uncertainty based on the similarity with local features in the support set.

---

Under review as a conference paper at ICLR 2022
PASCAL-5'
COCO-20
60
Segnentation Performauce (mloU)
75-
55-
654g
60
35
68-
30
5
Number of Support Exsmples
DCPNet (Ours)
RePri (CVPR 2021)
ASCNer (CVPR 2021)
SAGNN (CVPR 2021)
PFENet (TPAMI 2020;
DENer (ACM MM 2020)
SCL (CVPR 2021)
Figure I: Performance of the proposed DGPNet approach on the PASCAL-5' and COCO-20' bench- marks, compared to the state-of-the-art. We plot the mIoU (higher is better) for different number of support samples. For our approach, Our GP-based method effectively leverages larger support sets, achieving substantial improvements in segmentation accuracy. Our method also excels in the extreme one-shot case, even outperforming all previously reported results on COCO-20' for any support size.
show the mean and standard deviation over 5 experiments.
Our FSS approach is learned end-to-end through episodic training, treating the GP as layer in a neural network. This further enables us to learn the output space of the GP. To this end, we encode the given support masks with neural network in order to achieve mulliidimeeniionll output representation. In order to generate the final masks, our decoder module employs the predicted mean query encodings, together with the covariance information. Our decoder is thus capable of reasoning about the uncertainty when fusing the predicted mask encodings with learned segmentation priors. Lastly, we further improve our FSS method by integrating dense GPs multiple scales.
We perform comprehensive experiments on two benchmarks: PASCAL-5' (Shaban et al. 2017 and COCO-20' (Nguyen & Todorovic 2011). Our proposed DGPNet outperforms existing methods for 1-shot and 5-shot by a large margin, setting a new state-of-the-ar on both benchmarks. When using the ResNet101 backbone, our DGPNet achieves an absolute gain of 14.9 for 5-shot segmentation on the challenging COCO-20' benchmark, compared to the best reported results in the literature. We further demonstrate the cross-dataset transfer capabilities of our DGPNet approach from COCO-204 to PASCAL and perform detailed ablative studies to probe the effectiveness of our contributions.
2 RELATED WORK
Few-Shot Segmentation The earliest work in few-shot segmentation (FSS), by Shaban et al. 2017) proposed a method for predicting the weights of a linear classifier based on the support set, which was further built upon in later works (Siam et al.] 2019 Liu et al.) 2020a; Boudial et al. 202T) Instead of learning the classifier directly, Rakelly al. (2018) proposed construct a global con- ditioning prototype from the support set and concatenate it to the query representation, with several
subsequent works (Dong & Xing 2019; Zhang al. 2020 Wang al. 2019 Nguyen Todor- lovic 2019: Zhang et al. 2019b] Liu et al., 2020c; Azad et al. [021 Liu et al. 2020b] Xie et al.]
2021 Wang et al. 2021p major limitation of these methods is the unimodality assumption. To alleviate this problem. Zhang et al. (202T) construct additional prototypes by a self-guided module, while Yang et al. (2200a) Liu et al. (2020c): Li et al. (202T) instead cluster multiple prototypes to create a richer representation. However, clustering introduces extra hyperparameters, such as the number of clusters, as well as optimization difficulties. In contrast, our method is not restricted in any such sense, and only requires to choose appropriate kernel. Some recent works consider pointwise correspondences between the support and query set. These works have mostly focused on
attention or attention-like mechanisms (Zhag al. 2019a Yang et al. 2020bb Hu et al. 2019 Tian
2

---

Under review as a conference paper at ICLR 2022
Suppoct images
Madk cnoeder
U
Decoder
ys
plye No. Xs. Xs.
image enooder
Denie GP
X.S
ispux-cutuu cutput function
E
Var(yo)
Query image
Query prediction
X
X @,shallow
Figure 2: Overview of our approach. Support and query images are fed through an encoder to produce deep features and x respectively. The support masks are fed through another encoder to produce y. Using Gaussian process regression, we infer the probability distribution the query mask encodings y2 given the support set and the query features (see equations 7 a representation of this distribution and feed it through a decoder. The decoder then predicts segmentation the original resolution.
We create
al. 2020 Wang et al. 2020p. In contrast with these methods, construct principled posterior over functions, which greatly aids the decoder.
Combining GPs and Neural Networks While early work focused on combining GPs and neural networks in the standard supervised classification setting (Salakhutdinov & Hinton 2009 Wilson
at. 2016 Calandra al. 2016), there has recently been an increased interest in utilizing Gaussian processes in the context of few-shot classification (Patacchiola et al., 20200) [nell & Zemel 2021
Previous works employ the GP in the classification setting and as the final output layer work, and optimize proxies of either the predictive or marginal log likelihood directly. Note that the classification likelihood is non-gaussian, and hence computing the exact posterior and marginal likelihoods of the model becomes intractable. Here, go beyond this limitation and propose internal dense GP model of the support features, where the posterior predictive distribution used input to CNN decoder. Moreover, this allows to learn the output space of the GP further increase its expressive power. To the best of our knowledge, we are the first to introduce a dense GP approach for the challenging dense prediction task of few-shot segmentation.
net-
METHOD
3.1 FEW-SHOT SEGMENTATION
Few-shot segmentation is a dense few-shot learning task (Shaban et al. 2017). The aim to learn segment objects from novel classes, given only small set of annotated images. single instance of this problem, referred to as an episode, comprises small set of annotated samples, called the support set, and set of samples on which prediction is be made, the query set. Formally, we denote the support set as {(Tsk, Msx) k= comprising K image-mask pairs Isk E R Ho Wax3 and w, and the aim is to predict its Ms: E {0. 1} 1 x Wo, A query image is denoted as Io E R. xo corresponding segmentation mask Mg (0, Mo xWo
To develop our approach, we first provide general formulation for addressing the FSS problem, which applies to several recent methods, including prototype-based et. 2021 Liu et al.. 2020a) and correlation-bast three steps: feature extraction, few-shot learning, and prediction. the first step, deep features are extracted from the given images,
(Tian et al. 2020) Wang et al. (2200) ones. Our formulation proceeds in
x F(I) HXXxX
These features provide a more disentangled and invariant representation, which greatly aids the problem of learning from limited number of samples.
The main challenge in FSS, namely how to most effectively leverage the annotated support samples, encapsulated in the second step. As general approach, consider learner module A that

---

Under review as a conference paper at ICLR 2022
employs the support set in order to find function f, which associates each query feature zQ R with an outpur v2 E R. The goal is to achieve an output ye that is strongly correlated with the ground-truth query mask, allowing it act as strong in the final prediction. Formally, express this general formulation as,
f f= A(fxsk Ms) ve = f(ze)
(2)
The learner A aggregates information in the support set {xSK. Ms h in order to predict the function In the final step of our formulation,
f. This function is then applied to query features equation 2 output from the function f on the query set is finally decoded by a separate network as Me = oy xo) predict the segmentation Mo.
The general formulation in equation Dencapsulates several recent approaches for few-shot segmen- tation. In particular, prototype-baie based methods, for instance PANet (Wang al. 2019). , are retrieved by letting represent maoklno operation. The function f thus computes the cosine- similaii between the pooled feature vector and the input query features. In general, the design of the learner represents the central problem in few-shot segmentation, since it is the module that extracts infor- mation from the support set. Next, distinguish three key desirable properties of this module.
3.2 MOTIVATION
As discussed above, the core component in few-shot segmentation is the few-shot learner A. Much research effort has therefore been diverted into its design (Nguyen & Todorovic, 2019; Liu et al.)
2020a Wang et al.) 2020: Liu et al., 2020c; Yang et al. 2020a Li et al.) 2021]. To motivate our approach, first identify three important properties that the learner should possess.
Flexibility of f The intent in few-shot segmentation is to be able to segment a wide range of classes, unseen during training. The image feature distributions of different unseen classes are not necessarily linearly separable (Allen et al., 2019). Prototypical few-shot learners, which are essentially linear classifiers, would fail in such scenarios. Instead, we need mechanism that can learn and represent more complex functions f.
Scalability in support set size K An FSS method should be able effectively leverage additional support samples and therefore achieve substantially better accuracy and robustness for larger support sets. However, many prior works show little to no benefit in the 5-shot setting compared 1-shot. As shown by Li et al. [02T) and Boudial et al. (022]) D,, it is crucial that new information is effectively incorporated into the model without averaging out useful cues.
Since only a small number of support samples are available in FSS, the network needs to regularly handle unseen appearances in the query images. For instance, the query may include novel backgrounds, scenes, and objects. Since the function f predicted by the learner is not expected to generalize to such unseen scenarios, the network should instead utilize neighboring predictions or learned priors. However, this can only be achieved if f models and communicates the uncertainty of its prediction to the decoder.
Uncertainty Modeling
3.3 DENSE GAUSSIAN PROCESS FEW-SHOT LEARNER
In this work, we propose a dense Gaussian Process (GP) learner for few-shot segmentation. As few-shot learning component, the GP possesses all three properties identified in the previous section. It can represent flexible and highly non-linear functions by selecting an appropriate kernel. As a non- parametric model, further effectively benefits from additional support samples since all given data is retained. The GP explicitly models the uncertainty in the prediction by learning probabilistic distribution over space of functions. Furthermore, both the learning and inference steps of the GP differentiable functions. While computational cost is well-known challenge when deploying GPs, we demonstrate that even simple strategies can be used to keep the number of training samples tractable level for the FSS problem.
Our dense GP learner predicts a distribution of functions f from the input features x to an output y. Let xs IR HW xD denote the matrix of all support image features and xQ E R #WXD D the matrix retaining the query features. Let ys and yo be the corresponding outputs. The former is obtained from the support masks and the latter is to be predicted with the GP. The key assumption
4

---

Under review as a conference paper at ICLR 2022
in the Gaussian process is that the support and query outputs ys. ye are jointly Gaussian according to,
ys y2
~ N
((H)
Kss K&q
kse) Kee
(3)
(1
For simplicity, set the output prior means, μs and μQ, zero. The covariance matrix K in equation 3 is defined by the input features at those points and a kernel x R x R + R. In our
experiments, adopt the commonly used squared exponential (SE) kernel
1 2F2 1
x(x") ,,") oj exp(-
x²1
(4)
with scale parameter of and length parameter The kernel can be viewed as a similarity measure. If two features are similar, then the corresponding outputs are correlated.
Next, the posterior probability distribution of the query outputs is inferred. The rules for condition- ing joint Gaussian gives us (Rasmussen & Williams
2006
yelys, Xs, XQ ~ N(H Ees)
(5)
where
HoIs K5g [sss + to o ys
(6)
EΦis = Kee - (Kss + o1
(7)
The measurements are assumed to have been obtained with some additive ii.d. Gaussian noise with variance of This corresponds to adding scaled identity matrix o to the support covariance matrix Kss. The equations 4 7 thus predict the distribution of the query outputs, represented by the mean value and the covariance. Note that the asymptotic complexity is ((KWH)))) We therefore need to work on sufficiently low resolution. Next, we introduce a way to incorporate more information into the predicted outputs while maintaining a low resolution.
3.4 LEARNING THE GP OUTPUT SPACE
The decoder strives to transform the query outputs predicted by the few-shot learner into accurate mask. This is a challenging task given the low resolution and the desire to generalize to classes unseen during offline training. We therefore explore whether additional information can be encoded into the outputs, yQ. in order to guide the decoder. This information could for instance include the shape of the mask which of the object's parts is at given location. To this end, we train mask-encoder to construct the outputs,
Ysk = G(Msk)
(8)
Here, G is neural network with learnable parameters.
The aforementioned formulation allows us to learn the GP output space during the meta-training stage. To the best of our knowledge, this has not previously been explored in the context of GPs. There is some reminiscence to the attention mechanism in transformers (Vaswani et al., 2017). The transformer queries, keys, and values correspond to our query features xQ. support features x and support outputs ys. A difference is that in few-shot segmentation, there is a distinction between the features used for matching and the output- the features stem from the image whereas the output stems from the mask.
As mask encoder G. we employ a lightweight residual convolutional neural network. The net- work predicts multi-dimensional outputs for the support masks that are then reshaped into ys e RKHW xE These are fed into equation 6 when the posterior probability distribution of the query outputs is inferred. The two matrix-vector multiplications are transformed into matrix-matrix mul- tiplications and the result Pois E IR HWW x E is a matrix containing the multi-dimensional mean. The covariance in equation is kept unchanged. This setup corresponds to the assumption that the co- variance is isotropic over the output feature channels and that the different output feature channels independent.
5

---

Under review as a conference paper at ICLR 2022
3.5 FINAL MASK PREDICTION
Next, we decode the predicted distributions of the query outputs in order to predict the final mask. Since the output of the GP is richer, including uncertainty and covariance information, we first need to consider what information to give to the final decoder network. We employ the following output features from the GP.
GP output mean The multi-dimensional mean Hois represents the best guess of y Ideally, contains representation of the mask Mo to be predicted. The decoder works 2D feature maps and MQl is therefore reshaped as x, E R xxx x
GP output covariance The covariance Eois captures both the uncertainty in the predicted query output v2 and the correlation between different query outputs. The former lets the decoder network neighbouring
to identify uncertain regions in the image, and instead rely on, e.g., learnt priors predictions. Moreover, local correlations can tell whether two locations of the image are similar. For each query output, we employ the covariance between it and each of its spatial neighbours in an N N region. We represent as 2D feature map 25 E IR. x XW (N²) by vectorizing the N2 covariance values in the N x N neighborhood. For more details, see Appendix E
Shallow image features Shallow image features, extracted from early layers of deep neural net- works, are of high resolution and permit precise localization of object boundaries (Bhat al. 2018). We therefore store feature maps extracted from earlier layers of F and feed them into the decoder. These serve to guide the decoder as it transforms the low-resolution output mean into precise, high-resolution mask.
We finally predict the query mask by feeding the above information into decoder U,
Me UZ Zo Zy, Q.hslooow
As a decoder U, we adopt DFN Yu et al. 2018). DFN processes its input at one scale at a time, starting with the coarsest scale. In our case, this the mean and covariance. At each scale, the result at the previous scale is upsampled and processed together with any input at that scale. After processing the finest scaled input, the result upsampled to the mask scale and classified with linear layer and a so softmax function.
3.6 FSS LEARNER PYRAMID
Many computer vision tasks benefit from processing at multiple scales or multiple feature levels. While deep, high-level features directly capture the presence of objects or semantic classes, the mid-level features capture object parts Tian et al. (020). In semantic segmentation, methods begin with high-level features and then successively adds mid-level features while upsampling (Long al. 2015). In object detection, a detection head is applied to each level in a feature pyramid (Lin et al.
2017 Tian et al. (020)) discusses and demonstrates the benefit of using both mid-level and high- level features for few-shot segmentation. We therefore adapt our framework to be able to process features different levels.
We take features from our image encoder F extracted at multiple levels A. Let the support and query features level a A be denoted x and x From the mask encoder G we extract corresponding support outputs y3. We then introduce a few-shot learner A" for each level a. For efficient inference, the support features and outputs are sampled on grid such that the total stride compared to the original image is 32. The query features retain their resolution. Each few-shot learner then infers a posterior distribution over the query outputs y² parameterized by ( and
Ljss These are then fed into the decoder, which processes them one scale at a time.
EXPERIMENTS
We validate our proposed approach by conducting comprchensive experiments on two FSS bench- marks: PASCAL-5' (Shaban et al. 2017) and COCO-20 (Nguyen & Todorovic 2019).

---

Under review as a conference paper at ICLR 2022
4.1 EXPERIMENTAL SETUP
Datasets We conduct experiments on the PASCAL-5' (Shaban et al. 2017] and COCO-20' Nguuen & Todorovic 2019) benchmarks. PASCAL-55 is composed of PASCAL VOC 2012 (Everingham
et al. 2010p with additional SBD (Harharan et al. 2011 annotations. The dataset comprises 20 categories split into 4 folds. For each fold. 15 categories are used for training and the remaining 5 for testing. COCO-20* Nguyen & Todorovic 2019 is more challenging and is built from MS- COCO (Lin et al., 2014 Similar to PASCAL-5 COCO-20' benchmark is split into folds. For each fold, 60 base classes are used for training and the remaining 20 for testing.
Implementation Details Following previous works, we employ ResNet-50 and ResNet- 101 (He et al. 2016) backbones pre-trained on ImageNet (Russakovsky et al. 2015) as image encoders. We let the dense GP work with feature maps produced by the third and the fourth residual module. In addition, place single convolutional projection layer that reduces the feature map down to 512 dimensions. As mask encoder, we use a light-weight CNN (see Appendix E. We use og = 0.1,
of 1, and e vD in our GP. We train our models for 20k and 40k iterations 8 episodes for PASCAL-5' and COCO-20', respectively. On one NVIDIA A10 GPU, this takes up to 10 hours. We use the AdamW (Kingma & Ba 2013 Loshchilov & Hutter 2019) optimizer with a weight decay factor of 10 33 and a cross-entropy loss with 1-4 backgrounddforegroundd weighting. We use learning rate of 5 10 for all parameters save for the image encoder, which uses learning rate of 10-6. The learning rate is decayed with a factor of 0.1 when 10k iterations remain. We freeze the batch normalization layers of the image encoder. Episodes evaluate. We randomly flip images horizontally followed by resizing to a size of 384 x 384 for PASCAL-5' and 512 x 512 for COCO-20', Well-documented code will be made publicly available.
sampled in the same way we
Evaluation We evaluate our approach on each fold by randomly sampling 5k and 20k episodes re- spectively, for PASCAL-5' and COCO-20', This follows the work of Tian et al. 2020) in which it is observed that the procedure employed by many prior works, using only 1000 episodes, yields fairly high variance. Additionally, following WWag et al. (2019; Liu et al. (020cc: Boudial et al. (2021); D); Zhang et al. 20.21, our results in the sfate-of-the-ar comparison are computed as the average of 5 runs with different seeds. We also report the standard deviation. Performance is measured in terms of mean Intersection over Union (mloU). First, the IoU is calculated per class over all episodes in fold. The mloU is then obtained by averaging the IoU over the classes. As in the original work on FSS by Shaban et al. (2017). we calculate the IoU on the original resolution of the images.
4.2 STATE-OF-TII-ARY COMPARISON
We compare our proposed DGPNet with existing FSS approaches on PASCAL-5' and COCO-20* benchmarks. Table α presents the state-of-the-art comparison on both benchmarks. Following prior works, we report results given a single support example, 1-shot, and given five support examples, 5-shot. Our proposed DGPNet sets a new satte-of-the-art on both benchmarks. Among the previ- ous approaches, SAGNN (Xie et al. 202T) achieves mloU scores of 62.1 and 62.8 for 1-shot and 5-shot segmentation, respectively on PASCAL-5* using ResNet50. RePri (Boudiaf et al., 2021 ob- tains mloU scores of 59.1 and 66.8 for 1-shot and 5-shot segmentation, respectively. Our DGPNNet. achieves superior performance with mloU scores of 63.5 and 73.5 for 1-shot and 5-shot segmenta- tion, respectively. When using the ResNet101 backbone, PFENet (Tian et al., 2020) achieves the best 1-shot segmentation performance of 60.1 among existing methods. Our DGPNet outperforms PFENet Tian et al. 2020) by achieving mloU score of 64.8. Similarly, our DGPNet significantly outperforms the best reported 5-ohot results in literature (65.6 by RePri (Boudiaf et al. 202T)), with an absolute gain of 9.8.
Support Set
Query
DGPNet
Support Set
Query
DGPNet
Figure 3: Qualitative I-shot results of our approach on PASCAL-5' (left) and COCO-20' (right).
7

---

Under review as a conference paper at ICLR 2022
Table 1: State-of-the-art comparison on PASCAL-5' and COCO-20* benchmark in terms of mloU (higher better). In each case, the best two results are shown in magenta and cyan font, respectively. Asterisk denotes re-implementation by Liu et al. (2020a). Our DGPNet achieves state-of-the-art results for 1-shot and 5-shot on both benchmarks. When using ResNet101, our DGPNet achieves absolute gains of 8.2 and 15.2 for 1-shot and 5-shot segmentation, respectively on the challenging COCO-20' benchmark, compared to the best reported results in the literature.
PASCAT.-5'
COCO-20
Backbone
Method
I-shot
5-shot
I-shot
5-shot
CANet
(hang ct al.
2019b
55,4
57.1
40.8*
42.0"
PGNet Zhang
2019a
56.0
58.5
36.7*
37.5*
RPMM Yang el al
2022
56.3
57.3
30.6
35.5
CRNet Luu et al.
2020b
55.7
58.8
DENet
et
2020
60.1
60.5
42.8
43.0
ResNe150
LTM Yang et al. 20200
57.0
60.6
PFENet Tian a..
2020
60,8
61.9
PPNet Liu al. 20Zuc
51.5
62.0
25.7
36.2
RePri (Boudial et al. [022
59.1
66.8
34.0
42.1
SCL (Zhan et al. 2021
61.8
62.9
SAGNN (Xıe et al.
2021
62.7
62.8
Det (Ours)
63.5 + 0.4
73.5
0.3
45.0
0.4
56.2
+ 0.4
FWB (Nuuyen & Todorovic 2019
56.2
60.0
21.2
23.7
DAN Wang et al.
2020)
58.2
60.5
24.4
29.6
PFENet
2020
60.1
61.4
38.5
42.7
RePri (oudha ct al
2021
59.4
65.6
ResNet101
VPI Wang ct al. 202
57.3
60.4
23.4
27.8
ASG Q1
al.
59.3
64.4
34.6
42.5
SCL Zhang et al.
37.0
39.9
SAGNN (Xie et al.
2021
37.2
42.7
DGPNet (Ours)
64.8 1
0.5
75.4
0.4
46.7
1
0.3
57.9
1 0.3
On the challenging COCO-20' benchmark, DENet Liu et al. 2020a) obtains mIoU scores of 42.8 and 43.0 for I-shot and 5-shot segmentation, respectively using ReNer50. Our DGPNet outper- forms DENet (Liu et al. 2020a), achieving mIoU scores of 45.0 and 56.2 for 1-shot and 5-shot segmentation, respectively. When using ResNet101, DGPNet obtains absolute gains of 8.2 and 15.2 for 1-shot and 5-shot segmentation, respectively, compared to the best reported results in literature. Our per-fold results, including the standard deviation for each fold, are provided in appendix A.3
Qualitative examples are found Figure 3 and Appendix F
Scaling Segmentation Performance with Increased Support Size As discussed earlier, the FSS approach desired to effectively leverage additional support samples, thereby achieving superior accuracy and robustness for larger shot-sizes. We therefore analyze the effectiveness of the proposed DGPNet by increasing the number of shots on PASCAL-5' and COCO-20', Figure shows the results the two benchmarks. We the model trained for 5-shot for in all cases, except for 1-shot, Compared to most existing works, our DGPNet effectively benefits from additional support samples, achieving remarkable improvement in segmentation accuracy with larger support sizes.
Cross-dataset Evaluation In Table the cross-dataset evaluation capabilities of DGPNet from COCO-20' to PASCAL. For this cross-dataset evaluation experiment, we followed the same protocol as in Boudial et al. (2021), where the folds are constructed to ensure that there is no overlap with COCO-20* training folds. When using the same ResNet50 backbone, our approach obtains 1-shot and 5-shot mloU of 68.9
we present
Table
Performance comparison (mIoU, higher is better), when performing cross- dataset evaluation from COCO-20' to PAS- CAL. When using the same ResNet50 back- bone, our approach achieves significant im- provements for both 1-shot and 5-shot settings, with absolute gains of 5.8 and 11.3 mloU over RePRI (Boudiaf al. 2021).
Method
I-Shot
5-Shot
and 77.5, respectively with absolute gains 5.8 and 11.3 over RePRI (Boudiaf et al., 2021). Fur- ther, DGPNet achieves segmentation mloU 70.1 and 78.5 in 1-shot and 5-shot setting, respec- tively using ResNet101. For more details on the experiment, Appendix A.
RPMM Yang et al. 2020a
49.6
53,8
PFENet [aa al. 2020
61.1
63.4
RePRI Boudral et 2021
63.7
66.2
ResNctSU
68.9 0.4
77.5 = 0.2
Ours, ResNet101
70.1
0.3
78.5 JL 0.3
8

---

Under review as a conference paper at ICLR 2022
4.3 ABLATION STUDY
Here, we analyze the impact of the key components in our proposed architecture. We first investigate the choice of kernel, Then, we analyze the impact of the predictive covariance provided by the GP and the benefits of learning the GP output space. Last, we experiment with dense GPs at multiple feature levels. All experiments in this section are conducted with the ResNet50 backbone. Detailed experiments are provided in the Appendix.
Choice of Kernel Going from linear few-shot leamer to more flexible function requires an appropriate choice of kernel. We consider the homogenous linear kernel as our baseline. Note that the homogenous linear kernel is equivalent to Bayesian linear regression under appropriate priors (Rasmussen & Williams| 2006). To make learner more flexible, consider two ad- ditional choices of kernels, the Exponential and Squared Exponential (SE) kernels. The kernel equations are given in Appendix C In Table B results on both the PASCAL and COCO bench- marks are presented. Both the Exponential and SE kernels greatly outperform the linear kernel, with the SE kernel leading to a significant gain of 5.5 in mloU. These results show the benefit of more flexible and scalable learner.
Table 3: Performance of different kemnlls the PASCAL-5' and COCO-20* benchmarks. No- tably, both the Exponential and SE kernels signif- icantly outperform the linear kernel, confirming the need for flexible learner. Measured in mloU (higher is better). Best results are in bold.
PASCAL-5'
COCO-20
Kernel Linear
1-shot
5-shot
1-shot
5-shot
58.6
61.8
37.1
45.3
0.0
Exponential
59.3
67.3
40.9
51.8
4.1
SE
62.1
69.9
41.7
51.2
5.5
Incorporating Uncertainty and Learning the GP Output Space We adopt the SE kernel and analyze the performance of letting the decoder process the predictive covariance provided by dense GP. With the covariance window, obtain gain of 1.7 mloU. Then, we add the mask-encoder and learn the GP out- put space (GPO). This leads to 1.7 improve- ment in isolation, or a 2.5 mloU gain together with the covariance.
Table 4: Analysis of learning the GP output space (GPO) and incorporating covariance (Cov). Per- formance in mloU (higher is better). Best results in bold.
PASCAL-5
COCO-20
Cov
GPO
1-shot
5-shot
1-shot
5-shot
62.1
69.9
41.7
31.2
0.0
62.5
71.8
43.8
53.7
1.7
61.7
72.7
43.1
54.2
1.7
62.5
72.6
44.7
55.0
2.5
Multilevel Representations Finally, we investigate the effect of introducing mul- tilevel hierarchy of dense GPs. Specifi- cally we investigate using combinations of stride 16 and 32 features. The results are reported Table 5 The results show the benefit of using dense GPs at different fea- ture levels, leading to an average gains 1.4 mloU.
Table 5: Performance of different multilevel configura- tions of the dense GP few-shot learner the PASCAL- 5' and COCO-20' benchmarks. Measured in mloU (higher is better). Best results are in bold.
PASCAL
COCO-20*
Stride 16
Stride 32
1-shol
S-shot
1-shot
S-shot
A
62.5
72.6
44.7
55.0
0.0
60.4
69.3
40.7
51.1
-3.9
63.9
73.6
45.3
56.4
5 CONCLUSION
We have proposed few-shot learner based Gaussian process regression for the few-shot seg- mentation task. The GP models the support set in deep feature space and its flexibility permits it to capture complex feature distributions. It makes probabilistic predictions the query image, pro- viding both a point estimate and additional uncertainty information. These predictions are fed into CNN decoder that predicts the final segmentation. The resulting approach sets new state-of-the- art on PASCAL-5' and COCO-20', with absolute improvements of up to 14.9 mloU. Our approach scales well with larger support sets during inference, even when trained for fixed number of shots.
9

---

Under review as a conference paper at ICLR 2022
Reproducibility Statement In addition to the data, implementation, and evaluation details provided in section 4.1 we also provide (i) pseudo-code for the Gaussian Process; (ii) the details of the input to the decoder, and (iii) table including all trainable layers of our network. We have tried to be complete and believe that this information is sufficient to reproduce our results. In addition, the code for both training and evaluation will be made publicly available. Furthermore, due to the stochastic nature of the task, we follow Tian et al. (2200) and try to reduce the latter by sampling more episodes, 5000 for PASCAL-5' and 20000 for COCO-20', We additionally train and evaluate our final approach with five different seeds. Both the performance mean and standard deviation are reported in Table L
Ethics Statement Computer vision has the potential to automate many tasks, both in the form of autonomous agents and in the form of dig-data processing. There is a wide range of potential benefits in doing so, such as enabling robots to perform hazardous tasks or via advanced driving assistance systems reduce the number of accidents in traffic. However, there are also challenges. Automation may require changes societal systems and can be used for ill. We do not perceive few-shot segmentation as particularly problematic. Instead, few-shot segmentation enables the introduction of new categories without the need of large-scale data collection. Approaches based on episodic training, including our approach, are instead trained only on standard computer vision datasets. We two potential benefits. First, it democratizes semantic segmentation the cost of data collection is reduced. Secondly, with large-scale data collection there is always a risk of privacy infringement. Reducing the amount of data necessary mitigates these risks.
REFERENCES
Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. In International Conference on Machine Learning, pp. 232-241. PMLR, 2019.
Reza Azad, Abdur R Fayjie, Claude Kauffmann, Ismail Ben Ayed, Marco Pedersoli, and Jose Dolz. On the texture bias for few-shot CNN segmentation. Proceedings of the IEEF/CVE Winter Conference on Applications of Computer Vision, pp. 2674-2683, 2021.
Goutam Bhat, Joakim Johnander, Martin Danelljan, Fahad Shahbaz Khan, and Michael Felsberg. Unveiling the power of deep tracking. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 483 4983,4984400 2018
Malik Boudiaf, Hoel Kervadec, Ziko Imtiaz Masud, Pablo Piantanida, Ismail Ben Ayed, and Jose Dolz. Few-shot segmentation without meta-learning: A good transductive inference is all you need? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13979-13988 13988, 2021.
Roberto Calandra, Jan Peters, Carl Edward Rasmussen, and Mare Peter Deisemroth. Manifold Gaus- sian Processes for regression. In Proceedings of the International Joint Conference on Neural Networks, 2016. ISBN 978 7781509006199) doi: 10. 1001199NN.0001..777700
Nanqing Dong and Eric P. Xing. Few-shot semantic segmentation with prototype learning. In British Machine Vision Conference 2018, BMVC 2018, 2019.
Mark Everingham, Luc Van Gool, Christopher K.I. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 2010. ISSN 09205691. doi: 10. 10.10077/11223-009-0225555
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji. and Jitendra Malik. Se- mantic contours from inverse detectors. In Proceedings of the IEEE International Conference on Computer Vision, 2011. ISBN 9781457711015, doi: 10. 10.1109/CCV.2011..1122
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Tao Hu, Pengwan Yang, Chiliang Zhang, Gang Yu, Yadong Mu, and Cees GM Snoek. Attention- based multi-context guiding for few-shot semantic segmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 8441-8448, 2019.
10

---

Under review as a conference paper at ICLR 2022
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, 2015.
Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim. Adap- tive prototype learning and allocation for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8334-8343, 2021.
Tsung Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. European Conference on Computer Vision, pp. 740-755. Springer, 2014. doi: 10.1007/978-3-319- 10602-1. 48.
Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Lizhao Liu, Junyi Cao, Minqian Liu, Yong Guo, Qi Chen, and Mingkui Tan. Dynamic Extension Nets for Few-shot Semantic Segmentation. pp. 1441-1449. Association for Computing Machin- cry (ACM), 2020a. doi: 10. (0.114//39947773341399
Weide Liu, Chi Zhang, Guosheng Lin, and Fayao Liu. CRNer: Cross-Reference Networks for Few-Shot Segmentation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 4pp.11644 4164-4172 2020b. doi: 10. 1011090CCVP444500000000000
Yongfei Liu, Xiangyi Zhang, Songyang Zhang, and Xuming He. Part-Aware Prototype Network for Few-Shot Semantic Segmentation. European Conference on Computer Vision. pp. 142 1588, 2020c. ISBN 9783030585440. doi: 10.1007/978-3 0010/99783-30---0 1-.-0300555-77
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings tfe IEEE conference on computer vision and pattern recognition, pp. 3431 -4400, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL httpps://openreviee...tt net. forum?id
Bkg6RiCqy7
Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEEE International Conference on Computer Vision, volume 2019-Octob, pp- doi: 10. 101109/9/2222219000077.
2019. ISBN 9781728 148038.
Massimiliano Patacchiola, Jack Turner, Elliot J. Crowley, and Amos Storkey. Bayesian meta- learning for the few-shot setting via deep kernels. In Advances Neural Information Processing Systems, 2020.
Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alexei Efros, and Sergey Levine. Conditional net- works for few-shot semantic segmentation. In 6th International Conference on Learning Repre- sentations, ICLR 2018- Workshop Track Proceedings, 2018.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. 2006. doi: 10.7551 /
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, pp. 1-42, April 2015. doi:
10.10077/11263--05-0055
Ruslan Salakhutdinov and Geoffrey Hinton. Using deep belief nets to learn covariance kernels for Gaussian processes. In Advances in Neural Information Processing Systems 20 Proceedings of the 2007 Conference, 2009. ISBN 160560352X.
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. In British Machine Vision Conference 2017, BMYC 2017, 2017. ISBN 190172560X. doi: 10.5244/c.31.167
11

---

Under review as a conference paper at ICLR 2022
Mennatullah Siam, Boris Oreshkin, and Martin Jagersand. AMP: Adaptive masked proxies for few- shot segmentation. Proceedings of the IEEE International Conference on Computer Vision, volume 2019-Octob, pp. 2248-,257, 2019. ISBN 9781728148038. doi: 10.1109/ICCV.2099 00535.
Jake Snell and Richard Zemel. Bayesian few-shot classification with one-vs-each pólya-gamma augmented gaussian processes. In International Conference Learning Representations, 2021.
URL httpttppss//ppeereiiieeee htt..nnetttoroomm1ooooooooooooo.
Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior Guided Feature Enrichment Network for few-shot segmentation. IEEE Trunsactions Pattern Analysis & Machine Intelligence, 2020. ISSN 23318422. doi: 10.1 10.1 10.1109/tmmi..2220.00 2000.30377.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 9998-6008, 2017.
Haochen Wang. Xudong Zhang, Yutao Hu, Yandan Yang. Xianbin Cao, and Xiantong Zhen. Few- Shot Semantic Segmentation with Democratic Attention Networks. In European Conference on Computer Vision, pp. 730-746, 2020. ISBN 9783030586003. doi: 10.1007/978-3-000-5860000 43.
Haochen Wang, Yandan Yang, Xianbin Cao, Xiantong Zhen, Cees Snoek, and Ling Shao. Varia- tional prototype inference for few-shot semantic segmentation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 525-534, 2021.
Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng, PANet: Few-shot image the IEEE International Con- semantic segmentation with prototype alignment. Proceedings the
ference Computer Vision, volume 2019-Octob, pp. doi: 10.1109/CCCV.2019.0009
2019. ISBN 01887221480033
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learn- ing. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, 2016.
Guo-Sen Xie, Jie Liu, Huan Xiong, and Ling Shao. Scale-aware graph neural network for few-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5475-5484, 2021.
Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype Mixture Models for Few-Shot Semantic Segmentation. In European Conference on Computer Vision, pp. 763-778. Springer, 2020a.
Yuwei Yang, Fanman Meng, Hongliang Li, Qingbo Wu, Xiaolong Xu, and Shuai Chen. A new local transformation module for few-shot segmentation. In International Conference on Multimedia Modeling, pp. 76-87. Springer, 2020b.
Changqian Yu. Jingbo Wang, Chao Peng. Changxin Gao, Gang Yu, and Nong Sang. Learning Dis- criminative Feature Network for Semantic Segmentation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018. ISBN 9781538664209. doi: 10. 10.1109/CVP22221.000099 201999
Bingfeng Zhang, Jimin Xiao, and Terry Qin. Self-guided and cross-guided learning for few-shot segmentation. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8312-8321, 2021.
Chi Zhang, Guosheng Lin, Fayao Liu, Jiushuang Guo, Qingyao Wu, and Rui Yao. Pyramid graph networks with connection attentions for region-based one-shot semantic segmentation. In Pro- ceedings of the IEEE International Conference on Computer Vision, volume 2019-Octob, pp- 9586-9594 2019a. ISBN 17817281480035 doi: 10.1100//CC222219.00950
12

---

Under review as a conference paper at ICLR 2022
Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. CANET: Class-agnostic seg- mentation networks with iterative refinement and attentive few-shot learning. In Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2019b. ISBN 9781728132938. doi: N0.1009CVVR..219.0000
Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang. Sg-one: Similarity guidance network for shot semantic segmentation. IEEE Transactions on Cybernetics, 50(9):3855-388 2020.
13

---

Under review as a conference paper at ICLR 2022
SUPPLEMENTARY MATERIAL
We first provide additional results in Appendix Then, in Appendix B we provide analyses of the covariance neighborhood size and the effect of training the image encoder. The equations for the different kernels used in ablation study runtimes of the different components in our approach. Appendix t
shown in Appendix C In Appendix D list the
contains pseudo-code for our dense GP, list detailing the trainable layers of our approach, and the details how the output of the GP is presented to the decoder. Finally, in Appendix F we provide a qualitative comparison and qualitative results.
A ADDITIONAL RESULTS
We provide the per-fold results on the COCO-20' to PASCAL transfer experiment and list of the classes included; the per-shot results used to generate Figure in the main paper; and the per-fold results used the state-of-the-ae comparison.
A.1 CROSS-DATASED E EVALUATION
We supply additional details on the cross-dataset evaluation experiment. This experiment was pro- posed by Boudiaf al. (2021) and we follow their setup. First, our approach is trained on each of the four folds of COCO-20', Next, we test each of the four versions on PASCAL, using only the classes held-out during training. We list the classes each fold Table The full per-fold results presented in Table 6
Table 6: The results of our approach in COCO-20* to PASCAL transfer experiment (mloU, higher better). Following Boudiaf et al. 2021}. the approach is trained on a fold of COCO-20* training set and tested on the PASCAL validation set. The testing folds are constructed include classes not present in the training set, and thus not the same as PASCAL-5'
Mothol
F-0
F-1
F-2
L-Shx
F-3
Mea
F-0
F-1
S-Shox
F-2
F-3
Meon
RPMM Sanne
36.1
350
13.
54.6
49.5
40.2
35.0
55.2
61.3
31.8
PCENe:
43.2
65.1
66.5
69.7
45.1
66.8
68.5
73,1
63.4
RaPRI
528
57.7
73.1
8588
Table 7: The classes used for testing in the COCO-20' to PASCAL transfer experiment, proposed by Boudiaf et al. 2021)). This split different from that of PASCAL-5' order to avoid overlap between the training and testing classes. Fold-O
Fold-1
Fold-2
Fold-3
Airplane, Boat, Chair, Bicycle, Bus, Horse, Sofa
Bird,
Car,
Potted
Bottle, Cat, Cow, Mo- torcycle
Dining Table, Dog,
Plant, Sheep, Train,
Person
TV-monitor
A.2 1-10 SHOT RESULTS
Here, provide the full results from 1-10 shots, which was used to generate Figure 1 and D our results on PASCAL-5' and COCO-20' are presented. Note that our approach was trained for shot in the 1-shot setting, and 5 shots in the other nine settings.
In Table
Table 8: to 10 -hot PASCAL-5 results.
T-thet
Z-hhot
1-hhot
4:thor
5-4hot
6-thot
7-thor
10-shot
DGPNNr (ResNei10))
6488 0.5 684
724 0.3 74.2 0.5
0.4
761
766
77.0
77.5 0.4 77.7
Table 9: to 10 shot COCO-20* results.
I-hhol
S-shot
S-ahol
Geshol
Meshol
10shht
16.8 0.3
31.7 1
550 0. 36.7
1 0.3 37.9
38.7
392
59.7 0.3
60.0 1 0.3 60.200
14

---

Under review as a conference paper at ICLR 2022
A.3 PER-FOLD RESULTS
In this section, we provide the full results for our state-of-the-art comparison. In Tables To and | the per-fold results for both ResNet50 and ResNet101 are presented. In general results stable for all folds.
Table 10: Per-fold results on PASCAL-55
Method
g1
Mean
51
52
Mean
CANer (ReiNethl)
52.5
66.9
5133
51.9
55.4
55.5
67.8
51.9
53.2
57.1
DENet (ResNec50)
55.7
69.7
63.6
51.3
60.1
54.7
71.0
64.5
PFFNet Na15)))
55.4
63.1
70.7
RePri (ResNet50)
ASGNet (ResNert01)
59.8
67.4
53.6
$4.4
50.3
64.6
713
64.2
51.3
64.4
SCI. (ResNer50)
63.0
70.0
36.5
37.7
6188
64.5
70.9
$7.3
58.7
62.9
SAGNN (ResNer50)
64.7
69.6
57.0
$7.2
62.1
64.9
70.0
57.0
59.3
62.8
DGPNeT (KssNe15))
71.1
01.2=07
769
73.2
(ResNett01)
Table 11: Per-fold results on COCO-20*
T-Shol
5-Shot
Method
20F
20'
Mean
20"
202
Mean
DENet (KesNet50)
42.9
45.8
42.2
50.2
42.8
45.4
44.y
41.6
40.3
41.0
PFENet (ResNet50)
36.8
11.8
18.7
36.7
38.5
10.1
46.8
43.2
40.5
12.7
(ReNNe10))
(ResNet50)
SCL (ResNett10))
36.4
38.6
37.5
35,4
37.0
38,9
40.5
41.5
38.7
29.9
SAGNN (ResNet101)
36.1
11.0
38.2
33.5
37.2
40.9
48.3
42.6
38.9
42.7
TXPPNer DXPPRer/ReNNit00 (RteNNi33)
23.5 0.5
47.8 E U.
44.3
08
44.2 0.6
45.0 0.4
34.7 0.7
39.1 = 03
56.8 3 0.6
34,4 1 0.6
DGPNNet (ReNNe111)
58.4
B DETAILED ANALYSIS
We present results from additional experiments. First, the size of the local covariance region is analyzed. Then, we investigate the impact of freezing the backbone during episodic training.
B. ADDITIONAL COVARIANCE NEIGHBORIOOOD EXPERIMENTS
In section 3.5 we show how the covariance with neighbors in a local region is fed to the decoder.
In Table 12 supply additional results with different sized windows. We experiment with N (1,3,5, 5, Larger windows improve the results but the improvement seems to saturate after N = 5.
Table 12: Performance for different configurations covariance windows on the PASCAL-5' and COCO-20' benchmarks. Measured in mloU (higher is better).
PASCAL-5
COCO-20'
1x1
3x3
5x5
7x7
1-shot
5-shot
1-shot
5-shot
A
62.1
69.9
41.7
51.2
0.0
60.0
70.3
42.2
51.7
-0.2
62.0
71.1
43.6
53.0
1.2
62.5
71.8
43.8
53.7
1.7
63.0
71.7
43.7
53.5
1.8
B.2 EFFECT OF TRAINING THE IMAGE ENCODER
In Table 1 we compare our final approach trained with a frozen and unfrozen backbone. Several prior works found beneficial freeze the image encoder during episodic training. In contrast, find it beneficial to not freeze it and keep learning visual representations. By differentiating through our GP learner during episodic training, the proposed method can thus refine the underlying feature representations, which is another important advantage of our approach. However, our approach still improves upon state-of-the when employing frozen backbone.
B.3 ADDITIONAL BASELINE EXPERIMENTS
We supply three additional baseline experiments validate the effect of the proposed DGP-module We evaluate three variants: (i) We remove the GP (the f-branch) and let the decoder rely only on the shallow features. (ii) We replace the GP with the Prior-mask learning mechanism proposed in
15

---

Under review as a conference paper at ICLR 2022
Table 13: Comparison between freezing the backbone and fine-tuning it with low learning rate.
PASCAL-5'
COCO-20*
Backbone
Method
1-shot
5-shot
1-shot
5-shot
DGPNet (Frozen Backbone)
61.9 0.3
72.4 = 0.3
43.1 = 0.3
54.5 0.2
ResNet50
DGPNet (Ours)
63.5 + 0.4
73.5 = 0.3
45.0 = 0.4
56.2 + 0.4
DGPNet (Frozen Backhone)
63.4 + 0.5
74.3 0.3
44.6 + 0.5
56.6 0.3
ResNet101
DGPNet (Ours)
64.8 + 0.5
75.4
0.4
46.7 + 0.3 57.9 + 0.3
PFENet. (iii) We replace the GP with a prototype-based approach where the support features are mask-pooled and the result compared to the query features using cosine-similarity, similar to e.g. PANet. Results on PASCAL and COCO are reported the Table Our approach outperforms all baselines by large margin. This further validates the superiority of our GP module.
Table 14: Additional baseline experiments to validate the effect of the DGP-module.
PASCAL-5*
COCO-20*
Method
I-shot
5-shot
1-shot
5-shot
No GP
42.9
43.0
23.5
23.5
PFENet prior mask
54.0
54.7
35.8
37.2
Prototype
57.5
63.0
41.5
49.9
DGPNet (Ours)
63.5
73.5
45.0
56.2
C KERNEL DETAILS
In this section we provide the definitions of the kernels used in our ablation study. First we define the homogenous linear kernel as,
(i, v) = x²
As was noted in the paper, this kernel corresponds to Bayesian Linear Regression with specific prior. One could also consider learning a bias parameter, however we chose not to learn such pa- rameters for reasons of method simplicity, Next we consider the exponential kernel,
|| v112
Keep (x, u) = exp
This kernel behaves similarly as the SE kernel, however with sharper peak and slower rate of decay. For completeness we additionally define the SE kernel here again,
|| 312
(x y) = exp
282
We chose the length of the exponential kernel g² VD, we found the performance in general to be robust different values of f. Note that used the same value of é for both benchmarks and for all number of shots.
DD and the squared exponential kernel as
provide a kernel hyperparameter sensitivity analysis. We run experiment per value and choose the values to cover one order of magnitude centered around the values adopted. Perturbing the hyperparameters with factor of 0.3 3.0 leads minor but statistically significant drops in performance. One exception is increasing of by a factor of 3.0, which does not adversely affect performance.
In Table 15
Table 15: Kernel hyperparameter sensitivity analysis for the SE-kernel, which adopted in the main
paper. The sensitivity analysis is based on the full, final approach with a ResNet50 backbone and performed on the 5-shot setting in PASCAL-5' The hyperparameters values are chosen to cover one order of magnitude of hyperparameter values, centered at the values adopted in the main paper.
& 03/D
P=
e 3.0/
of = 0.3
o7
= 1.0
of = 3.0
o² = 0.03
d² = 0.1
0.3
72.9
73,5= 0.3 72.8
72.8
73,5
0.3
73.6
73.0
73,5
0,3 72.9
D RUNTIMES
We show the runtime our method in Table 16 We partition the timing into different parts. The Gaussian process (GP) split into two. One part preparing and decomposing the support matrix
16

---

Under review as a conference paper at ICLR 2022
Table 16: Runtimes of the different functions in approach, measured in milliseconds (ms). Tim-
ings are measured in evaluation mode on 512 x 512 sized images from COCO-20'.
Batch size
Batch size 20
1-shot
5-shot
1-shot
5-shot
Image encoder on support
15.5
23.4
48.5
197.5
Mask encoder on support
2.1
2.1
1.6
6.6
GP preparation support
8.3
13.9
7.2
113.5
Image encoder on query
9.7
13.6
39.8
39.9
GP inference query
7.9
9.4
12.1
38.1
Decoder
6.2
6,9
40.0
40.5
Total
49.7 ms
69.3 ms
149.2 ms
436.1 ms
in equation X and equation 7 of the main paper, and another part computing the mean and covariance of the query given the pre-computed matrix decomposition.
The timings are measured in the 1-shot and 5-shot settings on images from COCO-20" of 512 x 512 resolution, using either single episode per forward batch of 20 episodes in parallel. We the method in evaluation mode via torch. no.grad () Timing is measured by injecting cuda events around function calls and measuring the elapsed time between them. The events are inserted via torch. cuda Event (enable t Liming-Truse V100 for 1000 episodes and report the average timings of each part.
We run our approach on single NVIDIA
E ADDITIONAL IMPLEMENTATION DETAILS
We provide code for the Gaussian Process inference, the neural network layers that make up the modules used in our approach, and the details of how the predictive output distribution from the GP is fed to the decoder.
E.1
CODE FOR GAUSSIAN PROCESS
Pseudo-code for the dense GP is shown in Listing !1 Equation / involves the multiplication with a matrix inverse. is in practice computed via the Cholesky decomposition and solving the re- sulting systems of linear equations. For brevity and clarity, we omit device casting and sim- plify the solve implementation. In practice, we use the standard triangular solver in PyTorch, torch. torch.triangullet www.hhtriigguaaasle..e...
def GP (x (x. q, Y_s, x x_s, sigma_y, y, kernel):
Produces the predictive posterior distribution of the GP. After cach line, we comment the shape of the output, with sizes defined as:
the batch-size
Q is the number of query feature vectors the query image
s is the number of support feature vectors across the support images
M is the number of channels in the GP output space.
is the number of channels in the feature vectors.
10
11
Args:
12
x_g: deep query features (B,0,D)
y_s: support outputs (B,S,M)
14
X_s: deep support features (B)
15
sigma_y: mask standard deviation kernel: the kernel function,
16
B, S, x_s.shape
19
torch.eye (S)
20
K_ss = kernel (x_s, x_s) $ (B,
K_ = kernel (x_g, x_g)
(B,0,
K_sc kernel (x_s, x_9)
$ (B,
s torch.cholesky sK_ss sigma_yx*2
(8,
17

---

Under review as a conference paper at ICLR 2022
All neural network blocks used by approach. The rightmost column shows the di- mensions of the output of each block, assuming a 512 x 512 input resolution. The image encoder is from He et al. 62 and the decoder from Yu al. 2018 The BottleNeck and BasicB ccclBBL lock blocks are from He et al. (2016), and the CAB and RRB blocks from Yu et al. (2018. See their works for additional details.
Table 17:
Image Encoder
conv bn1
Conv2a
64 246 x 256
BatchNorm2d
64 256 256
relul
64 256 256
maxpool
MaxPoe12d
64 128 x 128
layerl
3x BottleNeec
256 128 x 128
layer2
4x BottieNeck
x 64
layer3
6x/23x BottleNeck
1024
32
layer4
3x BettieNeck
2048 x 16 16
layer3 out
Conv2a
512 x 32 32
layerd out
Conv2d
512 x 32 32
Mask Encoder
convi
Conv2a
16 256
bnl
BatchNorm2d
16 256 256
relul
ReLU
16 256 x 256
maxpool
MaxPoci2d
128
layerl
BasicBlock
32 x 64
layer2
BasicBlock
64 x 32 32
layer3
BasicBlock
64 x 16 16
layer2 out
Conv2G-Bat.chaarze
64 x 32 32
layer3 out
Canv2a-BttchNo.n22
64 16
Decuder
rrb in cub
RRB
256 x 16 16
CAB
256 x 16 16
mbb up upsamplel
x x
Upsample
256 32
rrh in 2 cub 2 rbb up upsample2
RRB
256 x 32 32
CAB
256 x 32 32
Upsample
256 64 64
rtb in 3 cab
RRB
256 x 64 64
CAB
256 x 64 64
mbb up upsample3 rth in 4 cab
x
Upsample
256 128 128
RRB
256 128 x 128
CAB
256 128 x
rub up
256 128
conv upsample4
Conv2a
x 128 x 128
Ups
2 512 x 512
mu_q (K_sq.1 solve (1_88.1, solve (1 y_8)) (H,
35
v solve (1_ss, K K_sq) (B, cov_g q_ - v.T 8 (B, return mu 9, cov
26
Listing PyTorch implementation of the Gaussian Process utilized in the proposed approach. Here, the learning and inference is combined in single step. The @ operator denotes matrix multiplication and solve ve the solving of linear system of equations. The T is the batched matrix transpose.
E.2 TRAINABLE LAYER LIST
we report the trainable neural network layers used in our approach. The image encoder listed either a ResNet50 (He et al. 2016) or a ResNet101 (He et al. 2016) with two projection layers. The results of layer3 and layer4 are each fed through linear projection layer, layer3 out and layer4 out respectively, to produce feature maps at stride 16 and stride 32. These two feature maps are then fed into one GP each in the GP pyramid. The support masks is fed through another ResNet (He et al. 2016) in order to produce the support outputs. The GPs are integrated as neural network layers, but do not contain any learnable parameters. The decoder is DFN (Yu al. 2018) We do not adopt the border network used in their work and we skip the global average pooling. We also feed it shallow feature maps extracted from the query. The shallow feature maps are the results of layerl and layer2 in the image encoder, at stride 4 and 8 respectively.
In Table 17
18

---

Under review as a conference paper at ICLR 2022
E.3 FINAL MASK PREDICTION DETAILS
In 3.5 we transformed the output of the GP before presenting to the decoder, restoring spatial structure. We present the equations for doing so. Let (-). denote tensor indexing. The mean repre- sentation fed to the decoder is found as
(z (Hos)aW+ur Zp E H
(10)
For the covariance we consider spatial neighbours within a square x N window. Let = (N 1)/2 and (i,j) {-d....,, .) be the possible offsets within the window. Then, the covariance representation fed to the decoder is
(4))) Arg. zg E BB
(11)
QUALITATIVE RESULTS
We provide qualitative results on PASCAL-5' and COCO-20'. First, we compare the our final approach to baseline on the COCO-20* 20' benchmark. The baseline also relies on dense GPs but uses linear kernel, does not utilize the predictive covariance or learn the GP output space, and uses single feature level (stride 32). Our final approach instead adopts the SE kernel, adds the predictive covariance local 5 x 5 region, learns the output space, and employs dense GPs two feature levels (stride 16 and stride 32). The results are shown in Figure Our final approach significantly outperforms the baseline in these examples, making only minor mistakes.
In Figure we show qualitative results of our final approach the PASCAL-5' dataset. Our ap- proach accurately segments the class of interested, even details such as sheep legs. In Figure 0 we show results COCO-20'.
We provide a qualitative comparison between different support set sizes in Figure / For this com- parison, COCO-20. In general, the 1-shot setting is quite challenging for several classes due to major variations in where the object appears and what it looks like. the single support sample different from the query sample, the model tends struggle. With five support samples, the model performs far better. At five, the benefit of additional support samples seems to saturate, with ten samples often bringing only marginal improvements. An example is the train episode Figure where minor improvements are obtained when going from five to ten support samples.
19

---

Under review as a conference paper at ICLR 2022
Support
Query
Baseline
Ours
Figure 4: Qualitative comparison between our final model and baseline in the 1-shot setting from the COCO-20' benchmark. Human faces have been pixelized in the visualization, but the model makes predictions on the non-pixelized images.
20

---

Under review as a conference paper at ICLR 2022
Support Set
Query
Ours
IIIH
Figure 5: Qualitative results on challenging episodes in the 1-shot setting from the PASCAL-55 benchmark.
21

---

Under review as a conference paper at ICLR 2022
Support Set
Query
Ours
3000
munool
Figure 6: Qualitative results on challenging episodes in the 1-shot setting from the COCO-20 benchmark. Human faces have been pixelized in the visualization, but the model makes predic- tions on the nin-lixelized images.
22

---

Under review as a conference paper at ICLR 2022
Support
Support 2-5
Support 6-10
Query
1-shot
5-shot
10-shot
Figure 7: Qualitative results of our approach given 5, and 10 support samples. The 1-shot results are based on Support 1; the 5-shot results on Support 1 and Support 2-5; and the 10-shot Support 1, Support 2-5, and Support 6-10. The results from COCO-20' and human faces have been pixelized in the visualization, but the model makes predictions on the non-pixelized images.
23


Reviews:
[{"review_id": "YHfvl9qbhBI", "reviewer": "Reviewer_Kjwm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pro:\n- A novel idea to combine GP and FSS.\n- Extensive experiments to validate the results on different benchmarks\n- State-of-the-art on two benchmarks, and gracefully extend to high-shot segmentation.\n\n\nCon:\n- Hyper-parameters sensitivity. Are the methods sensitive to different hyper-parameter in GP? The author should mention this in the paper.\n- What is your implementation without Cov and GPO in Table 4? The detail is missing here.\n- For me, it is still not clear how the input are interplayed in encoder.(E.3 only introduced about restoring spatial structure of the output of the GP)\n- In contribution part, mentioned the probabilistic modeling, I don't see its importance in experimental parts.\n- Visualization for 5 shot is not provided, what's the main improvement from 1shot to 5shot to 10shot?\n- How about removing the contribution of f branch, only keeping shallow x?", "summary_of_the_paper": "The paper strives for solving the few-shot segmentation problem, they propose to incorporate the Gaussian Process(GP) into the framework of few-shot segmentation. Except that, they also exploit the high-dimensional output space for GP, the result reaches state-of-the-art in two benchmarks,  one bonus is that  the segmentation quality scales gracefully as increasing the support set size.", "summary_of_the_review": "The paper overall presents a novel idea with solid empirical validations. Except some minor concern I raised in previous parts. In total, I am inclined to accept this paper.  I am happy to raise the score if my concern is fully resolved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SzXbOOjQULT", "reviewer": "Reviewer_nhTR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strengths:\n\n    1. It's a solid idea to introduce uncertainty modeling into few-shot segmentation.\n\n    2. The exact implementation details of the experiments are provided. Although the source code is not presented, the authors show its pseudo-code in SM.\n\n\n\n- Weaknesses:\n\n    1. The way of the GP outputs being used is somehow confusing. The authors claim that \\mu_{Q|S} and \\Delta_{Q|S} are the predicted mean and covariance of the mask of query, i.e., y_Q. However, the prediction of y_Q is fulfilled through DFN (a deep CNN), which means it's not ensured that the predicted \\hat{y}_Q can be sampled from a Gaussian distribution with a relatively high probability.\n\n    2. Since the input of the decoder is a concatenation of the GP outputs and shallow-level query image features, it's necessary to conduct another ablation study to show the influence on the predicted result cast by the GP outputs, i.e., the effectiveness of introducing the GP outputs.\n\n   3. The use of 1-4 background-foreground weighting in the loss function actually performs hard-example mining, which is empirically found beneficial in binary segmentation tasks. As the baseline has already surpassed many of the other methods, the author should apply the proposed method on a standard training scheme, such as the SGD optimizer with the CE loss without weighting, following the practice of PANet, PPNet, PFENet and CANet.\n\n   4. The inference speed should be discussed since an additional network DFN is appended as the decoder.", "summary_of_the_paper": "This paper proposes a special Gaussian process (GP) named dense GP, to model a mapping between dense local deep features and their corresponding mask values. Based on this dense GP, a few-shot segmentation method named DGPNet is proposed.\nThe authors claim that DGPNet is novel in that it can be applied to situations that unseen classes are not linearly separable, and can produce the uncertainty of its prediction as well. To support this they conduct series of experiments on PASCAL-5^i and COCO-20^i.", "summary_of_the_review": "This paper is novel mostly in that it utilizes a Gaussian process to generate an extra coarse prediction of the query image. However, an ablation study is lacking to prove the effectiveness of the dense GP module alone. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AjGYEAOMc48", "reviewer": "Reviewer_b84d", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a dense Gaussian Process approach for few shot segmentation scenario. Overall promising results are achieved comparing to the recent efforts. On the other hand, the paper has some issues to be discussed below.\n*lack of clarification of the major contributions. It seems the main contribution lies in being first to apply the combination of GPs and Neural Networks to the few short semantic segmentation task. It is yet unclear what is the main technical contribution in this paper, despite reading Sec.3 multiple times. \n*It is also surprising that despite being a much simpler method as shown in Fig.2 when comparing to existing methods such as SAGNN (CVPR'21), and despite the internal Gaussian representation is very coarse-scale and lacking-of-details as in Fig.2, the final results of this paper are much better. Is there any intuition of why this is the case?\n*The authors should also present more visual results of e.g. their 5-short segmentation in the appendix/supplementary. It also lacks visual evidence of the claimed uncertainty reasoning benefits.\n*What is the computational cost/time complexity for the training/inference processes, respectively?\n*The authors should provide the implementation publicly available. Otherwise it is difficult for others to validate and reproduce the same results and performance.\n", "summary_of_the_paper": "This paper proposes a dense Gaussian Process approach for few shot segmentation scenario. Overall promising results are achieved comparing to the recent efforts. ", "summary_of_the_review": "many aspects are unclear", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8LFfyQ_GD36", "reviewer": "Reviewer_uRb5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The idea of adopting dense Gaussian Process (GP) regression to learn the mapping from local deep image features to mask values, as well as considering uncertainty for the final segmentation is interesting. \n- The paper is well written and the methodology technically sound.\n- Authors report extensive experiments and the results achieved are very competitive.\n\nWeaknesses:\n- The idea that utilizing Gaussian processes in the context of few-shot classification has been exploited in [r1,r2]. Although the proposed method focuses on the segmentation task, it seems like a dense classification setting of r1 and r2. Authors need to clarify the main differences. \n- Does this method only focus on 1-way (binary segmentation) setting? Can it solve the multiple novel classes in a single episode?", "summary_of_the_paper": "Authors propose a novel few-shot segmentation method by adopting dense Gaussian process (GP) regression to capture complex appearance distributions. To boot the performance, authors consider the uncertainty in the final segmentation. Authors exploit the end-to-end learning capabilities of the proposed method to learn a high-dimensional output space for the GP. \nAuthors report state-of-the-art results in two public few shot segmentation benchmarks.", "summary_of_the_review": "The idea of adopting dense Gaussian Process (GP) regression to learn the mapping from local deep image features to mask values, as well as considering uncertainty for the final segmentation is interesting. Although the proposed method seems like a dense setting of the existing methods, the paper is well written and the results achieved are very competitive. I'd like to enhance my rating if the authors address all my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes the use of Gaussian process regression embedded into a neural network architecture for few-shot segmentation. In more detail, support and query images and support masks are fed through their encoders and their corresponding features are then used for Gaussian process regression to infer the distribution of the query mask encoding given the support set and the query images. The mean and the variance characterizing the GP predictive distribution is then fed into a CNN-based decoder to make the final prediction (segmentation).  The method is evaluated on PASCAL-5^i and COCO-20^i datasets, showing the superiority of the proposed approach wrt several competitive baselines. 

Overall, the reviewers found the approach of using GPs within the proposed architecture interesting and somewhat significant and novel to the few-shot segmentation community. Technically, the proposed method does not develop a new algorithm and simply uses standard Gaussian process regression. The authors seemed to have addressed several concerns raised by the reviewers including the ablation study evaluating the influence of the GP module. However, the reviewers felt that there were quite a few changes/clarifications to the paper and new results that were not highlighted in the revised version, which made it difficult to provide a new assessment of the paper. Furthermore, the reviewers also thought that the authors did not provide convincing explanations in terms of the improvements from 1-shot to 5-shots, the not-so-good results when the model was trained with standard SGD without loss weighting and the rationale behind the success of the 5-shot setting.
Decision: Reject