Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a special Gaussian process (GP) named dense GP, to model a mapping between dense local deep features and their corresponding mask values. Based on this dense GP, a few-shot segmentation method named DGPNet is proposed.
The authors claim that DGPNet is novel in that it can be applied to situations that unseen classes are not linearly separable, and can produce the uncertainty of its prediction as well. To support this they conduct series of experiments on PASCAL-5^i and COCO-20^i.
main_review: - Strengths:

    1. It's a solid idea to introduce uncertainty modeling into few-shot segmentation.

    2. The exact implementation details of the experiments are provided. Although the source code is not presented, the authors show its pseudo-code in SM.



- Weaknesses:

    1. The way of the GP outputs being used is somehow confusing. The authors claim that \mu_{Q|S} and \Delta_{Q|S} are the predicted mean and covariance of the mask of query, i.e., y_Q. However, the prediction of y_Q is fulfilled through DFN (a deep CNN), which means it's not ensured that the predicted \hat{y}_Q can be sampled from a Gaussian distribution with a relatively high probability.

    2. Since the input of the decoder is a concatenation of the GP outputs and shallow-level query image features, it's necessary to conduct another ablation study to show the influence on the predicted result cast by the GP outputs, i.e., the effectiveness of introducing the GP outputs.

   3. The use of 1-4 background-foreground weighting in the loss function actually performs hard-example mining, which is empirically found beneficial in binary segmentation tasks. As the baseline has already surpassed many of the other methods, the author should apply the proposed method on a standard training scheme, such as the SGD optimizer with the CE loss without weighting, following the practice of PANet, PPNet, PFENet and CANet.

   4. The inference speed should be discussed since an additional network DFN is appended as the decoder.
summary_of_the_review: This paper is novel mostly in that it utilizes a Gaussian process to generate an extra coarse prediction of the query image. However, an ablation study is lacking to prove the effectiveness of the dense GP module alone.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a dense Gaussian Process approach for few shot segmentation scenario. Overall promising results are achieved comparing to the recent efforts. 
main_review: This paper proposes a dense Gaussian Process approach for few shot segmentation scenario. Overall promising results are achieved comparing to the recent efforts. On the other hand, the paper has some issues to be discussed below.
*lack of clarification of the major contributions. It seems the main contribution lies in being first to apply the combination of GPs and Neural Networks to the few short semantic segmentation task. It is yet unclear what is the main technical contribution in this paper, despite reading Sec.3 multiple times. 
*It is also surprising that despite being a much simpler method as shown in Fig.2 when comparing to existing methods such as SAGNN (CVPR'21), and despite the internal Gaussian representation is very coarse-scale and lacking-of-details as in Fig.2, the final results of this paper are much better. Is there any intuition of why this is the case?
*The authors should also present more visual results of e.g. their 5-short segmentation in the appendix/supplementary. It also lacks visual evidence of the claimed uncertainty reasoning benefits.
*What is the computational cost/time complexity for the training/inference processes, respectively?
*The authors should provide the implementation publicly available. Otherwise it is difficult for others to validate and reproduce the same results and performance.

summary_of_the_review: many aspects are unclear
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors propose a novel few-shot segmentation method by adopting dense Gaussian process (GP) regression to capture complex appearance distributions. To boot the performance, authors consider the uncertainty in the final segmentation. Authors exploit the end-to-end learning capabilities of the proposed method to learn a high-dimensional output space for the GP. 
Authors report state-of-the-art results in two public few shot segmentation benchmarks.
main_review: Strengths:
- The idea of adopting dense Gaussian Process (GP) regression to learn the mapping from local deep image features to mask values, as well as considering uncertainty for the final segmentation is interesting. 
- The paper is well written and the methodology technically sound.
- Authors report extensive experiments and the results achieved are very competitive.

Weaknesses:
- The idea that utilizing Gaussian processes in the context of few-shot classification has been exploited in [r1,r2]. Although the proposed method focuses on the segmentation task, it seems like a dense classification setting of r1 and r2. Authors need to clarify the main differences. 
- Does this method only focus on 1-way (binary segmentation) setting? Can it solve the multiple novel classes in a single episode?
summary_of_the_review: The idea of adopting dense Gaussian Process (GP) regression to learn the mapping from local deep image features to mask values, as well as considering uncertainty for the final segmentation is interesting. Although the proposed method seems like a dense setting of the existing methods, the paper is well written and the results achieved are very competitive. I'd like to enhance my rating if the authors address all my concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper strives for solving the few-shot segmentation problem, they propose to incorporate the Gaussian Process(GP) into the framework of few-shot segmentation. Except that, they also exploit the high-dimensional output space for GP, the result reaches state-of-the-art in two benchmarks,  one bonus is that  the segmentation quality scales gracefully as increasing the support set size.
main_review: Pro:
- A novel idea to combine GP and FSS.
- Extensive experiments to validate the results on different benchmarks
- State-of-the-art on two benchmarks, and gracefully extend to high-shot segmentation.


Con:
- Hyper-parameters sensitivity. Are the methods sensitive to different hyper-parameter in GP? The author should mention this in the paper.
- What is your implementation without Cov and GPO in Table 4? The detail is missing here.
- For me, it is still not clear how the input are interplayed in encoder.(E.3 only introduced about restoring spatial structure of the output of the GP)
- In contribution part, mentioned the probabilistic modeling, I don't see its importance in experimental parts.
- Visualization for 5 shot is not provided, what's the main improvement from 1shot to 5shot to 10shot?
- How about removing the contribution of f branch, only keeping shallow x?
summary_of_the_review: The paper overall presents a novel idea with solid empirical validations. Except some minor concern I raised in previous parts. In total, I am inclined to accept this paper.  I am happy to raise the score if my concern is fully resolved.
review_title: null

Meta-review: This paper proposes the use of Gaussian process regression embedded into a neural network architecture for few-shot segmentation. In more detail, support and query images and support masks are fed through their encoders and their corresponding features are then used for Gaussian process regression to infer the distribution of the query mask encoding given the support set and the query images. The mean and the variance characterizing the GP predictive distribution is then fed into a CNN-based decoder to make the final prediction (segmentation).  The method is evaluated on PASCAL-5^i and COCO-20^i datasets, showing the superiority of the proposed approach wrt several competitive baselines. 

Overall, the reviewers found the approach of using GPs within the proposed architecture interesting and somewhat significant and novel to the few-shot segmentation community. Technically, the proposed method does not develop a new algorithm and simply uses standard Gaussian process regression. The authors seemed to have addressed several concerns raised by the reviewers including the ablation study evaluating the influence of the GP module. However, the reviewers felt that there were quite a few changes/clarifications to the paper and new results that were not highlighted in the revised version, which made it difficult to provide a new assessment of the paper. Furthermore, the reviewers also thought that the authors did not provide convincing explanations in terms of the improvements from 1-shot to 5-shots, the not-so-good results when the model was trained with standard SGD without loss weighting and the rationale behind the success of the 5-shot setting.
Decision: Reject