Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper develops a way to learn Gaussian processes with dense inducing variables by meta-learning for few-shot learning.
The algorithm provides a strong inductive bias for learning new tasks by specifying a shared Gaussian process prior over prediction functions of a set of tasks. It adapts the inducing variables to each task by efficient gradient descent to achieve task specific predictions.
Experiments show that this approach can achieve competitive performance in all the tasks considered, and state-of-the-art results in certain tasks.
main_review: 1. The paper is relatively well written. 
2. In terms of originality, I am not aware of other similar works.
3. The empirical results are pretty strong, achieving state-of-the-art results in certain settings.

**Detailed comments**:
1. Are there any theoretical guarantees for algorithm 1?
2. In equation 5, should it be $p(y|f_m)$ instead of $p(y|f)$?
3. Minor: some formatting issues at Line 20, 24.
limitations_and_societal_impact: There is no discussion on the limitations and potential negative societal impact of the work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a meta-learning algorithm based on Gaussian processes. The meta-learning or few-shot image classification algorithms based on Gaussian processes have been developed recently due to their principled uncertainty modeling mechanism, which can be particularly useful for few-shot learning settings where data are scarse. The core idea of the presented algorithm is its use of dense inducing variables; usually, the inducing point methods for GPs are employed when we have large training data so the direct computation of predictive or marginal distribution is costly. Hence, the inducing points methods are usually set to be sparse because they are introduced mainly for a computational reason. On the other hand, This paper interprets the inducing points as a shared knowledge that can be transferred between tasks and proposes to encode such meta-knowledge into them. To this end, the proposed method uses a dense set of inducing points so that rich meta-knowledge obtained during meta-training can be embedded into the inducing point and can be used to quickly adapt to new tasks. The resulting algorithm is a typical gradient-based meta-learning where the goal is to maximize marginal likelihood in the inner loop and the predictive likelihood in the outer loop, and the algorithm is trained via episodic training. The proposed algorithm is validated on common benchmarks on few-shot image classification and demonstrated decent performance.
main_review: I enjoyed reading the paper. The paper is well-written and clear, so easy to follow. The motivation is well-set, and I like the idea of transferring knowledge through inducing points. In my opinion, the main message of the paper is the use of dense inducing points whose size is much larger than typical support sets, and I think this idea can be applied to other related approaches.

Although not explicitly related, but I'd like to point out one related work that is worth citing. Set transformer (Lee et al., 2019) introduces an inducing point-based self-attention block to process set-structured data and is applied to meta-learning problems such as amortized clustering. There, the authors showed that their method with inducing points performs better than the full self-attention blocks without inducing points, and claim that the inducing points act as transferrable knowledge that can be shared among different tasks. I guess this view of interpreting inducing points is somewhat similar to the method proposed in this paper.

The experimental results are not that compelling; I'm not saying this because the tables in the paper do not list all the recent SOTA few-shot learning results (which would probably take an entire page). As the authors emphasized at the beginning of the text, meta-learning-based Gaussian processes provide a principled way of estimating uncertainties. In order to properly assess such aspects of GP-based meta-learning, I think the competing algorithms should be evaluated with the metrics related to uncertainty estimates (e.g., expected calibration error, negative log-likelihood, out-of-distribution detection performance, active learning, ...). It would have been good to have a minimal example on toy-regression tasks with meta-learned credible intervals displayed. 

Another thing that needs to be checked is the complexity of the algorithm. Figure 2 reports that the performance starts to saturate with the number of inducing points $m=256$. Then the time-complexity of computing marginal likelihood is roughly $O(nm^2)$, which is quite heavy since $m \gg n$. How are the actual running times (training time and inference time) compared to the baselines? I guess the proposed method can be quite slower than the baselines without GPs (which just typically requires a few forward passes through feature extracting neural nets) and GP-based without inducing points ($O(n^3)$).
limitations_and_societal_impact: The authors did not address the limitations and potential negative societal impact as shown in the checklist.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a solution to few shot learning by using Gaussian processes with dense inducing points from the perspective of few shot learning, which allows for learning a shared set of inducing points which are then specialized into task specific inducing points. 
main_review: ## Pros

- The results are strong and outperform all baselines methods.
- The method build off of work in GP's and meta learning
- The authors do a good job of citing past work in both GP and meta learning literature.

## Cons

- L40 (about GP's) ... "without suffering from computational cost in regular learning tasks" --> What does this mean? There is no computational cost? Even if you are saying there is reduced computational cost, is this even true? GP's scale as $\mathcal{O}(n^3)$
- Equation 7 has the order of arguments for the variational lower bound changed from equation 6. Is this a typo?
- The paper mentions dense inducing variables many times, but the inducing variables are actually quite sparse w.r.t. the task distribution. They are only dense when considered against the typical few shot learning task sizes. I think this distinction should be made clearer. 
- L299 says that figure 3 indicates the efficiency of the proposed solution. This is a very bold claim that lacks evidence. One could easily also say that it indicates the method overfits easily to tasks, as the performance drops quickly with more inner steps. This overfitting is something that other methods such as [1] do not show even with large numbers of adaptation steps.
- There is no analysis of the added runtime for training/inference, as well as trainable variable counts. This information should be included somewhere in order to better understand the relation to other methods. The method leads me to suspect that there would be a minimal parameter increase, but a large increase in training/inference times. 
- The authors don't describe any limitations of the work. 
- The authors do not apply the problem to the very common few shot regression tasks. Although these are usually simple, I think the GP aspect make a particularly compelling use case for regression tasks which should be discussed.

My score is reflective of mostly bullet points 5 and 7 which need to be addressed in order to make the submission stronger. 

 ## Minor

- L14 significantly --> significant
- L28 progresses --> progress
- L29 the few shot learning --> few shot learning
- L35 "widely developed" --> What does "widely" developed mean?
- L50 "we learning to learn" --> we learn to learn
- L73 "learning new task" --> learning new tasks
- L131 "few gradient descent update" --> few gradient descent updates
- L148 "an learned" --> a learned
- L154 "achieves better prior fitting the novel dataset" --> sentence needs to be rewritten to make sense

## References

[1] Nichol, A., Achiam, J., & Schulman, J. (2018). On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999.

limitations_and_societal_impact: There is no mention of limitations or societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an extension of the Gaussian process (GP) approach presented in [1] for the setup of few-shot learning. With each new task, the data is first processed by a neural network (NN) to obtain low-dimensional feature representations, and then a GP classifier acts on these features. Since the number of data points per task is limited, the authors suggest augmenting the model with inducing inputs that are shared among all tasks. Updating the parameters follows the standard meta-learning procedures (e.g., MAML). Using the lower bound presented in [1], an internal loop updates the task-specific inducing inputs. And by maximizing the predictive likelihood on the query set, an outer loop updates both the global inducing inputs and the network parameters. Experiments on standard FSL and cross-domain few-shot benchmarks show an improved accuracy over relevant baseline methods.  
main_review: I think that using Gaussian processes with NNs for few-shot challenges is a promising direction. However, there are some issues I would like the authors to address before I can pass it. 

This paper presents the following merits:  
- The paper shows strong results in a variety of experiments and against many baselines.  
- Using inducing variables to generate a richer dataset is a good idea.  
- The paper is written clearly and is easy to follow besides several points that will be described next.

Some points that I found problematic:  
- There are two crucial points that were not entirely clear to me, and since the code wasn't provided with the submission I could not get an answer myself. I will be happy to get clarification for both issues.  
(1) This paper builds on [1] which was designed for regression tasks with Gaussian likelihoods. To use this method, it seems that the authors referred to the discrete class labels as if they are Gaussian distributed. If indeed I understood correctly, it does not seem like a natural thing to do. Can you please clarify that point and further explain the intuition.   
(2) Unlike training in which the inducing inputs are jointly learned across tasks, when testing the model the classes are new and there should be no information sharing between tasks. Is that correct? If so, can you please clarify if and how does your method gain from sharing the inducing inputs of these new classes?  If the inducing location is only adjusted based on the inner loop iterations of the current task, does that suffice? I think that some analysis and more explanations regarding this part are required.  
- Many implementation details are missing. For example, did you perform a grid search over hyperparameters (besides the ones shown in the ablation study)? what temperature did you use in the softmax? how many epochs of training did you use? etc. In general, the text should provide much more details about the datasets, the experimental setups, and the baseline methods.  
- I think this paper omits important GPC references [2, 3, 4]. [2] handled a few-shot scenario as well, and all methods used inducing inputs in the embedding space of a NN.  
- From my experience temperature scaling often impact significantly on the classifier performance (for example, see [5] for a recent study on this topic). If indeed you adjusted the temperature for your method, did you try to tune it for relevant baseline methods as well? Specifically, I think that the performance of the main competitor OVE-GP (PL) should also be examined under the same conditions.  

Minor comments:
- There are some typing mistakes: gradient decend (L25), to learned (L152), Our (Table 1), double citation of the same reference (L234), the cross-domain tasks in Table 3 should be: Omniglot -> EMNIST & miniImageNet -> CUB.
- It is not clear from the text what network did you use to obtain the results in Table 1.


[1] Titsias, M. (2009, April). Variational learning of inducing variables in sparse Gaussian processes. In Artificial intelligence and statistics (pp. 567-574). PMLR.  
[2] Achituve, I., Navon, A., Yemini, Y., Chechik, G., & Fetaya, E. (2021). GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning. arXiv preprint arXiv:2102.07868.  
[3] Wilson, A. G., Hu, Z., Salakhutdinov, R., & Xing, E. P. (2016, December). Stochastic variational deep kernel learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems (pp. 2594-2602).  
[4] Bradshaw, J., Matthews, A. G. D. G., & Ghahramani, Z. (2017). Adversarial examples, uncertainty, and transfer testing robustness in Gaussian process hybrid deep networks. arXiv preprint arXiv:1707.02476.  
[5] Wenzel, F., Roth, K., Veeling, B., Swiatkowski, J., Tran, L., Mandt, S., ... & Nowozin, S. (2020, November). How Good is the Bayes Posterior in Deep Neural Networks Really?. In International Conference on Machine Learning (pp. 10248-10259). PMLR.
limitations_and_societal_impact: The authors did not address the potential limitations and negative societal impact of their work.  I think that authors should address this in the rebuttal.  
To the best of my judgment, there is no potential negative societal impact for this paper.
review_title: null

Meta-review: The reviewers all agree that the idea of using a dense set of inducing points to share information across tasks in the few-shot setting is novel and interesting. The discussions yielded a number of clarifications as well as new experiments on few-shot regression and uncertainty quantification. Please be sure to add these to the final draft.
Decision: Accept (Poster)