Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an ensemble GP method (where each model is termed a 'module') that can fuse a set of pre-existing GPs to train a new "meta-GP".  The authors also present a closed form ensemble lower bound that considers the parametrisations of the 'modules'.
main_review: I rather enjoyed the motivating dialogue and examples. 

"Each module is considered to be a sparse variational GP, containing only variational parameters and hyperparameters." Is it a necessary condition that each module be specifically a sparse variational GP? Could this approach be extended/directly applied to spectral approximations like in [R1]? I ask because given the generality of the proposal "We present a framework based on modular Gaussian processes" it is not so clear that the method applies only to a particular _class_ of GP derivatives (of which sparse vi methods are just one), or a more general class of GP approximations (of which, e.g., [R1] is also one).

Clarity:
The introduction starts with a rather involved sentence: "We develop a module-based method...revisiting any data". The sentence is rather convoluted and I would suggest condensing or splitting it up.

"n. However, GP models are not immune to settings where we need to adapt to irregular ways of observing data, e.g. asynchronous samples or missing inputs." I'm not so sure this is true (or at least the way it is phrased here). GPs are typically derived on continuous domains; i.e. ones that do not need data to be 'gridded' (although approximate methods have been proposed to accommodate structured data such as KISS-GP).

Related work:
I think [R0] should be included in the related work since it is addressing a similar problem but from the optimal transport perspective and applies in a potentially more general sense than KL divergence between stochastic processes.
Also, the approach of a "meta" GP seems directly related to the classic concept of model stacking. This has been attempted in the past (e.g. [R2]) and has also been termed a meta-gp. Could the authors comment on the relation to such prior work?

Compression: It seems to me that using a sparse VI gp, there is a consequent loss of information. If it was a full GP, there would be no information loss in the sense of the full covariance (of the 'past') data. 

[R0]  Mallasto, Anton, and Aasa Feragen. "Learning from uncertain curves: The 2-Wasserstein metric for Gaussian processes." Advances in Neural Information Processing Systems. 2017.
[R1] Lázaro-Gredilla, Miguel, et al. "Sparse spectrum Gaussian process regression." The Journal of Machine Learning Research 11 (2010)
[R2] Neumann, Marion, et al. "Stacked Gaussian process learning." 2009 Ninth IEEE International Conference on Data Mining. IEEE, 2009.
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Summary: the paper proposes a modular decomposition of the conventional GP process. The author claims that such novel decomposition can empirically improve the prediction accuracy. However, the paper lacks certain theoretical reasoning behind the claims. Many critical definitions are missing and the presentation quality of the paper is below the NeuRIPS standard. 
main_review: Weakness:

The presentation made the paper hard to follow, without a clear story line to seize. For example, why is it necessary to introduce the inducing variable $z_k$, and how are $z_k$ computed from D_k given x and y? Things get even worse when the subscript notation * is introduced, making the notations harder to interpret. Many notations are not defined，such as $L_k^*$, $f_{+\neq u_*}, V_{+ \neq *}$. Most importantly, the paper lacks discussions that why decompostion on the data is advantageous for GP learning. I listed some of my confusions below. 

79-81 “consider that N_k for all k ∈ {1, 2, . . . , K} is sufficiently large for not accepting exact GP inference due to temporal and computational demands.” What does this sentence mean? 

89-90 ”…being $u_k$ values of the same function $f(\cdot)$” (grammarly wrong).

144: notation $f_{+\neq u_*}$ is not defined here.   

153: why $p(y|f_+)=\sum_k \log p(y_k|f_+)$? Are the marginal of $ p(y) $and $p(y_k)$ expected to be same (I think so, because k seems to be indexing a subsets of the entire data., i.e., samples of size $N_k$ of the entire data)? If so, then there should be an additional multiplicative 1/K constant on the r.h.s term.

178-179: “Without the need of revisiting any data, the GP predictive $q_C(u_k)$ is playing a different role in contrast with the usual one.” This sentence is seemingly trying to support the novelty claim, but failed to deliver the message. To be honest, the whole discussion paragraph that claims the advantage of such modular method (175-182) is difficult to interpret even if one with proper GP background is 100% focused. I spent hours on this paragraph but still cannot get the insights behind. 

189: $L_k^*$ is not defined. 

240: the reasoning behind such factorization is unclear, partly because the notation V_{+ \neq *} is undefined. 

It leaves one wonder how the GP helps the transfer learning task in the empirical section at all. Please clarify this point. It seems to me the tasks are conventional Regression/Classification GP tasks, without the need of transfer learning. Please clarify how the advantage of the proposed method is reflected in the experiments section.

The usage of word "accept" has been abused everywhere. 
limitations_and_societal_impact: Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: A novel approach to learn from multiple tasks (involving different partitions of predictor datasets and different types of target variables) that share the same type of input and domain. The method approximates and bounds the overall (cross-task) log-likelihood with ELBOs and posteriors of each task/module. Using inducing points posteriors, the data in each task can be summarized by inducing points of each task and is not required in later tasks. This is an appealing property in practice since it obviates the need to store all datasets of all tasks, and the training can be performed in a task-distributed fashion. 
main_review: To be able to learn first on a large dataset and then quickly transfer to other tasks of smaller datasets is critical for modern machine learning tasks. Thus, I think the problem addressed by the authors is of high importance. The method proposed is novel and interesting (with small caveats not fully exposed), combining inspirations from previous work to address the transfer learning problem. In particular, the transfer does not require storing the full datasets of previously encountered tasks, which is appealing to many practical problems. Further, this method can be extended to multi-output and multi-dimensional GP latent, and can be stacked on top of each other according to tasks splitting to obtain a hierarchical meta-GP model. 

The experiments are thorough, covering small and large datasets. although I am very curious about the results in other well-controlled settings. Despite this small imperfection and other minor issues I list below, I really enjoyed reading this paper. The paper is also written fairly clearly with nice figures explaining the intuitions, but some descriptions are not intuitive to readers like me, so please provide more details. 

I am not completely up-to-date with the literature in this area, but I have not seen any methods that are trivially similar to this proposed here. The benchmark methods compared in the experiments are also quite different in methodology (though distributed in different senses), some are significantly less well-developed than this work. This argues for the novelty of this work, and if it is true that this method is that novel, the authors should further highlight this new contribution. 

Could the authors provide details to the questions below? If addressed well, I am willing to further raise my score. 
1. How does the approximation in (4) impact the bound? This is probably the only loose end in this method, and I hope to see some analysis in this respect. Are there cases when this approximation is particularly bad so that the objective is no longer a bound on the likelihood? Are there any tricks used to ensure the quality of this approximation?
2. Although we do not need to retain all the data for each task, we do retain the inducing points to compute $q_k$ for (6) and (7). How does the relative size of the inducing set to the task data set affect the performance of this method?
3. In the toy concatenation experiment, I think it would be great to analyse all the moving parts of this method. What are the effects of 
   1) inducing points and its approximate posterior,
   2) approximation made in (4) 
   3) modular training of hyperparameters? Importantly, I'd love to see a comparison to the best solution possible without modular training, for example, a full posterior computed exactly without inducing points. 
If it's not quite possible to clearly dissect the contribution of these factors, please explain why, as it is not immediately clear to me. 
4. In the description of the banana dataset experiment, why is it not possible to control the number of inducing points? How would the two methods compare if this number is controlled?
4. All the figures need proper captions. I know the space is limited, but the figures make less sense and the readers have to guess. 
5. The bold and normal Z's refer to unrelated objects, please consider using other symbols?
6. In line 229, what does $i$ index? Is this related to LMC?
7. In line 313, what is the exact inference solution? Exact in what sense? 
8. In line 303, what is the "combination of solutions" referring to? 
9. Figure 5 is not referred to. 
10. There should at least be some discussions in the main text on training and optimization. 

===== post rebuttle ======

I thank the authors for their reply. I also do not see serious technical concerns from other reviewers, except for the presentation and minor details (I now recall some initial difficulty in reading which is expressed in other reviews). This is a great paper and I raise my score and confidence. The value of this paper may be more obvious in the near future. 

I would strongly encourage the authors to revise the paper to include the content of ALL their responses provided here.
limitations_and_societal_impact: The authors mentioned data retention its impact on privacy. The authors could discuss how inducing points encapsulate or hides the information about each individual data, and if the dataset in each task reflects properties of a user group, does the inducing point reveal specific information about this group?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a framework of reusing previously trained sparse variational GP models, without the need for revisiting the original data.
Each variational sparse GP model is called a "module".
The authors assume K modules, each of which is trained individually on a dataset $D_k$ (for $1 \leq k \leq K$) by means of ELBO maximisation (equation (1)).
The trained modules are subsequently combined into a meta-GP, which is defined as a global variational approximation of a large-dimensional GP.
This is achieved by deriving a global variational bound, where the global likelihood is approximated by utilising the local variational distribution in eq. (4).
Eventually, the evidence lower bound of eq. (6) is produced, which is optimised with respect to the global variational parameters $\phi = \{\mu, S\}$, as well as the hyperparameters $\psi $ and the (global) inducing point locations $Z$.
The framework is also extended to a multi-output context following MOGP.
The experimental campaign demonstrates that the methodology proposed is highly competitive on a number of large-scale datasets. 
main_review: The variational framework that is proposed to combine pre-trained GP modules into a single approximating model is well-motivated and novel, to the best of my knowledge.
The paper is well-written and the related work is adequately discussed.
The methodology proposed is demonstrated on a number of examples, while the numerical experiments confirm that the method is competitive to the state-of-the-art on a number of datasets, some of which involve millions of instances.
I think this work is of interest to the community, as it may have significant applications not only due to its scalability properties, but also due to the possibility of reusing trained models based on a rigorous framework.


The are however a few points that need to be clarified.

I found the discussion of the VEM algorithm (Algorithm 1) in the appendix somewhat confusing.
No EM algorithm has been discussed or motivated in the main paper.
Before reading the appendix, I had been under the impression that the variational parameters ($\mu$ and $L$), the hyperparameters $\psi$ and the inducing locations $Z$ and were being jointly optimised with respect to the ELBO in Equation (6).
In the joint optimisation case, $\psi$ and $Z$ are treated in terms of empirical Bayes (due to the ELBO).
However, according to the VEM algorithm, these three classes of parameters are optimised independently in three nested optimisation loops.
In particular, the optimisation of $\mu$ and $L$ is called as the E-step, followed by 2 M-steps for $\psi$ and $Z$, which are treated as latent variables.
I do not understand the motivation behind this choice.

In line 176, it is stated that the sum of expectations in (5) works as an average contrastive indicator.
I am not sure that I understand that "contrastive" means in this context. 
Could you elaborate on this?
limitations_and_societal_impact: In terms of societal impact, the authors comment on the need for protection of fitted models, as these may contain sensitive information.
review_title: null

Meta-review: This paper proposes a framework for reusing previously learned sparse variational GP models without revisiting any data and a further extension to the multi-output context. All four knowledgeable reviewers recommend acceptance and I agree with them. As a clarification to the authors, yes, you are allowed and encouraged to incorporate amendments/additions to address the reviewers feedback in the final version of the paper. 

Decision: Accept (Poster)