Modular Gaussian Processes for Transfer Learning

1 Introduction . Imagine a supervised learning problem , for instance regression , where N data points are processed for training a model . At a later time , new data are observed , this time corresponding to a binary classification task , that we know are generated by the same phenomena , e.g . using a different sensor . Having kept the observations from regression stored , a common approach would be to use them in combination with the classification dataset to generate a new model . However , this practice might be inconvenient because of i ) the need of centralising data to train the model , ii ) the rising data-dependent computational cost as the number of samples increases and iii ) the obsolescence of fitted models , whose future usability is not guaranteed for the new set of observations . Looking at the deployment in large-scale scenarios , by any organization or use case , where data are ever changing , this solution becomes prohibitive . The main challenge is to incorporate unseen tasks as former models are recurrently discarded . Alternatively , we propose a framework based on modules of Gaussian processes ( GP ) ( Rasmussen and Williams , 2006 ) . Considering the current example , the regression model ( or module ) is kept intact . Once new data arrives , one fits a meta-GP using the module , but without revisiting any sample . If no data are observed , combining multiple modules is also allowed — see Figure 1 . Under this framework , a new family of module-driven GP models emerges . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Background . Much recent attention has been paid to the notion of efficiency , e.g . in the way of accessing to data . The mere limitation of data availability forces learning algorithms to derive new capabilities , such as , distributing the data for federated learning ( Smith et al. , 2017 ) , recurrently observe streaming samples for continual learning ( Goodfellow et al. , 2014 ) and limiting data exchange for private-owned models ( Peterson et al. , 2019 ) . A common theme is the idea of model memorising and recycling , that is , using the already fitted parameters for an additional task without revisiting any data . If we look to the probabilistic view of this idea , uncertainty is harder to be repurposed than parameters of functions . This is the point where Gaussian processes play their role . The flexible nature of GPs for defining a prior distribution over non-linear function spaces has made them a suitable alternative in probabilistic regression and classification . However , GP models are not immune to settings where we need to adapt to irregular ways of observing data , e.g . noncentralised observations , asynchronous samples or missing inputs . Such settings , together with the well-known computational cost of GPs , typically O ( N3 ) , has motivated plenty of works focused on parallelising inference . Before the modern era of GP approximations , two seminal works distributed the computational load with local experts ( Jacobs et al. , 1991 ; Hinton , 2002 ) . While the Bayesian committee machine ( BCM ) of Tresp ( 2000 ) focused on merging independently trained GP regression models on data partitions , the infinite mixture of GP experts ( Rasmussen and Ghahramani , 2002 ) combined local GPs to gain expressiveness . Moreover , the emergence of large datasets , with size N > 104 , led to the introduction of variational methods ( Titsias , 2009a ) for scaling up GP inference . Two recent works that combined sparse GPs with distributed regression ideas are Gal et al . ( 2014 ) ; Deisenroth and Ng ( 2015 ) . Their approaches are focused in exact GP regression and the distribution of computational load , respectively . Contribution . We present a framework based on modular Gaussian processes . Each module is considered to be a sparse variational GP , containing only variational parameters and hyperparameters . Our main contribution is to keep such modules intact , to build meta-GPs without accessing any past observation . This makes the computational cost minimal , becoming a competitor against inference methods for large-datasets . The key principle is the augmentation of integrals , that can be understood in terms of the Kullback-Leibler ( KL ) divergence between stochastic processes ( Matthews et al. , 2016 ) . We experimentally provide evidence of the benefits in different applied problems , given several GP architectures . The framework is valid for multi-output settings and heterogeneous likelihoods , that is , handling one data-type per output . Our idea follows the spirit of not being data-driven any longer , but module-driven instead , where the Python user specifies a list of models as input : models = { model1 , model2 , . . . , modelK } . This will be later called the dictionary of modules . 2 Modular Gaussian processes . We consider supervised learning problems , where we have an input-output training dataset D = { xn , yn } Nn=1 with xn ∈ Rp . We assume i.i.d . outputs yn , that can be either continuous or discrete variables . For convenience , we will refer to the likelihood term p ( y|θ ) as p ( y|f ) where the generative parameters are linked via θ = f ( x ) . We say that f ( · ) is a non-linear function drawn from a zeromean GP prior f ∼ GP ( 0 , k ( · , · ) ) , and k ( · , · ) is the covariance function or kernel . Importantly , when non-Gaussian outputs are considered , the GP output function f ( · ) might need an extra deterministic mapping Φ ( · ) to be transformed to the appropriate parametric domain of θ . Data modules . The dataset D is assumed to be partitioned into an arbitrary number of K subsets or modules , that are observed and processed independently , that is , { D1 , D2 , . . . , DK } . There is not any restriction on the number of modules , { Dk } Kk=1 do not need to have the same size , and we only restrict them to be Nk < N . However , since we deal with a huge number of observations , we still consider that Nk for all k ∈ { 1 , 2 , . . . , K } is sufficiently large for not accepting exact GP inference due to temporal and computational demands . Notice that k is an index while k ( · , · ) is the kernel . 2.1 Sparse variational approximations for independent modules . For each data module , we adopt the sparse variational GP approach based on inducing variables , and the framework of Titsias ( 2009b ) , where posterior mean vectors and covariance matrices are computed as parameters . The use of auxiliary variables with approximate inference methods is widely known in the GP literature ( Hensman et al. , 2013 , 2015 ; Matthews et al. , 2016 ) . In the context of K independent data modules , we define subsets of Mk Nk inducing inputs Zk = { zm } Mkm=1 , where zm ∈ Rp . Their non-linear function evaluations by f ( · ) are denoted as uk = [ f ( z1 ) , f ( z2 ) , . . . , f ( zMk ) ] > . As a first approach , we assume that f is stationary across all modules , being all uk ∀k ∈ { 1 , 2 , . . . , K } function evaluations of f ( · ) . This will be later relaxed in the following section . To obtain the independent approximation to the posterior distribution p ( f |Dk ) of the GP , we introduce a kth variational distribution qk ( f ) for each data module Dk . In particular , this variational density factorises as qk ( f ) = p ( f 6=uk |uk ) qk ( uk ) , with qk ( uk ) = N ( uk|µk , Sk ) and p ( f6=uk |uk ) being the standard conditional GP prior distribution given the hyperparameters ψk of the kth kernel . To fit the GP modules , and their variational distributions qk ( uk ) , we build lower bounds Lk on the marginal log-likelihood ( ELBO ) of every dataset Dk . Then , we use gradient-based optimisation methods to maximise the K objective functions Lk , one per module , in an separate asynchronous manner . The ELBOs , for either regression or classification tasks are obtained as follows Lk = Nk∑ n=1 Eqk ( fn ) [ log p ( yn|fn ) ] − KL [ qk ( uk ) ||pk ( uk ) ] , ( 1 ) with pk ( uk ) = N ( uk|0 , Kkk ) , where Kkk ∈ RMk×Mk has entries k ( zm , zm′ ) with zm , zm′ ∈ Zk and conditioned to certain kernel hyperparameters ψk that we also aim to optimise . The variable fn corresponds to f ( xn ) and the marginal posterior comes from qk ( fn ) = ∫ p ( fn|uk ) qk ( uk ) duk . In practice , the distributed bounds Lk are identical to the one presented in Hensman et al . ( 2015 ) and can be combined with stochastic variational inference ( Hoffman et al. , 2013 ; Hensman et al. , 2013 ) . Dictionary of modules . The principal goal here is to obtain a dictionary , containing the already fitted GP modules , for their later use . Such dictionary consists , for instance , of a list of module objects { M1 , M2 , . . . , MK } without any specific order , where eachMk = { φk , ψk , Zk , L∗k } , being φk the corresponding variational parameters µk and Sk . Notice that we also include L∗k in addition to the fitted inducing points and hyperparameters . This is the ELBO obtained during optimization . It will play a key role later for the module-driven learning process . 2.2 Module-driven lower bound . Ideally , to obtain an inference solution given the GP modules included in the dictionary , the resulting posterior density should be valid for all data modules { Dk } Kk=1 . This is only possible if we consider the entire dataset D in a maximum likelihood criterion , that is , using the global log-evidence . Specifically , our goal is now to obtain an approximate posterior distribution q ( f ) ≈ p ( f |D ) , by maximising a lower bound LM under the log-marginal density log p ( D ) . This bound should not revisit any data but only the objects in the dictionary of modules , that are models . Notice that the GP model will be no longer data-driven , but module-driven instead . This is , we build a new model from models , what we call a GP meta-model or meta-GP . We begin by considering the full posterior distribution of the stochastic process , similarly as Burt et al . ( 2019 ) does for obtaining an upper bound on the Kullback-Leibler ( KL ) divergence . The idea is to use the large-dimensional integral operators introduced by Matthews et al . ( 2016 ) in the context of variational inference , and previously by Seeger ( 2002 ) for standard GP error bounds . The use of the large-dimensional integrals is equivalent to an augment-and-reduce strategy ( Ruiz et al. , 2018 ) . It consists of two steps : i ) we augment the model to be conditioned on the high-dimensional index set of the stochastic process and ii ) we apply properties of Gaussian marginals to reduce the integral operators to a finite amount of GP function values of interest . Similar strategies have been previously used in continual learning for GPs ( Bui et al. , 2017 ; Moreno-Muñoz et al. , 2019 ) . Global evidence objective . The construction considered is as follows . We first denote y as all the output targets { yn } Nn=1 in the dataset D and f+ as the augmented large-dimensional GP . Notice that f+ sufficiently large and it contains the function values taken by f ( · ) at both { xn } Nn=1 and { Zk } Kk=1 for all data modules . We consider it finite . The log-marginal expression is therefore log p ( y ) = log p ( y1 , y2 , . . . , yK ) = log ∫ p ( y , f+ ) df+ , ( 2 ) where yk = { yn } Nkn=1 are the output data-points already used for training each GP moduleMk . The joint distribution in ( 2 ) factorises according to p ( y|f+ ) p ( f+ ) , where the l.h.s . term is the augmented likelihood distribution and the r.h.s . term would correspond to the GP prior over the large index set of the stochastic process . Then , we introduce the global variational distribution q ( u∗ ) = N ( u∗|µ∗ , S∗ ) that we aim to fit by maximising a lower bound under log p ( y ) given the dictionary of modules . The variables u∗ correspond to function values of f ( · ) given the new subset of inducing inputs Z∗ = { zm } Mm=1 , where M is the free-complexity degree of the global variational distribution . To derive the bound , we exploit the reparametrisation introduced by Gal et al . ( 2014 ) for distributing the computational load of the expectation term with GP latent variable models ( GPLVM ) ( Lawrence , 2005 ) . This is based on the decoupling of data-points conditioned on the inducing inputs u∗ . Applying Jensen ’ s inequality , it is obtained as log p ( y ) = log ∫∫ q ( u∗ ) p ( f+6=u∗ |u∗ ) p ( y|f+ ) p ( u∗ ) q ( u∗ ) df+6=u∗du∗ ≥ Eq ( u∗ ) [ Ep ( f+ 6=u∗ |u∗ ) [ log p ( y|f+ ) ] + log p ( u∗ ) q ( u∗ ) ] , ( 3 ) where we applied the properties of Gaussian conditionals to factorise the GP prior as p ( f+ ) = p ( f+6=u∗ |u∗ ) p ( u∗ ) . Notice that f+ = { f+6=u∗ , u∗ } . Here , the last prior distribution is p ( u∗ ) = N ( u∗|0 , K∗∗ ) where [ K∗∗ ] m , n : = k ( zm , zn ) , with zm , zn ∈ Z∗ . We also consider k ( · , · ) conditioned to the global kernel hyperparameters ψ∗ that we also aim to estimate . The double expectation in ( 3 ) comes from the factorization of the large-dimension integral given f+ and the application of Jensen ’ s inequality twice . Its derivation is in the appendix . Likelihood reconstruction . The augmented likelihood distribution p ( y|f+ ) is perhaps the most important part of the derivation . It allows us to apply conditional independence ( CI ) between the data modules Dk , and particularly , the output variables yk . This gives a factorized term that we will later use for introducing the GP modules Mk stored in our dictionary , that is , log p ( y|f+ ) = ∑K k=1 log p ( yk|f+ ) . To avoid revisiting each kth likelihood term , and hence , evaluating the corresponding data which might be unavailable , we want to estimate the likelihood distributions . In particular , we use Bayes theorem conditioned to the large-dimensional augmentation f+ . Bayes theorem indicates that each kth likelihood can be approximated as p ( yk|f+ ) ≈ Zk qk ( f+ ) pk ( f+ ) = Zk ( ( ( ( ( ( p ( f+6=uk |uk ) qk ( uk ) ( ( ( ( ( ( p ( f+6=uk |uk ) pk ( uk ) = Zk qk ( uk ) pk ( uk ) , ( 4 ) since qk ( f+ ) ≈ p ( f+|yk ) . Here , Zk refers to the normalization constant or evidence of Dk , which is still difficult to compute . We also assumed that the augmented approximate density factorises as qk ( f+ ) = p ( f+6=uk |uk ) qk ( uk ) , using the structure of the GP prior pk ( f+ ) , as in the proposition of Titsias ( 2009a ) . Similar approximations based on the stochastic process conditionals were previously used in Bui et al . ( 2017 ) and Matthews et al . ( 2016 ) , with emphasis on the theoretical consistency of augmentation . Importantly , all parameters and variables needed for the approximation in ( 4 ) are given , and stored in the kth moduleMk . Then , the nested conditional expectation in ( 3 ) turns out to be Ep ( f+6=u∗ |u∗ ) [ log p ( y|f+ ) ] ≈ K∑ k=1 Ep ( uk|u∗ ) [ logZk qk ( uk ) pk ( uk ) ] , where we applied properties of Gaussian marginals to reduce the large-dimensional expectation . For instance , the integral ∫ p ( f+ ) df+6=uk is analogous to ∫ p ( f+6=uk , uk ) df+6=uk=p ( uk ) via marginalisation . Variational contrastive expectations . The introduction of K expectation terms over the log-ratios given the likelihood approximations in ( 4 ) , leads to particular advantages . Having a nested integration in ( 3 ) , first over u∗ given the variational density q ( u∗ ) , and second over uk for the log-ratio qk ( uk ) /pk ( uk ) , we can exploit the GP predictive equation to write down K∑ k=1 Eq ( u∗ ) [ Ep ( uk|u∗ ) [ logZk qk ( uk ) p ( uk ) ] ] = K∑ k=1 EqC ( uk ) [ logZk qk ( uk ) pk ( uk ) ] , ( 5 ) where we obtained qC ( uk ) via the integral qC ( uk ) = ∫ q ( u∗ ) p ( uk|u∗ ) du∗ , that coincides with the approximate predictive GP posterior . The computation can be obtained analytically for each kth subset uk using the following expression qC ( uk ) = N ( uk|K > ∗kK−1∗∗µ∗ , Kkk + K > ∗kK−1∗∗ ( S∗ −K∗∗ ) K−1∗∗K∗k ) , where , once again , φ∗ = { µ∗ , S∗ } are the global variational parameters that we aim to learn . One important detail of the sum of expectations in ( 5 ) is that it works as an average contrastive indicator that measures how well the global q ( u∗ ) is being fitted to the modular approximations qk ( uk ) . Without the need of revisiting any data , the GP predictive qC ( uk ) is playing a different role in contrast with the usual one . Typically , we assume the approximate posterior fixed and fitted , and we evaluate its performance on some test data points . In this case , it goes in the opposite way , the approximate variational distribution is unfixed , and it is instead evaluated over the inducing inputs Zk provided by each kth moduleMk . Module-driven lower bound . We are now able to simplify the initial bound in ( 3 ) by substituting the first term with the contrastive expectations presented in ( 5 ) . This substitution gives us an initial version of the lower bound under the log-marginal likelihood of the GP , where no data intervene , log p ( y ) ≥ K∑ k=1 logZk + K∑ k=1 EqC ( uk ) [ log qk ( uk ) − log pk ( uk ) ] − KL [ q ( u∗ ) ||p ( u∗ ) ] . However , since normalization constants { Zk } Kk=1 are difficult to compute , this immediately implies that the bound is not well defined without them . We therefore , make use of the information provided by the modules . One can show that using the alternative construction of variational lower bounds on sparse GP models , the constant satisfies logZk = L∗k + KL [ q ( f , uk ) ||p ( f , uk|yk ) ] , where L∗k is obtained from Eq . ( 1 ) . Since we know the KL divergence is greater or equal to zero , and each L∗k per module is a lower bound of logZk , we have that ∑K k=1 L∗k ≤ ∑K k=1 logZk . This allows us to write down a closed-form bound LM ≤ log p ( y ) , LM = K∑ k=1 L∗k + K∑ k=1 EqC ( uk ) [ log qk ( uk ) − log p ( uk ) ] − KL [ q ( u∗ ) ||p ( u∗ ) ] . ( 6 ) The maximisation of ( 6 ) is w.r.t . the parameters φ∗ , the hyperparameters ψ∗ and Z∗ . To assure the positive-definitiness of variational covariance matrices { Sk } Kk=1 and S∗ on both local and global cases , we consider that they all factorize according to the Cholesky decomposition S = LL > . We can then use unconstrained optimization to find optimal values for the lower-triangular matrices L. A priori , the module-driven bound is agnostic with respect to the likelihood model chosen . There is a general derivation in Matthews et al . ( 2016 ) of how stochastic processes and their integral operators are affected by projection functions , that is , different linking mappings of the function f ( · ) to the parameters θ . In such cases , the local lower bounds Lk in ( 1 ) might include expectation terms that are intractable . Since we build the framework to accept any possible data-type , we propose to solve the integrals via Gaussian-Hermite quadratures as in Hensman et al . ( 2015 ) ; Saul et al . ( 2016 ) , and if this is not possible , an alternative would be to apply Monte-Carlo methods .

Summary: A novel approach to learn from multiple tasks (involving different partitions of predictor datasets and different types of target variables) that share the same type of input and domain. The method approximates and bounds the overall (cross-task) log-likelihood with ELBOs and posteriors of each task/module. Using inducing points posteriors, the data in each task can be summarized by inducing points of each task and is not required in later tasks. This is an appealing property in practice since it obviates the need to store all datasets of all tasks, and the training can be performed in a task-distributed fashion.