Learning What Not to Model: Gaussian Process Regression with Negative Constraints

We empirically demonstrate that our GP-NC framework performs better than the traditional GP learning and that our framework does not affect the scalability of Gaussian Process regression and helps the model converge faster as the size of the data increases . Gaussian Process ( GP ) regression fits a curve on a set of datapairs , with each pair consisting of an input point ‘ x ’ and its corresponding target regression value ‘ y ( x ) ’ ( a positive datapair ) . But , what if for an input point ‘ x̄ ’ , we want to constrain the GP to avoid a target regression value ‘ ȳ ( x̄ ) ’ ( a negative datapair ) ? This requirement can often appear in real-world navigation tasks , where an agent would want to avoid obstacles , like furniture items in a room when planning a trajectory to navigate . In this work , we propose to incorporate such negative constraints in a GP regression framework . Our approach , ‘ GP-NC ’ or Gaussian Process with Negative Constraints , fits over the positive datapairs while avoiding the negative datapairs . Specifically , our key idea is to model the negative datapairs using small blobs of Gaussian distribution and maximize its KL divergence from the GP . We jointly optimize the GP-NC for both the positive and negative datapairs . We empirically demonstrate that our GP-NC framework performs better than the traditional GP learning and that our framework does not affect the scalability of Gaussian Process regression and helps the model converge faster as the size of the data increases . 1 INTRODUCTION . Gaussian process are one of the most studied model class for data-driven learning as these are nonparametric , flexible function class that requires little prior knowledge of the process . Traditionally , GPs have found their applications in various fields of research , including Navigation systems ( e.g. , in Wiener and Kalman filters ) ( Jazwinski , 2007 ) , Geostatistics , Meteorology ( Kriging ( Handcock & Stein , 1993 ) ) and Machine learning ( Rasmussen , 2006 ) . The wide range of applications can be attributed to the property of GPs to model the target uncertainty by providing the predictive variance over the target variable . Gaussian process regression in its current construct fits only on a set of positive datapairs , with each pair consisting of an input point and its desired target regression value , to learn the distribution on a functional space . However , in some cases , more information is available in the form of datapairs , where at a particular input point , we want to avoid a range of regression values during the curve fitting of GP . We designate such data as negative datapairs . An illustration where modeling such negative datapairs would be extremely beneficial is given in Fig 1 . In Fig 1 ( b ) , an agent wants to model a trajectory such that it covers all the positive datapairs marked by ‘ x ’ . However , it is essential to note that the agent would run into an obstacle if it models its trajectory based only on the positive datapairs . We can handle this problem of navigating in the presence of obstacles in two ways , one way is to get a high density of positive datapairs near the obstacle , and the other more straightforward approach is to just mark the obstacle as a negative datapair . The former approach would unnecessarily increase the number of positive datapairs for GP to regress . Hence , it may run into scalability issues . However , in the latter approach , if the point is denoted as a negative datapair with a sphere of negative influence around it as illustrated by Fig 1.c , the new trajectory can be modeled with less number of datapairs that accounts for all obstacles on the way . Various GP methods in their current framework lack the ability to incorporate these negative datapairs for the regression paradigm . Contributions : In this paper , we explore the concept of negative datapairs . We provide a simple yet effective GP regression framework , called GP-NC which can fit on the positive datapairs while avoiding the negative datapairs . Specifically , our key idea is to model the negative datapairs using a small Gaussian blob and maximize its KL divergence from the GP . Our framework can be easily incorporated for various types of GP models ( e.g. , exact , SVGP ( Hensman et al. , 2013 ) , PPGPR ( Jankowiak et al. , 2019 ) ) and works well in the scalable settings too . We empirically show in §5 that the inclusion of negative datapairs in training helps with both the increase in accuracy and the convergence rate of the algorithm . 2 REVIEW OF GAUSSIAN PROCESS REGRESSION . We briefly review the basics of Gaussian Process regression , following the notations in ( Wilson et al. , 2015 ) . For more comprehensive discussion of GPs , refer to ( Rasmussen , 2006 ) . A Gaussian process is a collection of random variables , any finite number of which have a joint Gaussian distribution ( Rasmussen , 2006 ) . We consider a dataset D with n D-dimensional input vectors , X = { x1 , · · · , xn } and corresponding n × 1 vector of targets y = ( y ( x1 ) , · · · , y ( xn ) ) T . The goal of GP regression is to learn a function f that maps elements from input space to a target space , i.e. , y ( x ) = f ( x ) + where is i.i.d . noise . If f ( x ) ∼ GP ( µ , kθ ) , then any collection of function values f has a joint multivariate normal distribution given by , f = f ( X ) = [ f ( x1 ) , · · · , f ( xn ) ] T ∼ N ( µX , KX , X ) ( 1 ) with the mean vector and covariance matrix defined by the functions of the Gaussian Process , as ( µX ) i = µ ( xi ) and ( KX , X ) ij = kθ ( xi , xj ) . The kernel function kθ of the GP is parameterized by θ . Assuming additive Gaussian noise , y ( x ) |f ( x ) ∼ N ( y ( x ) ; f ( x ) , σ2 ) , then the predictive distribution of the GP evaluated at the n∗ test points indexed by X∗ , is given by f∗|X∗ , X , y , θ , σ2 ∼ N ( E [ f∗ ] , cov ( f∗ ) ) , E [ f∗ ] = µX∗ +KX∗ , X [ KX , X + σ 2I ] −1 y , cov ( f∗ ) = KX∗ , X∗ −KX∗ , X [ KX , X + σ 2I ] −1 KX , X∗ ( 2 ) KX∗ , X represents the n∗ × n covariance matrix between the GP evaluated at X∗ and X . Other covariance matrices follow similar conventions . µX∗ is the mean vector of size n∗ × 1 for the test points and KX , X is the n × n covariance matrix calculated using the training inputs X . The underlying hyperparameter θ implicitly affects all the covariance matrices under consideration . 2.1 GPS : LEARNING AND MODEL SELECTION We can view the GP in terms of fitting a joint probability distribution as , p ( y , f |X ) = p ( y|f , σ2 ) p ( f |X ) ( 3 ) and we can derive the marginal likelihood of the targets y as a function of kernel parameters alone for the GP by integrating out the functions f in the joint distribution of Eq . ( 3 ) . A nice property of the GP is that this marginal likelihood has an analytical form given by , L ( θ ) = log p ( y|θ , X ) = −1 2 ( yT ( Kθ + σ 2I ) −1 y + log ( ∣∣Kθ + σ2I∣∣ ) +N log ( 2π ) ) ( 4 ) where we have used Kθ as a shorthand for KX , X given θ . The process of kernel learning is that of optimizing Eq . ( 4 ) w.r.t . θ . The first term on the right hand side in Eq . ( 4 ) is used for model fitting , while the second term is a complexity penalty term that maintains the Occam ’ s razor for realizable functions as shown by ( Rasmussen & Ghahramani , 2001 ) . The marginal likelihood involves matrix inversion and evaluating a determinant for n× n matrix , which the naive implementation would require a cubic order of computations O ( n3 ) and O ( n2 ) of storage . Approaches like Scalable Variational GP ( SVGP ) ( Hensman et al. , 2013 ) and parametric GPR ( PPGPR ) ( Jankowiak et al. , 2019 ) have proposed approximations that lead to much better scalability . Please refer to Appexdix A for details . 3 GP REGRESSION WITH NEGATIVE DATAPAIRS . As shown in Fig . 1 , we want the model to avoid certain negative datapairs in its trajectory . In other words , we want the trajectory of the Gaussian Process to have a very low probability of passing through these negative datapairs . In this section , we will first formalize the functional form of the negative datapairs and then subsequently describe our framework called GP-NC regression . 3.1 DEFINITION OF POSITIVE & NEGATIVE DATAPAIRS . Positive datapairs : The set of datapairs through which the GP should pass are defined as positive datapairs . We assume a set of n datapairs ( input , positive target ) with D-dimensional input vectors , X = { x1 , · · · , xn } and corresponding n × 1 vector of target regression values y = { y ( x1 ) , · · · , y ( xn ) } . Negative datapairs : The set of datapairs which the GP should avoid ( obstacles ) are defined as negative datapairs . We assume a set of m datapairs ( input , negative target ) with D-dimensional input vectors X̄ = { x̄1 , · · · , x̄m } and corresponding set of negative targets ȳ = { ȳ ( x̄1 ) , · · · , ȳ ( x̄m ) } . The sample value of GP at input x̄i , given by f ( x̄i ) , should be far from the negative target regression value ȳ ( x̄i ) . Note that it is possible that a particular input x can be in both the positive and negative data pair set . This will happen , when at a particular input we want the GP regression value to be close to its positive target regression value y ( x ) and far from its negative target regression value ȳ ( x ) . 3.2 FUNCTIONAL REPRESENTATION OF NEGATIVE DATAPAIRS . For our framework , we first get a functional representation of the negative datapairs . We define a Gaussian distribution around each of the negative datapair , q ( ȳ|x̄ ) ∼ N ( ȳ ( x̄ ) , σ2neg ) , with mean equal to the negative target value ȳ ( x ) and σ2neg is the variance which is a hyperparameter . The Gaussian blob can also be thought of as the area of influence for the negative datapair with the variance σneg indicating the spread of its influence .

Summary: This paper in concerned with Gaussian process regression under constraints that aim to discourage the model from learning certain values (negative constraints). These are called negative data pairs, and the authors propose an extension to the standard GP methodology to incorporate these constraints in the model. This is done by iteratively training a standard GP and maximising the KL between the GP and blobs of the negative data pairs.