Deconditional Downscaling with Gaussian Processes

1 Introduction . Spatial observations often operate at limited resolution due to practical constraints . For example , remote sensing atmosphere products [ 2 , 3 , 4 , 5 ] provide measurement of atmospheric properties such as cloud top temperatures and optical thickness , but only at a low resolution . Devising methods to refine low-resolution ( LR ) variables for local-scale analysis thus plays a crucial part in our understanding of the anthropogenic impact on climate When high-resolution ( HR ) observations of different covariates are available , details can be instilled into the LR field for refinement . This task is referred to as statistical downscaling or spatial disaggregation and models LR observations as the aggregation of an unobserved underlying HR field . For example , multispectral optical satellite imagery [ 6 , 7 ] typically comes at higher resolution than atmospheric products and can be used to refine the latter . Statistical downscaling has been studied in various forms , notably giving it a probabilistic treatment [ 8 , 9 , 10 , 11 , 12 , 13 ] , in which Gaussian processes ( GP ) [ 14 ] are typically used in conjunction with a sparse variational formulation [ 15 ] to recover the underlying unobserved HR field . Our approach follows this line of research where we do not observe data from the underlying HR groundtruth field . On the other hand , deep neural network ( DNN ) based approaches [ 16 , 17 , 18 ] study this problem from a different setting , where they often assume that both HR and LR matched observations are available for training . Then , their approaches follow a standard supervised learning setting in learning a mapping between different resolutions . ∗Indicates equal contribution †Department of Statistics , Oxford , UK , OX1 3LB . < siu.chau @ stats.ox.ac.uk , shahine.bouabid @ stats.ox.ac.uk , dino.sejdinovic @ stats.ox.ac.uk > 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , both lines of existing methods require access to bags of HR covariates that are paired with aggregated targets , which in practice might be infeasible . For example , the multitude of satellites in orbit not only collect snapshots of the atmosphere at different resolutions , but also from different places and at different times , such that these observations are not jointly observed . To overcome this limitation , we propose to consider a more flexible mediated statistical downscaling setup that only requires indirect matching between LR and HR covariates through a mediating field . We assume that this additional field can be easily observed , and matched separately with both HR covariates and aggregate LR targets . We then use this third-party field to mediate learning and downscale our unmatched data . In our motivating application , climate simulations [ 19 , 20 , 21 ] based on physical science can serve as a mediating field since they provide a comprehensive spatiotemporal coverage of meteorological variables that can be matched to both LR and HR covariates . Formally , let bx = { x ( 1 ) , . . . , x ( n ) } ⊂ X be a general notation for bags of HR covariates , f : X → R the field of interest we wish to recover and z̃ the LR aggregate observations from the field f . We suppose that bx and z̃ are unmatched , but that there exists mediating covariates y , ỹ ∈ Y , such that ( bx , y ) are jointly observed and likewise for ( ỹ , z̃ ) as illustrated in Figure 1 . We assume the following aggregation observation model z̃ = EX [ f ( X ) |Y = ỹ ] + ε with some noise ε . Our goal in mediated statistical downscaling is then to estimate f given ( bx , y ) and ( ỹ , z̃ ) , which corresponds to the deconditioning problem introduced in [ 1 ] . Motivated by applications in likelihood-free inference and tasktransfer regression , Hsu and Ramos [ 1 ] first studied the deconditioning problem through the lens of reproducing kernel Hilbert space ( RKHS ) and introduced the framework of Deconditional Mean Embeddings ( DME ) as its solution . In this work , we first propose a Bayesian formulation of deconditioning that results into a much simpler and elegant way to arrive to the DME-based estimator of Hsu and Ramos [ 1 ] , using the conditional expectations of f . Motivated by our mediated statistical downscaling problem , we then extend deconditioning to a multi-resolution setup and bridge the gap between DMEs and existing probabilistic statistical downscaling methods [ 9 ] . By placing a GP prior on the sought field f , we obtain a posterior distribution of the downscaled field as a principled Bayesian solution to the downscaling task on indirectly matched data . For scalability , we provide a tractable variational inference approximation and an alternative approximation to the conditional mean operator ( CMO ) [ 22 ] to speed up computation for large multi-resolution datasets . From a theoretical stand point , we further develop the framework of DMEs by establishing it as a two-staged vector-valued regressor with a natural reconstruction loss that mirrors Grünewälder et al . [ 23 ] ’ s work on conditional mean embeddings . This perspective allows us to leverage distribution regression theory from [ 24 , 25 ] and obtain convergence rate results for the deconditional mean operator ( DMO ) estimator . Under mild assumptions , we obtain conditions under which this rate is a minimax optimal in terms of statistical-computational efficiency . Our contributions are summarized as follows : • We propose a Bayesian formulation of the deconditioning problem of Hsu and Ramos [ 1 ] . We establish its posterior mean as a DME-based estimate and its posterior covariance as a gauge of the deconditioning quality . • We extend deconditioning to a multi-resolution setup in the context of the mediated statistical downscaling problem and bridge the gap with existing probabilistic statistical downscaling methods . Computationally efficient algorithms are devised . • We demonstrate that the DMO estimate minimises a two-staged vector-valued regression and derive its convergence rate under mild assumptions , with conditions for minimax optimality . • We benchmark our model against existing methods for statistical downscaling tasks in climate science , on both synthetic and real-world multi-resolution atmospheric fields data , and show improved performance . 2 Background Materials . 2.1 Notations . Let X , Y be a pair of random variables taking values in non-empty sets X and Y , respectively . Let k : X × X → R and ` : Y × Y → R be positive definite kernels . The closure of the span of their canonical feature maps kx : = k ( x , · ) and ` y : = ` ( y , · ) for x ∈ X and y ∈ Y respectively induces RKHSHk ⊆ RX andH ` ⊆ RY endowed with inner products 〈· , ·〉k and 〈· , ·〉 ` . We observe realizations bD1 = { bxj , yj } Nj=1 of bags bxj = { x ( i ) j } nj i=1 from conditional distribution PX|Y=yj , with bag-level covariates yj sampled from PY . We concatenate them into vectors x : = [ bx1 . . . bxN ] > and y : = [ y1 . . . yN ] > . For simplicity , our exposition will use the notation without bagging – i.e . where D1 = { xj , yj } Nj=1 – when the generality of our contributions will be relevant to the theory of deconditioning from Hsu and Ramos [ 1 ] , in Sections 2 , 3 and 4 . We will come back to a bagged dataset formalism in Sections 3.3 and 5 , which corresponds to our motivating application of mediated statistical downscaling . With an abuse of notation , we define feature matrices by stacking feature maps along columns as Φx : = [ kx1 . . . kxN ] and Ψy : = [ ` y1 . . . ` yN ] and we denote Gram matrices as Kxx = Φ > x Φx = [ k ( xi , xj ) ] 1≤i , j≤N and Lyy = Ψ > y Ψy = [ ` ( yi , yj ) ] 1≤i , j≤N . The notation abuse ( · ) > ( · ) is a shorthand for the elementwise RKHS inner products when it is clear from the context . Let Z denote the real-valued random variable stemming from the noisy conditional expectation of some unknown latent function f : X → R , as Z = EX [ f ( X ) |Y ] + ε . We suppose one observes another set of realizations D2 = { ỹj , z̃j } Mj=1 from PY Z , which is sampled independently from D1 . Likewise , we stack observations into vectors ỹ : = [ ỹ1 . . . ỹM ] > , z̃ : = [ z̃1 . . . z̃M ] > and define feature map Ψỹ : = [ ` ỹ1 . . . ` ỹM ] . 2.2 Conditional and Deconditional Kernel Mean Embeddings . Marginal and Joint Mean Embeddings Kernel mean embeddings of distributions provide a powerful framework for representing and manipulating distributions without specifying their parametric form [ 22 , 26 ] . The marginal mean embedding of measure PX is defined as µX : = EX [ kX ] ∈ Hk and corresponds to the Riesz representer of expectation functional f 7→ EX [ f ( X ) ] . It can hence be used to evaluate expectations EX [ f ( X ) ] = 〈f , µX〉k . If the mapping PX 7→ µX is injective , the kernel k is said to be characteristic [ 27 ] , a property satisfied for the Gaussian and Matérn kernels on Rd [ 28 ] . In practice , Monte Carlo estimator µ̂X : = 1N ∑N i=1 kxi provides an unbiased estimate of µX [ 29 ] . Extending this rationale to embeddings of joint distributions , we define CY Y : = EY [ ` Y ⊗ ` Y ] ∈ H ` ⊗ H ` and CXY : = EX , Y [ kX ⊗ ` Y ] ∈ Hk ⊗ H ` , which can be identified with the crosscovariance operators between Hilbert spaces CY Y : H ` → H ` and CXY : H ` → Hk . They correspond to the Riesz representers of bilinear forms ( g , g′ ) 7→ Cov ( g ( Y ) , g′ ( Y ) ) = 〈g , CY Y g′〉 ` and ( f , g ) 7→ Cov ( f ( X ) , g ( Y ) ) = 〈f , CXY g〉k . As above , empirical estimates are obtained as ĈY Y = 1 NΨyΨ > y = 1 N ∑N i=1 ` yi ⊗ ` yi and ĈXY = 1 NΦxΨ > y = 1 N ∑N i=1 kxi ⊗ ` yi . Again , notation abuse ( · ) ( · ) > is a shorthand for element-wise tensor products when clear from context . Conditional Mean Embeddings Similarly , one can introduce RKHS embeddings for conditional distributions referred to as Conditional Mean Embeddings ( CME ) . The CME of conditional probability measure PX|Y=y is defined as µX|Y=y : = EX [ kX |Y = y ] ∈ Hk . As introduced by Fukumizu et al . [ 27 ] , it is common to formulate conditioning in terms of a Hilbert space operator CX|Y : H ` → Hk called the Conditional Mean Operator ( CMO ) . CX|Y satisfies by definition CX|Y ` y = µX|Y=y and C > X|Y f = EX [ f ( X ) |Y = · ] , ∀f ∈ Hk , where C > X|Y de- notes the adjoint of CX|Y . Plus , the CMO admits expression CX|Y = CXY C −1 Y Y , provided ` y ∈ Range ( CY Y ) , ∀y ∈ Y [ 26 , 27 ] . Song et al . [ 30 ] show that a nonparametric empirical form of the CMO writes ĈX|Y = Φx ( Lyy +NλIN ) −1Ψ > y , ( 1 ) where λ > 0 is some regularisation ensuring the empirical operator is globally defined and bounded . As observed by Grünewälder et al . [ 23 ] , since CX|Y defines a mapping from H ` to Hk , it can be interpreted as the solution to a vector-valued regression problem . This perspective enables derivation of probabilistic convergence bounds on the empirical CMO estimator [ 23 , 25 ] . Deconditional Mean Embeddings Introduced by Hsu and Ramos [ 1 ] as a new class of embeddings , Deconditional Mean Embeddings ( DME ) are natural counterparts of CMEs . While CME µX|Y=y ∈ Hk allows to take the conditional expectation of any f ∈ Hk through inner product EX [ f ( X ) |Y = y ] = 〈f , µX|Y=y〉k , the DME denoted µX=x|Y ∈ H ` solves the inverse problem3 and allows to recover the initial function of which the conditional expectation was taken , through inner product 〈EX [ f ( X ) |Y = · ] , µX=x|Y 〉 ` = f ( x ) . The associated Hilbert space operator , the Deconditional Mean Operator ( DMO ) , is thus defined as the operator DX|Y : Hk → H ` such that D > X|Y EX [ f ( X ) |Y = · ] = f , ∀f ∈ Hk . It admits an expression in terms of CMO and cross-covariance operators DX|Y = ( CX|Y CY Y ) > ( CX|Y CY Y ( CX|Y ) > ) −1 provided ` y ∈ Range ( CY Y ) and kx ∈ Range ( CX|Y CY Y C > X|Y ) , ∀y ∈ Y and ∀x ∈ X . A nonparametric empirical estimate of the DMO using datasets D1 and D2 as described above , is given by D̂X|Y = Ψỹ ( A > KxxA +M IM ) −1 A > Φ > x where > 0 is a regularisation term and A : = ( Lyy + NλI ) −1Lyỹ can be interpreted as a mediation operator . Applying the DMO to expected responses z̃ , Hsu and Ramos [ 1 ] are able to recover an estimate of f as f̂ ( x ) = k ( x , x ) A ( A > KxxA +M IM ) −1 z̃ . ( 2 ) Note that since separate samples ỹ can be used to estimate CY Y , this naturally fits a mediating variables setup where x and the conditional means z̃ are not jointly observed .

Summary: The paper considers the problem of learning a function  $$f:\mathcal{X}\to \mathbb{R}$$ from the relationship  $$g(y) = \int_x f(x) \mathbb{P}[x|y]dx$$ where the training data consists of two datasets. The first set is  $\mathcal{D}_1 = ( x^b_i, y_i )$ where each $x^b_i$ is a bag of points in $\mathcal{X}$ and $y_i$ is a single point in a space $\mathcal{Y}$, these are empirical samples from some distribution $\mathbb{P}[x|y]\mathbb{P}[y]$. The second dataset is of $\mathcal{D}_2 = (\tilde y_j, \tilde z_i)$ with $\tilde y_j \in \mathcal{Y}$ and $\tilde z_i \in \mathbb{R}$ is a set of input-output pairs to from $\tilde z_i = g(\tilde y_i)$.  Learning models for $f(x)$ from aggregate data, (input bag, output) pairs e.g. $( x^b_i, z_i )$ has been studied, similarly the use of  there are intermediate covariates $y_i \in \mathcal{Y}$ with a dataset of the form $( x^b_i, y_i, z_i )$ has been studied. As I undertsand, the novel constributino of this work is the extension to the case where there are two datasets, $( x^b_i, y_i)$ and $(\tilde y_j, \tilde z_j )$ that both contain points from the domain $\mathcal{Y}$ yet they are not the same points, they are mismatched.  The paper proposes a "conditional mean process", a Gaussian process model that interpolates aggregated data as well as the marginalization operater in order to infer the un-aggregated latent function under this above problem setting.