Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors consider the problem of refining low-res (LR) spatial observations using high-res (HR) covariates. 

Basically, the problem is as follows. The analyst observes LR outcome Z, HR covariate X, and LR covariate Y from the process Z=E[f(X)|Y]+e. The analyst wishes to recover the function f.

In more detail: the analyst observes two datasets: D1 and D2. D1 consists of N observations (x_j, y_j) where x_j may be a bag.  D2 consists of M observations (y_j, z_j). The function f is modelled as a function in an RKHS (with Bayesian or frequentist approach).

The authors characterize the Bayesian version of this problem and derive the GP posterior. The authors characterize the frequentist version of this problem and prove minimax optimal finite sample rates.

main_review: Originality: It seems that the estimator previously appears in Hsu and Ramos. Prop 3.2 and Prop 3.3 appear to be new characterizations of the Bayesian approach. Prop 4.1 demonstrates that the estimator is the solution to a 2-stage (frequentist) regression. Theorem 4.2 relates the problem considered in this paper to the nonparametric instrumental variable problem in order to match symbols with an existing theorem.

Quality: The theoretical results are complete and thorough. Why is H_\ell finite dimensional? This seems to be unnecessary when appealing to the quoted theorem. There seems to be a small clash between the theory and empirics. In theory, k is characteristic and ell is finite. In empirics, k and ell are sums of Guassian and Matern kernels, so they are infinite dimensional and not characteristic.

Clarity: It took me a long time to understand the problem set up: what is observed, what is the goal, and what will be estimated. Figure 1 helps, and should be larger. Please improve the introduction to this learning problem. Theorem D.4 should be in the main text since otherwise its assumptions are hidden.

Significance: It is difficult for me to assess the significance of this paper in the spatial observation resolution literature. Within the RKHS literature, the theoretical contributions are modest: the estimator and main theorem are essentially from previous papers, but the Bayesian characterization and connection to nonparametric instrumental variable regression are new and insightful. I think these should be emphasized more. If there is a strong significance for practitioners, I will raise the score.

limitations_and_societal_impact: What assumptions could be violated, and how would that impact conclusions and policy recommendations?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper extends on the work of deconditional mean embeddings (DMEs) and task transformed Gaussian processes (TTGPs) from Hsu and Ramos (2019) from both an application and theoretical stand point.

From a theoretical stand point, the authors developed the framework around deconditional mean embeddings further by three ways: (1) formulating another elegant way to arrive at deconditional posteriors from Hsu and Ramos (2019), (2) establishing deconditional mean operators (DMOs) as vector-valued regressors in a manner that mirrors Grunewalder et al (2012) which highlights its interpretation as a reconstruction operator, and (3) enhancing guarantees on the convergence rate of deconditional mean operators from what was established in Hsu and Ramos (2019) using the setup and results from  Caponnetto and De Vito (2007), Szabó et al (2016), and Singh et al (2019).

From an application stand point, the authors apply DMEs for refining low resolution spatial fields with high resolution information. This is the special case of DMEs and TTGPs where the task dataset consists of collections of low resolution covariates ($\tilde{y}$) and aggregated targets ($\tilde{z}$), and the transformation dataset consists of collections of (another potentially unmatched set of) low resolution covariates ($y$) each matched with bags of high resolution covariates ($^{b}\bf{x}$). The latter constructions regarding the transformation dataset is what makes this a non-trivial special case due to having bagged high resolution covariates, which allows for a slightly different empirical estimator for the cross-covariance operator. This leads to an alternative conditional mean operator (CMO) which they call the conditional mean shrinkage operator. Finally, the authors then apply their work to toy experiments (swiss roll) and downscaling of atmospheric temperature (CMIP6), where to scale it they also derive a variational formulation to approximate the deconditional posterior.

main_review: Overall, the contributions of the paper is well motivated and well developed. The paper presents non-trivial extensions to the original DME paper on a theoretical level, and at the same time successfully showcases a particularly useful type of application for the theory developed through refining low-resolution spatial fields with high-resolution information.

In particular, of all the theoretical contributions, I find the first contribution on an alternative view to arrive at deconditional posteriors using conditional mean processes very simple and elegant. In Hsu and Ramos 2019, this posterior was derived by defining and performing inference on the graphical model of the task transformed Gaussian process. Instead, this paper arrives at the posterior by first considering the conditional mean process constructed by taking the conditional mean of an original latent Gaussian process which we would like to infer. Due to linearity of the conditional mean, the resulting process is also a Gaussian process, and the authors derive their mean and covariance functions. This naturally leads to the notion of a joint Gaussian field between noisy observations of the resulting Gaussian process after the conditional mean and the original latent Gaussian process, which means the deconditional posterior can be easily derived from Gaussian conditioning. The authors show that this recovers the same deconditional posterior, also termed the posterior of the task transformed Gaussian process, from Hsu and Ramos (2019), establishing another elegant view for interpreting deconditional posteriors. An added bonus is that the authors also explicitly kept a non-zero mean function throughout this derivation.

The second and third contributions are also helpful in the sense that it casts the deconditional problem as a reconstruction problem, which provides a different two-staged reconstruction regression view to the two-staged task transformed regression view originally discussed in Hsu and Ramos (2019). The theorems developed for capturing the convergence rate under this reconstruction loss is well appreciated and indeed non-trivial.

The main weakness of the paper predominantly lies in its presentation, which hopefully should be easy to fix. Understandably, the presentation is dense due to the amount of content and contributions the authors are trying to communicate. However this is especially where the structure and presentation of the paper can vastly improve the communication of the core contributions of the paper. Unfortunately, as it currently stands it can be difficult to appreciate the contributions and their relationships all at once especially because they operate at different levels of generality.

For instance, all three theoretical contributions (as summarised in my summary section) are in fact highly general to the theory of deconditional mean embeddings, and not limited to the core application discussed in the paper, which is refining low-resolution spatial fields with high-resolution information. To me, these contributions are of higher significance compared to the specific application of focus, which is a special case (albeit non-trivial) to the original DME and TTGP setup. For the author's benefit, I would suggest to emphasize the former more than the latter, in contrast to how it is currently communicated. At the very least, it seems strange that the title, abstract, and introduction focuses heavily on setting up this specific application, while the majority of the remainder of the paper develops or extends a theorical framework that is much more general than the application aforementioned.

Furthering this point with a specific example, section 3.3 seems quite out of place in that this is the only sub-section in the major contribution sections (sections 3 and 4) where the multi-resolution application setting comes into play. It seems more appropriate to delay this to Section 5 instead when the specific application is discussed, and keep section 3 and 4 focused on the more general theoretical developments.

[Update] In light of the shared concerns on the paper's clarity and presentation which I agree with as initially noted, I have revised my score to give more weight to this concern, although this does not change my overall stance on the paper.
limitations_and_societal_impact: The work is motivated by applications in statistical downscaling with unmatched multi-resolution data. However, its contributions are general in nature in terms of the technique and not limited to this application. In fact, as highlighted in my review, I believe the general theoretical contributions are more significant than the specific application discussed.

The paper does address societal impact in the submission, citing positive benefits in understanding climate phenomenon from unmatched multi-resolution data which can be quite common, as well as potential negative impacts in using this technique to recover sensitive information from low resolution to high resolution where privacy is a concern.

Overall, I agree with the author's self assessment on limitations and societal impact of the paper.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper considers the problem of learning a function 
$$f:\mathcal{X}\to \mathbb{R}$$ from the relationship 
$$g(y) = \int_x f(x) \mathbb{P}[x|y]dx$$
where the training data consists of two datasets. The first set is  $\mathcal{D}_1 = ( x^b_i, y_i )$ where each $x^b_i$ is a bag of points in $\mathcal{X}$ and $y_i$ is a single point in a space $\mathcal{Y}$, these are empirical samples from some distribution $\mathbb{P}[x|y]\mathbb{P}[y]$. The second dataset is of $\mathcal{D}_2 = (\tilde y_j, \tilde z_i)$ with $\tilde y_j \in \mathcal{Y}$ and $\tilde z_i \in \mathbb{R}$ is a set of input-output pairs to from $\tilde z_i = g(\tilde y_i)$.

Learning models for $f(x)$ from aggregate data, (input bag, output) pairs e.g. $( x^b_i, z_i )$ has been studied, similarly the use of  there are intermediate covariates $y_i \in \mathcal{Y}$ with a dataset of the form $( x^b_i, y_i, z_i )$ has been studied. As I undertsand, the novel constributino of this work is the extension to the case where there are two datasets, $( x^b_i, y_i)$ and $(\tilde y_j, \tilde z_j )$ that both contain points from the domain $\mathcal{Y}$ yet they are not the same points, they are mismatched.

The paper proposes a "conditional mean process", a Gaussian process model that interpolates aggregated data as well as the marginalization operater in order to infer the un-aggregated latent function under this above problem setting. 


main_review: I have two major concerns, the overall writing and the novelty of the proposed conditional mean process.

Regarding the Conditional Mean Process

  -  the integral of a Gaussian process is another Gaussian process is a standard result (eg convolution of a GP is another GP), I feel I have seen variations of equation (3) in multiple previous works [1] [2].

  - I believe the following "deconditioning" step is standard (as rightly acknowledged to be Gaussian conditioning) and equations (5), (6) are textbook results.

  - the sparse inducing point variational approach was applied in Hsu and Ramos

  - I believe the main practical novelty mainly lies in using the kernel $l_y$ as a means to model $\mathbb{P}[X|Y]$ away from the training data thereby bridging the gap between the unmatched $y_i$ and $\tilde y_j$, I think this is really cool and the inclusion of VBAGG baseline (where $l_y$ is effectively "switched off") demonstrates the efficacy. However this seems rather unsurprising and rather incremental.

[1] https://arxiv.org/abs/1902.07908

[2] https://ieeexplore.ieee.org/abstract/document/9383978


Overall I found the paper initially rather difficult to read, I felt the mathematical statements were very dense (mainly Section 2.2) with minimal explanation and required multiple passes to follow (I am largely familiar with the notation though not an expert). I had to read the paper multiple times to first acquire an intuitive understanding of the problem definition, I felt there were multiple subtle ambiguities that threw me off course and minor clarifications would have made the paper much easier to read.

Here are some of my personal writing nitpicks.
 -  the first dataset is is introduced as $(x, y)$. This would traditionally imply that $x$ is predicting $y$, i.e. modelling $\mathbb{P}[Y|X]$, Figure 1 seems to imply this as well, however $y$ is an "input" for the output $x^b$, it is used to Monte Carlo approximate $\mathbb{P}[X|Y]$ instead. (I felt the introductory paragraph of Hsu and Ramos [19] was far easier to follow), perhaps using $y$ as function outputs (like [3]) as in  and $z$ (or anything else) as the "middle man" variables.

 - $z = \mathbb{E}[f(X)|Y] + \epsilon$ does not explicitly state what the expectation is over, though it slowly becomes clear that is is $X$.

  - the word "conditional" is used very frequently and I believe it is referring to the _marginal_ over $X$ using $\mathbb{P}[X|Y]$ (which happens to be a conditional distribution although $Y$ could be any parameter). Again upon first reading this led to a lot of ambiguity for me. (I would personally find "de-marginalizing" or "de-aggregation" to be much more intuitive than "de-conditioning")

  - given the two datasets (x, y), (y, z) and the "two stage regression" statements, I was partly expecting deep Gaussian Processes, again, this misinterpretation took a long time to un-misinterpret

  -  a "hands on" example application in the introduction with example $x$, $x$ and $z$ values would help a clear up much speculation from thee start avoiding many misinterpretations and blind alleys, upon first reading I didn't fully understand the problem setting until the experiments.



I have given a low confidence score to reflect my difficulty in reading the paper.


[3] Variational Learning on Aggregate Outputs with Gaussian Processes, Law et.al. NeurIPS 2019

limitations_and_societal_impact: The authors do not appear to explicitly raise possible failure modes, below is one example.

  - if $l_y$ is poorly chosen, this may cause the method to perform worse than VBAGG in matched data setting and worse than naive methods (like the imputation method on line 338) in the unmatched data setting.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a GP model for refining coarse-grained spatial data, which can be used for handling aggregated data that are unmatched spatially and temporally. The authors also develop the two-stage regression algorithm for downscaling. The effectiveness of the proposed model is demonstrated using synthetic and real-world datasets.
main_review: The task of recovering spatial fields from aggregated data is interesting and important. This paper is well-written and technically sound. The use of Conditional and Deconditional Kernel Mean Embeddings to the downscaling problem is novel. The experiments on atmospheric temperature data are encouraging. Accordingly, my opinion tends to accept; but there are some concerns about technical issues.

The detailed comments are as follows.

(1) It seems that the input of the proposed method is limited to data aggregated at grid cells. But, in my understanding, the proposed method allows data aggregated over regions that have various shapes; is that right? If that is the case, it would be helpful to clarify it; this idea may be similar to those of [e.g., 10, R1]. 

Actually, the spatially aggregated data (e.g., disease incidence [7]) are often associated with irregularly shaped regions such as districts. As in [10, R1], can the proposed model handle the effects from the region's size and shape? That could be critical for estimating prediction uncertainty. 

(2) I support you in that GP-based modeling is useful, but it would be better to mention the recently published downscaling methods based on DNN [e.g., R2] and clarify the advantages of the proposed approach.

(3) In [6], learning from aggregated data is more generally formulated. Can the authors consider the extension of the proposed method to other machine learning tasks such as classification and ordinal regression?

(4) Could you please elaborate on the reason why the predictive performance of BAGG-GP and VBAGG is worse than vanilla GP?

[R1] M. T. Smith, M. A. Álvarez, and N. D. Lawrence. Gaussian process regression for binned data. In arXiv e-prints, 2018.

[R2] Vandal, T.; Kodra, E.; Ganguly, S.; Michaelis, A.; Nemani, R.; and Ganguly, A. R. 2017. DeepSD: Generating high resolution climate change projections through single image super-resolution. In KDD, 1663–1672.
limitations_and_societal_impact: Please see comments (1), (2), and (3) in my main review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a novel Gaussian process approach for settings in which we have two datasets, D_1={(x_i, y_i)}_i=1 ^N and D_2=((y_i,z_i))_i=1^M  that are linked by the mediating variable Y. The datasets are not matched, and we want to learn a function from X to Z. In contrast to existing work, the algorithm generalizes to settings in which the resolution of the two datasets differ.
main_review: The paper comes along with a theoretical analysis of the proposed approach and a careful experimental validation. In addition, the paper is accomodated by a github repository that looks well documented and easy to run. While the paper seems to be theoretically sound, I found parts of it quiet confusing. However, this might also be attributed to my rather scarce background knowledge on conditional mean embeddings.

The title and abstract of the paper suggest that the main innovation of the paper is a new GP formulation that allows for different resolutions. However, for large parts of the paper, this seems to not be very relevant. In fact, the down-scaling only enters in Section 3.3 and in the experiments and is not even part of the main notation.

The major contribution of the paper seems to me is more the definition of the conditional mean process which allows for mediating variables in general. Interestingly,  one arrives at a GP formulation including analytical solutions for mean and covariance. This was surprising to me because chaining two Gaussian processes together and marginalizing out the intermediate variable leads in general to non-Gaussian distributions. 

The posterior mean recovers a solution that has been published previously using the deconditional mean embedding [19].
For ease of readibility, I would suggest to move the complete deconditional mean embedding discussion in Section 4 and not scattered between Section 1, Section 2.2 and Section 4. It is not strictly needed for understanding the conditional mean process and distracts from it. 

I also wonder if the paper would not be better split into two separate contributions - one with the general theoretical framework and one with the specific down-scaling application in mind. For me, the two parts seem to be a bit disconnected.

In the experimental section, I would suggest to add a second metric that takes the uncertainty estimates of the GP into account. After all, this is one of the main benefits of the GP-based method. It would also be interesting to compare the uncertainty estimates between the proposed solution and the GP in [19] for the special case that the granularity levels are the same on both datasets.
limitations_and_societal_impact: The societal impact paragraph is sufficient.
review_title: null

Meta-review: 
After the rebuttal and discussion, the consensus among the high-confidence reviewers is that the paper should be accepted. The meta-reviewer agrees. On the theoretical side, the paper makes valuable contributions that clearly go beyond previous work. On the practical side, the application in climate science is valuable.

For the community to fully benefit from the paper, serious efforts need to go into making the paper more accessible. First, the explanation of the problem setup and the background needs to be improved to make the paper accessible to non-experts. In addition to the advice given by the reviewers, which needs to be included, the authors could make use of the appendix to further clarify the background material and previous work. Second, as promised by the authors in the rebuttal, the presentation of the two theoretical contributions to (1) the theory of deconditional mean embeddings and (2) statistical downscaling needs to be improved.



Decision: Accept (Poster)