Title: Optimal Rates for Regularized Conditional Mean Embedding Learning
Authors: Zhu Li
Dimitri Meunier
Mattes Mollenhauer
Arthur Gretton
Venue: NeurIPS 2022
Keywords: Kernel Methods
Learning Theory

Abstract: We address the consistency of a kernel ridge regression estimate of the  conditional mean embedding (CME), which is an embedding of the conditional distribution of $Y$ given $X$ into a target reproducing kernel Hilbert space $\mathcal{H}_Y$. The CME allows us to take conditional expectations of target RKHS functions, and has been employed in nonparametric causal and Bayesian inference.
We address the misspecified setting, where the target CME is
in the space of Hilbert-Schmidt operators acting from an input interpolation space between $\mathcal{H}_X$ and $L_2$, to $\mathcal{H}_Y$. This space of operators is shown to be isomorphic to a newly defined vector-valued interpolation space. Using this isomorphism, we derive a novel and adaptive statistical learning rate for the empirical CME estimator under the misspecified setting. Our analysis reveals that our rates match the optimal $O(\log n / n)$ rates without assuming $\mathcal{H}_Y$ to be finite dimensional. We further establish a lower bound on the learning rate, which shows that the obtained upper bound is optimal.


Review discussion:
[Strengths]
1. The presentation of the paper is clear, and estimating the CME is an important problem.
2. The authors provide an operator-theoretic definition of the CME in the misspecified case where the target CME can be well approximated by Hilbert-Schmidt operators.
3. A lower bound on the learning rate is established, which reveals that when the target CME operator is smooth, the obtained upper bound is optimal.
[Weaknesses]
1. It seems that most of the results follow from [8, 39] as well as direct adaption of the interpolation spaces. The background section could be reduced.
2. I am not convinced about the advantages of adapting the interpolation space for approximation. How does the proposed approach compare with related work [39]?
3.
While most of the work is theoretical, it could be useful to provide numerical results to demonstrate the performance on real-life data.
See the main comments.
N/A
No
We thank the reviewer for their time and effort in reading our manuscript. We agree that certain aspects of the analysis make use of techniques from Fischer and Steinwarts (2020) and Talwai et al., (2022), and we have been careful in citing these earlier results where they are used.
There are several key steps that differentiate our paper from Fischer and Steinwarts (2020) and Talwai et al., (2022), which we hope we have made clearer in our revised submission:
First, a real-valued interpolation space is defined in Fischer and Steinwarts (2020), in which the misspecified results were studied.
However, the CME requires a vector-valued interpolation space, which we show is isometric to the Hilbert-Schmidt operator space. This allows us to define in a systematic way what it means to be well-specified and misspecified in the case where the output is itself an RKHS, rather than a scalar as in Fischer and Steinwarts (2020) . This then required significant changes to the proofs of Fischer and Steinwarts (2020) , as detailed in the paper.
In relation to Talwai et al., (2022), we have added a new remark 5, which replaces the discussion at the end of p. 8 in our original submission. There are two important points that distinguish our work: first, our construction avoids the $\beta > p$ requirement of Talwai et al., (2022): this limitation meant that the earlier proof was restricted to “near-well-specified” settings even for reasonable choices of kernel (see the remark for a more rigorous discussion).
Second, we provide a lower bound, which establishes that our result is minimax optimal. No such proof was provided in Talwai et al., (2022).
In respect of numerical applications of conditional mean embeddings, these may be found in some of the cited work, for instance Gr{\"u}new{\"a}lder et al., (2012), Sejdinovic et al., (2013), Song et al., (2013), Song et al., (2010), Song et al., (2009). We focus here on providing a theoretical foundation to support these empirical studies, as was done for the scalar case in Fischer and Steinwart (2020).
Strengths:
- The theoretical analysis is rigorous and improves on the understanding of CME learning in misspecified settings.
- Assumptions are milder and seem easier to satisfy than the ones found in existing theoretical analyses in the literature.
- The first lower bound on the learning rate for the CME estimator is derived.
Weaknesses:
- The derived results do not hold for the case of the constant function, i.e., when $Y$ is independent of $X$, which is a problem in the case of radial basis function and other characteristic kernels popular in kernel embeddings applications.
- The paper is quite dense at some points and a few details get lost.
* I've missed some important details:
- It is not specified whether the samples in the dataset for learning CME estimator need to be independent and identically distributed (i.i.d.) w.r.t. the joint the distribution of $X, Y$ or not. Would the results hold for a non-i.i.d. case?
- The statement of Theorem 2 is a bit confusing regarding $\alpha$. Does Thm. 2 hold for any $\alpha > 0$? Or does it imply there is an $alpha > 0$ such that its result holds?
- It is not clear to me how to get to the formulation of $C_{XX}$ in terms of $(\mu_i)_i$ and $(e_i)_i$ in line 135.
* Minor details:
- In the proof sketch after Theorem 4, are we missing an identity operator next to $\lambda$ in the definition of $F_{Y|X,\lambda}$?
Most limitations are discussed in the paper.
No
We thank the reviewer’s time and effort in providing such encouraging feedback. We address the questions raised by the reviewer below.
For the problem of constant functions, we have included a new discussion in the updated draft (Remark 6) and accompanying Appendix D. This remark discusses the case where $Y$ is independent of $X$ and shows that the constant function does not belong to the space $[\mathcal{G}]^{\beta}$ unless $\beta = 0$, hence consistency in this corner case indeed remains to be established (as we now clarify).
Regarding the denseness of the paper, we have updated our manuscript with the aim of improving clarity. We believe the revised manuscript is easier to follow, but we welcome further suggestions for improvement.
Regarding the questions raised by the reviewer:
In our study, the samples are indeed required to be i.i.d from the joint distribution of $X$ and $Y$. There is interesting work for non-i.i.d. data in the regression setting, see for instance: Hang, Steinwart, “Fast learning from alpha-mixing observations,” Journal of Multivariate Analysis, Volume 127, May 2014, Pages 184-199. The generalization of our result to the non-i.i.d. case will be the focus of future study.
Re Theorem 2: this theorem is no longer needed in the updated manuscript.
C_XX can be derived in the following way
$$C_{XX} = \mathbb{E}(\phi_X(x) \otimes \phi_X(x)) = \mathbb{E}( \sum_i \mu_i e_i(x)e_i(\cdot) \otimes \sum_j \mu_j e_j(x)e_j(\cdot) )  = \sum_i \mu_i \mu_i^{1/2} e_i(\cdot) \otimes \mu_i^{1/2} e_i(\cdot).$$
Thanks for the comment, we have addressed the typo.
Finally, please see the notes above for revisions and improvements we’ve now made to our manuscript.
I'd like to thank the authors for addressing mine and other reviewers' concerns. I believe the paper makes a significant contribution to the theory of kernel mean embeddings in misspecified settings and should be accepted.
Overall the paper is well-written and organized.
Though technical, enough details are provided to be self-contained. The upper bound is interesting as it explicilty involves the mis-specification parameter. The lower bound on learning rate is also seems to fill a gap in existing literature.
Please verify minor typos:
Consider five equations between eqn(14) and line 479. The fourth equation misses a C_YX term.
But it is minor because it is back in the next equation.
Limitations have been discussed (case of low \beta).
No
We thank the reviewer for their kind assessment of our manuscript. The mentioned typo is now fixed in the new version.
In addition, please see the notes above for further revisions and improvements we’ve made to our original submission.
This is a very technical paper and as such I have not verified all the details.
Despite being technical the paper does a great job in being clear and precise about the statements.
Conditional mean embeddings (CME), although used frequently in machine learning, have only recently been defined and studied rigorously. This paper is part of that literature and it makes some valuable contributions. The most important of them is using interpolation space theory to study an estimator for the CME operator and provide optimal learning rates.
Not relevant
No
We thank the reviewer for their encouraging comments. Please see the notes above for revisions and improvements we’ve now made to our manuscript.
Dear Reviewers,
We thank you for your time and effort in reviewing our manuscript, and sincerely appreciate your encouraging feedback and insightful comments.  We reply to reviewer questions in individual responses below.
We have updated our manuscript, incorporating changes made in response to reviewer suggestions,  and with additional improvements that  broaden the scope of the results. A high level description of the latter updates is as follows:
First, we have extended our analysis of the upper bound from the vector-valued interpolation RKHS $\mathcal{G}^{\beta}$ (which maps from an “enlarged” RKHS) to the vector-valued interpolation space $[\mathcal{G}]^{\beta}$ (which maps from subspaces of $L_2$). The proofs follow largely as before (in fact, the Bernstein conditions for the variance proof are now a bit simpler!), however this is more than a cosmetic update, since in certain cases $\mathcal{G}^{\beta}$ might not exist.
Thus the new result is strictly more general. Please refer to Remark 5 in the updated manuscript.
Second, we discovered that our original proof for the lower bound makes an implicit restrictive requirement on the RKHS $\mathcal{H}_Y$, which is that the required Gaussian distributions on  $\mathcal{H}_Y$ must be representable by distributions on  $E_Y$. This requirement arises on line 522 of the original supplementary material. Unfortunately, while the construction is possible in some RKHS  $\mathcal{H}_Y$ it may not hold across all of the infinite dimensional RKHSs $\mathcal{H}_Y$ that interest us, notably the RKHS induced by the exponentiated quadratic kernel.
Fortunately, we found that there is a far easier way to get the required lower bound. We simply perform a scalar regression to $k(y,a)$, rather than directly addressing the problem of predicting the potentially infinite dimensional $\phi_Y(y)$. This straightforwardly gives a lower bound on the original problem (eq. 33 in the revised submission),  which can then be addressed using the scalar regression results of Fischer and Steinwart (2020) and Caponnetto and De Vito (2007). In the revised submission, we cite all the results we need, since many steps now carry across with little or no modification once the original step of eq. 33 has been made.
We hope these revisions will strengthen the paper, and we’d be happy to clarify any further points that might arise.

Meta-review:
role: AC
data: metareview: There is a wide consensus among reviewers, after the discussion period, that this submission has strong and novel results where a clear minimax optimality is established for the conditional mean embedding problem.

For the camera ready version, the authors are strongly encouraged to clearly implement all revisions mentioned in the discussion related to the correctness of the lower bound. 
role: PC

Decision: Accept