Meta Two-Sample Testing: Learning Kernels for Testing with Limited Data
Year: 2021

Abstract:
Modern kernel-based two-sample tests have shown great success in distinguishing complex, high-dimensional distributions by learning appropriate kernels (or, as a special case, classifiers). Previous work, however, has assumed that many samples are observed from both of the distributions being distinguished. In realistic scenarios with very limited numbers of data samples, it can be challenging to identify a kernel powerful enough to distinguish complex distributions. We address this issue by introducing the problem of meta two-sample testing (M2ST), which aims to exploit (abundant) auxiliary data on related tasks to find an algorithm that can quickly identify a powerful test on new target tasks. We propose two specific algorithms for this task: a generic scheme which improves over baselines, and a more tailored approach which performs even better. We provide both theoretical justification and empirical evidence that our proposed meta-testing schemes outperform learning kernel-based tests directly from scarce observations, and identify when such schemes will be successful.

heading: 1 Introduction
text: Two-sample tests ask, “given samples from each, are these two populations the same?” For instance, one might wish to know whether a treatment and control group differ. With very low-dimensional data and/or strong parametric assumptions, methods such as t-tests or Kolmogorov-Smirnov tests are widespread. Recent work in statistics and machine learning has sought tests that cover situations not well-handled by these classic methods [1–18], providing tools useful in machine learning for domain adaptation, causal discovery, generative modeling, fairness, adversarial learning, and more [19–34]. Perhaps the most powerful known widely-applicable scheme is based on a kernel method known as the maximum mean discrepancy (MMD) [1] – or, equivalently [35], the energy distance [3] – when one learns an appropriate kernel for the task at hand [10, 16]. Here, one divides the observed data into “training” and “testing” splits, identifies a kernel on the training data by maximizing a power criterion Ĵ , then runs an MMD test on the testing data (as illustrated in Figure 1a). This method generally works very well when enough data is available for both training and testing.
In real-world scenarios, however, two-sample testing tasks can be challenging if we do not have very many data observations. For example, in medical imaging, we might face two small datasets of lung ∗These authors contributed equally. †Now at Department of Statistics, University of Oxford. ‡Work done partially while at the Toyota Technological Institute at Chicago.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
computed tomography (CT) scans of patients with coronavirus diseases, and wish to know if these patients are affected in different ways. If they are from different distributions, the virus causing the disease may have mutated. Here, previous tests are likely to be relatively ineffective; we cannot learn a powerful kernel to distinguish such complex distributions with only a few observations.
In this paper, we address this issue by considering a problem setting where related testing tasks are available. We use those related tasks to identify a kernel selection algorithm. Specifically, instead of using a fixed algorithm A to learn a kernel (“maximize Ĵ among this class of deep kernels”), we want to learn an algorithm Aθ from auxiliary data (Figure 1b):
arg max Aθ
E(P,Q)∼τ [ Ĵ(SteP , S te Q ;Aθ(S tr P , S tr Q )) ] . (1)
Here P,Q are distributions sampled from a meta-distribution of related tasks τ , which asks us to distinguish P from Q. The corresponding observed sample sets SP, SQ are split into training (tr) and testing (te) components. Aθ is a kernel selection algorithm which, given the two training sets (also called “support sets” in meta-learning parlance), returns a kernel function. Ĵ finally estimates the power of that kernel using the test set (“query sets”). In analogy with meta-learning [36–42], we call this learning procedure meta kernel learning (Meta-KL). We can then apply Aθ to select a kernel on our actual testing task, then finally run an MMD test as before (Figure 1b).
The adaptation performed by Aθ, however, might still be very difficult to achieve with few training observations; even the best Aθ found by a generic adaptation scheme might over-fit to StrP , S tr Q . For more stable procedures and, in our experiments, more powerful final tests, we propose meta multi-kernel learning (Meta-MKL). This algorithm independently finds the most powerful kernel for each related task; at adaptation time, we select a convex combination of those kernels for testing (Figure 1c), as in standard multiple kernel learning [43] and similarly to ensemble methods in few-shot classification [44, 45]. Because we are only learning a small number of weights rather than all of the parameters of a deep network, this adaptation can quickly find a high-quality kernel.
We provide both theoretical and empirical evidence that Meta-MKL is better than generic Meta-KL, and that both outperform approaches that do not use related task data, in low-data regimes. We find that learned algorithms can output kernels with high test power using only a few samples, where “plain” kernel learning techniques entirely fail.

heading: 2 Preliminaries
text: We will now review the setting of learning a kernel for a two-sample test, following [16]. Let X ⊂ Rd and P, Q be (unknown) Borel probability measures on X , with SP = {xi}mi=1 ∼ Pm and SQ = {yj}mj=1 ∼ Qm observed i.i.d. samples from these distributions. We operate in a classical hypothesis testing setup, with the null hypothesis that P = Q. Maximum mean discrepancy (MMD). The basic tool we use is a kernel-based distance metric between distributions called the MMD, defined as follows. (The energy distance [3] is a special case of the MMD for a particular choice of k [35].)
Definition 1 (MMD [1]). Let k : X × X → R be the bounded1 kernel of an RKHS Hk (i.e., supx,y∈X |k(x, y)| <∞). Letting X,X ′ ∼ P and Y, Y ′ ∼ Q be independent random variables,
MMD(P,Q; k) = sup f∈Hk,‖f‖Hk≤1
E[f(X)]− E[f(Y )] = √ E [k(X,X ′) + k(Y, Y ′)− 2k(X,Y )].
If k is characteristic, we have that MMD(P,Q; k) = 0 if and only if P = Q.
We can estimate MMD using the following U -statistic estimator, which is unbiased for MMD2 (denoted by M̂MD 2
u) and has nearly minimal variance among unbiased estimators [1]:
M̂MD 2 u(SP, SQ; k) := 1 m(m− 1) ∑ i 6=j H (k) ij , (2)
H (k) ij := k(xi,xj) + k(yi,yj)− k(xi,yj)− k(yi,xj). (3)
Testing. Under the null hypothesis H0, mM̂MD 2
u converges in distribution as m → ∞ to some distribution depending on P and k [1, Theorem 12]. We can thus build a test with p-value equal to the quantile of our test statistic mM̂MD 2
u under this distribution. Although there are several methods to estimate this null distribution, it is usually considered best [10] to use a permutation test [46, 47]: under H0, samples from P and Q are interchangeable, and repeatedly re-computing the statistic with samples randomly shuffled between SP and SQ estimates its null distribution.
Test power. We generally want to find tests likely to reject H0 when indeed it holds that P 6= Q; the probability of doing so (for a particular P, Q, k and m) is called power. For reasonably large m, Sutherland et al. [10] and Liu et al. [16] argue that the power is an almost-monotonic function of
J(P,Q; k) := MMD2(P,Q; k) σH1(P,Q; k)
, σ2H1(P,Q; k) := 4 ( E [ H (k) ij H (k) i` ] − E [ H (k) ij ]2) . (4)
Here, σ2H1 is the asymptotic variance of √ m M̂MD 2
u under H1; it is defined in terms of an expectation of (3) with respect to the data samples SP, SQ, for i, j, ` distinct. The criterion (4) depends on the unknown distributions; we can estimate it from samples with the regularized estimator [16]
Ĵλ(SP, SQ; k) := M̂MD 2 u(SP, SQ; k) /√ σ̂2H1(SP, SQ; k) + λ, (5)
σ̂2H1(SP, SQ; k) := 4
m3 m∑ i=1  m∑ j=1 H (k) ij 2 − 4 m4  m∑ i=1 m∑ j=1 H (k) ij 2 . (6) Kernel choice. Given two samples SP and SQ, the best kernel is (essentially) the one that maximizes J in (4). If we pick a kernel to maximize our estimate Ĵ using the same data that we use for testing, though, we will “overfit,” and reject H0 far too often. Instead, we use data splitting [2, 5, 10]: we partition the samples into two disjoint sets, SP = StrP ∪ SteP , obtain ktr = A(StrP , StrQ ) ≈ arg maxk Ĵλ(S tr P , S tr Q ; k), then conduct a permutation test based on MMD(S te P , S te Q ; k
tr). This process is summarized in Algorithm 2 and illustrated in Figure 1a.
This procedure has been successfully used not only to, e.g., pick the best bandwidth for a simple Gaussian kernel, but even to learn all the parameters of a kernel like (8) which incorporates a deep network architecture [10, 16]. As argued by Liu et al. [16], classifier two-sample tests [8, 48] (which test based on the accuracy of a classifier distinguishing P from Q) are also essentially a special case of this framework – and more-general deep kernel MMD tests tend to work better. Although presented here specifically for M̂MDu, an analogous procedure has been used for many other problems, including other estimates of the MMD and closely-related quantities [2, 5, 49].
When data splitting, the training split must be big enough to identify a good kernel; with too few training samples mtr, Ĵλ will be a poor estimator, and the kernel will overfit. The testing split,
1For the given expressions to exist and agree, we in fact only need Bochner integrability; this is implied by boundedness of either the kernel or the distribution, but can also hold more generally.
however, must also be big enough: for a given P, Q, and k, it becomes much easier to be confident that MMD(P,Q; k) > 0 as mte grows and the variance in M̂MDu(P,Q; k) accordingly decreases. When the number of available samples is small, both steps suffer. This work seeks methods where, by using related testing tasks, we can identify a good kernel with an extremely small mtr; thus we can reserve most of the available samples for testing, and overall achieve a more powerful test.
Another class of techniques for kernel selection avoiding the need for data splitting is based on selective inference [15]. At least as studied by Kübler et al. [15], however, it is currently available only for restricted classes of kernels and with far-less-accurate “streaming” estimates of the MMD, which for fixed kernels can yield far less powerful tests than M̂MDu [50]. In Section 5.4, we will demonstrate that in our settings, the data-splitting approach is empirically much more powerful.

heading: 3 Meta Two-Sample Testing
text: To handle cases with low numbers of available data samples, we consider a problem setting where related testing tasks are available. We use those tasks in a framework inspired by meta-learning [e.g. 36], where we use those related tasks to identify a kernel selection algorithm, as in (1). Specifically, we define a task as a pair T = (P,Q) of distributions over X we would like to distinguish, and assume a meta-distribution τ over the space of tasks T . Definition 2 (M2ST). Assume we are assigned (unobserved) training tasks T = {T i = (Pi,Qi)}Ni=1 drawn from a task distribution τ , and observe meta-samples S = {(SiP, SiQ)}Ni=1 with SiP ∼ (Pi)ni and SiQ ∼ (Qi)ni . Our goal is to use these meta-samples to find a kernel learning algorithm Aθ, such that for a target task (P′,Q′) ∼ τ with samples SP′ ∼ (P′)n ′ and SQ′ ∼ (Q′)n ′ , the learning algorithm returns a kernel Aθ(SP′ , SQ′) which will achieve high test power on (P′,Q′).
We measure the performance of Aθ based on the expected test power criterion for a target task: J (Aθ, τ) = E(P,Q)∼τ [ EStrP ∼Pm,StrQ ∼Qm [ J(P,Q;Aθ(StrP , StrQ )) ]] . (7)
If τ were in some sense “uniform over all conceivable tasks,” then a no-free-lunch property would cause M2ST to be hopeless. Instead, our assumption is that tasks from τ are “related” enough that we can make progress at improving (7).
By assuming the existence of a meta-distribution τ over tasks, it is promising to learn a general rule across different tasks [36]. Furthermore, we can quickly adapt to a solution to a specific task based on the learned rule [36], which is the reason why researchers have been focusing on meta-learning for several years. Specifically, in the meta two-sample testing, we hope to find “what differences between distributions generally look like” on the meta-tasks, and then at test time, use our very limited data to search for differences in that more constrained set of options. Because the space of candidate rules is more limited, we can (hopefully) find a good rule with a much smaller number of data points. For example, if we have meta-tasks which (like the target task) are determined only by a difference in means, then we want to learn a general rule that distinguishes between two samples by means. For a new task (i.e., new two samples), we hope to identify dimensions where two samples have different means, using only a few data points.
We will propose two approaches to finding an Aθ. Neither is specific to any particular kernel parameterization, but for the sake of concreteness, we follow Liu et al. [16] in choosing the form
kω(x, y) = [(1− )κ(φ(x), φ(y)) + ] q(x, y), (8) where φ is a deep neural network which extracts features from the samples, and κ is a simple kernel (e.g., a Gaussian) on those features, while q is a simple characteristic kernel (e.g. Gaussian) on the input space; ∈ (0, 1] ensures that every kernel of the form kω is characteristic. Here, ω represents all parameters in the kernel:2 most parameters come from deep neural network φ, but κ and q may have a few more parameters (e.g. length scales), and we can also learn .
Meta-KL. We first propose Algorithm 1 as a standard approach to optimizing (7), à la MAML [36]: Aθ takes a small, fixed number of gradient ascent steps in Ĵλ for the parameters of kω, starting
2We use kω when discussing issues relating to the parameters ω. When no ambiguity arises, we use k and kω interchangeably for deep neural network parameterized kernels. If we write only k, then ω still means the parameters of k by default.
Algorithm 1 Meta Kernel Learning (Meta-KL) Input: Meta-samples S = {(SPi , SQi)}Ni=1; kernel architecture (8) and parameters ω0; regularization λ 1. Initialize algorithm parameters: θ := [ωstart]← [ω0] 2. Define a parameterized learning algorithm Aθ(SP, SQ) as:
Algorithm 3 Meta Multi-Kernel Learning (Meta-MKL) Input: Meta-samples S = {(SPi , SQi)}Ni=1; kernel architecture as in (8) for i = 1, 2, . . . , N do
1: Optimize ωi ← ̂argmaxωĴλ(SPi , SQi ; ki) with some approximate optimization algorithm; end for 2: Define K := { ∑N i=1 βiki : β ∈ R N ≥0, ∑ i βi = 1}; # convex combinations of kernels 3: return the algorithm A(SP, SQ) = ̂argmaxk∈KĴλ (SP, SQ; k);
from a learned initialization point ωstart ∈ θ (lines 1-2). We differentiate through Aθ, and perform stochastic gradient ascent to find a good value of θ based on the meta-training sets (lines 3-6, also illustrated in Figure 2). Once we have learned a kernel selection procedure, we can again apply it to a testing task with Algorithm 2.
As we will see in the experiments, this approach does indeed use the meta-tasks to improve performance on target tasks. Differently from usual meta-learning settings, as in e.g. classification [36], however, here it is conceivable that there is a single good kernel that works for all tasks from τ ; improving on this single baseline kernel, rather than simply overfitting to the very few target points, may be quite difficult. Thus, in practice, the amount of adaptation that Aθ actually performs in its gradient ascent can be somewhat limited.
Meta-MKL. As an alternative approach, we also consider a different strategy for Aθ which may be able to adapt with many fewer data samples, albeit in a possibly weaker class of candidate kernels. Here, to select an Aθ, we simply find the best kernel independently for each of the meta-training tasks. Then Aθ chooses the best convex combination of these kernels, as in classical multiple kernel learning [43] and similarly to ensemble methods in few-shot classification [45]. At adaptation time, we only attempt to learn N weights, rather than adapting all of the parameters of a deep network; but, if the meta-training tasks contained some similar tasks to the target task, then we should be able to find a powerful test. This procedure is detailed in Algorithm 3 and illustrated in Figure 1c.

heading: 4 Theoretical analysis
text: We now analyze and compare the theoretical performance of direct optimizing the regularized test power from small sample size with our proposed meta-training procedures. To study our learning objective of approximate test power, we first state the following relevant technical assumptions, [16].
(A) The kernels kω are uniformly bounded as follows. For the kernels we use in practice, ν = 1.
sup ω∈Ω sup x∈X
kω(x, x) ≤ ν.
(B) The possible kernel parameters ω lie in a Banach space of dimension D. Furthermore, the set of possible kernel parameters Ω is bounded by RΩ: Ω ⊆ {ω | ‖ω‖ ≤ RΩ}. (C) The kernel parameterization is Lipschitz: for all x, y ∈ X and ω, ω′ ∈ Ω, |kω(x, y)− kω′(x, y)| ≤ Lk‖ω − ω′‖.
See Proposition 9 of Liu et al. [16] for bounds on these constants when using e.g. kernels of the form (8), in terms of the network architecture.
We will use σ2ω to refer to σ 2 H1 (P,Q; kω) of (4), and analogously for σ̂2ω , for the sake of brevity. Proposition 3 (Direct training with approximate test power, Theorem 6 of [16]). Under Assumptions (A) to (C), suppose ν ≥ 1 is constant, and let Ω̄s ⊆ Ω be the set of ω for which σ2ω ≥ s2. Take the regularized estimate σ̂2ω,λ = σ̂ 2 ω + λ with λ = m −1/3. Then, with probability at least 1− δ,
sup ω∈Ω̄s
∣∣∣Ĵλ(SP, SQ; kω)−J(P,Q; kω)∣∣∣ ≤ ξm = O( 1 s2m1/3 [ 1 s + √ D log(RΩm) + log 1 δ + Lk ]) .
Then, letting k̂ ∈ arg max Ĵλ(SP, SQ; k̂), we have 0 ≤ supk∈K J(P,Q; k)− J(P,Q; k̂) ≤ 2ξm.
Since m is small in our settings, and s may also be small for deep kernel classes as noted by Liu et al. [16], this bound may not give satisfying results.
The key mechanism that drives meta-testing to work, intuitively, is training kernels on related tasks. How do we quantify the relatedness between different testing tasks? Definition 4 (γ-relatedness). Let (P,Q) and (P′,Q′) be the underlying distributions for two different two-sample testing tasks. We say the two tasks are γ-related w.r.t. learning objective J if
sup k∈K |J(P,Q; k)− J(P′,Q′; k)| = γ. (9)
The relatedness measure is a (strong) assumption that two tasks are similar, because all kernels perform similarly on the two tasks, in terms of the approximate test power objective. It also implies the two problems are of similar difficulty, since for small γ, the ability of our MMD test statistics to distinguish the distributions (with optimal kernels) are similar. Definition 5 (Adaptation with Meta-MKL). Given a set of kernels {k1, . . . , kN}, the Meta-MKL adaptation is the kernel k̂ = ∑N i=1 β̂iki, where β̂ = arg maxβ Ĵλne(S tr P , S tr Q ; ∑ i βiki).
This adaptation step uses the same learning objective, Ĵλ, as directly training a deep kernel in Proposition 3 (though with a potentially different regularization parameter, λne).
To analyze the Meta-MKL scheme, we will make the following assumption, which Proposition 26 in Liu et al. [16] shows implies Assumptions (A) to (C) with ν = KRB √ N and Lk = K √ N .
(D) Let {ki}Ni=1 be a set of base kernels, each satisfying supx∈X ki(x, x) ≤ K for some finite constant K. Define the parameterized kernel k as
kβ(x, y) = N∑ i=1 βiki(x, y) (10)
where β ∈ RN , and let B be the set of parameters β such that kβ is positive semi-definite (guaranteed if each βi ≥ 0) and ‖β‖ ≤ RB for some RB <∞.
Theorem 6 (Performance of Meta-MKL). Suppose we have N meta-training tasks {(Pi,Qi)}i∈[N ], with corresponding optimal kernels k∗i ∈ arg maxk∈K J(Pi,Qi; k), and use n samples to learn kernels k̂i ∈ arg maxk∈K Ĵλ(SPi , SQi ; k) in the setting of Proposition 3. Let (P,Q) be a test task, with optimal kernel k∗ ∈ arg maxk∈K J(P,Q; k), from which we observe m samples SP, SQ. Call the Meta-MKL adapted kernel k̂β̂ = ∑ i β̂ik̂i, as in (10), with β̂ found subject to Assumption (D). Let (Pj ,Qj) be a meta-task which is γ-related to (P,Q). Then, with probability at least 1− 2δ, J(P,Q; k∗)− J(P,Q; k̂β) ≤ 2(γ + ξjn + ξm)
where ξjn is the bound of Proposition 3 for learning a kernel on (Pj ,Qj), and ξm is the equivalent bound for multiple kernel learning on (P,Q), which has D = N , RΩ = RB , and Lk = K √ N .
The ξjn term depends on the meta-training sample size n m. With enough (relevant) meta-training tasks (as N grows), γ is expected to go to 0. So, the overall uniform convergence bound is likely to be dominated by the term ξm, giving an overall m−1/3 rate: the same as Proposition 3 obtains for directly training a deep kernel only on (P,Q). This is roughly to be expected; similar optimization objectives are applied for both learning and adaptation, which are limited by sample size m. However, the other components of ξm are likely much smaller than the corresponding parts of ξjn where the kernels are defined by a deep network: the variance must be lower-bounded over a much larger set of kernels, D will be the number of parameters in the network rather than the number of meta-tasks, and the bound on Lk from Proposition 23 of Liu et al. [16] is exponential in the depth of the network. Altogether, we expect MKL adaptation to be much more efficient than direct training.
We also expect that Theorem 6 is actually quite loose. The proof (in Appendix A) decomposes the loss relative to k̂j , picking just a single related kernel; it does not attempt to analyze how combining multiple kernels can improve the test power, because doing so in general seems difficult. Given this limitation, however, we also prove in Appendix A a bound on the adaptation scheme which explicitly only picks the single best kernel from the meta-tasks (Theorem 10), which is of a similar form to Theorem 6 but with the ξm term replaced with one even better.

heading: 5 Experiments
text: Following Liu et al. [16], we compare the following baseline tests with our methods: 1) MMD-D: MMD with a deep kernel whose parameters are optimized; 2) MMD-O: MMD with a Gaussian kernel whose lengthscale is optimized; 3) Mean embedding (ME) test [5, 51]; 4) Smooth characteristic functions (SCF) test [5, 51], and 5) Classifier two-sample tests, including C2ST-S [8] and C2ST-L as described in Liu et al. [16]. None of these methods use related tasks at all, so we additionally consider an aggregated kernel learning (AGT-KL) method, which optimizes a deep kernel of the form (8) by maximizing the value of Ĵλ averaged over all the related tasks in the meta-training set S.
For synthetic datasets, we take a single sample set for StrP and S tr Q , and learn parameters once for each method on that training set. We then evaluate its rejection rate (power or Type-I error, depending on if P = Q) using 100 new sample sets SteP , SteQ . For real datasets, we train on a subset of the available data, then evaluate on 100 random subsets, disjoint from the training set, of the remaining data. We repeat this full process 20 times for synthetic datasets or 10 times for real datasets, and report the mean rejection rate of each test and the standard error of the mean rejection rate. Implementation details are in Appendix B.1; the code is available at github.com/fengliu90/MetaTesting.

heading: 5.1 Results on Synthetic Data
text: We use a bimodal Gaussian mixture dataset proposed by [16], known as high-dimensional Gaussian mixtures (HDGM): P and Q subtly differ in the covariance of a single dimension pair. Here we consider only d = 2, since with very few samples the problem is already extremely difficult. Specifically,
P = 1 2 N ([ 0 0 ] , [ 1 0 0 1 ]) + 1 2 N ([ 0.5 0.5 ] , [ 1 0 0 1 ]) ,
Q(∆h) = 1 2 N ([ 0 0 ] , [ 1 −∆h −∆h 1 ]) + 1 2 N ([ 0.5 0.5 ] , [ 1 ∆h ∆h 1 ]) .

heading: 50 75 100 125 150 175 200 225 250
text: In this paper, our target task is T = (P,Q(0.7)) and meta-samples are drawn from the N = 100 meta tasks T = {T i := (P,Q(0.3 + 0.1× i/N))}Ni=1; note that the target task is well outside the scope of training tasks. To evaluate all tests given limited data, we set the number of training samples (StrP , S tr Q ) to 50, 100, 150 per mode, and the number of testing samples (S te P , S te Q ) from 50 to 250.
Figure 3 illustrates test powers of all tests. Meta-MKL and Meta-KL are the clear winners, with both tests much better when mte is over 100 per mode. It is clear that previous kernel-learning based tests perform poorly due to limited training samples. Comparing Meta-MKL with Meta-KL, apparently, we can obtain much higher power when we consider using multiple trained kernels. Although AGT-KL performs better than baselines, it cannot adapt to the target task very well: it only cares about “in-task” samples, rather than learning to adapt to new distributions. In Appendix B.2, we report the test power of our tests when increasing the number of tasks N from 20 to 150. The results show that that increasing the number of meta tasks will help improve the test power on the target task.

heading: 5.2 Distinguishing CIFAR-10 or -100 from CIFAR-10.1
text: We distinguish the standard datasets of CIFAR-10 and CIFAR-100 [52] from the attempted replication CIFAR-10.1 [53], similar to Liu et al. [16]. Because only a relatively small number of CIFAR10.1 samples are available, it is of interest to see whether by meta-training only on CIFAR-10’s training set (as described in Appendix B), we can find a good test to distinguish CIFAR-10.1, with mtr ∈ {100, 200}. Testing samples (i.e., SteP and SteQ ) are from test sets of each dataset. We report test powers of all tests with 200, 500, 900 testing samples in Table 1 (CIFAR-10 compared to CIFAR-10.1) and Table 2 (CIFAR-100 compared to CIFAR-10.1). Since Liu et al. [16] have shown that CIFAR-10 and CIFAR-10.1 come from different distributions, higher test power is better in both tables. The results demonstrate that our methods have much higher test power than baselines, which is strong evidence that leveraging samples from related tasks can boost test power significantly. Interestingly, C2ST tests almost entirely fail in this setting (as also seen by Recht et al. [53, Appendix B.2.8]); it is hard to learn useful information with only a few data points. In Appendix B.3, we also report results when meta-samples are generated by the training set of CIFAR-100 dataset.

heading: 5.3 Analysis of Closeness between Meta Training and Testing
text: This subsection studies how closeness between related tasks and the target task affects test powers of our tests. Given the target task T in synthetic datasets, we define tasks T with closeness C as
T (C) = {T i := (P,Q((0.6− C) + 0.1× i/N))}Ni=1. (11)
It is clear that T (0) will contain our target task T (i.e., the closeness is zero). We also estimate the γ-relatedness between the target task and T (C), where C ∈ {0.1, 0.2, 0.3, 0.4, 0.5}, and the results show that γ grows roughly linearly with C. Specifically, for C = 0.1, 0.2, 0.3, 0.4, 0.5, the estimate γ̂ is 0.035, 0.067, 0.076, 0.104, 0.134, respectively. (Details can be found in Appendix B.4.)
In Figure 4, we illustrate the test power of our tests when setting closeness C to 0.1, 0.2 and 0.3, respectively. It can be seen that Meta-MKL and Meta-KL outperforms AGT-KL all the figures, meaning that Meta-MKL and Meta-KL actually learn algorithms that can quickly adapt to new tasks. Another phenomenon is that the gap between test powers of meta based KL and AGT-KL will get smaller if the closeness is smaller, which is expected since AGT-KL has seen closer related tasks.

heading: 5.4 Ablation Study
text: In previous sections, we mainly compare with previous kernel-learning tests and have shown that the test power can be improved significantly by our proposed tests. We now show that each component in our tests is effective to improve the test power.
First, we show that MKL can help improve the test power, and the data splitting used in Meta-MKL is much better than using the recent test of Kübler et al. [15]. The comparison has been made in synthetic datasets studied in Section 5.1 and the results can be found in Table 3. Meta-MKL-A is a test that takes all βi = 1/N in Algorithm 3, so that kernels are weighted equally. AGT-MKL uses
multiple kernels in AGT-KL (learning weights like Meta-MKL), and AGT-MKL-A does not learn the weights but just assigns weights 1/N directly to all base kernels. Meta-MKLSI is a kernel two-sample test using the selective inference technique of Kübler et al. [15] rather than data splitting in its Aθ.
From Table 3, we can see that introducing the multiple kernel learning (MKL) scheme substantially improves test power, as it combines useful features learned from base kernels covering different aspects of the problems. Moreover, learning with approximate test power with data-splitting in the meta-setting also outperforms the non-splitting testing procedure MetaMKLSI, since MetaMKLSI requires a linear estimator of MMD. The result also indicates that leveraging related tasks is also important to improve the test power, even though we only need a small set of training samples.
Then, we show that the labels used for constructing meta-samples are useful in the CIFAR datasets. We consider another test here: MMD-D with all CIFAR-10 (MMD-D w/ AC), which runs the MMD-D test using the same sample from CIFAR-10 as did the meta-learning over all tasks together. Compared to Meta-MKL, Meta-KL and AGT-KL, MMD-D w/ AC does not use the label information contained in the CIFAR-10 dataset. The test power of MMD-D w/ AC is shown in Table 4. We can see that the test power of MMD-D w/ AC clearly outperforms MMD-D/MMD-O since MMD-D w/ AC sees more CIFAR-10 data in the training process. It is also clear that our methods still perform much better than MMD-D w/ AC. This result shows that the improvement of our tests does not solely come from seeing more data from CIFAR-10. Instead, the assigned labels for the meta-tasks indeed help.

heading: 6 Conclusions
text: This paper proposes kernel-based non-parametric testing procedures to tackle practical two-sample problems where the sample size is small. By meta-training on related tasks, our work opens a new paradigm of applying learning-to-learn schemes for testing problems, and the potential of very accurate tests in some small-data regimes using our proposed algorithms.
It is worth noting, however, that statistical tests are perhaps particularly ripe for mis-application, e.g. by over-interpreting small marginal differences between sample populations of people to claim “inherent” differences between large groups. Future work focusing on reliable notions of interpretability in these types of tests is critical. Meta-testing procedures, although they yield much better tests in our domains, may also introduce issues of their own: any rejection of the null hypothesis will be statistically valid, but they favor identifying differences similar to those seen before, and so may worsen gaps in performance between “well-represented” differences and rarer ones.
Acknowledgments and Disclosure of Funding
FL and JL are supported by the Australian Research Council (ARC) under FL190100149. WX is supported by the Gatsby Charitable Foundation and EPSRC grant under EP/T018445/1. DJS is supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Canada CIFAR AI Chairs Program. FL would also like to thank Dr. Yanbin Liu and Dr. Yiliao Song for productive discussions.

Review summaries:
State of the art kernel based two-sample tests operator as two-stage procedures. The available data is split, then first a kernel is optimized, and then an MMD test is performed on the other split of the data. With small available data, however, it is hard to reliably learn a good kernel in the first stage. This problem is particularly relevant if the class of kernels is very large, for example deep kernels.  Assuming abundant access to data from related tasks, the paper proposes to pretrain the kernel selection on the related tasks. This meta learning is argued to meaningfully constrain the space of available kernels. Thus for the actual task we can hope to reliably learn a good kernel even with little training data.  The proposed method “Meta multi-kernel learning” essentially learns a kernel for each related task. These then serve as candidate kernels. With the data from the actual problem they simply optimize a linear combination of the candidate kernels. The latter is a standard apprach. Usually, however, the candidate kernels are some default kernels, potentially chosen by expert knowledge. The propsed meta learning approach thus provides a principled and data-driven way to find suitable candidate kernels.  The authors provide some theoretical analysis and include synthetic and real-data experiments to support the benefits of Meta two-sample testing. 

It is a common issue to choose a proper kernel in learning problems or hypothesis testing problems. Learning a kernel is a good way to overcome this issue. However, the learning process needs a lot of data to ensure the effectiveness of the learned kernel. This paper considers another way to learn a kernel, i.e., learning a kernel from similar tasks rather than the target task, and implement this way in the kernel two-sample testing. 

The paper introduces a new way of doing 2 sample testing in the setting where we are given multiple related 2 sample testing tasks and in the setting where we only have limited data. They propose 2 novel methods that allow them to use related information from the "meta training" tasks in order to learn the "best" kernel for 2 sample testing for a new unseen task. The main idea they are trying to stress is how to do 2 sample testing, in the setting where one is only given little data. They show that their 2 proposed method beats standard 2 sample tests by leveraging related tasks. They also provide a theoretical justification and bounds on their method.