Meta Two-Sample Testing: Learning Kernels for Testing with Limited Data 
Authors: Feng Liu
Wenkai Xu
Jie Lu
Danica J. Sutherland
Venue: NeurIPS 2021 Poster (2021)
Keywords: distinguishing distributions
statistical testing
meta-learning
deep kernel methods

Abstract: Modern kernel-based two-sample tests have shown great success in distinguishing complex, high-dimensional distributions by learning appropriate kernels (or, as a special case, classifiers). Previous work, however, has assumed that many samples are observed from both of the distributions being distinguished. In realistic scenarios with very limited numbers of data samples, it can be challenging to identify a kernel powerful enough to distinguish complex distributions. We address this issue by introducing the problem of meta two-sample testing (M2ST), which aims to exploit (abundant) auxiliary data on related tasks to find an algorithm that can quickly identify a powerful test on new target tasks. We propose two specific algorithms for this task: a generic scheme which improves over baselines, and a more tailored approach which performs even better. We provide both theoretical justification and empirical evidence that our proposed meta-testing schemes outperform learning kernel-based tests directly from scarce observations, and identify when such schemes will be successful.

Meta Two-Sample Testing: Learning Kernels for Testing with Limited Data
Feng Liu" Australian Al Institute, UTS feng. liuuuuss..... edu. au
Wenkai Xu" Gatsby Unit, UCL xwk4813ggmail.com com
Jie Lu Australian AI Institute, UTS jie. lwwuts.uuu....o edu. au
Danica J. Sutherland UBC and Amii dsuthêcs .bc.uucc..o ca
Abstract
Modern kemnel-based two-sample tests have shown great success in distinguishing complex, dithemimeniin distributions by learning appropriate kernels (or, as a special case, classifiers). Previous work, however, has assumed that many samples are observed from both of the distributions being distinguished. In realistic scenar- ios with very limited numbers of data samples, it can be challenging to identify a kernel powerful enough to distinguish complex distributions. We address this issue by introducing the problem of meta /wo-sample testing (M2ST), which aims to exploit (abundant) auxiliary data on related tasks to find an algorithm that can quickly identify a powerful test on target tasks. We propose two specific algorithms for this task: a generic scheme which improves over basclines, and a more tailored approach which performs justification and empirical evidence that our proposed meta-testing schemes out- perform learning kernel-base tests directly from scarce observations, and identify when such schemes will successful.
better. We provide both theoretical
Introduction
Two-sample tests ask, "given samples from each, are these two populations the same?" For instance, one might wish to know whether a treatment and control group differ. With very low-dimensional data and/or strong parametric assumptions, methods such as f-tests or Kolmogorov-Smiroo tests are widespread. Recent work in statistics and machine learning has sought tests that cover situations not well-handled by these classic methods [1-18], providing tools useful machine learning for domain adaptation, causal discovery, generative modeling, fairness, adversarial learning, and more [19-34]. Perhaps the most powerful known widely-applicable scheme based on a kernel method known as the maximum mean discrepancy (MMD) [1] - or, equivalently [35], the energy distance [3] when one learns an appropriate kernel for the task at hand [10, 16). Here, one divides the observed data into "training" and "testing" splits, identifies kernel on the training data by maximizing a power criterion J, then runs an MMD test on the testing data illustrated Figure la). This method generally works very well when enough data is available for both training and testing.
In real-world scenarios, however, two-sample testing tasks be challenging if do have very many data observations. For example, in medical imaging, we might face two small datasets of lung
'These authors contributed equally.
Now at Department of Statistics, University of Oxford. Work done partially while the Toyota Technological Institute Chicago.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Samples
Samples from Q
Training Samples
Testing Samples
Meta Samples
Data Spitting
Data Spitting
Data Splitting
Learn weightt
input
input
max
B
outpul
i
MMD Test
Ap
oulput
k
MMD Test
def.
Apply
Metattrain A on reteted nasia
max J
k)
arg max As
MMD Test
6 K
(a) Traditional kemee learning testing
(b)
Meta kernel learning for testing
(c) Meta multi-kernel learning testing
Figure I: Comparison among (a) traditional kernel learning [10. 16), (b) meta kernel learning, and (c) meta multi-kernel learning for kernel two-sample testing, where or h; are the learned kernel.
computed tomography (CT) scans of patients with coronavirus diseases, and wish to know if these patients are affected in different ways. If they are from different distributions, the virus causing the disease may have mutated. Here, previous tests are likely to be relatively ineffective; we cannot learn a powerful kernel to distinguish such complex distributions with only a few observations.
In this paper, we address this issue by considering problem setting where related testing tasks available. We use those related tasks to identify a kernel selection algorithm. Specifically, instead of using fixed algorithm A to learn a kernel ("maximize J among this class of deep kernels"), we want learn algorithm Ag from auxiliary data (Figure 1b):
arg max ((P.Q)=T [ Sh: Ao(S st))
Ag
Here PP, Q are distributions sampled from a meta-distribution of related tasks т, which asks us to distinguish P from Q. The corresponding observed sample sets Sr, Sc are split into training (Ir) and testing (te) components. Ag is kernel selection algorithm which, given the two training sets (also called "support sets" in meta-learning parlance), returns kernel function. j finally estimates the power that kernel using the test set ("query sets"). In analogy with meta- meta-learning [36-42], we call this learning procedure meta kernel learning (Meta-KL, KL). We can then apply Ag to select a kernel on our actual testing task, then finally run an MMD test as before (Figure lb).
The adaptation performed by Ag. however, might still be very difficult to achieve with few training observations; even the best Ag found by generic adaptation scheme might over-fit to sfr S For more stable procedures and, in our experiments, more powerful final tests, we propose meta multi-kernel learning (Meta-MKL). This algorithm independently finds the most powerful kernel for each related task; at adaptation time, we select a convex combination of those kernels for testing (Figure Ic), as in standard multiple kernel learning [43] and similarly ensemble methods in few-shot classification [44, 45]. Because we are only learning a small number of weights rather than all of the parameters of deep network, this adaptation can quickly find a high- kernel.
We provide both thcoretical and empirical evidence that Meta- MKL is better than generic Meta-KL, and that both outperform approaches that do not related task data, in low-data regimes. We find that learned algorithms can output kernels with high test power using only a few samples, where "plain" kernel learning techniques entirely fail.
2 Preliminaries
We will now review the setting of learning kernel for a two-sample test, following [16]. Let X C od and PP, Q be (unknown) Borel probability measures on X, with Sp = {m ~ pm and Sq {u, 1 @ observed i...d. samples from these distributions. We operate in classical hypothesis testing setup, with the null hypothesis that IP = Q
Maximum mean discrepancy (MMD). The basic tool we use is kernel-based distance metric between distributions called the MMD, defined as follows. (The energy distance [3] is a special case of the MMD for particular choice of k [35].)

---

Definition 1 (MMD [T]) Let k. X x X + R be the bounded' kernel of an RKHS Hy (i.e.,
< < 00). Letting X' and Y, Y' Q be independent random variables.
(M Q:
sup fEHL.Ifllm, 21
-
+(Y -
If k is characteristic, we have that MMD(F, Q: k) = 0 if and only if Q
We can estimate MMD using the following U-statistic estimator, which is unbiased for MMD (denoted by MMD,)
2 and has nearly minimal variance among unbiased estimators [1]:
MMD Se; k) = m(m- 1)
Σ
(k)
(2)
1 == k(x,, + k(y. H/) k( y/) x3).
Testing. Under the null hypothesis Su. mMMD,, converges in distribution as m - oo to some distribution depending on IP and lo [1, Theorem 12]. We can thus build test with p-value equal to the
quantile of our test statistic mMMD,, u. under this distribution. Although there are several methods to usually considered best [10] to use a permutation test [46, 47]:
estimate this null distribution, under Ho. samples from IP and Q are interchangeable, and repeatedly re-computing the statistic with samples randomly shuffled between Sp and Sy estimates its null distribution.
Test power. We generally want to find tests likely to reject Ho when indeed it holds that IP & Q; the probability of doing so (for particular IP, Q, and m) is called power. For reasonably large m, Sutherland et al. [10] and Liu et al. [16] argue that the power is an almost-monotonic function of
[(P, Q A3) = MMD (I,, Q: 0s, (P. Q:k)
of (P) = (B H
H H
(k) E [m!]]' (4)
Here, of is the asymptotic variance of /mn MMD under hı; it is defined in terms of an expectation of (3) with respect to the data samples SF, Sq. for i, E distinct. The criterion (4) depends on the unknown distributions; we can estimate it from samples with the regularized estimator [16]
2 (Sp, = MMD (Sp, Siz; k)
V
(Sr, Sqy k)
X,
(5)
2
m
m
6%, (Sp, Soj k)
(K)
(6)
m
m
i=j=1
Kernel choice. Given two samples SF and Sq. the best kernel is (essentially) the one that maximizes J in (4). If we pick a kernel to maximize our estimate j using the same data that we use for testing, though, we will "overfit," " and reject Ho far too often. Instead, we use data splitting [2, 5, 10]: we partition the samples into two disjoint sets, Sp S U Stc obtain h = A(S S R arg max; (x : ke), then conduct a permutation test based on MMD(S St k²)). This process is summarized in Algorithm 2 and illustrated in Figure la.
This procedure has been successfully used not only to, e.g., pick the best bandwidth for a simple Gaussian kernel, but even to learn all the parameters of kernel like (8) which incorporates a deep network architecture [10, 16). As argued by Liu et al. [16), classifier two-sample tests [8, 48] (which test based on the accuracy of classifier distinguishing IP from Q) are also essentially special case of this framework and more-general deep kernel MMD tests tend to work better. Although presented here specifically for MMD,,. .. an analogous procedure has been used for many other problems, including other estimates of the MMD and closely-related quantities [2, 5, 49].
When data splitting, the training split must be big enough to identify a good kernel; with too few training samples m JA will be a poor estimator, and the kernel will overfit. The testing split,
For the given expressions to exist and agree, we in fact only need Bochner integrability; this is implied by boundedness of either the kernel or the distribution, but also hold more generally.
3

---

however, must also be big enough: for given P, Q, and k, it becomes much easier to be confident that (I, Qi k) > 0 as m² grows and the variance in MMD,, Q: k)) accordingly decreases. When the number of available samples is small, both steps suffer. This work seeks methods where, by using related testing tasks, we can identify good kernel with an extremely small m thus we can reserve most of the available samples for testing, and overall achieve more powerful test.
Another class of techniques for kernel selection avoiding the need for data splitting is based on selective inference [15]. At least as studied by Kübler et al. [15], however, it is currently available only for restricted classes of kernels and with far-less-accurate "streaming" estimates of the MMD, which for fixed kernels can yield far less powerful tests than MMD, [50]. In Section 5.4, we will demonstrate that in our settings, the data-splitting approach is empirically much more powerful.
3 Meta Two-Sample Testing
To handle cases with low numbers of available data samples, we consider a problem setting where related testing tasks are available. We use those tasks in framework inspired by meta-learning [e.g. 36], where we use those related tasks to identify a kernel selection algorithm, as in (1). Specifically, we define a lask as pair 'T (P, Q) of distributions over X' we would like to distinguish, and assume distribiititt r over the space of tasks T.
Definition 2 (M2ST). Assume we are assigned (unobserved) training tasks T drawn from a task distribution T, and observe meta- meta-samples S = {(S), S with S (Pr) and S ym Our goal is to use these meta-samples to find kernel learning algorithm As. such that for a target task (P', ( ~ T with samples ST. ~ (P')'' and Soy ~ (Q))) the learning algorithm returns kernel Ao Soy)) which will achieve high test power on (P', (.)
We measure the performance of Ag based on the expected test power criterion for a target task:
J(A.. -) =Ep Esjr "ast"="msst""""" -0 [J Q: As (.St..s()))) s(D)]]]
If were in some sense "uniform over all conceivable tasks," then no-free-lunch property would cause M2ST to be hopeless. Instead, our assumption is that tasks from T are "related" enough that we can make progress at improving (7).
By assuming the existence of a meta-distribution T over tasks, it is promising to learn a general rule across different tasks [36]. Furthermore,
can quickly adapt to a solution to specific task based on the learned rule [36], which is the reason why researchers have been focusing on meta-learning for several years. Specifically, in the meta two-sample testing, we hope find "what differences between distributions generally look like" on the meta-tasks, and then at test time, use our very limited data to search for differences in that more constrained set of options. Because the space of candidate rules is more limited, we can (hopefully) find a good rule with a much smaller number of data points. For example, if we have meta-tasks which (like the target task) are determined only by a difference in means, then we want to learn a general rule that distinguishes between two samples by means. For a new task (i.e., new two samples), we hope to identify dimensions where two samples have different means, using only few data points.
We will propose two approaches to finding Ag. Neither is specific to any particular kernel parameterization, but for the sake of concreteness, we follow Liu et al. [16] in choosing the form
(8) where @ is deep neural network which extracts features from the samples, and is simple kernel (e.g., Gaussian) on those features, while is simple characteristic kernel (e.g. Gaussian) on the input space; E (0, 1] ensures that every kernel of the form k. characteristic. Here, w represents all parameters in the k most parameters come from deep neural network Φ, but k and may have a few more parameters (e.g. length scales), and we can also learn c.
(z, v) = [(1 c)m(o(z), ((v)) + c| qxx,)),
Meta-KL. KL. We first propose Algorithm as standard approach to optimizing (7), à la MAML [36]: Ag takes a small, fixed number of gradient ascent steps in JA for the parameters of k starting
We use k_ when discussing issues relating to the parameters w. When no ambiguity arises, we use and kw interchangeably for deep neural network parameterized kernels. If we write only k. then w still means the parameters of l by default.

---

Algorithm Meta Kernel Learning (Meta-KI.)
Input: sata-ssmples [(Sp., Sqi) } kernel architecture (8) and parameters we: regularization [Waer]] +- [Wb]
1. Initialize algorithm parameters:
2. Define a parameterized learning algorithm Ao(Se, Sa) as:
w - Wammr for 1 I...., Napp do < w + V (Sr. So: kw); end for; return kw
for T= 1,2, T Tax do
3: Sample I as set of indices in {1, 2, N of size Meatch
for E I. do
4: Split data as Spt = S U Spz and Sor St U S
Apply the learning algorithm: &s < As As Sir) end for
6: Update 0 c- + VV ExEz Ju(S 21 S ki):
# update O, i.e. start, to maxinize I (Aa,T)
end for
7: return Ao
Input
Oulput
Mela-Updates
Batch of Meta-Samples
Algorithm 2 Testing with a Kernel Learner
1: Input: Two samples: Sp. So: algorithm Aa
2: Split data as Sp = S U US$ and Se = S U
3: Learn a kernel = AS S5
4: Compute
MMD (S S k);
A
MMD
Aa
MMD
for i = 1,2, 1.2..... Rpeer do
Meta- Updaless
5: Shuffle SF U 5 into X and Y
Compute perm, + (X, end for
(x(D
+4) (D kg)
;
7: Output: p-value
71perns
Eir 1(perm, 2 est);
1083
Figure 2: Illustration of Algorithm 1.
Algorithm 3 Meta Multi-Kerne Learning (Meta-MKL)
Input: Mcta-samples 5 {(Sps, Sox) > kernel architccture as in (8)
for i =
N do
1: Optimize w. <- arg max JA (Sp,, Six kt) with some approximate optimization algorithm:
end for
2: Define = I> 8,k: : 8 E R Σ, B, = 11;
# convex combinations of kernels
3: return the algorithm A(S,, Sq) = arg mäx Sn So:k);
from a learned initialization point Wstart E (lines 1-2). We differentiate through Ag, and perform stochastic gradient ascent to find good value of 0 based the meta-training sets (lines 3-6, also illustrated in Figure 2). Once we have learned a kernel selection procedure, we can again apply it to a testing task with Algorithm 2.
As we will see the experiments, this approach does indeed use the meta-tasks to improve perfor- mance target tasks. Differently from usual meta-learning settings, as in e.g. classification [36], however, here it is conceivable that there is a single good kernel that works for all tasks from т: improving on this single baseline kernel, rather than simply overfitting to the very few target points, may be quite difficult. Thus, in practice, the amount of adaptation that Ag actually performs in its gradient ascent can be somewhat limited.
Meta-MKL. As an alternative approach, we also consider a different strategy for Ag which may be able to adapt with many fewer data samples, albeit in possibly weaker class of candidate kernels. Here, to select an Ag, we simply find the best kernel independently for each of the meta-training tasks. Then Ag chooses the best convex combination of these kernels, as in classical multiple kernel learning [43] and similarly to ensemble methods in few-shot classification [45]. At adaptation time, we only attempt to learn N weights, rather than adapting all of the parameters of a deep network; but, the meta-training tasks contained some similar tasks to the target task, then we should be able to find a powerful test. This procedure is detailed in Algorithm and illustrated in Figure Ic.
5

---

Theoretical analysis
We now analyze and compare the theoretical performance of direct optimizing the regularized test power from small sample size with our proposed meta-training procedures. To study our learning objective of approximate test power, we first state the following relevant technical assumptions, [16].
(A) The kernels hn are uniformly bounded as follows. For the kernels we use in practice, v = 1.
sup sup hw (z, S wEitaEX
(B) The possible kernel parameters lie in Banach space of dimension D. Furthermore, the set of possible kernel parameters Ω is bounded by Ro: Q C {w ||w||| S Ro).
(C) The kernel parameterization is Lipschitz: for all x, y X and w. w/ E Q
(x, y) (x, v)1 S Lu|lw - w'll
See Proposition 9 of Liu al. [16] for bounds these constants when using e.g. kernels of the form (8), in terms of the network architecture.
We will use o to refer to o (P, Qū kw) of (4), and analogously for o for the sake of brevity.
Proposition 3 (Direct training with approximate test power, Theorem 6 of [16]). Under Assump-
tions (A) to (C), suppose 2 1 is constant, and let 22 C Z be the set of for which o s² Take the regularized estimate o2 oā + > with X m-mmm1// -1/3, Then, with probability at least 1 6,
(e:8p,Ssikka kw) Qi kw)) s Em - ( 5
1 1 + V (Dlo(( log 8
sup wElt
Li
Then, letting i E arg max J₁ (Sp, Sig ki), we have S suPREk J(P,,
Q: Since m is small in our settings, and s may also be small for deep kernel classes as noted by Liu et al. [16], this bound may not give satisfying results.
2mm-
The key mechanism that drives meta-testing to work, intuitively, is training kernels on related tasks. How do quantify the relatedness between different testing tasks?
Definition 4 (y-relatedness). Let (P, Q) and (P', Q') be the underlying distributions for two different two-sample testing tasks. We say the two tasks are r--related w.rt. learning objective J if
sup J[P, Q: - J(P', (TP',(/k)) [/=))==>
keK
The relatedness measure is (strong) assumption that two tasks are similar, because all kernels perform similarly on the two tasks, in terms of the approximate test power objective. It also implies the two problems of similar difficulty, since for small y, the ability of our MMD test statistics to distinguish the distributions (with optimal kernels) are similar.
Definition 5 (Adaptation with Meta-MKL), Given a set of kernels (ki,..., KNN, the Meta-MKL adaptation is the kernel & = Σm= Biks, where 3 = arg max. JAM (Str 5 E, B,k.).
This adaptation step the same learning objective, .r. directly training deep kernel in Proposition 3 (though with potentially different regularization parameter, X*)).
To analyze the Meta-MKL scheme, we will make the following assumption, which Proposition 26 in Liu et al. [16] shows implies Assumptions (A) to (C) with v = K KRBVN N and L3 KVN.
(D) Let (k.) IN be a set of base kernels, each satisfying suprex (x( K for some finite constant K. Define the parameterized kernel k as
N
ka v) Σ (,k(
(10)
where 3 R and let B the set of parameters B such that kş positive semi-definite (guaranteed if each 8, 2 0) and ||||| < Ry for some Ry

---

Theorem 6 (Performance of Meta-MKL). Suppose we have N meta-training tasks {(P', O') HEIM with corresponding optimal kernels kg E arg maX (IPi :K;k), and use n samples to learn kernels k; E ang ma (x(Sr,, Sq. k) in the setting of Proposition 3. Let (P, Q) be a test task, with optimal kernel k' E arg ma (P, , Q k). from which we observe m samples Sp, Sq. Call the Meta-MKL adapted kernel kig =
-, >, Beki, as in (10), with 3 found subject to Assumption (D). Let (TPS, (e) be a meta-task which is 7-related to (P, Q). Then, with probability at least 1 26,
J(P. (+) [IP, Q R2) S 2(T + Sh. + Em)
where E is the bound of Proposition for learning a kernel on (IP) Q²), and Em is the equivalent bound for multiple kernel learning on (IP, Q) which has D N, Ra = RB, and T.g = K KvN.
The &d term depends on the meta-trainin sample size n > m. With enough (relevant) meta-trainin tasks (as N grows), y is expected to go to O. So, the overall uniform convergence bound is likely to be dominated by the term Em giving an overall m 1/3 rate: the same Proposition 3 obtains for directly training a deep kernel only on (P, Q) This is roughly to be expected; similar optimization objectives are applied for both learning and adaptation, which are limited by sample size m. However, the other components of Sm are likely much smaller than the corresponding parts of Eú where the kemells are defined by deep network: the variance must be kernels, D will be the number of parameters in the network rather than the number of meta- tasks, and the bound on La from Proposition 23 of Liu et al. [16] is exponential in the depth of the network. Altogether, we expect MKL. adaptation to be much more efficient than direct training.
over a much larger set of
We also expect that Theorem 6 is actually quite loose. The proof (in Appendix A) decomposes the loss relative to ky, picking just a single related kemnl;; it does not attempt to analyze how combining multiple kernels can improve the test power, because doing so in general seems difficult. Given this limitation, however, we also prove in Appendix A a bound on the adaptation scheme which explicitly only picks the single best kernel from the meta-tasks (Theorem 10), which is of a similar form to Theorem 6 but with the f. term replaced with one even better.
5 Experiments
Following Liu et al. [16], we compare the following baseline tests with our methods: 1) MMD-D:
MMD with a deep kernel whose parameters are optimized; 2) MMD-O: MMD with a Gaussian kernel whose lengthscale is optimized: 3) Mean embedding (ME) test [5. 51]: 4) Smooth characteristic functions (SCF) test [5, 51], and 5) Classifier two-sample tests, including C2ST-S [8] and C2ST-L as described in Liu et al. [16]. None of these methods use related tasks at all, so we additionally consider an aggregated kernel learning (AGT-KL) method, which optimizes a deep kernel of the
form (8) by maximizing the value of J2 averaged over all the related tasks in the meta-training set S.
For synthetic datasets, we take single sample set for Str and Str, and learn parameters once for each method on that training set. We then evaluate its rejection rate (power or Type-I error, depending on if P Q using 100 new sample sets Stc Ste. Ste. For real datasets, we train data, then evaluate on 100 random subsets, disjoint from the training set, of the remaining data. We repeat this full process 20 times for synthetic datasets or 10 times for real datasets, and report the mean rejection rate of each test and the standard error of the mean rejection rate. Implementation details are in Appendix B.I; the code is available gi thub.. woan/fengliu90/// Maw.e////MMeMeeMiaan.naa.
subset of the available
5.1 Results on Synthetic Data
We use a bimodal Gaussian mixture dataset proposed by [16], known as high-dimensional Gaussian mixtures (HDGM): IP and @ subtly differ in the covariance of single dimension pair. Here we consider only d = 2, since with very few samples the problem is already extremely difficult. Specifically,
1
0
1
9)
0
1
0.5
1
1)
0
P = N
1
1
0
0
2
0.5
0
1
(0)
0
1
1
0..5
((A4) N
2
-A
1
+
2
N
0.5
A
1
7

---

MMD-D
ME
C2ST-S
MMD-O
SCF
C2ST-L
Meta-MKL
Meta-KL
AGT-KL
1.00
1.00
1.00
0.80
0.80
Average tast pow
0.80
Average test
0.60
Average test
0.60
0.60
0.40
0.40
0.40
0.20
0.20
0.20
0.09 50
100
150
200
250
0.09
1000
150
200
250
0.00
50
100
150
200
250
Samples (Mhe) per mode
Samples (mhi) per mode
Samples (mee] per mode
(a) Test power when Wer 50.
(b) Test power when Mar 100. (c) Test power when Wer 150.
Figure 3: Test power synthetic datasets for = 0.05. Average test power when increasing the number of testing samples mie while only using the 50 (a). 100 (b) and 150 (c) training samples per mode (i.e., mar 50, 100, 150). Shaded regions show standard errors for the mean.
In this paper, our target task is T = (IP, Q(0.7)) and meta-samples are drawn from the N 100 meta tasks T = (T' == (P, (0.3 + 0.1 x i/N)) note that the target task is well outside the scope of training tasks. To evaluate all tests given limited data, we the number of training samples (Spr.57 to 50, 100, 150 per mode, and the number of testing samples (S) from 50 to 250.
Figure 3 illustrates test powers of all tests. Meta-MKL and Meta-KL KL are the clear winners, with both tests much better when me is over 100 per mode. is clear that previous kernel-lcarning based tests perform poorly due to limited training samples. Comparing Meta-MKL with Meta-KL. KL, apparently, we can obtain much higher power when we consider using multiple trained kernels. Although AGT-KL performs better than baselines, it cannot adapt to the target task very well: it only cares about "in-task" samples, rather than learning to adapt to new distributions. In Appendix B.2, we report the test power of our tests when increasing the number of tasks N from 20 to 150. The results show that that increasing the number of meta tasks will help improve the test power on the target task.
5.2 Distinguishing CIFAR-10 or -100 from CIFAR-10.1
We distinguish the standard datasets of CIFAR-IO and CIFAR-100 [52] from the attempted replication CIFAR-10.1 [53], similar to Liu et al. [16]. Because only a relatively small number of CIFAR- 10.1 samples are available, it is of interest to see whether by meta-training only on CIFAR-10's training set (as described in Appendix B). we can find a good test to distinguish CIFAR-10.1, with m²' E {100, 200}. Testing samples (i.c., S and 5t() are from test sets of each dataset. We report test powers of all tests with 200, 500, 900 testing samples Table (CIFAR- 10 compared to CIFAR- 10.1) (10.1))al and Table 2 (CIFAR-100 compared to CIFAR-10.1) Since Liu et al. [16] have shown that CIFAR-10 and CIFAR- 10.1 come from different distributions, higher test power is better in both tables. The results demonstrate that our methods have much higher test power than baselines, which is strong evidence that leveraging samples from related tasks can boost test power significantly, Interestingly, C2ST tests almost entirely fail in this setting (as also seen by Recht et al. [53, Appendix B.2.8J); it is hard to learn useful information with only few data points. In Appendix B.3, we also report results when meta-samples generated by the training set of CIFAR-100 dataset.
5.3 Analysis of Closeness between Meta Training and Testing
This subsection studies how closeness between related tasks and the target task affects test powers of our tests. Given the target task T in synthetic datasets, we define tasks T with closeness C' as
T(C) (T = (P. Q C) + i/N)))
(11)
clear that (0)) will contain our target task (i.e., the closeness is zero). We also estimate the g-relatedness between the target task and T(C). where C' E 0., 0.3, 0.4, 0.5), and the results show that 7 grows roughly linearly with C. Specifically, for = 0.1, 0.3, 0.5, the estimate is 0.035, 0.067, 0.076, 0.104 0.134, respectively. (Details can be found in Appendix B.4.)
8

---

Table 1: Test power of tests on CIFAR-I0 vs CIFAR-10.1 given very limited training data (a = 0.05, M+r 100, 200). The mie represents number of samples when testing. Bold represents the highest mean per column.
Mar
Methods
Th 200
mir 500
Mec = 900
Mis 200
Wis 500
Wit 900
ME
0.084:1:0009
0.096:10.016
0.160:20.035
0.104=0.013
0.202=0.020
0.326=200339
0.047:100013
0.037::0:011
0.047::00015
0.026=00009
0.018=0.006
0.026=200012
C2ST-S
0.059=00.009
0.062=0.007
0.059=0.007
0.052=00011
0.054=0.011
0.057=0.008
C2ST-
0.064:1:0.009
0.064:1:0.006
0.063:100007
0.075=00014
0.066:20.011
0.067
MMD.0
0.091110.011
0.141110.009
0.279.10.018
0.004
0.16010.011
0.01910.020
MMD-D
0. 004:2:0007
0.222:10.020
0.418:20.046
0.117:00013
0.226:20.021
0.447
AGT-KL
0.170:10.052
0.457:10.052
0.765:20.045
0.152
0.463-00060
0.058
KL.
0.2.5.10.010
0.671110.026
0.959-10:013
0.2261100015
0.668
0.072.10.006
Metn-MK1.
0.277-10.016
0.728:10.020
0.973-10.008
0.255
0.724
0.993
Table 2: Test power of tests on CIFAR-100 CIFAR-10.1 given very limited training data (a 0.05,
Mer = 100, 200). The mie represents number of samples when testing. Bold represents the highest mean per column.
Mlr = 100
Mer = 200
Methods
200
500
900
200
500
900
ME
0.211+00.200
0.459=00.045
0.751=000054
0,236=0.033
0.512=00006
0.744=0.090
SCF
0.006:1:0.027
0.132:1:0000
0.240:1:00095
0.136:20.036
0.065
0.416:20.114
C
0.064.10.007
0.063-10.010
0.067.10.008
0.024
0.0000
0.005
C2STL
0.009:10.010
0.077:100010
0.075:100010
0.378:200042
0,273:10.032
0.062
MMD.O
0,214.10.012
0.624.10.013
0,970.10.005
0.1991000016
0.614110.017
0.965.10000
MMD-D
0.244:10.011 I
0.644:10.030
0,970=10.010
0.223-10.016
0.627=00.331
0.005
AGT-KI.
0.596:10.044
0.979-10.010
1.000:10.000
0.635-00.038
0.994-00002
1.00000.000
Meta-KL
0,771=:0.188
0.999=10.001
1.000:10.000
0.806=0.017
1.000=0.000
1.000=0.0000
Metn-MK1.
0.820:10.015
1.000:10.000
1.000:10.000
0.018
1.000:0.000
1.000:0.0000
1.00
1.00
1.00
Average test power
0.80
Average test power
0.80
Average power
0.80
0.60
0.60
0.60
0.40
0.40
0.40
0.20
0.00
50
100
150 200
250
0.00
50
100
150
200
250
0.00
50
100
150
200
250
Samples (Ma) mode
Samples (ma) mode
Samples (ma) per mode
(a) Closeness is 0.1.
(b) Closeness 0.2.
(c) Closeness 0.3.
Figure 4: Test power when changing closeness between related tasks and the target task (a 0.05). Average test power when closeness 0.1 (a), 0.2 (b) and 0.3 (c), where only using the 50 training samples per mode (i.e., My 50). Shaded regions show standard errors for the mean.
In Figure 4, we illustrate the test power of our tests when setting closeness C to 0.1, 0.2 and 0.3, respectively. It can be seen that Meta-MKL and Meta-KL outperforms AGT-KL all the figures, meaning that Meta-MKL and Meta-KL actually learn algorithms that can quickly adapt to new tasks. Another phenomenon is that the gap between test powers of meta based KL and AGT-KL will get smaller if the closeness is smaller, which is expected since AGT-KL. has seen closer related tasks.
5.4 Ablation Study
In previous sections, we mainly compare with previous kernel-learning tests and have shown that the test power can be improved significantly by our proposed tests. We now show that each component in our tests is effective to improve the test power.
First, we show that MKL can help improve the test power, and the data splitting used in Meta-MKL is much better than using the recent test of Kübler et al. [15]. The comparison has been made in synthetic datasets studied Section 5.1 and the results can be found in Table Meta-MKI-A is a test that takes all B 1/N in Algorithm 3, so that kernels are weighted equally. AGT-MKL uses

---

Table 3: The test power of various tests on the synthetic dataset, where for data-splitting methods we use only 10 samples to select kernels and 200 to test. Meta-MKL-A, AGT-MKL-A and Meta-MKI-sI use 210 test samples. Bold represents the highest mean.
Tests
MeIn-MKI.
Meta- MKta-MMKL-AA
Meta-KI.
AGT-MKI.
AGT-MKI.--A
AGT-KI.
Meta-MKI.s
Power
0.702
0.701
0.509:10.044
0.364110.016
0.0022
0.253:1..025
0.058:10.008
Table 4: The test power of various tests on the CIFAR-10 10 vs CIFAR-10. task (Mar = 100). Bold represents the highest mean per row.
Tests
Meta MKL
Meta-KL.
AGT-KL
MMD-D w/ AC
MMD-D
MMD-O
200
0.007.10.016
0.245.10.010
0.170.10.032
0.134:10.010
0.104:10.07
0.091.10.011
mic 500
0.728:10.020
0.671:0.026
0.457:000052
0.325::0.028
0. 0.222:10.020
0. 0.009
Mee 900
0.973==00088
0.959=:00013
0.765+:0045
0.745=00.09
0.418:
0.279::0.018
multiple kernels in AGT-KL (learning weights like Meta-MKL), and AGT-MKL-A does not learn the weights but just assigns weights 1/N directly to all base kernels. Meta-MKLsI is a kernel two-sample test using the selective inference technique of Kübler et al. [15] rather than data splitting in its Ag.
From Table 3, we can see that introducing the multiple kernel learning (MKL.) scheme substantially improves test power, as it combines useful features learned from base kernels covering different aspects of the problems. Moreover, learning with approximate test power with data-splitting the meta-setting also outperforms the non-splitting testing procedure MetaMKLsI. since MetaMKLst requires a linear estimator of MMD. The result also indicates that leveraging related tasks is also important to improve the test power, even though we only need a small set of training samples.
Then, we show that the labels used for constructing meta- samples are useful in the CIFAR datasets. We consider another test here: MMD-D with all CIFAR- 10 (MMD-D w/ AC), which runs the MMD-D test using the same sample from CIFAR-10 as did the meta-learning over all tasks together. Compared to Meta-MKI., Meta-KL KL. and AGT-KL,, MMD-D w/ AC does not use the label information contained in the CIFAR- 10 dataset. The test power of MMD-D w/ AC shown in Table We can see that the test power of MMD-D w/ AC clearly outperforms more CIFAR- 10 data the training process. It is also clear that our methods still perform much better than MMD-D w/ AC. This result shows that the improvement of our tests does not solely come from seeing more data from CIFAR-10. Instead, the assigned labels for the meta-tasks indeed help.
MMDDD//MD-O since MMD-D w/ AC sees
Conclusions
This paper proposes kernel-based non- con-parmmttric testing procedures to tackle practical two-sample problems where the sample size is small. By meta-training on related tasks, our work opens a new paradigm of applying learning-to-lear schemes for testing problems, and the potential of very accurate tests in some small-data regimes using our proposed algorithms.
is worth noting, however, that statistical tests are perhaps particularly ripe for mis-application, c.g. by over-interpreting small marginal differences between sample populations of people to claim "inherent" differences between large groups. Future work focusing reliable notions of interpretability in these types of tests is critical. Meta-testing procedures, although they yield much better tests in our domains, may also introduce issues of their own: any rejection of the null hypothesis will be statistically valid, but they favor identifying differences similar to those seen before, and so may worsen gaps in performance between "well-representtd" differences and rarer ones.
Acknowledgments and Disclosure of Funding
FL and JL are supported by the Australian Research Council (ARC) under FL190100149. WX is supported by the Gatsby Charitable Foundation and EPSRC grant under EP/T018445/1 DJS is supported in part by the Natural Sciences and Engincering Research Council of Canada (NSERC) and the Canada CIFAR AI Chairs Program. FL would also like thank Dr. Yanbin Liu and Dr. Yiliao Song for productive discussions.
10

---

References
[1] Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander J. Smola. "A kernel two-sample test." Journal of Machine Learning Research 13 (2012), pp. 723- 773.
[2] Arthur Gretton, Bharath Sriperumbudur, Dino Sejdinovic, Heiko Strathmann, and Massimil- iano Pontil. "Optimal kernel choice for large-scale two-sample tests." NeurIPS. 2012.
[3] Gábor J. Székely and Maria L. Rizzo. "Energy statistics: A class of statistics based on dis-
tances." Journal of Statistical Planning and Inference 143.8 (2013), pp. 1249-1272.
[4] Ruth Heller and Yair Heller. "Multivariate tests of association based univariate tests." NeurIPS. 2016. arXiv: 1603.03418.
[5] Wittawat Jitkrittum, Zoltan Szabo, Kacper Chwialkowski, and Arthur Gretton. "Interpretable distribution features with maximum testing power." NeurIPS. 2016. arXiv: 1605. 06796.
[6] Hao Chen and Jerome Friedman. "A new graph-based two-sample test for multivariate and object data." Journal of the American Statistical Association 112.517 (2017), pp. 399-409. arviv: 1307. 6294..
[7] Debarghya Ghoshdastidar, Maurilio Gutzeit, Alexandra Carpentier, and Ulrike von Luxburg. "Two-sample tests for large random graphs using network statistics." COLT. 2017. arXiv:
1705.06168.
[8] David Lopez-Paz and Maxime Oquab. "Revisiting Classifier Two-Sample Tests." " ICLR. 2017. arXiv: 1610. 06545.
[9] Aaditya Ramdas, Nicolás Gareía Trillos, and Marco Cuturi. "On Wasserstein Two-Sample Testing and Related Families of Nonparametric Tests." Entropy 19.2 (Jan. 2017), p. 47. arXiv: 1509. 02237.
[10] Danica J. Sutherland, Hsiao- Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton. "Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy." ICLR. 2017. arXiv: 1611. 04488.488.
[11] Rui Gao, Liyan Xie, Yao Xie, and Huan Xu. "Robust Hypothesis Testing Using Wasserstein Uncertainty Sets." NewrIPS. 2018. arXiv: 1805. 10611.
[12] Debarghya Ghoshdastidar and Ulike von Luxburg. "Practical Methods for Graph Two-Sample
Testing." NeurIPS. 2018. arXiv: 1811. 12752.
[13] Shang Li and Xiaodong Wang. "Fully Distributed Sequential Hypothesis Testing: Algorithms and Asymptotic Analyses." IEEE Transactions on Information Theory 64.4 (2018), pp. 2742- 2758.
[14] Matthias Kirchler, Shahryar Khorasani, Marius Kloft, and Christoph Lippert. "Two-sample Testing Using Deep Learning." AISTATS. 2020. arXiv: 1910. 06239.
[15] Jonas M. Kübler, Wittawat Jitkrittum, Bernhard Schölkopf. and Krikamol Muandet. "Learning Kernel Tests Without Data Splitting." " NeurIPS. 2020. arXiv: 2006. 02286.
[16] Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J. Sutherland. "Learning Deep Kernels for Non-Parametric Two-Sample Tests." ICML. 2020. arXiv: 2002.
09116.
[17] Xiuyuan Cheng and Yao Xie. "Neural Tangent Kernel Maximum Mean Discrepancy." 2021. arXiv: 2106. 03227.
[18] Jonas M. Kübler, Wittawat Jitkrittum, Bernhard Schölkopf, and Krikamol Muandet. "An Optimal Witness Function for Two-Sample Testing." 2021. arXiv: 05573.
[19] Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Behrnhard Schölkopf. "Domain adaptation with conditional transferable components." ICML. 2016.
[20] Mikolaj Binkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. "Demystifying
MMD GANs." ICL.R 2018. arXiv: 1801. 01401.
[21] Anjin Liu, Jie Lu, Feng Liu, and Guangquan Zhang. "Accumulating regional density dissim- ilarity for concept drift detection in data streams." Pattern Recognition 76 (2018), pp. 256
272.
[22] Jiahua Dong, Yang Cong, Gan Sun, and Dongdong Hou. "Semantic-Transferi Weakly- Supervised Endoscopic Lesions Segmentation." ICCV. 2019. arXiv: 076699
[23] Petar Stojanov, Mingming Gong, Jaime G. Carbonell, and Kun Zhang. "Data-Driven Approach to Multiple-Souree Source Domain Adaptation." AISTATS. 2019.
11

---

[24]
Alberto Cano and Bartosz Krawezyk. "Kappa updated ensemble for drifting data stream mining." Machine Learning 109.1 (2020), pp. 175-218.
[25] Jiahua Dong, Yang Cong, Gan Sun, Yuyang Liu, and Xiaowei Xu. "CSCL: Critical Semantic- Consistent Learning for Unsupervised Domain Adaptation." ECCV. 2020. arXiv: 2008.10464
[26] Luca Oneto, Michele Donini, Giulia Luise, Carlo Ciliberto. Andreas Maurer, and Massimiliano Pontil. "Exploiting MMD and Sinkhorn Divergences for Fair and Transferable Representation Learning." NeurIPS. 2020.
[27] Zhen Fang. Jie Lu, Anjin Liu, Feng Liu, and Guangquan Zhang. "Learning Bounds for Open-Set Learning." ICML. 2021. arXiv: 2106. 15792.
(28) Zhen Fang, Jie Lu, Feng Liu, Junyu Xuan, and Guangquan Zhang. "Open Set Domain Adapta- tion: Theoretical Bound and Algorithm." IEEE Transactions on Neural Networks and Learning Systems 32.10 (2021), pp. 4309-4322
arXiv: 1907. 08375.
[29] Ruize Gao, Feng Liu, Jingfeng Zhang, Bo Han, Tongliang Liu, Niu Gang, and Masashi Sugiyama. "Maximum Mean Discrepancy is Aware of Adversarial Attacks." ICML. 2021.
arXiv: 2 11415.
[30] Yiliao Song, Jie Lu, Anjin Liu, Haiyan Lu, and Guangquan Zhang. "A Segment-Based Drift Adaptation Method for Data Streams." " IEEE Transactions on Neural Networks and Learning Systems Early Access (2021). DOI: 10. 1109 1ew111111NNNAA TTNNSS 2021 . 3062062.
[31] Gan Sun, Yang Cong, Jiahua Dong, Yuyang Liu, Zhengming Ding, and Haibin Yu. "What and How: Generalized Lifelong Spectral Clustering via Dual Memory." IEEE Transactions on Pattern Analysis and Machine Intelligence (2021). DOr: 10. 1009 / 2021 3058852..
[32] Ashraf Tahmasbi, Ellango Jothimurugesan, Srikanta Tirthapura, and Phillip B Gibbons. "Drift- Surf: 10064.
Learning under Concept Drift." ICML. 2021, pp. 10054-
[33] Bahar Taskesen, Man-Chung Yue, Jose H. Blanchet, Daniel Kuhn, and Viet Anh Nguyen. "Sequential Domain Adaptation by Synthesizing Distributionally Robust Experts." ICML. Ed. by Marina Meila and Tong Zhang. 2021. arXiv: 00322.
[34] Li Zhong. Zhen Fang, Feng Liu, Jie Lu, Bo Yuan, and Guangquan Zhang. "How Does the Combined Risk Affect the Performance of Unsupervised Domain Adaptation Approaches?"
AAAI 2021. arXiv: 2101. 2101.01104.
[35] Dino Sejdinovic, Bharath Sriperumbudur, Arthur Gretton, and Kenji Fukumizu. "Equivalence of distance--ased and RKHS-basee statistics in hypothesis testing." " The Annals of Statistics arXiv: 1207.6076.
41.5 (2013). pp. 2263-2291
[36] Chelsea Finn, Pieter Abbeel, and Sergey Levine. "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks." ICML. 2017. arXiv: 1703. 03400.
[37] Jake Snell, Kevin Swersky, and Richard S. Zemel. "Prototypical Networks for Few-shot Learning.' NeurIPS. 2017. arXiv: 1703.05173
[38] Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang. Sung Ju Hwang, and Yi Yang. "Learning to propagate labels: Transductive propagation network for few-shot learning." ICLR. 2019. Xivi 1805. 10002.
[39] Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. " "Attribute Propagation Network for Graph Zero-shot Learning." AAAI. 2020. arXiv: 2009. 11816.
[40] Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, Xuanyi Dong, and Chengqi Zhang. "Isometric Propagation Network for Generalized Zero-shot Learning." ICLR. 2021. arXiv: 2102. 02038.
[41] Yanbin Liu, Juho Lee, Linchao Zhu, Ling Chen, Humphrey Shi, and Yi Yang. "A Multi-Mode Modulator for Multi-Domain Few-Shot Classification." " 2021.
[42] Tean-Francoss Ton, Lucian Chan, Yee Whye Tch, and Dino Sejdinovic. "Noise Contrastive Meta-Learning for Conditional Density Estimation using Kernel Mcan Embeddings." AISTATS.
2021. arXiv: 1906. 02236.
[43] Mehmet Gönen and Ethem Alpaydın. "Multiple kernel learning algorithms." The Journal of Machine Learning Reseurch 12 (2011), pp. 221 1-2268.
[44] Minseop Park, Sachoon Kim, Jungtaek Kim, Yanbin Liu, and Seungjin Choi. "TAEML:
Task-Adaptive Ensemble of Meta-Learners." NewrlPS 2018 Workshop on Meta-learning. 2018.
[45] Nikita Dvomik, Cordelia Schmid, and Julien Mairal. "Selecting Relevant Features from Multi-domain Representation for Few-Shot Classification." " ECCV. Vol. 12355. 2020, pp. 769-
786. arxiv: 2003. 09338.
12

---

[46] Meyer Dwass. "Modified Randomization Tests for Nonparametric Hypotheses." The Annals of Mathematical Statistics 28.1 (Mar. 1957). Pp. 181-187.
[47] V. Alba Fernández, M.D. Jiménez Gamero, and J. Muñoz García. "A test for the two-sample problem based empirical characteristic functions." Computational Statistics & Data Analysis 52.7 (2008), PP. 3730-3748.
[48] Ilmun Kim, Aaditya Ramdas, Aarti Singh, and Larry Wasserman. "Classification Accuracy Proxy for Two Sample Testing." " Annals of Statistics 49.1 (2021). pp. 411-434, arXiv: 1602. 02210.
[49] Wittawat Jitkrittum, Wenkai Xu, Zoltan Szabo, Kenji Fukumizu, and Arthur Gretton. "A linear-time kernel goodness-of-ft test." NeurIPS. 2017. arXiv: 1705.
[50] Aaditya Ramdas, Sashank J. Reddi, Barnabas Poczos, Aarti Singh, and Larry Wasserman. "Adaptivity and Computation-Stattsiic Tradeoff's for Kernel and Distance based High Dimen- sional Two Sample Testing." 2015. arXiv: 1508. $508.00655.
[51] Kacper Chwialkowski, Aaditya Ramdas, Dino Sejdinovic, and Arthur Gretton. "Fast two- sample testing with analytic representations of probability measures." " NeurIPS. 2015. arXiv:
1506. 04725.
[52] Alex Krizhevsky. "Learning Multiple Layers of Features from Tiny Images." 2009, URL: attps://wwee cs. toronto..... awtrrmtt.eduu/ttcciaaaaaaainnnia
[53]
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. "Do ImageNet Classifiers Generalize to ImageNet?" ICML. 2019. arXiv: 1902. 10811.
[54]
Antonio Torralba, Rob Fergus, and William T. Freeman. "80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition." JEEE Transactions Pattern Analysis and Machine Intelligence 30.11 (2008), pp. 1958-1970.
[55] Alec Radford, Luke Metz, and Soumith Chintala. "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks." " ICLR. 2016. arXiv: 1511.0643.
[56] Diederík Kingma and Jimmy Ba. "Adam: Method for Stochastic Optimization." ICLR.
2015, arxiv: 1980.
13


Reviews:
[{"review_id": "t4Ur0o85nrf", "reviewer": "Reviewer_Ye4V", "summary": "State of the art kernel based two-sample tests operator as two-stage procedures. The available data is split, then first a kernel is optimized, and then an MMD test is performed on the other split of the data. With small available data, however, it is hard to reliably learn a good kernel in the first stage. This problem is particularly relevant if the class of kernels is very large, for example deep kernels.\n\nAssuming abundant access to data from related tasks, the paper proposes to pretrain the kernel selection on the related tasks. This meta learning is argued to meaningfully constrain the space of available kernels. Thus for the actual task we can hope to reliably learn a good kernel even with little training data.\n\nThe proposed method “Meta multi-kernel learning” essentially learns a kernel for each related task. These then serve as candidate kernels. With the data from the actual problem they simply optimize a linear combination of the candidate kernels. The latter is a standard apprach. Usually, however, the candidate kernels are some default kernels, potentially chosen by expert knowledge. The propsed meta learning approach thus provides a principled and data-driven way to find suitable candidate kernels.\n\nThe authors provide some theoretical analysis and include synthetic and real-data experiments to support the benefits of Meta two-sample testing.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I will first address the general dimensions and then include a list of comments and questions that I encourage the authors to address in order to improve the paper. Overall I can imagine the method a worthwhile contribution to the NeurIPS community. But not in the current state. I have some concerns regarding the objectiveness of the experiments. Since the paper is mainly supported based on the experiments, at the current stage I am reluctant to support acceptance.\n\n**Originality**:\nThe proposed method is new. The theory is, however, largely build on the results of Liu et al [16]. Also it seems that the work Kirchler et al [14] deserves more explicit discussion. They also propose a method that potentially translates well to a related task. It would be good to discuss the similarities and differences to that method. \n\n**Quality**:\n\n* The theoretical claim (Theorem 7) largely follows from prior work of Liu et al [16] and seems correct.\n* The provided code is not executable (missing requirements.txt or guidance on how to install required packages for example “freqopttest”). I was unable to reproduce the results or playing around with the hyperparameters.\n* The empirical results only highlight cases where the proposed methods outperform existing methods. However, to more thoroughly judge the approach it would be good to understand (and include experiments) that show when exisiting methods are to be preferred. On the same note, what happens if the tasks are *not* related.\n* I think the construction of the meta-tasks in the real application CIFAR should be thoroughly discussed in the main paper. Arguably this is the most critical step for a practicioner.\n\n\n**Clarity**: \n\n* The paper is generally clearly written and easy to understand for readers familiar with the challenges in two-sample testing.\n* the presentation of Meta-KL is not so easy to understand (and the motivation and theoretical treatment not really provided). Do I understand correctly that in case of deep kernel, it essentially optimizes the parameter intialization? If so, maybe it is worth explicitly stating that.\n\n**Significance**:\nThe paper defines a new task (improving two-sample tests with data from related tasks) and proposes a viable solution. However, in the current form I think it will be hard for practitioners to apply. It is not really clear how such situations look like where one has abundance of data of relevant related tasks. The construction of such related task should be discussed much more.\n\n\n## Comments & Questions (please try to address in rebutal or resubmission):\n\n1. CIFAR experiments: \n    * For the meta learning samples from, say, P (the CIFAR1-10 dataset) are used. For a fair comparison, we should also allow the existing methods to see this data. This is indeed possible as at least some of them do not require balanced samples. \n    * I assume that the differences between classes within CIFAR 10 is not the same as the difference between CIFAR-10 and CIFAR-10.1. Thus the Meta-Task are probably not perfect. Can it happen that we overfit to the meta-tasks? That is, if we use to much data and too aggressively train the meta kernels, does the performance decrease again?\n2. Please include a requirement.txt in the code to make it easier to use. The implementation does not provide guidance on how to install required packages (eg. freqoptest). I was not able to run the code.\n3. What is the motivation for including Meta-KL in the paper? For me Meta-MKL seems like a much more reasonable/natural approach and is much easier to grasp. The theory is only provided for the Meta-MKL, the experimental results show much better performance of Meta-MKL, and I think the implementation and application should be more straightforward. Based on this my impression is that the paper might be better without Meta-KL.\n4. Defintion 4: is the maximization of \\tilde{k} defined over all possible kernels? Or is the \\gamma relatedness defined with respect to some class of kernels? Is it trivial that J(P||Q;k) is bounded for all k?\n5. I am struggeling to exactly understand the messge of Theorem 7. What does the supremum mean here? Can you include an intuitive interpretation? Is this essentially guaranteeing, that with the proposed approach we will with high probability find a kernel such that J is close to the optimal J?\n6. Line 196: I don’t think the claimed results on Meta-MKL is actually included in the appendix. Did I miss sth?\n7. Figure 3: How is it possible that the results of Meta-MKL are significantly worse in panel c) than in panel b)? This raises some doubts about the implementation. I would expect that things can only be better if we have more data.\n8. Table 2: Why does the power of MMD-D not improve when using $m_{tr}=200$ as opposed to $m_{tr}=100$. Again this raises some concerns about the implementation of the experiments.\n\n## Minor comments (not relevant for score)\n\n1. Algorithm 3: Why do you define the class of kernels with softmax weights. Simply saying we allow for a convex combination seems a bit easier (and is also used later on). don’t forget do adjust in line 159 accordingly.\n2. Definition 5: The notation seems a bit confusing. What does the index $w$ indicate in the list of base kernels? Also in the maximizatin they are suddenly denotes as $k^{(i)}$. I don’t think $\\lambda^{ne}$ was defined.\n3. Definition 6: related to above. $w$ is used to both denote the linear weights in the combination as well as a subscript of the individual kernels.\n4. line 219: Typo? I think $n$ and $N$ are the same?\n5. Caption of Table 3: there is an “are” too much.\n\n\n### ------ Update after Discussion phase -------- \nI updated the papers score from 5 to 6 - marginally above acceptance threshold.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As argued above the limitations of the approach should be stated more clearly.", "review_text": "", "extra_scores": {"ethical_concerns": "no concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8-10"}, "extra_text": {}}, {"review_id": "ssrEnEOT-nq", "reviewer": "Reviewer_v3Mq", "summary": "The paper introduces a new way of doing 2 sample testing in the setting where we are given multiple related 2 sample testing tasks and in the setting where we only have limited data. They propose 2 novel methods that allow them to use related information from the \"meta training\" tasks in order to learn the \"best\" kernel for 2 sample testing for a new unseen task. The main idea they are trying to stress is how to do 2 sample testing, in the setting where one is only given little data. They show that their 2 proposed method beats standard 2 sample tests by leveraging related tasks. They also provide a theoretical justification and bounds on their method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Reasons to accept:\n-  Propose a new scenario for 2 sample testing in the meta learning setting,  which to the best of my knowledge has not been done before.\n-  Propose a theoretical bound on the error for the meta learning setup (I am not familiar with this and hence i will rely on the other reviewers comments on this, however it does look promising)\n-  Apply a simple MAML style algorithm to learn from related task and show significant improvements on synthetic data\n-  Use an ensemble style algorithm to further improve their proposed method.\n-  Analyse their algorithm on a synthetic dataset where closeness of the meta learning tasks can be measured\n\nReason to reject and clarification:\n-  How does your algorithm depend on N? It would be interested to see if a decrease in N actually decreases the performance of the algorithm and if so by how much. N=100 seems rather arbitrary on the synthetic datasets and hence I would like the authors to comment on this.\n-  What is the N in the CIFAR example? I tried to look up the experiment in the code provided but did not find the relevant info, neither in the appendix nor in the code base (I apologise if i missed this detail). The way i understand it m_tr means the number of samples in the meta training task and not the amount of meta training tasks. In line 219 you mention N for the synthetic datasets but not for the CIFAR dataset\n-  You also mention that one could use the idea of having little data in the scenario of medical imaging in your introduction line 28-33. However your \"real\" example is only on CIFAR dataset and I do not see a straight forward way to apply it on some more real scenarios. If it is straight forward i would ask the authors to provide proof that it actually works on other that CIFAR datasets and maybe consider the medical imaging one that they described in the introduction. \n-  This is might main criticism of this paper as it contains interesting ideas but they do not seem to have any application to test that the method works outside standardised synthetic datasets such as CIFAR. (They do create a new task that has not been done before) If what you said in the intro is true, to motivate your work, why not use that example in the experiments? You might not be able to have access to that data but in that case you would only have 3-5 images per patient no? and not 100+ as in your experiments. In these cases how does you algorithm work? What i want to say is, could you please provide extra motivation why CIFAR example is enough to convince me that it would work on \"real\" medical data as well?\n\n-  I will also be checking out the code and in particular i am curious why no dropout was used in the NN and how the learning rates as well as the number of epochs affect the performance. I assume you used a validation set to determine the num_epochs etc etc. Please comment on the above.\n\nOverall i think the paper is well written and presents a new avenue for meta learning. You provided theoretical bounds as well as good synthetic experiments in terms of testing power. I am more than happy to raise my score if the above concerns have been addressed. And will be looking forward to your rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No societal impact", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "OY1xvsFAUWo", "reviewer": "Reviewer_D4sZ", "summary": "It is a common issue to choose a proper kernel in learning problems or hypothesis testing problems. Learning a kernel is a good way to overcome this issue. However, the learning process needs a lot of data to ensure the effectiveness of the learned kernel. This paper considers another way to learn a kernel, i.e., learning a kernel from similar tasks rather than the target task, and implement this way in the kernel two-sample testing. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1.  The main contribution of this paper is to introduce meta-learning procedures to the field of kernel hypothesis testing. As known in the literature, kernel choice influences the learning performance/test power in the learning/hypothesis testing. To overcome this issue, researchers aim to split data into training and test sets and train a kernel using the training data. However, this solution has a natural drawback: it is not possible to use a lot of data to test, reducing the test power [r1]. The proposed idea overcomes this drawback wisely. \n\n2. The proposed theory demonstrates what assumptions can ensure that a good kernel can be learned from similar tasks. This is a good start to apply meta-learning-like methods in the field of hypothesis testing.\n\n3. Two-sample testing might become another task to test future meta-learning algorithms (e.g., few-shot learning, reinforcement learning). This might increase the significance of meta-learning algorithms, as two-sample testing is a fundamental problem in both statistic and machine learning.\n\nCons:\n\nQ1. Although the proposed idea is novel, there are no new algorithms introduced in this paper. Algorithm 1 is mainly from MAML [r2]. I am not sure which part of MAML should be redesigned to fit the two-sample testing problem.\n\nQ2. In the theory part, gamma-relatedness seems a very important definition. However, there are no illustrations to show what it means in synthetic data.\n\nQ3. In Proposition 3, why is it reasonable to set lambda to be m^{1-/3}? How does the lambda influence the test power?\n\nQ4. It is unclear why s’ should be greater than s. This is the key for Theorem 7.\n\nQ5. Eq. (11) also measures similarity between two tasks. The difference between gamma-relatedness and Eq. (11) is unclear. In addition, hyperparameters should be analyzed.\n\n[r1] Learning Kernel Tests Without Data Splitting, NeurIPS, 2020.\n\n[r2] Model-agnostic meta-learning for fast adaptation of deep networks, ICML, 2017.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main positive aspect of this paper is its inherent novelty: it is a new and unique way of improving two-sample testing by utilizing many related tasks. The corresponding theory also helps to demonstrate that the method is sound. Through thorough discussion with the reviewers, some additional experiments have been provided, and I ask that you please include these in the paper. The main lingering issue amongst the reviewers is the real-world motivation, or lack thereof. Some of the reviewers struggled to find a real-world scenario in which this kind of data would be present. During internal discussions, one reviewer did mention detecting adversarial examples, or changing distributions in streaming data as possible applications, which I thought worth mentioning here in case it is helpful.
Decision: Accept (Poster)