Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a new way of doing 2 sample testing in the setting where we are given multiple related 2 sample testing tasks and in the setting where we only have limited data. They propose 2 novel methods that allow them to use related information from the "meta training" tasks in order to learn the "best" kernel for 2 sample testing for a new unseen task. The main idea they are trying to stress is how to do 2 sample testing, in the setting where one is only given little data. They show that their 2 proposed method beats standard 2 sample tests by leveraging related tasks. They also provide a theoretical justification and bounds on their method.
main_review: Reasons to accept:
-  Propose a new scenario for 2 sample testing in the meta learning setting,  which to the best of my knowledge has not been done before.
-  Propose a theoretical bound on the error for the meta learning setup (I am not familiar with this and hence i will rely on the other reviewers comments on this, however it does look promising)
-  Apply a simple MAML style algorithm to learn from related task and show significant improvements on synthetic data
-  Use an ensemble style algorithm to further improve their proposed method.
-  Analyse their algorithm on a synthetic dataset where closeness of the meta learning tasks can be measured

Reason to reject and clarification:
-  How does your algorithm depend on N? It would be interested to see if a decrease in N actually decreases the performance of the algorithm and if so by how much. N=100 seems rather arbitrary on the synthetic datasets and hence I would like the authors to comment on this.
-  What is the N in the CIFAR example? I tried to look up the experiment in the code provided but did not find the relevant info, neither in the appendix nor in the code base (I apologise if i missed this detail). The way i understand it m_tr means the number of samples in the meta training task and not the amount of meta training tasks. In line 219 you mention N for the synthetic datasets but not for the CIFAR dataset
-  You also mention that one could use the idea of having little data in the scenario of medical imaging in your introduction line 28-33. However your "real" example is only on CIFAR dataset and I do not see a straight forward way to apply it on some more real scenarios. If it is straight forward i would ask the authors to provide proof that it actually works on other that CIFAR datasets and maybe consider the medical imaging one that they described in the introduction. 
-  This is might main criticism of this paper as it contains interesting ideas but they do not seem to have any application to test that the method works outside standardised synthetic datasets such as CIFAR. (They do create a new task that has not been done before) If what you said in the intro is true, to motivate your work, why not use that example in the experiments? You might not be able to have access to that data but in that case you would only have 3-5 images per patient no? and not 100+ as in your experiments. In these cases how does you algorithm work? What i want to say is, could you please provide extra motivation why CIFAR example is enough to convince me that it would work on "real" medical data as well?

-  I will also be checking out the code and in particular i am curious why no dropout was used in the NN and how the learning rates as well as the number of epochs affect the performance. I assume you used a validation set to determine the num_epochs etc etc. Please comment on the above.

Overall i think the paper is well written and presents a new avenue for meta learning. You provided theoretical bounds as well as good synthetic experiments in terms of testing power. I am more than happy to raise my score if the above concerns have been addressed. And will be looking forward to your rebuttal.
limitations_and_societal_impact: No societal impact
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: It is a common issue to choose a proper kernel in learning problems or hypothesis testing problems. Learning a kernel is a good way to overcome this issue. However, the learning process needs a lot of data to ensure the effectiveness of the learned kernel. This paper considers another way to learn a kernel, i.e., learning a kernel from similar tasks rather than the target task, and implement this way in the kernel two-sample testing. 
main_review: Pros:

1.  The main contribution of this paper is to introduce meta-learning procedures to the field of kernel hypothesis testing. As known in the literature, kernel choice influences the learning performance/test power in the learning/hypothesis testing. To overcome this issue, researchers aim to split data into training and test sets and train a kernel using the training data. However, this solution has a natural drawback: it is not possible to use a lot of data to test, reducing the test power [r1]. The proposed idea overcomes this drawback wisely. 

2. The proposed theory demonstrates what assumptions can ensure that a good kernel can be learned from similar tasks. This is a good start to apply meta-learning-like methods in the field of hypothesis testing.

3. Two-sample testing might become another task to test future meta-learning algorithms (e.g., few-shot learning, reinforcement learning). This might increase the significance of meta-learning algorithms, as two-sample testing is a fundamental problem in both statistic and machine learning.

Cons:

Q1. Although the proposed idea is novel, there are no new algorithms introduced in this paper. Algorithm 1 is mainly from MAML [r2]. I am not sure which part of MAML should be redesigned to fit the two-sample testing problem.

Q2. In the theory part, gamma-relatedness seems a very important definition. However, there are no illustrations to show what it means in synthetic data.

Q3. In Proposition 3, why is it reasonable to set lambda to be m^{1-/3}? How does the lambda influence the test power?

Q4. It is unclear why s’ should be greater than s. This is the key for Theorem 7.

Q5. Eq. (11) also measures similarity between two tasks. The difference between gamma-relatedness and Eq. (11) is unclear. In addition, hyperparameters should be analyzed.

[r1] Learning Kernel Tests Without Data Splitting, NeurIPS, 2020.

[r2] Model-agnostic meta-learning for fast adaptation of deep networks, ICML, 2017.

limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: State of the art kernel based two-sample tests operator as two-stage procedures. The available data is split, then first a kernel is optimized, and then an MMD test is performed on the other split of the data. With small available data, however, it is hard to reliably learn a good kernel in the first stage. This problem is particularly relevant if the class of kernels is very large, for example deep kernels.

Assuming abundant access to data from related tasks, the paper proposes to pretrain the kernel selection on the related tasks. This meta learning is argued to meaningfully constrain the space of available kernels. Thus for the actual task we can hope to reliably learn a good kernel even with little training data.

The proposed method “Meta multi-kernel learning” essentially learns a kernel for each related task. These then serve as candidate kernels. With the data from the actual problem they simply optimize a linear combination of the candidate kernels. The latter is a standard apprach. Usually, however, the candidate kernels are some default kernels, potentially chosen by expert knowledge. The propsed meta learning approach thus provides a principled and data-driven way to find suitable candidate kernels.

The authors provide some theoretical analysis and include synthetic and real-data experiments to support the benefits of Meta two-sample testing.

main_review: I will first address the general dimensions and then include a list of comments and questions that I encourage the authors to address in order to improve the paper. Overall I can imagine the method a worthwhile contribution to the NeurIPS community. But not in the current state. I have some concerns regarding the objectiveness of the experiments. Since the paper is mainly supported based on the experiments, at the current stage I am reluctant to support acceptance.

**Originality**:
The proposed method is new. The theory is, however, largely build on the results of Liu et al [16]. Also it seems that the work Kirchler et al [14] deserves more explicit discussion. They also propose a method that potentially translates well to a related task. It would be good to discuss the similarities and differences to that method. 

**Quality**:

* The theoretical claim (Theorem 7) largely follows from prior work of Liu et al [16] and seems correct.
* The provided code is not executable (missing requirements.txt or guidance on how to install required packages for example “freqopttest”). I was unable to reproduce the results or playing around with the hyperparameters.
* The empirical results only highlight cases where the proposed methods outperform existing methods. However, to more thoroughly judge the approach it would be good to understand (and include experiments) that show when exisiting methods are to be preferred. On the same note, what happens if the tasks are *not* related.
* I think the construction of the meta-tasks in the real application CIFAR should be thoroughly discussed in the main paper. Arguably this is the most critical step for a practicioner.


**Clarity**: 

* The paper is generally clearly written and easy to understand for readers familiar with the challenges in two-sample testing.
* the presentation of Meta-KL is not so easy to understand (and the motivation and theoretical treatment not really provided). Do I understand correctly that in case of deep kernel, it essentially optimizes the parameter intialization? If so, maybe it is worth explicitly stating that.

**Significance**:
The paper defines a new task (improving two-sample tests with data from related tasks) and proposes a viable solution. However, in the current form I think it will be hard for practitioners to apply. It is not really clear how such situations look like where one has abundance of data of relevant related tasks. The construction of such related task should be discussed much more.


## Comments & Questions (please try to address in rebutal or resubmission):

1. CIFAR experiments: 
    * For the meta learning samples from, say, P (the CIFAR1-10 dataset) are used. For a fair comparison, we should also allow the existing methods to see this data. This is indeed possible as at least some of them do not require balanced samples. 
    * I assume that the differences between classes within CIFAR 10 is not the same as the difference between CIFAR-10 and CIFAR-10.1. Thus the Meta-Task are probably not perfect. Can it happen that we overfit to the meta-tasks? That is, if we use to much data and too aggressively train the meta kernels, does the performance decrease again?
2. Please include a requirement.txt in the code to make it easier to use. The implementation does not provide guidance on how to install required packages (eg. freqoptest). I was not able to run the code.
3. What is the motivation for including Meta-KL in the paper? For me Meta-MKL seems like a much more reasonable/natural approach and is much easier to grasp. The theory is only provided for the Meta-MKL, the experimental results show much better performance of Meta-MKL, and I think the implementation and application should be more straightforward. Based on this my impression is that the paper might be better without Meta-KL.
4. Defintion 4: is the maximization of \tilde{k} defined over all possible kernels? Or is the \gamma relatedness defined with respect to some class of kernels? Is it trivial that J(P||Q;k) is bounded for all k?
5. I am struggeling to exactly understand the messge of Theorem 7. What does the supremum mean here? Can you include an intuitive interpretation? Is this essentially guaranteeing, that with the proposed approach we will with high probability find a kernel such that J is close to the optimal J?
6. Line 196: I don’t think the claimed results on Meta-MKL is actually included in the appendix. Did I miss sth?
7. Figure 3: How is it possible that the results of Meta-MKL are significantly worse in panel c) than in panel b)? This raises some doubts about the implementation. I would expect that things can only be better if we have more data.
8. Table 2: Why does the power of MMD-D not improve when using $m_{tr}=200$ as opposed to $m_{tr}=100$. Again this raises some concerns about the implementation of the experiments.

## Minor comments (not relevant for score)

1. Algorithm 3: Why do you define the class of kernels with softmax weights. Simply saying we allow for a convex combination seems a bit easier (and is also used later on). don’t forget do adjust in line 159 accordingly.
2. Definition 5: The notation seems a bit confusing. What does the index $w$ indicate in the list of base kernels? Also in the maximizatin they are suddenly denotes as $k^{(i)}$. I don’t think $\lambda^{ne}$ was defined.
3. Definition 6: related to above. $w$ is used to both denote the linear weights in the combination as well as a subscript of the individual kernels.
4. line 219: Typo? I think $n$ and $N$ are the same?
5. Caption of Table 3: there is an “are” too much.


### ------ Update after Discussion phase -------- 
I updated the papers score from 5 to 6 - marginally above acceptance threshold.

limitations_and_societal_impact: As argued above the limitations of the approach should be stated more clearly.
review_title: null

Meta-review: The main positive aspect of this paper is its inherent novelty: it is a new and unique way of improving two-sample testing by utilizing many related tasks. The corresponding theory also helps to demonstrate that the method is sound. Through thorough discussion with the reviewers, some additional experiments have been provided, and I ask that you please include these in the paper. The main lingering issue amongst the reviewers is the real-world motivation, or lack thereof. Some of the reviewers struggled to find a real-world scenario in which this kind of data would be present. During internal discussions, one reviewer did mention detecting adversarial examples, or changing distributions in streaming data as possible applications, which I thought worth mentioning here in case it is helpful.
Decision: Accept (Poster)