Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a new framework for solving few-shot regression problems. The proposed framework is based on deep kernel learning, which lies in the intersection of neural networks and kernel methods. The authors introduced adaptive deep kernel learning, which learns kernel for multiple task collection and computes the correct kernel for a task in a data-driven manner. The method is evaluated on three tasks collections — Sinusoids (synthetic dataset), Binding (real dataset of 5717 task where each task represents a binding affinity of small molecules against a given protein) and Antibacterial (real dataset of 3255 tasks where each task represents antimicrobial activity against given bacterium).


Pros:
1. The proposed framework introduces a new adaptive method for few-shot drug discovery regression problems. 
2. The paper addressed the question of uncertainty estimation for drug discovery tasks.


Cons:
1. The authors only evaluate the performance of the proposed method against other DKL-based methods. They do not consider a comparison with widely used in computational chemistry classical machine learning methods such as random forest, gradient boosting, SVM (which is also a kernel method), etc.
2. The experimental results in Tables 1-3 show only marginal improvement for real datasets. Also, it’s not clear if the numbers in the Tables 1-3 are on the original scale or on the transformed scale. To estimate how well the models perform it’s useful to transform the targets back to the original scale. 
3. Overall the paper is written in a somehow confusing manner and some details in the description of the experiments important for understanding are omitted.

Questions:
1. How well the proposed method performs against classical machine learning methods?
2. How was MSE in Tables 1-3 calculated? 
3. It would be useful to see the histogram of MSEs instead of just a single number.

review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: The submission is at the intersection of few-shot learning, kernel regression methods, and computational biology.
The main contributions are:
  - A few-shot learning algorithm combining several ideas and features:
    - Combining metric learning (shared across tasks) and kernel regression within each task
    - Learning a task representation by maximizing an estimate of the mutual information between the train (support) and valid (query) sets of a task
    - The addition of learned, synthetic "pseudo-examples"
  - Two datasets for few-shot regression, from real-life biological assays
The proposed algorithm outperforms (or is competitive with) mainstream few-shot learning methods on a synthetic 1D regression dataset, as well as the two proposed datasets.

This paper should be accepted, because it significantly expands the field of few-shot learning by proposing both a novel problem to tackle (few-shot regression from a high-dimensional, noisy input), with public datasets, and novel algorithm to solve it (combining several recent advances in different sub-fields of machine learning).

Overview
The overall problem is clearly stated, as well as its main challenges: noisiness of the data, different behaviors of the same input across tasks.
Despite the complexity of the proposed algorithm, its different pieces and their motivations are clearly motivated, introduced, and tested in ablation studies. I liked the clarity of section 2.
The "related work" section is clear and presents a good overall picture of the field. Additional papers that may be of interest:
  - Learned hallucination (Low-shot learning from imaginary data, Wang et al., CVPR 2018) seems to relate to the "pseudo-representations"
  - TADAM (Task dependent adaptive metric for improved few-shot learning, Oreshkin et al., NeurIPS 2018) is another example of task representation used in conjunction with metric learning (in the context of classification, not regression, though)
The proposed datasets are an interesting new benchmarking task, that naturally requires learning a task description, I hope it will get traction.

Questions:
  - The backpropagation through the kernel regressor (in order to train the parameters of the embedding function and other networks) seems unexpectedly straightforward. Is that only because there is a closed-form solution for the optimal regressor? Were the specific algorithms (KRR, GP) chosen because of that? Or could something like MetaOptNet (Meta-Learning with differentiable convex optimization, Lee et al., CVPR 2019) be used to relax that constraint?
  - What "generalization guarantees of kernel-based models" (l. 122) would be applicable here? Do they hold even when the kernel is applied on top of a learned embedding ($\phi_\theta$), or even when it depends on a trained model itself ($C_t$)?
  - What does "correlations > 0.8" mean exactly? (l. 258, 263)
  - In the "Binding" dataset, is it possible that the same protein is used in different bio-assays? In that case, are the different experiments "merged" into the same task, or be considered different tasks? Could it be possible that tasks involving the same protein would be in different (meta-)splits, or has that been taken care of during the data collection?
  - In the meta-test splits, how are examples split between the train/support and valid/query parts of each task?
  - How were the specific architectures of the different neural networks designed, or selected? Between $\phi$, v, r, $MLP_\rho$, the space of hyper-parameters seems huge, and the effects of these choices might be drastic.

Additional feedback
  - The caption of Figure 1 could (re-)introduce a definition of the notation. U and C_t for instance have not been introduced yet.
  - I'm not sure I agree that FSDKL only "share[s] characteristics with the metric learning framework", I see it more as being in that framework, but incorporating other elements as well (like other methods do, e.g., TADAM or RelationNet).
  - The horizontal axis of Figure 4 (a) and (c) are not clear until we see Table 4 of the appendix, and suggest an ordering of the different configurations, rather than 10 different categories. I'm not sure how to improve it though, maybe letters instead of numbers?

On notation:
  - On l. 155, is $\phi$ the same as $\phi_\theta$, or a different embedding function for x?
  - In Eq. 11, should $D^t_{trn}$ be $D^{t_j}_{trn}$ instead in both terms? Similarly, $D^{t_j}_{val}$ in the first term?
  - In Eq. 12, and l. 173, $\phi_x'$ suggests it is a different embedding $\phi'$ of the same $x$, if we want to convey that there are two inputs x and x' instead, $\phi_{x'}$ may be clearer.
  - In Fig. 4 (a) and 5, the vertical axis is labeled $\gamma_{mine}$ instead of (I assume) $\gamma_{task}$.
review_title: Official Blind Review #6

final_score: rating: 3: Weak Reject
review_content: This is an interesting paper that proposes the use of few shot regression to predict complicated experimental measurements such as protein-ligand binding affinity from very small, noisy real-world datasets. It is great that the authors make the effort to apply their approach to these important questions in drug-discovery. However, the paper as currently written is difficult to follow, and in particular it is hard to distinguish between places where existing methods are combined from novel contributions made by this paper. It would be helpful if the authors could explicitly delineate the novel contributions that they make. Moreover, there is a large body of work in the drug discovery literature that uses sparse experimental data on the interactions of multiple target proteins and multiple ligands to build models that predict the outcome of biological assays for held out protein targets, where this problem is known as drug-target interaction prediction, but these papers are not referenced in this work (e.g. reviewed in Chen et al. Molecules 23(9):1-15, 2018, Ezzat et al. 2017, 2018, 2019). 

In addition, it is hard to understand the results of the experiments that the authors carry out in this paper using data from BindingDB and PubChem. I don't understand the scaling that is applied to the MSE metric for the binding or antibacterial datasets. Why are the reported MSEs so low for all the methods? What does this metric mean? If the targets are first log2-scaled, then scaled linearly, then how different are they after this process? Given the tiny amount of data available, it seems surprising to me that the MSE is so low. From the brief description given, these problems appear significantly harder than the artificial Sinusoids task, yet the reported MSEs are orders of magnitude smaller. How is molecule similarity measured in Figure 2c? It would be useful for the authors to visualize performance for the Binding and Antibacterial tasks - for example how different are the different protein targets? What are the molecular ligand structures in the train and test set in each case - could the authors provide some examples? Using random splits into train and test likely means that some test data points are very close to train data points - can the authors stratify their analysis to provide some insight into the performance beyond the average MSE? 

Overall this paper makes a potentially interesting contribution, but it is not well situated within the drug discovery literature, and the results are not explained in enough detail to be understandable by experts in the drug discovery field. 

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: To tackle data scarcity issue in biological assay modeling, this paper proposes a method to episodically train Deep Kernel Learning models such that at meta-test time the models requires less examples. 

The proposed Adaptive Deep Kernel Learning extends Deep Kernel Learning (end-to-end train Gaussian Process/Kernel Ridge Regression using Neural Network's final layer embedding) by:

1. using a learnable task-specific kernel generator network 
2. applying episodic few-shot/meta learning to train the system end-to-end

Here are the major issues for rejecting:

Model:
- Section 3.1 and 3.2 about "Task Specific Kernel" are very hard to follow.
  -- (line 170) equation 12, What is the size of the MLP that is used as the kernel network 
  -- Is there an ablation to show the contribution of using learnable kernel network vs hand-picked kernel for ADKL-GP and ADKL-KRR? 
  -- (line 181) "where U is a set of unlabeled inputs" Where does the unlabeled inputs come from?
  -- Is there an ablation about |U|  <= 50 and its effect?

Experiments and Datasets:

- The ProtoMAML code provided by the authors has only a few lines of code. And in the comment, it says: "It turns out that ProtoMAML is the same as the light version of BMAML with one particle." Such implementation detail should be mentioned in the main paper.

- Section 5 lacks details. 
 -- How each meta-train episode is formed? Did they stochasticall sample a subset of molecules for each episode?
 -- What is the total number of tasks in the meta-train and meta-test split respectively? Are there any overlapping between the two splits?


Presentation Issues:
- Section 2 (line 76) "We extended it (Deep Kernel Learning) to few-shot learning and discuss its advantages over the metric learning framework" This sounds like the author developed a brand new framework, while the rest of paper is about proposing a specific realization of few-shot learning: Few-shot Deep Kernel Learning, which meta-learn through a differentialble kernel learning process (with the task kernel adaptation network novelty). 

- Section 3 (Figure 1) "The blue and orange colors show the procedure..." However there are more than two colors (including different shades of blue and orange colors) in the figure, which makes the figure hard to parse .  

- Section 4 (line 201) "DKL methods lie between neural networks and kernel methods". To me, at high-level, DKL adds kernel learning as a differentiable layer in the end of a neural network. See how a similar model is categorized in Bertinetto et al. ICLR 2019 paper "Meta-learning with differentiable closed-form solvers":  "The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression,as part of its own internal model, enabling it to quickly adapt to novel data." The delta of the paper is adding 1) Gaussian Process and 2) task-specific adaptation. If Bertinetto et al. categorize their method as part of a deep network then I think so should this work.

- Section 4 (line 239) "Our work goes beyond ... by proposing ADKL: a data-driven manner for computing the correct kernel for a task." This is probably a grammar mistake. "a data driven manner for" sounds odd. 

- Section 5 (line 272) "Fig 2 highlights three aspects of the collections that make them better benchmarks for evaluating the readiness of FSR methods for real-world applications relative to toy collections." What is the other benchmarks the authors are comparing with? Are they generic benchmarks for all types few-shot regression tasks/applications beyond biological assays (e.g. computer vision tasks like: object detection). Maybe defining the proposed benchmark as "complimentary" and "bio-assay application focused" would be more appropriate. 


review_title: Official Blind Review #5

final_score: rating: 6: Weak Accept
review_content: The authors of this work are attempting to solve the problem of few shot regression for drug discovery problems. This is an important problem in the field of deep drug discovery, since low data issues are very common. While previous papers have addressed the challenges of low data drug classification, there hasn't been much progress made so far on low data regression thus far.

The authors propose formulating the problem at hand as a few shot deep kernel regression problem (FSDKL). This framework has some similarities to past work deep metric learning, which has been used previously for few shot classification problems in drug discovery. In particular, the authors propose a new method, adaptive deep kernel learning to help improve task specific learning. ADKL is claimed to improve over FSDKL since it helps learn a task specific kernel function. There are a number of interesting methodological adaptations here, such as the use of an idea similar to DeepSets to encode order invariance in the training set (an important challenge when dealing with small "support" sets in training). The authors also incorporate some unlabeled data during training to improve on representation learning.

To test their contributions, the authors gather two new dataset collections, Binding and Antibacterial from publicly available sources. Some detail is provided about these collections, but a priori, it's not easy for me to judge the quality of these data sources since they haven't been benchmarked previously in the literature.

In the experimental section, the authors compare ADKL against a number of past low data learning methods. The benchmarks in section 6.1 would benefit from confidence intervals. In my experience, few-shot algorithms can be quite noisy, so statistical tests are important to distinguish architectural improvements from noise. Repeated trials could be taken over the choice of random seed for the experiments to gauge robustness. Without error bars, and on a new dataset, it's not possible to gauge if ADKL really provides an improvement. 

The experiments having to do with active learning in section 6.2 are interesting. Why is only CNP measured though? Do other methods not make sense with active learning? 

In conclusion, the authors consider an important problem for deep learning in drug discovery and offer a useful advance by adapting the framework of deep kernel learning to this space. However, the work could still use a good bit of polish to really shine. It's not clear to me that the suggested ADKL framework really makes an improvement over past few shot regression methods. Adding some statistical significance tests would be useful. It would also help if the authors benchmarked against datasets that were better known in this field, such as those from the moleculenet.ai suite. It's not easy to judge how easy/hard the new datasets the authors propose are, which makes it challenging to gauge the real improvement. I'm also not sure that this paper is a clear fit for ICLR. I think there's a real contribution to the field of deep drug discovery by the adaptation of the DKL framework to few shot drug discovery, but I'm not convinced that ADKL is superior to other DKL methods as a pure learning technique. 

All that said, I'm comfortable marking the paper as a "weak accept" since I think there is a real scientific contribution here, but I encourage the authors to make a serious effort to improve their presentation and tighten-up their experiments.

Detailed Suggestions:
- Section 2 on Deep Kernel Learning and Section 4 on Related work should likely be merged together. Going back and forth between the literature and author contributions was a little confusing. 
- There are a lot of acronyms in the paper; I found myself having to refer back and forth to all the methods considered. It might be worthwhile to make a cleanup pass to make the discussions a little easier to read. 
- Figure 4 would benefit from a legend. At first glance, it wasn't clear what gamma_mine and gamma_pseudo mean here. The color plot is also a little difficult to make sense. Perhaps consider an alternative plot.
review_title: Official Blind Review #7

final_score: rating: null
review_content: I unfortunately can't provide technical substance. But I do feel the presentation makes understanding unusually hard.
review_title: I don't quite follow this line of work.

Meta-review: This work applies deep kernel learning to the problem of few shot regression for modeling biological assays. To deal with sparse data on new tasks, the authors propose to adapt the learned kernel to each task. Reviews were mixed about the method and experiments, some reviewers were satisfied with the author rebuttal while others did not support acceptance during the discussion period. Some reviewers ultimately felt that the experimental results were too weak to warrant publication. On the binding task the method is comparable with simpler baselines, and some felt that the gains on antibacterial were unconvincing. 
Other reviewers felt that there remained simpler baselines to compare with, for example ablating the affects of learning the kernel with simple hand picking one. While authors commented they tried this, there were no details given on the results or what exactly they tried. 

Based on the reviewer discussion, the work feels too preliminary in its current form to warrant publication in ICLR. However, given that there are clearly some interesting ideas proposed in this work, I recommend resubmitting with stronger experimental evidence that the method helps over baselines.
Decision: Reject