Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents ADKF-IFT -- a new method to train deep kernel GPs using meta-learning. This method splits the parameters into two sets, where only one set is adapted to each task. This leads to a bi-level optimization problem, that the authors solve using the Implicit Function Theorem. The number of parameters which are adapted can help in balancing between over- and under-fitting, which helps the model perform well on tasks with small training datasets. The paper also shows that prior methods for training deep kernel GPs, DKT and DKL, are special cases of the proposed method.

Experimental results from chemistry show that the model generally performs competitively on small data domains, and also perform well on OOD tasks.
strength_and_weaknesses: ### Strengths
* The paper presents a novel method for training deep kernel GPs that is competitive on small data domains.
* The proposed method can help balance over- and under-fitting.
* The paper demonstrates that the proposed method subsumes the existing DKT and DKL methods as special cases.

### Weaknesses
* It would be instructive to understand how the model performs on tasks with larger training datasets. I understand that is not the goal of the paper, but understanding the failure modes of the method (if there are any) can be useful.
* How sensitive is the method to the choice of \Psi_{adapt}, the parameters that are adapted to each task? Providing a discussion on this would be useful
* Provide more details about the hyperparameters & architectures for reproducing the results.
clarity,_quality,_novelty_and_reproducibility: ### Clarity
The writing is clear and easy to follow. The paper provides enough background to understand the paper.

### Novelty
The primary contributions of the paper are the interpolation between meta-learning & conventional deep kernel learning, and the use of implicit function theorem to solve the resulting bi-level optimization problem. These are both novel and interesting.

### Reproducibility
The paper does not provide all of the hyperparameter and architecture details to reproduce the results. However, the authors promise to make the code available upon publication.
summary_of_the_review: The paper presents a novel method for training deep kernel GPs which obtains strong results on small datasets. The proposed method is shown to be a generalization of existing methods. Overall, I find this work to be sufficiently novel and useful for many chemistry problems to justify acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In order to fit molecular property prediction models that give good uncertainty estimates but also overcome issues associated with neural network methods that either overfit or underfit Gaussian Processes, the authors propose Adaptive Deep Kernel Fitting with Implicit Function Theorem (ADKF-IFT), a generalization of Deep Kernel Learning and Deep Kernel Transfer that meta-learns a subset of parameters (usually those associated with a feature extractor) across many tasks, and transfer learns the rest (usually those associated with the base kernels of Gaussian Processes). The authors propose a bilevel optimization method that avoids costly inner optimization gradient calculations by invoking Cauchy's Implicit Function Theorem. The authors demonstrate how ADKF-IFT outperforms a large variety of existing methods of molecular property prediction tasks in MoleculeNet and few-shot prediction tasks in FS-MOL. They also show how the feature representation learned by ADKF-IFT can be used with Gaussian Process surrogate models to provide optimal or near-optimal results in out-of-domain molecular optimization tasks like molecular docking and antiviral drug design. Using ablation studies, the authors also show that the bilevel optimization is what gives the reported performance gains.
strength_and_weaknesses: Strengths:
- ADKF-IFT provides the means to effective training of Gaussian Processes that overcomes the shortcomings of the two methods it generalizes, DKL and DKT while achieving state-of-the-art predictive power.
- ADKF-IFT achieves state-of-the-art results on many molecular property prediction benchmarks, including few-shot tasks.
- ADKF-IFT appears to learn informative molecular features that perform well for entirely different tasks.

Weaknesses:
- The authors do not include any examples that demonstrate that ADKF-IFT is neither over- or underfit, nor do they demonstrate that the uncertainty estimates derived from the learned GPs are better than those of GPs learned in other ways.
- MoleculeNet is somewhat out-of-date - the authors should consider the superset of tasks proposed in the Therapeutics Data Commons (https://tdcommons.ai/) database.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to understand. The work appears to be high-quality and well-executed, though no code has yet been shared.

Generalizing the DKL and DKT methods into a bilevel optimization problem in order to overcome the shortcomings of both is an original and important contribution.
summary_of_the_review: This paper makes a significant contribution to Gaussian Process training methods by proposing a novel bilevel optimization method that achieves state-of-the-art molecular property prediction. Such models could have significant impact on the work of researchers applying ML to drug discovery, or other related topics.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new meta-learning framework formulated as a bilevel optimization problem. The implicit function theorem is utilized to efficiently compute the gradient of the bilevel optimization problem. It is shown that in the context of deep kernel learning the proposed framework encompasses DKL and DKT as extreme cases. In the wide range of models under this framework, the author proposes ADKF-IDF which meta-learns the parameters of the feature extractors and fine-tunes the parameters of the base kernel. This specific design choice avoids the computational hassle in the gradient computation and induces a natural interpretation of deep kernel learning in meta-learning. The ADKF-IDF is extensively compared to existing methods on few-shot learning tasks predicting molecular properties and on Bayesian optimization of molecular properties.
strength_and_weaknesses: ### Strengths
- The paper proposed a meta-learning framework that is formulated via bilevel optimization and optimized using the implicit function theorem with general applicability.
- In its application to deep kernel models, the authors propose a deep kernel model with a meta-learned feature extractor and fine-tunable GP parameters. This not only circumvents the computational hassle of the proposed but also gives a quite interpretable hierarchical structure. While including existing methods, DKL and DKT as special cases, this specific model is a balanced interpolation of both under the proposed meta-learning framework.
- The proposed ADKF-IFT is extensively compared with existing methods on molecular property prediction few-shot learning. 
Especially, in the experiments on FS-MOL benchmarks, it seems that much effort and resources have been put into the experiment, which provides also good baselines for others as the authors benefited from the baseline results in MoleculeNet experiments.
- In addition to its strengths in meta-learning, the authors also conduct Bayesian optimization experiments which supports the good quality of learned representation in the proposed meta-learning framework and the well-calibrated uncertainty of ADKF-IFT.




### Weaknesses
- Is a random initialization of $\psi_{adapt}$ enough to guarantee the below?
$$
IF(\psi_{adapt} | \psi_{meta}, S_T) = \frac{\partial L_{train} (\psi_{meta}, \psi_{adapt}, S_T)}{\partial \psi_{adapt}} = 0
$$
    - In Appendix B, it says that in ADKF-IFT, the fine-tuned parameter $\psi_{adapt}$ is initialized using the median heuristic.
    - It seems that such initialization does not guarantee $IF(\psi_{adapt} | \psi_{meta}, S_T) = 0$.\
Does the median heuristic generate a point $\psi_{adapt}$ such that $IF(\psi_{adapt} | \psi_{meta}, S_T) = 0$?\
Is there a mechanism that drives a point $\psi_{adapt}$ such that $IF(\psi_{adapt} | \psi_{meta}, S_T) \neq 0$ to a point such that $IF(\psi_{adapt} | \psi_{meta}, S_T) \neq 0$?
- How is it guaranteed that $\psi_{adapt}^*$ is a minimum?
    - From Thm.1, it seems that when the implicit function theorem is used, implicitly defining function is 
$$
IF(\psi_{adapt} | \psi_{meta}, S_T) = \frac{\partial L_{train} (\psi_{meta}, \psi_{adapt}, S_T)}{\partial \psi_{adapt}} = 0
$$
This implies that for a given $\psi_{meta}$, $\psi_{adapt}$ satisfying the above equation is a maximum, a minimum, or a saddle point.
    - I was wondering how $\psi_{adapt}$ is guaranteed to be a minimum.\
Even if the initialization guarantees $IF(\psi_{adapt} | \psi_{meta}, S_T) = 0$ or the gradient descent eventually makes the point to satisfy $IF(\psi_{adapt} | \psi_{meta}, S_T) = 0$, 
    - It seems unlikely that the minimization of upper validation loss is obtained with the maximization of lower train loss. \
This may imply that some care is necessary for the choice of two losses in this bilevel meta-learning framework.\
I agree that the specific instantiation of ADKF-IFT seems reasonable also in this respect.\
Still, I am curious about the authors' thoughts and opinions on this.
- Sensitivity to the initialization
    - Even if it is guaranteed that $\frac{\partial L_{train} (\psi_{meta}, \psi_{adapt}, S_T)}{\partial \psi_{adapt}} = 0$, it is imaginable that, in some case, $\psi_{adapt}$ is initialized to a point of a connected component of $\\{\psi_{adapt} | \frac{\partial L_{train} (\psi_{meta}, \psi_{adapt}, S_T)}{\partial \psi_{adapt}} = 0 \\}$ which consists of maximum points.
Considering this case, I am curious about the sensitivity of the method to the initialization or the choice of hyperparameters (some hyperparameters may exclude such bad cases).





### Minor points
- It seems that this somewhat seemingly relevant work is missing
    - Few-Shot Bayesian Optimization with Deep Kernel Surrogates; ICLR 2021
    - I am not asking to include this as an additional baseline. 
    - The comparison with this seems to be able to further highlight the benefit of ADKF-IFT.
- Significance of the differences in the ablation study.
    - I can check visually the consistency of the superiority of ADKF-IFT in Figure 3. But in many cases, the confidence intervals (I guess they are from repeated experiments with some randomness, it is reasonable to call them confidence intervals) overlap. Maybe providing some simple statistical results may strengthen the argument of the ablation study.
- Duplicated names
    - In the paper, ADKF-IFT stands for two things, 1) meta-learning framework formulated by a bilevel optimization problem and trained using the implicit function theorem, 2) deep kernel GP model with meta-learned feature extractor and fine-tuned GP parameters using the proposed meta-learning framework
    - Saving ADKF-IFT to the proposed specific deep kernel methods, it may be less confusing to use another name for the proposed meta-learning framework reflecting its generality, for example, Meta-BIFT (meta-learning formulated via BIlevel optimization and optimized via Implicit Function Theorem)

clarity,_quality,_novelty_and_reproducibility: Except for the questions in 'Weaknesses', the paper is well-written in general. Its technical novelty efficiently combines existing methods and tools under a new meta-learning framework proposed by the authors. The empirical contribution of the paper seems quite significant by providing other few-shot learning baselines on FS-MOL for various support set sizes. It seems that implementation details are provided at a reasonable level of detail. Still, the reproducibility can be improved by disclosing the code.
summary_of_the_review: The paper proposes a new meta-learning framework and a specific instance of it that is shown to be effective in few-short learning and out-of-sample prediction demonstrated via Bayesian optimization experiment. With a good combination of many technical tools under the new meta-learning framework, the paper has good empirical contributions. As long as my concerns mentioned in 'Weakness' can be addressed, I am willing to increase my score and support the acceptance of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Authors propose new meta-learning methods for deep kernel Gaussian processes called ADKF-IFT (Adaptive Deep Kernel Fitting with Implicit Function Theorem). Proposed method uses a meta-learning framework that is formulated via bi-level optimization and optimized using the implicit function theorem with general applicability. With ADKF-IFT, authors obtain SoTA results on small dataset molecular property prediction problems. 

Strengths:
- Effective training methods for Gaussian Processes overcoming shortcomings of DKL, DKT
- Achieves state-of-the-art results on many molecular property prediction tasks including few-shot tasks
- Method learns informative molecular features that performs well on out-of-domain tasks
- Well written and easy to understand
- High quality and well executed
- Generalizing DKL and DKT into bilevel optimization overcoming issues of both is novel and important contribution
- Utilizing meta-learning (bi-level optimization) to interpolate between conventional deep kernel learning and meta-learning is an interesting and novel method. 

Weakness
- MoleculeNet is out-of-date and a reviewer suggested new tasks (tasks proposed in the Therapeutics Data Commons database)
- No code for reproduction and details for reproduction is lacking (hparams, architecture details)
    - Author promised code upon publication
- Lack of evaluation of larger training size datasets. 

AC encourages the authors to provide code and experiment as promised.

justification_for_why_not_higher_score: Still limitation to small dataset only; solving scalability issues and maintaining solid performance would have put the paper on the next score. 

justification_for_why_not_lower_score: Authors did a great job of summarizing highlights of the reviewers. All reviewers are in support for acceptance for the paper

- Reviewer BY35: “Overall, I find this work to be sufficiently novel and useful for many chemistry problems to justify acceptance.”
- Reviewer sNf8: “This paper makes a significant contribution to Gaussian Process training methods by proposing a novel bilevel optimization method that achieves state-of-the-art molecular property prediction. Such models could have significant impact on the work of researchers applying ML to drug discovery, or other related topics.”
- Reviewer fnkh: “With a good combination of many technical tools under the new meta-learning framework, the paper has good empirical contributions.”

Results are solid and provide interesting methodology to overcome issues in existing methodology.
Decision: Accept: poster