GAUCHE: A Library for Gaussian Processes in Chemistry
Authors: Ryan-Rhys Griffiths
Leo Klarner
Henry Moss
Aditya Ravuri
Sang T. Truong
Yuanqi Du
Samuel Don Stanton
Gary Tom
Bojana Ranković
Arian Rokkum Jamasb
Aryan Deshwal
Julius Schwartz
Austin Tripp
Gregory Kell
Simon Frieder
Anthony Bourached
Alex James Chan
Jacob Moss
Chengzhi Guo
Johannes P. Dürholt
Saudamini Chaurasia
Ji Won Park
Felix Strieth-Kalthoff
Alpha Lee
Bingqing Cheng
Alan Aspuru-Guzik
Philippe Schwaller
Jian Tang
Venue: NeurIPS 2023 poster (2023)
Keywords: Gaussian processes
Bayesian optimization
Chemistry
Molecular Machine Learning
Applications
Software
TL;DR: A software library for Gaussian processes in chemistry

Abstract: We introduce GAUCHE, an open-source library for GAUssian processes in CHEmistry. Gaussian processes have long been a cornerstone of probabilistic machine learning, affording particular advantages for uncertainty quantification and Bayesian optimisation. Extending Gaussian processes to molecular representations, however, necessitates kernels defined over structured inputs such as graphs, strings and bit vectors. By providing such kernels in a modular, robust and easy-to-use framework, we seek to enable expert chemists and materials scientists to make use of state-of-the-art black-box optimization techniques. Motivated by scenarios frequently encountered in practice, we showcase applications for GAUCHE in molecular discovery, chemical reaction optimisation and protein design. The codebase is made available at https://github.com/leojklarner/gauche.

GAUCHE: A Library for
Gaussian Processes in Chemistry
Ryan-Rhys Griffiths
Leo Klarner
Henry Moss
Aditya Ravuri
Sang Truong 1*
Samuel Stanton S* Gary To
Bojana Rankovic
Yuanqi Du 10-
Arian Jamasb
Aryan Deshwal"
Julius Schwartz
Austin Tripp"
Gregory Kell
Simon Frieder2
Anthony Bourached
Alex J. Chan
Jacob Moss
Chengzhi Guo
Johannes Durholt
Saudamini Chaurasia
Ji Won Park
Felix Strieth-Kalthoffs
Alpha A. Lee
Bingqing Chenglé 16
Alán Aspuru-Guzik 6.7.17
Philippe Schwaller
Jian Tang 18,
Meta Uniieeriity Oxford Univeesity of Cambridge Sttanford University GGeenteech
University of Toronto
Vector Institute *EPFL 9CCR Catalysis
Cooomell University
Il Washington State University 'iinn'ss College London
Universsiyy College London
Evonik Industries AG
Syracuse University
1IST Austria 'CCIFAR AI Research Chair
18MILA Quebee Al Institute
19 HEC Montreal
Equal contributions
fryangriff123, Rgoa com
Abstract
We introduce GAUCHE, an open-source library for GAUssian processes in CHEmistry. Gaussian processes have long been comerstone of probabilistic machine learning, affording particular advantages for uncertainty quantification and Bayesian optimisation. Extending Gaussian processes to molecular representations, however, necessitates kernels defined over structured inputs such as graphs, strings and bit vectors. By providing such kernels in modular, robust and easy-to-use framework, we seek to enable expert chemists and materials scientists to make use of state-of-the-an black-box optimization techniques. Motivated by scenarios frequently encountered in practice, we showcase applications for GAUCHE molecular discovery, chemical reaction optimisation and protein design. The codebase made available http:///giuuub.oom/ooollreeeeeee
Introduction
Early-stage scientific discovery is often characterised by the limited availability of high-quality experimental data [1, 2, 3], meaning that there is much knowledge to gain from targeted experiments. As such, machine learning methods that facilitate discovery in low data regimes, such as Bayesian optimisation (BO) [4, 5, 6, 7, 8, 9] and active learning (AL) [10, 11], have great potential to expedite the rate at which useful molecules, materials, chemical reactions and proteins can be discovered.
At present, Bayesian neural networks (BNNS) and deep ensembles are typically the method of choice to generate uncertainty estimates for molecular BO and AL loops [10, 12, 13, 14]. For small datasets, however, Gaussian processes (GPs) may often be preferable and more appropriate choice [15, 16]. Furthermore, GPs possess particularly advantageous properties for BO; first, they admit exact as opposed to approximate Bayesian inference and second, few of their parameters need to be determined by hand. In the words of Sir David MacKay [17].
"Gaussian processes are useful tools for automated tasks where fine tuning for each problem is not possible. We do not appear to sacrifice any performance for this simplicity."
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Reactions
Molecules
Proteins
Fingerprint
RXX
ECFP Fingerprint
Moteoutor Groph
SMILES String
Sequencee String
Structural Graph
[
I]
NC1=00=00=C1
RASHLMQA
Goussian Processes
Bayesian Optimisation
Figure An overview of the applications and representations available in GAUCHE.
The iterative model refitting required in BO makes a prime example of such an automated task. However, canonical GPs typically assume continuous input spaces of low and fixed dimensionality, hindering their application to standard molecular representations such as SMILES/SELFIES strings [18, 19, 20], topological fingerprints [21, 22, 23) and discrete graphs [24, 25J.
With GAUCHE, we provide modular, robust and easy-to-use framework rapidly prototype GPs with 30+ GPU-accelerated string, fingerprint and graph kernels that operate
range of molecular representations (see Figure 1). Furthermore, GAUCHE interfaces with the GPyTorch [26] and BoTorch [27] libraries and contains extensive set of tutorial notebooks to make state-of-the-art probabilistic modelling and black-box optimization techniques more easily accessible scientific experts in chemistry, materials science and beyond.
2 Background
We briefly recall the fundamentals of Gaussian processes and Bayesian optimisation in Sections 2.1 and 2.2, respectively, and refer the reader to [28] and [29, 30, 31] for a more comprehensive treatment.
2.1 Gaussian Processes
Notation X E Raxd is design matrix of training examples of dimension d. A given row of the design matrix contains a training molecule's 's representation x, A GP is specified by mean function, m(x) = (x)] and covariance function k(x, x') Ex(f(x) m(x))
mxx)))] (X X) is kernel matrix, where entries are computed by the kernel function as [K] k(x,, x,) and 0 represents the of kernel hyperparameters. The GP specifies the full distribution over the
function f to be modelled as
f(x) GP (m(x))
Training Hyperparameters for GPs comprise kernel hyperparameters, 6, in addition to the likeli- hood noise, o These hyperparameters are chosen by optimising an objective function known as the negative log marginal likelihood (NLML)
log Xog((y 0) - 2 (Ko XX +
log (x, + + -
log (2TT),
2
encourages with
controls model capacity

---

where I represents the variance of i.idd Gaussian noise on the observations y. The NLML embodies Occam's razor for Bayesian model selection [28] in favouring models that fit the data without being overly complex.
Prediction At test locations X., assuming zero mean function obtained following the standard- ization of the outputs the GP returns a predictive mean, f. = (XXXX.X))(XXX of - y, and predictive uncertainty cov(f,) ((,,,X,) (XX,, X) X
2.2 Bayesian Optimisation
In molecular discovery campaigns, we are typically interested in solving problems of the form
x' arg max fxx, xEX
where f()) : X + R is an expensive black-box function over a structured input domain X. In our setting the structured input domain consists of a set of molecular representations (graphs, strings, bit vectors) and the expensive black-bor -bx function is an experimentally determined property of interest that wish to optimise. Bayesian optimisation (BO) [32, 33, 34, 35, 29, 36] is data-efficient methodology for determining x". BO operates sequentially by selecting input locations at which to query the black-box function f with the aim of identifying the optimum in few queries as possible. Evaluations are focused on promising areas of the input space as well as areas with high uncertainty- balancing act known as the exploration/exploitttion trade-off.
The two components of a BO scheme are probabilistic surrogate model and an acquisition function. The surrogate model is typically chosen to be GP due to its ability to maintain calibrated uncertainty estimates through exact Bayesian inference. The uncertainty estimates of the surrogate model are then leveraged by the acquisition function to propose function is a heuristic that trades off exploration and exploitation, well-known examples of which include expected improvement (EI) [33, 35] and entropy search [37, 38, 39, 40]. After the acquisition function proposes an input location, the black-box is evaluated at that location, the surrogate model is retrained and the process is repeated until solution is obtained.
input locations query. The acquisition
3 Molecular Representations
We review commonly used representations for molecules (Section 3.1), chemical reactions (Sec- tion 3.2) and proteins (Section 3.3), before describing the kernels that operate on them in Section An overview of the representations considered by GAUCHE is provided in Figure 1.
3.1 Molecules
Graphs A molecule can be represented an undirected, labelled graph G = (V.&) where vertices V = (ri,-.... .uN] represent the atoms of an N-atom molecule and edges & C V x V represent covalent bonds between these atoms. Additional information may be incorporated in the form of vertex and edge labels L : V E + Σv x Σg by specifying e.g. atom types or bond orders.
OH
NH2
bags of subgraphs g' =²= (V' C V. C E) of a certain type and then hash them into machine-readable bit or count vectors. Extended connec- tivity fingerprints (ECFPs) [21]. for example, enumerate all circular subgraphs up to a pre-specified radius parameter by assigning initial numeric identifiers to each atom in molecule and iteratively updat- ing them based on the identifiers of their neighbours. Each level of iteration appends substructural features of increasing non-locality to an array, which is then hashed to a bit vector reflecting the presence or absence of those substructures in the molecule (see Figure 2). We choose a radius of 3 for all experiments the main text and provide more detailed ablation of the radius parameter in Appendix E.3.
Fingerprints Molecular fingerprints enumerate
OH
NH₂
Figure 2: Visualisation of the ECFP subgraph enumeration and hashing procedures.
3

---

Additionally, we make use of fragment descriptors, which are count vectors in which each component indicates the count of particular functional group present in molecule, well as the concatenation of the fingerprint and fragment feature vectors, representation termed fragprints [3], which has shown strong empirical performance. Example representations of Xip for fingerprints, X for fragments and Xtrp for fragprings are given as
Xfp
Xfr
3
0
Xfrp
1 3
2
Strings The Simplified Molecular-Input Line-Entry System (SMILES) is a text-based representa- tion of molecules [18. 19]. examples of which are given in Figure Self- Seff-Referencing Embedded Strings (SELFIES) [20] is an alternative string representation to SMILES such that bijective mapping exists between a SELFIES string and a molecule.
2H
NH
NCI=CC-CC-CI
NCI-CC-CC-NI
Figure 3: SMILES strings for structurally similar molecules. Similarity is encoded in the string through common contiguous subsequences (black). Local differences are highlighted in red.
3.2 Reaction Representations
Chemical reactions consist of (multiple) reactants and reagents that react to form one or more products. The choice of reactant/reagent typically constitutes categorical design space. Taking the thgh-throughput experiments by [41] ] on Buchwald Hartwlg reactions, the reaction design space consists of 15 aryl and heteroaryl halides, Buchwald ligands, bases, and 23 isoxazole additives.
example
Concatenated Molecular Representations the number of reactants and reagents constant, the molecular representations discussed in Section 3.1 may be used to represent them, and the vectors for the individual reaction components can be concatenated to construct representation of the reaction [41. 42]. An additional and commonly used concatenated representation is the one-hot-encoding (OHE) the reaction categories where bits specify which of the components in the different reactant and reagent categories is present. In the Buchwald- Harttig example, the OHE would describe which of the aryl halides, Buchwald ligands, bases and additives are used in the reaction, resulting in a 44- diddisiiiionl bit vector [43].
Differential Reaction Fingerprints Inspired by the hand-engineered difference reaction finger- prints by [44] and [45] recently introduced the differential reaction fingerprint (DRFP). This reaction fingerprint constructed by taking the symmetric difference of the sets containing the molecular substructures both sides of the reaction arrow. Reagents are added to the reactants. The size of the reaction bit vector generated by DRFP is independent of the number of reaction components.
Data-Driven Reaction Fingerprints [46] described data-driven reaction fingerprints using Trans- former models such as BERT [47] trained in supervised SMILES. Those models can be fine-tuned on the task of interest to learn more specific reaction representations [48] (RXNFP). Similar the DRFP, the size of the data-driven reaction fingerprints is independent of the number of reaction components.
unsupervised fashion on reaction
3.3 Protein Representations
Proteins are large macromolecules that adopt complex 3D structures. They can be represented in string form describing the underlying amino acid sequence. Graphs varying degrees of coarseness may also be used for structural representations that capture spatial and intramolecular relationships between structural elements, such as atoms, residues, secondary structures and chains. GAUCHE interfaces with Graphein [49], a library for pre-processing and computing graph representations of structural biological data, thereby enabling the application of graph kernel-based methods to protein structure. We provide experiments protein fitness prediction in Appendix E.

---

Molecular Kernels
The choice of kernel is an important inductive bias for the properties of the function being modelled. common choice for continuous input domains the radial basis function kernel
ARBE (x, x') = 03 -x (- x )
2P
where of is the signal amplitude hyperparameter (vertical lengthscale) and & is the (horizontal) lengthscale hyperparameter. However, in order to train GPs on the molecular representations covered in Section 3, hespoke kernel functions that able operate non-continuous input spaces needed.
4.1 Fingerprint Kernels
The simplest kernel operate on fingerprints is the scalar product or linear kernel defined for vectors x, x' R as
Scalar Product Kernel
&scalar Produce x') = of (x, x'):
where of is a scalar signal variance hyperparameter and (.. +) is the Euclidean inner product.
Tanimoto Kernel Introduced general similarity metric for binary attributes [50], the Tanimoto kernel was first used chemoinformatics in conjunction with non-GP-based kernel methods [51].
is defined for binary vectors x, {0, 1 for d 2 1 as
xx,x'
Tamimooto (x, x) = o3 | + |X/| (x,x')"
where || is the Euclidean norm.
In addition to the Tanimoto kernel, GAUCHE provides parallelisable and batoh-Gi--ompaiiblee implementations of 12 other bit and count vector kernels that are presented in Appendix G.
4.2 String Kernels
String kernels [52, 53] measure the similarity between strings by examining the degree to which their sub-strings differ. In GAUCHE, we implement the SMILES string kernel [54] which calculates an inner product between the occurrences of sub-strings, considering all contiguous sub-strings made from at most characters (we set featurisation & + TR, also known as bag-of-characters representation [55], SMILES string kernel between two strings S and S' is given by
5 in our experiments). Therefore, for the sub-string count
kitiing (S) == o (
More complicated string kernels do exist in the literature, for example, GAUCHE also provides an implementation of the subset string kernel [56] which allows non-contiguous matches. However, we found that the significant added computational cost of these methods did not provide improved performance over the more simple SMILES string kernel in the context of molecular data. Note that although named the SMILES string kernel, this kernel can also be applied to any other string representation of molecules e,g. SELFIES or protein sequences.
4.3 Graph Kernels
Graph kernels define mapping Φx 9 - H. from graph domain & to feature space H. in which the inner product between pair of graphs g.g' E G serves as a similarity measure
Kurphh (g. g') = oz (dx(g) (x(g)))
where À denotes kernel-specific hyperparameters. Depending on how dx is defined, the kernel captures different substructural motifs and is characterised by different hyperparameters. The Weisfeiler-Lehmnn (WL) kernel [57], for instance, is given by the inner products of label count vectors over À iterations of the Weisfeiler-Lehmarl algorithm [58].
To maximise the number of graph kernels available in GAUCHE we implemented the SIGP class, which enables PyTorch-based GPS to trained on non-tensorial inputs with any kernel from the
GraKel library [59] (see Appendix F for more details).
5

---

Table 1: Predictive accuracy and calibration of Gaussian process and probabilistic deep learning models across four different molecular property prediction benchmarks. RMSE (L) and NLPD (L) are reported as mean and standard error over 20 random 80/20 train/test splits. The best models up
to statistical significance are highlighted in bold. Numerical issues were encountered with the WI. kernel on the large lipophilicity dataset and the corresponding entries are left blank.
Photoswitch
ESOL
FreeSolv
Lipophilicity
Model
Representation
RMSE
NLPD
RMSE
NIPD
RMSE
NI.D
RMSE
NLPD
fragprinns
20.0 20.9.
0 1222::000
0.71g00
0.33
0.28ao.8
0.67
0. a.71
Tanimoto
fingerprints
23.4
0.33g000
101:000
0.71
1. 1.93
0.58
0.70a00:
0.80.001
Tragments
26.3+ 3en.
0.50
0.91=0.0:
0.57
1. 1.1953.00
0.44
0.80
0.94
fragprints
22.5
0. 0.23:so.m
0.88
0.53ea.0
+.27
0.25ax..0
0.77
0.92=m0:
GAUSSIAN
Scalar Product
fineerprints
24.8
0.33ga0s
1.17:00:
0.84
1,03
0.64
0.84ame1
1.03a00
Tragments
36.6e110
0.80
1.13:0.0
0.82
1. 1.6353.00
0.5da0.co
0.97
0.88c0.1
String WL Kernel
SMILES
24.8aa7
0.30
0.66=0.01
0.29:0.0s
1.31
0.16
0.08::0.1
0. 0.72s0.01
graph
22.4:1.4
0.30
1.4:0.02
0.76
1.47a000
0.47ax.0e
fragprints
20.9..o6
1.63.04
0.88
1.701011
1.39
1.41
0.751004
3.82_
FC-BNN
fingerprints
22.4187
2.22_030
1.08.002
2.5018.4
1.83
2.65
1.81
1.74
fragments
0.601.000
1.03200
1,8310
1.48.9.00
D.872001
5.02.0m
GNN-BNN
graph SELFIES SELFIES
28.5+112
1.00=0.1s
0.88=6.0:
1.70s0.11
1.01e0.c2
1.1:001
Ensemble
26.4+10
4.31=0a8
0.07=0.01
2.91e0.14
1.2900.4
2.21s0.31
0.75e0.:1
2.60=00s
DKL
0.48:005
0.94=6.04
0.90sa.13
0.33
0.91a0.1
1:46:005
Experiments
We evaluate GAUCHE range of regression, uncertainty quantification (UQ) and Bayesian optimisation (BO) tasks. The principle goal in conducting regression and UQ benchmarks is to gauge whether performance on these tasks may be used proxy for BO performance. BO is powerful tool for automated scientific discovery but one would prefer to avoid model misspecification in the surrogate when deploying a scheme in the real world. We make of the following datasets with experimentally determined labels:
Photoswitch The labels y are the values of the isomer π n² transition wavelength for 392 photoswitch molecules [3].
ESOL The labels y are the logarithmic aqueous solubility values for 1128 organic small molecules [60].
FreeSolv The labels y are the hydration free energies for 642 molecules [41].
Lipophilicity The labels y are the octanol/water distribution coefficient (log at PH 7.4) of 4200 compounds curated from the ChEMBL. database [61, 62).
Buchwald-Hartwig reactions The labels are the yields for 3955 Pd-catalysed Buch-
Hard-Hartwig C-N cross-couplings [41].
Suzuki-Miyaura reactions The labels y are the yields for 5760 Pd-catalysed Suzuki- Miyaura C-C cross-couplings [63].
5.1 Regression and Uncertainty Quantification (UQ)
Experimental setup For the regression and uncertainty quantification experiments, all datasets were randomly split into training and test sets with a ratio of 80/20. (Note that validation sets are not required for the GP models, since hyperparameters objective on the train set). All GP models were trained using the L-BFGS- B optimiser [64] and, if not stated otherwise, the default settings in the GPyTorch and BoTorch libraries apply.
chosen using the marginal likelihood
To quantify model performance, the predictive accuracy and the calibration of the predictive uncer- tainty estimates of the fitted models were evaluated on the held-out test set and summarised as the root-mean-square error (R MSE) and the negative log predictive density (NIPPD), respectively. The mean and standard error of these metrics over 20 different random splits are reported in Table 1.

---

1.6
640
1.4
2.6
080
2.2
280
0.6
233
Random
Rassom
Kundon
Eigertss
27-
teraltine
kumber
(a) Photoswitch
(b) ESOL
(c) Buchwald-Hartwig reactions
Figure 4: BO performance reporting the standard error from 50 randomly initialised trials (20 for Buchwald- -artwwi.-- kernel density estimate over the trials shown on the right axis. EI fragprints results use the Tanimoto kernel. The random search baseline is indicated in blue.
Alternative metrics to measure the quality of the predictive uncertainty estimates such the mean standardised log loss (MSLL) and the quantile coverage error (QCE) are reported in Appendix A, while additional results for the reaction yield prediction datasets and scaffold splits are presented in Appendices B and C. We also benchmark the performance of GP models against range of Bayesian neural network- (BNN) and deep ensemble-based methods that are detailed in Appendix D.
Results Summarising our results in Table 1, we find that Tanimoto- based GPs generally outperform Scalar Product ones in terms of RMSD and NLDP, while string kernel-based GPs often yield even better performance. We additionally note that the quality of the predictive uncertainty estimates roughly correlates with predictive accuracy in the case of GP-based models.
While deep probabilistic models attained competitive results in terms of RMSD, we found their uncertainty estimates be consistently less reliable than those of GP-based models with discrete string kernels or shallow continuous kernels on hand-crafted features (e.g. fragprints). limiting their suitability for Bayesian optimisation and active learning. Our results suggest that for small to mid-sized molecular datasets the Tanimoto kernel combined with fragprint representations in particular very compelling option, with good accuracy, calibration, and runtime across all tasks.
5.2 Bayesian Optimisation
Building these results, employed the two best-performin kernels, namely the Tanimoto- fragprint kernel and the SMILES string kernel, to undertake Bayesian optimization BO over the photoswitch and ESOL datasets. BO is for 20 iterations of sequential candidate selection (EI acquisition) where candidates are drawn from 95% of the dataset. The models are initialised with 5% of the dataset. In the case of the photoswitch dataset, this corresponds to just 19 molecules. The results are provided in Figure 4. In this ultra-low data setting common to many areas of synthetic chemistry bothboth both models significantly outperform the random search baseline, highlighting the real-world use-case for such models in supporting human chemists to prioritise candidates for synthesis.
Furthermore, one may observe that BO performance is tightly coupled to regression and UQ perfor- mance. In the case of the photoswitch dataset, the better-performing Tanimoto model on regression and UQ also achieves better BO performance, while on the ESOL dataset, the string kernel performs best. Additionally, we run BO on the Buchwald- Hartwig dataset using the Tanimoto kernel for the bit-vector representations DRFP and OHE, and the RBF kernel for RXNFP. All three representations perform similarly and outperform the random search.
Finally, we also investigate the performance of GP-based models for preferential BO a setting in which the acquisition strategy only requires rank-based preferences of candidates. as opposed to their absolute objective function values [65, 66, 67, 68). We use a TTanimotofragprint kernel GP model to perform molecular Bayesian optimization on the photoswitch dataset using binary preference data alone and present the full results in Appendix E.4.

---

Table 2: An overview of existing open-source GP, BO and molecular machine learning libraries. With GAUCHE, we provide a modular, robust and easy-to-use framework that combines state- of-the-art probabilistic modelling and black-box optimization techniques with bespoke molecular representations and kernels to make them more easily accessible to the broader scientific community.
Gaussian Processes
Bayesian Optimisation
Molecular
Chemistry Tutorials
Graph Kernels
Bit Vector String Kerelss
Library
Representations
Kernels
GPyTorch [26]
GPilow [69, 70]
BoTorch [27]
X
DeepChem [71]
X
X
X
GraKel [59]
x
X
X
FlowMO [72]
X
X
GAUCHE (ours)
Related Work
General-purposes GP and BO libraries do not cater for molecular representations. Likewise, general- purpose molecular machine learning libraries do not consider GPs and BO. Here, we review existing libraries, highlighting the niche GAUCHE fills in bridging the GP and molecular machine learning communities. The closest work to ours is FlowMO [72], which introduces basic molecular GP library in the GPflow framework. In this project, extend the scope of the library broader class of molecular representations (graphs), problem settings (BO) and applications (reaction optimisation and protein engineering). An overview of how GAUCHE fits into the existing open-source GP, BO and molecular machine learning stack is presented in Table 2.
Gaussian Process Libraries GP libraries include GPy (Python) [73], GPflow (TensorFlow) [69, 70], GPyTorch (PyTorch) [26] and GPJax (Jax) [74] while examples of recent BO libraries include BoTorch (PyTorch) [27], Dragonfly (Python) [75]. HEBO (PyTorch) [76] and Trieste (Tensorflow) [77]. The aforementioned libraries do not explicitly support molecular representations. Extending them to cover molecular representations, however, requires implementations of bespoke GP kernels for bit vector, string and graph inputs together with modifications BO schemes to consider acquisition function evaluations over a discrete set of held-out molecules, setting commonly encountered in virtual screening [78, 79).
Molecular Machine Learning Libraries Molecular machine learning libraries include DeepChem [71], DGL-LifeSci [80] and TorchDrug [81]. DeepChem features a broad range of model imple- mentations and tasks, while DGL-LifeSci focuses on graph neural networks. TorchDrug caters for applications including property prediction, representation learning, retrosynthesis, biomedical knowledge graph reasoning and molecule generation. However, none the aforementioned libraries includes GP implementations. In terms of atomistic systems, DScribe [82] features, amongst other methods, the Smooth Overlap of Atomic Positions (SOAP) representation [83], which is typically used in conjunction with GP model to learn atomistic properties. Automatic Selection And Predic- tion (ASAP) [84] also principally focuses on atomistic properties as well as dimensionality reduction and visualisation techniques for materials and molecules. Lastly, the Graphein library focuses on graph representations of proteins [49].
Graph Kernel Libraries Graph kernel libraries include GraKel [59], graphkit-leara [85], graphk- and Chemo [88]. The aforementioned libraries focus on CPU implementations in Python. Extending graph kernel computation GPUs has been noted as an important direction for future research [89]. In our work, we build on the GraKel library to construct graph- graph-structured inputs. It is worth noting that GAUCHE extends the applicability of GPU-enabled GPs general graph-structured inputs beyond just molecules and proteins.
ernels [86], graph-kernels [87], pykernels hhtpss:/github..om//
GPS that can be trained on non-tensorial,
8

---

Molecular Bayesian Optimisation BO over molecular space can be divided into two classes. In the first class, molecules are encoded into the latent space of variational autoencoder (VAE) [4]. BO is then performed over the continuous latent space and queried molecules are decoded back to the original space. Much work on VAE-BO has focussed on improving the synergy between the surrogate model and the VAE [90, 5, 91, 92, 93, 94, 95, 96]. One of the defining characteristics of VAE-BO is that enables the generation of new molecular structures. In the second class of methods, BO is performed directly over the original discrete space of molecules. In this setting it is not possible to generate new structures and so a candidate set of queryable molecules defined. The inability to generate new structures however, is not bottleneck to molecule discovery as the principle concern is how best to explore existing candidate sets. These candidate sets are also known as molecular libraries in the virtual screening literature [97]. To date, there has been little work on BO directly over discrete molecular spaces. In [56], the authors use a string kemne GP trained on SMILES to perform BO select from a candidate set of molecules. In [98]. optimal transport
kernel GP is used for BO over molecular graphs. In [99] a surrogate based on the Nadarya- Watson estimator is defined such that the kernel density estimates are inferred using BNNs. The model is then trained on molecular descriptors. Lastly, in [100] and [101] a BNN and a sparse GP respectively are trained on fingerprint representations of molecules. In the case of the sparse GP the authors select an ArcCosine kernel. It is a longstanding aim of the GAUCHE Project to compare the efficacy of VAE-BO against vanilla BO on real-world molecule discovery tasks.
Chemical Reaction Optimisation Chemical reactions describe how reactants transform into products. Reagents (catalysts, solvents, and additives) and reaction conditions heavily impact the outcome of chemical reactions. Typically the objective is to maximise the reaction yield (the amount of product compared to the theoretical maximum) [41], in asymmetric synthesis, where the reactions could result different enantiomers, to maximise the enantiomeric excess [102], or to minimise the E-factor, which is the ratio between waste materials and the desired product [ [103]. A diverse set of studies have evaluated the optimisation of chemical reactions in single and multi- mujti-objecctve settings [103, 104]. [105] and [106] benchmarked reaction optimisation algorithms in low low-dimensional settings including reaction conditions, such as time, temperature, and concentrations. [6] suggested BO general tool for chemical reaction optimisation and benchmarked their approach against human experts. [107] compared the yield prediction performance of different kernels and [108] the impact of various molecular representations. In all reaction optimisation studies above, the representations of the different categories of reactants and reagents are concatenated to generate the reaction input vector, which could lead to limitations if another type of reagent is suddenly considered. Moreover, most studies concluded that simple one-hot encodings (OHE) perform at least on par with more elaborate molecular representations in the low-data regime [6. 108, 109]. In GAUCHE, we introduce reaction fingerprint kernels, based on existing reaction fingerprints [46, 45] and work independently of the number of reactant and reagent categories.
7 Limitations
One potential limitation of GAUCHE is the focus on core implementations that are likely to remain robust across many applied problems as possible which will enable the library to have the most impact. As such, there is less of focus on bespoke GP implementations for more targeted problems [110, 111, 112, 113]. Nonetheless, we hope that GAUCHE will function an active development platform for such implementations.
8 Conclusion
We have introduced GAUCHE, a library for Gaussian Processes in Chemistry, with the aim of providing a user-friendly and robust library of state-of-the-ee uncertainty quantification and Bayesian optimisation tools that may hopefully be deployed for screening in laboratory settings. Our aim is to maintain a lean, well-tested and up-to-date codebase and invite community-driven contributions principally as pull requests in the form of notebooks that reflect the needs and considerations that researchers come across in practice. In this fashion, we may support more advanced features without bloating the codebase and increasing maintenance requirements.

---

Acknowledgements
BR and PS acknowledge support from the NCCR Catalysis (grant number 180544), a National Centre of Competence in Research funded by the Swiss National Science Foundation. ARJ is funded by a Biotechnology and Biological Sciences Research Council (BBSRC) DTP studentship (BB/M01 1 194/1), GT acknowledges the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), and the Vector Institute. FS-K is postdoctoral fellow in the Eric and Wendy Schmidt AI in Science Postdoctoral Fellowship Program, program by Schmidt Futures. AA-G acknowledges the generous support of Anders G. Froseth, the Canadian Institute for Advanced Research (CIFAR), and the Canada 150 Research Chair program. LK acknowledges support from the University of Oxford's Clarendon Fund.
R-RG declares that this work was done exclusively at the University of Cambridge. Code/data was not created/accessed by Meta.
References
[1] Ying Zhang and Chen Ling. A strategy to apply machine learning to small datasets in materials science. Npj Computational Materials, 2018
[2] Ryan-Rhys Griffiths, Jiachen Jiang, Douglas JK Buisson, Dan Wilkins, Luigi C Gallo, Adam Ingram, Dirk Grupe, Erin Kara, Michael L Parker, William Alston, et al. Modeling the multiwavelength variability of Mrk 335 using Gaussian processes. The Astrophysical Journal, 914(2):144, 2021.
[3] Ryan-Rhy Rhys Griffiths, Jake L Greenfield, Aditya R Thawani, Arian R Jamasb, Henry B Moss, Anthony Bourached, Penelope Jones, William McCorkindale, Alexander A Aldrick, Matthew J Fuchter, al. Data-driven discovery of molecular photoswitches with multioutput Gaussian processes. Chemical Science, 2022.
[4] Rafael Gómez-Bombarelli. Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengelingg Dennis Sheberla, Jorge Aguilera-Iparraguiee Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science, 2018
[5] Ryan-Rhys Griffiths and José Miguel Hernández-Lobato. Constrained Bayesian optimization for automatic chemical design using variational autoencoders. Chemical Science, 2020.
[6] Benjamin Shields, Jason Stevens, Jun Li, Marvin Parasram, Farhan Damani, Jesus I Martinez Alvarado, Jacob M Janey, Ryan P Adams, and Abigail G Doyle. Bayesian reaction optimization as tool for chemical synthesis. Nature, 2021.
[7] Yuanqi Du, Tianfan Fu, Jimeng Sun, and Shengehao Liu. Molgensurvey: A systematic survey in machine learning models for molecule design. arXiv preprint arXiv:2203. 2022.
[8] Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sun- daram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable Bayesian optimization using deep neural networks. In International Conference on Machine Learning, 2015.
[9] Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian opti- mization with robust Bayesian neural networks. Advances in Neural Information Processing Systems, 2016.
[10] Yao Zhang et al. Bayesian semi eumi-superiised learning for uncertaintyy prediction of molecular properties and active learning. Chemical Science, 2019.
[11] Kevin Maik Jablonka, Giriprasad Melpatti Jothiappan, Shefang Wang, Berend Smit, and Brian Yoo. Bias free multiobjective active learning for materials design and discovery. Nature Communications, 2021.
[12] Seongok Ryu, Yongchan Kwon, and Woo Youn Kim. A bayesian graph convolutional network for reliable prediction of molecular properties with uncertainty quantification. Chemical Science, 2019.
10

---

[13] Doyeong Hwang, Grace Lee, Hanseok Jo, Seyoul Yoon, and Seongok Ryu. A bench- mark study on reliable molecular supervised learning via Bayesian learning. arXiv preprint urXiv:2006.002)) 2020.
[14] Gabriele Scalia, Colin A Grambow, Barbara Pernici, Yi-Pei Li, and William H Green. Eval- uating scalable uncertainty estimation methods for deep learning- based molecular property prediction. Journal of Chemical Information and Modeling, 2020.
[15] Yoshua Bengio. What are some Advantages of Using Gaussian Process Models vs Neural Networks?, 2011. https//www.qqurraa
Gaasssinn-
[16] Gary Tom, Riley J Hickman, Anizet Zinzuwadia, Afshan Mohajeri, Benjamin Sanchez- Lengeling, and Alan Aspuru-Guziie Guzik. Calibration and generalizability of probabilistic models on low-data chemical datasets with dionysus. arXiv preprint arXiv22212.01574, 2022.
[17] David JC MacKay, David JC Mac Kay, et al. Information theory, inference and learning algorithms. Cambridge University Press, 2003.
[18] Eric Anderson, Gilman D) Veith, and David Weininger. SMILES, line notation and computer- ized interpreter for chemical structures. Environmental Research Laboratory, 1987.
[19] David Weininger. SMILES, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences, 1988.
[20] Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik. Self-referencing embedded strings (SELFIES): 100% robust molecular string representation.
Muchine Learning: Science and Technology, 2020.
[21] David Rogers and Mathew Hahn. cxteniid- fingerprints. Journal of Chemical Information and Modeling, 2010.
[22] Daniel Probst and Jean-Louis Reymond. A probabilistic molecular fingerprint for big data settings. Journal of Cheminfornatics, 2018.
[23] Alice Capecchi, Daniel Probst, and Jean-Louis Reymond. One molecular fingerprint to rule them all: Drugs, biomolecules, and the metabolome. Journal of Cheminformatics, 2020.
[24] David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. Advances in Neural Information Processing Systems, 2015.
[25] Steven Kearnes, Kevin McCloskey, Marc Berndl, Vijay Pande, and Patrick Riley. Molecular graph convolutions: Moving beyond fingerprints. Journal of Computer-Aided Molecular Design, 2016.
[26] Jacob Gardner, Geoff Pleiss, Kilian Q Weinberger, David Bindel, and Andrew Wilson. GPyTorch: Blackbox matrix-mate matrix Gaussian process inference with GPU acceleration. In Advances in Neural Information Processing Systems, 2018
[27] Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy, BoTorch: A framework for efficient Monte-Carlo Bayesian optimization. Advances in Neural Information Processing Systems, 2020.
[28] Carl Edward Rasmussen and Zoubin Ghahramani. Occam's razor. In Advances in Neural Information Processing Systems, 2001.
[29] Eric Brochu, Vlad M Cora, and Nando De Freitas. A tutorial on Bayesian optimization of ex- pensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint /0 2010.
[30] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan Adams, and Nando de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 2016.

---

[31] Peter I Frazier. A tutorial on Bayesian optimization. arXiv preprint 1,.10:1807.02811,1 2018.
[32] Harold Kushner. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Joint Automatic Control Conference, 1963.
[33]
Jonas Močkus. On Bayesian methods for seeking the extremum. In Optimization techniques IFIP technical conference, 1975.
[34] A. Zhilinskas. Single-step Bayesian search method for an extremum of functions of a single variable. Cybernetics, 1975.
[35] Donald R. Jones, Matthias Schonlau, and William Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 1998.
[36] Antoine Grosnit, Alexander I Cowen- Rasul Tutumov, Ryan-Rhys Griffiths, Jun Wang, and Haitham Bou-Ammar. Are we forgetting about compositional optimisers in Bayesian optimisation? arXiv preprint arXiv:2012.08244: 2020.
[37] Philipp Hennig and Christian J Schuler. Entropy search for information-efficient global optimization. Journal of Machine Learning Research, 2012.
[38] José Miguel Hernández-Lobatt Matthew W Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. Advances in Neural Information Processing Systems, 2014.
[39] Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient Bayesian optimization. In International Conference on Machine Learning, 2017.
[40] Henry B Moss, David S Leslie, Javier Gonzalez, and Paul Rayson. Gibbon: Gereral-purpose information-based Bayesian optimisation. Journal of Machine Learning Research, 2021.
[41] Derek T Ahneman, Jesús G Estrada, Shishi Lin, Spencer D) Dreher, and Abigail G Doyle. Predicting reaction performance in C-N cross-coupling using machine learning. Science, 2018.
[42] Frederik Sandfort, Felix Ktrthth-Kalthoff, Marius Kühnemund, Christian Beecks, and Frank Glorius. A structure-based platform for predicting chemical reactivity. Chem, 2020.
[43] Kangway Chuang and Michael Keiser. Comment on "predicting reaction performance in c-n cross-coupling using machine learning". Science, 2018.
[44] Nadine Schneider, Daniel M Lowe, Roger A Sayle, and Gregory A Landrum. Development of novel fingerprint for chemical reactions and its application to large-scale reaction classification and similarity. Journal of Chemical Information and Modeling, 2015.
[45) Daniel Probst, Philippe Schwaller, and Jean-Louis Reymond. Reaction classification and yield prediction using the differential reaction fingerprint DRFP. Digital Discovery. 2022.
[46] Philippe Schwaller, Daniel Probst, Alain C Vaucher, Vishnu H Nair, David Kreutter, Teodoro Laino, and Jean-Louis Reymond. Mapping the space of chemical reactions using attention- based neural networks. Nature Machine Intelligence, 2021.
[47] Jacob Devlin Kenton, Chang Ming- and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.
[48] Philippe Schwaller, Alain C Vaucher, Teodoro Laino, and Jean-Louis Reymond. Prediction of chemical reaction yields using deep learning. Machine learning: Science and Technology. 2021.
[49] Arian R. Jamasb, Ramon Viñas, Eric J. Ma, Charlie Harris, Kexin Huang, Dominic Hall, Pietro Lió, and Tom L. Blundell. Graphein a Python library for geometric deep learning and network analysis on protein structures and interaction networks. bioRxiv, 2021.
[50] John C Gower. A general coefficient of similarity and some of its properties. Biometrics, 1971.
12

---

[51] Liva Ralaivola, Sanjay J Swamidass, Hiroto Saigo, and Pierre Baldi. Graph kernels for chemical informatics. Neural networks, 2005.
[52] Huma Lodhi, Craig Saunders, John Shawe- Shawe-Taylor, Nello Cristianini, and Chris Watkins. Text classification using string kernels. Journal of Machine Learning Research, 2002.
[53] Nicola Cancedda, Eric Gaussier, Cyril Goutte, and Jean Michel Renders. Word sequence kernels. Journal of Machine Learning Research, 2003.
[54] D-S Cao, J-C Zhao, Y-N Yang, C-X Zhao, J Yan, S Liu, Q-N Hu, Q-S Xu, and Y-Z Liang, In silico toxicity prediction by support vector machine and SMILES representation based string kernel. SAR and QSAR in Environmental Research, 2012.
[55] Daniel Jurafsky and James H Martin. An introduction to natural language processing, compu- tational linguistics, and speech recognition, 2000.
[56] Henry Moss, David Leslie, Daniel Beck, Javier Gonzalez, and Paul Rayson. BOSS: Bayesian optimization over string spaces. Advances Neural Information Processing Systems, 2020.
[57] Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(9), 2011.
[58] Boris Weisfeiler and Andrei Leman. The reduction of graph to canonical form and the algebra which appears therein. nti, Series, 2(9):12-16, 1968.
[59] Giannis Siglidis, Giannis Nikolentzos, Stratis Limnios, Christos Giatsidis, Konstantinos Skianis, and Michalis Vazirgiannis. Grakel: A graph kernel library in Python. Journal of Machine Learning Research, 2020.
[60] John S Delaney. ESOL: Estimating aqueous solubility directly from molecular structure. Journal of Chemical Information and Computer Sciences, 2004.
[61] Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. ChEMBL.:
A large-scale bioactivity database for drug discovery. Nucleic Acids Research, 2012.
[62] A Patrícia Bento, Anna Gaulton, Anne Hersey, Louisa Bellis, Jon Chambers, Mark Davies, Felix A Krüger, Yonnn Light, Lora Mak, Shaun McGlinchey, et al. The ChEMBL bioactivity database: An update. Nucleic Acids Research, 2014.
[63] Damith Perera, Joseph W Tucker, Shalini Brahmbhatt, Christopher Helal, Ashley Chong, William Farrell, Paul Richardson, and Neal W Sach. A platform for automated nanomole-scale reaction screening and micromole-scale synthesis in flow. Science, 2018
[64] Dong C Liu and Jorge Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical Programming, 1989.
[65] Eric Brochu, Nando Freitas, and Abhijeet Ghosh. Active preference learning with discrete choice data. Advances in Neural Information Processing Systems, 20, 2007.
[66] Eric Brochu. Interactive Bayesian optimization: Learning user preferences for graphics and animation. PhD thesis, University of British Columbia, 2010.
[67] Javier González, Zhenwen Dai, Andreas Damianou, and Neil D Lawrence. Preferential Bayesian optimization. In International Conference on Machine Learning, pages 1282-1221 1291. PMLR, 2017.
[68] Zhiyuan Jerry Lin, Raul Astudillo, Peter Frazier, and Eytan Bakshy. Preference exploration for efficient Bayesian optimization with multiple outcomes. In International Conference on Artificial Intelligence and Statistics, pages 4235-4258. PMLR, 2022.
[69] A. G. Matthews, Mark van der Wilk, Tom Nickson, Keisuke Fujii, Alexis Boukouvalas, Pablo León- Vill-Villagrá, Zoubin Ghahramani, and James Hensman. GPflow: Gaussian process library using TensorFlow. Journal of Machine Learning Research, 2017.
13

---

[70] Mark van der Wilk, Vincent Dutordoir, ST John, Artem Artemev, Vincent Adam, and James Hensman. A framework for interdomain and multioutput Gaussian processes. arXiv preprint urXiv2200300//33 15, 2020.
[71] Bharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu. Deep Learning for the Life Sciences. O'Reilly Media, 2019.
[72] Henry B. Moss and Ryan Rhys Griffiths. Gaussian process molecule property prediction with FlowMO, arXiv preprint arXiv:2010.00118, 2020.
[73] GPy. GPy: A Gaussian process framework in /t GPy, since 2012.
[74]
Thomas Pinder and Daniel Dodd. Gpjax: A gaussian process framework in jax. Journal of Open Source Software, 7(75):4455, 2022.
[75] Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christo- pher R Collins, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Tuning hyperparameters without grad students: Scalable and robust Bayesian optimisation with Dragonfly. Journal of Machine Learning Research, 2020.
[76] Alexander Cowen-Rivers, Wenlong Lyu, Rasul Tutunov, Zhi Wang, Antoine Grosnit, Ryan Rhys Griffiths, Alexandre Max Maraval, Hao Jianye, Jun Wang, Jan Peters, et al. An empirical study of assumptions Bayesian optimisation. arXiv preprint arXiv:2012.038: 0.38226 2020.
[77] Joel Berkeley, Henry B. Moss, Artem Artemev, Sergio Pascual-Diaz, Uri Granta, Hrvoje Stojic, Ivo Couckuyt, Jixiang Qing, Nasrulloh Loka, Andrei Paleyes, Sebastian W. Ober, and Victor Picheny. Trieste. http://gghuub..oo////coommt -abssiiiiiees/s//eeteteeeee 2022.
[78] Edward Krapp. Using Bayesian optimization to accelerate virtual screening for the discovery of therapeutics appropriate for repurposing for COVID-19. arXiv preprint arXiv:2005.0712/. 2020.
[79] David E Graff, Matteo Aldeghi, Joseph A Morrone, Kirk E Jordan, Edward O Kyzer-Knapp, and Connor W Coley. Self-focusing virtual screening with active design space pruning. arXiv preprint arXiv:2205.0175. 2022.
[80] Mufei Li, Jinjing Zhou, Jiajing Hu, Wenxuan Fan, Yangkang Zhang, Yaxin Gu, and George Karypis. DGL-LifeSed An open-source toolkit for deep learning on graphs in life science. ACS Omega, 2021.
[81] Zhaocheng Zhu, Chence Shi, Zuobai Zhang, Shengehao Liu, Minghao Xu, Xinyu Yuan, Yangtian Zhang, Junkun Chen, Huiyu Cai, Jiarui Lu, Chang Ma, Runcheng Liu, Louis- Xhonneux, Meng Qu, and Jian Tang. TorchDrug: powerful and flexible machine learning platform for drug discovery. arXiv preprint 2rXiv:2202.08320 2022.
[82] Lauri Himanen, Marc OJ Jäger, Eiaki Morooka, Filippo Federici Canova, Yashasvi Ranawat, David Z Gao, Patrick Rinke, and Adam S Foster. DScribe: Library of descriptors for machine learning in materials science. Computer Physics Communications, 2020.
[83] Albert P Bartók, Risi Kondor, and Gābor Csányi. On representing chemical environments. Physical Review B, 2013.
[84] Bingqing Cheng, Ryan-Rhys Griffiths, Simon Wengert, Christian Kunkel, Tamas Stenezel, Bonan Zhu, Volker L. Deringer, Noam Bernstein, Johannes T Margraf, Karsten Reuter, et al. Mapping materials and molecules. Accounts of Chemical Research, 2020.
[85] Linlin Jia, Benoit Gaüzère, and Paul Honeine. graphkit-leam: A Python library for graph kernels based on linear patterns. Pattern Recognition Letters, 2021.
[86] Mahito Sugiyama, Elisabetta Ghisu, Felipe Llinares-Lóópez, and Karsten Borgwardt. graphk- emels: R and Python packages for graph comparison. Bioinformatics, 2018.
14

---

[87]
Mahito Sugiyama and Karsten Borgwardt. Halting in random walk kernels. Advances in Neural Information Processing Systems, 2015.
[88] Benoit Gaüzére, Luc Brun, Didier Villemin, and Myriam Brun. Graph kernels based on relevant patterns and eycle information for chemoinformaties. In Proceedings of the 21 International Conference on Puttern Recognition, 2012.
[89] Swarnendu Ghosh, Nibaran Das, Teresa Gonçalves, Paulo Quaresma, and Mahantapas Kundu. The journey of graph kernels through two decades. Computer Science Review, 2018.
[90] Ryan-Rhys Griffiths, Philippe Schwaller, and Alpha A Lee. Dataset bias the natural sciences: A case study in chemical reaction prediction and synthesis design. arXiv preprint SoXiv:2105.02633 2021.
[91] Austin Tripp. Erik Daxberger, and José Miguel Hernández-Lobato. Sample-efficient optimiza- tion in the latent space of deep generative models via weighted retraining. Advances in Neural Information Processing Systems, 2020.
[92] Aryan Deshwal and Jana Doppa. Combining latent space and structured kernels for Bayesian optimization over combinatorial spaces. Advances in Neural Information Processing Systems, 2021.
[93] Antoine Grosnit, Rasul Tutunov, Alexandre Max Maraval, Ryan-Rhys Griffiths, Alexander I Cowen-Rivers, Lin Yang. Lin Zhu, Wenlong Lyu, Zhitang Chen, Jun Wang, et al. High- dimensional Bayesian optimisation with variational autoencoders and deep metric learning. arXiv preprint arXiv:2106.03609 2021.
[94] Ekansh Verma and Souradip Chakraborty. Uncertainty-awart labelled augmentations for high dimensional latent space Bayesian optimization. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[95] Natalie Maus, Haydn Jones, Juston S Moore, Matt Kusner, John Bradshaw, and Jacob R Gardner. Local latent space Bayesian optimization over structured inputs. arXiv preprint WrXiv:2201111822 2022.
[96] Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating Bayesian optimization for biological sequence design with denoising autoencoders. arXiv preprint arXiv:2203. 12742, 2022.
[97] Edward O Pyzer-Knaph Changwon Suh, Rafael Boanez-Bommbrrlli.. Jorge Aguilera- Iparraguirre, and Alán Aspuru What high-throughput virtual screening? A perspec- tive from organic materials discovery. Annual Review of Materials Research, 2015.
[98] Ksenia Korovina, Sailun Xu, Kirthevasan Kandasamy, Willie Neiswanger, Barnabas Poczos, Jeff Schneider, and Eric Xing. ChemBO: Bayesian optimization of small organic molecules with synthesizable recommendations. International Conference on Artificial Intelligence and Statistics, 2020.
[99] Florian Häse, Matteo Aldeghi, Riley J Hickman, Loic M Roch, and Alán Aspuru- Guzik. Gryffin: An algorithm for Bayesian optimization of categorical variables informed by expert knowledge. Applied Physics Reviews, 2021.
[100] José Miguel Hernández-Lobate James Requeima, Edward O Knapp. and Alán Aspuru- Guzik. Parallel and distributed Thompson sampling for large-scale accelerated exploration of chemical space. In International Conference on Machine Learning. 2017.
[101] Sattar Vakili, Henry Moss, Artem Artemev, Vincent Dutordoir, and Victor Picheny. Scalable Thompson sampling using sparse Gaussian process models. Advances in Neural Information Processing Systems, 2021.
[102] Andrew F Zahrt, Jeremy J Henle, Brennan T Rose, Yang Wang, William T Darrow, and Scott E Denmark. Prediction of higher-selectivity catalysts by computer-driven workflow and machine learning. Science, 2019.
15

---

[103] Artur M Schweidmann, Adam D Clayton, Nicholas Holmes, Eric Bradford, Richard A Bourne, and Alexei A Lapkin. Machine learning meets continuous flow chemistry: Automated optimization towards the pareto front of multiple objectives. Chemical Engineering Journal, 2018.
[104] Pia Müller, Adam D Clayton, Jamie Manson, Samuel Riley, Oliver May, Norman Govan, Stuart Notman, Steven V Ley, Thomas W Chamberlain, and Richard A Bourne. Automated multi-objective reaction optimisation: Which algorithm should use? Reaction Chemistry & Engineering, 2022.
[105] Kobi C Felton, Jan G Rittig, and Alexei A Lapkin. Summit: Benchmarking machine learning methods for reaction optimisation. Chemistry-Methodss 2021.
[106] Florian Häse, Matteo Aldeghi, Riley J Hickman, Loïe M Roch, Melodic Christensen, Elena Liles, Jason E Hein, and Alán Aspuru-Guzik, Olympus: A benchmarking framework for noisy optimization and experiment planning. Machine Learning: Science and Technology, 2021.
[107] Alexe L. Haywood, Joseph Redshaw, Magnus WD Hanson-Heine, Adam Taylor, Alex Brown, Andrew M Mason, Thomas Gärtner, and Jonathan Hirst. Kernel methods for predicting yields of chemical reactions. Journal of Chemical Information and Modeling, 2021.
[108] Alexander Pomberger, Antonio Pedrina McCarthy, Ahmad Khan, Simon Sung, Connor Taylor, Matthew Gaunt, Lucy Colwell, David Walz, and Alexei Lapkin. The effect of chemical representation on active machine learning towards closed-loop optimization. ChemRxiv, 2022.
[109] Riley Hickman, Jurgis Ruža, Loic Roch, Hermann Tribukait, and Alberto Gareía-Durán. Equipping data-driven experiment planning for self-driving laboratories with semantic memory: Case studies of transfer learning in chemical reaction optimization. ChemRxiv, 2022.
[110] Natalie S Eyke, William H Green, and Klavs Jensen. Iterative experimental design based on active machine learning reduces the experimental burden associated with reaction screening. Reaction Chemistry Engineering, 2020.
[111] Bojana Rankovic, Ryan-Rhys Griffiths, Henry B. Moss, and Philippe Schwaller. Bayesian optimisation for additive screening and yield improvements in chemical reactions - beyond one-hot encodings. ChemRxiv, 2022.
[112] Ryan-Rhys Griffiths. Gaussian Processes at Extreme Lengthscales: From Molecules to Black Holes. PhD thesis, University of Cambridge, 2023.
[113] Henry B Moss, Sebastian W Ober, and Victor Picheny. Inducing point allocation for sparse Gaussian processes in high-throughput Bayesian optimisation. In Artificial intelligence and statistics, 2023.
[114] Guy W Bemis and Mark A Murcko. The properties of known drugs. 1. molecular frameworks. Journal of medicinal chemistry, 39(15):2887-288 1996.
[115] W Patrick Walters and Regina Barzilay. Applications of deep learning in molecule generation and molecular property prediction. Accounts of Chemical Research, 54(2):263 2020.
[116] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre- trainininn graph neural networks. In International Conference on Learning Representations, 2019.
[117] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pages 370-378, PMLR, 2016.
[118] Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are bayesian neural network posteriors really like? In International conference machine learning, pages 4629-4640. PMLR, 2021.
[119] Yinglu Cui, Yanchun Chen, Xinyue Liu, Saijun Dong, Yu'e Tian, Yuxin Qiao, Ruchira Mitra, Jing Han, Chunli Li, Xu Han, et al. Computational redesign of a petase for plastic biodegradation under ambient condition by the GRAPE strategy. ACS Catalysis, 1 11(3):1340- 1350, 2021.
16

---

[120] Seongjoon Joo, Injin Cho, Hogyun Seo, Hyeoncheol Francis Son, Hye- Young Sagong, Tae Joo Shin, So Young Choi, Sang Yup Lee, and Kyung-Jin Kim. Structural insight into molecular mechanism of poly(ethylene terephthalate) degradation. Nature Communications, 9, 2018.
[121] Bing Liu, Lihui He, Liping Wang, Tao Li, Chang-Chenn Li, Huayi Liu, Yunzi Luo, and Rui Bao. Protein crystallography and site-direct mutagenesis analysis of the poly(ethylene terephthalate) hydrolase PETase from Ideonella sakaiensis. ChemBioChem, 19, 2018.
[122] Shosuke Yoshida, Kazumi Hiraga, Toshihiko Takehana, Ikuo Taniguchi, Hironao Yamaji, Yasuhito Maeda, Kiyotsuna Toyohara, Kenji Miyamoto, Yoshiharu Kimura, and Kohei Oda. A bacterium that degrades and assimilates poly (ethylene terephthalate). Science, 351 1199, 2016.
[123] Urszula Chajewska, Daphne Koller, and Ronald Parr. Making rational decisions using adaptive utility elicitation. In Proceedings of the Seventeenth National Conference on Artificial Intel- ligence and Twelfih Conference on Innovative Applications of Artificial Intellignce, pages 363 363-369, 2000.
[124] Johannes Fürnkranz and Eyke Hüllermeier. Pairwise preference learning and ranking. In Machine Learning: ECML 2003: 14th European Conference on Machine Learning. Cavtat- Dubrovnik, Croatia, September 22-26, 2003. Proceedings 14, pages 145-156. Springer, 2003,
[125] Craig Boutilier. A POMDP formulation of preference elicitation problems. In Eighteenth National Conference Artificial Intelligence, page 239-246. American Association for Artificial Intelligence, 2002.
[126] Johannes Fürnkranz and Eyke Hüllermeier. Preference Learning. Springer Science & Business Media, 2010.
[127] G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature Chemistry, 4(2):90-98, 2012.
[128] Ryan-Rhys Griffiths, Philippe Schwaller, and Alpha A Lee. Dataset bias the natural sciences: case study in chemical reaction prediction and synthesis design. arXiv preprint OrXiv:2105.02637 2021.
[129] Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. GuacaMol: bench- marking models for de novo molecular design. Journal of chemical information and modeling, 1996) 2019.
[130] liris Sundin, Alexey Voronov, Haoping Xiao, Kostas Papadopoulos, Esben Jannik Bjerrum, Markus Heinonen, Atanas Patronov, Samuel Kaski, and Ola Engkvist. nouman-in-the-blee assisted de novo molecular design. Journal of Cheminformatics, 14(1):1-16, 2022.
[131] Oh-Hyeon Choung, Riccardo Vianello. Marwin Segler, Nikolaus Stiefl, and José Jiménez-Luna, Learning chemical intuition from humans in the loop. ChemRxiv, 2023.
[132] Wei Chu and Zoubin Ghahramani. Preference learning with Gaussian processes. In Proceed- ings of the 22nd international conference on Machine learning, pages 137-144, 2005.
[133] Raul Astudillo, Zhiyuan Jerry Lin, Eytan Bakshy, and Peter Frazier. qEUBO: A decision- theoretic acquisition function for preferential Bayesian optimization. In International Confer- ence on Artificial Intelligence and Statistics, pages 1093-1114. PMLR, 2023.
[134] Seung-Seok Choi, Sung-Hyuk Cha, Charles C Tappert, al. A survey of binary similarity and distance measures. Journal of systemics, cybernetics and informatics, 8(1):43-48, 2010.
[135] Rafic Nader, Alain Bretto, Bassam Mourad, and Hassan Abbas. On the positive semi remi-definite property of similarity matrices. Theoretical Computer Science, 755:13 28, 2019.
[136] Daniel P Faith, Peter R Minchin, and Lee Belbin. Compositional dissimilarity robust measure of ecological distance. Vegetatio, 69:57-68, 1987.
[137] Stephen Forbes. On the local distribution of certain Illinois fishes: an essay statistical ecology, volume 7. Illinois State Laboratory of Natural History, 1907.
17

---

[138] Stephen A Forbes. Method of determining and measuring the associative relations of species. Science, 61(1585):518-5555 5204, 1925.
[139] Robert R Sokal, Peter HA Sneath, et al. Principles numerical taxonomy. Principles of numerical laxonomy, 1963.
18

---

Supplementary to:
GAUCHE: A Library for Gaussian Processes in Chemistry
A Uncertainty Quantification Experiments
In Table 3 we present further uncertainty quantification results using the quantile coverage error (QCE) metric. Numerical errors were encountered with the WL kernel the large lipophilicity dataset and the corresponding entry is left blank.
Table 3: UQ benchmark. QCE values (4) for 80/20 train/test split across 20 random trials.
GP Representation
Kemel
Photoswitch
ESOL
FreeSolv
Lipophilicity
Tanimoto
fragprints
0.019
0.023
0.002
0.023
0.002
0.002
fingerprints
0.023 = 0.003
0.022 + 0.002
0.018 = 0. 0.003
0,006 + 0.001
fragments
0.025 0.005
0.012 :: 0.002
0.014 0.002
0.009 0.002
Scalar Product
fragprints
0.033 = 0. .006
0,010 + 0.002
0.017 = 0. 0000
0,010 + 0.001
fingerprints
0.036 0.0 .006
0.014 0.002
0.016 0.002
0.009 0.001
fragments
0.027 0.004
0.012 + 0.008
0.021 = 0.003
0.010 0.001
String
SMILES
0.024 0.003
0.016 0.002
0.019 0.003
0.005
WL Kernel (GraKel)
gruph
0.025 = 0.007
0.011 + 0.004
0.019 = 0.009
B Chemical Reaction Yield Prediction Experiments
Further regression and uncertainty quantification experiments are presented in Table 4. The differ- ential reaction fingerprint in conjunction with the Tanimoto kernel is the best-performing reaction representation.
Table 4: Chemical reaction regression benchmark. 80/20 train/test split across 20 random trials.
GP Model
Buchwald-
Kemel
Representation
RMSH |
1t score
QCE
Tanimoto
7.94 0.05
0.001
DRFP
6.48 1 0.45
0.94 0.015
1.0002
Scalar Product
OHE
15.23 0.062
0.69 0.002
0.008 0.000
DRFP
14.63 0.050
0.010 0.002
RBF
RXNEP
1..00
0.84 = 0.001
0.024
Suzuki-Miyaura
Tanimoto
OHE
0.038
0.83 0.001
0.007 0.001
DRFP
0..46
0.001
0.019 1: 0.000
Scalar Product
OHE
10.01100002
0.47 0.003
0.012
DRFP
19.60
0.52 = 0.003
0.014 1: 0.001
RBF
RXNFP
13.8510018
0.75 = 0.002
0.007 0.000
C Scaffold Split Experiments
To investigate how GP models behave on more challenging train/test splits, we have re-run parts of our experimental evaluation with 80-20 Bemis-Mureko [114] scaffold splits instead of random splits. As only the lipophilicity dataset exhibits sufficient seaffold diversity to perform this analysis (the skewness of the scaffold distribution in the others makes 80-20 split impossible), the results Table focus on the predictive accuracy (R (RMSF) and calibration (NLPD) of GP models this setting.
19

---

While this more challenging evaluation setup leads to slightly higher RMSES and NLPDs, we note that one can observe the same trends as with random splits: Tanimoto-bra based GPs generally outperform Scalar Product ones, while string kernel-based GPs are better than both.
Table 5: Regression and uncertainty quantification experiments on scaffold splits.
Kernel
Representation
RMSE (1)
NLPD (L)
Tanimoto
Fragprints
0.8611.00
1.02
Fingerprints
0.8810.01
1.1210.04
Fragments
0.89
2.10
Scalar Product
Fragprints
0.89+0.01
1.7540.08
Fingerprints
0.95+0.01
1.99+0.09
Fragments
1.0010.01
String
SMILES
0.82=0.01
1.08+0.01
D
Deep Probabilistic Models
There is a growing body of work applying deep learning to molecular property prediction [115]. Therefore addition evaluating GPs with varying shallow kernel functions, we repeat the regression experiments with range of deep Bayesian models, varying both the network architecture and the Bayesian inference procedure. We evaluate the following models:
FC-BNN + VI is fully connected neural network with single variational inference (VI) Bayesian layer with 100 nodes, followed by the rectified linear unit (RcLU) activation, and final output layer. Trained with early stopping.
GNN-BNN VI utilises the same network and graph features used for the graph embed- dings [116] followed by a final VI Bayesian layer. Trained with early stopping.
CNN DKL GP is the same approach and architecture used by [96] to predict molccular properties for Bayesian optimisation. Using SELFIES representations, 1D CNN encoder is shared and trained jointly through a generative masked language model (MLM) head [47] and discriminative deep kernel GP head [117].
CNN Ensemble is deep ensemble of 1D CNN networks, also implemented by [96], where each ensemble component uses the SELFIES molecule representation and is trained independently to minimize the MSE loss. Deep ensembles have been shown provide high-fidelity approximations of Bayesian model averages relative to alternative approaches such as Laplace approximation VI [118].
E Further Experiments
E.1 Protein Fitness Prediction
We consider the task of protein fitness prediction where the fitness function (target label) takes the form of the melting point in degrees Celsius. We collate dataset of 151 PETase protein sequences from values reported in [119, 120, 121]. PETases, recently discovered in 2016 [122], are a class of esterase enzymes which, via hydrolysis, catalyse the breakdown of polyethylene terephthalate (PET) 2-hydroxyethy/ m hundreds of years to degrade naturally, PETases are capable of degrading PET in a matter of days but their melting point is key property of interest for deployed applications. Each sequence consists of an amino acid chain of length 290. We use the 'bag of amino acids' representation with a max n-gram value of 5 which gives rise to a count vector where each component represents the number of a given n-gram contained in the sequence. A max n-gram value of means that all n-grams up to and including length 5 are included in the fcaturisation. We subsequently train Tanimoto kernel GP
plastic to monomeric mono-2-
terephthalate (MHET). While PET plastics can require
20

---

on the featurisation. The results are provided in Table 6 and indicate that the GP model obtains low generalisation error on the melting point prediction task.
Table 6: PETase melting point prediction experiment. 80/20 train/test split across 20 random trials.
GP Model
PETase Test Set
Kemel
Representation
RMSE J
MAE +
score f 0.81 0.02
Tanimoto
Bag of Amino Acids
3.68 0.13
2.54 + 0.10
E.2 Molecular Preference Learning
In many optimisation problems, while it may be challenging define mathematical utility function, human feedback on pairwise comparisons can be leveraged to learn a latent utility function. In the machine learning literature, this observation has inspired work on preference learning [123, 124, 125, 126] where the goal is to learn a utility function 9xx) using binary preference data r(xi, X2) obtained from human feedback on inputs x. In the case of molecular design, there are many situations in which it is difficult to articulate a utility function [127, 128, 129] which has motivated recent work on human-in -i---ie---oooopppp preference learning [130, 131]. this setting, pairwise preferences r r(x1, X2) are collected from a human chemist by presenting with many choices over pairs of molecules, In this section, we highlight use-case for GAUCHE GP-based preference learning [132]. Using the photoswitch dataset as a case study, we simulate noiseless pairwise comparisons from a human chemist on the E isomer и x² transition wavelength (latent utility function). We train Tanimoto kernel GP on the fragprints representation of the molecules, using probit likelihood and the Laplace approximation [132]. The results are provided in Table 7. We report the Kendall- Tau rank correlation as the preference model learns ordinal rankings in place of absolute values of the transition wavelength. The results indicate that is possible to learn accurate models of molecular properties through binary feedback alone.
Table Photoswitch preference learning experiment. 80/20 train/test split across 20 random trials with 5000 training set pairwise comparisons. Kendall-Tau rank correlation (K-T) values reported.
GP Model
Photoswitch Dataset
Kernel
Representation
K-T Train 1 KeT
Tanimoto
Fragprints
0.95 0.001 0.79 +: 0.01
E.3 Ablation over the ECFP Radius Parameter
In addition the experimental results presented in Section we have performed an ablation study over the radius parameter of connecttitt fingerprints (ECFPs). Specifically, we trained Tanimoto-kernel GPs on the Photoswitch dataset using series of five increasing radius parameters. In Table 8, we report the mean and standard error of the RMSE and NLPD over 50 different 80-20 train-test-splits..
Table 8: RMSE and NLPD of Tanimoto-kemell GPs trained on the Photoswitch dataset using a
series of five increasing ECFP radius parameters.
Fingerprint
RMSE (L)
NLPD (L)
ECFP4
22.6510..55
0.41:0.05
ECFP6
23.50+0.55
0.47+0.05
ECFP8
24.43:10.54
0.52+0.05
ECFP10
25.1740..54
0.56+0.04
ECFP12
25.70+00.5
0.58+0.04
Intriguingly, these results show a strong negative correlation between the fingerprint radius and predictive performance. We hypothesize that this is caused by the fact that an expanding feature
21

---

space leads to lower and less informative Tanimoto similarity scores, making it more difficult to train generalisable models.
E.4 Preferential Bayesian Optimisation
In many Bayesian optimisation problems, the acquisition strategy only requires rank-based pref- erences of candidates opposed their absolute objective function values) to operate. Such an observation has motivated the field of Preferential Bayesian optimisation (PBO) [65, 66, 67, 68] where Bayesian optimisation is performed using binary preference data in lieu of the absolute values of the objective function. In Figure 5 we present the results of Bayesian optimisation loop on the photoswitch dataset featuring Tanimoto kernel surrogate with probit likelihood and the Laplace approximation [132], combined with the EUBQ ( acquisition function from [68, 133]. The results indicate that it possible to perform molecular Bayesian optimization using binary preference data alone.
Dest objecttre Value
610
590
Randiom
EUBO
580
Number of Rerations
Figure 5: PBO results on the Photoswitch dataset using the EUBO acquisition function. 20 random trials reporting the 1.96 standard error bands. The surrogate initialized with 98 molecules with 100 pairwise comparisons. 294 molecules are retained in a heldout set to be selected as part of the BO loop. single molecule is selected on each iteration and an additional 100 pairwise comparisons over the augmented train set are recorded.
22

---

Coding Kernels in GAUCHE
We provide an example of the class definition for the Tanimoto kernel in GAUCHE below
class Tanimote (ExactGP):
def in_itt___ (self, auper (TaninotoGP, self) int........tttttttn..... train_y, likelihood)
train_y, likelihood):
nean_nednle ConstantNean()
Ve use the Tanimoto hernel to work uktA
noleculor fingerprint representations
self covar_nodula ScaleKarnel ("aaninotokarmal
def forward(self,
nean_x self (ean__odull(x))
covar_x covar modulle() reture hultivariatallormal (nean_x, covar_x)
and an example definition of black box kernel (where gradients with respect to hyperparameters and input labels required).
class WiKernel (gauche. Kernel):
def _init_____ __elnt
super i__itt_____
self kernel karnel. Welaeellailstttlerehhhnnn
0iru_cache (maxeize3)
def kern(solf,
reture tensor kerle. kernel. fit_trranstorn( data))
class CraphdP(gauchaa SICP))
def -. i.,nit_.. (self, train.x, train.y, likellhood); super inttt train.y, likelihood)
self nean ConstantHean()
WLKernel()
def forward(self, I):
<a gauche. Isputa instance, with I.data holdsng 1ist of grakel. insfancee. nean
(zeroe (len (a.daa)) covariance self. colffaoorriancc (X) Maltivarriattllonall (maan, covariance)
Importantly, GAUCHE inherits all the facilities of GPy'Torch and GraKel allowing broad range of of models to be defined on molecular inputs such as deep GPs, multioutput GPs and heteroscedastic GPs.
23

---

G
Additional Bit and Count Vector Kernels
GAUCHE provides parallelisable and
implementations of the following simi-
larity measures from [134] that are provably symmetric and positive semi-definite [135]. All kernels defined for binary vectors x, x' {0. 1)dd for d 2 1, where n represents the number of common is the Euclidean norm.
zeros between x and x' and ||
Braun- -rauu-Blanquet Kernel
(x, x²) max | ||x(1|
B Blanquei (x,x') == =0
Dice Kernel
2 x') ||x|| + ||x'||
Ruice (x,,'xxxx))) 0j.
Faith Kernel [136]
kvaith (x,, (x,xx) = o3 2. (x,xx) x²) + n
2d
Forbes Kernel [137, 138]
krorbes x) == 07. d. (x, x') |||| + ||'|||
Intersection Kernel
Kintersoction (x,, (x,x') = == ((x, x) + xx(xx))
MinMax Kernel
|
+ |
x'
MinnMax
(X,MMaxXX,)))))))) (x, = 07
|
+ |x"
+
x
Otsuka Kernel
(x,x')
Kowska x') = 03
||| + xx'
Rand Kernel
(x,x) +n
BRand (x, x)) :=)== 07
d
Tanimoto Kernel
xx) +
Rogers-Tanimate Tanimato (x, x') =0z
o.. 2. ||x + 2. ||x||
3 (xxx')
Russell-Rao Kernel
BRussell-RRo = 07 (x,x') d
Sorgenfrei Kernel
(x, x²²
Ksorgenfrei (x, x') = =og
|x|| + ||x
Sokal-Sneath Kernel
[139]
(x, x")
kisukal Sneath (x, x') == =0}
2. ||x| + 2- ||x ||
- 3 (x,x')
24


Reviews:
[{"review_id": "uS3KY6fKpV", "reviewer": "Reviewer_rK4e", "summary": "The authors present a framework called GAUCHE with comprehensive exploration of Gaussian Processes (GP) and their application to molecular machine learning. The authors thoroughly examine different ways of representing molecular structures - through hand tuned fingerprints, string notations (SMILES/SELFIES/Protein sequences), and undirected graphs. The authors evaluate the proposed kernels and representation schemes across several diverse benchmarks involving regression tasks, uncertainty quantification, and Bayesian optimization (BO). The experiments use diverse datasets, representing a span of property prediction from single molecule and reaction yield prediction from sets of molecules. Finally, the authors evaluate the best-performing GAUCHE kernels in a Bayesian optimization framework using three distinct datasets. Results reveal that GAUCHE's kernels outperforms random baseline, especially in low data regime, highlighting its practical utility in aiding chemists to prioritize synthesis candidates. Overall, this is an impressive work paper with strong theoretical framework with rigorous empirical evaluation.", "questions": "1. How does the splitting strategy affect the results? For example, what if the data were split based on molecular scaffolds, which would ensure the structural diversity of the test set, instead of a random split? \n2. In the context of the Uncertainty Quantification benchmark, how does the predicted variance of the model relate to OOD samples? Could you provide a plot of the Tanimoto distance to the training dataset versus the predicted variance?\n3. The authors touch upon the potential use of embeddings of molecules from pretrained models as inputs for the kernels, but there doesn't seem to be any results for this approach. Is this something the authors are considering for future investigations?\n4. How would the performance of the kernels be impacted with the changes in the hand tuned fingerprint radius? Have any ablation studies been conducted to investigate this?", "limitations": "1. The GPR models can be computationally intensive, especially for larger datasets if used in conjunction with large dimensional fingerprints etc. This could limit the applicability in scenarios where abundant high-quality data is available. \n2. The performance of predictive models in low data regimes is heavily affected by the choice of dataset splitting method. The results obtained using random splitting might not hold if other splitting methods like scaffold or temporal splitting are used.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper introduces a novel, theoretically robust kernel design for Gaussian process regression tailored for molecular data types.\n2. The authors demonstrated the utility of the GAUCHE kernels in Bayesian optimization tasks which highlights their real-world applicability, particularly in the context of supporting chemists in candidate selection for synthesis.\n3. The authors ensure reproducibility by providing clear explanations and thorough empirical evaluations as well as a very well designed codebase.\n4. The paper uses a variety of datasets ranging from properties applicable to drug discovery and material discovery domain, which  showcases the adaptability of the GAUCHE kernels across different data.\n5. The authors evaluate the GAUCHE kernels across a wide array of molecular input types ranging from fingerprints, strings and graphs highlighting the versatility of the proposed framework.", "weaknesses": "This paper presents a strong methodological framework and robust empirical evaluation, but it does fall short in some key areas. \n\n1. The authors have shown that GAUCHE framework has been proven to be effective in a number of experiments, the comparison against SOTA neural network is notably lacking. For instance, chemprop, and ChemBERTa. \n2. The authors emphasizes GAUCHE's applicability for a low data regime, which is certainly a critical need in many scientific domains. However, in cases where large, high-quality datasets are available. For instance, internal databases at big pharma companies or public repositories like BindingDB. It would be great to see experiments around how the computational requirements, training time, and model performance would be impacted when scaling up the dataset size. \n3. It would be great if the authors could explore comparisons between RDKit/FragPrints and proprietary fingerprint techniques such as Dragon FP and Schrödinger, provided the licenses and costs are manageable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;\n\nThank you for taking the time to review our manuscript and for providing detailed, helpful and constructive feedback. We were happy to see you emphasizing the quality of our codebase and empirical evaluation, as well as the practical importance of our work.\n\nThe main suggestions you raised in your review relate to a characterization of the computational requirements as the dataset size increases, as well as a comparison to state-of-the-art deep learning algorithms and proprietary fingerprints. We sincerely appreciate the feedback and will try to address your points below.\n\n&nbsp;\n\n## __Computational Requirements for Larger Datasets__ ##\n\n&nbsp;\n\n> The authors emphasizes GAUCHE’s applicability for a low data regime, which is certainly a critical need in many scientific domains. However, in cases where large, high-quality datasets are available. For instance, internal databases at big pharma companies or public repositories like BindingDB. It would be great to see experiments around how the computational requirements, training time, and model performance would be impacted when scaling up the dataset size.\n\n> The GPR models can be computationally intensive, especially for larger datasets if used in conjunction with large dimensional fingerprints etc. This could limit the applicability in scenarios where abundant high-quality data is available.\n\n&nbsp;\n\nWhile black-box optimization tasks in low-data regimes are of substantial practical importance, we agree that it would be very interesting to investigate and extend GAUCHE to work well on larger datasets - for example, as you suggest, large public or private databases or even large DFT- or MD-derived datasets. As standard Gaussian Process inference scales cubically in the number of datapoints, it is likely too expensive to be a viable option in such settings. We have added a tutorial featuring the use of sparse GPs on the lipophilicity dataset (ca. 4200 molecules), which reduce this complexity by only performing inference over a subset of inducing points.\n\n&nbsp;\n\n## __Comparison to Proprietary Fingerprints__ ##\n\n&nbsp;\n\n> It would be great if the authors could explore comparisons between RDKit/FragPrints and proprietary fingerprint techniques such as Dragon FP and Schrödinger, provided the licenses and costs are manageable.\n\nWhile we have not explored the cost and licensing considerations of proprietary molecular fingerprints, we hope that the modularity of GAUCHE helps to facilitate such a comparison, as it would be straight-forward to apply any of our kernels to a dataset of molecules featurized as Dragon or Schrödinger fingerprints. At the reviewer's suggestion to expand the set of featurizations, we have now included Mordred descriptors which achieve state-of-the-art performance on the photoswitch benchmark. We include these results in the attached pdf.\n\n&nbsp;\n\n## __Comparison to SOTA Neural Networks__ ##\n\n&nbsp;\n\n> The authors have shown that GAUCHE framework has been proven to be effective in a number of experiments, the comparison against SOTA neural network is notably lacking. For instance, chemprop, and ChemBERTa.\n\n&nbsp;\n\nAs most SOTA deep learning frameworks (such as Chemprop and ChemBERTa) do not provide out-of-the-box support for BO/UQ it may be difficult to include them directly. However, the ChemBERTa embeddings could be added as an additional featurization. Additionally, we agree that it would be interesting to identify uncertainty-aware and BO-compatible deep learning frameworks for future benchmarking studies.\n\n&nbsp;\n\n## __Questions__ ##\n\n&nbsp;\n\n> How does the splitting strategy affect the results? For example, what if the data were split based on molecular scaffolds, which would ensure the structural diversity of the test set, instead of a random split?\n\n> In the context of the Uncertainty Quantification benchmark, how does the predicted variance of the model relate to OOD samples? Could you provide a plot of the Tanimoto distance to the training dataset versus the predicted variance?\n\n&nbsp;\n\nThis is an excellent point. While we have chosen random splits for our current experimental setup to mimic late-stage molecular optimization within a chemical series, we agree that characterizing and comparing the predictive accuracy and calibration of different models in an out-of-distribution regime is a useful additional consideration. We will try to set up the corresponding experiments and add them to the appendix of our manuscript. For the Tanimoto kernel, given that it is using the Tanimoto distance metric directly, we would expect there to be a direct correlation albeit it is unclear what we would expect for non-binary representations.\n\n&nbsp;\n\n> The authors touch upon the potential use of embeddings of molecules from pretrained models as inputs for the kernels, but there doesn’t seem to be any results for this approach. Is this something the authors are considering for future investigations?\n\n&nbsp;\n\nThere is a tutorial notebook, *pretrained_kernel.py* in the notebooks folder of the library that considers the case of pretrained embeddings. Seeing as this methodology is of interest, we can run more extensive experiments and include them as an additional point of comparison.\n\n&nbsp;\n\n> How would the performance of the kernels be impacted with the changes in the hand tuned fingerprint radius? Have any ablation studies been conducted to investigate this?\n\n&nbsp;\n\nWe have not carried out any kernel-specific ablations yet, but agree that this would be another interesting plot to add to our appendix, as it would illustrate how the performance of Gaussian Process models changes when they are trained on progressively more expressive molecular representations.\n\n&nbsp;\n\n## __Summary__ ##\n\n&nbsp;\n\nWe hope that this additional discussion addresses all of your points. Please let us know if you have any further questions!\n\n&nbsp;\n\nSincerely,\nThe Authors\n\n&nbsp;\n"}, {"review_id": "LQnpmkVOPK", "reviewer": "Reviewer_jdZg", "summary": "This paper presents a library for Gaussian processes on chemistry data. In this library, a number of kernels are implemented over chemical representations such as graphs, strings and bit vectors. Regression and Bayesian optimization experiments are shown using the library.\n", "questions": "Neurips is probably not the right venue for publishing software libraries.", "limitations": "The limitation of the proposed method has been explicitly discussed.\n", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- GP has been widely used as a ML tool for Chemistry.\n- A reliable library can lower the barrier for chemistry experts on using ML tools.\n", "weaknesses": "- All of the implemented kernels are known in the literature.\n- Neurips is probably not the right venue for publishing software libraries.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;\n\nThank you for taking the time to review our manuscript. We were happy to see you emphasize both the general usefulness of Gaussian Process models as robust molecular machine learning tools, as well as the practical impact that a well-designed and easy-to-use library can have by making them more accessible to scientific experts in chemistry and materials science.\n\nThe concerns you raised in your review relate to the suitability of NeurIPS as a venue for software library papers and the originality of the kernels we have implemented. We will address each of these points in turn below.\n\n&nbsp;\n\n## __Publishing Software Library Papers at NeurIPS__ ##\n\n&nbsp;\n\n> Neurips is probably not the right venue for publishing software libraries.\n\n&nbsp;\n\nOur interpretation of the NeurIPS 2023 Call For Papers, which explicitly calls for “libraries” in the Infrastructure section, as well as a call for manuscripts on “Machine learning for sciences”, leads us to believe that our contribution falls within the remit of the current iteration of NeurIPS, albeit we acknowledge that the aforementioned calls are a new addition in recent years. We refer to [1-3] as examples of software libraries that were recently published at NeurIPS.\n\n&nbsp;\n\n## __Originality of our Kernels__ ##\n\n&nbsp;\n\n> All of the implemented kernels are known in the literature.\n\n&nbsp;\n\nBuilding on the previous point, we would like to emphasize that our main contribution is the provision of a robust and easy-to-use library to make Gaussian Processes more easily accessible to expert chemists and materials scientists. While we strongly believe that the development of novel and more performant kernels is an important area of future research, we would like to point out that this is an orthogonal gap in the literature to the one we aim to address with this manuscript - though one that could strongly benefit from the robust foundation we aim to provide with GAUCHE.\n\n&nbsp;\n\nWe hope that these additional clarifications address all of your questions and concerns. Please let us know if you have any further questions!\n\n&nbsp;\n\nSincerely,\n\nThe Authors\n\n&nbsp;\n\n## __References__ ##\n\n&nbsp;\n\n[1] Jamasb, Arian, et al. “Graphein-a python library for geometric deep learning and network analysis on biomolecular structures and interaction networks.” Advances in Neural Information Processing Systems 35 (2022): 27153-27167.\n\n[2] Pineda, Luis, et al. “Theseus: A library for differentiable nonlinear optimization.” Advances in Neural Information Processing Systems 35 (2022): 3801-3818.\n\n[3] Feydy, Jean, et al. “Fast geometric learning with symbolic matrices.” Advances in Neural Information Processing Systems 33 (2020): 14448-14462.\n\n&nbsp;"}, {"review_id": "SvzSBF4Itc", "reviewer": "Reviewer_oVt5", "summary": "This article presents a library for Gaussian process-based inference with a special focus on chemistry applications. At heart, the library contains two classes of objects: kernel, and data loaders. The article introduces Gaussian processes and chemistry-specific kernels and discusses the interfacing of the library with other frameworks, in particular relating to Bayesian optimisation.", "questions": "As discussed in the weaknesses, my main reasons for the negative rating are\n(i) that the supported methods are not well documented within the article\n(ii) the future of the library is really unclear: for instance\n> We seek to further grow our userbase and solicit feedback from laboratory practitioners on the most common use-cases for BO and GP modelling in molecular discovery campaigns.\n\nis an admission of stale development and of an unclear vision for the future of the library.\nSome clarity on this would be much appreciated.\n\n\nA remark on my confidence level: I am by no means in the position to judge the relevancy of introducing a package for chemistry-specialised kernels and do not really understand the end applications. I will therefore fully defer to other reviewers when it comes to this.", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed library seems to fill a gap in the open-source GP stack, providing specialised building blocks for use in chemistry. The code cleanliness is rather high and the code is well-tested.\n\nThe paper is fairly clear (see however my questions in the weaknesses), and provides a reasonable introduction to the problem and to the GP literature around it, some illustration of the superior performance of GPs (compared to Grakel) is adequately shown. The relation to prior works and existing solutions is also well documented.", "weaknesses": "As it stands, there is no clear indication in the article that the library has had any impact on the practice of data-driven chemistry. If possible, the authors should include examples of real-life uses of GAUCHE, and if there is none, it then would mean that the library is likely not mature enough to warrant a full-size article (although it would likely be a good workshop paper).\n\nI have found it hard to understand which \"20+ bespoke\" kernels were in fact implemented. For instance, the graph kernels are described in the article, but the code provided shows nothing under \"gauche/kernels/graph_kernels\". Similarly, I cannot count 20 kernels in the code and would like for the authors to specifically list these.\n\nThe future of the library is also not mentioned: what is the governance model and what are the next steps? Is it to implement more kernels? When new kernels are implemented, then what? As it stands it seems that the authors suggest they will \"wait and see\", depending on the feedback they receive from the practitioners. This makes me believe that publishing an article on the library itself is then fairly premature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;\n\nThank you for taking the time to review our manuscript and for providing helpful and constructive feedback. We were happy to see you emphasize how our library complements the current open-source molecular machine learning stack and acknowledge the high quality of our code and tests.\n\nThe concerns you raised in your review relate to examples of real-life use-cases, our open-source governance model, and the claims we make regarding the number of available kernels. We will address each of these points in turn below.\n\n&nbsp;\n\n## __Examples of Real-World Use-Cases__ ##\n\n&nbsp;\n\n> If possible, the authors should include examples of real-life uses of GAUCHE\n\n&nbsp;\n\nThis is a great point and central to our motives for introducing GAUCHE. We are eager to share at least four instances (for which public references are available) in which GAUCHE has been utilized in real-world research and production settings. As stated in the general comment, we will liaise with the AC/SAC to determine the best way to provide these references!\n\n&nbsp;\n\n## __Plans for Future Development__ ##\n\n&nbsp;\n\n> the future of the library is really unclear\n\n&nbsp;\n\nPending the advice of the AC on the double-blind policy, if we are able to share some recent applications of GAUCHE this should give some flavour of the user-inspired extensions we have recently implemented! Additionally, in the attached pdf we have included a new SOTA performance on the photoswitch benchmark motivated by **reviewer rK4e** suggestion to investigate additional fingerprints/descriptors.\n\n&nbsp;\n\n## __Number of Available Kernels__ ##\n\n&nbsp;\n\n> I have found it hard to understand which \"20+ bespoke\" kernels were in fact implemented.\n\n&nbsp;\n\nFor the graph kernels alone, the total count subsumes the total number of kernels available in the GraKel library (18). In the \"**external_graph_kernels.ipynb**\" notebook we showcase the application of our wrapper around GraKel, the SIGP class which allows any kernel from GraKel to be used as a component of a PyTorch-based Gaussian process. Since GPU-enabled GP libraries do not natively support non-tensorial inputs, the SIGP class required a substantial amount of engineering work, but enables autodiff-based, end-to-end learning over the kernel hyperparameters of the GraKel library. Certain kernels such as the Weisfehler-Lehman (WL) kernel have concrete feature functions, and so a WL kernel-GP can be implemented by fitting a linear kernel to those features. However, given that many of GraKel's kernels do not possess distinctive feature functions, we decided to implement a more general wrapper framework. Additionally, we are in the process of integrating an additional suite of bit vector kernels in the coming week.\n\n&nbsp;\n\n## __Summary__ ##\n\n&nbsp;\n\nWe hope that our additional clarifications and discussion address all of your questions and concerns.\n\nPlease let us know if you have any further questions!\n\n&nbsp;\n\nSincerely,\n\nThe Authors\n\n&nbsp;"}, {"review_id": "aSb6jJeLwc", "reviewer": "Reviewer_rHxF", "summary": "The authors discuss Molecular, Reaction, and Protein Representations and provide a unified framework for these models. Python's GPyTorch library is used to train the Gaussian processes. The authors define certain kernels for Gaussian processes to fit and perform several experiments to evaluate the performance. The paper also contains a brief overview of  Gaussian processes and Bayesian optimization.", "questions": "What are the major limitations of the available packages in R/Python  so that they can not be used in the chemistry problems and how this package addresses them? If the kernels are omitted, what is the main advantage of the library compared to the others?", "limitations": "The authors addressed the limitations of the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors properly analyze the related kernel functions used in chemistry. Indeed, they provide an overview of the applications and representations available in the GAUCHE library. The library seems easy to follow and can adapt to proteins, molecules, and chemical reactions.", "weaknesses": "The major weakness is the contribution. The training task uses GPyTorch, a powerful library in Python for Gaussian processes. \nThe main contribution of the paper is to provide some important kernels in chemistry. It seems some of the kernels have been developed in other libraries or their scripts are available on the internet. The theory part of the paper is not rich enough. Section 3 only explains the relevant kernels and does not provide novel ideas or solutions for the methodological challenges. The Gaussian process methods and kernels are already there without new inventions, and the paper does not describe how to adapt them to chemistry problems. It would be great if the authors describe the major benefits of the proposed solution compared to the other libraries related to the Gaussian processes, except the kernels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;\n\nThank you for taking the time to review our manuscript and for providing valuable and helpful feedback. We were happy to see that you appreciated the thorough treatment of the representations, kernels and applications we cover in our work.\n\nThe main concern you raised in your review is the novelty of our contributions, which we aim to clarify below.\n\n&nbsp;\n\n## __Clarifying the Added Value of GAUCHE__ ##\n\n&nbsp;\n\n> It would be great if the authors describe the major benefits of the proposed solution compared to the other libraries related to the Gaussian processes, except the kernels.\n\n&nbsp;\n\nThe main contribution of our work is to create an open and unified community resource that makes it as easy as possible to combine current (and future) state-of-the-art molecular representations and kernels with existing Gaussian Process and Bayesian Optimization infrastructure. The central motivation behind this approach is to create a public repository that enables expert chemists and materials scientists with little background in GPs or BO to make use of state-of-the-art black-box optimization techniques, as well as to minimize the time and effort spent on re-implementing redundant optimization pipelines.\n\nWe refer to Section 5 of the manuscript for a thorough review of prior art that aims to convey the added value that GAUCHE provides over existing libraries. For enhanced clarity, we have summarised this comparison as a table in the general comment above as well as in the attached pdf.\n\n&nbsp;\n\n> The main contribution of the paper is to provide some important kernels in chemistry. It seems some of the kernels have been developed in other libraries or their scripts are available on the internet. (…) Section 3 only explains the relevant kernels and does not provide novel ideas or solutions for the methodological challenges.\n\n&nbsp;\n\nWhile it is correct that we mostly build on kernels that are well-established in the literature, we would like to point out that \n\n1. Discrete kernels are not straight-forwardly compatible with the design assumptions of GPyTorch/BOTorch (see below for more), and that a unified and open-source repository of compatible implementations is very helpful\n2. That many of these kernels have never been used in the context of GP regression\n3. That fewer still have been used for Bayesian optimization - especially in the context of chemistry, materials science and structural biology.\n\n&nbsp;\n\nWe strongly agree that the development of novel and more performant kernels and representations is an important area of future research, but would like to point out that this is an orthogonal gap in the literature to the one we aim to address with this manuscript - though one that could strongly benefit from the robust foundation we aim to provide with GAUCHE.\n\n&nbsp;\n\n> The Gaussian process methods and kernels are already there without new inventions, and the paper does not describe how to adapt them to chemistry problems.\n\n&nbsp;\n\nWe would like to contest the claim that we do not describe how to adapt our framework to chemistry problems, as we put significant effort into providing a range of easy-to-adapt Jupyter Notebook tutorials that demonstrate how to use GAUCHE for various real-world tasks in medicinal chemistry and reaction optimization. Going even further, we would like to point out that the library has already been used in a range of real-world production settings. We are unsure of how to link to these examples without violating the double-blindness of the review process and are currently liaising with the AC.\n\n&nbsp;\n\n## __Questions__ ##\n\n&nbsp;\n\n> What are the major limitations of the available packages in R/Python so that they can not be used in the chemistry problems and how this package addresses them?\n\n&nbsp;\n\nOne important and major limitation of existing GP frameworks is that they are built with continuous data in $\\mathbb{R}^d$ in mind. For instance, the kernel base class of GPyTorch assumes that custom kernel sub-classes are based on Euclidean distance metrics. In GAUCHE, we provide a parallelizable and batch-GP-compatible alternative to this base class that can be easily extended to implement arbitrary bit and count vector kernels. As another example, we would like to point out that the associated GP optimization utilities only work for tensor-valued input spaces and required substantial adjustment to work on graph-structured inputs.\n\n&nbsp;\n\n## __Summary__ ##\n\n&nbsp;\n\nWe hope that these points clarify the objective of the paper and address the presentational concerns you raised. Please let us know if you have any further questions!\n\n&nbsp;\n\nSincerely,\n\nThe Authors\n\n&nbsp;"}, {"review_id": "DfYwE3gJxW", "reviewer": "Reviewer_JWPZ", "summary": "Gaussian processes are widely used for black-box optimization when data is scarce. On the other hand, effectively representing molecules, proteins, and chemical reactions is a dedicated research area in molecular machine learning.\n\nAlthough separate tools exist to address these two challenges, this paper introduces a novel library that integrates both. By doing so, it enables chemists without extensive knowledge of Bayesian optimization to harness its benefits.", "questions": "* `l. 60`, the covariance $\\sigma_y$ is not introduced. I understand that this assumes a Bayesian linear model of the form $f({\\bf x}) = \\phi({\\bf x})^\\top {\\bf w} + \\sigma_y \\epsilon$ where $\\epsilon$ follows a standard normal distribution and $K({\\bf x}, {\\bf x}') = \\phi({\\bf x})^\\top \\phi ({\\bf x}')$, as in [1, Eq. 2.21]. I am not familiar with the BO literature, and perhaps it's the only model considered, but it would be worth mentioning that you are considering a Bayesian linear model, along with a proper citation to justify the formula. \n* `l. 68`: again, adding a reference e.g. [1, Eq. 5.8] seems necessary to justify the NLML formula. \n* Fig. 1 is not informative for someone who does not have GP or chemistry background. After reading sec. 2.{3, 4, 5}, we understand that the 3rd row show the possible representation for the different applications considered in the 2nd row (even though \"SMARTS\" in not described in sec. 2. 4). In any case, the figure would benefit from a more detailed caption, so that the message it is trying to convey appears clearly. \n* Table 1 and 2: why the font is so small? It would be convenient to have the same font size as the rest of the document.\n* `l. 227`, I don't know the 3 metrics for measuring the quality of the uncertainty estimates. A word of introduction explaining what they measure, what are their strength and limitation would be welcomed here. \n\n**Related literature.**\n\n* You may be interested in [2] which provides an convolutional kernel network for graph-structured data. \n\n**Typos.**\n\n* `l. 55`: $m(\\mathbf{x'})$ instead of $m(\\mathbf{x})$ \n* `l. 80`: extra \"the\"\n\n**References.**\n\n[1] Rasmussen and Williams - 2006 - Gaussian processes for machine learning\n\n[2] Convolutional Kernel Networks for Graph-Structured Data – 2020 – Dexiong Chen, Laurent Jacob, Julien Mairal", "limitations": "The limitations identified by the authors regarding specific algorithms for certain problems are noteworthy but can be left for future research. Aside from that, my main concerns are summarized in the \"Weaknesses\" section mentioned previously.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The library's objective is clearly defined and holds potential for the chemistry and machine learning research communities. The persuasive comparison with existing work in Section 5 strengthens its position. Sections 2.3 and 2.4 provide thorough explanations of molecular and chemistry reactions.", "weaknesses": "I find the objective of the paper somewhat unclear. Here is my understanding and suggested improvements:\n\n* The paper's main aim, as summarized in Section 5, is to create a unified library called GAUCHE that combines chemistry libraries (for molecule/chemical reaction representation) with Bayesian optimization libraries. Such libraries exist separately, GAUCHE aims to unify them.\n* Sections 2 and 3 effectively describe the library's provided representations and kernels.\n* However, I fail to grasp the purpose of the experiments described in Section 4. The message seems to be that \"the library works and enables Bayesian optimization on various chemistry benchmarks.\" This leaves room for improvement: what sets GAUCHE apart? Why should users choose it over manually combining a molecular representation with a GP library? Section 5 partially answers these questions – and should be presented earlier in the paper, in my opinion.\n* Overall, it appears that the library does not offer new theoretical insights or algorithms. While this is not necessarily a problem (as GAUCHE fills a relevant niche, allowing non-GP-expert chemists to utilize state-of-the-art black-box optimization for chemical reactions), **the paper should clearly emphasize the library's strengths**. These may include (a) ease of use, (b) a modular code structure enabling user extension, or (c) superior performance on public benchmarks. In that spirit, a helpful addition would be a table comparing different libraries, with the different libraries (containing GAUCHE) in rows and the different features (GP, molecular representation, etc) in columns, and cells indicating whether the feature is implemented or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;\n\nThank you for taking the time to review our manuscript and for providing detailed, helpful and constructive feedback. We were happy to see that you appreciate the practical usefulness of a well-designed and easy-to-use library that enables scientific experts to make use of Bayesian optimisation in low-data regimes.\n\nThe main concern you raised in your review relates to the way we present the objectives and strengths of our work and we sincerely appreciate the feedback. We will try to address your points below.\n\n&nbsp;\n\n## __Clarifying the Added Value of GAUCHE__ ##\n\n&nbsp;\n\n> The paper’s main aim, as summarized in Section 5, is to create a unified library called GAUCHE that combines chemistry libraries (for molecule/chemical reaction representation) with Bayesian optimization libraries. Such libraries exist separately, GAUCHE aims to unify them.\n\n&nbsp;\n\nAs you correctly stated, our main aim is to create an open and unified community resource that makes it as easy as possible to combine current (and future) state-of-the-art molecular representations and kernels with Gaussian Process and Bayesian Optimization libraries. The central motivation behind this approach is to create a public repository that enables expert chemists and materials scientists with little background in GPs or BO to make use of state-of-the-art black-box optimization techniques, as well as to minimize the time and effort spent on re-implementing redundant optimization pipelines.\n\n&nbsp;\n\n> However, I fail to grasp the purpose of the experiments described in Section 4. (…) What sets GAUCHE apart? Why should users choose it over manually combining a molecular representation with a GP library?\n\n&nbsp;\n\nThe purpose of the experiments in Section 4 (and the corresponding Jupyter Notebook tutorials) is to provide prospective users with a convincing demonstration that GAUCHE presents a modular and user-friendly platform for the rapid exploration and prototyping of different molecular representations and similarity kernels to establish which setup (if any) works best for a given application.\n\n&nbsp;\n\n> (…) the paper should clearly emphasize the library’s strengths. These may include (a) ease of use, (b) a modular code structure enabling user extension, or (c) superior performance on public benchmarks.\n\n&nbsp;\n\nWe will make sure to refine these points in Section 5 and to also clearly state them in both the Abstract and Introduction of the paper.\n\n&nbsp;\n\n> In that spirit, a helpful addition would be a table comparing different libraries (…)\n\n&nbsp;\n\nThank you for the suggestion. We agree that clearly stating the value that GAUCHE adds over existing libraries by summarizing Section 5 into an intuitive table would help to further clarify the points we make above. We have included a draft of this table below and a more polished version in the pdf submitted with our general response.\n\n&nbsp;\n\n| Library  | Gaussian Processes | Bayesian Optimisation | Molecular Representations | Chemistry Tutorials | Graph Kernels | Bit Vector Kernels | String Kernels |\n|----------|--------------------|-----------------------|---------------------------|---------------------|---------------|--------------------|----------------|\n| GPyTorch | ✓                  | ✗                     | ✗                         | ✗                   | ✗             | ✗                  | ✗              |\n| GPflow   | ✓                  | ✗                     | ✗                         | ✗                   | ✗             | ✗                  | ✗              |\n| BoTorch  | ✓                  | ✓                     | ✗                         | ✗                   | ✗             | ✗                  | ✗              |\n| DeepChem | ✗                  | ✗                     | ✓                         | ✓                   | ✗             | ✗                  | ✗              |\n| GraKel   | ✗                  | ✗                     | ✗                         | ✗                   | ✓             | ✗                  | ✗              |\n| FlowMO   | ✓                  | ✗                     | ✓                         | ✓                   | ✗             | ✓                  | ✓              |\n| GAUCHE   | ✓                  | ✓                     | ✓                         | ✓                   | ✓             | ✓                  | ✓              |\n\n\n&nbsp;\n\n## __Questions__ ##\n\n&nbsp;\n\n1. Great spot on the $\\sigma_y$ term. This indeed represents a coefficient for additive Gaussian noise and we will update the manuscript to include this definition.\n2. Many thanks for the reference! We have opened a GitHub issue to introduce convolutional kernel networks.\n\n&nbsp;\n\n## __Summary__ ##\n\n&nbsp;\n\nWe hope that these points clarify the objective of the paper and address the presentational concerns you raised.\n\nPlease let us know if you have any further questions!\n\n&nbsp;\n\nSincerely,\n\nThe Authors\n\n&nbsp;"}]

Author rebuttal:
&nbsp;

## __Overview__ ##

&nbsp;

We would like to thank all reviewers for the time and effort put into reviewing our manuscript and for the valuable and constructive feedback they have provided.

We are delighted that all reviewers recognized the practical significance of our work, highlighting that GAUCHE “fills a gap in the open-source GP stack” (**reviewer oVt5**) and provides a “strong theoretical framework with rigorous empirical evaluation” (**reviewer rK4e**) that “allows non-GP-expert chemists to utilize state-of-the-art black-box optimization tools” (**reviewers JWPZ and jdZg**) and is able to “adapt to proteins, molecules, and chemical reactions” (**reviewer rHxF**).

We are also happy to see that reviewers appreciated our scope, noting that we “provide thorough explanations of molecular and chemical reaction” tasks (**reviewer JWPZ**), “thoroughly examine different ways of representing molecular structures” (**reviewer rK4e**) and “properly analyze the related kernel functions” (**reviewer rHxF**), which we evaluate “across several diverse benchmarks involving regression tasks, uncertainty quantification, and Bayesian optimization (BO)” (**reviewer rK4e**).

Finally, we were pleased to hear that reviewers appreciated our engineering effort, noting that we provide “a very well designed codebase” (**reviewer rK4e**) that “seems easy to follow and adapt” (**reviewer rHxF**), further emphasizing that “the code cleanliness is rather high and the code is well-tested” (**reviewer oVt5**) and that we “ensure reproducibility by providing clear explanations” and tutorials (**reviewer rK4e**).

&nbsp;

## __Summary of Concerns__ ##

&nbsp;

The main concerns raised by reviewers related to:

&nbsp;

1. The suitability of NeurIPS as a venue for publishing software libraries (**reviewer jdZg**),
2. The need for a clearer emphasis of the library’s strengths (**reviewers JWPZ and rHxF**) and
3. Examples of real-world case studies that use GAUCHE (**reviewer oVt5**).

&nbsp;

**(1)** We have addressed the first point by referring to the NeurIPS Call For Papers - which explicitly invites contributions on “libraries” (and “machine learning for sciences”) - as well as a range of software libraries that were recently published at NeurIPS [1-3].

**(2)** In response to the feedback we received regarding the second point, we will further refine our abstract, introduction and discussion of prior art to more clearly highlight the strengths and added value that GAUCHE provides (namely ease-of-use, modularity and robust empirical performance). As requested by reviewers, we have also summarized the advantages our library provides over existing packages in the table below, which we will add to Section 5.

&nbsp;

| Library  | Gaussian Processes | Bayesian Optimisation | Molecular Representations | Chemistry Tutorials | Graph Kernels | Bit Vector Kernels | String Kernels |
|----------|--------------------|-----------------------|---------------------------|---------------------|---------------|--------------------|----------------|
| GPyTorch | ✓                  | ✗                     | ✗                         | ✗                   | ✗             | ✗                  | ✗              |
| GPflow   | ✓                  | ✗                     | ✗                         | ✗                   | ✗             | ✗                  | ✗              |
| BoTorch  | ✓                  | ✓                     | ✗                         | ✗                   | ✗             | ✗                  | ✗              |
| DeepChem | ✗                  | ✗                     | ✓                         | ✓                   | ✗             | ✗                  | ✗              |
| GraKel   | ✗                  | ✗                     | ✗                         | ✗                   | ✓             | ✗                  | ✗              |
| FlowMO   | ✓                  | ✗                     | ✓                         | ✓                   | ✗             | ✓                  | ✓              |
| GAUCHE   | ✓                  | ✓                     | ✓                         | ✓                   | ✓             | ✓                  | ✓              |

&nbsp;

**(3)** In response to the request for real-world case studies, we are eager to share at least four instances (for which public references are available) in which GAUCHE has been used by other researchers in real-world research and production settings. As the review guidelines state that we should not include any links to external pages, we will liaise with the AC/SAC to determine the best way to provide these references.

We have added additional discussions and clarifications of these and all other points raised by the reviewers under the respective reviews. Additionally, our attached rebuttal document includes a more polished version of the markdown table above as well as new results attaining a new **SOTA on the photoswitch benchmark** following the suggestions of **reviewer rK4e**.

We hope that our response addresses all reviewer questions and concerns and are happy to answer any further questions!

&nbsp;

Sincerely,

The Authors

&nbsp;

## __References__ ##

&nbsp;

[1] Jamasb, Arian, et al. “Graphein-a python library for geometric deep learning and network analysis on biomolecular structures and interaction networks.” Advances in Neural Information Processing Systems 35 (2022): 27153-27167.

[2] Pineda, Luis, et al. “Theseus: A library for differentiable nonlinear optimization.” Advances in Neural Information Processing Systems 35 (2022): 3801-3818.

[3] Feydy, Jean, et al. “Fast geometric learning with symbolic matrices.” Advances in Neural Information Processing Systems 33 (2020): 14448-14462.

&nbsp;

Meta-review: The authors provide a new library for GAUssian processes in CHEmistry. 

The paper is well-written and easy to follow. 

As indicated by their scores, the reviewers really liked the paper and the reviewers noted that GAUCHE “fills a gap in the open-source GP stack”, it “allows non-GP-expert chemists to utilize state-of-the-art black-box optimization tools”, and it is “very well designed codebase”.

The only negative concern from one of the reviewers was that NeurIPS might not be the best fit for this paper since it is not a traditional NeurIPS paper in the sense that it is rare to see software libraries published at NeurIPS.

The authors responded with a citation from the NeurIPS call from papers where it is listed that NeurIPS welcomes software libraries and they also listed some recent examples where NeurIPS published software libraries:

[1] Jamasb, Arian, et al. “Graphein-a python library for geometric deep learning and network analysis on biomolecular structures and interaction networks.” Advances in Neural Information Processing Systems 35 (2022): 27153-27167.

[2] Pineda, Luis, et al. “Theseus: A library for differentiable nonlinear optimization.” Advances in Neural Information Processing Systems 35 (2022): 3801-3818.

[3] Feydy, Jean, et al. “Fast geometric learning with symbolic matrices.” Advances in Neural Information Processing Systems 33 (2020): 14448-14462.

The GAUCHE software package indeed might be a great tool for working on certain chemistry problems with machine learning and the authors also provided a list of papers from various authors where this library has been already used.
Decision: Accept (poster)