How Much Space Has Been Explored? Measuring the Chemical Space Covered by Databases and Machine-Generated Molecules
Authors: Yutong Xie
Ziqiao Xu
Jiaqi Ma
Qiaozhu Mei
Venue: ICLR 2023 poster (2023)
Keywords: molecular generation
drug discovery
coverea measures
chemical space measures

Abstract: Forming a molecular candidate set that contains a wide range of potentially effective compounds is crucial to the success of drug discovery. While most databases and machine-learning-based generation models aim to optimize particular chemical properties, there is limited literature on how to properly measure the coverage of the chemical space by those candidates included or generated. This problem is challenging due to the lack of formal criteria to select good measures of the chemical space. In this paper, we propose a novel evaluation framework for measures of the chemical space based on two analyses: an axiomatic analysis with three intuitive axioms that a good measure should obey, and an empirical analysis on the correlation between a measure and a proxy gold standard. Using this framework, we are able to identify #Circles, a new measure of chemical space coverage, which is superior to existing measures both analytically and empirically. We further evaluate how well the existing databases and generation models cover the chemical space in terms of #Circles. The results suggest that many generation models fail to explore a larger space over existing databases, which leads to new opportunities for improving generation models by encouraging exploration. 


Published as a conference paper at ICLR 2023
How MUCH SPACE HAS BEEN EXPLORED? MEASURING THE CHEMICAL SPACE COVERED BY DATABASES AND MACHINE-GENERATED MOLECULES
Yutong Xie', Ziqiao Xu", Jiaqi Ma², Qiaozhu Mei
School of Information, University of Michigan, Ann Arbor, Michigan, USA 2Chemistry Department, University of Michigan, Ann Arbor, Michigan, USA
"School of Information Sciences, University of Illinois Urbana-Champaign. Champaign, Illinois, USA
ABSTRACT
Forming a molecular candidate set that contains a wide range of potentially effective compounds is crucial the success of drug discovery. While most databases and machine-learinng-aase
generation models aim to optimize particular chemical properties, there is limited literature on how properly measure the coverage of the chemical space by those candidates included or generated. This problem is challenging due the lack of formal criteria select good measures of the chemical space. In this paper, we propose a novel evaluation framework for measures of the chemical space based on two analyses: an axiomatic analysis with three intuitive axioms that good measure should obey, and an empirical analysis on the correlation between a measure and proxy gold standard. Using this framework, we are able to identify #Circles, a new measure of chemical space coverage, which is superior to existing measures both analytically and empirically. We further evaluate how well the existing databases and generation models cover the chemical space in terms of #Cireles. The results suggest that many generation models fail to explore a larger space over existing databases, which leads new opportunities for improving generation models by encouraging exploration.
INTRODUCTION
To efficiently navigate through the huge chemical space for drug discovery, machine learning (MI.) based approaches have been broadly designed and deployed, especially de novo molecular generation methods (Elton et al. 2019, Schwalbe-Koda & Gómez- Bombareli 2020 [Bian & Xie 2021 Deng al. 2022). Such generation models learn to generate candidate drug designs by optimizing various molecular property scores, such as the binding affinity scores. In practice, these scores can be computationally obtained using biological activity prediction models (Olivecrona et al. 2017 Li jet al. 2018), which is the key to obtaining massive labeled training data for machine learning. However, high in silico property scores are far from sufficient, as there is usually a considerable misalignment between these scores and the in vivo behaviors. Costly wet-lab experiments are still needed verify potential drug hits, where only a limited number of drug candidates can be tested.
In light of this cost constraint, it is critical to select or generate drug candidates not only with high in silico scores, but also covering a large portion of the chemical space. As functional difference between molecules is closely related to their structural difference (Huggins et al., 201T; Wawer al. 2014). a better coverage of the chemical space will likely lead to a higher chance of hits in wet experiments. For this purpose, quantitative coverage measures of the chemical space become crucial. Such measures can both be used to evaluate and compare the candidate librarie and be incorporated into training objectives to encourage MI. models better explore the chemical space.
In this paper, we investigate the problem of quantitatively measuring the coverage of the chemical space by a candidate library. There have been few such coverage measures of chemical space. For example, richness counts the number of unique compounds in a molecular set, and it has been used describe how well a model is able to generate unique structures (Shi & von Itzstein 2019
"Correspondence to: Qiaozhu Mci <qmei wwwch.cdulhhaa A candidate library a collection of drug candidates filtered by rules or generated by MI. models.

---

Published as a conference paper at ICLR 2023
Polykovskiy et al. 2020). In addition, molecular fingerprints have been used to calculate pairwise similarity or distance between two compounds, and the average of these pairwise distances has been used to describe the overall internal diversity of a molecular set (Brown et al. 2019 Polykovskiy
et al. 2020).
However, most existing coverage measures are heuristically proposed and the validity of these measures is rarely justified. In fact, defining the "right" measure for the coverage of chemical space coverage is challenging. Unlike the molecular property scores, there is no obvious "ground truth" about the coverage of chemical space. Moreover, the chemical space is complex and combinatorial, making the design of a good measure even more difficult.
To address the fundamental problem of properly measuring the coverage of chemical space for drug discovery, we propose novel evaluation framework with two complementary criteria for evaluating the validity of coverage measures. We first formally define the concept of coverage measures on the chemical space (referred to as chemical space measures), where many existing heuristic measures fall into our definition (Section 3). Then we introduce the two criteria and compare various (existing and new) chemical space measures based on the criteria (Section 4 Specifically, the first criterion (Section 4.1 is based on axiomatic analysis with three intuitive axioms that good chemical space measure should satisfy. Surprisingly. most heuristic measures that are commonly used in literature, such as internal diversity, fail satisfy these intuitive axioms. The second criterion (Section 4.2) compares the chemical space measures with a proxy of the gold standard: the number of unique biological functionalities covered by the set of molecules. We find that #Circles, a new chemical space coverage measure (defined in Section 3.2.3) that has a strong basis in the mathematical literature, not only satisfies both axioms but also better correlates with the gold standard.
Finally, we apply the #Circles measure to evaluate how well the existing databases and ML models cover the chemical space (Section 5). Interestingly, the evaluation results suggest that many ML models fail to explore a larger portion of chemical space compared to drug candidates obtained from virtual screening over existing databases. We believe these findings lead to new direction to improve ML-based drug candidate generation models on better exploring the chemical space.
RELATED WORK
Molecular databases and sechhinn-genrratee compounds are rich sources of drug candidates for forming a candidate library in drug discovery. To evaluate the quality of molecular databases and molecular generation methods, variety of metrics proposed. In general, four categories of evaluation metrics can be identified in the literature, which are related to: (1) bioactivities, (2) molecular properties, (3) data likelihood, or (4) the coverage of the chemical space, respectively.
In this paper, we mainly focus on the fourth category of metrics, the metrics that are more or less related to the degree of coverage (or exploration) in the chemical space (other metrics are discussed in Appendix A In this category, commonly used measures include richness. uniqueness, internal diversity, external diversity, KL divergence, and Fréchet ChemNet Distance (FCD) (Olivecrona et al.
2017. You et al. 2018 De Cao & Kipi| 2018 Elton et al., 2019
Brown et al.) 2019 Popova et al. 2019 Polykovskiy et al., 2020; (Shi et al. |020 Jin al. 2020 Xie et al. (021). Besides, Zhang
et al. 2021) propose to use the number of unique functional groups or ring systems to estimate the
chemical space coverage and to compare several recent generative models. Similarly in Blaschke
et al. (2020), the number of unique Bemis-Murcko scaffolds is used to measure the variety of drug candidates. Koutsoukas et al. (2014) study the effect of molecular fingerprinting schemes on the internal diversity of compound selection. These measures usually mix the concepts of diversity, coverage, or novelty, and their validity as a measure of exploration is not not justified.
To the best of our knowledge, this is the first work that formally investigates the validity of molecular chemical space measures. In particular, axiomatic approaches are used to analytically evaluate various designs of a measurement, such as utility functions (Herstein & Milnor [155)) cohesiveness (Alcalde-Unzu & Vorsatz 2013), or document relevance (Fang et al. 2004). While one study applies axiomatic analysis to the design of diversity measures, with a particular focus on the domain of science of science Yan 2021), the analysis of chemical space measurements remains novel. Using axiomatic analysis to evaluate the chemical space measures in the chemical space novel. With an empirical analysis in addition to the axiomatic analysis, we make practical recommendations on effective chemical space measures, including two novel measures.
2

---

Published as a conference paper at ICLR 2023
3 DEFINING CHEMICAL SPACE MEASURES
3.1 DEFINITION OF CHEMICAL SPACE MEASURES
To define a chemical space measure, first formalize the notion of chemical space by assuming distance metric exists. This assumption is widely adopted in cheminformatics, using distance metrics such the Tanimoto distance (Tanimoto 1968 Bajusz al. 2015).
Assumption 3.1 (Chemical space). The chemical space 24 contains all possible molecules and is metric space with a distance metric function d U - [0, +.0).
Definition 3.2 (Tanimoto distance). For molecules #1, 22 U. whose binary molecular finger- print vectors are 11, 22 E {0, 1 where n is the dimensionality of the fingerprint, their Tanimoto distance is defined
Eラ=1 T1j 22
x2), = 1 Σj=1 max (11j F22
The Tanimoto distance is also referred to as the Jaceard distance (Jaccard, 1912) other domains. Its range is [0, 1]. For finite sets (e.g., molecular fingerprints). the Tanimoto distance is metric Lipkus (999).
function (Kosub] 2019
Our definition and following analysis are generic to all distance functions and are not limited to the Tanimoto distance. One can also use the distance of latent hidden vectors (Preuer et al. 2018 Samanta et al., (2200) the root-mean-squees -uuurr deviation (RMSD) of three-dimensional molecular conformers (Fukutani et al. 2021).
We then define chemical space measure the following.
Definition 3.3 (chemical space measure). Given the universal chemical space U. a chemical space measure is function that maps a of molecules to a non-negative real number that reflects to what extent the set spans the chemical space, i.e., P(U) + [0, oo), where P(-) is the notation of power set. In particular, p0))
3.2 EXAMPLES OF CHEMICAL SPACE MEASURES
intentionally left general. Many measures related to the coverage, diversity, or novelty of molecular set fall into this definition. We summarize these various measures used literature into three categories: reference-based measures, aggregation sggegattion-base based measures, and localiy-bbase measures. We are also able define new chemical space measures under this formulation.
Definition 3.3
3.2.1 REFERENCE-BASEE BASED MEASURES
The first broad category of chemical space measures compare the generated molecules S with a reference set R. In such context, reference-based measure be defined as the coverage of references:
Coverage(S, R) == Σ
max cover(ı, #ES
VER
where cover(æ, y) indicates how molecule I can cover the reference y.
When the reference set R is taken as collection of molecular fragments, the coverage function can be written as nover(x, y) == Tmoleeule x contains fragment y), where T[-] is the indication function. A large body of drug discovery literature
the number of distinct functional groups (FG), ring
systems (RS), or Bemis-Murcko scaffolds (BM) in S to gauge the size of explored chemical space (Zhang al. 2021 Blaschke al. 2020), corresponding to the cases where R. is the collection of all measures as #FG, #RS, and #BM respectively. Another example is the Richness of the molecular set, i.e.. Richness := [
possible FG, RS, or BM fragments. We denote these specific reference-
(Sh & von Itzstein 2019 Polykovskiy et al. 2020), where cover(ar, ar, y) == T[ v) and R = 44
3.2.2
AGGREGATION--ASEE MEASURES
Though reference-based measures estimate chemical space coverage in an intuitive way, such measures highly rely on the characteristics of the reference set and do not consider the relations between the compounds. In contrast, the aggregation-based measures utilize the pair-wise distances
3

---

Published as a conference paper at ICLR 2023
among molecules and reflect the extent of chemical space coverage by aggregating the distances. For a S with n molecules, several aggregation based measures can be defined as follow:
2 Diversity(S:d) n(n 1)
d(c,y) v): (3a)
SunDiversity(S:d)
d(a, y), (3d)
3.yES spy
17
Diameter(S;d)) == max d(x,
(3b)
SunDiameter(S;c) = Σ d(z, v),
(3e)
288
(St == min d(z, v). 2,vES any
(3c) Su
min d(x, v), yES xES uyka
(31)
where z, y molecules in S, d is distance metric as defined previously.
Among these measures, Diversity (also referred to as the internal diversity), the average distance
between the molecules, is widely used in the literature YYou al. 2018 De Cao & Kipf 2018 Popova
et al. 2019 Polykovskiy et al. [2200] Shi et al. 2020: Xie et al.] 2021). We follow the topology theory and also introduce Diameter and Bottleneck the maximum and minimum distance between any pair of molecules, respectively (Edelsbrunner & Harer 2010), since they can also reflect the dissimilarity between molecules in S. We further introduce three Sum- variants for the above measures. The Sum- variants will tend to increase when new molecules are added into the set. In addition, the determinant of the similarity matrix of molecules also measure of dissimilarity, which is often employed in diverse subset selection as key concept of the determinantal point processes (DPP) (Kulesza & Taskar 2011] Kulesza et 2012). The DPP measures is defined as DPP(S) == dSt(S). where S is the Tanimoto similarity matrix of candidate molecules (i.e., 1 - d(æ, y) for Tanimoto distance).
3.2.3 LOCALITY-BAEED MEASURES
Inspired by the sphere exclusion algorithm used in compound selection (Snarey et al. 1997 Gobbi Lee 2003). we introduce new chemical space coverage measure that highlights the local neighborhoods covered by a set of molecules:
#Circles(S; d, t) := max
|C|
s.t. d(r,y) t, Vx VyC
(4)
CgS
where E (0,1) is distance threshold that corresponds to the diameter of circle.
Intuitively, #Circles counts the maximum number of mutually exclusive circles that can fit into S as neighborhoods, with a subset of its members C as the circle centers. When the threshold = 0, #Circles becomes the richness measure Richness == (S]. which is the number of unique molecules (Shi & von Itzstein 2019 PPolkkoskiyy et al. 2020). Interestingly, #Circles has close relation to the concepts of covering and packing in mathematics, which is in particular related to the definition of packing number in topology (Vershynin) 2018). We discuss their detailed connections in
Appendix B
SELECTING THE RIGHT MEASURE
While all the aforementioned measures can heuristically reflect the degree of exploration, they do not always agree with each other. We need a principled way to select the most suitable measures from variety of possible choices. Ideally, the selection criteria should not depend on the particular molecule properties of the target or the algorithm used to generate the candidates. We achieve this through a qualitative axiomatic analysis and quantitative correlation comparison.
4.1 CRITERION #1: AN AXIOMATIC ANALYSIS CHEMICAL SPACE MEASURES
We propose three simple and intuitive principles that good chemical space measure should satisfy. First, including more molecules should not decrease the degree of chemical space coverage. As it is easy filter large candidate libraries into smaller subsets, including more molecules (even similar ones) is not harmful, and can render a higher probability of containing drug hits. Second, when adding new molecules, the coverage should increase properly, instead of inflating too much. Third, molecular sets with more dissimilar molecules should have higher degree of chemical space coverage. These three principles are formalized below as three axioms and can be tested analytically.
"The SumBottlencek measure is sometimes abhreviated as "SumBot" in the main paper.
4

---

Published as a conference paper at ICLR 2023
Axiom 4.1 (Monotonicity). A good chemical space measure μ. should be monotonic, i.e., for any two molecular sets S1, S2 C U1 it holds that
r(S US max(u(S)).
A monotonic measure tends to increase when more molecules are included. This tendency is intuitive drug discovery as testing more molecules means having a higher probability to discover a drug hit.
Axiom 4.2 (Subadditivitt). A good chemical space measure should be subadditive, i.e., for any two molecular sets St, S2 S U. it holds that
p(S, U.) S ((S)) s(S2).
(6)
Note that the coverage of chemical space spanned by molecular set may correlate with the probability of containing drug hits. So correspondingly, the coverage (uS1 U S2) should not exceed the sum of (((s) and j((2)) Some direct corollaries and discussions on how the first and second axioms connect to mathematical measures are in Appendix
Axiom 4.3 (Dissimilarity)) good chemical space measure should have a preference dissimilar elements, ie., for any three molecules 20, 31, D2 E 14, if d(zo, x1) 2 d(zo, T2), it holds
M({zo, x1)) 2 p({zo, #2))
where d is the distance metric.
Intuitively, considering adding new molecule into an existing molecular
Monotonicity
Subadditivity
Reference bused E.g., JFG, aRS,
set S (ma), where there two choices, namely 2 and Z2, the more dissimilar one 2 will be preferred.
CCircles
SumDiversity
Bottloock
Depending on whether each chemical space measure satisfies monolocinicy, subadditivity and/or dissimilarity. we can put it into a Venn diagram as shown in Figure 1 The formal proofs are provided in Appendix(D) [D] Despite that the three axioms are simple and intuitive, surprisingly, the proposed #Circles is the only measure that sat- isfies all these three axioms.
Diameter
Sumbotileneck
Suml Siameere
Diversity
Dissimilurity
Figure 1: Chemical space measures, categorized by whether
they satisfy monotonicity, subadditivity, and/or dissimilarity. #Circles is the only measure that satisfies all simultaneously.
4.2 CRITERION #2: CORRELATION WITH BIOLOGICAL FUNCTIONALITY
While only measure stands out in the axiomatic analysis, it does not guarantee its empirical performance. Meanwhile, there are still multiple measures that satisfy one or two of the three axioms.
In this subsection, further investigate the validity of the chemical space measures by testing their correlations with the biological functionality variety of molccules. Such functionalities can provide valuable information in distinguishing molecules and implicating chemical space coverage. We correlate the chemical space measures to the number of unique biological functionality labels covered by molecular set. A better chemical space measure should have a higher correlation the variety of biological functionalities, even though it is proxy gold standard.
4.2.1 EXPERIMENT SETUP
We base the analysis on the BioActivity dataset that also used to compare different compound selection algorithms (Koutsoukas et al. 2014). This dataset contains 10,000 compound samples extracted from the ChEMBL database (Gaulton et al. 2017) with bio-activity labels, which annotate 50 activity classes with 200 samples each. Following Koutsoukas et al. (2014), for subset of this dataset S, we take the number of unique class labels as a proxy "gold standard" the variety of the molecules in S, i.e., GS(S): #unique labels in S, which represents the number of biological functionality types covered by S. We then compare the behavior of the gold standard and the chemical space measures in two settings find out which measures have the highest correlations with the coverage (or variety) of biological functionalities: (1) a fixed-size setting where random subsets of the dataset with fixed sizes are measured, and (2) growing-size setting where we sequentially
5

---

Published as a conference paper at ICLR 2023
Diversity
Diversity
Sumbiversity
SumDiversily
SumDiameter
Diameter
SumDiameter
Diameter
Bottlenack
SumBottlenek
SumBottleneck
DPP
DPP
4FG
AR5
#BM
&irrles
Richness
Richness
-0.2
0.0
0.2
0.4
0.6
0.8
50
Correlation
DTW distance
(a) Spearman's correlations between the gold standard GS and chemical space measures in the fixed-size setting. A higher correlation is better.
(b) DTW distances between the gold standard GS and chemical space measures in the growing-siz size setting, the smaller the better. A smaller distance is better.
Correlation between each chemical space measure and biological functionality coverage. The chemical space measure with the highest correlation highlighted in green. Results are aggregated by running experiments independently ten times.
Figure 2:
add molecules into a molecular set. We compare the measuring results of GS and chemical space measures with Spearman's correlation coefficient and dynamic time warping (DTW) distance. The experiment details are listed in Appendix
4.2.2 RESULTS AND DISCUSSION
Table Regarding the two suggested criteria, #Circles is the most recommended chemical space measure. "Mono", "subadd", and "dissim" are abbreviations for monotonicity, subadditivity, and dissimilarity respectively.
Experiment results in Figure 2 suggest that #Circles stands out for both empir- ical experiment settings. For the fixed- size setting, #Circles and SumBottleneck have notably higher correlations to the gold standard than other measures. This indicates that the locality information is critical, as these two measures both prefer new molecules that are arm's length from their nearest neighbors. For the growing-size setting, #Circles sur- passes all other chemical space measures. Reference-based measures such #FG also perform prominently. Both #Circles and reference- measures satisfy subadditivity, making them suitable for "growth" setting. The Sum- variants of aggregation-baan measures outperform their original forms, as they tend to increase when adding new molecules.
(CI) Axiomatic properties (Sec. 4.1
(C2) Correla
-tions (Sec.
4.2
Measures
Axioms sotished
Fixed
Growing
Diversity SumBot #FG
Dissimilarity
Medium
Low
Subadd, dissim Mono, subadd
High
Medium
Medium
Medium
Richness #Circles
Mono, subadd, dissim
Low
Medium
Mono, subadd, dissim
High
High
More experiment results can be found in Appendix B where we also discuss the impact of the distance metric d. We also study the sensitivity of the subset size n, the way of adding new molecules, and the choice of the #Circles's 's threshold t. Results suggest that f = 0.75 is a good choice for both two settings. Correlations between chemical space measures visualized in Figure and 11
4.3 WHICH MEASURE THE RIGHT CHOICE?
The empirical analysis shows that the locality-bart based #Circles measure is robust choice for all tested scenarios, which reconfirms the conclusion of the axiomatic analysis. Besides, SumBottleneck may be effective choice when a fixed number of candidates may be good alternatives if -urefully-designed and comprehensive reference sets are available. Surprisingly, the widely used Diversity measure rendered inferior both analytically and empirically, casting doubts on its efficiency in measuring and encouraging exploration for molecular generation. Our analysis and comparison results are summarized Table
the target. Reference-based measures
MEASURING CHEMICAL SPACE COVERAGE
In this section, we apply the chemical space measures to evaluate how well the existing databases and ML-based molecular generation models cover the chemical space.

---

Published as a conference paper at ICLR 2023
Original Databases
Filtered Databases
ZINC-250K
249K
10K
ZINC-250K
174K
2053
ENAMINE
ENAMINE
MOSES
1937K
MOSES
1845K
ChEMBL
2505K
ChEMBL
967K
4501
GDB-17
1448K
175K
GDB-17
1671K
8115
10
10
10
10f
4x10
Richness
#Circles
Richness
#Circles
Figure 3: Richness and #Circles of molecular databases. Databases are listed vertically. The horizontal axes are chemical space measures (plotted in logarithmic scales). #Circles has the threshold value (t = 0.75). We find discrepancy between Richness and #Circles, mcaning that a larger database does not necessarily span wider range of the chemical space.
5.1 MEASURING CHEMICAL SPACE COVERED BY MOLECULAR DATABASES
Other than being rich source for virtual screening for drug purposes, molecular databases are also bonanzas of data for training ML models. both scenarios, the coverage of chemical space is crucial descriptor for comparing various databases. With higher coverage of the chemical space, we are able construct candidate libraries with better varieties. Also, the ML model would benefit from the diverse training data, obtaining better understanding of the overall chemical space. Here, compare five commonly used molecular databases with the selected measures.
5.1.1 EXPERIMENT SETUP
Databases and filtering rules. We include five widely used drug and compound databases in our measurement: (1) ZINC-250K (a random subset of the ZINC database) (Irwin & Shoichet 2005), (2) MOSES (Polykovskiy et al. 2020), (3) ChEMBL (Gaulton et al. 2017), (4) GDB-17 (Ruddigkeit et al. 2012), and (5) the Enamine Hit Locator Library (ENAMINE). More details about the databases are listed in Appendix F.I
We further apply two filtering rules these databases to select potential drug candidates. In drug discovery, drug-likenes. and synthesizability of compounds are both important considerations. The quantitative drug-likeness (QED) (Bickerton et al. (012)) and the synthetic accessibility (SA) (Brd & Schuffenhauer 2009) are computed. The molecules are filtered with QED 2 0.6 and SA S 4.
5.1.2 RESULTS AND DISCUSSION
Figure compares Richness and Circles of molecular databases. From the results, we conclude the findings as follows. (1) A larger database does not necessarily span wider range of the chemical space: There is discrepancy between Richness and #Circles, and the #Circles measuring results suggest that, in terms of chemical space coverage, ZINC-250K covers larger area than ENAMINE and MOSES. (2) ZINC-250K and GDB-17 are recommended: With the smallest amount of compounds, ZINC-250K renders relatively high coverage of the chemical space, which is highly recommended for model training with limited computation resources. Constructed by enumeration, GDB-17 substantially expands the known chemical space great level, which is suitable for training MI. models for chemical space representations. More details and findings on other chemical measures are provided in Appendix F
5.2
MEASURING CHEMICAL SPACE EXPLORED BY GENERATION MODELS
5.2.1
PROBLEM FORMULATION
In drug discovery, the search for active molecules towards a druggable target is often formulated as the following unconstrained optimization problem (Olivecrona et al. 2017 Gömez- Bombarelli et al.
2018 [iu et al. 2018 You et al. 2018 Jin et al. 2018 De Cao & Kipt 2018 Popova et al. 2019
Shi et al. 2020
Xie et 2021):
arg max self
$1 (1(x) 82(x) o .......ssa).) (2),
(8)
where x is a molecule in the chemical space 41 S : 4 - R a function scoring particular biological properties, and the "o" operator indicates the combination of multiple scores (e.g.. summation or multiplication). This scoring function can be binding affinity to protein targets, drug-likenee synthe- sizability, When wet-lab experiments not available, these scores obtained computationally,
7

---

Published as a conference paper at ICLR 2023
Table 2: Measuring results of the chemical space explored by molecular generation methods. In- corporating chemical space measures into the objectives increases the exploration of the chemical space measures, but none of the tested ML-based models surpasses the virtual screening baseline (Databases) in terms of #Circles when 0.75 (as suggested in Sec. 4.2 highlighted in grey). Numbers in the parentheses following #Circles are values for the threshold l. each chemical space measure, the larger value the better. Bold indicates the best performance in each measure.
Method
Richness
#Cireles (0.70)
#Circles (0.75)
#Circles (0.80)
Diversity
Databases
1,250
78.0 + 4.4%
55.7 + 3.7%
30.3 = 5.0%
0.827
RationaleRL,
526 + 1.7%
14.2 + 6.8%
12.1 + 6.5%
9.9 + 0.3%
0.762 + 0.3%
DST
29
2.0 + 0.0%
1.0 + 0.0%
1.0 + 0.0%
0.559
JANUS
261
15.0 + 11.6%
6.7 + 8.7%
4.0 + 25.0%
0.759
MARS
134K + 10.2%
75.3 + 4.4%
22.6 + 13.3%
7.3 + 16.0%
0,736 + 0.8%
Diversity
177K + 13.7%
93.8 + 7.6%
24.4 + 9.8%
8.8 = 13.9%
0.752 = 0.2%
+SumBot
221K + 13.4%
128.3 + 8.1%
29.3 + 7.6%
8.8 + 13.0%
0.752 = 0.3%
*#Circles
179K + 17.4%
129.9 +
19.1%
30.2 + 18.0%
9.7 + 16.2%
0.749 + 0.7%
Unfortunately, the majority of the computational scoring functions are merely approximations and cannot accurately predict the wet-lab results. As pointed out in the chemistry literature, rather than focusing on optimizing the estimated property scores, it is crucial to generate variety of compounds that span wider range of the chemical space (Huggins et al.) 2011: Wawer et al. [0144] Ashenden 2018). We therefore propose the following objective as the goal of molecular generation, and use this new objective to compare models:
arg max SCu
p(S) S.L. Sx(x) 2 C₁. Vk E [K], x E
where S C l is a molecular candidate set, ((-) is a chemical space measure, s4(z) 2 C₁ means the estimated property score of a molecule x is at or above threshold Ck.
5.2.2 EXPERIMENT SETUP
Following Li et al. 2018) and Jin et al. (2020), we consider the inhibition against an Alzheimer-related target protein c-Jun N-terminal kinase-3 (JNK3) as the biological objective. The JNK3 binding affinity score is predicted by a random forest model based on Morgan fingerprint features of molecule (Rogers & Hahn 2010). We also consider drug likeness (QED) and synthetic accessibility (SA) as they are important in practical drug discovery scenarios.
Generation objective.
Models, variations, and the baseline. We employ four recently proposed molecular generation models to optimize the generation objective (JNK3, QED, and SA): (1) RationaleRL (Jin et al.
2020 generates molecules by combining rationales with reinforcement learning (RL); (2) DST (Fu et al. 2022) supports pradient-based optimization
chemical graphs; (3) JANUS (Nigam et al. 2022) uses the genetic algorithm (GA) to design drugs inversely with parallel tempering; (4) MARS (Xie et al. 2021) generates fragment- based molecular structures based on Markov chain Monte Carlo (MCMC) sampling. Due to MARS' great ability to simultaneously optimize multiple drug discovery objectives, also consider three variants of it MARS+ where the chemical space measures are incorporated into Equation to encourage exploration. The virtual screening setting is introduced as a baseline, where the five aforementioned molecular databases are combined, forming a candidate set of 1 11M compounds.
HARS+SumBottlenee
and
Evaluation. To validate the ability in exploring the chemical space, we compare the molecules generated by models using Equation 9 with multiple chemical space measures. The molecules (including molecules generated by models as well as compounds from databases) are filtered with the constraints JNK3 2 0.5, QED > 0.6, and SA < 4, representing the positive candidates. We examine seven different measures, and report #Cireles, Richness, and Diversity here; the others are listed in Appendix G). Note we cannot directly measure the biological functionality coverage in Section 4.2 as many of the generated molecules are not in the database. For RationaleRI. and DST, 5000 molecules are generated as suggested in their papers. For JANUS and MARS, the population size and the number of MCMC chains are set as 5000. JANUS evolves for 10 generations, and MARS iterates 2000 steps. More implementation details are listed in Appendix G
5.2.3 RESULTS AND DISCUSSION
8

---

Published as a conference paper at ICLR 2023
Table 2 lists the chemical space mea- sures estimated for the positive candi- date compounds (i.e., JNK3 2 0.5, QED 2 0.6, and SA v 4) gener- ated by models from databases. Based on these results, we conclude the following. (1) ML-based mod- els fail to explore a larger effec- databases:
Mathod
104
Witu Bereening
RationaRL
DST
JANUS
10a
MARS
tual area compared Unexpectedly, we find the molecules obtained by virtual screening over databases span the largest area of the chemical space in terms of #Circles when & 0.75. The most compet- itive model among the tested ones, MARS+#Circles,, explores only about half of the space compared to virtual screening. (2) Chemical space mea- sures can encourage the model to explore: By incorporating chemical space measures into the optimization objective of MARS, all measures are significantly improved during molecular generation. (3) Diver- sity should be avoided as a descriptor for exploration: During sampling, MARS and its variants all converge quickly in Diversity (Figure 19). despite a large number of new molecules discovered. Again, it suggests that the widely used Diversity is not suitable for measuring chemical space.
109
101
109
0.4
0.5
0.6
Threshold
Figure 4: #Circles in the log scale and its threshold / selection for various models. Curves of the #Circles measure reflects characteristics of molecular generation methods.
We want to stress that the parameter 1 of #Cireles has an interpretable meaning. When is smaller, #Circles tends to reflect the "spread" of molecules at finer granularity. When t is larger, the measure focuses more on the global picture. Therefore, we can intuitively characterize different molecular generation methods by examining #Circles values with different thresholds i as displayed in Figure From the results, we find MARS and JANUS present similar characteristics as they both have re relatively sharp decrease around = 0.75. This might be because both two models based on sampling algorithms and tend to exploit locally in the chemical space. In contrast, RationaleRL tends explore globally because it designed to combine active fragments (rationales). We can also compare the models by adopting the concept of Pareto optimization. We find the DST, JANUS, and the vanilla MARS model are completely dominated by the variant MARS+#Circles. Besides, the RationaleRI. method is dominated by the virtual screening approach.
few limitations of the measurement with #Circles should be noted. For example the running time of calculating #Circles is exponential in theory. We implement a fast approximation by sacrificing accuracy (Appendix H.. More experiment details and findings are provided in Appendix G
6 CONCLUSION AND FUTURE WORK
In this paper, we have presented a systematic study on the coverage measures of chemical space in the context of drug discovery. The core contribution of this study is novel evaluation framework for selecting proper coverage measures. Using the proposed framework, we have identified a new chemical space coverage measure, #Cireles, which is superior existing heuristic measures commonly used in the drug discovery literature. We have also quantitatively compared molecular databases and ML-based generation models in terms of #Circles. The results suggest that many ML-based generation models fail to explore a larger chemical space compared to virtual screening, since they tend to exploit locally instead of exploring widely.
Measuring the chemical space is a fundamental problem in drug discovery. Our work in this direction opens even more research questions that are worth investigating in the future. For example, it would be interesting to design and evaluate more chemical space measures under our framework. Moreover, the newly suggested objective for molecular generation (Equation (9) inspires further study, since computationally challenging as a combinatorial constrained optimization problem. In general, the proposed chemical space measures and the generic measure selection framework are not restricted to the application of drug discovery, but can also be used in other domains, such as material design, science of science, and text or image generation.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We would like thank Professor Kevyn Collins-Thompesr Professor Paramveer Dhillon, and Professor Aaron Frank for their helpful feedback and suggestions. We also thank the anonymous reviewers for their constructive comments, and particular for pointing out the relation between #Circles and packing number. This work was in part supported by the National Science Foundation under grant number 1633370.
REFERENCES
Shahin Ahmadi, Zohreh Moradi, Ashwani Kumar, and Ali Almasirad. Smiles-based qsar and molecular docking study of xanthone derivatives as a-glucosidase inhibitors. Journal of Receptors and Signal Transduction, pp. 1-12, 2021.
Jorge Alcalde-Unzi and Mare Vorsatz. Measuring the cohesiveness of preferences: an axiomatic analysis. Social Choice and Welfare, 41(4))965-988, 2013.
Stephanie Kay Ashenden. Screening library design. Methods in encymology, 610:73-96, 2018.
Dávid Bajusz, Anita Rácz, and Károly Héberger. Why tanimoto index an appropriate choice for fingerprint-baeed similarity calculations? Journal of cheminformatics, 7(1):1-13 2015.
Yuemin Bian and Xiang -un Xie. Generative chemistry: drug discovery with deep learning generative models. Journal of Molecular Modeling, 27(3):1-18, 2021.
G. Richard Bickerton, Gaia V. Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L. Hopkins. Quantifying the chemical beauty of drugs. Nature Chemistry, 4(2):90-98, 2012.
Thomas Blaschke, Ola Engkvist, Jürgen Bajorath, and Hongming Chen. Memory-assisted reinforce- ment learning for diverse molecular de novo design. Journal of cheminformatics, 12(1):1-17, 2020.
Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling. 59(3): 1096-1108, 2019.
Nicola De Cao and Thomas Kipf. MoIGAN: An implicit generative model for small molecular graphs. ICMI 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018.
Jianyuan Deng, Zhibo Yang, Iwao Ojima, Dimitris Samaras, and Fusheng Wang. Artificial intelligence in drug discovery: applications and techniques. Briefings in Bioinformatics, 23(():bbab430, 2022.
Herbert Edelsbrunner and John Harer. Computational topology: an introduction. American Mathe- matical Soc., 2010.
Daniel C Elton, Zois Boukouvalas, Mark D Fuge, and Peter W Chung, Deep learning for molecular design a review of the state of the art. Molecular Systems Design & Engineering, 4(4):828 849, 2019.
Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based molecular complexity and fragment contributions. Journal of Cheminformatics, 1(1):8 8, 2009.
Hui Fang, Tao Tao, and ChengXiang Zhai. formal study of information retrieval heuristics. In Proceedings of the 27th annual international ACM SIGIR conference Research and development in information retrieval, pp. 49-56, 2004.
Tianfan Fu, Wenhao Gao, Cao Xiao, Jacob Yasonik, Connor W. Coley, and Jimeng Sun. Differ- entiable scaffolding tree for molecule optimization. In International Conference on Learning Representations, 2022.
10

---

Published as a conference paper at ICLR 2023
Tomonori Fukutani, Kohei Miyazawa, Satoru Iwata, and Hiroko Satoh. G-rmsd: Root mean square deviation based method for diree-iimnsiionl molecular similarity determination. Bulletin of the Chemical Society of Japan, 94(2):655-66) 2021.
Anna Gaulton, Anne Hersey, Michal Nowotka, A. Patrícia Bento, Jon Chambers, David Mendez, Prudence Mutowo, Francis Atkinson, Louisa J. Bellis, Elena Cibrián-Uhalte, Mark Davies, Nathan Dedman, Anneli Karlsson, María Paula Magariños, John Overington, George Papadatos, Ines Smit, and Andrew R. Leach. The chembl database in 2017. Nucleic Acids Research, 45, 2017.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Alberto Gobbi and Man-Ling Lee. Dise: directed sphere exclusion. Journal of chemical information and computer sciences, 43(1):317-323, 2003.
Rafael Kömeez-oomareeli, Jennifer
Jennifer Nansean Wei, David Duvenaud, José Miguel Hermández-Lobato Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguire Timothy D. Hirzel, Ryan Prescott Adams, and Alan Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 4(2):268 (77, 2018.
Paul R Halmos. Measure theory, volume 18. Springer, 2013.
Israel N Herstein and John Milnor. An axiomatic approach to measurable utility. Econometrica, Journal of the Econometric Society, pp. 291-297, 1953.
David John Huggins, Ashok Ramakrishnan Venkitaraman, and David Robert Spring. Rational methods for the selection of diverse screening compounds. ACS Chemical Biology, 6(3):208-217, 2011.
John J Irwin and Brian K Shoichet. Zinc- free database of commercially available compounds for virtual screening. Journal of chemical information and modeling, 45(1):177-112, 2005.
Paul Jaccard. The distribution of the flora in the alpine zone. 1. New phytologist, 11(2):37-50, 1912.
Huaipan Jiang, Jian Wang, Weilin Cong, Yihe Huang, Morteza Ramezani, Anup Sarma, Nikolay V Dokholyan, Mehrdad Mahdavi, and Mahmut Kandemir. Predicting protein-ligand docking structure with graph neural network. Journal of Chemical Information and Modeling, 62(12): 2923-2932, 2022.
Wengong Jin, Regina Barzilay, and Tommi S. Jaakkola. Junction tree variational autoencoder for molecular graph generation. In Proceedings of the 35th International Conference on Machine Learning, 2018.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures. In International Conference on Machine Learning, pp. 4849-4859. PMLR, 2020.
Sven Kosub. A note on the triangle inequality for the jaccard distance. Pattern Recognition Letters, 120:36-38, 2019.
Alexios Koutsoukas, Shardul Paricharak, Warren RJD Galloway, David Spring, Adriaan P IJzerman, Robert C Glen, David Mareus, and Andreas Bender. How diverse are diversity assessment methods? comparative analysis and benchmarking of molecular descriptor space. Journal of chemical information and modeling, 54(1):230 242, 2014.
Alex Kulesza and Ben Taskar. Learning determinantal point processes. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence, pp. 41944449444 2011.
Alex Kulesza, Ben Taskar, et al. Determinantal point processes for machine learning. Foundations and Trends Machine Learning, 5(2-3): 2012.
Yibo Li. Liang Ren Zhang, and Zhenming Liu. Multi-objective de novo drug design with conditional graph generative model. Journal of Cheminformatics, 10(1):33, 2018.

---

Published as a conference paper at ICLR 2023
Alan H Lipkus. A proof of the triangle inequality for the tanimoto distance. Journal of Mathematical Chemistry, 26(1):266 11:263-26655 1999.
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander Gaunt. Constrained graph variational autoencoders for molecule design. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Leland McInnes, John Healy, and James Melville. Umap:
Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 1802.03426, 2018.
AkshatKumar Nigam, Robert Pollice, and Alán Aspur--Guzik Parallel tempered genetic algorithm guided by deep neural networks for inverse molecular design. Digital Discovery, 144):390-404 2022.
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatics, 9(1):88444 2017.
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengelingg Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, et al. Molecular sets (moses): benchmarking platform for molecular generation models. Frontiers in pharmacology. 11:1931, 2020.
Mariya Popova, Mykhailo Shvets, Junier Oliva, and Olexandr Isayev. Molecularrnn: Generating realistic molecular graphs with optimized properties. arXiv: Learning, 2019.
Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Günter Klambauer. Fréchet chemnet distance: metric for generative models for molecules in drug discovery. Journal of chemical information and modeling, 58(9):1736-177) 1741, 2018.
David Rogers and Mathew Hahn. Extended-commecttvvgy fingerprints. Journal of Chemical Informa- fion and Modeling. 50(5):742-754, 2010.
Lars Ruddigkeit, Ruud Van Deursen, Lorenz Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of chemical information and modeling, 52(1 18:2664 2875, 2012.
Sergio Caiz-Carmona, Daniel Alvarez-Garcia, Nicolas Foloppe, A Beatriz Garmendia- Doval, Szil- veszter Juhos, Peter Schmidtke, Xavier Barril, Roderick E Hubbard, and S David Morley. rdock: fast, versatile and open source program for docking ligands to proteins and nucleic acids. PLoS computational biology, 10(4)=e100357,, 2014.
Soumitra Samanta, Steve O' O'Hagan, Neil Swainston, Timothy Roberts, and Douglas B Kell. Vae-sim: novel molecular similarity measure based on variational autoencoder. Molecules, 25(15):3446, 2020.
Daniel Schwalbe-Kode Koda and Rafael Gómeez-Bmbarelle Generative models for automatic chemical design. In Machine Learning Meets Quantum Physics, pp. 445-467. Springer, 2020.
Jie Shen, Feixiong Cheng, You Xu, Weihua Li, and Yun Tang. Estimation of adme properties with substructure pattern recognition. Journal of chemical information and modeling, 1034 2010.
Chence Shi. Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: a flow-base base autoregressive model for molecular graph generation. In Proceedings of the 8th International Conference Learning Representations, 2020.
Yun Shi and Mark von Itzstein. How size matters: diversity for fragment library design. Molecules, 24(15))2838, 2019.
Brian K Shoichet, Irwin D Kuntz, and Dale L Bodian. Molecular docking using shape descriptors. Journal of computational chemistry, 13(3):380-397, 1992.
12

---

Published as a conference paper at ICLR 2023
Michael Snarey, Nicholas K Terrett, Peter Willett, and David J Wilton. Comparison of algorithms for dissimilarity-base based compound selection. Journal of Molecular Graphics and Modelling, 15(6): 372-385, 1997.
TT Tanimoto. An elementary mathematical theory of classification and prediction, ibm report (november, 1958), cited in: G. salton, automatic information organization and retrieval, 1968.
Oleg Trott and Arthur J Olson. Autodock vina: improving the speed and accuracy of docking with new scoring function, efficient optimization, and multithreading. Journal of computational chemistry, (1(2):55-461,2000 2010.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. Smiles-bert: large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics, pp. 429 436, 2019.
Mathias J Wawer, Kejie Li, Sigrun Gustafsdottir, Vebjorn Ljosa, Nicole E Bodycombe, Melissa Marton, Katherine L Sokolnicki, Mark-Anthoni Bray, Melissa M Kemp, Ellen Winchester, et al. Toward performance-diverse small-molecus libraries for cell-based phenotypic screening using multiplexed high-dimensional profiling. Proceedings of the National Academy of Sciences, 11 (30):
2014.
Scott A Wildman and Gordon M Crippen. Prediction of physicochemical parameters by atomic contributions. Journal of chemical information and computer sciences, 39(5):868-88 1999.
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. In International Conference on Learning Representations, 2021.
Shiyan Yan. Axiomatic Analysis of Unsupervised Diversity on Large-Scale High-dimensional Data. PhD thesis, 2021.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay S. Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems. 2018.
Jie Zhang, Rocío Mercado, Ola Engkvist, and Hongming Chen. Comparative study of deep generative models on chemical space coverage. Journal of Chemical Information and Modeling, (6):2222 2581, 2021.
13

---

Published as a conference paper at ICLR 2023
A EVALUATION METRICS FOR MOLECULAR DATABASES AND MOLECULAR GENERATION METHODS
Molecular databases and machine- -eteerrteee compounds are rich sources of drug candidates for forming a candidate library in drug discovery. To evaluate the quality of molecular databases and molecular generation methods, variety of metrics proposed. In general, four categories of evaluation metrics can be identified in the literature, which are related to: (1) bioactivities, (2) molecular properties, (3) data likelihood, or (4) the coverage of the chemical space, respectively.
The first category describes the biological actívities of the included or generated molecules toward certain protein targets. Using docking simulators, researchers can calculate docking scores to determine the binding affinity between the given compound structure and the target (Trott & Olson 2010). However, the simulation usually takes a long time to run. To obtain fast feedback and sufficient amount of data to train molecular generation models, machine learning (MI.) methods are also proposed predict the bioactivity of molecules, including linear regression, supported vector machines (SM random forests, and neural networks (NNs) (Shoichet et al. 1992 Olivecronal ct al. 2017 Li et al. 2018 Ahmadi et al. 2021 Jiang et al. 2022). The second category of metrics evaluates the moleculan ar properties of compounds. Some of them are calculated based on heuristic rules, such as the validity of compounds, molecular weight, octanol-water partition coefficient (logP) (Wildman & Crippen| 1999). drug- mingllikeness score (QED) (Bickerton et al., 2012), and synthetic accessibility (SA) (Ertl & Schuffenhauer 2009). With the advent of ML. and deep learning, MIL prediction models like SVMs, random forests, recurrent neural networks (RNNs), and graph neural networks (GNNs) are also employed to predict the properties of compounds (Shen et al., al.||2111 22100] Wang et al., 2019; Gilmer al., 2017). Another branch of metrics uses data likelihood to evaluate the molecules output by generative models 2018 Liu et al., 2018 Jin et al.) 2018 De Cao & Kipf 2018). A few other metries are particularly related to the coverage of the chemical space or the extent of exploration in the chemical space. These enverageannd-exploratioo
(especially deep (eariing--based
Commez-Bombarelll et al.
metries are introduced in Section 2
B COVERING NUMBER AND PACKING NUMBER
In mathematics, given a metric space (T, d), where T is the universal set and d is the distance metric, the coverage of a subset K C T can be described in terms of e-nets (Vershynin) 2018) defined below. Definition B.1 (s-net), Given metric space (T, d), for subset K C T' and positive number an E-net of K is subset N S K, where every point in K is within distance E of i.e.
Vz E K, Bzo E N: d(x,, zo)= S.
Two related concepts, covering number and the packing number, are then defined as follows.
Definition B.2 (Covering number). For subset K of the metric space (T, d)., its covering number, denoted N (K, d, e), is the smallest possible cardinality of an s-net of K.
Definition B.3 (Packing number). For a subset V the metric space (T", d), is called e-separated if d(x. v) > for all points + y E N. The packing number of subset K T, denoted P(K, d, E). is the largest possible cardinality of an E-separated subset K.
Our definition of #Circles is equivalent the packing number when setting the distance threshold in #Circles equal to the diameter of the balls E in the packing number. To our best knowledge, however, the use of this notion chemical space coverage measure is novel. In light of this connection, further leveraging insights from topology to investigate the chemical space would be an interesting future direction.
C AXIOM COROLLARIES AND DISCUSSIONS
Some direct corollaries of monotonicity and subadditivity follows. Corollary C.1 (Subtraction). If a chemical space measure is subadditive, then for any two molecular sets VS S2 C U, we have
(( 2 (1) 82) 2 (((i)) (((2))
14

---

Published as a conference paper at ICLR 2023
Corollary C.2 (Monotonicity [Single Molecule). J chemical space measure μu is subadditive, thenfor any molecular set VS _ l and a single molecule I E lt we have p (z}) 2 p(S).
Corollary C.3 (Dominance). If chemical space measure μa is subadditive, then for any two molecular sets S2 C if $1 S S2, then p((1)) S {((2)).
A subadditive chemical space measure is special case of outer measures in the context of mathemat- ical measure theory. Outer measures are relaxations of measures, where the latter requires a stricter additivity property, i.e., Ju(1 U S2) (((1) + (((2) for any two disjoint sets $1, S2 (Halmos) 2013). We consider additivity to be too strong and can conflict with intuitions in drug discovery: an additive measure defined
discrete space must take the form of μ(S) Enes w(x), in which wx)) is weight function that independently assigns score to each element in the space, meaning additive measures defined on the discrete chemical space cannot capture the interrelationship between molecules in a candidate set, which counters the reality, thus can hardly tell the variety of compounds and how much the chemical space is been covered.
D PROOFS FOR MONOTONICITY, SUBADDITIVITY AND DISSIMILARITY
In Figure Te subadditivity, and/or dissimilarity. In this section, provide proofs verify each measure's monotonicity, subadditivity, and dissimilarity.
we show the chemical space measures according to whether they will satisfy monotonicity,
Note that for aggregation- based measures, when [ 1, M(S) not defined. So without loss of generality, we assume μ({z}) = w(r), x E U in such cases, where w U + R is an importance function.
Proposition D.1 (Monotonicity of chemical space measures.). Reference- chemical space measures, SumDiversity, Diameter, SumDiameter, and #Circles are monotonic. Diversity, Bottleneck, SumBottleneck, and DPP are not monotonic.
Proof. In the proof, for two arbitrary molccular sets S₁ and S₂ as stated in the monotonicity axiom, consider combining them into S := S U.2.
For the reference- chemical space measures, We prove the monotonicity and subadditivity simultaneously for reference-based measures by first proving the monotonicity and subadditivity of the maximum of cover(..)).
To prove the monotonicity and subadditivity of the maximum of cover(., -), consider combining any two molecular sets $1 C U. For any reference E R. we have
max cover(ar, v). max cover(æ, u) xES;
max
2e8;
max USS
cover v)
max cover(x, (z, y) + max cover(z,
#E82
Therefore,
max (Coverage(Si),, Coverage(S2))
Coverage U US)
(Soeraag((S))) + Coverage(S,),
thus proving the Coverage measure is monotonic and subadditive.
For the SumDiversity chemical space measure, note that the monotonicity corollary (Corollary 42 can also entail the monotonicity property defined in Axiom (by sequentially adding molecules from S2 to S from S₁ S2). So we prove the monotonicity (defined in Axiom 4.1) by proving
15

---

Published as a conference paper at ICLR 2023
the monotonicity [single molecule] (defined in C.2
For molecule 24 and molecular set
C l4 with size n, using the fact that SumDiversity S) = n Diversity(S),, we have
SunDiversity(S U (a]) SumDiversity(S)
=(n D)iversity(()) 2
d(x,8) v) Diversity(S)
VES
Diversity(S) 2
d(z, v).
yES
Comparing Diversity(S) and Eyes d(z, is equivalent comparing Eystwes (v V2) and (n 1) EyEs d(x y). Consider each "triangle" tuple (y1, 4)), due the metric characteristics of d, we have d(y), 82) S d(x, v1) + d(x, 32). meaning that Σ fwes d(y) V22 is less or equal to (n 1) Eves d(x, y). When w(x) = 0, SumDiversity($ {x}) SumDiversity($) 2 0, proving monotonicity SumDiversity.
For the Diameter chemical space measure, when the importance function satisfies w(x 0, Va E u, we have
Diameter(S) max d(x, y) z.yES aşy
2 max
max a.yESt an
((x,)):
max d(r,y) z.yESa zzy
max (Diameter(Si), Diameter(S2)),
proving the monotonicity of Diameter.
For the SumDiameter chemical space measure, when the importance function satisfies w(x) 0, Vo E U, we have
SumDiameter(S)
Σ max d(z, y) wES yptizr
Σ max d(z, v), JES
Σ max d(z, y) WES ZES, vst
max
zES1 váz
max maa (S)mmDiametter()))
SumDiameter((S.)
proving the monotonicity of SumDiameter.
For the #Circles chemical space measure, we define (*)S) C S as an arbitrary set that satisfies [C)( #Circles(S)) For any two molecular sets S S2 C U, since (S,) S S1 S2, according to the definition of #Circles, we have
#Circles(S,) < #Circles(S, U.).
Similarly, we also have
#Circles(Sa) #Circles(S, U US),
proving the monotonicity of #Circles.
For the Diversity chemical space measure, disprove its monotonicity by proving it violates the monotonicity [single molecule] corollary (Corollary C.2). For a molecule x and a molecular set S C l4 with size
1, if S, we have
16

---

Published as a conference paper at ICLR 2023
Diversity(S (SU, U (x})
Σ d(v.8)) + Ed(aa d(a,y)
(n + 1)m
BYY'S way
WES
Diversity(S) +
Σ d(z,y). v)
n+1
(n + 1)n yES
And the change in Diversity is
Diversity U {x}) - Diversity(S)
11
n + 1
Diversity(S) +
2
Σ d(x,v)
+
(n + 1)n
BES
2 n +1 1
Diverrity(S))
Σ d(x, v) n yES
1
When the average distance of x and S, i.e., 1/2 E,, d(x y). is less than Diversity (e.g., adding molecule on the "segment" between two existing molecules), Diversity would decrease, thus violating the monotonicity corollary and proving Diversity is not monotonic.
For the Bottleneck chemical space measure, we disprove its subadditivity by proving violates the monotonicity (single molecule] corollary (Corrolary C.2) Consider adding molecule into molecular set with size n > 1. If & S, we have
Bottleneck(SU U (z}) min (Bottleneck(S), min d(x, d(z,v)) v)) yES
When x introduces more restricting bottleneck, i.e., miny d(z, y) < Bottleneck(S), we will have Bottleneck (S (a:)) < (ottlenecc((S), violating the monotonicity corollary, thus proving Bottleneck is not monotonic.
For the SumBottleneck chemical space measure, we disprove its monotonicity by proving it violates the monotonicity [single molecule] corollary (Corrolory C.2). Consider adding molecule into molecular set with size n If & S, we have
SumBottleneck(SU {x))
Σ min yES
min d(y, v'), d(z,v) v) ) g'eS y/ s/
+
min d(x, v), wES
and
SumBottleneck($)=
Σ min d(y. s/). WES nf ES s'
When x introduces some more restricting bottlenecks, i.e., for many y E S, d(x, y) is small (e.g.. adding molecule into set whose size is two, and the molecule is added near one of the two
17

---

Published as a conference paper at ICLR 2023
molecules), will have SumBottleneck(SUU()) SumBottleneck(3)) violating the monotonicity corollary, thus proving SumBottleneck is not subadditive.
For the DPP chemical space measure, disprove its monotonicity by proving it violates the monotonicity [single molecule] corollary (Corrolory C.2 Consider adding x into {ao} where Zo d(2, ro) is denoted as b. We have
(P 1 b = =1-8
b². -
When b 0 will have DPP(( (z), x}) DPP((x0)) violating the monotonicity corollary, thus proving DPP is not monotonic.
Proposition D.2 (Subadditivity of chemical space measures.). Reference-based chemical space mea- sures, Bottleneck, SumBottleneck, and #Circles are subadditive. Diversity, SumDiversity, Diameter, and SunDiameter are not subadditive.
Proof. For the reference-based chemical space measures, the subadditivity provided under Propo- sition D.I
For the Bottleneck chemical space measure, when the importance function satisfies w(x) 2 M, Vo E 4, where M is the upper bound of Bottlenkeck (e.g., w(w) oo), we have
Bottleneck(S)
min d(x,y)
x.yES
zzy
min
min d(x, v), min d(z,y) s.yES, zzy
2,yES, xFy
<Bottleneck(S.) Bottleneck(S:)),
proving the subadditivity of Bottleneck.
For the SumBottleneck chemical space measure, when the importance function satisfies w(x) 2 M, Vx E U, where M is the upper bound of SumBottleneck (e.g., w(x) oo), we have
SumBottleneck(())
Σ
min yES,USS シボ
d(z, v)
SES
Σ
min VESLUSa
d(z, y) + Σ
min
d(z,w) v)
yESLUS2
zES
wES,
Σ
min d(x, y) + Σ BES
min d(x, y) BESt
S
zESa
=SumBottleneck(S) Sum (82)),
proving the subadditivity of SumBottleneck.
For the #Circles chemical space measure. prove its subadditivity by contradiction. For any two molecular sets S₁. S. U, assume #Circles(S, US2) #Circles(S,) + #Circles(S,). Use the notations Ci = C* (S) U S2) Si and C2 = (Si U.)2) n Sz. We have
Ca |C2 2 #Circles(S, U S2) #Circles(St) + (CCcdees))))
Since all values non-negative, must have |C, #Circles(SI) or |C2] #Circles(S.). (S2). contradicting with the definition #Circles(S,) or #Circles(S,). ), thus proving the subadditivity of #Circles.
18

---

Published as a conference paper at ICLR 2023
For the Diversity and SumDiversity chemical space measures, disprove their subadditivity by providing a counter-example. For two disjoint molecular sets with two molecules each, i.e. {21, and {2z, I4 denote dı d();, Then have
Diversity f0 02, 23, 23, Fza)
Diversity( (( L2, F3. F2a
(d12 + d13 d1a + d23 + d2s + d34),
4-3 3
and
Diversity(f33,21 d12 d31-
Similarly,
SumDiiversty((1)),2)))
= 4. Diversityy (2,1, T3,
2
4 (d12 + d13 d11 + d23 + d21 d31),
4.3
and
SumDiversity((zi.a2)) SummDiverstty(zzs..r x13) 2- 2-d12 d31-
When the inter-set distances are larger than the inner-set distances. i.e., d1a > 5-(122
d3a), will have Diversity((=111 T2 #4)) SumDiversity( x22 + Diversity 2a). Simi- larly, when d1s + d14 + d23 + d21 > 2- (d12 + d3a). we will have SumDiversity ((I1, Z2, 23. 24}) > SumDiversity x22) SunDiversity 23, 21 thus proving both Diversity and SumDiversity mea- sures are not subadditive.
For the Diameter chemical space measure, For two disjoint molecular sets Si, S2 S U whose sizes are larger than one, we have
Diameter(S, U822
d(r.y). v),
2.yES zœ
and
Diameter(S,) + Diameter(S,) max d(z,x') + max d(y.y'). 2. ES,
u.y E.y ES,
w*y"
When the maximum inter-set distance is larger than the maximum inner-set distance, i.e., s..us d(x, y) maxx,x' ES, d(x, x') maxy,y' ES d(y. v'), we will have Diameter(S, U S2) > Diameter (S₂) Diameter(Sz), thus proving the Diameter measure is not subadditive.
For the SumDiameter chemical space measure, disprove its subadditivity by providing counter- example. For two disjoint molecular sets with two molecules each, i.e., {21, Ø2 and {03. x4), we denote dij d(x;, m.).. Then we have
19

---

Published as a conference paper at ICLR 2023
SumDiversity((11,,22)
Σ max ((14, sE
LE|1] jths
and
SumDiversity((21: + SumDiversity((33,22) 24}) = 2- d12 + 2- d34.
When the inter-set distances are larger than the inner-set distances, i.e., d13, d14, d23, d21 d12, d3s. we will have SumDiversity( (z,, 22, 33, 33,444 x44) SumDiversity(an + SumDiversity (2a, T4). thus proving the SunDiameter measure is not subadditive.
Proposition D.3 (Dissimilarity property of chemical space measures.). Diversity, SumDiversity, Diameter, SumDiameter, Bottleneck, SumBottleneck, DPP, and #Circles have preferences to dissimi- larity.
Proof. In the proof, for the three molecules To. 22 U as stated in the dissimilarity axiom in Section 4.1 we assume d(zo, x1) 2 d( xz2).
For the Diversity chemical space measure, we have
Diversity((2o. x1)) = d(z0, x1) 2 d(zo, x2) (2))
proving the dissimilarity property of Diversity. For the SumDiversity chemical space measure, we have
SumDiversity((2o..:]] -2 x1) 2 2 d()o, SumDiversity x2)),
proving the dissimilarity property of SumDiversity. For the Diameter chemical space measure, we have
Diameter((2o, x1 d(zo, x1) 2 d(zo, Diameter {0o, x2)),
proving the dissimilarity property of Diameter. For the SumDiameter chemical space measure, we have
SumDiameter((zooz1]) 2dd( x11 2 d(xo, x2) SumDiameter( ((20, x22))
proving the dissimilarity property of SumDiameter. For the Bottleneck chemical space measure, we have
Bottleneck( (2o, x11) = d( x1) 2 d(z0, Bottleneck (fwo, x22),
proving the dissimilarity property of Bottleneck.
For the SumBottleneck chemical space measure, we have
SumBootleneck(zoo.z))) x1)) 2-d(xx, x1) 2 2 d(xo, x22) BumBottleneck((zn,
20

---

Published as a conference paper at ICLR 2023
proving the dissimilarity property of SumBottleneck.
For the DPP chemical space measure defined with the Tanimoto similarity, denoting d(zo, æ1) and d(zo, z2) as d1 and d2 respectively, when d1, d2 [0. (Tanimoto distances),
we have
1 (PP((20,21)) = 1-d1
1- d1
2d1 - di 2d2 - d3
1
1- d2
(22
d2
proving the dissimilarity property of DPP. For the #Circles chemical space measure, we have
#Circles((xoo xi )) I[d(ro, zi) 2 1+ 1 > t #Circles(faog, x2)),
proving the dissimilarity property of #Circles.
Discussion on the dissimilarity property of reference-based measures. For a reference- based measure, considering adding new molecule 21 d2 the existing moelculer set S {2o), have
(0 (max(cover(ETo y), cov.rooi...)).ooooooo v)))
VER
Coverage((no,
Σ
(max(cover(xo: 8). cover v))),
DER.
where the values Coverage((20)) ]. R) and Coverage( (20, x2), R) will depend on the particular definition of cover(.,)) :) and the choice of the reference set R.
Generally, an arbitrary coverage function and an arbitrary reference set do not necessarily meet the dissimilarity requirement. In our study, we define the cover function cover(æ, y) I[molecule x contains fragment y/. where I[|] is the indicator function, and y is a fragment con- tained in the reference set R. Some counter-examples far away from the reference set R while 22 is very close to R. (even contained in R), then the Coverage measure will prefer the molecule d2 instead of the more dissimilar molecule #1.
be easily constructed, for instance, if
Therefore, the dissimilarity property does not hold for chemical space measures #FG, #RS, and #BM.
Discussion on new definition of reference- based measure. In addition to counting the number of fragments in the reference set R contained in the molecular set S, another way define reference- based measure is to investigate the distance from S to R. Particularly, we could define such a reference-based measure as below:
Coverage R; = Σ F E S such that d(x, y) < t).
(10)
DER
where [[[:] is the indicator function, d is the distance metric, and is the distance threshold.
Similar to the discussion #FG, #RS, and #BM, the characteristics of the reference-ber based measure defined by Eq. 10 highly relies on the property of R, and an arbitrary reference set R can not guarantee the dissimilarity property of this measure.
We now prove that the reference- based measure defined by Eq. 10 satisfies both monotonicity and subadditivity. In the proof, Coverage(-) refers the definition Eq. 10
21

---

Published as a conference paper at ICLR 2023
Proof. We first prove the monotonicity and subadditivity of the indicator function I[-], where consider combining any two molecular sets S2.S2 S U. For any reference y E and a distance threshold t, we have
max a S1 such that d(x,y) v) < t), I S2 such that d(z, v) < t))
$ Ar E Si U.2 such that d(x, v) <]
Br E S such that d(x, v) < + [[3 E S2 such that d(x, < t].
Therefore,
max (Coverage(SI). Coverage((S))
Coverage U2)
Coverage S1) Coverage(Si).
thus proving the Coverage measure defined by Eq. 10 is monotonic and subadditive.
E RANDOM SUBSET EXPERIMENT DETAILS
E. BIO-ACTIVITY DATASET
The 10K BioActivity dataset (Koutsoukas al. 2014) contains 10,000 compound samples excerpted from the ChEMBL database (Gaulton al. (017) with bio-activity labels. These labels are the 50 largest ChEMBL activity classes, including enzymes (e.g.. proteases, lyases, reductases, hydrolases, and kinases) and membrane receptors (e.g., GPCRs and non-GPCRs). The label distribution shown in Figure 5
Distribution ChEMBL activity classes
Hydrolases
Transcription
4%
Reductase
Membrane receptons' Non-GPCRs
Transporters 4%
Lyases
Membrane receptons' GPCRs 40%
Kinases 12%
24%
Figure 5: Label distribution of the BioActivity dataset KKousookaa et al. 2014). 50 bio-activity functionality classes are included.
We use UMAP Mclnnes et al. 2018) to visualize the molecules in this dataset based on their Morgan that fingerprint similarity is
fingerprints displayed Figure From the visualization, indeed correlated with bio-activity similarity.
22

---

Published as a conference paper at ICLR 2023
Figure 6: UMAP visualization of compounds in the BioActivity dataset. Different colors stand for different bio-actyvity labels.
E.2 RANDOM SUBSETS WITH FIXED SIZES
We first consider randomly sampled molecular subsets of the BioActivity dataset with fixed size n. randomly sample n molecules S from the dataset and compute the biological functionality coverage GS(S) well each chemical space measure μ(S). By repeating the randomization, can calculate Spearman's 's correlation between the gold standard GS and each individual measure We the experiment for three different fixed sizes 50, 200, and 1000 to represent different molecular distribution density. The pair-wise correlations between measures are displayed in Figure g
Furthermore, when n. = 50 and the molecules are distributed sparsely, all chemical space measures positively correlated with the gold standard. However, when the subset size increases and n 1000, Bottleneck and DPP becomes negatively correlated, while other measure tend to perform better. This because these measures will be bounded by the most similar molecular pair, thus severely conflicting with the subadditivity axiom.
200
In this experiment, we repeat Algorithm for ten times obtain reliable correlations.
Algorithm 1 Calculating chemical space measures for random subsets with fixed sizes.
The fixed subset size n; The bio-activity dataset ((r., yo)) where yı E y are bio-activity labels and [ = 50; K chemical space measures {Px 1K k=1" repeat
Input:
Sample a number m uniformly from {1,...,50) 50).
Sample labels ) uniformly from V.
Sample n molecules S with labels in y" uniformly.
Compute GS(S) and μa(S) for ls E [K]. until repeated for 1000 times
Calculate the correlations between GS and {px K based on the 1000-times experiment results.
Experiment results. We show the experiment results for different fixed random set size n in Figure 7 We find the #Circles and SumBottleneck measures perform constantly better than all other measures.
When the fixed size increases, most chemical space measures' performances also increase, except for Bottleneck and DPP, meaning they are not suitable for measuring the variety when the molecules distributed crowdedly.
23

---

Published as a conference paper at ICLR 2023
Diversity
SumDiversity
Diameter
SumDeemeter
Bottleneck
SumBottleneck
DPP
#FG #RS
#BM
# Circles Richness
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
Correlation
(a) Random subsets with fixed size
50.
Diversity SumDiversity Diameter
Sum Diameere
Bottleneck
SumBottleneck
DPP
# FG #RS
#BM
# Circles Richness
0.2
0.0
0.2
0.4
0.6
0.8
1.0
Correlation
(b) Random subsets with fixed size n 200.
Diversity
SumDiversity Diameter
SumDiameter
Bottleneck
SumBoltleneck
DPP
# FG
#RS
#BM
# Circles Richness
0.2
0.0
0.2
0.4
0.6
0.8
1.0
Correlation
(c) Random subsets with fixed size 1000.
Figure 7: Correlations between the gold standard GS and chemical space measures in the fixed-size random subset setting. The fixed size is set different values. A larger correlation indicates the better. The average results are obtained by running experiments independently for ten times.
Correlation between chemical space measures. We also visualize the pairwise correlation between chemical space measures Figure From the figure can that, the gold standard GS, #Circles, and SumBottleneck are most similar with each other in the fixed-size setting.
Threshold / for #Circles. The #Circles threshold is selected to maximize the correlation to the gold standard GS. Taking 200 as example, we test different values as Figure 9 displays and select = 0.70 as the threshold. We can see #Circles works well for wide range of thresholds like [0.40, 0.70]. Olivecrona al. (2017), the authors suggest to use threshold 0.60 decide whether two molecules are dissimilar with each other") which aligns our results. For n 50 and 1000, the threshold set as 0.70 and 0.65 respectively.
In the original text, the authors suggest similarity threshold of 0.40 that is equivalent to distance threshold of 0.60.
24

---

Published as a conference paper at ICLR 2023
0
Battleneck
Richness
Diameter
SumDiameter
#FG
Diversity
SumDiversity
GS
SumBottleneck
#Circles
.RSS
#8M
DPP
Richness
Diameter
SumDiameter
#FG
Diversity
SumDiversity
GS
SumBottenneck
#Crrcles
#RS
#BM
Bottl
Figure 8: Correlations between chemical space measures in the fixed-size random subset setting. The fixed size is set as n 200. larger correlation indicates the better. The average results are obtained by running experiments independently for ten times.
0.8
0.6
Correlation
0.4
0.2
0.0
-0.2
0.2
0.4
0.5 Threshold t
0.8
1.0
Figure 9: Correlations between the gold standard GS and the #Circles measure in the fixed-size setting with different threshold t. The fixed size is set as 200. A larger correlation indicates the better. The average results are obtained by running experiments independently for times.
Distance metric d. We also study the impact of distance metric d. In Table 3 experiment results for both fingerprint-based Tanimoto distance and VAE-based latent space dissim- ilarity (Samanta al. 2020). We find the experiment results obtained with the VAE dissimilarity remain consistent with the results obtained with the Tanimoto distance.
we listed the
E.3 RANDOM SUBSETS WITH GROWING SIZES
To mimic the molecular generation process, we also grow the size of the subsets. Specifically, for maximum size n, sequentially sample molecules without replacement to form subsets
25

---

Published as a conference paper at ICLR 2023
Table 3: Correlations between the gold standard and chemical space measures in the fixed-size random subset setting. The fixed size is set as n 200. A larger correlation indicates the better. Results are obtained by averaging ten independent experiments. Italic texts indicate molecular representation and the distance metric. The top three measures are highlighted in green and the best measure is printed in bold.
Aggregation-basee
Tanimoto distance
VAE dissimilarity
Diversity
0,478 1 0.011
0.388 + 0.040
SumDiversity
0.478 + 0.011
0.388 0.040
Diameter
0.179 + 0.031
0.112 = 0.022
SumDiameter
0.228 + 0.028
0.201 = 0.029
Bottleneck
-0.293 = 0.015
-0,298 = 0.027
SumBottleneck
0.821 = 0.010
0.527 0.013
DPP
-0.183 + 0.021
-0.244 0.030
Reference-based
Fgragment
#FG
0.421 + 0.033
#RS
0,574 + 0.025
#BM
0.610 + 0.028
Locality- based
Tanimoto distance
VAE dissimilarity 0.745 0.014
#Cireles
0.831 1 0.008
SMILES
Richness
-0,207 = 0.025
For both the gold standard GS and a chemical space measure μ., record their values as S grows into time series. e.g., {(,, (( Comparing the trajectory of chemical space measure with the trajectory of the gold standard, we can observe which measure behaves more similarly to GS. We quantitatively estimate the similarity of their trajectories with the dynamic time warping (DTW) distance of the two time series.
(S, = (
In this experiment, we repeat Algorithm for ten times to obtain reliable DTWW distances.
Algorithm 2 Calculating chemical space measures for random subsets with growing sizes.
Input:
The maximum subset size The bio-activity dataset { vs) v))) where y1 ) are K
bio-activity labels and | 50: K chemical space measures {11 {p k=1- Sample a number m uniformly from (1,..., 50).
Sample labels y" from y.
for i in {l..... n do
Sample unseen molecule x, whose label is in y
Set Se := { . Compute GS(S,) and (S.) for l [K].
end for
Plot chemical space measure curves for GS and (μx) KK where the axes are E (1,..., n) and the y axes are chemical space measure values GS(S,) and (S.)..
Transform the cumulative curves into incremental ones.
Calculate DTW distances between incremental curves.
Experiment results. To mimic the way in which generation models propose new molecules, in Algorithm 2, we require the newly sampled molecule z; to be similar to the already sampled molecules {T1..... 1 The specific implementation can be found our code Moreover, we
also test the following two cases: (1) All molecules are sampled uniformly; (2) The newly sampled
molecule r; have be most similar to the already sampled molecules (x1,.... T 1 1). The results of
DTW distances for these two cases are shown in Figure 10
*The code will be released after publication.
26

---

Published as a conference paper at ICLR 2023
Diversity
SumDiversity
Diameter
SumDiameter
Bottleneck
SumBottieneck
DPP
#FG
#RS
# BM
#Crclles
Richness
1
2
3
5
10
50
DTW distance
(a) The new molecule is uniformly sampled from all unseen molecules.
Diversity SumDiversity
Diameter
SumDiameter
Bottleneck
SumBottleneck
DPP
#RS
#BM
#Circles
Richness
2
3
5
10
50
DTW distance
(b) The new molecule z; needs to be similar to the already sampled ones
Ø1
Diversity
SumDiversity
Diameter
SumDiameter
Bottleneck
SumBottleneck
DPP
# FG
# RSS
#BM
#Circles Richness
2
3
DTW distance
(c) The new molecule x; needs to most similar the already sampled ones
Ti=1-
Figure 10: DTW distances between the gold standard GS and chemical space measures in the growing-size random subset setting. The maximum size is set 1000. A smaller distance indicates the better. The average results are obtained by running experiments independently for ten times.
From Figure 10 we can see that, #Circles performs the best. Also, as the new molecules to add become more similar the existing ones, the advantage of #Circles other measures becomes larger. This makes #Circles especially suitable for measuring molecular generation models.
27

---

Published as a conference paper at ICLR 2023
5
Bottleneck
DPP
Richness
SumDiversity
SumDiameter
SumBattleneck
#BM
#Circles
#FG
Diversity
Diameter
Bottleneck
DPP
Richness
SumDiversity
SumDiameter
SumBottlenes lentleeeck
#BM
GS
#Circcles
#FG
#RS
Diversity
Diameter
Figure 11: DTW distances between chemical space measures in the growing-size random subset
setting. The maximum size is set as 1000, and the new molecule needs be similar to the already samples ones. A smaller distance indicates the better. The average results are obtained by running experiments independently for ten times.
15
DTW distance
10
5
0.4
0.5
0.6
0.7
0.8
0.9
Threshold
Figure 12: DTW distances between the gold standard GS and the #Circles measure in the growing- size random subset setting with different threshold t. The maximum size is set as n 1000, and the new molecule needs to be similar the already samples ones. A smaller distance indicates the better. The average results are obtained by running experiments independently for ten times.
DTW distances between chemical space measures. We visualize the pairwise DTW distances between chemical space measures in Figure 11 From the figure we can see that, the gold standard GS and the #Circles measure are most similar with each other in the growing-size setting.
In addition, we find the Richness, SumDiversity, SumDiameter, SumBottleneck, and #BM are forming large cluster, while #FG and #RS tend to similar with each other.
28

---

Published as a conference paper at ICLR 2023
Threshold for #Circles. The #Circles threshold is selected to minimize the DTW distance to the gold standard GS. Taking the second scenario as an example, we test different values as Figure 12 displays and select 0.76 the threshold. For the first and the third scenarios, the threshold is set as 0.84 and /. = 0.78 respectively.
Distance metric d. We also study the impact of distance metrie d. In Table 4 listed the experiment results for both fingerprint-based Tanimoto distance and VAE-based latent space dissim- ilarity (Samanta al., 2020). We find the experiment results obtained with the VAE dissimilarity remain consistent with the results obtained with the Tanimoto distance.
Table 4: DTW distances between the gold standard and chemical space measures in the growing-st random subset setting. The maximum size set as n 1000, and the new molecule needs to be similar to the already samples ones. A smaller distance indicates the better.
Aggregation-base
Tanimoto distance
VAE dissimilarity
Diversity
18.668 + 6.973
30.063 + 4.284
SumDiversity
5.425 + 0.404
5.484 + 0.296
Diameter
17.299 4.801
28.071 3.917
SumDiameter
5.328 0.396
5,472 0.297
Bottleneck
38.668 5.769
37.168 + 5.422
SumBottleneck
5.167 1 0.353
5,432 + 0.293
DPP
18.845 3.962
12.052 + 2.176
Reference-based
Fgragment
#FG
3.797 0.295
#RS
4.382 0.247
#BM
5.365 + 0.396
Locality-basee
Tanimoto distance 2.173 0.910
VAE dissimilarity 2.470 0.629
#Circles
SMILES 5.454 0.347
Richness
F MEASURING MOLECULAR DATABASES
F.1 MOLECULAR DATABASES
We measure the chemical space coverage for five molecular databases that are commonly used in virtual screening and generative model training:
(1) ZINC-2501 random subset of the ZINC database (Irwin & Shoichet 2005) and consists of 249K commercially-availabee compounds from different vendors for virtual screening:
(2) MOSES (Polykovskiy et al., 2020) another sub-collection of 7INC molecules. contains approximately 2M molecules in total, filtered by molecular weights (ranged range from 250 to 350 Daltons), the number of rotatable bonds (not greater than 7), water-octanol partition coefficient (logP, less equal than 3.5), atom types (C, N. S, O, F, CI, Br, and H), ring cycle sizes (no larger than 8), medicinal chemistry filters (MCFs), and PAINS filters.
(3) ChEMBL (Gaulton al., 2017 is manually curated database of 2M bioactive molecules with known experimental data, especially with the inhibitory and binding properties against macromolecule targets. This database contains not only drug and drug-like molecules, but also natural products and biopolymers.
(4) GDB-17 (Ruddigkeit et al. 2012) enumerates chemical structures that contain up to 17 atoms of C, N, O, S, and halogens, overlapping with the molecular weight range typical for lead
$7INC-250k: 440a 6bc wwwd33333331dd333eeeee333 1
e75e f7a1 1d0c5ff3ddCer
29

---

Published as a conference paper at ICLR 2023
(0.7)
(0.8)
Diveraity
SumBottieneck
ZINC-250U
240456
10281
3883
1035
0.71
1623.23
3490
6076
136430
10210
3670
1033
0871
1697.10
10236
1602.87
ENAMINE
400150
0145
1700
377
0.53
2011.00
150
1435
206497
0883
2555.70
MOSES
1936952
8090
2480
623
0.889
3316.20
1158
6816
448026
8142
2486
0.857
5025
$284.05
CHEMBL
23048/5
32583
10856
2510
0.879
12003 99
17006
46614
718657
12078.51
32518
11082
2708
12019:48
GDB-17
1140778
1/5300
29092
3296
0.896
73099 70
82000
1802502
3539100
20116
2048
0.885
73730.80 85
Figure 13: Measuring results on original databases.
Database
Richness
#Circles (0.75)
Diversity
SumBottleneck #FG
#RS
#BM
ZINC-250K
173731
2035
0.861
1106.10
1952
2940
92715
1098.43
2040
0.863
1115.64
ENAMINE
427366
1229
0.851
2461. 75
142
1268
187742
2402.39
1263
0.851
2375.03
MOSES
1846492
1714
0.856
8832 55
938
4513
413521
1698
0.855
8955.37 37
1720
0.855
CHEMBL
987322
4570
0.872
5423.73
6304
14004
317255
5464.92
5368.
GDB-17
1621490
8094
0.889
10733.83
5969
167795
425582
8138
0.889
10870.31
0.889
Figure 14: Measuring results filtered databases.
compounds. The lead-like subset filters 11M compounds with lead-like properties (100-350 MW & 1-3 clogP).
(5) Enamine is a company that provides compound libraries for high-throughput screening (HTS). The Enamine Hit Locator Library their largest diversity library with high MedChem tractability, and the compounds in the library are readily available for purchase and are guaranteed synthesizable at reasonable cost.
F.2 RESULTS ON CHEMICAL SPACE MEASURES
The measuring results are listed in Figure 13 and 14
G MOLECULAR GENERATION EXPERIMENT DETAILS
G.1 MODEL IMPLEMENTATION
We implement the models with the official repositorie All hyperparameters are default.
'ENAMINE libraries:
ibraries
RationaleRL:
D
JANUS: htN.ah/nw///..ttato.cccooooooooa
MARS:
samp
30

---

Published as a conference paper at ICLR 2023
G.2 MARS VARIANTS
The terms incorporated into objectives are computed as follows 8
Hovertyyiwerssyy (I, ()
Σ d(r.v)
(11)
[S]
yES
Novelty Sumbotttencck (2, (x, min d(x, yES
(12)
Hovelty (x, min d(x, y) > yes
(13)
For the model variants, we test different a values from {0.1, 0.3, 1.0, and report the best performance. Specifically, we use a = 1.0 for MARS+Diversity, = 0.3 for MARS+SumBottlenkk, and a = 0.1 for MARSe#Ci The threshold we use for the #Circles measure is = 0.60.
G.3 RESULTS ON OTHER MEASURES
Table 5: Measuring results of the chemical space explored by molecular generation methods. In each chemical space measure, the larger value the better. Bold indicates the best performance in each measure.
Method
SumBottleneck
#FG
#RS
#BM
Databases
280
58 0.0%
132
502
RationaleRL.
0.3%
207 0.0%
442 2.3%
DST
8
9
26
JANUS
76 +
93 +
73 =
133 +
MARS
535 = 7.9%
346 + 16.0%
67 = 3,5%
19.5K + 23.1%
+Diversity
12.9%
463 17.2%
20.1K 25.2%
SumBot
12.5%
868 + 15.3%
2.3%
14.0K 20.8%
+#Circles
742 + 12.9%
601 + 35.0%
69 + 5.5%
18.9K + 15.3%
The results on other chemical space measures are listed in Table 5
G.4 MOLECULAR PROPERTY AND BINDING AFFINITY DISTRIBUTIONS
To investigate how incorporating chemical space measures into the objective Eq. 11 can influ- ence the optimization for molecular properties, we examine the property distributions of molecules generated by MARS and its variants.
As shown in Figure [5 16, our resulting libraries from the joint optimizations with chemical space measures (i.e.. MARS+#Circles, and MARS+Diversity)) are not showing a significant downward shift on JNK3 binding affinity as well QED scores and SA scores. It is expected that compared with vanilla MARS, the property scores of molecules generated by MARS variants would decrease slightly, because instead of solely optimizing the property scores, the model needs to make sacrifices for more diverse library. However, this sacrifice would insignificant compared to the diversity gain in the resulting library. In practice, these molecules would pass the scoring threshold and would be considered qualified in these regards. In Table 2, the experiment results show that by incorporating chemical space measures into optimization objectives, the richness of discovered qualified molecules (INK3 2 0.5, QED 2 0.6, and SA < 4) got significantly improved.
G.5 VISUALIZATION
To provide a more intuitive view of the effect that adding a joint objective exploration can encourage the molecular generation model to discover wider space, visualize the chemical space explored by the baseline (MARS) and MARS-SumBotleneck as well as MARS+#Circles in Figure 17] where
Eqquations 11 13 are approximations of [n(S {x}) ((S) defined to avoid numerical issues and for
computational effecincy.
31

---

Published as a conference paper at ICLR 2023
JNK3 Prediction Scores (All Samples)
rDock Docking Scores (30k Samples)
1.0
10
Interactive Cocking Score kcal/moll
Predicted Activity Soote
08
07
06
30
04 MARS
MiveS+Diverrity
MARS
MARS-Diverrely
Figure 15: Left: Distributions JNK3 machine learning model predicted score. We observe downward trend in the MARS variant models, which are jointly optimized with chemical space measure. However,
generated molecules are above the score threshold, and in practice, due the limitation of the prediction model, they should be considered equally qualified candidates for further analysis. Right: Distributions of interaction scores of the ligands and the receptor docking (lower energy score indicates
favorable interactions). For the docking experiments, randomly sampled 30K molecules from each generated library and performed molecular docking on the JNK3 target (PDB: 7KSI) using rDock (Ruiz- (Ruiz-Carmona Carmona et al.] 2014). The distributions of docking scores show that the joint optimizations on chemical space measures do nor affect the binding affinity significantly when predicted by in silico docking.
show the 2D layout of the Morgan fingerprints of the functional groups in the molecules generated through principal component analysis (PCA). The larger number unique functional groups and wider spread of the data points obtained by adding exploration-based novelty terms indicate more diverse structures being explored.
We show the molecular clustering results in Figure 18 Clusters calculated based Morgan fingerprints of the generated molecules and their Tanimoto similarity. Compared to the baseline model, larger number of clusters be obtained from MARS+Sum
Figure shows the dynamics of measures for MARS and its variants.
H FAST APPROXIMATION OF #CIRCLES
The computation of #Circles combinatorial and the running time is exponential in theory. We implement two fast approximations by sacrificing accuracy Algorithm and Algorithm 4
Algorithm 3 Approximation of #Circles (sequential).
Input:
The molecular set S to be measured, the distance metric d, the distance threshold t. Randomly reorder the molecular set S. Initialize an empty set C. Set : (S).
for i in (1...., n do
Add S, to if minyec d(a, u)
end for
Return: [C] and
32

---

Published as a conference paper at ICLR 2023
Nemaloee Symbert Soww (S44)
0.00
0.80
6.88
0.76
0.0
8.00
696
670
6400
MARS
200
330
300
230
MAPS
MARS-#Drddo
MA-S-Chardly
Figure 16: Selected molecular property distributions. Except for SAS (normalized from to the larger the better), all property distributions do not show significant changes across different models. The downward shift of the SAS is likely due to the increased diversity in the molccular librarics.
MARS
MARS +#Circles
+SumBottleneck
Optimizing chemical space measures encourages the molecular generation model to explore larger span of the chemical space. This figure shows principal component analysis (PCA) of functional groups discovered by different models.
Figure 17:
Algorithm 4 Approximation of #Circles (recursive).
Input: The molecular set S be measured, the distance metric d, the distance threshold maximum number recursive layers L, number processors m.
Call Algorithm
3 with arguments (8, d. t) if L 0.
Randomly reorder the molecular set S.
Evenly split S into m subsets Si,.... Sm. Call Algorithm 4 with arguments (S,, d, t, L 1, m) for = 1....,m and collect the returns C,
Set = C₁ U C
Return: Results obtained by calling Algorithm with arguments (C, d, t).
33

---

Published as a conference paper at ICLR 2023
MARS
MARS+SumBottennek
Figure 18: Hierarchical clustering results of generated molecules. Different colors stand for different clusters. The clustering threshold is set as 0.7x x the height of the highest linkage accordance with SciPy's 's default value.
Aichnes
Diversity
SARR
30
100c
2000
Step
Step
Step
Figure 19: The dynamics of chemical space measures over generated molecules. Incorporating chemical measures into generation objectives increases the exploration of the chemical space.
34


Reviews:
[{"review_id": "pEhQSazI_B", "reviewer": "Reviewer_cGEX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The present paper is concerned about measuring how much a set of molecules covers the chemical space. All of the metrics that they consider take a set of molecules $\\mathcal{S}$ as input and output a coverage metric. Among many existing heuristically-designed metrics, the authors propose a metric called #Circles, which amounts to the maximum number of non-overlapping circles of radius t/2 whose center are some of the molecules in $\\mathcal{S}$.\n\nThe authors claim the validity of the proposed metrics in two ways. First, they propose two axioms that such a metric should satisfy, and conclude that the proposed metric satisfies both of them, while the others do not. Second, they compute the correlation between the coverage metric and a proxy gold standard of the variety of the molecules (which is the number of unique biological functionality types in $\\mathcal{S}$), and show that the proposed metric correlates with the gold standard better than the others. Based on these two findings, the authors suggest that the proposed metric, #Circles, is preferred to the others.\n\nThe authors finally measure the coverage by the existing databases as well as the molecules generated by existing generative models. The analysis on the existing databases suggests that ZINC-250k and GDB-17 are recommended in terns of coverage. The analysis on generative models suggest that the sets of molecules discovered by existing ML-based methods have less coverage than the virtual screening.", "summary_of_the_review": "This paper focuses on the validity of metrics used to evaluate the performance of generative models, and I like the problem setting. Although there are several issues to be resolved or to be further discussed in the paper, this paper could be a good starting point for us to understand and enhance evaluation metrics for molecular generation. Therefore, I would suggest to accept this paper, given the authors discuss the issues on the axioms in the paper (or if there is any misunderstanding, please correct me).", "strength_and_weaknesses": "# Strengths\n- The problem setting itself is good, because the validity of existing metrics has been less focused in the domain.\n- The proposed metric is intuitive and looks reasonable.\n- Analyses on both the existing databases and molecules discovered by generative models are insightful and will be a basis for enhancing generative models.\n\n# Weaknesses\n### Axioms are not very convincing to me\nThe sub-additivity axiom implies that adding more and more molecules to $\\mathcal{S}$ does not hurt the score, which may be reasonable for some purposes but may not be reasonable for others. For example, some of the existing methods aim to construct a focused library where molecules in it are more likely to satisfy the requirements. In such a case, adding irrelevant molecules to the generated library may improve the proposed coverage score, but is not desirable considering the original purpose. This indicates that \"axioms\" (which, I suggest, should be replaced by \"assumptions\" or \"requirements\") of the metrics depend on how the metrics are used. Therefore, I would ask the authors to clarify the usages of the metrics and deduce requirements for those usages.\n\nIn fact, it seems that the authors assume the scenario where the (focused) library is an input to the virtual screening module; given that scenario, adding irrelevant molecules to $\\mathcal{S}$ is ok. In contrast, some of existing generative models may wish to substitute not only the library but also the virtual screening, i.e., they wish to generate hits directly. The proposed metric is not suitable to measure the diversity of the generated hits.\n\nIn summary, I would like the authors to discuss the scenario they consider and relate the axioms to the scenario tightly. In addition, I would like the authors to discuss the limitation of the proposed axioms and metrics, i.e., it may not be applied to other scenarios including evaluation of generative models of hits.\n\nRegarding Axiom 4.2, what I don't understand is whether there always exists $x^\\star$ such that $d(x^\\star, x_1)=d(x^\\star, x_2)=\\frac{1}{2}d(x_1, x_2)$. If this statement does not hold, the axiom is not well-defined, and should be refined.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "n7T6DRtLCn", "reviewer": "Reviewer_yinU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new metric called #Circles for chemical spacing measurement. The authors analyzed and compared existing metrics and their metric mathematically, proving that their metric has two good properties. They also conducted experiment to analyze current datasets and molecule generation models by their metric.", "summary_of_the_review": "The paper provides enough mathematical proof to confirm their proposed metric has good theoretical properties. But the experimental results indicate that this metric is sensitive to the parameter, which make is impractical to use this metric to evaluation molecule generation models. And the model equipped with this metric did not show significant improvement over other metrics.", "strength_and_weaknesses": "Strengths:\n\nThe authors provide a new perspective on the metrics for measuring the chemical space and conduct various experiments for molecule generation to demonstrate the argument.\n\nWeaknesses:\n\n 1) #Circles is  sensitive to the distance threshold parameter t, according to Table 2. MARS with #Circles is notably better than RationaleRL when t is 0.70; but theses two models achieve similar #Circles scores when t is 0.80. Such sensitivity to distance threshold parameter will make #Cricles impractical to evaluate model performance, i.e., the score can be controlled by selecting various distance threshold parameter to present inconsistent results.\n\n 2) Is the setting of Axiom4.2(Dissimilarity) necessary in defining the measure of the topological space and what is the significance in the field of realistic drug design?\n\n 3) Are there any relevant experiments that can demonstrate the formulation of Eq. 3 to generate higher quality molecules? What kind of model will the authors design for this purpose.\n\n************************\nConsidering the reviewers' detailed response and revision, I have raised the score. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OurK4to4EGU", "reviewer": "Reviewer_74LY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper the authors propose an axiomatic analysis of some of the metrics most commonly used in ML-based de novo design algorithms. Specifically, they lay down several axioms that measures of coverage of chemical space should obey and based on those propose a new metric for evaluation (#circles). ", "summary_of_the_review": "In general, I believe this to be a positive contribution to the field and I think the work provided here is novel enough to recommend publication. However, there are a few things/questions that the authors should address:\n\n* In Figure 2, the #circles metric seems to outperform many of the other proposed metrics. Richness, in particular, as the authors mention, is a specific case of the #circles metric when t=0, yet performance-wise its correlation is even negative. Could the authors elaborate why they think this happens?\n* On the threshold t=0.75. The authors claim that this choice is good for both the fixed size and the dynamically growing set settings - however this seems to have only been tested on the first setting (Fig 9).\n* On section 5.2.3 the authors claim that ML models fail to explore a larger effectual area compared to databases. This conclusion is drawn after setting up a large baseline database, which will obviously cover a larger portion of chemical space than the generative methods, since the space of JNK3 binders with predefined characteristics should intuitively be smaller. I personally do not see this as a failure of the ML models, but I would love to have to see this further discussed in the manuscript. Along those lines, instead of using all 5 databases as a baseline, have the authors considered using a subset of these data sources where only JNK3 inhibitors are included?\n* Another interesting insight is that the authors claim that the #circles metric can be tuned to explore either global or local regions of chemical space depending on which threshold t is chosen. I believe further attention should be paid to this, as it is only briefly mentioned in the results section. Along those lines, instead of choosing one threshold t, have the authors considered using a weighted average of the metric as a function of t? \n\n\nOther minor points:\n\n* Typo in section 4.2.2 (space before comma, after \"is critical\")\n* In table 1, should the \"axiomatic properties\" header not point to Sec 4.1, instead of 3?\n* Figure 9 (appendix) could use some rewriting\n* Missing comma after Eq. 2\n* Figure 4 is somewhat hard to read - consider increasing its size.", "strength_and_weaknesses": "Overall this is a good contribution to the field, w\n\nStrengths: \n\n* Provides new insight on the information that commonly-used metrics for de novo design provide.\n* Formal, thorough analyses.\n* Easy to read manuscript, provides enough context to understand the topic. \n* Proposes new metric to evaluate chemical libraries addressing limitations of the existing ones\n\nWeaknesses:\n\n* Motivation for the presented axioms somewhat absent\n* No accompanying code provided for the #circles metric, at least at time of submission.\n* Exponential compute time for the proposed metric. Approximation not provided.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BqYtG_tQV-j", "reviewer": "Reviewer_UhAH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper discusses techniques to measure the coverage and diversity of a subset of chemical space (i.e. a set of molecules). The authors give a mathematical definition of a diversity measure, then postulate two axioms which these measures should satisfy: sub-additivity and dissimilarity. This leads to their 3 main contributions:\n\n1. Classifying 3 different families of diversity measures\n2. Showing that many popular diversity measures do not satisfy their axioms\n3. Proposing the #Circles measure which does satisfy both axioms.\n\nThe authors then perform several molecular screening and design experiments with the diversity measures discussed above, including the #Circles measure.", "summary_of_the_review": "To me the good parts of the paper are the #Circles diversity and the criticism of other diversity measures, which I think the ML community needs to hear and digest to improve the quality of work in this area. However, numerous small issues make me think the paper is not quite ready to be accepted in its current form. Chief among these is that the axioms are not very well-justified in my opinion. This part of the paper is too important to be glossed over.\n\nI really like the #Circles measure and would like to accept this paper, so I would propose the following changes for my score to increase:\n1. Change axioms\n2. Fix notation issues and errors which I pointed out above\n3. Better discussion of dependence on $d,t$ for all metrics (these are somewhat arbitrary choices which will have a large influence on the results unfortunately)", "strength_and_weaknesses": "**Strengths**\n\n- _Important insight about previous diversity measures_: specifically, the proof/demonstration that the average pairwise distance between molecules in a set is not sub-additive is very important, because many previous papers published at ML conferences have used this measure. I think this paper makes a good case that this measure is fundamentally flawed and should not be used, which is itself an important contribution.\n- _#Circles diversity measure is principled and useful_: it is interpretable, satisfies the proposed axioms, and intuitively makes a lot of sense. I think it can be used as a drop-in replacement for the flawed diversity measure above.\n- _Classification of previously-used diversity measures is a nice insight_: I think that assembling and classifying many previously proposed methods to measure diversity is itself a useful contribution for people who work on molecules. While I am not certain about the novelty of this, I think many members of the ML community would benefit from reading it.\n- _Implications beyond molecules_: although this paper focuses just on molecules, the findings about diversity measures and the proposal of the #Circles metric are completely general, and could be used to measure the diversity of data from other domains (e.g. images, point clouds, etc). I think the authors should state this more explicitly.\n\n**Weaknesses**\n\n- _Many un-numbered equations_: this made writing this review difficult and will make it difficult for people referring to parts of your paper to point to specific equations. Please give all equations numbers.\n- _Questionable assumption of distance metric in section 3.1_: throughout the paper, the authors assume there is a reference distance metric $d$ on chemical space. I think this assumption is problematic: in reality there are _many_ possible distance metrics on chemical space, for example Tanimoto distances with many different types of fingerprints, descriptor-based distances (e.g. Euclidean distance between `rdkit` descriptors), or optimal-transport type distances. It is not really clear that any one metric is better than the others. What makes this problematic in my opinion is that the authors assume a single canonical metric $d$ and make this central to their analysis: for example in axiom 4.2. Clearly many diversity measures which satisfy axiom 4.2 for a given distance metric $d_1$ would no longer satisfy it if a new metric $d_2$ was used (but $d_1$ was still used internally by the diversity measure). I think it would be appropriate for the authors to take a more agnostic stance on this (e.g. simply saying that a metric $d$ can be used to define diversity, but it can also be defined in other ways).\n- _Various confusing terminology/errors_.\n  - The equation for Tanimoto distance in definition 3.2 actually seems to define Tanimoto similarity (i.e. $d(x,x)=1$ instead of $d(x, x)=0$. The authors probably meant to use $1-d$ as the definition.\n  - The term \"measure\" in definition 3.3 is a poorly chosen name in my opinion because \"measure\" is already a loaded term from measure theory describing a similar function which also satisfies other properties (notably sub-additivity). Given that the authors intend to define a measure here as a function which is _not_ necessarily sub-additive, I think this may confuse many readers. I would suggest instead \"diversity measure\" or \"diversity metric\" to clarify this.\n  - The term \"similarity matrix\" is used on page 4 without being defined in general. For the Tanimoto metric there is an obvious notion of similarity ($1-d$), but for other distance metrics I don't think this is the case.\n  - \"Cover\" is not really defined in section 3.2.2: it is unclear what the range of a \"cover\" function is, not what its properties should be. I would instead try to define this using a suitable distance metric (e.g. Tanimoto distance with fingerprints which explicitly encode the presence of scaffolds). Coverage could then be restated in a similar way to circles, e.g. $\\mathrm{Coverage}(\\mathcal{S}, \\mathcal{R}) = \\sum_{y\\in\\mathcal{R}} \\mathbb{I}\\left[\\exists x\\in\\mathcal{S}: d(x, y) < t\\right]$. This is counting the number of molecules in the reference set with a sufficiently similar molecule in $\\mathcal{S}$\n  - #Circles in equation 1 will always equal 0. I think the authors need to add the additional condition $x\\neq y$ to get the desired behaviour.\n- _Small issues with classification system in section 3.2_: I don't understand why #Circles is not a distance-based measure: it also uses a distance metric, and is essentially using the matrix of all pairwise distances between molecules like most of the methods in section 3.2.1. I think that putting it in its own section is a bit confusing, and downplays the connection between #Circles and all the other diversity measures. Second, I think that the parameters of many of these metrics should be more clearly stated.\n  - I would give a subscript $d$ to all measures in section 3.2.1 (e.g. $\\mathrm{Diversity}_d, $ ) to clearly emphasize the dependence on a chosen distance metric $d$.\n  - Similarly, I think you should write $\\mathrm{Circles}_{d,t}$\n- _Axioms in section 4 are not well-justified_:\n  - Axiom 4.1 is justified by saying that increasing the size of the set should not decrease the diversity.  First, while I believe this, I don't know that everybody would believe this intuitively. I think you need to argue more explicitly that diversity should be measuring \"coverage\" and not \"average similarity\". If I were to write the argument, I would essentially say \"a large set of molecules can be easily filtered to a smaller set so having many similar molecules is not a bad thing: what really matters is the region of chemical space explored by the list\". Second, this intuitive justification only justifies the first inequality, not the second ($\\mu(S_1\\cup S_2) \\leq \\mu(S_1) + \\mu(S_2)$). The reason for requiring this inequality is completely unclear to me. I know that a similar inequality appears in measure theory but I don't know why it applies here. I actually think it should be removed. This slightly weaker axiom is less disputable, and would still allow corollaries A.2 and A.3 to hold. It would slightly change the classification in Figure 1 (e.g. diameter would now be sub-additive). I would then also rename this axiom to \"monotonicity\".\n  - Axiom 4.2 does not seem to match the description given: the description states that more dissimilar molecules should be given a higher diversity when added to a set, but the axiom seems to state that when trading off distance between 2 elements of a set and a third element, the trade-off with an even distance between the two points should be favoured. This seems sort of reasonable to me, but it is not at all clear that this should be required. I would certainly not describe it as \"simple and intuitive\". I think the authors need to provide better justification for this axiom or re-formulate it. Also, it depends on the distance metric used which I think is undesirable (c.f. my discussion above).\n- _No details on how to compute circles diversity_: the authors state that computing#Circles is combinatorial with exponential running time, and that they use a fast approximation instead. However this approximation does not appear to be described anywhere. Given that #Circles is a key contribution of this paper, I think the authors need to explicitly state this, ideally in the main text instead of the appendix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster