Venue: ICLR 2023 Conference

Reviews:

final_score: rating: null
review_content: In a recent discussion of this paper, a colleague of mine pointed out that the `#circles` metric described in this paper is essentially the same as the _packing number_ in topology. [Wikipedia](https://en.wikipedia.org/wiki/Covering_number) defines the packing number with radius $r$ as equivalent to #circles at $2r$, while [these lecture notes](http://www.stat.yale.edu/~yw562/teaching/598/lec14.pdf) define it as being equivalent to #circles. Overall I am pretty sure that the definition of #circles is _exactly_ equivalent up to a change of variables, but the authors should double check this. At the very least, it is extremely similar. My guess is that the authors of the paper were not aware of these connections.

I feel slightly embarrassed that I did not catch this during my original review. I think it detracts slightly from the paper (because #circles is not actually a _novel_ metric), but nonetheless _I still think that accepting this paper was the correct decision_: the idea to use it as a diversity measure for molecule generation is a valuable insight, and the analysis of other diversity measures is valuable.That being said, in light of this, I think the following changes would be appropriate:

1. The authors should acknowledge that #circles and packing number are equivalent (or at least very similar), and add some sort of citation to refer a reader to relevant work in this area (perhaps a topology textbook of some sort). I also think the authors should not say that their metric is "novel".
2. Previous work has proved many theorems about packing number, for example that it is non-decreasing, and have also characterized the relationship with the _covering number_. Perhaps these studies could be leveraged to strengthen the claims and conclusions of the paper?
review_title: Suggestion to discuss relationship between #Circles and packing number

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new metric called #Circles for chemical spacing measurement. The authors analyzed and compared existing metrics and their metric mathematically, proving that their metric has two good properties. They also conducted experiment to analyze current datasets and molecule generation models by their metric.
strength_and_weaknesses: Strengths:

The authors provide a new perspective on the metrics for measuring the chemical space and conduct various experiments for molecule generation to demonstrate the argument.

Weaknesses:

 1) #Circles is  sensitive to the distance threshold parameter t, according to Table 2. MARS with #Circles is notably better than RationaleRL when t is 0.70; but theses two models achieve similar #Circles scores when t is 0.80. Such sensitivity to distance threshold parameter will make #Cricles impractical to evaluate model performance, i.e., the score can be controlled by selecting various distance threshold parameter to present inconsistent results.

 2) Is the setting of Axiom4.2(Dissimilarity) necessary in defining the measure of the topological space and what is the significance in the field of realistic drug design?

 3) Are there any relevant experiments that can demonstrate the formulation of Eq. 3 to generate higher quality molecules? What kind of model will the authors design for this purpose.

************************
Considering the reviewers' detailed response and revision, I have raised the score. 


clarity,_quality,_novelty_and_reproducibility: 
The paper is clear written and it provides the reader a new perspective on this problem.
 The experiment results indicate that this metric is too sensitive to the parameter, which make is impractical to use this metric to evaluation molecule generation models. 

summary_of_the_review: The paper provides enough mathematical proof to confirm their proposed metric has good theoretical properties. But the experimental results indicate that this metric is sensitive to the parameter, which make is impractical to use this metric to evaluation molecule generation models. And the model equipped with this metric did not show significant improvement over other metrics.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discusses techniques to measure the coverage and diversity of a subset of chemical space (i.e. a set of molecules). The authors give a mathematical definition of a diversity measure, then postulate two axioms which these measures should satisfy: sub-additivity and dissimilarity. This leads to their 3 main contributions:

1. Classifying 3 different families of diversity measures
2. Showing that many popular diversity measures do not satisfy their axioms
3. Proposing the #Circles measure which does satisfy both axioms.

The authors then perform several molecular screening and design experiments with the diversity measures discussed above, including the #Circles measure.
strength_and_weaknesses: **Strengths**

- _Important insight about previous diversity measures_: specifically, the proof/demonstration that the average pairwise distance between molecules in a set is not sub-additive is very important, because many previous papers published at ML conferences have used this measure. I think this paper makes a good case that this measure is fundamentally flawed and should not be used, which is itself an important contribution.
- _#Circles diversity measure is principled and useful_: it is interpretable, satisfies the proposed axioms, and intuitively makes a lot of sense. I think it can be used as a drop-in replacement for the flawed diversity measure above.
- _Classification of previously-used diversity measures is a nice insight_: I think that assembling and classifying many previously proposed methods to measure diversity is itself a useful contribution for people who work on molecules. While I am not certain about the novelty of this, I think many members of the ML community would benefit from reading it.
- _Implications beyond molecules_: although this paper focuses just on molecules, the findings about diversity measures and the proposal of the #Circles metric are completely general, and could be used to measure the diversity of data from other domains (e.g. images, point clouds, etc). I think the authors should state this more explicitly.

**Weaknesses**

- _Many un-numbered equations_: this made writing this review difficult and will make it difficult for people referring to parts of your paper to point to specific equations. Please give all equations numbers.
- _Questionable assumption of distance metric in section 3.1_: throughout the paper, the authors assume there is a reference distance metric $d$ on chemical space. I think this assumption is problematic: in reality there are _many_ possible distance metrics on chemical space, for example Tanimoto distances with many different types of fingerprints, descriptor-based distances (e.g. Euclidean distance between `rdkit` descriptors), or optimal-transport type distances. It is not really clear that any one metric is better than the others. What makes this problematic in my opinion is that the authors assume a single canonical metric $d$ and make this central to their analysis: for example in axiom 4.2. Clearly many diversity measures which satisfy axiom 4.2 for a given distance metric $d_1$ would no longer satisfy it if a new metric $d_2$ was used (but $d_1$ was still used internally by the diversity measure). I think it would be appropriate for the authors to take a more agnostic stance on this (e.g. simply saying that a metric $d$ can be used to define diversity, but it can also be defined in other ways).
- _Various confusing terminology/errors_.
  - The equation for Tanimoto distance in definition 3.2 actually seems to define Tanimoto similarity (i.e. $d(x,x)=1$ instead of $d(x, x)=0$. The authors probably meant to use $1-d$ as the definition.
  - The term "measure" in definition 3.3 is a poorly chosen name in my opinion because "measure" is already a loaded term from measure theory describing a similar function which also satisfies other properties (notably sub-additivity). Given that the authors intend to define a measure here as a function which is _not_ necessarily sub-additive, I think this may confuse many readers. I would suggest instead "diversity measure" or "diversity metric" to clarify this.
  - The term "similarity matrix" is used on page 4 without being defined in general. For the Tanimoto metric there is an obvious notion of similarity ($1-d$), but for other distance metrics I don't think this is the case.
  - "Cover" is not really defined in section 3.2.2: it is unclear what the range of a "cover" function is, not what its properties should be. I would instead try to define this using a suitable distance metric (e.g. Tanimoto distance with fingerprints which explicitly encode the presence of scaffolds). Coverage could then be restated in a similar way to circles, e.g. $\mathrm{Coverage}(\mathcal{S}, \mathcal{R}) = \sum_{y\in\mathcal{R}} \mathbb{I}\left[\exists x\in\mathcal{S}: d(x, y) < t\right]$. This is counting the number of molecules in the reference set with a sufficiently similar molecule in $\mathcal{S}$
  - #Circles in equation 1 will always equal 0. I think the authors need to add the additional condition $x\neq y$ to get the desired behaviour.
- _Small issues with classification system in section 3.2_: I don't understand why #Circles is not a distance-based measure: it also uses a distance metric, and is essentially using the matrix of all pairwise distances between molecules like most of the methods in section 3.2.1. I think that putting it in its own section is a bit confusing, and downplays the connection between #Circles and all the other diversity measures. Second, I think that the parameters of many of these metrics should be more clearly stated.
  - I would give a subscript $d$ to all measures in section 3.2.1 (e.g. $\mathrm{Diversity}_d, $ ) to clearly emphasize the dependence on a chosen distance metric $d$.
  - Similarly, I think you should write $\mathrm{Circles}_{d,t}$
- _Axioms in section 4 are not well-justified_:
  - Axiom 4.1 is justified by saying that increasing the size of the set should not decrease the diversity.  First, while I believe this, I don't know that everybody would believe this intuitively. I think you need to argue more explicitly that diversity should be measuring "coverage" and not "average similarity". If I were to write the argument, I would essentially say "a large set of molecules can be easily filtered to a smaller set so having many similar molecules is not a bad thing: what really matters is the region of chemical space explored by the list". Second, this intuitive justification only justifies the first inequality, not the second ($\mu(S_1\cup S_2) \leq \mu(S_1) + \mu(S_2)$). The reason for requiring this inequality is completely unclear to me. I know that a similar inequality appears in measure theory but I don't know why it applies here. I actually think it should be removed. This slightly weaker axiom is less disputable, and would still allow corollaries A.2 and A.3 to hold. It would slightly change the classification in Figure 1 (e.g. diameter would now be sub-additive). I would then also rename this axiom to "monotonicity".
  - Axiom 4.2 does not seem to match the description given: the description states that more dissimilar molecules should be given a higher diversity when added to a set, but the axiom seems to state that when trading off distance between 2 elements of a set and a third element, the trade-off with an even distance between the two points should be favoured. This seems sort of reasonable to me, but it is not at all clear that this should be required. I would certainly not describe it as "simple and intuitive". I think the authors need to provide better justification for this axiom or re-formulate it. Also, it depends on the distance metric used which I think is undesirable (c.f. my discussion above).
- _No details on how to compute circles diversity_: the authors state that computing#Circles is combinatorial with exponential running time, and that they use a fast approximation instead. However this approximation does not appear to be described anywhere. Given that #Circles is a key contribution of this paper, I think the authors need to explicitly state this, ideally in the main text instead of the appendix.
clarity,_quality,_novelty_and_reproducibility: **Clarity**: the writing of this paper is generally good. My main recommendation would be for the authors to slightly change their terminology (e.g. change the word "measure", see above for more)

**Novelty/Originality**: from what I understand, the #Circles diversity measure and the classification of previously-used diversity measures is original and in my opinion highly valuable. I think that many people in ML for molecules would benefit from reading this paper (I personally benefitted from reading it).

**Quality**: There are good parts and bad parts. I think the #Circles diversity and the criticism of existing diversity measures is a very high-quality contribution. I think that other papers should immediately start using #Circles. However, to me the bad parts of the paper are the axioms, which are not super well-justified. I think the only indisputable axiom is monotonicity; the authors should either justify the rest or provide a more nuanced analysis which does not require the reader to agree with the strong forms of both axioms.

**Reproducibility**: code is not provided, and some key details are missing (e.g. the fast approximation to #Circles). I would consider this work not very reproducible.
summary_of_the_review: To me the good parts of the paper are the #Circles diversity and the criticism of other diversity measures, which I think the ML community needs to hear and digest to improve the quality of work in this area. However, numerous small issues make me think the paper is not quite ready to be accepted in its current form. Chief among these is that the axioms are not very well-justified in my opinion. This part of the paper is too important to be glossed over.

I really like the #Circles measure and would like to accept this paper, so I would propose the following changes for my score to increase:
1. Change axioms
2. Fix notation issues and errors which I pointed out above
3. Better discussion of dependence on $d,t$ for all metrics (these are somewhat arbitrary choices which will have a large influence on the results unfortunately)
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The present paper is concerned about measuring how much a set of molecules covers the chemical space. All of the metrics that they consider take a set of molecules $\mathcal{S}$ as input and output a coverage metric. Among many existing heuristically-designed metrics, the authors propose a metric called #Circles, which amounts to the maximum number of non-overlapping circles of radius t/2 whose center are some of the molecules in $\mathcal{S}$.

The authors claim the validity of the proposed metrics in two ways. First, they propose two axioms that such a metric should satisfy, and conclude that the proposed metric satisfies both of them, while the others do not. Second, they compute the correlation between the coverage metric and a proxy gold standard of the variety of the molecules (which is the number of unique biological functionality types in $\mathcal{S}$), and show that the proposed metric correlates with the gold standard better than the others. Based on these two findings, the authors suggest that the proposed metric, #Circles, is preferred to the others.

The authors finally measure the coverage by the existing databases as well as the molecules generated by existing generative models. The analysis on the existing databases suggests that ZINC-250k and GDB-17 are recommended in terns of coverage. The analysis on generative models suggest that the sets of molecules discovered by existing ML-based methods have less coverage than the virtual screening.
strength_and_weaknesses: # Strengths
- The problem setting itself is good, because the validity of existing metrics has been less focused in the domain.
- The proposed metric is intuitive and looks reasonable.
- Analyses on both the existing databases and molecules discovered by generative models are insightful and will be a basis for enhancing generative models.

# Weaknesses
### Axioms are not very convincing to me
The sub-additivity axiom implies that adding more and more molecules to $\mathcal{S}$ does not hurt the score, which may be reasonable for some purposes but may not be reasonable for others. For example, some of the existing methods aim to construct a focused library where molecules in it are more likely to satisfy the requirements. In such a case, adding irrelevant molecules to the generated library may improve the proposed coverage score, but is not desirable considering the original purpose. This indicates that "axioms" (which, I suggest, should be replaced by "assumptions" or "requirements") of the metrics depend on how the metrics are used. Therefore, I would ask the authors to clarify the usages of the metrics and deduce requirements for those usages.

In fact, it seems that the authors assume the scenario where the (focused) library is an input to the virtual screening module; given that scenario, adding irrelevant molecules to $\mathcal{S}$ is ok. In contrast, some of existing generative models may wish to substitute not only the library but also the virtual screening, i.e., they wish to generate hits directly. The proposed metric is not suitable to measure the diversity of the generated hits.

In summary, I would like the authors to discuss the scenario they consider and relate the axioms to the scenario tightly. In addition, I would like the authors to discuss the limitation of the proposed axioms and metrics, i.e., it may not be applied to other scenarios including evaluation of generative models of hits.

Regarding Axiom 4.2, what I don't understand is whether there always exists $x^\star$ such that $d(x^\star, x_1)=d(x^\star, x_2)=\frac{1}{2}d(x_1, x_2)$. If this statement does not hold, the axiom is not well-defined, and should be refined.


clarity,_quality,_novelty_and_reproducibility: # Clarity
The paper is well-organized and well-written, and it is very comfortable to read through.

# Quality
The quality is satisfactory except that there is a minor issue to be resolved. The equation in Definition 3.2 is actually Tanimoto similarity, and Tanimoto distance should be 1 - Tanimoto similarity. I assume that this is a typo, but I would appreciate if the authors re-confirm their software implementation.

# Novelty
Discussion on the coverage metric in this domain is novel to me.

# Reproducibility
It seems there exists enough information to reproduce the results. I would appreciate if the authors publish the source code.
summary_of_the_review: This paper focuses on the validity of metrics used to evaluate the performance of generative models, and I like the problem setting. Although there are several issues to be resolved or to be further discussed in the paper, this paper could be a good starting point for us to understand and enhance evaluation metrics for molecular generation. Therefore, I would suggest to accept this paper, given the authors discuss the issues on the axioms in the paper (or if there is any misunderstanding, please correct me).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper the authors propose an axiomatic analysis of some of the metrics most commonly used in ML-based de novo design algorithms. Specifically, they lay down several axioms that measures of coverage of chemical space should obey and based on those propose a new metric for evaluation (#circles). 
strength_and_weaknesses: Overall this is a good contribution to the field, w

Strengths: 

* Provides new insight on the information that commonly-used metrics for de novo design provide.
* Formal, thorough analyses.
* Easy to read manuscript, provides enough context to understand the topic. 
* Proposes new metric to evaluate chemical libraries addressing limitations of the existing ones

Weaknesses:

* Motivation for the presented axioms somewhat absent
* No accompanying code provided for the #circles metric, at least at time of submission.
* Exponential compute time for the proposed metric. Approximation not provided.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well-written and provides a clear description of the problem that the authors are trying to address. In terms of novelty, some of the analyses provided I believe deserve credit:

* A deep look into metrics that measure coverage of chemical space
* An axiomatic analysis for the aforementioned metrics, with the proposal of two that intuitively most metrics should fulfill.
* A criteria to evaluate coverage metrics based on their correlation with biological functionality.
* Recommendations on which widely-known chemical databases cover the most space according to the metrics defined
* Insights into how large the coverage of molecules generated by de novo design algorithms 


Sadly, none of the current study is reproducible as the authors claim that code will be released in the future. I believe that having an implementation of the provided metric (#circles) could already benefit other ongoing molecular de novo design studies.
summary_of_the_review: In general, I believe this to be a positive contribution to the field and I think the work provided here is novel enough to recommend publication. However, there are a few things/questions that the authors should address:

* In Figure 2, the #circles metric seems to outperform many of the other proposed metrics. Richness, in particular, as the authors mention, is a specific case of the #circles metric when t=0, yet performance-wise its correlation is even negative. Could the authors elaborate why they think this happens?
* On the threshold t=0.75. The authors claim that this choice is good for both the fixed size and the dynamically growing set settings - however this seems to have only been tested on the first setting (Fig 9).
* On section 5.2.3 the authors claim that ML models fail to explore a larger effectual area compared to databases. This conclusion is drawn after setting up a large baseline database, which will obviously cover a larger portion of chemical space than the generative methods, since the space of JNK3 binders with predefined characteristics should intuitively be smaller. I personally do not see this as a failure of the ML models, but I would love to have to see this further discussed in the manuscript. Along those lines, instead of using all 5 databases as a baseline, have the authors considered using a subset of these data sources where only JNK3 inhibitors are included?
* Another interesting insight is that the authors claim that the #circles metric can be tuned to explore either global or local regions of chemical space depending on which threshold t is chosen. I believe further attention should be paid to this, as it is only briefly mentioned in the results section. Along those lines, instead of choosing one threshold t, have the authors considered using a weighted average of the metric as a function of t? 


Other minor points:

* Typo in section 4.2.2 (space before comma, after "is critical")
* In table 1, should the "axiomatic properties" header not point to Sec 4.1, instead of 3?
* Figure 9 (appendix) could use some rewriting
* Missing comma after Eq. 2
* Figure 4 is somewhat hard to read - consider increasing its size.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new metric called #Circles for chemical spacing measurement. The authors analyzed and compared existing metrics and their metric mathematically, proving that their metric has two good properties. They also conducted experiments to analyze current datasets and molecule generation models by their metric. 

The paper originally obtained diverse reviews. By considering the author rebuttals and after several rounds of discussions, the reviewers eventually come to a consensus and believe the paper has its merits and is worth accepting. In particular,
1)	It is working on an important topic – the quality of the metric of ML-based de novo drug design.
2)	The authors have significantly improved the motivation of their work during their revisions.
3)	The paper should have implications for the ICLR audience.
justification_for_why_not_higher_score: The paper is acceptable, but still with some concerns from the reviewers.
justification_for_why_not_lower_score: The paper has its merits, and after the discussions, one of the reviewers raised his/her score to reflect the consensus.
Decision: Accept: poster