Peer review of: Understanding the diversity of the metal-organic framework ecosystem (2020)

REVIEWERS' COMMENTS: 
 
Reviewer #1 (Remarks to the Author): 
 
The authors quantify the diversity of chemical space in common MOF databases by using machine learning 
techniques. While this is a very interesting topic - which is furthermore relevant for a wide community -, I 
am a bit disappointed in the actual results and conclusions of the study. The analysis is definitely a welcome 
and necessary addition to the proliferation of MOF databases, but it does not provide substantially new 
insights as suggested in the abstract and introduction. Many of the observations were already known from 
earlier studies, but are now combined into an exhaustive study. However, I appreciate how they highlight 
some of the issues in MOF screening studies with a number of well-chosen examples, and it is a nice wake-
up call for all computational scientists who routinely use one specific (MOF) database. 
 
In conclusion, it is an instructive manuscript that clearly points out some potential problems of screening 
studies, and it is supported by a lot of data and explanatory figures. Nevertheless, as the current manuscript 
is written very specifically for MOFs, I am not convinced by the general importance of this work for the more 
general audience Nature Communications aims at and I feel that this manuscript might be better suited in a 
more specific materials science/physical chemistry journal. I leave it to the Editor to decide based on this 
journal aims and scope. 
 
Below a number of suggestions are given that could help the authors to strengthen their work: 
 
1. The authors have chosen to study CH4 and CO2 adsorption because of their differences in chemistry. 
Furthermore, they decide to work in three different adsorption regimes (infinite dilution, low pressure, high 
pressure). It is well-known that generic force fields are typically unable to describe the chemically specific 
interactions between the framework and CO2. For example, this is the case for MOFs with coordinatively 
unsaturated metal sites. Selecting a different generic force field could deliver very different results. 
Achieving quantitative and qualitative correct predictions for the right reasons is therefore a challenging 
task. Of course, this is the price that has to be paid when working with large databases. However, how can 
the authors be sure that their analysis will not change when selecting a different force field? How sensitive is 
e.g. the analysis on the importance of the different variables (Figure 4) on the chosen level of theory? 
 
2. Many observations mentioned in the manuscript (especially in the section on ‘Predicting adsorption 
properties of MOFs’) are already known in the MOF community. The authors should refer to previous 
literature. 
 
3. The authors select a diverse set of structures from the CoRe-2019 and BW-DB database in their third 
example to illustrate that models trained on such a subset possess a good transferability. I think that 
generating diverse subsets without bias towards some regions is a very interesting product of their study, 
and I believe such subsets should be used in the future as sanity checks for screening studies. Is it possible 
to provide a tool to automatically sample such a set of structures from a given larger database? 
 
4. One of the central questions of this study is about how one would like to be sure that a new MOF adds 
relevant information. Can the authors provide an easy (quantitative of qualitative) way for experimental 
researchers to test a new structure to the large set of structures that they considered? 
 
 
Reviewer #2 (Remarks to the Author): 
 
In this excellent contribution the authors develop tools to quantify and analyze the structural and chemical 
diversity MOF databases and apply these tools to some prominent MOF databases providing an evidence-
based perspective on past large-scale simulation studies on MOFs, which will be probably eye-opening for 
many readers. The authors take advantage of the available information about MOF diversity to answer 
questions regarding how well the MOF-space has been explored in previous high throughput screening 
efforts, to what extent bias is present in different databases, what the implications of these biases are on 
conclusions that emerged from earlier studies, and what are key ways to improve diversity and reduce bias 
in future studies. The manuscript is filled with numerous useful “bits” of information that I am sure will guide 
future molecular simulation and machine learning studies in MOFs, in a way that will probably make MOF 
discovery more effective. In my opinion, this is the kind of work that I would expect to find in Nature 
Communications, so I am recommending the publication of this manuscript. I have, however, some 
comments that I would like the authors to consider. They are presented below not necessarily in order of 
importance. 
 
1) I am just curious about why no neural network models were considered for the machine learning part of 
this work. Was there a particular rationale for the chosen machine learning methods? Was it because it is 
less straightforward to examine variable importance? 
 
2) I think that the higher metal diversity of the CoRE MOFs, or otherwise the lack of metal diversity in 
hypothetical databases is something that has been pointed out here and there in the past. In that regard, 
the mining of nodes for future generation of MOFs is really informative. I wonder if the authors have any 
thoughts on why the metal diversity is so low in hypothetical databases. Sometimes I am under the 
impression that CoRE MOFs have numerous metals, but in single atom form with “flimsy” linker connections, 
such as only one oxygen of a carboxylate, that lead to easy collapse upon activation. This might be a reason 
why so many CoRE MOFs remain in obscurity, while most of the experimental MOF literature overwhelmingly 
focus on the usual suspects (MOF-5, MOF-74, UiO-66, Cu-BTC, MIL-100, ZIF-8, etc). In this case, it would 
seem to me that hypothetical databases tend to emphasize on “robust” nodes from well-known MOFs that 
are known to withstand activation. Other times, though, I am under the impression that the connectivity of 
CoRE MOFs is simply much more complicated than what MOF generation methods can handle. Can the 
authors comment? 
 
3) How common are the nodes presented in the SI? That is how many CoRE MOFs have them? What kind of 
linkers are associated with them? 
 
3) About the discussion around Figure 4. Specifically, the ARABG database. In the cited publication, 
Anderson et al report variable importance for some CO2 adsorption metrics (selectivity, working capacity), 
with a pore descriptor having the largest importance (kind of in line with the variable importance here for 
BW-20K). These are not the CO2@0.15bar adsorption metrics the authors are using here, but it does make 
me wonder how sensitive general conclusions regarding whether chemistry or pore geometry are more 
important are to how the chemical and geometrical descriptors are defined, as well as the type of machine 
learning method. 
 
4) One implication of the discussion in this work is that simply trying to get high number of structures in a 
database for the sake of getting a high number is not the best use of resources. A more targeted MOF 
generation for the application of choice may be the smarter strategy. For example, the ToBaCCo database 
seems to have been originally explored for H2, CH4 and Xe/Kr, which are not expected to be affected by 
functionalization, making it OK for the database not to have functional groups. On the other hand, the 
ARABG database seems small but emphasizes on functional groups, which base on the discussion of this 
paper seems very important for CO2 adsorption. For ARABG, machine learning was done with a small set 
(considering typical databases), so my question is whether in the current work the authors have unveiled 
some sort of, at least semi-quantitative, insights or rules of thumb about critical database size (assuming 
maximum diversity) beyond which machine learning models will not learn much more. (This perhaps relates 
to the kind of work presented in Fig. 5) 
 
4) I think it would be a good idea to present histograms in the supplementary information as complementary 
way to present the information given in Figure 2 and Figure 3. Some readers may find those histograms 
easier to interpret. 
 
5) Arguably, one way to read Fig. 4 could be that the screening the CoRE MOFs fails to capture that linker 
functionalization is important for CO2 adsorption as noted by screening of ARABG or BW-20K, or as thought 
of my experimentalists who, for example, consider strategies such as amino functionalization to boost CO2 
sorption. In inspecting Fig S13 and S14, I can’t help but notice that the CoRE MOFs fail to cover an area of 
functional groups that is covered by the BW database (I don’t know about ARABG). Moreover, the area the 
CoRE MOFs do cover tend toward the area that ToBACCo covers, which we know does not really have 
functional groups. So I wonder if the CoRE MOFs are just lacking in those key functional groups that truly 
have an impact on CO2 adsorption, which allow metal chemistry to take such an outsized role. 
 
6) The authors state: “This explains the relatively poor performance in prediction of CO2 adsorption 
properties using only geometric descriptors as electrostatic interaction plays a crucial role.” Based on an 
earlier sentence, I think the authors may have meant to say: “(…) prediction of CO2 adsorption properties at 
low pressure”. 
 
 
 
Reviewer #3 (Remarks to the Author): 
 
The authors have created a well written paper that describes a new way to classify and organize the many 
MOFs available in popular databases, and to study their diversity in doing so. In addition to being well 
written, the paper is well organized and the results are clearly presented. I think this paper can be accepted 
essentially without revisions. 
 
That being said, notions of MOF diversity are necessarily (?) subjective. It's hard to imagine how one can be 
write or wrong when measuring MOF diversity, so the aim is more to be more or less useful, but that will 
then depend on what applications MOF scientists are interested in. To their credit, the authors do get into 
applications and comment on the utility of their measures for those purposes... but it would have been nice 
to see an at least short discussion in the introduction that acknowledges this fundamental subjectivity. 
Conversely, if I am wrong and MOF diversity can be objectively measured, then I would certainly want the 
authors to explain that early in the paper! 
 
In summary, unsurprisingly good work from great authors. Looking forward to seeing it published!