dataesr/scientific-paragraphs-categorization huggingface dataset:
Patrick Walter used a selection of drugs of the ChEMBL as a reference. Such restrictions can be useful to drug likeness. However, for organic molecular materials such restrictions could be too harsh. So, we propose two different lters. Filter 1 is based on the full ChEMBL25 compounds as a reference dataset. This corresponds to 556 187 unique connectivity features. Filter 2 is based on both ChEMBL25 and ZINC20. ZINC20 encompasses approximately 800k unique features, and the union of both encompasses 1 156 416 unique connectivity features of many atom types. To illustrate such lters, we have tested several molecules studied in the MOLTECH-Anjou laboratory in molecular materials for electronics and photonics. The azobenzene, thioindigo, tetracyanoquinone, and perylene molecules are composed of only connectivity features present in ChEMBL and thus pass both lters, see Fig.
2. However, the most iconic electron donor molecule, the tetrathiafulvalene presents two connectivity features that do not exist in ChEMBL25, see Fig. 3. Since this molecule exists in the ZINC20 dataset, it passes lter 2. It is worth pointing out here how the ECFP works. An ECFP2 takes into account the central atom, its bonds, and the type of bonds of its rst neighbours. Potentially in the case of a carbon bonded to four atoms which are themselves bonded to three other atoms, the largest connectivity feature in an ECFP2 would be dened by 17 atoms as in 2,2-dimethylpropane. With the ECFP4 ltering, we take also into account the chemical nature of the second neighbours and their bonds. Theoretically, it can include implicitly up to 53 atoms. In the whitelists, the ECFP4 max radius has been considered to ensure a short and medium range ltering.
However, as it will appear soon in the results, the connectivity features keep the information of atoms being in a cycle but not its size. The evolutionary algorithm can therefore propose highly constrained unsaturated cycles while respecting this ltering. The use of ECPF6 might have limited the use of a whitelist based on cyclic properties, but ECFP6 includes explicitly the atoms up to three of the central atom. This descriptor is therefore very specic. For small molecules, ltering by ECFP6 is almost equivalent to having a list of allowed SMILES. In order to further rene our realistic chemical space, a second ltering based on generic cyclic features (denoted GCF) is proposed. The process is illustrated on the ribavirin molecule in Fig. 4. Using the NetworkX python library,  35  the vertices (bonds) that do not belong to a cycle are deleted (Fig.
4a ). RDKit is then used to compute the Murcko scaffold on each remaining subgraph that includes a cycle (Fig. 4b ). At this point, the cyclic feature contains information on the bond and atom types. In order to work with more generic cyclic features, all atoms with a coordination number of 4 or less are converted to carbon atoms. Since hypervalent carbon produces RDKit errors, hypervalent atoms are le unchanged. Contrary to the connectivity features that include all atom types and are based on the whole datasets, the GCF lists have been built on the CNOFS chemical subspace of ChEMBL (GCF1) or ChEMBL and ZINC (GCF2). Therefore, the only hypervalent case corresponds to sulphur with a coordination of 5 or 6. To avoid unstable fused small cycles with double and triple bonds, we have decided to keep the bond type information (Fig.
4c ). During this procedure, SMILES writing cleaning steps are performed to produce clean cycles. The GCF generation program and lists are available on GitHub at the following address https://github. com/ BenoitDamota/gcf. There are 11 013 generic cyclic features in the ChEMBL subset that is formed by only H, C, N, O, F, and S atoms. Only eleven of those represent more than 1% of the total number of cycles, see Fig. 5. In fact, the distribution is very uneven with the 6-membered aromatic ring accounting for more than 44% of the ring features, followed by the 5-membered rings (10%) that are probably heteroaromatic. The top trio is completed by a 6membered ring without unsaturation (10%). Although our method of generation is somewhat different we nd similar results to a previous analysis of CAS scaffolds.
Molecule generation The use of whitelists based on connectivity and cycle properties is not limited to a single method of molecule generation. The available chemical space is however dependent on this method. We used here EvoMol, an evolutionary algorithm based on RDKit graph objects. 21  EvoMol is available at https:// github. com/jules-leguy/EvoMol. In EvoMol, hydrogen atoms are treated implicitly and the bond orders are integers. The valence of the atoms is used as a reference to place the Fig. 3  The tetrathiafulvalene molecule used in organic electronic materials does not pass filter 1. To the right, the missing ECFP in ChEMBL. hydrogen atoms. The generator does not generate radicals and charged atoms. Ions and zwitterions have therefore been le out of the datasets during the chemical space comparison part.
The action space of EvoMol is also set with a limit on the heavy atoms count (Z > 1) and with the list of allowed chemical elements. For the rst objective of the QED optimisation, the search space is set to contain molecules with up to 38 heavy atoms among C, N, O, F, P, S, Cl, and Br. The population size is set to 1000 and the optimisation is run for 1000 steps. At each optimisation step, 10 individuals are replaced. One mutation consists in applying up to two operations on the molecular graph. For the rediscovery objective of the reference datasets, the limit on the HAC has been set to 9 (for QM9 W PC9) and then 10 heavy atoms for the other reference datasets. H, C, N, O, F, and S atoms form the chemical elements list of our chemical space search. At each optimisation step, 10 random individuals are replaced.
7. Visually the obtained solutions are overall much more pleasant. The combinations of hetero-elements seem more reasonable. However, we observe that there is some assembly of cycles of very different sizes that seem peculiar. This is due to the fact that the ECFP does not encode the cycle size information. For example, in the rst row, there is a molecule with a 5membered ring fused with a 3-membered ring which includes a double bond. This combination is highly constrained and probably not very stable. In order to ll this gap in the connectivity features, we propose to add a second ltering, for the moment a posteriori, based on the cyclic features present in ChEMBL for GCF1. That means that a generic cyclic feature score (GCFscore) will be 1 only if all contained GCF exist in ChEMBL. Keep in mind that this whitelist of cyclic features was set to contain information concerning double and triple bonds since there are saturated fused rings of many kinds.
On the 20 experiments, only 9 pass the cyclic feature ltering with a score of 1, highlighted in red. We can observe three different molecules with a top score of 0. 948. Interestingly, a derivative of commercially available scaffolds has been found twice, the In the 9 solutions that pass the lter of cyclic features, we can also notice the presence of heterocycles of sizes 8, 11, and 14. Indeed, we nd in the ChEMBL database heterocyclic derivatives of large sizes such as oxacyclooctane (PubChem CID 12677196, CHEMBL148748) and oxacycloundecane (PubChem CID 20080726). The compound 1-oxa-4-azacyclooctane-3,8dione is commercial (PubChem CID 55299436). The advantage of the ltering method presented here is that it is possible to trace the connectivity or cycle properties of the proposed solutions and then nd these properties in the reference datasets.
QM9 is a subset of molecules of the GDB enumerations. 30  QM9 was built from a constrained combinatorial approach while PC9 is a subset taken from PubChem data with the same number of heavy atom limits and chemical element constraints. Once the radicals, the stereochemical information, and the duplicates are removed, the union of these two sets contains 190 300 different molecules. Depending on the lter used, it can be seen from Table  1  that the number of molecules that contain only known connectivity features in ChEMBL25 (column passing lter 1) or even the ChEMBL25 and ZINC20 (column passing lter 2) is a minority of this set of molecules. The impact of adding the ZINC connectivity features is evident. For other combinatorial reference datasets like GDBChEMBL and GDB11, the ltering is also quite drastic removing at least 84% of the molecules while accepting all the ChEMBL25 and ZINC20 connectivity features.
Starting from only a methane molecule, the generator randomly mutates the molecules of its population and if the solution belongs to the QM9 W PC9 dataset, the score for this solution is 1 and if not, it is 0. It is thus a pure enumeration study based on random actions on randomly chosen individuals. The important thing here is to check whether the limitation to certain connectivity features as dened in the ECFP2 and ECFP4 still allows navigation in chemical space. For each heavy atom limit and each lter, ten experiments were performed. We also tested the impact of the number of actions on the molecular graphs for each mutation. As an example, for a depth of four actions, the generator can chain up to four operations such as adding or removing an atom (see the Methodology section) before ltering.
In view of the generation exibility of EvoMol, we took the opportunity to also test the impact of adding a new chemical element, sulphur, onto the search space. In Table  1 , the numbers of unique molecules in the reference datasets and in the enumerations are reported depending on the heavy atoms allowed and the lter. Even considering lter 2 which presents more connectivity features, the expansion induced by the addition of the sulphur atom is not overwhelming. If we look at the numbers in ChEMBL and ZINC, we can expect that adding sulphur represents a size increase of around 45% of the chemical space. If this increase seems important, it can be put in perspective with the huge combinatorial that exists without lters. An enumeration by EvoMol without constraints other than the valence rules managed to generate 5433 different SMILES with C, N, O, and F having HAC 5 or less.
Adding sulphur allows us to generate 139 689 SMILES with the same size limit. So, lters on connectivity features have a clear and major impact on the denition of the search space and strongly limit exotic combinations. Rather condent in our ability to enumerate a realistic chemical space of reasonable size, we transformed the objective of random generation into a systematic enumeration of all neighbouring solutions of an initial dataset. With a limit of HAC of 10 and a maximum of 2 actions on the molecular graph, it takes a few days of computation to go over the 600 thousand or so molecules already generated. However, increasing the number of actions to 3 multiplies the combinatorial by several orders of magnitude and makes this approach unreasonable even for datasets of small molecules.
All the molecules generated under constraints (either from lter 1 or lter 2) with the two lists of atoms considered, CNOF and CNOFS, were gathered in a single set called Evo10. We propose here to take advantage of this rather large enumeration of realistic chemical spaces associated with the connectivity properties of ChEMBL and ZINC and to compare it to the same size molecules of reference datasets listed in Table  1. The cumulative evolution of the number of SMILES as a function of the HAC is plotted in Fig. 8  for the CNOF search space. Similar tendencies are observed for the CNOFS search space. We have also added the enumeration tests without lters (denoted as Evo10 no lter) and it can be seen that thanks to the lters the chemical space is several orders of magnitude smaller. The Evo10 dataset is larger than the ltered GDB11 which has the most molecules passing the lters.
It can also be seen that QM9 W PC9 contains only a small part of what is possible with realistic lters. Despite the impressive size of ZINC, it contains only a very small part of the chemical space dened by these connectivity features. A logarithmic regression based on Evo10 (lter 2 W GCF2) allows us to estimate a size of 5 × 10 9 molecules with CNOFS and a HAC limit of 17 and several 10 22 molecules with CNOFS and a HAC limit of 40 still excluding radicals, stereoisomers, ions, and zwitterions. With the connectivity lters, we managed to enumerate more compounds respecting the chemistry of either ChEMBL or ChEMBL plus ZINC. Yet, one could ask if we managed to rediscover every molecule up to HAC 10 present in these reference datasets. In Table  2 , the few SMILES that were not found have been reported.
Considering the molecules of the reference datasets that pass lter 2 in the CNOFS search space, EvoMol managed to recover all molecules of GDBChEMBL and GDB11. Only 8 molecules have not been found with two actions mutations. Two belong to ChEMBL and 6 to ZINC. The reported SMILES present a large number of heteroelements and correspond probably to connectivity features that are quite isolated. Table 2 SMILES of the filtered reference datasets that were not found during the EvoMol enumeration with 2 actions. The last column is the number of neighbours at 3 actions that belong to Evo10 SMILE Dataset Neighbours CC1]NOC(C)(O)C1]NO ChEMBL 103 O]NN(O)S(]O)(]O)O ChEMBL 20 CN1ON(C)ON(C)O1 QM9 W PC9, ZINC 4 c1coc2occoc]2o1 ZINC 65 CSC1N]NC(SC)N]N1 ZINC 10 S]c1[nH]ssc2nnc1]2 ZINC 8 N1]NC(]C2N]NN]N2)N]N1 ZINC 1 N1]S]NC2]C1N]S]N2 ZINC 0 If we enumerate the neighbours at three actions of the missing SMILES, we nd that only one of them cannot be mutated to another Evo10 molecule with 3 actions.
This peculiar bicycle is commercially available on demand. In the end, it is the only molecule of the ltered reference datasets that we would not nd with a mutation depth set at 3 actions. Searching for its neighbours at 4 actions on the molecular graph leads to 17 neighbours present in Evo10. However, just this search took several hours. It is fair to say that there are probably only very few chemical compounds isolated at more than 3 actions away from the rest of the ltered chemical space. Furthermore, for specic problems, it might be relevant to select an initial population that is tted for the task. Having demonstrated that lters reduce the search space but do not visibly create unreachable compounds, it would be interesting to study the chemical diversity associated with these lters.
We have already shown in the past that the chemistry of QM9 and PC9 is somewhat different and that some chemical functions are missing in QM9. 28  This is a problem for the generalizability of machine learning methods using QM9 as a training set. However, QM9, GDB11, and GDBChEMBL by their combinatorial construction present a more important diversity in specic associations like the combinations of alkenes with other functions. We can compare the chemical environments that exist in datasets with those that pass the lters. We can consider here that the number of connectivity features as dened in the ECFP2 is a measure of the small range of chemical environment diversity. The ECFP2 takes into account the central atom and its rst neighbours and their bond types. We have chosen the ECFP2 since the ECFP4 can include up to 53 atoms and can be very specic.
In Table  3 , the number of connectivity features before and aer ltering is reported. There are initially between 7 and 10 thousand different connectivity features as dened in the ECFP2. Most of them in QM9 W PC9, GDBChEMBL, and GBD11 do not belong to ChEMBL and or ZINC. Thanks to the EvoMol enumeration, we can also see in Table  3  that small sizes datasets cannot reproduce the full range of possible coordinations even at the scale of the ECFP2 since we just managed to generate half of the connectivity features of ChEMBL. Let us now discuss the impact on the chemical diversity of a posteriori ltering by cyclic features. If we look at the most common cyclic features of the QM9 dataset, see Fig. 9 , we can immediately notice a very different distribution from those of ChEMBL where the phenyl group was predominant (see Fig.
5 ). Table  1  shows that in terms of the number of molecules, ltering removes on average around 10% of the molecules passing the lters. However, if we look at the number of cyclic features in each of the datasets before and aer the lters, see Table  4 , we can see a drastic reduction in the number of cycles (column fx compared to column fx W GCFx). The 10% or so of molecules removed are associated with the majority of the GCF. The qualitative results on the QED indicated that it was easy for an evolutionary algorithm to construct exotic tangles of nested cycles while respecting the connectivity lters. Again, the correct denition of extensive whitelists can be discussed. But it should be noted that GCF2 includes all the cyclic features of hundreds of millions of known compounds. In total, we have counted 15 431 cyclic features.
Two-thirds of them concern more than 17 atoms. Small molecule datasets will not be able to represent the topological diversity of ChEMBL and ZINC. To conclude the study of the chemical diversity passing the lters, we have calculated the SAscore, CLscore, and RAscore for all molecules of the Evo10 dataset according to the chosen lter. In Fig. 10 , the comparative distributions between these two scores for the lters allowing the most compounds to pass, i. e. lter 2 and GCF2, show a concentrated area around 2. 5 to 4. 5 in the SAscore and 2. 5 and 4 in the CLscore. A low SAscore should denote better synthesisability and in the SYBA article, the authors propose a threshold value of 4. 4 in the SAscore above which the molecule would be too complex to be synthesised. 13, 23  According to these approaches, the vast majority of Evo10 could become reality.
In fact, these distributions resemble those of ChEMBL and ZINC. 22  Evo10 only contains connectivity and cyclic features existing in these datasets while the two scores favour the most popular environments. Thus, there is a trail of points with a high CLscore and a low SAscore as expected. A past optimisation of these scores showed us that this corresponds to simple molecules consisting only of alkyl and aryl environments with very few cycles. 21  The RAscore is a score between 0 and 1 based on a neural network designed as a pre-screening tool to avoid a retrosynthetic analysis of all generated compounds. 17  Values near 1 should indicate a compound possessing a synthetic route. The RAscore proportions for all compounds of Evo10 are reported in Table  5. The vast majority of the compounds in this dataset have a score greater than or equal to 0.
99. The amount of molecules that the classier considers without any synthetic route is very low, below 1%. To be fair, it should be noted here that the descriptors used to dene the RAscore in the neural network are ECFP6 and that it was trained on a portion of ChEMBL. The two approaches are different but based on similar descriptors and the reference dataset. It is therefore not very surprising that the RAscore rates favourably a large proportion of Evo10.
Conclusions When we consider several types of heavy atoms, such as the simple set of C, N, O, F, and S, a phenomenal amount of combinations are then possible while respecting the valence rules. Many of these combinations contain very exotic associations. It is fairly accepted that scores can be used to select more realistic molecules and some of these scores are based on connectivity descriptors such as ECFP4. We present in this work new descriptors, called generic cyclic features (GCF), to complete the local information of the ECFP4 with the cyclic substructures of a molecule. For both ECFP4 and GCF, we have chosen to limit the possible combinations to those that exist in two datasets based on real molecules, ChEMBL ZINC. These two full datasets encompass 1 156 416 connectivity features (based on ECFP4).
And on a ltered subset (see the Methodology section) composed of 288 000 000 molecules of the ChEMBL and ZINC, 15 431 cyclic features were extracted. With these whitelists, it is possible to evaluate the amount of features of a molecule belonging to ChEMBL, or ChEMBL and ZINC. It is not a matter of assessing whether on average a molecule resembles the most common in these databases, but whether each piece that constitutes it exists, even rarely. This is how we have dened a realistic molecule. We have implemented this approach in our evolutionary generator EvoMol, but we believe that all generation methods can benet from these whitelists. The chemical space of realistic molecules is much smaller than the set of possible combinations. It still contains an estimated amount of more than 10 22 molecules with at least 40 heavy atoms.
We were able to enumerate starting from methane more than 676 000 molecules having up to 10 heavy atoms of the set C, N, O, F, and S, and only having connectivities and cyclic features known in ChEMBL and ZINC. We were able to navigate through the chemical space of realistic molecules and indeed rediscover all molecules passing these same lters from the reference datasets which are here ChEMBL, ZINC, QM9, PC9, GDB11, and GDBChEMBL. It cannot be said that all synthesizable molecules are represented in this set. This is a common limit of whitelistbased ltering. The list needs to be updated if features that exist in reality happen to have been overlooked. However, our work is based on a very large sample of the known chemical space. It is especially the visualisation of the proposed solutions aer ltering that convinces us of the relevance of this approach.
Acknowledgements This work was supported by the  University of Angers , and the  French Ministry of Education and Research  (JL PhD grant). The calculation resources were provided by the  LERIA  and  MOLTECH-Anjou laboratories. The authors would therefore like to thank  Jean-Mathieu Chantrein  for his technical help. TC and BDM would especially like to thank the  BOINC  community involved in our project, called QuChemPedIA@home (https://.
Data availability The code for the molecular generator EvoMol can be found at https://github. com/jules-leguy/EvoMol. The version of the code employed for this study is version 1. 4. 1. The code for the general cyclic feature scoring program GCF can be found at https:// github. com/BenoitDamota/gcf. Data as JSON dictionaries for this paper are available as a collection in gshare at https:// doi. org/10. 6084/m9. gshare. c. 6041117.
Introduction 1. Statistical modeling of spatial extremes The availability of increasingly detailed spatial and spatio-temporal datasets has motivated a recent surge in methodological developments to model such data. In this work, we are concerned with modeling extreme values of spatial or spatio-temporal processes, which we denote by {Y (s) : s ∈ S ⊆ R 2 } and {Y (s, t) : (s, t) ∈ S × T ⊆ R 2 × R + }. The goal of modeling spatio-temporal extremes is often to enable extrapolation from observed extreme values to future, more intense episodes, and consequently requires careful selection of models suited to this delicate task. Early work on spatial extremes focused almost exclusively on max-stable processes  (Smith, 1990; Coles, 1993; Schlather, 2002; Padoan et al. , 2010; Davison and Gholamrezaee, 2012).