dataesr/scientific-paragraphs-categorization huggingface dataset:
Since there is a considerable reduction in the number of relevant instances after the classification step, this step alters the ratio between the relevant instances and the retrieved instances. Thus, not only in F1 but also a significant drop in precision is observed across all the models, when compared with the ground-truth results. The drop in precision is due to a number of relevant documents being discarded by the classifier. Since our best results were not obtained after applying document classification for the relevant article detection, we consider that our best models are those applied on the initial dataset comprised of relevant and irrelevant documents. Thus, we continue by presenting the performance of these models for each language in the dataset. As shown in Table  3 , BERT-multilingual-uncased obtained the highest scores for three out of the four low-resource languages, while BERT-multilingual-cased was more fitted for Polish.
Model-wise Analysis As demonstrated in the results, different models perform differently on different datasets. Thus, we move beyond the holistic score assessment (entity F1-score) and compare the strengths and weaknesses of the models at a fine-grained level. We analyzed the individual performance of the models and the intersections of their predicted outputs by visualizing them in several UpSet plots  7. As seen in Figure  3  (a), there are approximately 70 positive instances that none of the systems was able to find. The highest intersection, approximately 340 instances, represents the true positives found by the three systems. BERT-multilingualcased was able to find a higher number of unique true positive instances, instances not detected by the other models. BERT-multilingual-uncased had the highest number of true positive instances cumulatively, the second-highest number of unique true positives, and the lowest number of false positive instances.
This reveals the ability of the BERTmultilingual-uncased model to find the relevant instances in the dataset and to correctly predict a large proportion of the relevant data points, thus the high recall and precision, and overall F1 performance. The overall performance is, generally, affected by the equally higher number of false positive and false negative results, as presented in Figures  3 (b, c ). XLM-RoBERTa-base recorded the highest false negative rate and the lowest number of true positive instances, which explains the low recall and F1 scores.
Attribute-wise Analysis We chose to utilize an evaluation framework for interpretable evaluation for the named entity recognition (NER) task  [16]  that proposes a fine-grained analysis of entity attributes and their impact on the overall performance of the information extraction systems  8. We conduct an attribute-wise analysis that compares how different attributes affect performance on the DAnIEL dataset, (e. g. , how entity or sentence length correlates with performance). The entity attributes considered are entity length (eLen), sentence length (sLen), entity frequency (eFreq), token frequency (tFreq), out-of-vocabulary density (oDen), entity density (eDen) and label consistency. The label consistency describes the degree of label agreement of an entity on the training set. We consider both entity and token label consistencies, denoted as eCon and tCon.
eCon represents the number of entities in a sentence. To perform the attribute-wise analysis, bucketing is applied, a process that breaks down the performance into different categories  [16, 17, 33]. The process involves partitioning the attribute values into m = 4 discrete parts, whose intervals were obtained by dividing the test entities equally, with in some cases the interval method being customized depending on the individual characteristics of each attribute  [16]. For example, in the entity length (eLen), entities in the test set with lengths of {1, 2, 3} and > 4 are partitioned into four buckets corresponding to the lengths. Once the buckets are generated, we calculate the F1 score with respect to the entities of each bucket. The results in Table  4  illustrate that for our dataset the performance of all models varies considerably and it is highly correlated with oDen, eCon, tCon, and eLen.
Semi-supervised Learning Experiments Due to the limited availability of annotated datasets in epidemic event extraction, we employ the self-training semi-supervised learning technique in order to analyze whether our dataset and models could benefit from having relevant unannotated documents. We then experiment with the mean teacher (MT) training method, a semi-supervised learning method where a target-generating teacher model is updated using the exponential moving average weights of the student model  [41]. As such, the approach can handle a wide range of noisy input such as digitized documents, which often are susceptible to optical character recognition (OCR) errors  [45, 32, 36]. First, we consider the documents in four low-resource languages from the DAnIEL dataset: Greek, Chinese, Russian, and Polish.
These languages are  around 80% less represented in this dataset when compared to French. For these experiments, we simulate several few-shot scenarios by implementing a strategy in which we split our training data into annotated and unannotated sets, starting from 20%, and increasing iteratively by 10 percentage points until 50%. Thus, we obtain four few-shot learning scenarios as detailed in Table  5. For example, in the 20% scenario, 20% of the data is considered annotated and 80% unannotated. The process of self-training, as presented in Figure  4 , has the following steps: -Step 1: Each of our models will be, at first, the teacher, trained and finetuned on the event extraction task using a cross-entropy loss and a small percentage of the DAnIEL dataset (i. e. , we keep 20% for training).
For the latter, we fine-tune all our models on between 800-3000 sentences of training data for each language (as shown in Table  5 ) and use it as a teacher model. Larger improvements in performance were noticed in the case of the XLM-roBERTa-base model, where self-training leads to 2. 29% average gains on Greek (from 67. 99% to 69. 55%), 4. 19% on Polish (from 71. 59% to 74. 59%), and on Russian, 3. 21% (from 46. 73% to 48. 23% while remaining unchanged for Chinese. In the majority of the cases and for all the models, the performance improvements can also be due to the fact that, because of the few-shot scenarios that are created from our initial dataset, the simulated unannotated data remains in-domain with the labeled data. It was proven that using biomedical papers for a downstream named entity recognition (NER) biomedical task considerably improves the performance of NER compared to using unannotated news articles  [20].
Catchments dataset A total of 685 catchments are used in this study: 527 French catchments and 158 Swedish catchments. In France, discharge series (Q) were extracted at a daily time step from the French Hydro database  (Leleu et al. , 2014). Only catchments that are not regulated (according to the classification within this database) were selected. Corresponding catchment areas vary between 5 km 2 and 26 900 km 2. Precipitation (P ) and temperature (T ) data were extracted from the SAFRAN atmospheric reanalysis produced by Météo-France (8 × 8 km and averaged over the catchment upstream areas;  Vidal et al. , 2010). In Sweden, discharge series were extracted at a daily time step from the SMHI database. Only stations having less than 5 % of their area regulated are used (information extracted from https://www.
smhi. se/, last access: 26 June 2019). The degree of regulation is the percentage of the volume of the mean annual flow that can be stored in reservoirs located upstream of the gauged catchment outlet. Catchment areas vary between 1 and 14 400 km 2. Precipitation and temperature data were extracted at a daily time step from the PTHBV database (4 × 4 km;  Johansson, 2002). For all stations, potential evaporation (E 0 ) was estimated from daily mean air temperature and daily extraterrestrial radiation following  Oudin et al. (2005). Daily hydroclimatic data were aggregated on an annual scale for the purposes of this study, with the hydrological year starting on 1 October. By defining the start of the year in this way, rather than by a calendar year, we aim to minimize a water volume that could be carried over 2 calendar years.
01), to decide whether Eq. (  2 ) improves significantly on Eq. (1). If the improvement is not significant, we conclude on the absence of multi-year memory for that catchment, and keep the simplest representation (that of Eq. 1). The objective function is a root mean square error (RMSE) of the Q/P anomalies. It is used for the optimization of the parameter values and for this model selection. Parameters of both equations were calibrated using particle swarm optimization through the "hydroPSO" R package (Zambrano-Bigiarini and Rojas, 2013).
Identification of hydroclimatic drivers of catchment memory Based on the literature review of Sect. 1. 5, we identified a few hydroclimatic descriptors that can explain catchment memory. Each catchment will be described with the average of the annual values of Q and P , E 0 , Q/P and P /E 0. In addition, the percentage of the catchment area covered by lakes is used as a supplementary descriptor. This information is provided by SMHI for Swedish catchments (extracted from https://www. smhi. se/data/hydrologi/ vattenwebb, last access: 26 June 2019). For France, the Lake Water Bodies according to the Water Framework Directive is used (extracted from https://geo. data. gouv. fr/fr/datasets/ cc23e393dc05180892f2bf04f0be0423b62ebd86, last access: 25 May 2021). Finally, the contribution of groundwater is assessed by a baseflow index (BFI) calculated according to the work of  Pelletier and Andréassian (2020a)  where the baseflow is estimated from the outflow of a quadratic reservoir.
The approach has two parameters (calibrated for each catchment): the reservoir capacity and the duration over which past effective precipitations filling this reservoir are taken into account. This baseflow filtering was performed with the associated "baseflow" R package  (Pelletier et al. , 2021). The statistical distribution of each of these descriptors will be compared among three catchment memory classes: no multi-year memory, short multi-year memory and long multiyear memory. The first class corresponds to catchments for which the Student's t test (from Sect. 2. 3) did not reveal a significant multi-year memory. The remaining catchments are split into two groups according to the median value of their memory in order to distinguish catchments with short multiyear memory and catchments with long multi-year memory.
1. 5. This could, for instance, be geomorphology, geology, vegetation, soil type and depth, aquifer interactions, groundwater levels, and human modifications (tile drainage, land degradation, reservoirs). Author contributions. AdL, VA and LC designed the study and performed the analysis. GL and BA provided the Swedish data set and analyzed the results in Sweden. AL wrote the paper, and all authors provided input on the paper for revision before submission. Competing interests. The contact author has declared that neither they nor their co-authors has any competing interests. Disclaimer. Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Acknowledgements. The authors wish to thank the associate editor,  Bettina Schaefli , and the three anonymous reviewers, for their constructive comments, which helped to improve the overall quality of the article. The authors also gratefully acknowledge the reviews of  Gaëlle Tallec ,  Charles Perrin  and  Antoine Pelletier. Financial support. This research has been supported by the project  AQUACLEW , which is part of  ERA4CS , an  ERA-NET  initiated by  JPI Climate , and funded by  FORMAS (SE) ,  DLR (DE) ,  BMWFW  (AT), IFD (DK),  MINECO (ES) ,  ANR  (FR) with co-funding by the  European Commission  (grant no  690462 ). Review statement. This paper was edited by  Bettina Schaefli  and reviewed by three anonymous referees.
Introduction The central dogma of molecular biology implies the transfer of the information contained within genes into the corresponding proteins. Although the amino acid sequence of a protein is important, the three dimensional structure is even more crucial for a protein to fulfill its cellular and biochemical functions. Indeed, misfolding or aggregation due to point mutations or other causes are known to be responsible for many pathologies including neurodegenerative disorders such as Alzheimer's disease  (Dobson, 2003 , Forman et al. , 2004). Hence, since the resolution of the three dimensional structure of myoglobin  (Kendrew et al. , 1958) , the first one to be determined, extensive efforts have been devoted to solve the structures of proteins from various organisms. As a consequence, almost 180,000 structures are currently deposited at the Protein Data Bank, a 50-years old database  (Berman et al.
Consequently, extensive efforts have been devoted for many decades to the development of protein structure prediction approaches either by template-based or template-free modeling. Improvements were constantly realized but very recently, a breakthrough step has been made as testified by the results of the recent Critical Assessment of protein Structure Prediction (CASP14) experiment  (Pearce & Zhang, 2021). Indeed, the accuracy of the models obtained by the machine learning method AlphaFold developed by the DeepMind company was impressive compared to those obtained by other methods  (Jumper et al. , 2021 , Lupas et al. , 2021 , Pereira et al. , 2021). This inspired the improvement of the RoseTTaFold program by the Baker's lab  (Baek et al. , 2021). The greatly improved accuracy of the structure predictions generated by AlphaFold and RoseTTAFold then offers the promise that many of these in silico three dimensional models can be good to excellent templates for researchers to conduct further studies.
Crystallization, data collection and processing Crystallization trials were performed by mixing 150 nL of protein with an equal volume of different crystallization solutions in 96-well TTP Labtech plates at 7°C using the Mosquito automate (TTP Labtech). Prior to X-ray exposure, the crystals were transferred into the crystallization solution supplemented with 20% ethylene glycol and 20% glycerol and flash-cooled in liquid-nitrogen. The data were collected at 100 K on Proxima-2a beamline (Synchrotron SOLEIL, Saint-Aubin, France). Several datasets collected from a single crystal were processed using the XDS program, merged and scaled using the XSCALE program  (Kabsch, 1993).
6, 0. 9-1 M Li2SO4 and 0. 6 M ammonium sulfate, within one day at 7°C. These crystals appeared within 6 days and grew for at least two more weeks to reach 200 µm length (Fig. 1B ). They diffracted on the Proxima-2a beamline from the French synchrotron SOLEIL in 2019  (Duran et al. , 2013). Taking advantage of their size and of this microfocus beamline, we collected several datasets from the same crystal. By merging three datasets together, we obtained a 2. 45 Å resolution dataset (Table  1 ). These crystals belong to the P321 space group or its related enantiomorphs (P3121 or P3221) with an estimated number of two KlNmd4 molecules per asymmetric unit assuming a 50% solvent content  (Matthews, 1968).
, 2018) , RosettaCM  (Song et al. , 2013)  and i-Tasser  (Yang et al. , 2015). For PHYRE-2 model, we selected the one generated using the crystal structure of the PIN domain from human SMG6 (PDB code 2HWW;  (Glavan et al. , 2006) ) as template (predicted sequence identity with KlNmd4 of 22%). The Swiss-Model server generated a single model from another crystal structure of the same protein domain (PDB code 2DOK;  (Takeshita et al. , 2007) ). The i-Tasser server used both structures to generate five different models while we selected the ab initio mode for the RosettaCM server to also generate five different models (when several models have been generated using the same software, there are annotated from a to e). More recently, we generated one and five additional models using the AlphaFold and RoseTTAFold softwares, respectively  (Baek et al.
-MOLREP To analyze the results for the different molecular replacement trials performed with MOLREP, we mostly focused on the contrast value calculated by the program as suggested in the tutorial. Basically, this value reflects the difference between the highest score and the mean score of the different solutions obtained after the translation fonction search has completed and the higher this score the more likely the solution is to be correct. In general, contrast values higher than 3 are strong indication that the proposed solution is correct. The distribution of the contrast values proposed by the MOLREP program for the different solutions obtained from the various models, clearly shows that the AlphaFold model largely outcompetes the other models and successfully led to the correct positioning of both KlNmd4 monomers (Fig.
2A ). However, the scores for these solutions were low and similar to the scores of incorrectly positioned monomers. This later observation clearly shows that although some monomers can be correctly positioned, it is difficult to identify them among all the wrongly positioned models. Finally, for those models where one molecule was correctly positioned, if we use 100 rotation functions in the translation function, i. e. more than when we use the default parameters defined in the MOLREP program, the correct position of the second monomer could only be found for the RoseTTAFold-b and RoseTTAFold-c models. Next, we refined the coordinates of the solutions proposed for these different molecular replacement trials using the BUSTER program and analyzed the R and Rfree values obtained for all these solutions (Fig.
2B ). Once again, the solution obtained with the AlphaFold model outcompetes the other ones as testified by the much lower R and Rfree values (37% and 40. 8%, respectively) compared to the solutions obtained with RoseTTAFold-e (R and Rfree values of 47. 9% and 51. 7%, respectively) and RoseTTAFold-a models (R and Rfree values of 51. 7% and 54. 3%, respectively). As expected, the other solutions, i. e. with one or two monomers incorrectly positioned, yielded significantly higher R and Rfree values (Fig. 2B ), typical to those obtained for incorrect solutions. -PHASER In parallel, we performed molecular replacement trials using the PHASER program and each of these 18 models. In that case, we analyzed the results by monitoring the TF Z-scores obtained for each solution. Indeed, it is considered that for the first positioned molecule, TF Z-score higher than 8 are indicative of correct solutions while TF Z-scores in between 6 and 8 indicate probable correct solutions  (Phaserwiki).
In our case, the solutions obtained using the AlphaFold model as well as the five models from RoseTTAFold yielded solutions with TF Z-scores higher than 10 for the first monomer and similar or higher for the second monomer (Fig. 2C ). This indicates that for each of these 6 models, two copies were correctly positioned. On the contrary, the statistics of the solutions obtained with any of the other models suggested incorrect solutions and this was confirmed by the theta, phi and chi angles of the selected rotation functions, which significantly differ from those of the correct solutions. The refinement of these different solutions with the BUSTER program revealed important gaps in the R and Rfree values between these different solutions (Fig. 2D ), which can then be divided into 3 groups. The first one with mean R and Rfree values of 68.
This is in line with the outstanding results obtained by AlphaFold during the CASP14 session  (Millan et al. , 2021 , Pearce & Zhang, 2021 , Pereira et al. , 2021). We managed to solve the structure of a small protein and to obtain 2Fo-Fc electron density maps of excellent quality within less than an hour while we were struggling for almost two years since the obtention of good quality diffraction datasets on this project. From discussions with other colleagues, we know that several other structural biologists have also succeeded in solving reluctant crystal structures thanks to the AlphaFold model of their favorite proteins. Other examples describing these successes will undoubtedly be published in the near future. Our simple routine can be easily automated as done using different molecular replacement and refinement programs in the CaspR server or MrBUMP program for instance  (Claude et al.
, 2004 , Keegan & Winn, 2008). As many other structural biology colleagues  (Cramer, 2021 , Perrakis & Sixma, 2021 , Thornton et al. , 2021) , we are convinced that this achievement will definitely revolutionize structural biology. It is then very likely that the future protein and multi-protein complexes crystal structures will most largely be determined by molecular replacement and that methods such as MIR/SIR using heavy metal derivatives, SAD/MAD using selenomethionine-labelled proteins will become more and more marginal or applied to specific cases such as low resolution data. This incredible breakthrough in the accuracy of predicted three dimensional structures opens amazing perspectives for biology in general. Indeed, thanks to the combined action of DeepMind and EMBL-EBI, every biologist has now access to the AlphaFold Protein Structure Database (https://alphafold.
ebi. ac. uk/) presenting the models of the almost complete proteomes from 21 prokaryotic and eukaryotic model organisms including the Methanocaldococcus jannaschii archeon, some bacteria (Escherichia coli, Mycobacterium tuberculosis…), Saccharomyces cerevisiae, Caenorhabditis elegans and of course Homo sapiens  (Tunyasuvunakool et al. , 2021 , Varadi et al. , 2021). No doubt that this ressource is already fueling experiments in many biology labs. Very importantly, this database gives easy access to the confidence score (pLDDT) for each amino acid in the predicted structure. In our case, there is a strong agreement between these pLDDT values and the similarity between the AlphaFold model and our experimental structure (Fig. 4B ). Among the regions from the KlNmd4 AlphaFold model with pLDDT values lower than 90, some adopt the same conformation as in the crystal structure, while others do not.
Acknowledgements We are grateful to the  SOLEIL Synchrotron (France)  staff (proposal numbers 20181001 and  20201046 ), in particular  Martin Savko  and  Serena Sirigu  (Proxima-2a beamline) for smoothly running the facility. MG acknowledges financial supports from the  Centre National pour la Recherche Scientifique (CNRS) , the  Agence Nationale pour la Recherche (ANR ;  ANR-18-CE11-0003-04 ) and  Ecole Polytechnique. IBB is supported by a PhD fellowship from the  French Ministère de l'Enseignement Supérieur et de la Recherche (MESR).
Here the context is x Þ Ñ vr and the step t1 means pλx. xq vr " vr. The step is proven by congruence (rule cong). Then congruence is applied again (step t2) to prove that pλx. f xq vr " f vr and step t3 concludes the renaming. During proof search each module of veriT appends steps onto a list. Once the proof is completed, veriT performs some cleanup before printing the proof. First, a pruning phase removes branches of the proof not connected to the root K. Second, a merge phase removes the duplicated steps. The final pass prepares the data structures for the optional term sharing via name annotations.