dataesr/scientific-paragraphs-categorization huggingface dataset:
Enhancement and optimisation of a dense cloud Depending on the number of images used in a photogrammetric project, the size of the point cloud can vary greatly. For projects with a lot of images (> 20 000), both UAV and terrestrial, the cloud can have around 1 billion points, which is difficult to display for standard computers, but especially difficult to process for any type of computer. It is then necessary to think about an optimisation method because the density of these clouds is too high for our needs. Cloud filtering has developed a lot in recent years, especially in the field of robotics and due to the increase in the size of the objects to be processed to have smaller datasets, to allow the acceleration of real-time calculations  (Leal et al. , 2017). Whatever algorithm is used, it will consider either a physical or statistical characteristic to reduce the number of points.
Most generally the reduction is done according to the neighbouring points except for the case of a random reduction  (Han et al. , 2017). For our project and in view of the different existing approaches, we have chosen to use the topological approach In this case, the reduction algorithms consider the position in space of the points in relation to each other when deciding whether to delete a point. These methods are not the most efficient for keeping the geometric information of the clouds, but they are still easily accessible and allow for simple density reduction without considering other more complex parameters. The CloudCompare software uses a topology-based approach in its spatial filter that searches for each point within a user-defined perimeter to remove. Indeed, in addition to being programmable, which is relevant for our automatic processing chain, the software is open source, and we know how the algorithm works.
White balance correction: The notion of balance is essential for the consistency of the products resulting from the various computations. This result is far from uniform and needs to be improved. After this project-specific study, we decided to use the Retinex algorithm, which proved to be the most suitable for our situation, especially for scenes with natural light. To apply this correction to all our 86,000 images, we developed a Python script. This script allows us to apply a correction to each image using both CPU and GPU at the same time without loss of metadata (EXIF). In addition, the process can correct approximately 3 images per second, making it a powerful script.
In its operation, Metashape can consider in the calculation process the coordinates of the camera centres as well as their rotation parameters. As it is a new generation software, it is possible to give it images with and without its parameters, which allows to add new images over time, but also to consider those that could not be aligned in the past. Our strategy is based on two main principles and allows us to considerably reduce the need for RAM, the calculation time, and the alignment errors; these are the georeferencing of the project and the calculation by tile. The aim is to pre-compute small projects, the number of images of which depends on the computer power at our disposal, and then to reuse the approximated external orientation parameters as source parameters in the final project.
Improvement and correction of 3D models in Metashape: Now that we can align large projects efficiently, we can easily create 3D models using the depth maps. Indeed, the algorithm will first compute a dense point cloud, then keep only robust points visible from at least five maps. Furthermore, if images are slightly misaligned, errors or multiple triangle layers may be found in the same location. Apart from the cleaning issues, there are also issues related to the size of the model and the number of faces. At the beginning of the project, we did not have any criteria for the number of faces per square meter. So, we gave the software a quality between low, medium and high. This black box operation provides us with models with several faces that is too high for our needs. The study of the different model reduction algorithms presented in paragraph 2.
Cleaning of a model: In general, cleaning a 3D model in Metashape is an action that can be easily performed using the various tools at our disposal, such as manual or automatic selection according to different criteria. The software is perfectly optimised to navigate within a model even if it has more than 100 million faces. This is a real advantage, as other less optimised software can make the experience more difficult. However, despite all its advantages, Metashape does not offer a "clipping box" that allows you to view and modify only a part of a model without influencing the rest. This lack can make the cleaning process longer, especially for details such as excavations. In our case, we chose to apply a pyramidal cleaning method, starting with a global cleaning and ending with a more detailed local cleaning.
Optimisation using external and Metashape-specific functions: The cleaning of the model is only one step in obtaining a final model. Indeed, once the cleaning is done, it is necessary to optimise the model either for the number of faces or for the topology. Metashape natively offers two very useful functions for topology correction: "Close Holes" and "Fix Topology". The first one allows to close the holes created during the cleaning phase; the software analyses the model and according to a value of percentage of size of the holes that we indicate, it closes them. The second function allows the topology of the model to be corrected, i. e. duplicate faces and edges to be removed, normals to be fixed and edges to be fixed. It is important to apply this function last to correct the final model.
Reducing the number of faces in the model can be done natively in Metashape. The problem with this approach is that we do not know the algorithms behind the function. We have therefore chosen to study different types of reduction algorithms seen in paragraph 2. 3. We have therefore developed a Python script allowing us to apply this filter to a model using the MeshLab python library. To determine the number of faces acceptable for our project, we carried out various tests in collaboration with the archaeologist in charge of the site. The objective was to determine the minimum number of faces necessary to ensure a good accuracy of the model for video and orthophotos. This requires: -6,000 faces per m2 for a model for normal use (to create textures, videos, orthophotos) -10,000 faces per m2 for a model representing excavations.
Production of true orthomosaics using Metashape: The Metashape software, which we are using for this project, offers the creation of orthomosaics from digital terrain models or 3D models. Using the first option produces a classical orthophoto without vertical correction. This option is ideal for purely aerial shots, but not for our complex site geometry. In our case, we use a 3D model as a basis to produce fully corrected true orthophotos.
Reducing overlap There are two ways of reducing the overlap and thus limiting the number of images for our project: the arbitrary method, without considering the calculated orientation of the images, and the a posteriori method, which considers the alignment performed. For our project, we chose to use the second approach. To carry out this approach, we have developed a method based on the overlap reduction algorithm proposed by Metashape natively. In addition, we have developed a Python script that allows us to apply this method to all the photo files automatically. Reduction values of around 60% can be achieved for some spaces. However, high values mean too much reduction and should never be reached. Despite the advantages of this algorithm, there are some limitations to its use. Overlay reduction is useful if all the images in the file are of the same quality and if you do not want to create complex or detailed models.
The fact that our projects are georeferenced makes it easy to make a geometric link between the different polygons, as the coordinates of their vertices will always be in the same system. Metashape provides a Python script to automatically create the footprint of each image aligned to the model as a polygon. The optimisation of the script consisted in implementing a multicore calculation adapted to our computer. The creation of the boundary polygons of each space is done manually, as they are set up by the archaeologist and ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume X-M-1-2023 29th CIPA Symposium "Documenting, Understanding, Preserving Cultural Heritage: Humanities and Digital Technologies for Shaping the Future", 25-30 June 2023, Florence, Italy depend on archaeological criteria.
Indeed, there is no information on the model that the automatic creation of these polygons. It was therefore necessary to create each of the 162 rooms boundary polygons by hand, taking care to ensure that each polygon slightly overlaps its neighbours. The use of the Python package shapely allows us to use the boolean function intersects() which tests the intersection between two polygons. We then test for each of the boundary polygons, the intersection with the set of camera footprint polygons. The output of the script is a Python dictionary with the name of the locus as a key and a list of all images that see the locus as a path to the images as a value. The entire process can be seen in Figure  3.
Complete retrospective reanalysis of clinical ES data after a defined period of time has elapsed, along with pipeline updates (especially to the OMIM and ClinVar databases), leads to an additional diagnostic yield ranging from 10. 5% to 32% (Table  1 )  (Costain et al. , 2018; Ewans et al. , 2018; Nambot et al. , 2018; Baker et al. , 2019; Li et al. , 2019; Salfati et al. , 2019). A recent review of published reanalysis papers reported 10% diagnostic yield, but with considerable heterogeneity due to the delay in reanalysis (after or before 24 months), which is the limitation to OMIM-morbid genes or research extension  (Dai et al. , 2022). The major reasons for novel diagnosis are not only the ongoing discovery of novel genes involved in human diseases  (Boycott et al. , 2017)  but also novel annotations of well-known OMIM-morbid genes extending their clinical phenotypes  (Fokstuen et al.
, 2016). However, genes that are newly associated with human disorders are not instantly implemented in the commonly used public databases  (OMIM, ClinVar, etc. )  or in laboratory databases, which limits the power of reanalysis in the diagnostic setting. Diagnostic yield may indeed be found to increase significantly, from 30% to more than 40% in DD/ID, when reanalysis is extended to translational research involving data-sharing for candidate genes  (Bruel et al. , 2019). Because systematically performing complete reanalysis of ES data represents a significant challenge for clinical laboratories, a semi-automated reanalysis pipeline that interrogates various databases could facilitate efficient reevaluation of undiagnosed individuals using up-to-date literature; this could be of significant value to clinical laboratories  (Costain et al.
, 2015). In rare diseases, targeted GREP queries can be performed over ES or GS data on request and/or for specific purposes  (Panagopoulos et al. , 2014; Bujakowska et al. , 2015; Panagopoulos et al. , 2015). Nevertheless, intensive use of GREP query strategies for certain genes or variants has never been reported. We present an innovative reanalysis strategy combining intensive prospective medical bibliographic monitoring with subsequent use of rapid GREP querying in a large ES database and international data-sharing to improve diagnostic yield and reduce diagnostic delay in individuals with DD/ID.
Prospective medical bibliographic monitoring From April 2019 to October 2020, manual monitoring of medical bibliographies was performed every day by one medical biologist. The biologist searched the PubMed database for the following terms: intellectual disability, (neuro)developmental delay, and (neuro)developmental disorder. The PubMed search output was sorted using the "most recent" display option, and only the 10 articles displayed on the first page were considered. When an article described a novel gene-disease association or broadened a known gene-disease phenotype, we considered this article to be relevant. In addition, the gene name was searched on OMIM to look for any existing association with human disorders. Finally, the gene in question was searched in the ES database using the GREP command-line tool.
GREP strategy Use of the daily GREP query strategy started in April 2019, with a basic GREP query used to search for every rare variant (variant allele frequency <1%) of genes of interest in all the vcf files available (hg19). These vcf files are text files storing gene sequence variants with a mandatory minimum of eight columns and custom annotation. The custom annotation included general population frequency and occurrence [in the Exome Sequencing Project (ESP), Exome Aggregation Consortium (ExAC), Genome Aggregation Database (gnomAD) exome/genome, and a control sample count from our own database]; a batch sample count; several pathogenicity scores [Polyphen  (Adzhubei et al. , 2010) , GERP  (Cooper et al. , 2005) , Grantham  (Grantham, 1974) , CADD  (Kircher et al. , 2014) , misZ, and pLI]; the OMIM pathology name, number, and mode of transmission; and lastly, the ClinVar classification.
This basic GREP query performed by the biologist is referred to as v1 in the manuscript figures and tables. For example, the basic GREP query "grep -w GENE1 *. tsv > GENE1. tsv" searches for the exact expression "GENE1" in all files ending with ". tsv" and creates a new file "GENE1. tsv". The -w option is included so that the query searches for whole-word matches for GENE1 and will not print lines such as GENE10, GENE11, etc. With the increasing amount of ES data (novel or reanalysis) available, the number of lines printed increases accordingly, leading to hardly readable files. To improve this command-line tool and make the resulting file more biologist-friendly, in early 2020, we developed a bash script set up for our specific working environment, referred to as v2. The novel command-line script only requires the gene name as an argument, meaning that the biologist only has to type "GENE1" (example query: "getthatgene.
bash GENE1", compared to the more complex v1 GREP query. The output of the v2 GREP query was reformatted using a python3 script called by the bash script. The most important steps included 1) removal of exome reanalysis duplicates to retain only the most recently analyzed file, and 2) filling of gaps in the additional annotations with dots to produce an intelligible spreadsheet (Figure  1A ). Three novel annotations were added during versioning of the database, namely, the and observed/expected with lower/upper threshold, Splicing Prediction Pipeline,  (Leman et al. , 2022)  and the CCR  (Havrilla et al. , 2019)  scores. Finally, in October 2020, we performed an end-point GREP query by searching again for all the genes previously searched with the daily GREP queries to see whether we could recover any missed diagnoses due to the time that had elapsed between the initial daily GREP queries and the end-point GREP query.
Prospective medical bibliographic monitoring From April 2019 to October 2020, prospective medical bibliographic monitoring identified 128 genes implicated in DD/ID, mainly from publications in the American Journal of Human Genetics (30/128 genes), the Journal of Medical Genetics (10/128 genes), Brain (10/128 genes), Clinical Genetics (8/128 genes), and Genetics in Medicine (8/ 128 genes) (Figure  2 ; Supplementary Table  S1 ). Among these 128 genes, 100 were associated with a novel human disorder (of which 37 were still not classified as morbid in the OMIM database and one was not reported in OMIM at all), nine had already been associated with a distinct human disorder, and 19 represented updates to the genotype/phenotype correlation of a known human disorder (Figure  3 ). The number of relevant manuscripts identified per week ranged from 0 to more than 5 (Figure  4A ).
GREP strategy The time interval between a positive GREP query and delivery of the diagnostic report to the clinician ranged from 1 to 11 weeks (overall mean and SD: 4. 2 ± 3. 1; mean in 2019: 3. 4 ± 2. 2; mean in 2020: 4. 5 ± 3. 5; Figure  4B ). For the daily GREP queries, the average time was 2. 6 months (SD ± 1. 8), compared with 6. 6 months (SD ± 3. 4) for the end-point GREP queries. We also calculated the number of printed lines resulting from GREP queries using v1 and v2. Overall, the v1 script resulted in 52 genes and 82,156 lines of output (mean: 1,579 output lines/gene) and the v2 script resulted in in 76 genes and 11,953 lines of output (mean: 157 output lines/gene). This represented a drastic reduction (mean: 10-fold) in the number of output lines between v1 and v2. There was a slight increase in the number of lines of output between the two sets of v2 searches (mean difference: 1.
Molecular results From the daily GREP queries, we obtained results of interest for 38 variants (29 premature stop codons and nine missenses) in 25/ 128 genes (20%) in 37 individuals (Figure  5 ). After Sanger validation and family segregation, 18 variants were classified as pathogenic or likely pathogenic (P/LP), 15 as variants of unknown significance (VUS), and four as benign or likely benign (B/LB). In three genes, we identified both P/LP variants and VUS (MAPK8IP3, SCAF4, and SUZ12), and in one gene, we identified P/LP, VUS, and B/LB variants (ZNF292) (Table  2 ). All 18 P/LP variants in the 16 genes were identified in 18 previously undiagnosed DD/ID affected individuals, with diagnostic odysseys ranging from 6 months to 6 years. Thirteen of the 16 genes had not previously been reported to be involved in human disorders in the OMIM database and therefore were not annotated in the pipeline and had not been considered in the first diagnostic analysis.
After Sanger validation and family  Frontiers in Genetics frontiersin. org segregation, five variants were classified as P/LP (in three genes for three affected individuals), eight as VUS (in eight genes for eight affected individuals), and five as B/LB (in four genes for five affected individuals). The five P/LP variants identified the three genes had not been considered in the initial analysis for two reasons. First, the queries for SMPD4 and WDFY3 were carried out using the v1 script, probably leading to output that was too complicated for interpretation (1,259 variants in the v1 GREP query for SMPD4 and 1,433 for WDFY3). Second, the query for NUP188 was carried out in February 2020 using the v1 script, but the individual in question was only added to our exome database in August 2020, so the variants could not have been detected previously.
Moreover, with the majority of the candidate genes being responsible for ultra-rare diseases, this strategy would probably prove to be more effective if the number of undiagnosed affected individuals was much greater. The novel diagnoses were mainly established in genes newly implicated in human diseases (32/36 genes). These genes had not been reported to be involved in human disorders in the OMIM database at the time of the initial clinical ES analysis and thus, could not be retained by the usual diagnostic analysis. Since these variants were present in the vcf files used in the initial ES analysis, a translational research analysis extended to non-OMIM (morbid) genes could focus on these candidate genes  (Fokstuen et al. , 2016). However, some variants appear to be very difficult to interpret, especially missense variants [39% of variants (22/56) identified in our GREP queries].
e. , one gene per day) in 2019/2020 according to OMIM statistics  (Boycott et al. , 2017). However, the time that elapses between online publication in PubMed and OMIM indexing can be up to several months, which delays the annotation of these genes and hampers their interpretation in clinical routine practice. It also requires periodic pipeline updates with information from different databases (OMIM, ClinVar, HGMD, and denovoDB) and reanalysis to improve diagnosis. Despite their recognized utility  (Nambot et al. , 2018; Bruel et al. , 2019) , periodic reanalysis strategies in diagnostic and research settings may represent a significant challenge for clinical laboratories. These appear to be very time-consuming, with systematization being possible only if a sufficiently large translational research team is available to work in partnership with the clinical laboratory and physicians.
A strategy combining intensive prospective bibliographic monitoring and targeted GREP queries appears to be a good compromise for the workload of clinical laboratories, especially since the usual diagnostic pipelines rely on updated data to aid in the interpretation of genes newly found to be involved in human disorders and/or flagged with pathogenic information in databases. In addition, this strategy enables faster diagnoses than periodic reanalysis. The time delay appears to be significantly reduced in the daily strategy, compared to published periodic reanalysis strategies, where it currently ranges from 6 to 18 months (Figure  1A )  (Costain et al. , 2018; Ewans et al. , 2018; Nambot et al. , 2018; Baker et al. , 2019; Li et al. , 2019; Salfati et al. , 2019)  or may be conducted on physicians' request.
They must belong to the clinical area of the cohort studied. Indeed, our keywords were suitable for DD/DI but did not capture a wide range of genetic diseases and would not be suitable for other databases. Limiting their number or combining them would reduce the chances of identifying suitable candidate genes and therefore reduce the scope of the GREP query strategy. Increasing their number would probably lead to the collection of additional manuscripts to be read, which would be more time-consuming and therefore difficult to implement in routine diagnosis. An alternative could be to subscribe to RSS feeds from selected journals in the field of interest  (Sobreira et al. , 2015). This strategy requires the selection of appropriate journals and subsequent filtering of manuscripts of interest from these journals  (Dubuque, 2011; Beller et al.
, 2018; Marshall and Wallace, 2019). Ultimately, the identification of manuscripts could be automated via a direct search in PubMed, which would facilitate the search and save time  (Bohle, 2018). While retrieving manuscripts from appropriate journals is one component of the strategy, it must be acknowledged that systematic review of these manuscripts to identify the few manuscripts of interest is another. Indeed, identifying the name of a particular gene in a manuscript title does not imply that the article presents a novel gene-pathology association. To address this caveat, several text mining software tools have been developed, each with advantages and disadvantages  (Rani et al. , 2015; Van der Mierden et al. , 2019). In addition, updates to public and private databases could lead to update description of variants or changes in classification.
Accordingly, DD and ID are likely the best groups of rare diseases in which to apply this strategy. Nevertheless, efficient reanalysis strategies have also been reported in other rare disorders, such as sudden death, DD/ ID, epilepsy, and Mendelian disorders  (Costain et al. , 2018; Ewans et al. , 2018; Nambot et al. , 2018; Baker et al. , 2019; Li et al. , 2019; Salfati et al. , 2019). Another advantage of the GREP query strategy is its ease of setup, since the GREP command-line tool is available in every Linux terminal. One disadvantage is the multiplicity of pipeline versions and/or reanalysis of ES data. Indeed, if different vcf files originating from the same affected individual, but with different versions of the pipeline (and most likely with additional annotations), are searched using a GREP query, then the output file resulting from the query will consist of multiples lines pertaining to the same variants (Figure  1B ).