dataesr/scientific-paragraphs-categorization huggingface dataset:
Introduction Biological institutes and museums of natural history around the world curate large collections, including type specimens. Their uniqueness, as well as the issues of physical access to and the protection of artefacts constitute an argument for their digitisation, even though there is remote accessibility for a wider group of beneficiaries interested in biodiversity. The material that is formed out of these resources feeds digital databases. With available and correct data at their disposal, nature conservation managers, policy-makers, scientists and local communities can consciously make decisions for managing natural resources  (Ballard et al. 2017). Digitisation revives the Natural History Collections (NHCs) documenting the biota of the Earth, which contributes to a better understanding and protection of biodiversity  (Butchart et al.
The use of such a standard also facilitates the subsequent interoperability of systems of similar purpose, which considerably contributes to increasing their possibilities in the area of data mining and analysis  (Hardisty et al. 2019). The amount of data necessary to process during a typical digitisation process requires automatisation by using specially developed procedures and tools. They cover the rules for processing information in the form of hard copy records, which are transferred to a predefined format and a subsequent verification of the prepared data. These tools are coupled with database systems that have the capability to incorporate this data and manage it effectively  (Zalani et al. 2019). The purpose of this paper is to present the key components of the AMUNATCOLL IT system for biodiversity data organisation, storage and processing.
The metadata and the database structure, as well as applications and procedures are used in the data flow. The work has been conducted within the AMUNAT-COLL project, which is co-financed by the European Union from the European Regional Development Fund under the Operational Program Digital Poland (ANC 2022;  Jackowiak et al. 2022). The overall goal is to make information on the natural collections at the Faculty of Biology, Adam Mickiewicz University  (FBAMU 2022)  openly available to all interested parties. The specific project aim is to develop a standardised set of metadata that enables the registration of unique NHC values to be compatible with international standards. On this basis, the design and development of a digital database with the structure corresponding to the metadata was carried out in the implementation phase.
Metodological assumptions The basis for developing the methods and the mechanisms of the data storage system is the proper understanding of the needs of target groups and the technological solutions that ensure the required functionality and compliance with the applicable solutions. In this study, they relate to metadata, database and supporting operational procedures. It is of utmost importance to recognise and use the standards for handling biodiversity data under IT system implementation. The standards allow us to follow common rules in the description of specimens, and due to their specificity, affect the database design and layout of the information management system. Moreover, they facilitate common and repeated use of data and enable following similar rules in various applications and processing methods, improving usability.
The semantic definitions of specimens defined in DwC facilitate their better use in a variety of contexts. In terms of definitions, Darwin Core does not specify data types and constraints; it relies on recommendations to property values that can be limited, e. g. , by a specific vocabulary. The most recent version of the Darwin Core Standard was released in 2009 and consists of 169 defined terms  (DwCS 2009). ABCD is a schema that was designed for describing taxon occurrence and data exchange in a highly structu red way supporting the diversity of databases and compatibility with other data standards. ABCD is considered an evolution of the Darwin Core, having a more refined structure and containing approximately 1200 elements. Processing of the ABCD information is facilitated by compliance with eXtensible Mark-up Language (XML 2016), which is maintained by a variety of applications.
The current major version, widely adopted and endorsed by Biodiversity Information Standards (BIS 2021), is 2. 06. ABCD is composed of three main concepts: mandatory, highly recommended and commonly used. The mandatory concept outlines parameters that are compulsory for creating valid ABCD documents. It is related to the dataset description and pertains to aspects, such as contact with the people responsible for the content and the technical issues and the metadata general description, while also specifying the institution from which the data comes from. The highly recommended concept describes a common entry point to the data. The data contains information about licence, language, units with details on identification, locality and URI. Commonly used concepts encompass more data on specimen descriptions, e.
Database The definition and the design of the database is the consecutive aspect after the metadata structure. Despite standardisation at the metadata level, the structure of a database does not fully correspond to the representation of its full dataset. In individual biodiversity systems, it reflects the specific needs and requirements underlying their definition. The need to adapt the scope and the structure of the stored data to the offered functionality in a way that ensures the highest efficiency frequently translates to relying directly on popular database management systems (DBMS), such as MySQL (2022),  Post-greSQL (2021) , Microsoft SQL Server (MSSQL 2022), and Oracle Database (Oracle 2022). The serving of multimedia files is facilitated by the Multimedia Database Management System (MMDBMS 2022) that is built upon the DBMS and offers additional functio nality, such as integration, data independence, concurrency control, persistence, privacy integrity control, recovery and query control.
Operational procedures A similar approach adjusting to the requirements is applied for the implementation of the operational procedures. These are used for facilitating data preparation, validation, casting to the database tables and securing multimedia files. Moreover, they can be designed in the form of regulations (e. g. , explaining the who, what and when is implemented in the digitisation procedure) and the accompanying set of tools for conducting this process separately or in a workflow by its automation. The implementation method depends on the dataset specificity (e. g. , the form of source materials being digitised), as well as the decisions and preferences of the developers themselves. Therefore, specific programming languages, libraries and frameworks cannot be named here just by indicating the only correct solution.
Metadata specification The ultimate purpose of the metadata definition task is to prepare the structure that links to the need for formal characterisation of the unique resources of the natural collections gathered at the Faculty of Biology of Adam Mickiewicz University in Poznań  (Jackowiak et al. 2022). Considering the high development complexity of the database dealt with, the proposed formalisation enables the unambiguous elaboration of essential information in line with international standards. Nevertheless, despite the will to comply with the standard, to meet specific requirements addressed by the project and the system specification itself, a regular set of features The Adam Mickiewicz University Nature Collections IT system (AMUNATCOLL). offered by ABCD were extended by a number of nonstandard fields, such as those related to protected areas.
Only a synthetic overview of the extensions that were implemented in the AMUNATCOLL database in relation to the ABCD standard are more thoroughly discussed. A crucial facet in the process of the metadata structure definition was the generation of the identification number (ID) of the described object. The number unequivocally identifies a single specimen, an individual in the observation or a sample. Multimedia files, such as scans or photographs, are identified twofold. Herbarium scans of specimens were marked by the same number as the specimen itself. It enabled joining a picture with the corresponding item under presentation in the portal. The different approach was used for the photos that did not represent the specimens stored in the NHCs (e. g. , landscape, habitats), for which a unique ID was generated.
In the next step, criteria necessary to quantify the taxonomic information were identified. The table of taxonomic specimens containing the attributes necessary to describe plants, fungi and animals was translated into an extensive structure. This approach resulted in defining individual taxonomic attributes allocated to separate database columns. Another identified challenge was the alignment of the metadata structure for taxonomic synonyms for species with multiple names in particular. A special case considered the taxonomic names appearing on the herbarium sheets are no longer valid today. This was solved by implementing a synonym list containing these names. The specimen section provides supplementary information detailing the storage of specimens up to the indication of a specific rack and shelf.
This makes it easier to find the specimen in case of a need to reach a physical object. In dealing with specimens originating from protected areas (e. g. , national and landscape parks, nature reserves, Natura 2000 area), such information is recorded in specially designated fields. Information related to or belonging to specific geographical divisions (administrative, physiographic, European biographic and potential natural vegetation of Poland) constitute a separate group of information increasing the data value in the context of the functioning groups of organisms and their interdependencies. In addition, information on the Distribution Atlas of Vascular Plants in Poland (ATPOL) grid system (Zając 1978) as a regional spatial division system of Poland and suitability for target groups is supplemented.
The sample data section is entirely an additional group of information to ABCD on the specification of the unexplored and unmarked material (e. g. , soil sample, timber, nest) from which the selected specimens are derived. Once marked, they are transferred to the "specimen" group. The attributes of the samples are characterised by the specificity that requires the definition of a separate form where parameters, such as merocenosis (considered organisms colonizing the merotope) or groups of animals marked from a given sample can be specified. Moreover, the data include identification (required to create a new record in the database), spatial references, protected areas, the ATPOL grid system, information about gathering, merocenosis data and belonging to geographical divisions, which are similar to specimens.
Another section of the spatial metadata enclosed particulars on an affiliation of records to the protected nature areas, administrative division, regional geographic and biogeographic divisions, or a location in relation to the UTM (2022) and ATPOL grid fields  (Zając 1978). Considering this type of metadata allows for more spatially precise searches of records in the database and provides a more accurate survey of the specimen distribution depending on the characteristics of the terrain. In a different way from the ABCD standard, the multi media data in the AMUNATCOLL database constitutes a valuable source of information about landscapes, habitats and objects present there. As such, these objects are subject to an identification process similar to that conducted for specimens. This means preparation of a full set of data, including identification, taxonomy of objects located there, their detailed description, spatial references, protected areas, ATPOL grid system, description of the habitat, target groups and geographical divisions.
Data on references, understood as information about publications where specimens from the AMUNAT-COLL database are described, have been extended with a set of parameters describing them in the context of a specific publication type, including a journal, monography or book chapter. This enables the addition of extra details about volume number, issue, pages, publisher, publication location, and book title for a chapter. Bibliographic data are kept in a different table. Each item receives its individual number, which is then used to create a link to an NHC object referred to in a publication. Additional value was obtained by defining the "Comments" field in several metadata sections. It permitted the inclusion of additional information that, for formal reasons, could not be entered into other attributes of a corresponding section.
For example, in the case of multimedia not associated with a specific specimen entity, the field enables us to describe the picture content (animate and inanimate objects, surroundings, environmental conditions) in a flexible way, which is essential for understanding the content. The metadata specification also includes information, such as the allowed range of values for a given attribute and the connections between the fields that define the conditionality of their use. It facilitates the process of data preparation and the input into the database by several dozen people, which usually leads to numerous mistakes. The most common mistakes were related to the wrong date format, the use of illegal abbre viations and the use of values that at the moment could not be used in connection with the values of other fields.
Database structure The database is a key element of the AMUNAT-COLL IT system, where all metadata are stored; therefore, much time and attention were devoted to its efficient implementation. It has been designed to store all 212 metadata fields describing specimen specifics and the organisational, technical and support fields to ensure an effective way to access and manage them. The implementation was based on the PostgreSQL server (PostgreSQL 2021). It is a free, open-source relational database management system that emphasises extensibility and SQL compliance. It has the functionality required by the specifics of the project, such as ACID (Atomicity, Consistency, Isolation, Durability) properties, automatically updated views, foreign key triggers and stored procedures. PostgreSQL is designed to handle heavy workloads such as data warehouses or web services with multiple concurrent users.
The database storage structure is divided into three logical databases (Fig. 2 ). Each database contains data related to a specific operational part of the project. The "amunatcoll" database is associated with the storage of specimen information, the "dlibra" database houses the multimedia data, and the "anc_portal" database is for the portal operational tasks. The "amunatcoll" database corresponds to storing data related to specimens and contains tables with information about specimen taxonomic names and their synonyms, samples, bibliographic data about publications, kinds of specimens, collections and subcollections of specimens, statistics, and specimen import history. The "dlibra" database is the database of the dLibra digital library, and its task is to store data on imported multimedia objects.
From all the tables available in the library, the project uses the data contained in the tables storing information about metadata related to multimedia files and metadata of publication scans. In this database, the attributes and their values are stored in two columns common to all attributes. Contrary to the "amunatcoll" database, the attributes do not have dedicated columns to store corresponding parameters in a separate way. The "anc_portal" database is accountable for storing information related to the portal and its functions. It contains tables storing information about user data, user permissions, resources created by users from the mobile application (projects, observations and their files), other resources created by users (albums, filters, base maps), teams and their members, and visiting statistics.
Configurations are divided into two types: Standard -the conversion creates missing columns and form sheets, sorts them in the order identical to the specification, and changes the cell value formatting to the text to ensure data correctness in the import process. Custom -in this case, apart from the operations performed in the standard mode, additional operations (custom defined upon user request) are conducted on the input data to facilitate the preparation of data in the older format for import into the database, which could be changing the date format, combining or splitting columns, etc. In case of errors during the conversion or when detecting any irregularities in the file, the converter displays a detailed report. Below, the main page of the tool is presented (Fig. 7 ), where the user specifies the file for the conversion and the configuration scheme.
Validator Validator is a tool for checking (validating) Excel files, which is available in the form of a web application. Its main purpose is to analyse the correctness of the data delivered to the form, considering the context of the sheet (specimen, sample, iconography and bibliography) and field. Moreover, it verifies the presence of mandatory fields, duplicates, and provides in-between relations with specific fields. The tool returns a report detailing errors by occurrence and description, which is then broken down into each detected sheet. The tool along with the converter is an indispensable part of the import procedure presented in the figure below (Fig. 8 ). It must be noted that the validator is also utilised during the automatic import process to revalidate the data, which prevents information that does not meet the assumptions appear in the database.
Aggregator The program "Aggregator" is used to combine XLSX files that are consistent with the AMUNAT-COLL specification. It is used in the digitisation process, mainly during the stage of describing records by a georeferencing team or a translation team. Due to the specifics of the data preparation process, it results in many relatively small files with information about specimens. Information added at a later stage is often repeated for many records from the same collection. It is evident that it is easier to work with large datasets by properly sorting and filtering them. The aggregation program combines many smaller files into a one bigger file or set of files with a given maximum size whenever it is exceeded. The complimentary operation is performing the basic file format verification (e.
Reporter The application "Reporter" is used to summarise the number of records in XLSX files that are consistent with the AMUNATCOLL specification and it is used by project coordinators for the purpose of the ongoing monitoring of the progress of digitisation work. Similar to the "Aggregator" application, "Reporter" is a command line solution. The argument to run the application is the name of the directory where the input files are located. The result of the program operation is the information about the number of unique records in the categories 'SPECIMEN', 'SAMPLE' or 'MULTI-MEDIA. ' It also displays information about errors found in the files, such as an incorrect header or duplicate records.
The descriptive data import process The overall purpose of the import process is to incorporate data assembled by the digitalisation team into the database structure following consistency defined by metadata specification. The benefit of this process is obtaining easier access to digitised resources by making them available for external systems, e. g. , portals, mobile applications or cooperating external databases. The process of importing data to the database is carried out in three steps. The first is the data preparation by a person with specific privileges called the data administrator. Second, files are checked against compliance with the documentation (this time in an automated process), and the data is placed in the database. Finally, the files containing specimen data are sent to the definite directory on the cloud disk (data buffer).
The iconographic data import process The iconographic data import process, to some extent, resembles that of species data. In this case, metadata are accompanied by multimedia (scans, pictu res), and from the import perspective, they are one. That means that if one of the two (a record in the file or a multimedia file) is missing, the import will not begin. In the first step of importing iconographic data, the files are placed in predefined directories in the data buffer, and then they are imported to the dLibra digital library. The import is carried out at the level of a single directory, and the files in it are processed before being imported. The directory contains a metadata file that describes the attributes of each file in that directory.
The preparation of multimedia files Before importing, multimedia files are processed, which allows them to be made eligible for the AMU-NATCOLL system (preparation in terms of design assumptions and requirements related to the software used) and to protect them against unauthorised use. In the case of still images, files are subjected to the following operations: • If the picture is delivered in the CR2 (RAW) format, conversion to the "TIFF" format is applied. • If a photo has nonstandard orientation information, it is rotated. The data import procedure starts automatically at a programmed time of the day, provided that changes in the input files compared to the previous import are detected under the scan process (Fig. 10 ). Newly provided records are validated, and in case of success (data conforming to the specification), they are placed in the database.
Software The AMUNATCOLL project is highly demanding in terms of the number and size of processed images. It is enough to mention that hundreds of thousands of images with a capacity of hundreds of terabytes were developed during the implementation of the design tasks. Therefore, due diligence was taken in selecting the appropriate software. As a result, the following open-source software was selected and used for graphic manipulation in the importing process, processing and storing multimedia files: libvips (2021) and ImageMagick (ImMag 2021). Libvips is capable of processing large pictures with small memory utilisation since it does it piece-by-piece on small rectangles or groups of lines. Moreover, the processing can be conducted in parallel (threads are run on separate cores) mode, which leads to a better use of the full available computing power.
AMUNATCOLL uses libvips to create thumbnails and TIFF pyramid files. ImageMagick also prioritises the performance of conducted operations; thus, a multiple threads computation is viable even on tera-pixel image sizes. As a result of a large number of possible operations, such as converting, filtering, and artistic effects, this software is used as an engine of graphic operations in many professional solutions. In the case of AMUNATCOLL, it is primarily used to implement security measures related to the legal protection of shared graphics. It includes the aforementioned cropping of the image border, adding EXIF metadata, applying a visible watermark and a hologram that is visible only after applying the decoding operation. Both applications are able to run on the Linux system and can be called directly from the command line, which is a favour of the AMUNAT-COLL infrastructure and processing routine, and it is also crucial for the project prerequisites.