dataesr/scientific-paragraphs-categorization huggingface dataset:
py to store and retrieve data from the database. The general module appion/Loop. py iterates over a set of images in an experiment and pause the processing until a new image is entered in the database. This functionality allows to perform some operations (e. g. Particle Picking or CTF estimation) concurrently with the data acquisition. While the Appion Python scripts can be executed from a command line, most of the processing steps can be launched from a web-based GUI. The web pages are generated using PHP [94], a server-side scripting language, from which Appion SQL database is accessed for reading and writing. This GUI provide access to the underlying processing pipeline, even if the command line can also be used. The Appion pipeline guides users over a wide variety of processing and analysis software packages.
Current research in EM depends on the analysis of huge amounts of data. These are in several formats and use different vocabularies. Furthermore, data are often stored or distributed using formats that leave implicit many important features relating to the structure and semantics of the data. It is really surprinsing that a formal data model has not been developed for EM data processing, independent of the underlying storage format. For example, in Leginon  [92] , a database is used to organize images acquired by the electron microscope. Nonetheless, all the Leginon's documentation found emphasizes in the benefits of the relational database and its implementation, but does not points out to a more logical data model. The same happens to Appion  [45] , that follows a similar architecture to Leginon.
In this case, a relational database is implemented for data processing but the data model is not clearly described. Another example is EMEN2  [95] , developed by the same group of EMAN2  [47] , that implements an electronic notebook using an object oriented database. This database is built around two concepts: protocols and parameters. Parameters are values that might be recorded during an experiment and they are defined by a description, data type and units. A protocol is composed by many parameters and defines the specific type of record that will be stored in the database. EMEN2 uses hierarchical relationships between parameters and protocols to create some kind of ontology. Nevertheless, from an image processing point of view, EMEN2 provides only some simple visualization and evaluation tasks.
Their authors claim that it is possible to incorporate portions of the processing pipeline, but currently tasks such as 3D reconstruction and refinement are treated as external processes. It seems that the ontology developed for EMEN2 is not directly used by EMAN2 for the implementation of image processing programs. Finally, it should be mentioned the EMX  [96]  initiative, that implemented an exchange format between EM software packages. EMX was one of the main results obtained from the I2PC Developers Workshop held in Madrid in 2012. In EMX, a reduced data model for EM image processing was described and implemented using XML. The proposed data model described some entities such as Particles and Micrographs, while others were left out of the initial standard. This work represents a first step in the direction of standardization and data modelling, although this model is only supported by current EM packages as an import/export format, but not as their processing model.
Scientific Workflows Scientific workflows have been increasingly used in the last decade for data intensive science  [97]. A variety of workflow systems, both open source (e. g. Taverna  [98] , Galaxy  [99] , Vistrails  [100] , Kepler  [101] , Triana  [102] , ASKALON  [103] ) and commercial (e. g. Pipeline Pilot2  [104]  ) are in use in many scientific disciplines such as genomics, astronomy, cheminformatics, etc. In this section three of these workflow systems are analyzed: Taverna, Galaxy and VisTrails. These were selected because some of their application domains have points in common with our field. Moreover, some ideas implemented in Scipion have been inspired from features in those systems.
Most Taverna workflows are composed from a mixture of distributed Web Services, local scripts and other service types. Nevertheless, in some cases the installation only uses local services, where data and software are placed together and network traffic is notably reduced. One example of such scenario is the usage of Taverna in cloud environments, where the input dataset needs to be uploaded once and the engine and services receive a reference to the data. The main advantage of using distributed services, however, is that computations can be off-loaded to remote service providers and there is no need to install tools or data locally. Consequently, complex analyses can be performed regardless of local infrastructure, using distributed tools and resources. A disadvantage of relying on external Web Services is that workflows can be broken by service unavailability or changes in their interface  [105].
Nonetheless, there is a large redundancy of web service functions that allows to identify reliable services and alternatives to make workflows more robust. The workflow engine in Taverna is implemented in Java as a multi-threaded object model, where processors are represented by objects, and data transfers are realized using local method invocations. Since each processor is a separate thread, it can autonomously start working and transfer output data without the need of centralized enactment system. Although execution is decentralized, a facade pattern is used to provide external components with a single point of contact (e. g, monitor and result presentation in Figure  3. 1). The Taverna Workbench is a graphical application that allows bioinformaticians to develop new workflows and test analysis methods, by either creating the workflows from scratch or by composing existing ones (See Figure  3.
Galaxy In 2005, the Galaxy Project started with the main goal of enabling researchers without programming and system administration expertise to perform computational analysis through the Web. It was designed as a system for the integration of genomic sequences, their alignments and functional annotations. Despite being available several genome browsers at that time for data visualization (  [106] , http://genome. ucsc. edu], NCBI MapViewer  [107] , and Ensembl  [108] , http://www. ensembl. org), more complex analyses still required programming and databases skills. By addressing this problem, Galaxy allowed users to gather and manipulate data from several sources, while storing every user action in the system history, a key element of the framework. One of the core features of Galaxy is its web-based GUI that provides users access to a large set of computational tools.
The main window (shown in Figure  3. 3) is divided into three panels: the left one shows available tools grouped into categories that can also be searched; the middle panel is used to fill the parameters of the selected tool or to inspect an output dataset; the right panel contains the full history of actions and produced outputs. All tools use a single interface to simplify new tools integration and to maximize usability. In the recent years the tools interface was rewritten to improve the user's experience. One addition was the possibility to include citations to facilitate researchers to reference the methods used in their analysis. Another important component of Galaxy is its graphical workflow editor (see Figure  3. 4). It greatly simplifies the task of creating multi-step processes by drag-and-drop different tools and connecting them so that the output of one becomes the input of another.
Workflows are critical to automate and repeat the analysis of large datasets. Workflows capabilities have been improved since the first versions of Galaxy. For example, workflows can now be paused and restarted. Moreover, entire workflows can be embedded into another workflow as a single step, what facilitates the creation and management of complex analysis in the Galaxy's workflow editor. Over the years the number of available tools (more than 650 at the time of the writing) for the Galaxy community has increased notably  [99]. Tools vary from simple text manipulation and statistical operations to the analysis of high-throughput genomic datasets, including quality control, genotyping and variant identification, motif finding, RNA-seq, ChIP-seq and metagenomics. The system can handle a large number of data formats (e.
The vistrails are stored using XML, which is a very flexible format to accommodate variability in the information being stored. A clear benefit of using an open and self-describing format is the ability to query, share and publish vistrails among scientists. The execution of a vistrail can take a long time depending on the size of the dataset and the complexity of the operations used. Taking advantage of the high-level specification of a vistrail, the system analyzes and optimizes the execution of dataflows. The Vistrails Cache Manager takes care of the scheduling of modules, it identifies previously computed subnetworks and replace costly operations by constant-time cache lookups  [100]. For the modules that needs to be computed, the Vistrail Player receives a dataflow instance (an XML file) and executes it using the underlying Visualization or Script APIs.
This model is a cornerstone of the entire Scipion framework to integrate of existing EM programs and to adapt to future methodologies and requirements. Moreover, having a good specification of the processed objects allows to go one level up and concatenate more efficiently different steps to form an entire workflow. By providing the general overview of the project as a workflow, users could track all the operations performed and reproduce them in a more automatic way. The first section of this chapter summarizes the evolution of this model design and the software technologies used for that. First, we started with a relational database, then we moved to a much powerful ontological database and finally we used Python to bring together the model definition and its usage in the implementation.
Design Evolution The initial Scipion design made use of a relational database (PostgreSQL) to represent the EM data model. Very soon we recognized that it was not a good approach due to the large number of tables required that were very difficult to maintain. Relational databases are very useful when the data model is well understood and it is known how the data will be used. However, for Scipion the developers can never quite know what requirements will be needed to accommodate the different algorithms of each software package (or for different methodologies), so high flexibility was mandatory. In this way, from an initial design in which database tables were strongly related to the different image-processing elements, we changed to a more flexible approach. The new schema reflected imageprocessing entities in a more abstract way using classes and its properties, that were stored in flat key/value pairs.
This new approach was much more flexible to adapt to new data entities and algorithms. The new design was implemented using the Jena-TDB ontological database (http: //incubator. apache. org/jena/documentation/tdb/index. html) for metadata storage. The persistence layer used Empire (an implementation of Java Persistence API), which defines ways for accessing, persisting and managing data between Java objects and a relational databaset. This layer queried the ontological database through SPARQL, a query language to retrieve and manipulate data stored in RDF (Resource Description Format) format. The Scipion ontology, that was created using Protégé (http: //protege. stanford. edu), contained around 300 classes and a similar number of properties. There were two main issues with this implementation that could seriously attempt against the project goals.
First, it was not easy to extend with new protocols since the classes (Java or Python) were automatically generated from the ontology definition. So, adding new protocols or data objects required an entire re-build of the ontology and code generation. This was not an easy procedure to be done each time a new protocol was added. The second big problem was the performance. When a new empty project was created in Scipion, it contained a database with hundreds of thousands rows. Later on, this huge number of rows and the intermediate layers used, reduce the queries performance and detriment the user experience for simple processing tasks. During this stage of the development of Scipion, our laboratory was also focused in the release of a new major version of Xmipp, our software package for image processing in EM.
Movies The Movie class is a specialization of Image that contains the location of the movies frames stored in the hard disk and can store information related with frames alignment. By storing only the relative shifts between frames to align them, we can later obtain the aligned binary movie on demand. This allow us to avoid several copies of the input movies, which could be of a considerable size (2TB for a one day session). Even if nowadays the disk space is relatively cheap, the raw movies data size is becoming a great concern when several projects are executed simultaneously or different processing strategies are explored. Addressing this concern, the Movie class in Scipion allows transparent access to compressed binary files.
Process Model Another cornerstone of our entire business model is the process model, which is complementary to the previously discussed data model. The central element of the process model in Scipion is the Protocol class, that can be defined as a container of algorithms and the engine for its execution. Protocol objects are composed by the same basic Python types discussed in Section 4. 2. 1. The main challenge when developing the protocols infrastructure was how to deal with a myriad of software packages and programs, with different conventions and file formats. At the same time, we wanted to simplify the task of adding a new protocol to the whole framework for developers and for expert users without a strong computational background. To reach this goal, we designed the protocol infrastructure following the Facade and Adapter design patterns  [114].
Both patterns address the problem of dealing with an external system that is not under our control that we want to simplify or adapt to fit our desired interface. In our specific case, the Protocol class acts as a wrapper over the underlying programs. On one side, each protocol should be consistent with the Scipion interface by using as inputs and output objects defined in our data model. On the other side, the protocols internally "adapt" the Scipion context to each program by converting to the required formats and calling the programs in the correct way for each case. In order to formalize the description of the process model we are going to define some of the more important concepts: Definition 4. 1. A step is the minimal amount of work that can be scheduled by the system. Each step could depend on other steps, for example, if a step s2 depends on s1, then s2 can only be executed after s1 is finished.
Streaming Processing The first Scipion release was designed so that if a run r2 depends on an object created by a run r1, then r2 could not be launched until r1 finished. Soon we discovered that, under certain circumstances, this approach introduced a performance limitation. For example, it prevented to execute pre-processing steps (e. g. movie alignment or CTF estimation) while the data were been acquired. To address this new requirement we made some changes in our data and process model. First, we introduced the property ST REAM M ODE to the Set class discussed in Section 4. 2. 1. This property could have the value OP EN , when still new elements are being added to the set, or CLOSED, when no more elements will be added. Furthermore, streaming protocols create the output set as soon as the first element is processed and update it with new elements later.
For example, in streaming mode, the import protocol keeps monitoring the data folder to add more elements as they are acquired by the microscope. To support the data stream processing, some protocols were modified to adopt a more proactive behaviour to discover new input data, process it and update their output accordingly. As illustrated in Examples 2 and 3, the developer of a protocol could add different steps depending on the parameters or the input data, but with the assumption that input objects were complete. In the case of Example 3, there are two steps added per input micrograph, but it will not process more micrographs if they are added after the insertAllSteps function is executed. We added a new function, stepsCheck, that will be called automatically by the underlying execution engine.
Packages As mentioned in previous chapters, Scipion was conceived with extensibility in mind. For EM we developed a basic data and process model to represent the processing pipeline, but Scipion design allows the extension of this model. In order to extend the current EM model, it is required to provide new packages (referring to EM packages such as Xmipp, Relion, EMAN2, SPIDER, etc), that contains mainly protocols and viewers. The flexibility of Python facilitated the task to design packages as a plug-and-play architecture, where all the information of a package is contained in a single folder. When a new package folder (Python module) is dropped into Scipion, the framework will automatically discover the protocols and viewers defined in that package. When there are many protocols implemented within a given package, it is useful to define some utility functions at the package level, such as converting particles from Scipion to the package format and vice-versa.
Protocols Together with the EM data model discussed previously, we also developed several EM protocol classes that serve as the base for further extension. The classes hierarchy diagram shown in Figure  4. 1 represents the main categories of protocols for EM. Protocols derived from P rotImport serve to incorporate external data into a Scipion project. The subclasses of P rotM icrographs represents movies and micrographs operations, while the subclasses of P rotP articles contains protocols related to particles. Protocols that perform 2D analysis (such as alignment and classification) should inherit from P rot2D, while in 3D (inital volume, 3D refinement and classification) should do it from P rot3D. These classes form the base from which any other EM protocol will be derived. In some cases, the base classes contains some code to perform common tasks, facilitating the development of new protocols with similar functionality.
Viewers Another type of objects that could be defined in the framework is the V iewer, which handle the visualization of other objects such as protocols or data. Viewers are also important objects since visualization and data analysis are essentials for EM processing. In Scipion, a viewer class can be linked to one or several objects types via its targets property. Moreover, for a given class C, more than one viewer can be defined. Then, in a given visualization context, the system can discover all viewers that are available for a given object and the user can choose among them. The viewers mechanism also allows us to quickly incorporate existing visualization tools from EM packages or from other scientific visualization applications. Furthermore, viewer actions are encapsulated as V iew objects to separate the logic of the data representation from the environment where it is going to be displayed.
Implementation This section describes the implementation of three important parts of the Scipion framework: (a) the persistence mechanism, (b) the graphical interfaces and, (c) the execution engine. Most Scipion code is written in Python since it is a very versatile scripting language. Python has been widely adopted for scientific applications and is also used by several EM packages in the field. The persistence layer uses Sqlite as the underlying storage and implements a simple ORM to work with Python objects that are mapped to the database. The execution mechanism is implemented purely in Python and it is designed to be used in high-performance environments such as multi-core machines or clusters. Graphical interfaces have been implemented to maximize modularity and usability so they can be reused in different parts of the framework.
Automatic Data Persistence In 3DEM, most software suites use text files for metadata storage. A less popular alternative is databases. Leginon and Appion use the Sinedon library to map Python classes onto a relational database. The underlying database was implemented with MySQL (https://www. mysql. com/), which requires the setup of a dedicated database server. The Sinedon library was developed using the SQLDict Python library that operates on dictionary-like objects. In EMAN2, the file-oriented Berkley Database (BDB) was used for some time, but it was removed from version 2. 1 because users needed to care too much about the BDB files structure. At present, EMAN2 stores information using JSON and HDF5 files. The SPIRE layer on top of SPIDER introduced a database to stored result files, with the drawback that it is not fully synchronized with all files in the project folder.
In Xmipp 3, the SQLite library was used internally to operate on image metadata. In Scipion, one of the design principles was to separate the conceptual model from the physical model. For achieving that goal, we looked into the object-relational mapping (ORM) concept, which allows data transfer between objects in a programming language to/from an underlying storage. In Python there are many libraries that provide ORM functionality, including SQLAlchemy (http://www. sqlalchemy. org/, Django's ORM (https://www. djangoproject. com/, Peewee (http://docs. peewee-orm. com/), PonyORM (http://ponyorm. com/) and SQLObject (http://sqlobject. org/), among others. Some of these libraries implement the Active Record pattern  [116] , which maps a given object into a row of the corresponding table. This pattern is very intuitive to link between the database structure and the programming objects, but have the downside that objects are too tied to the database structure.
Another pattern is the Data M apper  [116] , in which there is a clear separation between the domain and the database and the objects do not have persistence methods. Both patterns have their pros and cons, as well as using ORM tools. One of the concerns that is associated with the use of ORMs is the potential for reduced performance, together with the danger of shifting the complexity from the database to the application code. After testing some of the above mentioned libraries we decided to implement our own mapper because performance was critical for us when dealing with large EM datasets. Furthermore, most of these ORM libraries require to use a particular data model philosophy that constrained too much the Scipion model design. We choose Sqlite, a relational database engine, with built-in functionality for querying, updating, sorting and searching.
Despite we have used Sqlite as the engine to store and retrieve objects, our implementation allows to replace this mapper with other approaches, either based on another database engine or even using text files (e. g, xml or json). In Scipion, we found two different contexts with different requirements for the ORM layer. In one hand, we wanted to store different kind of objects and their relations in a simple manner for the entire project. On the other hand, we wanted a very efficient mapper to handle huge datasets, that are common in EM processing. We have developed two different strategies that fit in each context, both of them are based on the basic objects hierarchy described in Section 4. 2. 1 and use SQLite as the database backend. In the following sections we describe both approaches.