dataesr/scientific-paragraphs-categorization huggingface dataset:
This can be an effective way of reducing per-node memory usage especially when using GPU acceleration, and is also desirable when a large number of processes are run in parallel, a typical scenario when using Adorym on a multi-node supercomputer. For the user-given number of tiles of N tile,y and N tile,x , the original image of size L y × L x is divided into interconnected tiles with size given by L tile,y = L y /N tile,y Directions and quantities are labeled with the index of dimension in the corresponding array; for example, O2 means that the associated object axis is stored as the 2nd dimension of the object array. and L tile,x = L x /N tile,x , where x is the ceiling function that returns the smallest integer larger or equal to x. If MDH data are being tiled, all the tiles of images acquired at different distances are collectively regarded as "scan spots", and are saved contiguous in the second dimension of the dataset following a "tile-then-distance" order.
Module LargeArray The LargeArray class for large arrays is used to hold the object (when it is kept or partially kept in memory; more about this will be introduced in subsection 1. 7), or arrays that are comparable in size as the object (such as the object gradient and the finite support mask). The parent class of LargeArray contains common methods such as array rotation (important for processing tomographic data), HDF5 read/write (used in HDF5-mediated distribution mode), and MPI-based data synchronization (used in distributed object mode). The latter two methods mentioned above are specifically relevant to distributed reconstruction modes where the object function is either scattered over processes or nodes, or stored in a HDF5 file on the hard drive; distribution schemes for these cases are discussed in Section 1.
More information about parameter refinement will be introduced in Section 1. 4. 2. When the size of the raw data is large, Adorym can be set to process only a subset, or a minibatch, of the measured data containing batch_size tiles at a time. Thus, we hereby define the terms we will use for describing the workload of reconstruction: an "iteration" is used to refer to the process of finishing a minibatch of data by all parallelized processes (the inner loop), and an "epoch" refers to the work done to cover the entire dataset (all tiles on all viewing angles; the outer loop). Propagation through the object is done using multislice propagation which is capable of simulating multiple scattering within the object volume and is useful when the object thickness is beyond the depth of focus  [15, 11].
Parameter refinement In Section 1. 4, we have shown the incorporation of probe position correction in the forward model, which is done using Fourier shift theorem. In addition to that, Adorym also supports the refinement of a range of experimental variables, and the list can be readily expanded by users upon needed. At present, the following parameters and variables can be refined: • Probe defocus: in imaging techniques like ptychography, it is often the case where the probe function retrieved from a previous experiment is used to reconstruct a new dataset. However, as the axial position of the sample may differ, the previously recovered probe might be deviated from the incident plane of the new sample. In this case, Adorym can Fresnel propagate (Eq. 6) the probe function according to a list of defocusing distances, z defoc , which stores the defocusing that should be applied for each tilt angle.
On the other hand, packages may differ in performance depending on the environment in which they run, in the ease of setting up the environment they require, and in the size of their user communities. Autograd builds its data type and functions on the basis of the popular scientific computation package NumPy  [42]  and SciPy  [43] ; these in turn can make use of hardware-tuned libraries such as the Intel Math Kernel library  [44]. However, Autograd does not have built-in support for graphical processing units (GPUs). Another AD package which provides GPU support is PyTorch, which has perhaps a larger user community. Autograd and PyTorch are dynamic graph tools which allow the computational graph or the forward model to be altered at runtime  [45]. This means the forward model structure can be modified on the fly using Python's native workflow control clauses (such as an if.
else. block), which is very helpful for implementing a multi-stage optimization strategy. In this way one can use one type of loss function for the first several iterations, and switch to another after that. Moreover, a dynamic graph tool means the runtime values of any intermediate variables can be easily retrieved by inserting a print function inside the forward model code. With a static graph tool, this can only be done outside the forward model, which makes debugging less intuitive and straightforward. Therefore, Adorym is designed to specifically work with dynamic graph tools. As we aim to incorporate both Autograd and PyTorch (and more dynamic graph AD libraries in the future) in Adorym, we have built a common front end for them in the wrapper module. This module provides a unified set of APIs to create optimizable or constant variables (arrays in Autograd, or tensors in PyTorch), call functions, and compute gradients.
In order to make sure all frontend APIs know the right backend to use, we keep the setting parameter backend in a specific module named global_settings. This common module is imported by reference (instead of by value) in every relevant module of Adorym, and once its value is changed in any module or script, it is changed for all modules after that point. For example, the line import adorym. global_settings is used in both ptychography and wrapper; once it is changed in ptychography by adorym. global_settings. backend = "pytorch" upon user setting, the value of adorym. global_settings. backend is changed in wrapper immediately as well, so all frontend functions in wrapper referring to the backend setting will use functions from PyTorch in this case.
Module Optimizer Optimizers define how gradients should be used to update the object function or other optimizable quantities; these choices are provided by the class Optimizer. Adorym currently provides 4 built-in optimizers: gradient descent (GD), momentum gradient descent, adaptive momentum estimation (Adam), and conjugate gradient (CG). Moreover, Adorym also has a ScipyOptimizer class that wraps the optimize. minimize module of the Scipy library  [43] , so that users may access more optimization algorithms coming with Scipy when using the Autograd backend on CPU. The simplest GD algorithm updates an optimizable vector n using the gradient of the loss function with regards to it, or x ← x -ρ∇ x L ( 19 ) where ρ is the step size. GD is very low in overhead in that it does not require any additional parameters to be stored and updated.
Specifically, in Adorym we use MPI4py Python API  [56]. The MPI interface allows one to conveniently communicate data or synchronize program workflow among multiple processes (or ranks) on CPU cores. Because each spawned MPI rank is independent from others on the Python level, it can individually send data back and forth between RAM and a specified GPU, and run GPU operations without interfering with other ranks as long as the GPU memory does not overflow. Therefore, our MPI-based parallelization is also inherently capable of working with an arbitrary number of GPUs. When it is necessary to communicate between all ranks, each rank needs to retrieve the data back from the GPU to the CPU, and use collective MPI operations such as Allreduce. This is inefficient compared to direct GPU-to-GPU communications using tools such as NVLink  [57] , but it works on a wider range of devices.
For such cases, Adorym comes with an alternative option of storing the full object function in a parallel HDF5 file  [1] , which is referred to as the H5 mode. The HDF5 library works with the MPI-IO driver to allow a file to be read or written by several MPI ranks, where any modifications to the file's metadata are done collectively by all ranks to avoid potential conflicts. This feature provides a viable scheme of further reducing the RAM usage of Adorym. At the beginning of the reconstruction job, the object function is initialized and saved as an HDF5 file. Before entering the differentiation loop, the object function is duplicated as a new dataset in the same HDF5 file, and rotated to the viewing angle being processed in a similar fashion as the DO mode: the rotation is done in parallel by all ranks, with each rank handling several y-slices.
Following that, object chunks are read from the rotated object by each rank, and sent to GPU if GPU acceleration is enabled. The calculated gradient chunks are written by each rank into a separate gradient dataset, which, after all tiles on the current angle are processed, is rotated back to 0 • collectively. To update the object function, each rank reads in a slice of the object and the corresponding gradient at a time, performs update using the selected optimizer, then writes the updated object slice back to the HDF5 file. The next slice is then processed, until the entire object is updated jointly by all ranks. The workflow [Fig. 3(c) ] in fact resembles that of the DO mode, except the distribution of object chunks and synchronization of gradient chunks is done through HDF5 file reading and writing instead of using MPI AlltoAll.
User interface If a reconstruction job can be done using the provided ForwardModel classes, users generally just need to call the reconstruct_ptychography function inside the ptychography. Optimizers for different variables can either be explicitly declared using the child classes of Optimizer, or be specified by providing the optimizer name (for object function) and step size while using default values for other parameters. Below is a code example that can be used to generate the fly-scan ptychography results shown in the main text: import adorym from adorym. ptychography import reconstruct_ptychography output_folder = "recon" distribution_mode = None optimizer_obj = adorym. AdamOptimizer("obj", output_folder=output_folder, distribution_mode=distribution_mode, options_dict={"step_size": 1e-3}) where l, c, and s respectively gauge the similarity of the images in terms of luminance, contrast, and structure.
Figure12: Adorym reconstruction of a tomographic x-ray fluorescence and ptychography dataset of a frozen hydrated alga cell Chlamydomonas reinhardtii acquired using 5. 5 keV X rays [76]. The phosphorus K fluorescence projection (a) at 0 • through the 3D volume highlights polyphosphate bodies, which are also seen at somewhat lower resolution in the 90 • projection through the reconstruction due to the limited tomographic tilt range of -68 • to +56 •. As in a previouslyreported reconstruction of this dataset [76] , this phosphorus reconstruction used refinement of the tomographic tilt angles, leading to angular corrections about all three axes as shown in (c). These refined angles were then employed in a subsequent ptychotomography reconstruction, yielding a 3D volume rendered in (d). Sub-images in (e) show optical sections from the reconstructed volume cut at indicated z-positions after it is rotated about the y-axis by 30 •.
Funding This research used resources of the  Advanced Photon Source (APS)  and the  Argonne Leadership Computing Facility (ALCF) , which are  U. S. Department of Energy (DOE) Office of Science User Facilities  operated for the  DOE Office of Science by Argonne National Laboratory  under Contract No. DE-AC02-06CH11357. It also used 3ID of the National Synchrotron Light Source II, a  U. S. Department of Energy (DOE) Office of Science User Facility  operated for the  DOE Office of Science by Brookhaven National Laboratory  under Contract No. DE-SC0012704. We also thank the  Laboratory Computing Resource Center (LCRC) at Argonne  for additional computational access. We thank the  Argonne Laboratory Directed Research and Development  for support under grant  2019-0441. We also thank the  National Institute of Mental Health ,  National Institutes of Health , for support under grant  R01 MH115265.
RNA tertiary structure is crucial to its many non-coding molecular functions. RNA architecture is shaped by its secondary structure composed of stems, stacked canonical base pairs, enclosing loops. While stems are precisely captured by free-energy models, loops composed of non-canonical base pairs are not. Nor are distant interactions linking together those secondary structure elements (SSEs). Databases of conserved 3D geometries (a. k. a. modules) not captured by energetic models are leveraged for structure prediction and design, but the computational complexity has limited their study to local elements, loops. Representing the RNA structure as a graph has recently allowed to expend this work to pairs of SSEs, uncovering a hierarchical organization of these 3D modules, at great computational cost.
loops, which are themselves stabilized by more sophisticated base interaction patterns. These networks are essential to shape RNA 3D structures but unfortunately still poorly understood. Here, we undertake the task to build a catalog of base interaction networks occurring in multiple structures. However, a pairwise comparison of all RNA structures is computationally heavy. Therefore, we devise an algorithm leveraging intrinsic properties of RNA base interaction networks that enables us to quickly mine full databases of 3D structures. Compared to previous methods, our techniques bring the total running time of the analysis from months to hours while performing more general searches. The data collected though this work will benefit molecular evolution studies and serve in structure prediction tools.
In synthetic biology, the availability of databases of autonomous structural modules is key for designing new molecules  [8]. The assembly of RNA binding sites may also require bringing together distant modules within the secondary structure  [9]. A comprehensive and indexed catalog of substructures would greatly facilitate studies of these sites. Some RNA modules have received a specific attention such as GNRA loops, Kink-turns, Gbulges, and the various types of A-minors. Moreover, several works have been presented, proposing computational methods to detect RNA modules in tertiary structures using either geometry or graph-based approaches  [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22]. A coarse grain graph representation of the secondary structure with its pseudoknots has already shown the modularity of ribosomal structures  [23] , and has been used recently for fragment based design applications  [24].
However, the purpose of the majority of those methods is to search for known modules in new structures. A couple of methods has been proposed that search local modules without any prior knowledge of their geometry or topology  [11, 15]. In addition to those methods, databases of RNA modules found in experimentally determined RNA tertiary structures have been proposed such as RNA 3D Motif Atlas  [5]  and RNA Bricks  [25]. We are interested in the whole landscape of RNA modules (known or not) rather than any RNA module in particular which distinguishes us from most of the works previously mentioned. Furthermore, we aim at extracting recurrent patterns in the secondary structure rather than in the sequence or in the tertiary structure. Those patterns capture a topological information that has been associated with similar tertiary structures and can be in turn used to derive consensus sequences.
As previously highlighted in several key structural studies  [26, 27] , they are therefore interesting RNA modules candidates that warrant further and more quantitative investigations. Our goal is to automatically and comprehensively capture this topological information to accelerate research in area. To our knowledge, the only published method similar in those aspects is CaRNAval  [28] , one of our previous work. In CaRNAval, we presented an algorithm to find all identical In  (1)  we show an RNA and its secondary structure with non-canonical interactions. Base pair interactions in blue are local (both nucleotides involved are in the same or in adjacent SSEs) while the ones in red are long range interactions (between two distant SSEs). The canonical base pair interactions are represented with double lines.
We made the results of CaRNAval available as an extensive organized catalogue of the Recurrent Interaction Networks (RINs) computed on all the nonredundant structures available in RNA3DHub  [29]. The method developed for CaRNAval is automated and does not use any prior knowledge of neither the topology nor the geometry of the structures it detects. CaRNAval was limited in its capacity to capture the whole landscape of RNA modules. The algorithmic work presented in this paper aims to remedy to this. Indeed, by approaching RNA secondary structures as graphs equipped with a proper edge coloring, we designed several graph matching algorithms and used them as the core of a modular automated pipeline. Leveraging the proper edge coloring of a structure graph allows to improve execution time a hundredfold compared to CaRNAval.
We introduce an abstract type/label b35 to complement the b53 label. We can thus define a bijection ι as follow: • ι(xYZ, C) = xZY, C • ι(xYY, C) = xYY, C • ι(b53, local) = b35, local • ι(b35, local) = b53, local An interaction of type t between nucleotides a,b (represented by nodes v a ,v b ), is represented by two directed edges {v a , v b } and {v b , v a } whose respective labels are t and ι(t). This property is important as a requirement of the algorithms we designed (cf. Section 5. 1 in S1 Text). We represent each RNA chain in the dataset as a RNA 2D structure graph, the annotations of the RNA base pair interactions corresponding to the labels of the edges of the graph (cf.
Exceptions We observed a few nucleotides annotated with two interactions involving the same Leontis-Westhof edges in some RNA structures (0. 02% of the nucleotides of our reference dataset cf. section 3. 1). Those interactions could either be annotation errors or biologically relevant. Given the rarity of those exceptions, we chose to duplicate the graphs concerned into different proper edge-colored versions, each covering a different interpretation. Details about the duplication procedure and the different versions are provided S2 Text.
From common subgraphs back to RNA structures By transposing the RNA structures to graphs and using our algorithms, we are thus able to obtain the set of All Maximal Common Subgraphs contained in any given dataset. However, the number of Maximal Common Subgraphs grows exponentially with the size of the dataset and quickly exceeds human capacities. As a consequence, we designed a restriction system to define more human-sized subsets of structural elements and designed our method to extract and organize such subsets specifically rather than the whole set All Maximal Common Subgraphs. Those subsets of structural elements are to be defined by users through rules or restrictions, according to the types of structures they want to study. This figure illustrates the extension process from a "starting point" (here ((g 0 , h 0 ), (g 0 , h 0 )), in blue).
Recurrent Interaction Network (RIN). We call Recurrent Interaction Network (RIN) any recurrent subgraph of RNA 2D Structure Graphs (i. e. observed in at least two RNAs of the dataset). A RIN is formally defined as a pair ðS; OÞ with: • S = {V S , E S } a connected graph with the properties of a RNA 2D structure graph • O a collection of occurrences. An occurrence records an observation of S in the dataset. We represent an occurrence as a pair (G, i) with G = {V G , E G } a RNA 2D structure graph and i an injection from V S to V G that respects the edge labels. • 9ðG; iÞ; ðH; i 0 Þ 2 O s. t. G 6 ¼ H (i. e. it should be recurrent) This minimal set of properties defines the RIN � class which can be seen as the mother-class from which all other classes are derived by adding additional restrictions.
Note that we will be using class to refer to subsets of structural elements from now on as the relations between subsets are similar to the ones between the classes of a class-oriented langage. To illustrate this let us consider a set of additional rules/restrictions R, designed to invalidate some structural elements we are not interested in. R thus defines RIN R which is a subclass of RIN �. For our method to extract RIN R from a dataset, R is to be translated into a filtering function f R : G ! C RIN R with G a graph that shares the same properties as an RNA 2D structure graph and C RIN R the collection of RIN R s in G that respects the rules in R (the properties defining RIN � are "built-in"). To put simply, the role of f R : G ! C RIN R in the pipeline is to extract the RIN R s from the maximal common subgraphs.
Additionally, we offer the possibility of providing a second filtering function f 0 R : G ! G 0 that takes as input an RNA 2D structures graphs G in the dataset and outputs another graph G 0 , which is a subgraph of G without the edges and nodes in G that already infringe a rule of R (and thus cannot possibly be part of any valid RIN R ). f 0 R is optional as it only improves performances by reducing the search space, albeit greatly in most cases. We will be using RIN R in the following sections to denote an arbitrary class of RINs currently being extracted. 2. 5. 2 Extraction of RIN R. For every pair of RNA 2D Structure Graphs in the dataset (after the application of f 0 R , if provided), we use our algorithm solving the maximal common subgraphs problem to extract the set of all maximal common subgraphs between the two graphs (as illustrated in  Fig 6).
Results & discussion In this section, we present the results of different applications of our method that fall into two categories. First are the results obtained from the dataset used in CaRNAval  [28]  that aim at validating our method, at illustrating the flexibility of the method in regards of defining families of substructures and at evaluating the impact of consecutive relaxations of constraints over the same dataset. Second are the results obtained from a recently published dataset that constitute an up-to-date corpus of structures.
Datasets We use two different datasets of RNA structures. Both datasets are produced from the nonredundant RNA database maintained on RNA3DHub  [29]. The difference between the two datasets is the version of this database used: the first dataset is based on version 2. 92 (Sept. 9 th 2016) whereas the second dataset is based on version 3. 137 (Jul. 29 th 2020). Our motivations in using two datasets (that will be referred as dataset 2. 92 and dataset 3. 137 from now on) instead of just using the more recent dataset 3. 137 lie in that dataset 2. 92 was the one used in CaRNAval  [28]. As a consequence, dataset 2. 92 was necessary to draw any meaningful comparison with CaRNAval. The non-redundant RNA database maintained on RNA3DHub  [29]  contains all-atom molecular complexes with a resolution of at worse 3Å(843 for version 2.
Assessment of the method on Dataset 2. 92 3. 3. 1 Reproduction of previous work. A natural first step in the evaluation of our method is to verify if it is able to reproduce the results presented in CaRNAval. In the notation s , it displays 337 structures. Indeed, it appears during our evaluation that 4 RINs were actually invalid and should not have have passed the filters of CaRNAval. The absence in our results actually validates our method. The 2 last RINs are a special case: they have only 2 observations with both observations inside a single RNA chain (whereas our definition requires at least two occurrences from distinct RNA chains). As such they are valid RINs but invalid RIN abc s. As a conclusion, our method reproduces previous results perfectly as the only discrepancies were due to either errors in said previous results or modifications in definitions.