dataesr/scientific-paragraphs-categorization huggingface dataset:
This practice might have some problems such as bug replications that increase maintenance costs  [11]. Ideally, reuse should correspond to a systematic practice where the language constructs that are used in more than one DSL are defined in interdependent language modules that can be used as plug-in pieces during the DSLs development process. In this paper, we present Puzzle, a tool to assist refactoring processes intended to remove specification clones in a given set of legacy DSLs. More precisely, Puzzle offers the following features: Detection of specification clones. Puzzle provides a set of comparison operators that enable automatic detection of specification clones in a given set of DSLs. These operators take into account not only the names of the constructs, but also the inter-constructs relationships and their semantics.
Additionally, the implementation of Puzzle is flexible enough to permit the definition of new comparison operators. Hence, the detection strategy can be easily improved or adapted to particular contexts. Quantification of potential reuse. Puzzle comes out with a set of metrics (inspired in  [1] ) to quantify the potential reuse emerging from the existing specification clones. The objective is to provide a mechanism that allows language designers to estimate (in an objective fashion) the benefit of a refactoring process intended to remove specification clones in a given set of DSLs. For example, Puzzle measures the amount and percentage of language constructs cloned in a set of DSLs, as well as how different is a given DSL with respect to the others. All these metrics are presented in the form of charts implemented as HTML reports that can be easily shared and published.
Extraction of reusable language modules. Puzzle enables a reverse engineering process to extract reusable language modules from the detected specification clones  [8]. This strategy is based on a principle illustrated in Figure  1 : if a DSL specification is viewed as a set of specification elements, then specification clones can be viewed as sets overlapping, and reusable language modules can be obtained by breaking down that overlapping  [10]. The language modules resulting from this refactoring process can be later assembled in the construction of new DSLs. Architecture. The architecture of Puzzle is composed of two parts illustrated in Figure  2 : the infrastructure and the superstructure. The infrastructure is a set of plug-ins that enable the specification of DSLs according to the technological space described above.
In turn, the superstructure is a set of plug-ins that provides analysis and reverse-engineering techniques on the DSLs specified on top of the infrastructure. The Puzzle's infrastructure is based on the Eclipse Modeling Framework (EMF). EMF provides a modeling language, called Ecore, which we use to specify metamodels. In turn, we use the notion of aspects provided Kermeta  [6]  to specify operational semantics. An aspect encapsulates a set of domain-specific actions that are weaved into a metaclass of the metamodel. The mapping between metamodels and aspects is specified in Melange 1. In turn, the superstructure of Puzzle corresponds to a set plug-ins that can be divided into three categories according to their functionalities: comparison, metrics, and reverse-engineering. Comparison plug-ins implement the comparison operators needed to detect specification clones at the level of abstract syntax and semantics (for the case of comparison of semantics, Puzzle uses JCCD  [2].
The metrics plug-ins compute a set of metrics for the detected specification clones and present the results as a set of HTML reports that display those metrics in the form of charts. The reverse-engineering plug-ins implement the algorithms that extract reusable language modules from the detected specification clones. Tool demonstration. In the rest of this section, we provide three videos (available in the papers' website 2  ) that show the way in which a set of DSL defined in the Puzzle's infrastructure is analyzed by the Puzzle's superstructure. The Puzzle's source code is available in the project's website  3. The input of Puzzle is a Melange script that references a set of DSLs. The analysis starts by comparing the DSLs specifications (at the level of the abstract syntax and the semantics) and produces a first report indicating whether there are any specification clones or not.
Suboptimal structures A straightforward way to deal with uncertainty in structure prediction is to generate a set of plausible structures instead of a single optimal one. The first approach to suboptimal structure was to produce a small set of (hopefully) representative structures that are optimal given that one base pair is enforced. This method was suggested independently by Steger et al. [104]  and Zuker  [144] , and made popular by the program mfold  [144]. A different, more exhaustive but computationally more expensive method was introduced in RNAsubopt  [136] , which enumerates all secondary structures within an energy increment d from the MFE.
The essential point lies in using a unique decomposition of the secondary structure space, ensuring that no structure is counted twice. This paved the way to apply a broad variety of statistical methods from thermodynamics to RNA secondary structures, such as the computation of base pair probabilities  [76]  and statistical sampling of secondary structures according to their equilibrium probabilities  [19]. However, the first practical implementation of the partition function and base pair probability computations that could be applied to RNAs of reasonable size became available in 1994 with the RNAfold program  [44].
The idea was extended in LocARNA  [132, 130]  to also reduce the memory consumption from quartic to quadratic. RAF  [22]  restricts both structure and alignment search space to achieve quadratic run time. The recent SPARSE  [131]  method achieves quadratic run time even without sequence-based heuristics. While most of the Fig. 1. Secondary structure predictions for the spliced leader RNA from Leptomonas collosoma  [59]. Displayed are secondary structures predicted by various methods, such as MFE, ensemble centroid, MEA structure, as well as suboptimal structures obtained from stochastic backtracking (marked by S), and the 5 best suboptimals sensu Zuker (marked by Z), all implemented in the programs RNAfold, and RNAsubopt of the ViennaRNA Package  [44, 64]. To account for their respective pairwise base pair distance, a hierarchical cluster tree is shown.
Each possible move x ! y is associated with a rate k yx. For RNA, the simplest move set consists of opening or closing of a single base pair. Folding Solving Eq. 11 directly is impractical for anything except toy examples, since the dimension of the rate matrix is equal to the number of possible structures. One possibility to address this issues is to perform stochastic simulations of RNA folding using a Monte Carlo method. This approach is taken e. g. by Kinfold  [27]  and KineFold  [50]. While the outcome of this method can be regarded as a gold standard, computing and analyzing a large number of trajectories can be time consuming and tedious. An alternative approach is available through direct investigation of the energy landscape in terms of local minima, energy barriers and transition rates, as done by the barriers program  [29].
Combining experimental data with RNA secondary structure prediction As many RNA structure probing methods became a mainstream technology, the demand for efficient and precise methods to combine them with computational methods in RNA structure determination is evident. Chemical probing, such as SHAPE, or DMS, usually yields per-nucleotide reactivities that, to some extent, reflect the structural context of a nucleotide. These reactivities are then used to either guide in silico RNA structure prediction methods directly, or determine which representatives fit the experiments best. Today, several approaches to incorporate chemical probing data into thermodynamics-based computational tools have been suggested  [31]. Available programs that allow for probing data guided structure prediction include Fold of the RNAstructure package  [92] , the MC-Fold/ MC-Sym  [84]  pipeline, RNAsc  [139] , RNApbfold  [123] , SeqFold  [83] , and Struc-tureFold  [115].
After all, empirical data on SHAPE reactivities shows, that both, paired and unpaired nucleotides, are more likely to have low reactivities  [107]. Thus one can not directly infer a small likelihood to be unpaired from low reactivity. A more recent implementation that takes up the idea of Eddy  [26]  to incorporate SHAPE, DMS, and PARS data was introduced with the RME program  [135]. Following a Bayesian approach to determine posterior pairing probabilities pðp i j r S ðiÞÞ the authors first fit prior distributions for the respective probing methods from known data. From that, they compute conditional probabilities pðiÞ ¼ pðp i ¼ 0 j r S ðiÞÞ to observe a nucleotide i being paired. Only then, the resulting probabilities are converted into pseudo energies DG S ðiÞ ¼ ÀRT Á m Á log pðiÞ 1 À pðiÞ ð16Þ to guide a partition function computation, where m serves as a scaling factor.
The benchmark set for the prediction consists of about 1900 pseudo-knot free sequence/structure pairs taken from RNA Strand database  [2]. The 95% confidence intervals for PPV, and sensitivity were estimated using bootstrapping with 1000 iterations. As visible in the plot, perfect (one-dimensional) hard constraints extracted from reference structures substantially increases the prediction performance, even if only unpaired positions in the reference are constrained during the prediction. Surprisingly, constraining just paired positions in the reference yields slightly lower PPV, while the sensitivity increases as expected. Application of constraints to both, paired and unpaired positions yields almost perfect predictions. that require adjustment, indicating potential post-transcriptional modification site, or inter-molecular interactions that are not explicitly handled by the nearest neighbor model.
In contrast to the other methods discussed above, the impact of on the diversity of the structure ensemble can be usually considered small, making it applicable to RNAs with several distinct low energy structures. Though, this method has a relatively high asymptotic time complexity of Oðn 4 Þ, uses an ad hoc probing data conversion, and strongly depends on the optimization technique to alleviate exploration of the rugged landscape of solutions. Our implementation of this method in the ViennaRNA Package alleviates these drawbacks by estimating p i ð Þ from structure samples, and allows for a variety of optimization techniques and conversions from probing data into probabilities (see supplementary material, Section 5 of  [65] ).
Limitations and future perspectives In theory, one would expect that perfect one-dimensional probing data, i. e. data that binarily distinguishes between paired and unpaired, yields almost perfect structure predictions. To test this hypothesis, we collected a dataset of about 1900 pseudo-knot free sequence/structure pairs consisting of 25 different classes of RNA from the RNAstrand database  [2] , see Table  1  for the composition of the benchmark set. For each reference structure, we constructed three sets of (one-dimensional) hard structure constraints to (i) prohibit unpaired positions in the reference from being paired, (ii) enforce paired positions in the reference to be paired, and (iii) a combination of both, respectively. We then applied the resulting constraints to MFE structure predictions using RNAfold, and assessed the prediction performance by means of Positive Predictive Value (PPV), and Sensitivity.
Unfortunately, none of the existing approaches takes these findings into account. Therefore, it remains unclear, whether more elaborate methods that distinguish more than two pairing states help to increase prediction performances. We have recently implemented a generic approach for guided structure prediction by means of hard and/or soft constraints into several programs of the ViennaRNA Package  [65]. This allows, for instance, an easy application of SHAPE data using the methods of Deigan et al. [16] , Zarringhalam et al. [139] , Washietl et al. [123]  with the programs RNAfold, RNAsubopt, and RNAalifold.
Acknowledgments This work was partly funded by the  Austrian Science Fund (FWF) :  F43  ''RNA regulation of the transcriptome", and the  joint project ''RNALands " by the  Austrian Scinece Fund (FWF) :  FWF-I-1804-N28  and the  French Agence Nationale de la Recherche   ANR-14-CE34-0011. Dedicated to  Prof. Bernhard Keppler  on the occasion of his 60th birthday.
SVM is a supervised machine-learning algorithm that can predict the category of test datasets based on the knowledge acquired from a learning dataset. The performance (accuracy) of the automated SVM approach was assessed, by comparing the SVM-predicted quality labels to investigator-determined quality labels. The accuracy for classifying 1457 3D-MRI volumes from our database using the SVM approach is around 80%. These results are promising and illustrate the possibility of using SVM as an automated quality assessment tool for 3D-MRI.
One approach to reduce the workload of visually inspecting large number of 3D-MRIs is parallel processing by multiple investigators, such that each investigator inspects a subset of the data. However, such an approach can be unreliable, as each investigator uses a different threshold for accepting or excluding data. Additionally, this approach is time consuming, which makes the task of maintaining and updating the image quality information of large growing 3D-MRI datasets in a timely manner challenging. Therefore, automating the image quality rating procedure can improve the reproducibility, reliability, and efficiency of morphometric analysis, particularly when maintaining large neuroimaging databases  (Cheng et al. , 2009). Mortamet et al. (2009)  implemented a univariate and automated approach using two quality indices to assess the quality of 3D-MRIs.
MATERIALS AND METHODS The analysis pipeline is outlined in Figure  1. In step 1, 1457 T1-weighted 3D-MRIs were retrieved from our group's database (acquisition details are provided in Supplemental Material). In step 2, investigators visually inspected the 3D-MRI volumes. Each 3D-MRI was preprocessed in step 3, in order to extract features in step 4. In step 5, the SVM was used to classify the 3D-MRIs. The performance of SVM-based classification was evaluated by comparing to the visual inspection procedure.
Visual Inspection, a Reference Point of Data Quality The visual inspection procedure was performed in two stages as shown in Figure  2. In Stage I, the 3D-MRI visual inspection task was distributed among a group of investigators, such that each investigator was assigned a subset of 3D-MRIs for initial inspection. Investigators searched each slice for recurring artifacts, such as eye movement, ringing, aliasing, grainy images, head movement, and teeth filling. The severity of each artifact was evaluated based on a 4 • rating scale: heavy, moderate, slight, or none. These descriptions provide details for the visual In Stage II, the green and red 3D-MRI volumes were labeled "usable" and "not-usable, " respectively. A group of 5-9 investigators further classified the yellow 3D-MRI volumes as "usable" or "not-usable" based on a majority vote.
3D-MRI Preprocessing In the remainder of the Methods, I w,i (x, y, z) denotes the intensity at coordinates (x, y, z), in the acquired 3D-MRI w, taken from an individual subject i. All 1457 3D-MRIs I w,i , were preprocessed using the voxel-based morphometry (VBM) toolbox, as implemented in statistical parametric mapping (SPM8) software  (Ashburner and Friston, 2000; Friston et al. , 2003). Briefly, a unified segmentation  (Ashburner and Friston, 2005)  approach was used for tissue classification, image registration, and nonuniformity image intensity corrections. The tissue classification procedure generated six tissue probability maps from each I w,i , denoted as p c,i , where c = { gray matter, white matter, cerebral spinal fluid (CSF), skull, fat, and background }. The image registration procedure generated a transformation matrix H i for each subject i to spatially normalize each 3D-MRI I w,i , from native (x, y, z) to normalized space (x ′ , y ′ , z ′ ).
SVM Classification Each iteration of the supervised classification procedure consisted of a training stage and a testing stage and is illustrated in Figure  8. In step (a), visual inspection categorized the dataset into 1267 "usable" 3D-MRIs and 190 "not-usable" 3D-MRIs. In (b), 190 of the 1267 "usable" 3D-MRI volumes were selected randomly to generate two equal size "usable" and "not-usable" datasets to reduce bias in the classification procedure. In step (c), a 10-fold cross validation procedure was used to generate a training set and a testing set composed of 90% and 10% of the data, respectively. In step (d), the features of the training set along with the visually inspected category were used as input   12 ). The minimum along the y-axis (coronal direction) was computed to define ASF3 as the maximum over the 30 sagittal slices x 1 , as in Equation (  13 ).
into the SVM to generate a classifying hyperplane. In step (e) the hyperplane was then used to classify the features from the testing set as "usable" or "not-usable. " This classification was then compared to the visually inspected category in order to compute accuracy, sensitivity, and specificity of the classification procedure. This entire procedure was repeated 1000 times in order to select different subsets of the "usable" 3D-MRI volumes and account for variability in the dataset.
DISCUSSION Compared to a univariate quality assessment approach that generates a single number, the multivariate approach (together with the global and ROI automated image quality features) presented in this paper is more informative as it provides details that categorize, localize, and quantify the extent of noise in the data. These parameters are key tools for assessing the quality of 3D-MRI in a neuroimaging database, where the brain 3D-MRI volumes are indexed and can be queried according to the artifact type. Moreover, since the features used in the classifier are regional, the affected regions can be highlighted in an automated plugin added to a 3D imaging viewer. This plugin can then be used to improve the efficiency of the quality assessment procedure of 3D-MRIs through visual inspection by helping investigators to focus in on the problematic regions.
Resolve the Misclassified Datasets The reasons for misclassification can either be related to human error, SVM classification error, or a combination of the two. Human error misclassification is an important motivation for this study and can occur for a number of reasons. Visual inspection of structural 3D-MRI volumes in large databases may require that this process be shared by a number of experts rather than an individual investigator, which can introduce inter-rater variability in how the 3D-MRI volumes are labeled. Even if the task were assigned to a single investigator, intra-individual variability though probably less than inter-individual variability is still an issue that needs to be addressed. Misclassification of a 3D-MRI due to machine error suggests the developed methods can be further refined, to improve performance.
In some of the 3D-MRIs of the dataset analyzed, there was aliasing in the axial direction, with 2-3 axial neck slices wrapping around to the top. The ring-mask used to define Equation (  11 ) was bounded at the top to avoid aliasing in the axial direction. While ASF3 was developed to capture aliasing in the coronal direction, this feature can easily be implemented to measure the extent of aliasing in all directions. While the artifact-specific features were developed to target artifacts found in 3D-MRI volumes acquired using a spoiled gradient recalled (SPGR) sequence, they can be used for assessing 3D-MRIs acquired using other pulse sequence. To extract meaningful features, investigators can follow a two-step approach: (1) identify recurring artifacts specific to the acquisition pulse sequence and (2) adapt the automated artifactspecific features to target the most prominent artifacts.
CONCLUSION A novel method to automatically assess the quality of images using a multivariate classifier has been proposed. This is the first study that uses a multidimensional machine-learning algorithm based on automated regional extracted features to automate quality assessment of structural magnetic resonance images (MRI). Compared to a univariate quality assessment approach that generates a single number, the approach presented in this paper is more informative as it provides details that categorize, localize and quantify the extent of noise in the data. By breaking the problem down into smaller problems, features have been developed that individually quantify each of the artifacts that can be used as inputs into the SVM classifier. These parameters are key tools for assessing the quality of 3D-MRI in a neuroimaging database where the brain images are indexed and can be queried according to the different types of artifacts.
ACKNOWLEDGMENTS The authors thank  Drs. Heike Toste ,  Hao Yang Tan ,  Lukas Kemph ,  Fabio Sambataro ,  Roberta Rasetti ,  Eugenia Radulescu ,  Shabnam Hakimi  and  Mr. Saumitra Das  ( Genes, Cognition and Psychosis Program, NIMH ) for their part in the visual inspection and quality ratings of the 3D-MRIs; and  Mr. Brad Zoltick  and  Dr. Yunxia Tong  ( Genes, Cognition and Psychosis Program, NIMH ) for data-basing and IT support.
In such settings, likelihood-free computation or approximate Bayesian computation (ABC)  (Tavaré et al. , 1997; Beaumont et al. , 2002; Marjoram et al. , 2003; Sisson et al. , 2007)  can be employed. ABC methods circumvent the explicit evaluation of the likelihood using a simulation-based approximation. The original ABC algorithm jointly simulates θ θ θ ∼ p(θ θ θ) and D s ∼ p(D s |θ θ θ ) and accept θ θ θ if and only if the simulated dataset D s is close to the observed dataset. The simple ABC scheme suffers from the same shortcamings as other rejection samplers: most of the samples are drawn from regions of parameter space, which cannot give rise to simulation outputs that resemple data. Therefore a number of computational schemes have been proposed to enhance the efficiency of ABC algorithms.
Approximate Bayesian Computation Approximate Bayesian Computation (ABC) methods  (Tavaré et al. , 1997; Pritchard et al. , 1999; Beaumont et al. , 2002)  replace the calculation of the likelihood with direct sampling from the posterior p(θ θ θ|D) ∝ p(D|θ θ θ)p(θ θ θ) through simulation. Given the prior distribution p(θ θ θ) on the parameter space, a candidate parameter vector θ θ θ * is sampled from p(θ θ θ) and a simulated data set D s is generated from the likelihood function p(D|θ θ θ * ). The parameter vector is accepted if the simulated data set is close to the observed data set D o according to a distance function d (d(D o , D s ) ≤ ). The distance function detrermines the discrepancy between the two datasets and is a tolerance value. If the data are too intricate or complicated it is common to replace a comparison of the observed and simulated data by a comparison of suitable summary statistics.