dataesr/scientific-paragraphs-categorization huggingface dataset:
As it was necessary to redesign the overall system, we tried to avoid the intrinsic schema limitations of relational database systems that hindered development of the TAGDB in the past and considered the data storage technologies that were developed in the academic and commercial computing world in recent years  [8]. The technologies of "NoSQL" databases are used by large commercial information companies and are well adapted to this type of applications. At the same time, one can separate the parts of the project that can be of interest to other generic scientific applications from the parts closely related to ATLAS, and create a system that can be potentially useful for several other scientific and technical applications.
3 The project starts from the knowledge accumulated over the years with the development and operation of the ATLAS TAGDB  [1] -  [7]. The work has been divided into several stages: • Studies of the patterns of data storage in terms of functionality, scalability and sustainability of the various types of NoSQL databases currently available; local tests of functionality and distributed test of scalability; system architecture design of both the database and the auxiliary services to enter the data and facilitate the data mining and extraction. • Implementation of a catalogue in the selected technology; population of the database with all LHC Run1 ATLAS events; complete functional tests of the scalability of databases containing billions of records; comparison of performance with the ATLAS TAGDB.
Adaptation and consolidation of services, including the automatic loading of data from each phase of reconstruction and event selection. • Definition of public interfaces to load, search and retrieve the data; packaging and release of the produced software so that it can be used by other scientific communities. The most innovative part of this project is the adaptation and application of the NoSQL technology for the cataloguing of data of a large experiment. ATLAS alone accumulated in 2011-2012 2⋅10 9 real and 6⋅10 9 simulated events per year. If the production of events were uniform in time, this would correspond to a rate of creation of the records in TAGDB of about 100 Hz. Since each event is processed multiple times, the number of records and the record creation rate in TAGDB is considerably larger.
Each reconstruction campaign produces new versions of every event, in different formats. For ATLAS, they are ESD (Event Summary Data, full reconstruction output), AOD (Analysis Object Data, summary of reconstruction to be used for physics analysis) and several versions of ROOT n-tuples  [9] -  [10]. For each reconstruction campaign and for each format one can generate a key-value pair (where the value is always the navigational information, i. e. the identifier of the file that contains the event in question and the internal structured pointer) and add it to the original record. The result is that an event will always correspond to one and only one logical record in the database that contains the entire processing history of the event. Two main applications must be developed simultaneously with the back-end database: the infrastructure to populate the database and the tools to search for the records and extract the.
Core architecture NoSQL technologies offer several advantages over relational databases for applications like the EventIndex: they scale linearly with the amount of data, there are many external tools to optimize data storage and data searches, they use clusters of low-cost Linux servers, including disks, and finally they belong to the world of open source software, so there is no license for the installation and use of these products. Using NoSQL technologies, it is possible to store information for each event in a single logical record. The record is created upon recording of the event from the online system, with initially only limited information, such as event number, run number, time stamp, luminosity block number, which triggers selected the event, and the identifier of the file that contains the event in RAW format.
ATLAS uses the GUID (Global Unique IDentifier) as identifier of each file but any other system can be used, provided it is capable of uniquely associating identifiers and files. This information can be stored as key-value pairs in NoSQL systems, together with the internal file navigation information. As CERN-IT is providing a Hadoop [11] service, and Hadoop and the many associated tools seem to be a good solution for the EventIndex, we started testing the performance of solutions based on plain HDFS (the Hadoop file system) and/or HBase  [12]  (the Hadoop database system), using different data formats. For these initial tests we imported 1 TB of data from the TAGDB, which correspond to the first processing of the full real data collected in 2011. The data are stored in Hadoop map files  2.
The Generator module creates new filesets and registers them into the Catalog. The Creator creates new filesets. The Extender adds information (new columns). The Appender adds new data. The Indexer creates new indices. The Convertor converts filesets to different formats, including the MapFiles. The Accessor allows remote access, through an HTTP server on a dedicated machine; a publicly visible WebService can be accessed through the CLI or the GUI. The Reader performs the search using one of the available tools (Searcher, Scanner or MapReducer), depending on the query.
PoS(ISGC2014)023 PoS(ISGC2014)023 Figure  3  shows the file organization and relationships in the Hadoop file system (HDFS). The data are stored in collections of "filesets", where each collection represents an event collection (grouping all events that have been processed by the same software version). The TagFiles can be files or directories of files. A collection of TagFiles makes a TagSet, each of which has: a master record, pointers to before/after filesets (vertical partitions), slave TagFiles (horizontal partitions) and index TagFiles. The catalogue checks the presence and consistency of all components. Each TagFile is represented by one entry in the HBase table.
Data collection and upload Data to be stored in the EventIndex are produced by all production jobs that run at CERN or on the Grid. For every permanent output file, a snippet of information, containing the file unique identifier and for each event the relevant attributes, has to be sent to the central server at CERN. The estimated rate (in May 2013, during the LHC shutdown) would be ~20 Hz of file records containing ~7 kHz of event records. During data-taking periods these numbers should be doubled; as both the data-taking rate and the Grid processing power are expected to increase by perhaps a factor two by 2015, the system should be ready to accept over 80 Hz of file records and insert into the back-end over 30 kHz of event records (plus contingency). The architecture is based on producers of the EventIndex information on the worker nodes where event data are processed, a messaging system to transfer this information and asynchronous consumers that effectively insert the data in the Hadoop database.
Data queries and retrieval The performance of data query interfaces is crucial for the usability of the EventIndex. For the initial tests, we imported a full year worth of ATLAS data (2011, first processing version) from TAGDB into Hadoop and catalogued them. This corresponds to just over 1 TB of data in CSV format, before internal replication. Simple data scan operations on this dataset using the available Hadoop cluster of 20 nodes take about 15 minutes; we consider this as the worst case, as normally no query would need to read in all data from disk for a full year of data taking. Queries that give the run number and event number (the event picking use case) are very fast; even using the so-far unoptimised "accessor" method these queries return their results in 3. 7 seconds (just "fast").
For comparison, the same queries using a full Map-Reduce job return in 90 seconds. Again, these should be considered as worst cases as they are the results of the first prototype implementation. An important use case is the query (or count) of the events that satisfied a given trigger condition, or a combination of trigger conditions. In ATLAS, every "fired" trigger condition is marked by a bit set to "1" in the relevant trigger word (Level-1, Level-2 and Event Filter). The correspondence between bits in the trigger words and the actual triggers depends on the run and luminosity block, as the list of active triggers and their possible prescale factors is adjusted according to the variations of the LHC luminosity, in order to maximize the collection of useful data for physics analyses. Queries to the EventIndex involving trigger chains need to be first decoded by knowing the trigger menu via the run number and finding in the COMA database.
INTRODUCTION Using genetics information in conjunction with brain imaging data is expected to significantly improve our understanding of both normal and pathological variability of brain organization. It should lead to the development of biomarkers and in the future personalized medicine. Among other important steps, this endeavor requires the development of adapted statistical methods to detect significant associations between the highly heterogeneous variables provided by genotyping and brain imaging, and the development of software components with which large-scale computation can be done. In current settings, neuroimaging-genetic datasets consist of a set of (1) genotyping measurements at given genetic loci, such as Single Nucleotide Polymorphisms (SNPs) that represent a large amount of the genetic between-subject variability, and (2) quantitative measurements at given locations (voxels) in three-dimensional images, that represent e.
However, some substantial challenges still have to be addressed to fully exploit the power of cloud infrastructures, such as data access, as it is currently achieved through high latency protocols, which are used to access the cloud storage services (e. g. , Windows Azure Blob). To sustain geographically distributed computation, the storage system needs to manage concurrency, data placement and inter-site data transfers. We propose an efficient framework that can manage inferences on neuroimaging-genetic studies with several phenotypes and permutations. It combines a MapReduce framework (TomusBLOB,  Costan et al. , 2013)  with machine learning algorithms (Scikit-learn library) to deliver a scalable analysis tool. The key idea is to provide end-users the capability to easily describe the statistical inference that they want to perform and then to test the model on their own computers before running the very same code in the cloud at a larger scale.
Scikit-learn Scikit-learn is a popular machine learning library in Python  (Pedregosa et al. , 2011)  designed for a multicore station. In the Scikit-learn vocabulary, an estimator is an object that implements a fit and a predict method. For instance a Ridge object (lines 12-13 of Figure  1 ) is an estimator that computes the coefficients of the ridge regression model on the train set and uses these coefficients to predict data from the test set. If this object has a transform method, it is called a transformer. For instance a SelectKbest object (lines 10-11 of Figure  1 ) is a transformer that modifies the input data (the design matrix X) by returning the K best explanatory variables w. r. t. a scoring function. Scikit-learn defines a Pipeline (lines 8-13 of Figure  1 ) as the combination of several transformers and an final estimator: It creates a combined estimator.
Finally, we put some constraints on the description of the machine learning estimator and the cross validation scheme: • The prediction score is computed using the Scikit-learn cross_val_score function and the folds for this cross validation loop are generated with a ShuffleSplit object. • An estimator is described with a Scikit-learn pipeline with one or more steps. • Python can dynamically load modules such that a program can execute functions that are passed in a string or a configuration file. To notify that a string contains a Python module and an object or function to load, we introduce the prefix DYNAMIC_IMPORT:: • To select the best set of parameters for an estimator, model selection is performed using Scikit-learn GridSearchCV and a 5-folds inner cross-validation loop.
Full example (cf. script in Figure 1) • General parameters (Lines 1-3): The model contains covariates, the permutation test makes 10,000 iterations and only one permutation is performed in a task. 10,000 tasks per brain target phenotypes will be generated. • Prediction score (Lines 4-7): The metrics for the cross-validated prediction score is R 2 , the cross-validation loop makes 10 iterations, 20% of the data are left out for the test set and the seed of the random generator was set to 0. • Estimator pipeline (Lines 8-13): The first step consist in filtering collinear vectors, the second step selects the K best features and the final step is a ridge estimator. • Parameters selection (Lines 14-16): Two parameters of the estimator have to be set: the K for the SelectKBest and the alpha of the Ridge regression.
The cloud infrastructure is composed of multiple data centers, which integrate heterogeneous resources that are exploited seamlessly. For instance, the Windows Azure cloud has five sites in United States, two in Europe and three in Asia. As resources are granted on-demand, the cloud gives the illusion of infinite resources. Nevertheless, cloud data centers face the same load problems (e. g. , workload balancing, resource idleness, etc. ) as traditional grids or clusters. In addition to the computation capacity, clouds often provide data-related services, like object storage for large datasets (e. g. , S3 from Amazon or Windows Azure Blob) and queues for short message communication.
NEUROIMAGING-GENETICS COMPUTATION IN THE CLOUD In practice, the workload of the A-Brain application 1  is more resource demanding than the typical cloud applications and could induce two undesirable situations: (1) other clients do not have enough resource to lease on-demand in a particular data center; (2) the computation creates performance degradations for other applications in the data center (e. g. , by occupying the network bandwidth, or by creating high number of concurrent requests on the cloud storage service). Therefore, we divide the workload into smaller sub-problems and we select the different datacenters in collaboration with the cloud provider. For balancing the load of the A-Brain application, the computation was distributed across four deployments in the two biggest Windows Azure datacenters.
In the cloud context, a deployment denotes a set of leased resources, which are presented to the user as a set of uniform machines, called compute nodes. Each deployment is independent and isolated from the other deployments. When a compute node starts, the user application is automatically uploaded and executed. The compute nodes of a deployment belong to the same virtual private network and communicate with the outside world or other deployments either through public endpoints or using the cloud storage services (i. e. , Windows Azure Blob or Queue). TomusBlobs  (Costan et al. , 2013)  is a data management system designed for concurrency-optimized PaaS-level (Platform as a Service) cloud data management. The system relies on the available local storage of the compute nodes in order to share input files and save output files.
IMAGEN: A NEUROIMAGING-GENETIC DATASET IMAGEN is a European multi-centric study involving adolescents  (Schumann et al. , 2010). It contains a large functional neuroimaging database with fMRI associated with 99 different contrast images for 4 protocols in more than 2000 subjects, who gave informed signed consent. Regarding the functional neuroimaging data, we use the Stop Signal Task protocol  (Logan, 1994)  (SST), with the activation during a [go wrong] event, i. e. , when the subject pushes the wrong button. Such an experimental contrast is likely to show complex mental processes (inhibition failure, post-hoc emotional reaction of the subject), that may be hard to disentangle. Our expectation is that the amount of Blood Oxygen-Level Dependent (BOLD) response associated with such events provides a set of global markers that may reveal some heritable psychological traits of the participants.
Eight different 3T scanners from multiple manufacturers (GE, Siemens, Philips) were used to acquire the data. Standard preprocessing, including slice timing correction, spike and motion correction, temporal detrending (functional data) and spatial normalization (anatomical and functional data), were performed using the SPM8 software and its default parameters; functional images were resampled at 3 mm resolution. All images were warped in the MNI152 coordinate space. Obvious outliers detected using simple rules such as large registration or segmentation errors or very large motion parameters were removed after this step. BOLD time series was recorded using Echo-Planar Imaging, with TR = 2200 ms, TE = 30 ms, flip angle = 75 • and spatial resolution 3 × 3 × 3 mm. Gaussian smoothing at 5 mm-FWHM was finally added.
Contrasts were using a standard linear model, based on the convolution of the time course of the experimental conditions with the canonical hemodynamic response function, together with standard high-pass filtering procedure and temporally auto-regressive noise model. The estimation of the first-level was carried out using the SPM8 software. T1-weighted MPRAGE anatomical images were acquired with spatial resolution 1 × 1 × 1 mm, and gray matter probability maps were available for 1986 subjects as outputs of the SPM8 New Segmentation algorithm applied to the anatomical images. A mask of the gray matter was built by averaging and thresholding the individual gray matter probability maps. More details about data preprocessing can be found in  Thyreau et al. (2012). DNA was extracted from blood samples using semiautomated process.
THE EXPERIMENT The aim of this experiment is to show that our framework has the potential to explore links between neuroimaging and genetics. We consider an fMRI contrast corresponding to events where subjects make motor response errors ([go wrong] fMRI contrast from a Stop Task Signal protocol). Subjects with too many missing voxels or with bad task performance were discarded. Regarding genetic variants, 477,215 SNPs were available. Age, sex, handedness and acquisition center were included in the model as confounding variables. Remaining missing data were replaced by the median over the subjects for the corresponding variables. After applying all exclusion criteria 1459 subjects remained for analysis. Analyzing the whole brain with all the genetic variants remains intractable due to the time and memory requirements and dimension reduction techniques have to be employed.
The cloud experimental setup The experiment was performed using the Microsoft Windows Azure PaaS cloud in the North and West US datacenters, that were recommended by the Microsoft team for their capacity. We use the Windows Azure storage services (Blob and Queue) in both datacenters in order to take advantage of the data locality. Due to our memory requirements, the Large VM type (4 CPU cores, 7 GB of memory and 1000 GB of disk) is the best fit regarding the Azure VMs offer 2.
The result is a simple generic code (few lines) that provides the user a quick way to conduct early, small-scale investigations on its own computer or at a larger scale on a high-performance computing cluster. With JSON we provide a standard format for the description of statistical inference so that no programming skills are required and so that it can be easily generated from a webpage form. (2) TomusBLOB permits to execute seamlessly the very same code on the Windows Azure cloud. We could also disable some parts of TomusBLOB to achieve a good compromise between the capabilities and the robustness. We demonstrate the scalability and the efficiency of our framework with a 2 weeks geographically distributed execution on hundreds of virtual machines. The results confirm that brain activation signals are an heritable feature.
ACKNOWLEDGMENTS This work was supported primarily by the  Microsoft INRIA  joint centre grant A-brain and secondarily by the  Digiteo ICoGeN grant  and the  ANR  grant  ANR-10-BLAN-0128. HPC investigations were carried out using the computing facilities of the  CEA-DSV  and CATI cluster. The data were acquired within the  IMAGEN  project, which receives research funding from the  E. U. Community 's  FP6 ,  LSHM-CT-2007-037286. This manuscript reflects only the author's views and the Community is not liable for any use that may be made of the information contained therein.
Introduction With the increasing complexity of systems on chips (SoC) and time-to-market pressure, rising the abstraction level to ESL (Electronic System Level) modeling is becoming indispensable. Platform-based design emerges as a new paradigm for the design and analysis of embedded systems  [1]. In that context, SystemC TLM (Transaction Level Modeling)  [2]  is gaining acceptance, in particular because the simulation of TLM models is several orders of magnitude faster, thus considerably improving productivity in SoC design  [3]. SystemC  [4]  is in fact a C++ library; a SystemC model is made of modules that have I/O ports and can run processes. They are interconnected using channels. A SoC model includes modules that are models of microprocessors (ISS, Instruction Set Simulator) which execute embedded software, busses, memories, hardware coprocessors like DSP (Digital Signal Processor), DMAs (Direct Memory Access, components that perform memory transfers),.
Then, the SystemC simulation is run as usual, and the assertion checkers dynamically report property violations, if any. In that context, simulation traces must be sampled according to communication actions involved in the assertions, hence the observation mechanism detects the communication functions calls and triggers the monitors accordingly. A prototype tool called ISIS has been developed; experimental results on industrial case studies have been reported in  [8] ,  [9]. However, this original version of the tool offers few flexibility to the user. It is only possible to select, at the SystemC compile time, the assertions/monitors to be attached to the design. We present here conceptual extensions of the underlying observation model, and their implementation, that enable the designers -in particular the software developers -to customize and optimize the verification process and to get concise and easily analyzable verification results.
the SystemC platform is used by the software designer to develop the application, but he usually does not wish to recompile the platform (or even has it only under executable form). Hence selecting the assertions to be attached to the design should be feasible at runtime, not at the SystemC compile time, more essentially, it should be possible to configure the instrumented simulation to take into account relationships between assertions to dynamically disable/enable monitors, thus clarifying and optimizing the verification, it should also be possible to store concise information about assertions activations, satisfactions, violations, and to easily analyze verification results.
Configuration without Platform Recompilation The assertion checkers generated from the PSL expression of the requirements, and the observation mechanism, are SystemC modules that instrument the Sys-temC platform. ISIS takes as input the platform source files and an XML configuration file that specifies the selected assertions, and produces this instrumented design. With the original version of the tool, the user must execute again this procedure, and recompile the resulting platform, every time he wants to choose other requirements to be checked. The tool has been improved to generate an instrumented source code that import all the assertion checkers but enables their optional runtime instantiation, according to choices given in a configuration file.