dataesr/scientific-paragraphs-categorization huggingface dataset:
Python Scripts e2pdbs2mrcs. py Automates generation of volumes from atomic coordinates using the EMAN2 python script e2pdb2mrc. py. Can be run in parallel batches. e2classesvsprojs. py Automates comparison of class averages with volume projections using the EMAN2 python script e2classvsproj. py. Can be run in parallel batches. e2classesvsprojs_best_scores. py Identifies the best score and volume projection for each class average as well as the overall best score and volume projection for all class averages using the output of e2classesvsprojs. py. e2classesvsprojs_extract_best. py Extracts the best scoring coordinate files and corresponding image stacks using the output of e2classesvsprojs_best_scores. py. e2classesvsprojs_generate_best_list. py Generates a list of the images for the best scoring volume projection versus class average comparisons using the output of e2class-esvsprojs_best_scores.
py. The resulting list in "fast LST format" can be used as input for compilation of the images into an image stack in EMAN2. extract_rg. py Extracts Rg values embedded in a text file containing output generated by the IMP program rg. filenames_rg. py Combines filenames from one file with Rg values from another. foxs_component_summation. py Sums FoXS partial profiles using c1 and c2 constants from MultiFoXS. foxs_resample. py Resamples a FoXS profile to match q values from a reference profile using linear interpolation. histogram_fractions. py Generates a histogram of values with corresponding fractions after sorting in ascending order. histogram. py Generates a histogram of values after sorting in ascending order. multifoxs_filenames. py Generates a file containing the filenames for input to the command line version of multi_foxs.
Pipelines The following "pipelines" are intended to illustrate the sequence of command line tools and scripts. Although they can be converted to a fully automated shell script if desired, we prefer to run the instructions individually to allow the output at each step to be monitored for quality control. e2classesvsprojs_pipeline. txt Example "pipeline" illustrating sequence of command line instructions used for comparison of 2D class averages with volume projections calculated from a pool of models generated by RRT_SAMPLE. multifoxs_pipeline. txt Example "pipeline" illustrating the sequence of command line instructions used for Multi_FoXS model generation, profile calculation and analysis with the IMP command line tools RRT_SAMPLE, foxs, and multi_foxs. multifoxs_mem_pipeline. txt Example "pipeline" illustrating the sequence of command line instructions used to prepare Multi_FoXS output for MEM in DELA.
ACKNOWLEDGMENTS In memory of  Pierre Chardin , who pioneered the field of small GTPases. We thank  Sagar Kathuria  for help with SAXS data collection,  Gregory Hendricks ,  Lara Stritmatter ,  Chen Xu , and  Roger Craig  at the  UMASS EM core facility  for technical support during EM data collection, and  Steven Ludtke  at  Baylor College of Medicine  for expert advice on EM data processing and analysis with EMAN2. The research on Grp1 was supported by grant  NIH   GM056324  (to D. G. L. ) and used resources of the  Advanced Photon Source , a  U. S. Department of Energy (DOE) Office of Science User Facility  operated for the  DOE Office of Science by Argonne National Laboratory  under contract no. DE-AC02-06CH11357. This project was supported by grant  9 P41 GM103622  from the  National Institute of General Medical Sciences  of the  NIH.
py This paper This paper SASBDB: SASDGC4 https://www. sasbdb. org N/A and MultiFoXS models foxs. sh Malaby et al. , 2018 N/A ARNO 2-400 + IP4 SAXS with DAMMIF, foxs_component_summation_resample. sh This paper This paper SASBDB: SASDG84 https://www. sasbdb. org N/A GASBOR and antiparallel CORAL models foxs_component_summation. py This paper N/A Grp1 14-399 + IP4 NS-EM Volume 1 with best antiparallel model foxs_resample. py This paper This paper EMDB: EMD-20628 PDB: 6U3E http://www. emdatabank. org N/A Grp1 14-399 + IP4 NS-EM Volume 2 with gasbor. sh This paper Malaby et al. , 2018 EMDB: EMD-20629 N/A best antiparallel model histogram. py This paper PDB: 6U3G http://www. emdatabank. org N/A Recombinant DNA histogram_fraction. py This paper N/A Plasmid: Modified pET15 (pDL2) multifoxs_filenames.
Data retrieval systems are facing a paradigm shift due to the proliferation of specialized data storage engines (SQL, NoSQL, Column Stores, MapReduce, Data Stream, and Graph)  supported by varied data models (CSV, JSON, RDB, RDF, and XML). One immediate consequence of this paradigm shift results into data bottleneck over the web; which means, web applications are unable to retrieve data with the intensity at which data are being generated from different facilities. Especially in the genomics and healthcare verticals, data are growing from petascale to exascale, and biomedical stakeholders are expecting seamless retrieval of these data over the web. In this paper, we argue that the bottleneck over the web can be reduced by minimizing the costly data conversion process and delegating query performance and processing loads to the specialized data storage engines over their native data models.
We propose a web-based query federation mechanism-called PolyWeb-that unifies query answering over multiple native data models (CSV, RDB, and RDF). We emphasize two main challenges of query federation over native data models: 1) devise a method to select prospective data sources-with different underlying data models-that can satisfy a given query and 2) query optimization, join, and execution over different data models. We demonstrate PolyWeb on a cancer genomics use case, where it is often the case that a description of biological and chemical entities (e. g. , gene, disease, drug, and pathways) spans across multiple data models and respective storage engines. In order to assess the benefits and limitations of evaluating queries over native data models, we evaluate PolyWeb with the state-of-the-art query federation engines in terms of result completeness, source selection, and overall query execution time.
I. INTRODUCTION The database experts have predicted the demise of ''One Size Fits All'' approach used in the data retrieval and management solutions  [1]. This prediction is now evident, as in the last couple of years several data models (e. g. , text, CSV, Graph, JSON, RDB, RDF, XML) and storage engines are proliferating with overlapping requirements, use-cases, and user-bases. This is particularly true in complex domains like healthcare and life sciences (HCLS) where the organisations are facing a major shift in the data retrieval requirements. A simultaneous use of different specialized data storage engines, data models, and supporting querying mechanism is needed to retrieve data from different interacting facilities (clinical measurement, medical history, laboratory test, demographics, etc.
There is wider availability of open-source and commercial tools that allow curating, publishing, aggregating, storing, and querying Linked Data. The typical linked data approach to query independent data silos is to convert all the underlying native data models into the RDF data model and devise querying mechanism through which these independent data silos can be queried in unison. While this approach can be practical for simpler verticals, in case of the HCLS domain it is already predicted that 2-40 exabytes of storage capacity will be needed by 2025 just for the human genomes which will continue to grow approximately 40 petabytes of additional genomic information each year  [4]. Nevertheless, raw storage is not the main concern, but the amount of variant data (text, relational, stream, graph, etc.
) being queried and analyzed is already seen as a major hurdle in the meaningful use of this vast amount of data. The normal federation approaches evaluate a given query over multiple data stores complying to a single data model. Recently, an atypical approach of federating queries over heterogeneous data models has been initiated -called Polystore 1   [5]  -that exploits different data models and storage engines in their native formats, i. e. , without converting them to a common data model. The early demonstrators of Polystore have shown promising outcomes in federating queries across disparate data models used in the Multiparameter Intelligent Monitoring in Intensive Care II Databases (MIMIC II)  [6]. It's important to note that, the concept of Polystore is an abstract idea of unifying querying over multiple data models which can be implemented using different technologies (RDB, Web, or Semantic technologies).
In this paper, we present a semantic approach, called PolyWeb, to federate query over Web polystores containing cancer and biomedical data sets. We devise a query federation approach that focuses on source selection and joins over different data models (e. g. , CSV, RDB, RDF). It is an open research problem to perform join across different data models. Further, it is important to understand the gain and loss of querying over data sources 2 in their native data models compared to the conventional approach of querying curated data sources -from several heterogeneous sources -using a common data model. We compare PolyWeb against the state-of-the-art SPARQL query federation engines. The rest of the paper starts with the description of a motivation scenario that requires to query over disparate cancer genomics data sets.
In the related work section, query federation approaches developed using relational database and semantic web methods are discussed. We then provide a brief overview of the PolyWeb architecture and discusses in detail 1 http://wp. sigmod. org/?p=1629 2 A data source encapsulates a data set complying to a particular data model/format (CSV, JSON, RDB, RDF, etc. ) the two algorithms designed to select prospective data sets, optimize, plan and execute a given query. The correctness and complexity of the PolyWeb algorithms are also discussed. We provide an extensive evaluation of PolyWeb by comparing it with the state-of-art query federation engines. Finally, we conclude our work and provide future directions to further enhance our work.
II. MOTIVATING SCENARIO -CANCER GENOMICS The Next Generation Sequencing (NGS) technologies are producing a massive amount of sequencing data sets. As said, there will be a top-up of approximately 40 petabytes of genomic information every year from a wide variety of data sources (hosting different databases, data formats, etc. ) published by human genome research centers worldwide. Often, these data sets are published from isolated and different sequencing facilities. Consequently, the process of sharing and aggregating multi-site sequencing data sets are thwarted by issues, such as the need to discover relevant data from different sources, built scalable repositories, the automation of data linkage, the volume of the data and efficient querying mechanism. PolyWeb is motivated by the needs of the BIOOPENER project  3  which is aiming to link cancer and biomedical data sets providing interlinking and querying mechanisms to understand cancer progression from normal to diseased tissue with pathway components, which in turn helped to map mutations, associated phenotypes, and diseased pathways  [7].
In cancer genomics, discoveries of biological and chemical entities (gene, pathway, drug, diseases, etc. ) are available in several overlapping data sources containing complex genomic features, studies, and associations of such features. In order to understand the tumorigenesis, it is often the case that several genetic features, diseases, medical history, etc. are studied together. Considering the exponential growth and variety of genomics and biomedical data sets, it is impractical to assume that all these isolated and disparate data sets will be available in a single data model. In order to tap the vast knowledge locked in these data sets, it is now important to exploit them in native formats. The example SPARQL query shown in Listing 1 retrieves association of a gene (MYH7) with a primary site (where tumor progression starts), disease, copy number variation (CNV), CNV location (start, end), CNV segment mean, and reported samples of patients.
The CNV information gives insight into the structural variation of a gene which helps analyses the progression of the cancer tumor, ultimately impacting on the prognosis and treatment of disease. Fig. 1  shows an unoptimised query plan that includes the type of databases or models (e. g. , CSV, RDB, RDF) that can satisfy individual triples from three data sources. In our previous work, we developed a SPARQL federation engine -called SAFE  [8]  -that federates queries over genomics and clinical trial repositories represented in RDF. Similarly, in our previous work, we evaluated a wide variety of proposals (FedX, LHD, SPLENDID, FedSearch, GRANATUM, Avalanche, DAW, ANAPSID, ADERIS, DARQ, LDQPS, SIHJoin, WoDQA and Atlas) on how to execute SPARQL queries in federated settings  [9]. However, we have no work to compare that federates over different native data models.
Thus the motivation for our research is to investigate how to enable query federation over native data models in Web like open-world scenarios. As argued in the introduction, off-the-shelf federation engines cannot be used since they are designed to federate over a single data model. Hence following are the core research questions we address in this paper: • How can we devise source selection in a Web like federated scenario where data sources comply with different data models? • How can we implement a web-based query federation plan that retrieves correct and complete results from different data models? • Can we optimize and execute the query federation plan for different native data models in a manner that allows us to compete with off-the-shelf RDF query federation engines? The benefit of PolyWeb is two-fold: (i) It reduces the data conversion cost: a given query is executed over the native data models without converting multiple data models to a common data model.
Therefore, the major advantage of PolyWeb holds in all the data retrieval scenarios where the data conversion cost is high. For instance, SPARQL-based federation engines listed in Table  1  execute a given query on the requirement (or assumption) that data sources are available in RDF; and (ii) it delegates data querying load to specialized data storage engines, instead of loading curated data from multiple data models to a general-purpose engine. In this article, we demonstrate that PolyWeb is helpful in reducing the data conversion cost and still be able to retrieve complete result sets from different native data models.
A. RELATIONAL DATABASES The main focus of relational database approaches to query federation is around the closed-world enterprise settings which require a predefined number of data sources, schemas, and data sets to process queries across different business units within or between organisations. Some early  [10]  and more recent query federation approach  [11]  target the challenge of distributed query processing, distributed transactions, performance, and replica management. The database community realized the importance of managing distributed and heterogeneous databases and proposed ways to address the heterogeneity found across databases  [12]. Several early data mediators  [13]  and middleware systems  [14]  -primarily based on the Global as View (GAV) and Local as View (LAV) approaches  [15]  -developed in the last three decades had the same motivation, namely, unifying the data retrieval and query process over different models and sources.
The assumptions behind -such as availability of global schema, availability of mappings between schemas (local and global), and revising all the mappings with addition or removal of data sources -the GAV and/or LAV based approaches are too restrictive for Web like querying scenarios. Recently the database community is exploring a new perspective -called Polystore  [5]  -to unify queries over multiple data models. The early demonstrators of Polystore have shown promising outcomes in federating queries across disparate data models used in the Multiparameter Intelligent Monitoring in Intensive Care II Databases (MIMIC II)  [6] ,  [16] ,  [17]. Similarly, NoSQL polyglot persistence  [18]  style solutions  [19]  are proposed to query over different data models. Domain specific solutions, such as streaming and sensor data processing  [20] , enterprise analytics  [21] , social media  [22]  have taken initiatives to query over multiple data stores which natively support different data models.
Source selection is typically enabled using a local index/catalogue and/or probing sources with queries at runtime. The former approach assumes some knowledge of the content of the underlying endpoints and requires update/synchronization strategies. However, the latter approach incurs a higher runtime cost, having to send endpoints queries to determine their relevance for various sub-queries. Thus, many engines support a hybrid approach of index and query-based source selection. Initiatives, such as Ontology-Based Data Access (OBDA) exploit ontologies specifically for accessing relational databases (RDB)  [34] ,  [35]  or semantic data lakes  [36] ; such initiatives are complementary to our PolyWeb proposal of querying different data models. The challenges of federating a given query over multiple data models are different in a Web like scenario.
Often the data sets are selected from a large pool of prospective data sources and there is no control over the availability of complete schemas and data setsunlike enterprises where data resources are controlled. It's important to note that database research is investigating the normal federation approach (using single RDB data model) vs. Polystore-based federation implemented using database technologies  [5] ,  [37]. Similarly, in this paper we are investigating normal federation (using single RDF data model) vs. Polystore-based federation implemented using semantic technologies. The obvious extension of our work is to investigate and compare database-style implemented Polystore vs. semantic technology implemented Polystore federation. This will help the wider research communities (database, semantic web, linked data) to understand the challenges of Polystore based query federation over different data models in a database-style enterprise like scenario vs.
Our key aim is to reduce (or possibly avoid) the cost and effort of massive data conversion needed in a typical data warehousing and/or single data model approach for federating queries over different data sources. The PolyWeb architecture is summarized in Fig. 2 , which shows its four main components: (i) Source Selection: performs source selection based on the capabilities of native data sources; (ii) Query Optimization: performs cost-effective arrangement of a query (triple patterns) in a manner that reduces query joins and remote requests on the network; (iii) Query Planning: builds a query plan based on joins found between query arguments; and (iv) Query Execution: performs the execution of sub-queries against the selected native data sources and merges the results returned. In addition to these components, PolyWeb can deal with the multiplicity of data models in the data sources by relying on mapping definitions that allow homogenising the interactions with databases in their native model.
An example of the data model is the relational model where data sets are relational databases (sets of relational tables), the query language corresponds to the relational algebra, given responses in the form of a relational table. We insist in particular on the RDF data model, where data sets are RDF graphs, that is, sets of RDF triples (s, p, o) where s is a blank node or an IRI, p is an IRI, and o is a blank node, an IRI, or a literal. The query language for RDF is SPARQL, which provides responses in the form of SPARQL results as defined by the SPARQL 1. 1 standard  [38]. We denote the RDF data model with rdf, the set of RDF graphs with G, the set of SPARQL queries sparql, the set of SPARQL results as RS sparql , and the evaluation of a SPARQL query q against an RDF graph G with [[q]] G.
Sometimes, it is not even possible to convert the source data because it is only made accessible via a query endpoint. Instead, the mapping definition should be used to define a query translation, defined as follows: Definition 3 (Query Translation): A query translation for a mapping definition md on a data model dm is a pair qt md , rt md where: • qt md : sparql → DS dm and • rt md : RS dm → RS sparql , such that for all q ∈ sparql and all D ∈ DS dm , [[q]] md(D) = rt md (Eval dm (qt md (q), D)). Mapping definitions can be written in dedicated mapping languages, such as R2RML  [39]  (for mapping relational databases to RDF), RML  [40]  (extending R2RML to other data models, such as XML, JSON, CSV, HTML), XSPARQL  [41]  (initially for XML), SPARQL-Generate  [42]  (CSV, JSON, XML, HTML, CBOR), XSLT  [43]  (if limited to transformations that results in RDF/XML), defined as a JSON-LD  [44]  @context for JSON, or using the CSVW vocabulary  [45]  for CSV.
For instance, the R2RML mapping shows that an RDF predicate genome : chr is mapped to a database column Chromosome in table cnv and the RML definition shows a mapping from RDF predicate genome : start to a CSV column Start_Position of the cnvd table. R2RML (RDB to RDF Mapping Language)  [39]  is a W3C standard for expressing customized mappings from relational databases to RDF. RML (RDF Mapping Language)  [40]  5 https://www. w3. org/TR/sparql11-query/#BasicGraphPatterns extends R2RML applicability to expressing mappings from CSV, HTML, JSON, and XML to RDF. Mappings defined using R2RML and RML are itself RDF graphs. PolyWeb employs a simple data-free 6 indexing mechanism where the index stores minimal information about given data sets. In the case of RDF data sets, it stores only predicate IRIs obtained from individual data set; otherwise, it exploits the mapping definitions to associate predicate IRIs to non-RDF data sets.
An overview of the experimental datasets is given in Table  4 , for instance, COSMIC-CNV consists of 37 million triples (6. 54 GB size in RDF format); with equivalent 29 million database records. The ''Raw Type'' column represents the raw data format and the ''No record'' column presents the total number of records (rows) in each raw data type. The last column represents the cost (''RDFisation Time'') associated with the transformation of raw data to RDF format. PolyWeb can avoid the data conversion cost (i. e. , total 3 hours for 3. 5 GB raw data). One of the main aims of PolyWeb is to avoid or reduce the data conversion cost in a federated environment. We have designed 10 queries to evaluate and compare the performance of PolyWeb with FedX and HiBISCuS engines. Table  5  summaries the characteristics of these queries following similar dimensions to that used in the Berlin SPARQL benchmark  [51] , which shows that the queries have a variety of characteristics and complexity.
1) DATA SUMMARIES GENERATION TIME AND COMPRESSION RATIO The method of generating data summaries for PolyWeb is compared with various state-of-the-art approaches and the comparison results are shown in Table  7. The comparison metrics used are data summaries generation time (time), data summaries size (size) and the data summaries reduction (ratio: computed as 1 -index size total dump size ). The results of Table  7  show that the different sizes of data summaries for all approaches are much smaller than the relative size of the RDF data dump. In the case of FedX, no data summaries are created since relevant sources are determined on-the-fly at runtime. PolyWeb produces the smallest data summaries by storing only the meta-data about predicates: for RDF datasets having a raw dump size of 7 GB, PolyWeb generates data summaries of size 2 KB, achieving 99.