dataesr/scientific-paragraphs-categorization huggingface dataset:
Essential amino acids (EAA) consist of a group of nine amino acids that animals are unable to synthesize via de novo pathways. Recently, it has been found that most metazoans lack the same set of enzymes responsible for the de novo EAA biosynthesis. Here we investigate the sequence conservation and evolution of all the metazoan remaining genes for EAA pathways. Initially, the set of all 49 enzymes responsible for the EAA de novo biosynthesis in yeast was retrieved. These enzymes were used as BLAST queries to search for similar sequences in a database containing 10 complete metazoan genomes. Eight enzymes typically attributed to EAA pathways were found to be ubiquitous in metazoan genomes, suggesting a conserved functional role. In this study, we address the question of how these genes evolved after losing their pathway partners.
Database Searches for Genes Involved in Essential Amino Acid Biosynthesis Based on previous articles  [2] , we performed sequence searches to find all the enzymes responsible for the biosynthesis of essential amino acids in yeast (Saccharomyces cerevisiae). Each gene in this set was used as a query for a BLAST-based similarity search  [10]  in metazoan genomes (i) to retrieve putative homologs and (ii) to search for deleted genes. Genes were considered homologs based on annotation and on being able to retrieve the original query using BLAST in the yeast genome.
Phylogenetic Analysis of ReGens BLAST searches in the NCBI Refseq database were conducted, in order to find putative homologs from other species of plants, fungi and metazoans for each of the eight curated genes. These protein sequences were multiply aligned using ClustalW  [14] , Muscle  [15]  and MAFFT  [16]  using default parameters. Phylogenetic analyses were then performed using the PhyML software  [17]. The evolutionary history was inferred using the Maximum Likelihood method based on the JTT matrix  [18]  of amino acid substitution. Five-hundred bootstrap replicates were used to evaluate the phylogenetic consistency.
Conservation Pattern A Python script was developed to visualize the amino acid conservation profiles of ReGens and their putative homologs in fungi and plants. Amino acid sequences from all clades were multiply aligned using ClustalW and each residue was compared, using the BLOSUM 62 matrix, with all residues located in the same alignment column (representing the same molecular character) (Supplementary material 1). Amino acid conservation scores were divided into groups for pairwise clade comparisons: Metazoan X Fungi, Fungi X Plants and Plants X Metazoan. The character-to-character BLOSUM value is summed for each alignment column and finally averaged for each pairwise clade comparison. This average value is further filtered to remove the high frequency noise using a low pass filter (See formula below).
𝑦𝑦 𝑛𝑛 = 𝛼𝛼𝑥𝑥 𝑛𝑛 + (1 -𝛼𝛼)𝑦𝑦 𝑛𝑛-1 (1) This filter is used to observe local tendencies of conservation, instead of the score of a single position. It filters the average comparison score for every position in the multiple alignment (xn), multiplying this value by a constant alpha, 0 < α < 1 (we used α = 0. 05, higher means less noise and less resolution) to produce the final value shown on the y-axis at the n th position (yn). As this filter is directional, we plot the average results for both the original alignment and the reverse alignment. This program uses the graphical library MatPlotLib  [19]  to plot the results.
Ka/Ks Estimation Using PAML Pairwise and branch specific Ka/Ks estimation was done using the codeml program of the PAML package, using the PhyML maximum likelihood tree and the back translated multiple alignment as inputs. Pairwise comparisons were separated into three categories, Fungi X Metazoans, Fungi X Plants and Metazoans X Plants. Finally, the Ka/Ks values were averaged for each clade comparison.
Finding and Describing Human ReGens A complete dataset containing all the yeast (Saccharomyces cerevisiae) protein sequences for EAA biosynthesis, consisting of 49 proteins, was produced through manual curation in the UniProt database  [11]  (see Supplementary material 2). All these sequences were used as BLAST queries to search the RefSeq  [20]  database for putative homologs in metazoan genomes. Eight of the 49 enzymes were found to have clearly identifiable homologs in metazoans, as estimated by sequence identity and protein coverage (Table  1 ). They were carefully annotated and their molecular function was manually investigated in both the literature and databases such as UNIPROT  [21] , KEGG  [13] , GO  [22] , Brenda  [12]  and CDD  [23]. We confirmed that five of the eight ReGens are currently annotated as members of other non-EAA metabolic pathways.
Conservation of Protein Sequences between Clades The sequences encoding the homologs of ReGens in a number of selected organisms were retrieved from the RefSeq database (Table  2 ). Manually curated orthologous groups were created for each ReGen. In the case where a given ReGen presented multiple family members (paralogs) in a given genome, we chose the version that was the most similar to the fungi sequence used as query. All multiple alignment tools provided efficient and congruent sequence alignments (data not shown) and we proceeded with ClustalW data (see Supplementary material 4). Using the Python programming language, together with Biopython and MatProtLib libraries, we developed a visualization script to provide an easy and intuitive view of sequence conservation between tens of proteins derived from distant clades.
Phylogenies of ReGens Using Maximum Likelihood Phylogenetic analysis was performed for all eight ReGens except AadAT (absent in plants), using maximum likelihood (ML) approaches based on the JTT method implemented in the PhyML package  [17]. As most gene trees agree with the species tree, metazoan ReGens were expected to cluster with their fungi orthologs. If non-ToL evolution occurred in the metazoan genes, fungi and plant enzymes (still being used for amino acid biosynthesis and therefore subject to similar functional constraints) might retain a higher level of similarity. Thus, a phylogenetic analyses of these genes would result in the autotrophic (plant and fungi) genes being clustered as sister groups, while metazoan ones would be seen as an outgroup. Corroborating the conservation analysis results, phylogeny data showed that most observed ReGens fit the ToL model, and metazoan proteins were found to cluster together with fungal proteins in the expected Opisthokonta clade.
Interactions between proteins from hosts and symbionts have been shown in a number of organisms, sometimes in a given tissue or during a brief development stage  [52, 53]. We observed favored permanence of genes at the start or end of EAA pathways, possibly because these are usually responsible for the regulation of the entire pathway. Moreover, the last genes of a biosynthetic pathway might also take part in the degradation of the final product, as is the case with BHMT and SD. On the other hand, several of our experiments suggested non-ToL evolution of the ALS and BHMT genes in metazoan genomes (Table  5 ). These results are further supported by the observation that amongst the 13 amino acids annotated as belonging to the catalytic site of Arabidopsis thaliana's ALS protein structure (deposited as 1YI1  [54]  in the PDB database  [55]  and verified using the Catalytic Site Atlas  [56] ), 10 were found to be conserved in fungi and only 5 in metazoan proteins.
The World Wide Web is the largest information repository available today. However, this information is very volatile and Web archiving is essential to preserve it for the future. Existing approaches to Web archiving are based on simple definitions of the scope of Web pages to crawl and are limited to basic interactions with Web servers. The aim of the ARCOMEM project is to overcome these limitations and to provide flexible, adaptive and intelligent content acquisition, relying on social media to create topical Web archives. In this article, we focus on ARCOMEM's crawling architecture. We introduce the overall architecture and we describe its modules, such as the online analysis module, which computes a priority for the Web pages to be crawled, and the Application-Aware Helper which takes into account the type of Web sites and applications to extract structure from crawled content.
The crawling architecture of ARCOMEM aims at solving these different issues by providing flexible, adaptive, intelligent content acquisition. This is achieved by interfacing traditional Web crawlers such as Heritrix with additional modules (complex resource fetching, Web-application-aware extraction and crawling, online and offline analysis of content, prioritization), as well as by adapting the internals of the crawlers when needed (typically for managing priorities of content relevance). The objective of this article is to present an overview of this crawling architecture, and of its performance (both in terms of efficiency and of quality of the archive obtained) on real-Web crawls. This article extends  [7]. The remainder of this work is organized as follows. We first discuss in Section 2 the related work.
Web Crawling Descriptions of early versions of Google's and Internet Archive's large-scale crawler systems appeared in  [9, 10] , respectively. However, one of the first detailed descriptions of a scalable Web crawler is that of Mercator by Heydon and Najork  [11] , who provide information on the various modules of the crawler and the design options. Najork and Heydon also describe a distributed crawler based on Mercator in  [12]. Shkapenyuk and Suel  [13]  introduce a distributed and robust crawler, managing the failure of individual servers. Heritrix  [14]  is an archival-quality and modular open source crawler, developed at the Internet Archive. In Section 4. 4 we will describe how we have adapted Heritrix in order to fit in ARCOMEM's crawling architecture. Boldi et al. [15]  describe UBICrawler, a distributed Web crawler, implemented in Java, which operates in a decentralized way and uses consistent hashing to partition the domains to crawl across the crawling servers.
Focused crawlers and crawlers in general can harvest data from the publicly indexable Web by following hyperlinks between Web pages. However, there is a very large part of the Web that is hidden behind HTML forms  [38]. Such forms are easy to complete by human users. Automatic deep-Web crawlers, however, need to complete HTML forms and retrieve results from the underlying databases. Barbosa and Freire [39]  develop mechanisms for generating simple keyword queries that cover the underlying database through unstructured simple search forms. Madhavan et al. [40]  handle structured forms by automatically completing subsets of fields, aiming to obtain small coverage over many hidden Web databases.
The combination of the original crawl specification with the extracted information from the reference crawl is called the Intelligent Crawl Definition (ICD). This specification, together with relatively simple semantic and social signals, is used to guide a broad crawl that is followed by a thorough analysis of the crawled content. Based on this analysis a semi-automatic selection of the content for the final archive is carried out. The translation of these steps into the ARCOMEM crawling architecture foresees the following processing levels: the crawler level, the online processing level, the offline processing level, and the cross-crawl analysis that revolve around the ARCOMEM database as depicted in Figure  2. Since the focus of this article is the crawling and the online analysis we will focus on these levels in the rest of the article and give only a brief overview on the other levels.
Crawling Level At this level, the system decides and fetches the relevant Web objects as these are initially defined by the archivists, and are later refined by both the archivists and the online processing modules. The crawling level includes, besides the traditional crawler and its decision modules, some important data cleaning, annotation, and extraction steps (we explain this in more detail in Section 4). The Web objects (i. e. , the important data objects existing in a page, excluding ads, code, etc. ) are stored in the ARCOMEM database together with the raw downloaded content.
Offline Processing Level At this level, most of the basic processing over the data takes place. The offline, fully-featured, versions of the entity, topics, opinions, and events analysis (ETOE analysis) and the analysis of the social contents operate over the cleansed data from the crawl that are stored in the ARCOMEM database. These processing tools perform linguistic, machine learning and NLP methods in order to provide a rich set of metadata annotations that are interlinked with the original data. In addition to processing of textual content, multimedia content can also be analyzed and enriched with meta-information. The respective annotations are stored back in the ARCOMEM database and are available for further processing and information mining. After all the relevant processing has taken place, the Web pages to be archived and preserved are selected in a semi-automatic way and transferred to the Web archive (in the form of WARC files).
Application-Aware Helper The goal of the application-aware helper (AAH) is to make the crawler aware of the particular kind of Web application it is crawling, in order to adapt the crawling strategy accordingly. The presence of the AAH in the crawling processing chain ensures Web content is crawled in an intelligent and adaptive manner. The AAH applies different crawling strategies for different types of social Web sites (Web forums, blogs, social networks, photo networks, music networks, video networks, etc. ), or for specific content management system (vBulletin, WordPress, etc. ). The AAH detects the Web application and Web page type within the Web application before deciding which crawling strategy is best for the given Web application. More precisely, this module performs the following steps: (1) it detects the Web application type (general type, content management system, etc.
In our experiments, we did not see any instance where a Web application was not successfully detected. The AAH deals with two different cases of structure adaptation: first, when (part of) a Web application has been crawled before, but recrawl of Web application fails after template changes; second, when a new Web application has been detected successfully, but (some of) the existing actions are inapplicable. The adaptation module for recrawls of Web application relearns appropriate crawling actions for each failed crawlable object. In ARCOMEM, crawled Web pages with their Web objects and metadata are stored in the form of RDF triples in the ARCOMEM database. Therefore, we have proposed an algorithm which utilizes the ARCOMEM database and first detects structural changes for already crawled Web applications by looking for the crawled content in the Web pages with crawling actions used during a previous crawl.
If the system detects the Web application with any relaxed expression then the system will apply the crawling actions adaptation as described above. The AAH has reduced bandwidth, time, and storage (by requiring fewer HTTP requests for known Web applications, avoiding duplicates) using limited computational resources in the process. Application-aware crawling also helps adding semantic information to the ARCOMEM database. More details about the functioning and independent evaluation of the AAH are provided in  [49, 50].
Online Analysis Within the online analysis, several modules analyze crawled Web objects in order to guide the crawler. The purpose of this process is to provide scores for detected URLs. These scores are used for guiding the crawler in order to obtain a focused crawl with respect to the intelligent crawl definition (ICD). The main modules used within the online analysis are the AAH, the GATE platform for text analysis  [51] , and a prioritization module. The actual online processing consists of three phases which are displayed in Figure  3 ; within Figure  2  these phases are related to the connections between the online processing and the crawler. (1) The AAH performs preprocessing steps on the crawled Web pages; (2) Online analysis modules run on relevant document parts; (3) The output of online analysis modules is aggregated and a score for each URL is provided.
A more detailed description of the these steps is provided in the remainder of this section. In the first phase we run the AAH on the Web page to detect regions of interest in the document and discard irrelevant parts. A detailed description of the functions provided by the AAH is given in Section 4. 1. The input document is split into one or more document parts. Each document part is processed separately from now on. In the second phase the online analysis modules are run on the content of the document part. Currently we use textual analysis modules using GATE, a URL scoring module using URL patterns and a simple spam link filter using a black list. Additional modules can be added easily. The textual analysis module performs basic NLP pre-processing on the text and allows the extraction of relevant entities.
The version of GATE used within the Online Analysis is a lightweight version of GATE since the performance and the processing time needs to be as fast as possible. The tasks carried out by the GATE component comprise basic linguistic processing steps, language identification and Named Entity Recognition (NER). In contrast to the use of the basic GATE functions which are needed to create word vectors describing the crawled objects, the use of the NER module is optional. With the NER module disabled the processing time of a Web document was reduced by about 70%. Based on these observations we disabled the NER module in the online phase for most crawls and moved this part to the offline analysis, where performance aspects are not as critical as during the online phase. Using the extracted keywords and the given crawl specification we calculate a score based on the cosine similarity of the term vectors.
The distribution module maintains a consistent hashing ring that reflects the current cluster topology. It forwards URLs to the appropriate node for them to be queued in the local URL store. The URL distribution being based on the pay level domain, it is easy to guarantee that no more than one fetcher in the whole cluster will be crawling a specific host at any time. However, there is no guarantee that different pay level domains are not mapped to the same IP address. To ensure rate control for IP addresses, we use a global IP address registry. The WARC files get copied asynchronously to a specific directory in a Hadoop file system (HDFS). A periodic import task will insert the content from the HDFS into HBase. This makes the crawler quite independent from the storage system. In particular, the crawler can continue to work without HBase for as long as it has available disk space.
All the modifications are available as open source software in the releases of the ARCOMEM project. In this article we focus on the two main features mentioned above. The default frontier of Heritrix employs a Berkeley DB backed hash table for storing URLs, typically grouped according to the domain or host they belong to. The key of a URL's record in the frontier is computed based on its domain, a flag indicating whether the URL should be crawled immediately, its priority (or precedence in Heritrix terminology), and a counter, which increases for every URL that is inserted in the frontier (Figure  5 ). The frontier implementation provides a next method for obtaining the next URL to crawl from a given domain or host, but there is no method to update the priority of an already scheduled URL.
To evaluate the focused crawler architectures, we perform simulated crawls on datasets created with three topics of DMOZ. We create three random samples of 20 seeds for each of the topics and the results we obtain for each configuration are the average of 9 simulations. For each set of seeds, we simulate a crawl of 10,000 Web pages. The topic vector we use to compute similarities between each topic and the crawled Web pages corresponds to the vector sum of the seed term vectors. For the evaluation of the results, we employ three measures: (a) harvest ratio, which we define as the ratio of Web pages whose cosine similarity with the topic vector is greater than 0. 333 over all crawled Web pages; (b) average similarity of crawled pages; and (c) fraction of DMOZ subtopics with at least one crawled page.
Comparison of ARCOMEM Versus Standard Crawl For evaluating the quality of the crawls in terms of how focused the collected documents are, we conducted a series of experiments. The main task of these experiments is to give an overview of the number of the crawled documents match the keywords given by the ICD, and how similar the textual content of the crawled documents is compared to the seed documents. Since there is no ground truth for the actual relevance of a document we have investigated some alternatives on how to measure the relevance of a document with respect to the given crawl definition. The dataset used for the experiment consists of two different crawls in the financial domain. One crawl was performed using the described ARCOMEM architecture while the other crawl is a standard Heritrix crawl.
Conclusions The scale of the Web, the volatility of information found in it, as well as the emergence of social media, require a shift in the way Web archiving is performed. Towards this goal, the ARCOMEM project has developed a scalable and effective framework that allows archivists to leverage social media and guide crawlers to collect both relevant and important information for preservation and future reference. In this article, we have presented ARCOMEM's crawling architecture, providing a detailed description of its main modules for extracting structured information from Web applications and prioritizing the URLs to be crawled. We have also outlined the main features of a large-scale distributed crawler, which can collect the content from billions of URLs while maintaining a high download rate.
The architecture we have described enables us to use either the large-scale crawler or an enhanced version of Heritrix, for which we have described the required modifications we have implemented to support the adaptive prioritization of URLs and the scheduling of URLs from remote processes. Our experimental results show that the adaptive and batch prioritization, which are employed in the proposed crawling architecture, are effective in acquiring relevant content. When comparing the quality of a crawl performed with the ARCOMEM architecture against a crawl performed with Heritrix, we have seen that the ARCOMEM crawler has downloaded earlier more relevant content. Overall, the proposed crawling architecture is both extensible, by adding new modules to expand the analysis, and scalable, offering a new approach to crawling content for Web archiving.
To reduce the number of premises and assumptions, Nam and Alur  [30]  proposed a method to automatically group the n components into m groups, where m < n, by reducing the problem to the hypergraph partition problem. Alur et al. [4]  proposed a symbolic implementation of AGR for the AGC proof rule. They used Binary Decision Diagrams (BDD)  [8]  to symbolically encode the observation table maintained by the L * algorithm. However, the works based on the L * algorithm mentioned above are only applicable for untimed systems. To infer timed assumptions for AGR, a learning algorithm for timed models is needed. Grinchtein et al. [16]  proposed three algorithms TL * sg , TL * nsg , and TL * s for learning ERAs. Their learning algorithms deal with timed words, while our TL * algorithm deals with guarded words.