dataesr/scientific-paragraphs-categorization huggingface dataset:
Background on NAND Flash Memory NAND flash memory has some constraints caused by its internal intricacies. Basically, the smallest addressable data unit in flash memory is a page (2 to 8 KB), and a fixed set of pages (usually 64) composes a block. Read and write NAND flash memory operations are executed on pages while erase operations are executed on blocks. The main constraints are: 1) Write/Erase (W/E) asymmetry: writes are performed on pages whereas erasures are realized on blocks. 2) Erase-before-write limitation: a costly erase operation is necessary before data can be modified. 3) Limited number of W/E cycles: the average number is between 5000 and 10 5 depending on the technology used. A flash memory specific component (either software or hardware) is generally used to overcome the aforementioned limitations: 1) The erase-before-write and the W/E granularity asymmetry constraints imply that data updates should be performed out-of-place.
Flash Chip Management with Linux Figure  1  illustrates the software / hardware stack involved in managing the secondary storage with embedded flash chips in Linux. User space applications (A on Figure  1 ) access files using system calls (C) which can be encapsulated in some library (B). System calls are received by VFS (D) in kernel space. VFS role is to abstract the use of multiple file systems in one operating system, and to maintain some caching and I/O optimization mechanisms. VFS dispatches requests to the corresponding file system, which in our case is a FFS (E). The FFS handles the requests and eventually performs flash operations on the memory chip (G) through the NAND driver, named the Memory Technology Device (MTD) layer (F). Traditional VFS layer: The VFS layer  [2]  allows various file systems operating on multiple storage devices to coexists in the OS, and to be transparently accessed by the user.
Several OS level optimization mechanisms are implemented by the VFS layer. The Linux page cache is a RAM data cache buffering read and written file data. There are two other mechanisms implemented at the VFS level which aim to respectively enhance read and write performance between the page cache and the file system. First, the page cache read-ahead mechanism mainly targets hard disk drives. During a file read operation, if the system detects a sequential pattern, it prefetches data sequentially from the file system. Read-ahead system is based on the fact that hard disks drives show better sequential pattern performance reducing time consuming head moves. Readahead, enabled by default, has proven to be ineffective when working with some FFS  [15]. Nevertheless it is still active in YAFFS2 and JFFS2, but deactivated in UBIFS.
The three main FFS supported by Linux are JFFS2  [16] , YAFFS2  [8]  and UBIFS  [14]. The way file data and metadata are read from / written to flash memory is very specific to the FFS model considered. Nevertheless, all FFS must cope with flash memory constraints. They all perform outof-place data updates and old data invalidation, leading to garbage collection (GC). GC is done online with I/O requests when the amount of free space is critically low, and asynchronously through a kernel thread. FFSs also implement wear leveling policies in their write strategies and GC victim block selection algorithms. MTD  [10]  is a generic driver for all NAND chips supported by Linux. It allows the FFS to perform flash operations. One can note that MTD maintains in RAM a read buffer which has the size of one underlying flash page.
RELATED WORKS Many tools have been developed to help understand Linux I/O behavior. Blktrace  [3]  is a reference tool allowing to trace at a block level. Unfortunately, this tool is not of help when using raw flash chips based on MTD layer. To our knowledge, there is currently no software specifically targetting the trace of FFS based systems. Many tools exist that allow to trace and monitor kernel system calls and events, one can cite SystemTap  [4] , Oprofile  [7] , Ftrace  [13] , etc. Those tools are very powerful and can help in understanding the I/O request flow. However, their use is cumbersome and their installation could be very hard on embedded platforms due to compiling and dependency issues. The objective of the presented toolset is: (1) to provide users with very easy to use and yet powerful tools, (2) to focus on FFS-based storage systems, (3) to be homogeneous as it is based on the same light (non intrusive) Kprobe facility, and (4) easy to install (one just needs to compile the modules against the kernel code and insert the modules).
Fundamental Design Concepts The three tools are Linux kernel modules written in C. Kernel modules are built against a kernel source directory tree without the need of (re)compiling the whole kernel. They are loaded and unloaded on demand at runtime. The three developed modules can run concurrently. Kprobes: All the tools rely on Kprobes  [6]. Kprobe stands for kernel probe and is a low-overhead functionality provided by Linux kernel to obtain information about kernel function execution at run-time. Kprobes were used because of three main reasons : their portability as they are natively supported by Linux, their simplicity as one does not need to familiarize with a complex framework to develop/modify them, and their non-intrusivity as they were reported to infer a very small overhead  [6]. Kprobes must be used in kernel code.
VFSMon VFSMon traces VFS related events using Jprobes. VFS events can be decomposed into three classes: VFS inputs, VFS internal events, and VFS outputs (which actually are the FFS inputs). VFS inputs are high level functions corresponding to the file system calls entry points at the VFS level. Functions such as vfs read(), vfs write() or vfs open() are traced along with the related parameters: file identifier (inode), offset and size of data read / written, etc. VFS internal events are related cache mechanisms (page cache readahead and write-back) mentioned earlier. VFSMon traces page cache hits and misses on read operations. Several read-ahead related events and parameters are also traced, allowing mainly to observe the prefetching window size evolution. There is no particular event traced concerning the page cache write-back mechanism as its behavior can be observed when tracing asynchronous write calls to the FFS, i.
e. FFS inputs. Traced FFS inputs, or VFS outputs, are the entry points of the FFS level. The three main functions at this level are xxx readpage() (xxx being the name of the FFS) which is called by VFS to ask the FFS to read a page of a file from the storage media and places it in the page cache. This function is called on a page cache miss or during a read-ahead prefetching pass. xxx write begin() and xxx write end() are called to write the data of a page on the storage media, for instance, in case of a write-back dirty page flush. VFSMon takes as input the partition for which the user needs to trace file I/Os. This allows to filter file operations related to other peripherals, partitions, pipes, etc by tracing a dedicated flash partition. The user can start, stop, pause, and reset VFSMon very easily through the /proc directory.
The user can start, stop, pause, and reset Flashmon through the /proc directory. One can also have the possibility to interact with Flashmon via /proc to isolate I/Os generated from a given program to make the trace more readable. For instance, one might develop a program that performs some read operations, after each read request, the user program can send a command to Flashmon via /proc in order to insert a marker allowing to clearly see flash operations generated for each applicative read operation. Flashmon allows to collect two types of outputs: a spatial view of the flash memory, and an event log. The spatial view represents the number of operations (read, write, and erase) sustained by each block of the flash memory since the module insertion. It allows to have a clear idea of the distribution of the I/O operations on the flash memory for a given application and/or partition.
FuncMon FuncMon is a multi-level kernel function execution time tracer. It works by placing Kretprobes on the (a) entry and (b) exit points of relevant functions of the I/O software stack. One Kretprobe is associated to each traced function. For a given traced function, to each trace point (entry and exit) is associated a Kretprobe handler which is executed (1) when the traced function is called and (2) when it returns. System time is obtained on both the entry and exit handlers, and the the execution time is then inferred. FuncMon can be used with any function of the I/O kernel storage stack. The user can start, stop, and reset FuncMon through the /proc directory. Figure  2  presents an example of FuncMon output, configured to track the jffs2 readpage() function (reading 4KB of data from the flash media and storing it as a page in the page cache).
Consequently with the three developed tools one can have a precise idea on the latencies induced by each layer on the I/O software stack. All on a synchronized homogeneous (from a user point of view) set of tools. The case of JFFS2 JFFS2  [16]  is a mature and widely used flash file system, mainlined into Linux sources since 2001. For obvious reasons we cannot describe all the internals of JFFS2 in this paper. Write operations are packed by JFFS2 into so-called data nodes which are written on flash. The data contained in one Linux page from one file can be scattered into several nodes at various on-flash offsets. This potential fragmentation can have a strong impact on JFFS2 read performance. Fragmentation occurs when a file is written/updated randomly. This phenomenon is studied in the next section.
In this work JFFS2 compression feature was disabled as it adds a potentially important and varying overhead on the performance. In the presented case-study, we are interested in the jffs2 readpage() kernel function execution time which reads a 4KB Linux page from flash memory and places it in the VFS page cache. As we are at the FFS level, we ignored the page cache and read-ahead mechanisms. The model is simple and focused on a specific operation to illustrate the use of the developed tools. Modeling Methodology The modeling methodology consists of : (A) an exploration phase: the tracing toolset is used to identify the various parameters impacting JFFS2 read performance ; (B) a modeling phase: based on the output of the exploration phase, a theoretical model is built ; (C) a validation phase: once again the tracing framework is used to create a performance profile for the tested hardware / software platform and validate the previously built model against a given I/O workload.
Exploration Phase The objective of the exploration phase is to determine the major factors having a significant impact on jffs2 readpage() call execution times. To do so, a series of tests has been conducted. The tests consist in simple C micro benchmark programs reading data from a file written on a clean JFFS2 formatted flash memory partition. The page cache was emptied before each test. We mainly varied three parameters: the access pattern (random and sequential), the inter arrival times, and the file fragmentation. We did not vary the request size as at the FFS level only 4KB system pages requests are issued. This behavior can be confirmed using VFSMon. Measures on the jffs2 readpage() execution times were achieved thanks to FuncMon. From the experimentations, three observations can be done: (1) the main factor influencing the jffs2 readpage() execution times is the file fragmentation.
Proposed Simple Model When comparing the previous FuncMon outputs (FFS level) with the related Flashmon outputs (driver level), one can conclude that the execution time of a jffs2 readpage() call is mostly related to the number of flash pages read operations. Then it can be modeled as follows: Tjffs2 readpage f,p = Nflash pages f,p * Tflash page read+Toverhead Where Tjffs2 readpage f,p is the execution time of a call to the jffs2 readpage() function on the Linux page of index p inside a file f. Nflash pages f,p is the number of flash pages concerned, Tflash page read is the time to read one flash page at the driver level (just below the FFS), and finally, Toverhead is the additional time related to JFFS2 internal operations. Nflash pages f,p depends on the on-flash physical location of the nodes composing the Linux page as each node can be written in one or many flash pages.
Moreover, the MTD  read buffer presence implies that the same flash page read consecutively many times would trigger only one flash page read operation. We define the term fragmentation map as being the ordered sequence of physical flash page numbers needed to be read in order to retrieve all the FFS nodes composing a Linux page. For example, if a Linux page p of a file f is composed of two 2 KB sized nodes, with the first one physically located on flash pages 12 and 13, the second on flash pages 13 and 14, and the fragmentation map for the Linux page in the file is F ragf,p = {12, 13, 13, 14}. So Nflash pages is equal to the size of the fragmentation map minus the number of potential MTD cache hits in case a flash page is accessed two or more times repeatedly. Nflash pages f,p = |F ragf,p| -Nread cache hits f,p Here |F ragf,p| is the size of the fragmentation map of the page p in file f.
Estimator and Model Validation An estimator implementing the model was developed and used to analyze the model accuracy. This program estimates the time taken to read a JFFS2 file in a given hardware / software environment. It takes as input an I/O trace a trace describing the order in which the Linux pages of the file are read, and a description of the environment in terms of: (A) the Tflash page read and Toverhead values for the model parameters, and (B) the fragmentation map for each Linux page of the file being read.
Results are presented in Table2. Fragmentation Map To build the fragmentation map for a given file, we relied on two software tools. The first is jffs2dump developed by JFFS2 contributors. Given the raw (binary) content of a jffs2 flash partition, it outputs the list of nodes and details about its location and state. We built a second tool that takes the output of jffs2dump and builds the fragmentation map for each Linux page of each file.
We started from the existing geneDistiller pipeline developed for vertebrates  (Gilchrist et al. , 2004) , and extended it for application to highly divergent and heterozygous organisms. We have in particular developed a novel algorithm for automated 5′ end recognition and a method for distinguishing paralogous differences from allelic variation. Here, we present these extensive conceptual improvements and characterize the ensuing first comprehensive collection of 19,163 full-ORF cDNA clone for a highly polymorphic marine invertebrate. The collection is organised into a set of fifty 384-well plates for copying and distribution; the identity and coordinates in the plate of the full-ORF clones can be found on each gene card page in the ANISEED database (www. aniseed. cnrs. fr), from which a table with the 384-well plate coordinates for each clone can be downloaded.
Sanger sequencing of primary cDNA libraries The GATEWAY cDNA clones were arrayed and sequenced by standard Sanger sequencing on equipment at JGI. All clones supplied were sequenced from both the 5′ and 3′ ends, with success rates passing the JGI submission quality threshold of 90% and 98% from the cien (also called XABT) and cima8 (also called CBWU) libraries, respectively. The cien library, made from mixed early embryonic stages sampled from North Atlantic animals, was sequenced to a depth of 359,578 ESTs. The second, or cima8 library, made from whole mature adults sampled from the Pacific population, was sequenced to a depth of 196,408 ESTs. Together these provided more than 500 million bases of long read sequence. EST sequences can be found in the EST section of GenBank.
Alignment of EST and cluster sequences to Kyoto Hoya (KH) C. intestinalis genome assembly and gene models The current C. intestinalis genome assembly, called Kyoto Hoya (KH)  (Satou et al. , 2008b) , was downloaded from the ANISEED database (http://www. aniseed. cnrs. fr/aniseed/download/down load_data). The corresponding gene models, named KH gene models, were initially manually curated in 2008 (KH2008) and have have been successively refined in 2010 and 2012 (KH2010, KH2012 models). The KH2010 (http://ghost. zool. kyoto-u. ac. jp/da tas/KH2010. 1. KHGene. gff. zip) and KH2012 (http://ghost. zool. kyo to-u. ac. jp/datas/KH. KHGene. 2012. gff3. zip) model sets used in this work are largely identical. The former was used for a more general quality assessment of the sequenced libraries, while the later was used for the more accurate analysis of clone coverage.
To estimate coverage of KH2010 gene models with ESTs, all available EST sequences were matched to the KH2010 transcripts model by best hit BLASTn analysis using an e-value threshold of e-20 and a word size of 20. EST sequences were then grouped by KH gene locus. To identify putative exons, the full-ORF clone EST sequences were aligned to the KH assembly using the EST2genome model of the Exonerate alignment programme. We ran the search at low stringency to allow for the high level of sequence divergence between type A and type B strains, using the following parameter list: -model est2genome -gapopen-15 -bestn 1 -quality 85 -percent 33 -geneseed 200 -subopt false -hspfilter 100 -maxintron 10000. 2. 5. Parallel de novo and reference EST clustering projects: (Fig. 2iii ) Clustering of ESTs was done largely as previously described  (Gilchrist et al.
, 2004)  to 95% sequence identity. Under these conditions 1,074,949 sequences (generated in this project and retrieved from GenBank) that passed a basic quality threshold were assembled into 26,186 gene clusters and 9380 singleton ESTs. 2. 7. Disentanglement of over-clustered back-to-back gene pairs: (Fig. 2v ) Analysis of the assembled EST clusters showed some erroneous co-clustering of pairs of opposite-strand, back-to-back genes with overlapping 3′ UTRs. These were detected after clustering, using protein BLASTx alignments to identify the respective coding regions, and EST strand orientation to detect the sense boundary for separation into pairs of clusters. For this, all assembled clusters were analysed for multiple open reading frames with significant matches to known proteins. Those with multiple open reading frames were further analysed for (a) a second downstream ORF with negative strand matches, and (b) the average orientation of assembled ESTs switching between predominantly positive strand to predominantly negative strand between the two ORFs (3′ ESTs were reverse complemented before assembly).
9. ORF start detection: use of SL trans-splicing data (Fig. 2viii ) To take advantage of the presence of a unique splice leader (SL) sequence at the 5′ end of transcripts for $50% of Ciona genes, we matched a set of 170,299 5′ 454 sequence reads starting from an SL sequence  (Matsumoto et al. , 2010)  against our 5′ EST sequences using BLASTn. Clones containing the matched 5′ EST were present in 5049 top level clusters which were marked as full-ORF, and the clones made available to the clone selection process (Table  1 ). 2. 10. ORF 3′end detection: (Fig. 2vii ) We found that genes with long mRNAs and/or low expression levels, frequently give rise to two clusters, one covering the 5′ end of the gene (upstream cluster), and the other the 3′ end (downstream cluster). To confirm the presence of the terminal stop codon of the open reading frame, or 3′ UTR sequence, in such downstream clusters, we devised two additional tests.
First, we reduced the BLASTx e-value sensitivity threshold used in the rest of the project (0. 001) to 0. 1 to detect short stretches of coding sequence at the 5′ edge of (probably incomplete) contigs. Secondly, we identified stretches of well-assembled, 3′ UTR-like (no significant protein matches) sequence with stop codons in each frame, thus providing a definitive 3′ limit of any possible open reading frame. This region is then assumed to be downstream of the actual open reading frame. Similarly, contig sequences where no coding sequence is detected and the cluster is predominantly assembled (466%) from 3′ ESTs, are assumed to be part of the 3′ UTR. 2. 11. Alternative transcript picks: (Fig. 2x ) The geneDistiller pipeline  (Gilchrist et al. , 2004)  allows gene clusters showing evidence of alternative splicing to be internally re-assembled with more stringent criteria into two or more transcript-based sub-clusters.
Defining genes with highly conserved N-termini for RefSeq validation of cliff algorithm To allow selection of genes with highly conserved N-terminal position, we used BLASTx (translated nucleotide query vs. protein database subjects) to match the set of downloaded C. intestinalis RefSeq transcripts to protein data from worm, fly, fish, chicken, mouse and human, using an e-value limit of 10 À 10 and taking the highest scoring match for each species. For each match we used the coordinates of the reported alignment to predict the likely start (P) of the coding sequence in the Ciona transcript from the other species protein. In the simple case where the protein aligns against the transcript from the start, P equals the position of the start of the alignment on the transcript. Where the reported alignment starts within the protein sequence, P is predicted by assuming that the length of the unaligned N-terminal part of the proteins is a conserved feature, and by calculating its position on the transcript sequence.
Using standard BLAST alignment notation: P¼ query_start À 3(subject_start À1), where query is the Ciona transcript and subject is the protein. Highly conserved proteins were then defined as those where proteins from all six other species gave the same value of P. There were 303 such transcripts in the Ciona NCBI RefSeq transcript set. Of these, 268 aligned to 227 EST contig sequences, which were both large enough (at least five 5′ ESTs) to apply the cliff algorithm and annotated by RefSeq as possibly full-ORF. These 268 RefSeq transcript sequences were used for our validation test.
Comparing EST contigs with RefSeq predictions for pipeline validation For a wider validation of our full-ORF pipeline, we again used the Ciona RefSeq data, but this time compared the whole data set (excluding those annotated in GenBank as truncated) with the whole set of contig sequences for which full-ORF predictions were made. We combined BLAST alignments between our contig sequences and the predicted RefSeq coding sequences with assembly depth and other data to identify identical and differing ORF predictions. To establish the relationship between our EST contigs and the Ciona RefSeq ORF predictions, we generated gapped and ungapped BLASTn alignments between these two data sets with an e-value limit of 10 À 20 , using the coding sequence extracted from the Re-fSeq data with the given GenBank coordinates.
Sequence alignment for coverage of Ciona genes as a function of transcript length and relative abundance Starting from the BLASTn alignment of all available ESTs to KH2010 transcript models, we estimated the relative expression level of each gene by summing up the numbers of ESTs from all GenBank deposited ESTs mapping to the transcripts in each gene model. To evaluate the relationship between abundance and coverage, we divided the 15,254 KH loci ranked by EST number into 20 equal bins. For transcript size estimation, we used the length of the longest predicted protein in each gene. 2. 17. GO analysis of KH genes covered with a full-ORF clone GO annotation of KH gene models was carried out as part of the ANISEED annotation pipeline  (Tassy et al. , 2010). Briefly, KH model proteins were analysed with InterproScan  (Hunter et al.