dataesr/scientific-paragraphs-categorization huggingface dataset:
International Airline Passengers To illustrate the selected aspects of the methodology, we used the International Airline Passengers dataset  [35]. It contains the number of international airline passengers in thousands per month, collected between January 1949 and December 1960, and has 144 data points in total, first 92 of which we used for training, 23 for validation and the remaining 29 for testing. The history window length was set to 5, and batch size was = 1.
Pruning/Quantization Optimization The optimization was done according to flow A-finding per-model weights pruning and quantization parameters. Due to the small size of the original dataset, the optimization process could use it as a whole. The RMSE (see Equation (A14) in Appendix C) was used as a quality measure, resulting in a following quality score: quality score = RMSE pq -RMSE orig 0. 01. ( 17 ) Optimization was run for 3600 steps and resulted in 49 unique Pareto optimal solutions, four of which retained the required quality score < 1. The results are shown in Table  1  and Figure  10a. Figure  10a  shows that the best result results range 12. 22 % to 12. 50 % with size of 4 to 5 bits. It is worth noting that the biggest contribution came from quantization, and the sparsity level due to pruning was relatively low.
Beijing PM 2. 5 Pollution This dataset contains hourly readings of the PM 2. 5 concentration from the US Embassy in Beijing, combined with the meteorological data from Beijing Capital International Airport  [36] , and includes missing data. Each sample contains 13 attributes, including date of the reading, hour, PM 2. 5 concentration (µg/m 3 ), dew point ( • C), temperature ( • C), pressure (hPa), combined wind direction, cumulated wind speed (m s -1 ), cumulated hours of snow and cumulated hours of rain. The combined wind direction is expressed as one of five broad categories: The gaps in the data shorter than 12 h were interpolated. Longer gaps split the data into 37 variable-length series, containing 42,168 samples in total. Series were then assigned to training, validation and testing sets, maintaining roughly a 64-16-20 split.
Pruning/Quantization Optimization Optimization was run for 3600 steps. Since the dataset is quite small, it was used whole during the optimization process. Similarly to the International Airline Passengers example, the RMSE was used in a quality score. Optimization resulted in 165 unique Pareto optimal solutions, 39 of which retained the required quality score < 1. For selected of those solutions, the hardware resources consumption and latency were simulated and presented in Table  2. The full results are shown in Table  A4  (Appendix E) and Figure  11a. In terms of percentage of original model size, the best result for this dataset is comparable to the one achieved for International Airline Passengers (Section 3. 1. 2), but the Pareto front is much larger.
Individualized Pruning and Quantization The individualized pruning and quantization optimization (flow B) was run for 10,800 steps. It resulted in 332 unique Pareto solutions, 11 of which retained the required quality. The results are presented in Table  A5  (Appendix E) and Figure  11b. In case of this dataset, the individualized pruning and quantization did not lead to as good results as applying the flow A. This inability to at least match the best per-model outcomes may indicate that the selected number of optimization steps was too low for the number of optimized parameters.
Superconducting Magnets This case study was based on the experiments conducted in  [17] , which was focused on exploring the influence of history length and various data quantization schemes on the models' performance. The data we used came from the PM database of CERN LHC, representing the state of the superconducting magnets. Four data channels were used: U DIFF (total voltage), U RES (resistive voltage), I DCCT (current measured using Hall sensor), and I DIDT (time derivative of the electric current). Anomalies were marked using QUENCHTIME field as a start and until the end of series. Overall, ≈26 % of used samples were marked as anomalous, which allowed using the accuracy (see Equation (A15) in Appendix C) as one of the quality measures. The currently used system can detect an anomaly (a quench) with a latency of ≈10 ms.
Initial Conditions Analysis The tested model comprises of two LSTM layers and a single dense layer. The exact tensors sizes for the used architecture can be seen in Table  3. Equation (  1 ) was used along with the data in Table  3  to compute the critical amount of memory size, which is essential to proceed with the proposed procedures. The calculations are as follows: memory = (17664 + 12416 + 66) * 1 bit * 0. 05 ≈ 1507 bit ≈ 188 B. ( 19 ) We also directly mapped the model to using 32-bit fixed-point precision without compression and it turned out that it exceeded available resources on the target FPGA, Xilinx Zynq UltraScale + MPSoC XCZU15EG. This result led us to the conclusion that the proposed methodology is essential in order to map the model to the FPGA.
Pruning/Quantization Optimization To speed up the optimization process (done according to flow A), we have extracted the testing subset containing ≈8 % of original samples. The subset size was selected based on the incremental dataset boosting method. At first, mean and variance were calculated for a single batch with size = 16,384). Then, the dataset was expanded by another batch, with mean and variance calculated again. If they changed more than a selected threshold (in our case 1%) when compared to the previous values, the dataset was expanded by a single batch again. In the end, the quality measures for the subset were compared with original ones (as seen in Table  4 ), and similarity confirmed. Optimization was run for 3600 steps, and resulted in 34 unique Pareto optimal solutions (Figure  12a ), 14 of which retained the required accuracy (drop < 1 p.
Pruning and Quantization with Retraining Pruning and retraining can be considered as one of the most flexible compression schemes since it enables weights shaping after each iteration step. However, it is worth noting that optimization with retraining can also be very time consuming, e. g. , in our case a single pruning-quantization-retraining-pruning-quantization step took ≈23 min, so the optimization running for 3600 steps was estimated to take nearly two months. Therefore, we decided to apply the retraining procedure only in conjunction with previously optimized parameters. The results are presented in Table  5  (in parentheses). For retraining, the full dataset was used, and in only in a few cases, the performance (accuracy) of the model improved. Based on those experiments, it can be seen that using the predefined pruning and quantization parameters, even previously optimized, does not guarantee a good starting point for the retraining.
Data Bucketization We also ran the experiments for flow D, using the recursive_adaptive data bucketization algorithm. It maps the input space to a fixed number of bins in such a way that all the resulting, uneven bins have similar cardinality if possible. For more details, please see Section 2. 1 and Appendix A of  [17] , and Section 4. 2 of  [16]. The tested model structure was the same (two LSTM layers, with 64 and 32 cells, respectively, and a two-unit dense layer), which allowed us to focus on the differences caused by the data bucketization itself. The training on bucketized data yielded results on par with the original dataset (see Table  4 ). Also, in this case, for optimization, a test subset of ≈8 % original size was used. Optimization was run for 3600 steps, and resulted in 42 unique Pareto optimal solutions, eight of which retained the required accuracy (Figure  13a ).
DL2HDL Comparison with Other Frameworks To our best knowledge, the only high-level tool that can achieve comparable latency for neural network inference in FPGA is hls4ml  [10]. This tool was designed to work in particle physics experiments, where response time is crucial. It provides similar functionality using a different approach. The main difference between our flow and the hls4ml processing architecture is the number of procedure's stages. In hls4ml, in the first stage, the model is created in one of the high-level DL frameworks, such as Keras or PyTorch, then it is mapped to Vivado HLS, and in the last step, the FPGA RTL structure is generated by the synthesis and implementation tool. Our DL2HDL tool skips the middle step and provides mapping directly from Python to HDL. It is worth noting that the number of conversion steps affects the flexibility of the tool and may a source of potential mapping challenges.
Thus, we decided to simplify the flow when designing the DL2HDL. Since at the time of this article submission the hls4ml support for recurrent neural networks was not yet available, we were not able to test it on network architectures examined in this work. However, it was possible to test the DL2HDL using the very similar network to the one presented in  [10]. The tested architecture was a four-layer fully connected network, with ReLU as activation of the first three layers and softmax as output activation. Although we believe that softmax activation can be omitted in this application, we included it for a fair comparison. We used the same 16 bit wide fixed-point representation and pruned the network to the same number of parameters (1338). After HDL generation, the implementation was performed using Vivado software, for Xilinx Kintex Ultrascale xcku115-flvb2104-2-i with a target frequency of 200 MHz, as in the original experiment.
While our setup accounts for this option in general, it should be extended with a series of meta-parameters to use retraining for multiple epochs properly. Careful selection of parameters, such as the number of epochs, learning rate adjustment schedule and dataset size, is vital because retraining, especially with a full dataset is very time consuming and may be counter-productive if not carefully implemented. Using a reduced dataset, possibly newly selected in each epoch, and introduction of sensitivity list for the model's layers may significantly boost the effects of retraining procedure. One of the possible compression flows for big models, not discussed in this work, is based on the idea of incremental pruning and quantization. In this flow, introducing the sensitivity list allows picking fragments of the model, which are pruning prone and should be addressed as first.
An alternative could be developing a different way calculating the maximum number of steps. An interesting observation in terms of FPGA resources consumption is irregularity of its distribution. For instance, in the course of the experiments, we found out that there is a component in the model, which contributed vastly to the overall resources consumption. Because of the non-hierarchical structure of the generated HDL code using MyHDL, it was hard to determine which section it was. Fortunately, after some examination we determined tanh activation function as the culprit (see Appendix D). As the next step, the module was re-implemented with tanh as a linear approximation of the original function rather than Taylor function expansion. This change significantly reduced the overall resources consumption (see Table  A1 ).
Conclusions The proposed mapping methodology and tools may be used in FPGA-based IoT devices for model compression. It can also be employed for distributed systems since its application in every node is independent and also can be ported across processing elements within the sensors network. The methodology also proposes steps to be taken in order to assess potential resources consumption of the system. It may be incorporated as a backbone element in a larger environment for a generation or ad-hoc reconfiguration of a network infrastructure. Our DL2HDL tool matches or supersedes the state-of-the-art solutions in terms of the achieved latency, but the architecture is more uniform and allows mapping directly from Python to HDL. Our solution generates HDL models with three orders of magnitude lower than LeFlow  [8]  and comparable to hls4ml  [10].
Results We created CNF files in DIMACS CNF format, a standard input format accepted by most SAT solvers. The files start with a line that begins with p cnf followed by the number of variables and the number of clauses. One line for each clause then follows. Each clause is expressed by listing the indices of the variables involved in the clause separated by spaces, using negative numbers for negated variables. A zero is added at the end of each clause line. The files were created with a Python script (source code available at github. com/etonello/regulatory-network-sat). (x 1 , x 2 , x 3 ) = (1 -x 2 x 3 , x 3 , x 1 x 2 x 3 -x 1 x 2 -x 1 x 3 -x 2 x 3 + x 1 + x 2 + x 3 ). The path of length 4 leaving the origin reaches a fixed point; none of the negative circuits admitted by regulatory graph are local.
Conclusions: By performing RNAi experiments for three poorly investigated EGFR isoforms, we were able to successfully predict 1140 putative target genes specifically regulated by EGFR isoforms II-IV using the developed Bayesian Gene Selection Criterion (BGSC) approach. This approach is easily utilizable for the analysis of data of other nested experimental designs, and we provide an implementation in R that is easily adaptable to similar data or experimental designs together with all raw datasets used in this study in the BGSC repository, https://github. com/ GrosseLab/BGSC.
The two-step BGSC procedure of first defining easily interpretable conceptual groups of genes associated with different EGFR isoforms and subsequently classifying genes based on the approximated posterior probability to these groups seems to be a promising approach in such a situation, and this approach is readily adaptable to other and more complex experimental designs. The datasets analyzed during the current study and the R-scripts for reproducing the results and plots of this work are available in the BGSC repository, https://github. com/GrosseLab/BGSC.
Illumina BeadChip analysis was conducted the EGFR-cDNA (NCBI-ref NM_005228. 3) for siRNA I at the microarray core facility of the Interdisciplinary at positions 4094-4116 and for siRNA ALL at posi- Center for Clinical Research (IZKF) Leipzig (Faculty of tions 1258-1278 (Table 2). To ensure that the EGFR- Medicine, University of Leipzig). 250 ng RNA per sam- specific siRNAs and the nonsense siRNA do not inter- ple were ethanol precipitated with GlycoBlue (Invitrogen) act with other transcripts, we used the sequences of as a carrier and dissolved at a concentration of 100- siRNA I , siRNA ALL , and nonsense siRNA to perform 150 ng/μl before probe synthesis using the TargetAmp™- a BLAST search with Nucleotide BLAST against the Nano Labeling Kit for Illumina Expression BeadChip (Epi- human-genome database (http://www.
Ribosome Allocation Is Largely Explained by the Copy Number and Length of ORFs. To uncover determinants of translation speed in exponentially growing yeast cells, we analyzed a recently published ribosome footprinting dataset obtained in this system (4), from the perspective of the TASEP model of translation (Fig. 1A ). Denoting by N the number of ribosomes bound to a coding region of L codons, and assuming that the rate with which a ribosome completes the polypeptide chain is given by the product of the frequency of finding a ribosome at the stop codon N=L, and the effective ribosome translocation (and termination) rate k el , the change in the number of ribosomes bound on the mRNA is given by the differential equation dN=dt = k in m -ðk el =LÞN. Here, m is the number of mRNA molecules, k in is the effective rate of translation initiation on an mRNA molecule, and we assume the broad region of parameter values where ribosome collisions are rare.
This model predicts that the number of ribosome-protected fragments (RPFs) mapping to a specific mRNA is proportional to the mRNA abundance, the length of the ORF, and the ratio of the effective rates of initiation and elongation, N ≈ ðk in =k el ÞLm. Testing this prediction with the experimental dataset mentioned above (4), we found that the number of RPFs mapped to a specific mRNA indeed correlated very well with the relative abundance of the mRNA estimated by mRNA sequencing. However, further incorporating the ORF length slightly but significantly (z score = -4. 74, Fisher z-transformation test) reduced the correlation, rather than improved it (SI Appendix, Fig. S1 ). As it was reported that the estimation of mRNA abundance by RNA sequencing is a critical aspect to control in the analysis of ribosome footprinting data (4), we repeated the analysis of the scaling behavior with estimates of mRNA abundance from another RNAseq dataset, obtained by sequencing of RNAs purified directly with oligo(dT) from yeast cell lysates  (17).
The 3′-end bias in ORF coverage by RNA-seq reads, which strongly affects the accuracy of mRNA abundance estimates (4), was limited in this dataset, comparable with that inferred from the data obtained with a Ribozero protocol (SI Appendix, Fig. S2 ). We found that, when using the RNA samples obtained by oligo(dT)-based purification, both the mRNA level and ORF length contributed to the number of RPFs, as expected (Fig. 1C ). We therefore used this mRNA-sequencing dataset for the analyses described below, but present similar results with the RNA-seq data from reference 4 in SI Appendix. The mRNA levels alone explained 65% of the variance in RPF numbers. Further taking into account the ORF length increased this number to 74% (Fig. 1C ), setting an upper bound of 25% on the variance in RPFs that could be due to differences in ribosome density along transcripts.
19  (Fig. 2A ). Results of protein synthesis rate and ribosome density for individual ORFs, computed using either oligo(dT) or Ribo-zero RNA-sequencing datasets, are given in Datasets S1 and S2. In the model, ORFs that initiate translation at very low rates have very low protein output and their ribosome coverage per codon reflects the rate of translation elongation (Fig. 3A ). Al-though our dataset contained only few proteins with very low synthesis rate, the ∼10-fold range in ribosome coverage that we infer this way is comparable to the 20-fold range that we observed for ORFs with the same RPF density (Fig. 3B ). The model also predicts that protein synthesis rate and ribosome coverage increase linearly with the initiation rate, as long as ribosome collisions do not halt elongation (Fig.
19. The black contour indicates 90% of the empirical distribution approximated through the 2D kernel density estimation from the R package "MASS. " Combining Protein Synthesis Rate and Ribosome Footprint Density Reveals the Speed of Translation Elongation. If ribosome queueing events are rare, proteins should be synthesized at the rate at which new chains are initiated s ' k in , the ribosome density can be approximated as ρ ' k in =k el , and the ratio of protein synthesis rate to ribosome density (SDR) will be the effective translation elongation rate, s=ρ ' k el. We used this relationship to uncover features of the mRNA or of the corresponding protein that most strongly affect elongation (Fig. 4 ). The positive correlation of the SDR with the average speed of ribosomes, calculated from the normalized ribosome densities at individual codons, served as control (Fig.
S5  and S6 ) reduced the predictive power of the model only marginally (correlation coefficient, 0. 65). The linear model also highlighted the most explanatory features, which were (in order of the significance of their correlation coefficient being different from zero) the ribosome density (P < 2e-16), isoelectric point pI (P = 3. 64e-15), codon adaptation index (P = 1. 22e-11), molecular weight of the encoded protein (P = 9. 83e-05), ORF length (P = 0. 000118), GRAVY index (P = 0. 000256), tRNA adaptation index (P = 0. 000977), domain coverage (P = 0. 006493), and fraction of optimal codons (P = 0. 009106). The weights of individual features in the linear model are shown in Dataset S3. Qualitatively similar but somewhat lower in magnitude correlation coefficients were obtained when the other RNA-seq dataset, from ref.
0. 68). The most informative amino acids were Arg (P value of the coefficient being different from zero in the linear fit = 2. 35e-10), Pro (P = 1. 68e-07), Ala (P = 1. 87e-07), Glu (P = 1. 14e-06), and Ser (P = 0. 00861). See Dataset S3 for the inferred weights of these features. Ribosomal Protein mRNAs Are Translated Slowly for Their Ribosome Densities. Assuming that all initiating ribosomes complete translation and that they elongate at similar rates across transcripts the ribosome footprint density is generally used as an estimate of the translation efficiency, defined as the number of protein molecules produced per mRNA molecule per unit time  (12). As RPs represent a high translation burden for the cell, their transcripts should be highly optimized for translation. Indeed, RP-encoding genes have a significantly higher tRNA adaptation index than other genes (Fig.
Perturbed Translation Dynamics in Ribosomal Protein Deletion Strains. Deletion of specific RP genes has been associated with changes in translation and replicative life span  (17, 38). In particular, deletion of rpl7a (Δrpl7a strain) led to ribosome assembly defects and overall decreased protein synthesis (measured by the incorporation of a methionine analog), whereas deletion of rpl6a (Δrpl6a strain) led to increased protein production  (17). To determine how the translation parameters of individual ORFs are affected in these strains, we measured the protein synthesis rates by pSILAC and analyzed them jointly with ribosome profiling data obtained before  (17). Results for individual ORFs are given in Datasets S4-S6, for the wild-type control, Δrpl6a, and Δrpl7a strains. We found that accumulation of light peptides in the mutant strains was less well explained by a linear fit compared with the wild-type strain, especially for the high-translation Δrpl6a strain (SI Appendix, Fig.
Taking advantage of abundant data generated for the yeast Saccharomyces cerevisiae and measuring protein synthesis rates with the high transcriptome coverage afforded by currently available methods, we evaluated the translation initiation and elongation rates for individual yeast ORFs. Using additional datasets to estimate absolute protein synthesis rates as well as ribosome densities per codon, we found that the translation initiation rate varies over a ∼100-fold range among yeast transcripts (Fig. 3 ). This is consistent with an initial estimation of translation efficiency based on ribosome profiling (2) as well as with the results of a study that used these data to parametrize a whole-cell model of translation, which found that the time between initiation events on individual mRNAs (5th to 95th percentile) is from 4 to 293 s  (45).
S2 ) and the transcript abundance estimates showed limited systematic differences between the 2 datasets (SI Appendix, Fig. S12 ). However, it is interesting to note that the data obtained with the optimized Ribo-zero protocol did not yield the expected scaling of RPFs with mRNA length and abundance, even though the mRNA abundances inferred from these data were very highly correlated with the number of RPFs. In the future, it will be interesting to determine the translation status of mRNA species that are preferentially enriched by different protocols. It is also unlikely that the wider range in translation initiation rate is due to error in estimating the protein synthesis rates because our analysis only included ORFs for which peptide accumulation was well described by a constant accumulation rate.