dataesr/scientific-paragraphs-categorization huggingface dataset:
Insertion of a new dataset While in theory, SBTs can be updated in O(height × b) (height being the tree height), in practice, the leave clustering might make it necessary to rebuild the whole structure from time to time. In theory, sequence Bloom matrices can be updated in O(b) time by adding a new Bloom filter to the others and updating the inverted index. From a practical point of view, neither the latest SBT nor matrix provide an insertion operation in their implementation. SeqOthello's update is implemented and in O(n), as it requires to re-compute the hash function. Query In any case, the worst-case time complexity for queries is in O(n). Thus, the number of random access impacts is critical for runtimes in practice. In the particular case where a single query is present in a single dataset, sequence Bloom tree structures allow a O(log(n)) random accesses.
Implementation Bloom filters settings and false positives In sequence Bloom trees structures, the BF size is adapted to all distinct k-mers over the database. In matrix structures, the BF size is chosen according to the largest k-mer set among datasets. However, a single size does not fit all Bloom filters in the general case. Therefore, COBS  [23] , an improvement of BIGSI, Bloom filters are folded in order to adapt to datasets of different sizes. Motivated by  [19]  work on false-positive tradeoffs, both SBT and matrix approaches chose to work with a single hash function to optimize the query time. The BF sizes can be set to control the average false positive rate for SBT and matrix structure. It is conversely not possible to control it in SeqOthello.
Compression The SBT structure implies redundancy in the internal nodes, following the consecutive unions of Bloom filters. This redundancy motivated the application of compression schemes in SBTs. Recent SBT approaches  [22, 24, 21]  propose compression strategies through an improved representation of the informative bits at each level of the tree. Compression does not change the false-positive rates but can increase the query time. Matrix structures' query requires rows of the same size in the matrix, and therefore it is not adapted to compression schemes. The folding strategy used in COBS allows to resize Bloom filters according to the dataset size to limit memory usage, but this optimization comes at the price of a higher false-positive rate. SeqOthello compresses the presence/absence vectors by buckets, using different compression strategies (RRR, delta encoding) according to the sparsity of the vectors.
Disk/RAM requirements Both matrix and tree structures take O(n×b) space. One can expect SBTs to be a little less efficient in space since they maintain ∼ 2n Bloom filters instead of n for matrix structures. However, their compression scheme can make them less costly than sequence Bloom matrices in practice at a fixed false-positive rate. SeqOthello requires a O(N log(n)) bits of space, with N the number of distinct k-mers. All three structures keep all preprocessed data (datasets BFs or k-mer sets) on the disk as temporary files during the indexes construction, which leads to a heavy temporary disk footprint. During the query, each approach is loaded partially in AMQ method clustering compression overhead update query COBS no no (folding) O(1) yes* O(b) O(n) HowDeSBT yes yes O(n) yes* O(height × b) O(n) SeqOthello NA yes O(n) yes O(n) O(n) PAC no yes O(log(n)) yes O(b) O(n) Table  1 : Comparison between the most recent AMQ approach and ours.
We give worst-case complexities. The "clustering" columns denote whether the methods necessitate a prior clustering of the Bloom filters. "compression" indicates if the method compresses the Bloom filters. "overhead" indicates the time overhead for constructing the index after input preprocessing. "update" gives the time complexity for inserting a new dataset (heigh is the tree height). "yes*" means that the operation is theoretically feasible but not yet implemented. NA stands for not applicable. "accesses for query" gives the expected order of magnitude of memory accesses for a single query in the general case. RAM. Using a parameter, COBS and previous generations of SBTs can be fully loaded on request. It is not the case for HowDeSBT and SeqOthello.
Use cases With their stacks of Bloom filters, sequence Bloom matrices are well adapted to datasets of similar sizes, such as collections of similar genomes. SBT and SeqOthello's compression factors will be particularly well suited to somewhat similar datasets, such as sequencing data from cohorts. The recursive query that prunes the query space at each level also makes SBT structures suitable for querying many similar datasets. Similarly, we can expect a batch of very similar queries to be accelerated in SBTs as they will all descend in identical (cached) subtrees. On massive query sequences and diverse query k-mer sets, sequence Bloom matrices or SeqOthello can be preferred because they ensure less random accesses in the worst case when k-mer look-ups become a bottleneck.
An introduction to PAC through conceptual comparison to other AMQs PAC is based on BFs as sequence Bloom trees and matrices. It inherits from SBTs by keeping the idea of organizing data in tree structures and finding an efficient encoding for the internal bits of the tree. As SBTs, it compresses its leaves, but contrary to all other methods, it never has high disk footprints during the construction. On the query side, PAC is also related to COBS because queries are handled using an inverted index on a matrix of BFs. During the query, its number of random accesses is constant as in COBS and SeqOthello's schemes. Table  2. 2. 3 summaries aspects of PAC and other methods' comparison. In the following, we expend on PAC's structure and novelties.
Aggregative Bloom Comb Trees PAC relies on comb-trees to structure the set of Bloom filters representing each dataset. It builds two comb-trees that share the same set of leaves. As in SBTs, these trees' inner nodes are unions of Bloom filters, but for PAC, they are organized in a binary left-comb tree and another binary Bloom right-comb tree (see Figure  2  for an example of the difference between a tree used by SBTs and a single binary Bloom comb-tree). Similarly to sequence "Bloom trees" we name these trees "Bloom comb-trees". Given a list of BFs BF 1 , BF 2 ,. BF n representing datasets, The binary Bloom left-comb tree is such that the leftmost inner node BF n+1 represents BF 1 ∪ BF 2. Each other inner node up to the root contains the union BF of its child inner node and of a leaf BF, then we have BF n+1 ⊆ BF n+2 ⊆.
Second, n Bloom filters are the leaves of both combs. Panel A of Figure  3  shows an example of two Bloom comb trees and the aggregative Bloom filters that represent them. An aggregative Bloom filter represents a run of 1's in an inclusion series of BFs. Figure  2 : Left: a SBT structure. Right: a Bloom left comb-tree for the same database. Four datasets of a database are represented using grey shapes. Toy example Bloom filters are represented as bit vectors associated with shapes, and the content of each node is recalled. In the case of the Bloom comb tree, the inner nodes' Bloom filters (highlighted zone in grey) are not explicitly represented but are encoded using an aggregative Bloom filter instead (integer vector). Runs of 1's corresponding to the integers are colored. For instance, the leftmost 1 is found at levels 1 and 2, therefore a run of length 2 in the vector.
Update PAC by inserting new datasets As our structure is accumulative, adding new datasets does not require changing the index structure. New datasets can be added by constructing their BFs and updating cells of the aggregative Bloom filters when needed. We implemented this functionality to allow the user to insert a dataset collection to an existing index. This insertion presents a very similar cost to building an index from the said collection. However it is way less costly than rebuilding from scratch an index including previous and novel datasets.
A B slice in the inverted index The aggregative Bloom comb trees constructed for a given partition are represented using leaves BFs (in grey) and two aggregative BFs V l and V r. Panel B: a single query in the same partition as Panel A. The index has been inverted. A k-mer enters a PAC by finding the partition corresponding to its minimizer. V l and V r are loaded, and a search space (here  [3]  only) is defined given the position obtained by hashing the k-mer (here the k-mer's hash value is 2). Then, a slice is extracted from the inverted index at the given position, and bits are checked to find 1's. Here there's a single position, so necessarily an 1. Memory usage and parallelization Bloom filters are computed in RAM and serialized on disk to avoid heavy memory usage. This way, only one Bloom filter is stored in RAM at a given moment (C if C threads are used to treat the datasets in parallel).
Preamble All experiments were performed on a single cluster node running with Intel(R) Xeon(R) CPU E5-2420 @1. 90GHz with 192GB of RAM and Ubuntu 16. 04. In the following experiments, we show RNA-seq indexing use cases for PAC. We compared PAC against the latest methods of the AMQ paradigm, i. e. , HowDeSBT  [21]  for SBTs and Se-qOthello  [15]. HowDeSBT is the third and most recent generation of SBTs. We also tested COBS  [23] , the most recent sequence Bloom matrix, although it is not designed to work on sequencing data but rather on genomes. We used kmtricks  [27]  for an optimized construction of HowDeSBT and its Bloom filters (commit number 532d545). SeqOthello (commit 68d47e0) uses Jellyfish  [28]  for its pre-processing, we worked with version 2. 3. 0. COBS's version was v0. 1. 2, and PAC's commit was a2a0112.
Indexing 2,500 human datasets and comparison to other AMQ structures The dataset first used in the initial SBT contribution  [19]  has become a de facto benchmark in almost any following paper describing a related method. We make no exception and benchmarked PAC and other AMQ structures on this dataset. It contains 2,585 human RNA-seq datasets, with low frequency k-mers filtered according to previous works  [22]   2  , resulting in a total of ∼ 3. 8 billions of distinct k-mers. The input files are represented as compacted de Bruijn graphs generated by Bcalm2  [29]  in gzipped FASTA files. We also settings used in previous benchmarks and used a k value of 21. We chose COBS, PAC, and HowDeSBT's settings to have an average 0. 5% false positive rate. Notably, SeqOthello's false positive rate cannot be controlled.
In Table  2 , we present the costs of indexing this database with the different tools. Several observations can be made. First, PAC does not generate any temporary disk footprint; only the final index is written on disk, while HowDeSBT, SeqOthello, and COBS generate many temporary files that are an order of magnitude larger than the index itself. This cost is not to neglect as it could become a hindrance in a disk-limited environment and should be seen as a scalability problem to index more significant instances. Another observation is that PAC and COBS's preprocessing and index construction steps are comparable and faster than the other tools, showing at least a two-fold improvement of required CPU time. The total computational cost of a PAC index is improved threefold over SeqOthello and six-fold over HowDeSBT.
Each tool was configured to filter unique k-mers before indexing to remove most sequencing errors (we used k = 31). Since COBS does not provide such an option and produces huge index files, we did not include it in this benchmark. To approximate the order of magnitude of the amount of (non-unique) distinct k-mers, we used ntcard  [30]. For example, the batch of 2 11 datasets contains approximately 3 billion k-mers to index, while the 2 14 dataset contains more than 40 billion k-mers. In Figure  5  (Panel A), we report the CPU time required for the indexing construction along with the end-to-end index production (including pre-processing) to display their evolution on databases of increasing size. On Panel B of Figure  5 , we report the disk usage of the index along with the total disk usage required during the construction (including pre-processing).
We can make similar observations to the first experiments. PAC is the only tool that managed to build an index of the 32,000 human RNA-seq samples, in ∼5 days (∼60 CPU days), using 22GB of RAM and for a total index size of 1. 1TB. PAC's construction is faster than state-of-theart for the whole construction process. It produces slightly bigger indexes but requires much less external memory. SeqOthello crashed during index construction on the 2 12 datasets experiment, and HowDeSBT failed to end before our ten-day timeout on the 2 11 datasets experiment. PAC was the only tool to build an index on the 2 13 , 2 14 , and 2 15 datasets.
Query results As pointed out in  [21] , SBTs' query time has a relatively high variance and is impacted by batch effects. For a fair comparison, we designed a similar experiment to the one presented in HowDeSBT's paper, with sequences batches of increasing sizes. We repetitively selected random transcripts from human RefSeq (using seqkit  [31] ), and gathered 10, 10, 5, 3 and 3 batches of sizes 1, 10, 100, 1,000 and 10,000 transcripts. We report HowDeSBT, SeqOthello, and PAC's CPU time on each batch size in Figure  6. We warmed the cache prior to profile the queries. We observe that, according to our description in Section 2. 2, sequence Bloom tree structures perform the best on small query sets. 5 : Results on increasing human dataset sizes. We report CPU hours for constructing the different indexes, including the pre-processing (methods preceded with "total") and the index alone.
Interestingly on a larger batch of 10,000 transcripts, SeqOthello and PAC presented very similar results (10 and 5 minutes, respectively). Those results denote that the query time of these two tools is dominated by loading the index. To assess when query time became predominant, we queried a large batch of 100,000 transcripts for which PAC lasted 11 minutes (2 CPU hours) and an even larger batch of 500,000 transcripts handled by PAC in 40 minutes (6. 5 CPU hours).
It stands among the fastest methods for index construction while maintaining low disk and memory footprints. Query We showed PAC's ability to query 500,000 human transcripts in less than an hour, being the fastest in wallclock time, and comparable to SeqOthello in CPU time. The worst-case query complexity remains the same for all methods, O(n) for a k-mer present in all n datasets. We saw that inverted indexes methods perform Θ(1) random accesses but still need to read a bit-slices of size O(n). Using aggregated Bloom filters, PAC improves these indexes, as in favorable cases (kmers present in a single dataset with no collision), PAC answers in O(1) since only one position can and will contain a 1. Moreover, since we implement our bit-slices as sparse vectors, enumerating the 1's is done in Θ(o) (o being the number of 1's) instead of O(n).
Future work AMQ's trade-off for efficiency is to accept false positives during the query. Recently, Findere  [32]  provided a solution for reducing the false-positive rate of any AMQ data structure indexing k-mers while speeding up the queries. We plan on integrating Findere to later versions of PAC and thus, reaching a negligible amount of false positives in the structure. While much effort has been put into designing efficient indexes, the query expressivity has remained practically the same since the first SBT paper. Two data-structures  [33, 34]  improved on that question by allowing to query k-mers abundances instead of presence/absences in collections. These methods are based on exact k-mer representations and show scalability up to 2,500 human RNA-seq. As a follow-up for PAC, we would like to augment the index with probabilistic structures such as count-min-sketches for efficient abundance storage per dataset.
Acknowledgements The authors would like to thank  Léonid  for a renewed sense of what tiredness, teamwork and joy are. We also thank  Daniel Gautheret  for providing us with tips on the  SRA API ,  Bastien Cazaux  for proofreading the manuscript, and  Anatoliy Kuznetsov  for his outstanding Bitmagic library and sustained support. This work was supported by a grant by the  Agence Nationale de la recherche  for the project " Transipedia  " [ ANR-18-CE45-0020 ].
We develop this approach on a 2D plate with a hole as well as a 3D case with spot-welded plates undergoing fast deformation, representing nonlinear elastoplasticity problems. Neural networks are trained using simulation data generated by explicit dynamic FEM solvers. The PGANN results are in good agreement with the FEM solutions for both test cases, including those in the training dataset as well as the unseen dataset, given the loading type is present in the training set.
g. , local plasticity  [9, 10] , local crack propagation  [11] ). An overview of such techniques can be found in  [12]. They allow the global mesh to remain unchanged while the local problem is refined in space and time. The non-intrusive local/global strategy relies on an iterative exchange of interface quantities between global and local computations. It was successfully applied to coupling of 2D and 3D models in thin composite panels with local stress concentration and debonding  [13]. The same approach was extended in the context of explicit dynamics in  [14, 15]. Sub-modelling techniques are another non-intrusive option widely available in Finite Element Method (FEM) software and used in industry. However, they are referred to as "one-way coupling" because the data exchange occurs from global to local, and the global solution is not updated after the local solution has been modified.
The ReLU activation function σ can be defined as: σ(a) = max(0, a) (5) where a is the input of the node. The NN training has been performed in the Python environment using TensorFlow. We used the TensorFlow 2. 0 API tf. distribute. MultiWorkerMirroredStrategy to distribute the training across multiple machines (up to 4 machines) with single NVIDIA Quadro RTX 4000 GPU cards. Source codes are available at https://github. com/afsalpt1/ROM-PGANN. Using the TensorFlow API tf. data. Dataset, efficient input pipelines are written for the data generated in Section 4. 1. The pipeline allows for easy access to the data to distribute the training across multiple machines. We set the training batch size to 128 to ensure better generalization of each batch. To train the NN, the Adam optimization algorithm is used.
Data generation and post-processing The data required for training the NNs are obtained by solving the reference model shown in Figure  6  using a nonlinear explicit FEM dynamic solver. The simulation spans from an initial time t 0 = 0 ms to a final time t end = 2. 5 ms. To discretize the time domain, a constant time step ∆t l = 2. 3 × 10 -4 ms is employed, calculated based on (2). Consequently, each simulation comprises approximately 10,800 time steps. For simplicity it is assumed here that the ratio ∆t g /∆t l is equal to 10. The first dataset comprises five principal loading directions, namely translational displacements (s x , s y , s z ) and rotational angles (θ x , θ y ). Each load direction undergoes eight simulations with varying magnitudes, including four positive and four negative magnitudes, resulting in a total of 40 simulations.
The second dataset is created to enrich the first dataset by including additional loading directions. Unlike the single-direction loading in the first set, the second dataset involves simultaneous monotonic loading in two directions, as outlined in Table  1. This results in a total of 24 distinct loading directions. For each combination of load directions, four simulations are performed, encompassing two positive and two negative magnitudes, thereby generating additional 96 simulations. Consequently, the second dataset comprises a total of 136 simulations. Further information regarding the magnitude and direction of loading can be found in Appendix A. While alternative approaches like Design of Experiment (DoE) methodologies could be employed to efficiently generate optimal data parameters, it is important to note that the simplified load cases used in this article primarily serve to present the core concept of our strategy.
Training and verification of PGANN In this section, we explore the individual training and verification of all four sub-networks of the proposed PGANNs. First we outline a systematic approach for determining the optimal hyper-parameters of a NN. The hyper-parameters we focus on include: the number of layers, the number of neurons or filters in dense or convolutional layers, the choice of activation functions, the type of loss function, the learning rate, and the batch size, among others. Then to evaluate the accuracy of the trained model, aside from the dedicated test dataset comprising 10% of the overall data, four additional test cases are generated, which are neither included in the first dataset nor in the second dataset. To avoid confusion, we will refer to the former as the test dataset and the latter as test cases.
This involves incorporating one deconvolutional layer with same shape as its preceding layer at positions A, B, or C as depicted in Figure  10. The results are tabulated in Table  3. The results indicate that adding a single layer at positions A and B leads to the lowest NMAE values. The effect of incorporating multiple layers at position A and B is shown in Figure  10 , which indicates that adding eight such layers is optimal. The inclusion of dense layers after each input layer does not seem to have a significant impact on the training process. After 1,000 epochs, the NMAEs of the optimized (NN I) on the first and second test datasets (10% of total data) are 6. 52 × 10 -4 and 4. 8 × 10 -3 , respectively. Furthermore, we investigated the robustness of the trained NN I on unseen data using the test cases presented in Table  2.
Using a learning rate of 5 × 10 -5 and a batch size of 256 yielded optimal results for both datasets. By comparing the error and its corresponding training cost, we concluded that it is best not to add layers at positions A, B, and C. Additionally, we found that including 2 dense layers after v t Γ and ω t Γ resulted in the best performance. One of the main hyper-parameters to optimize in autoencoder is the size of latent vector z t u. Figure  13a  shows the performance with respect to the size of the latent vector after 100 epochs. It can be observed that the model has the best performance when using 64 latent variables. We choose a latent vector of size 50 as an optimal value as there is no significant decrease in NMAE with increase in size of z t u. Figure  13b  displays the latent vector of test case 1 with size 50.
After 1,000 epochs, the NMAEs of the optimized model on the first and second test datasets are 6. 38 × 10 -4 and 4. 38 × 10 -3 respectively. Figure  14  shows the performance of the network and the result of reconstructed displacement using 50 latent variables. From the first two test cases, it can be observed that, although the input is scaled by a large magnitude, the network is able to reconstruct local displacement accurately just using 50 latent variables. However, the reconstruction is poor in test case 3. Notably, in test case 4, where a combination loading similar to that in the training set is present, the reconstruction performance improves. The use of skip connections in an autoencoder architecture can have a significant impact on its performance  [34]. Skip connections allow for the flow of information from the input layer directly to the output layer, bypassing the bottleneck or compressed representation in the middle layers.