dataesr/scientific-paragraphs-categorization huggingface dataset:
Datasets In this work we generate datasets using two different sequence simulators, Seq-Gen  (Rambaut, 2017)  and Evosimz  (Zou et al. , 2020). The first model of evolution we tested our method on is a standard one, based on the PAM exchangeability matrix  (Dayhoff et al. , 1978) , widely used in the phylogenetics community, and implemented in several phylogenetics tools  (Rambaut, 2017; Huelsenbeck and Ronquist, 2001; Yang, 2007). We simulated trees as in  Zou et al. (2020) , i. e. , for each n, random n-leaves tree topologies were generated with the ete3 (Huerta-Cepas et al. , 2016) populate() function and branch lengths were drawn uniformly between 0 and 1. The corresponding alignments were then obtained by simulating evolution along the branches with the PAM evolution model implemented in the software Seq-Gen  (Rambaut, 2017).
We used such simulations to train Phyloformer and investigate the effect of the number of leaves or the number of sites on the performances of various pieces of software. The second model was presented in  Zou et al. (2020)  and implemented in their software Evosimz. It is based on a model that combines 9 different amino acid replacement matrices, a Gamma distribution for rate heterogeneity across sites, and site-specific amino acid equilibrium frequencies across sites. Further, the model can simulate heterogeneity across branches in both the rate of sequence evolution and equilibrium frequencies. In this simulator, the branch lengths found on the trees do not exactly correspond to expected numbers of per site substitutions, see  Zou et al. (2020). However, this peculiarity has no consequence over our evaluation of the reconstruction methods, because we rely on Robinson-Foulds distances between true and estimated phylogenies that do not depend on branch lengths.
We relied on existing datasets that make phylogenetic reconstruction challenging  (Zaharias et al. , 2022). These datasets have short sequences (200 amino acids) and long underlying branch lengths. Zaharias et al. (2022)  looked at three different combinations of parameters (with distinct branch length distributions and amounts of time heterogeneity). For each of these parameter configurations they moreover considered different branch lengths distributions (corresponding to increasingly difficult phylogenetic estimation), multiplying by 10, 100 and 1000 their average, obtaining therefore 12 test datasets in total, each made of 20 trees along with the corresponding alignments containing 20 sequences of length 200. These datasets allowed us to evaluate the applicability of our method to a fairly complex model and to directly compare our results to those reported in  Zaharias et al.
(2022). 5. 2 Performance on alignments simulated using Seq-Gen 5. 2. 1 Reconstruction accuracy on alignments simulated using Seq-Gen We trained our network on 100, 000 alignments generated using Seq-Gen (Rambaut and Grass, 1997) (see section 5. 1) with n = 20 and a sequence length of L = 200. We then generated additional simulations to investigate the effect of the number of leaves or the number of sites on phylogenetic reconstruction accuracy and on the resources required by various pieces of software. To investigate the effect of the number of leaves, we generated 1500 alignments of length 200 sites, varying the number of leaves between 10 and 100 by increments of 10. To investigate the effect of the number of sites, we generated datasets with 30 leaves, but varied sequence length between 200 and 1500 by increments of 100.
In both cases, 150 tree-alignment pairs were generated for each parameter setting. We compared our method to the IQ-TREE (Minh et al. , 2020) implementation of maximum likelihood, and to a standard distance method implemented in the R phangorn package  (Schliep, 2011)  which computes the maximum likelihood estimates of pairwise evolutionary distances and then reconstructs a tree using NJ  (Saitou and Nei, 1987). Both methods were run with the correct model of sequence evolution. Figure  4  shows the normalized Robinson-Foulds distance  (Robinson and Foulds, 1981)  between the true and reconstructed trees for all three methods. Phyloformer performs better than the standard distance method across all parameter settings, and rivals with maximum likelihood in settings that most closely resemble its training data: when the number of leaves is close to 20, and when the number of sites close to 200.
The degraded performance of Phyloformer on larger trees could be explained by the fact that larger trees have a longer average path length between leaves, which results in longer distances (given by the cumulative branch lengths along the path). These long distances can fall outside the range of distances seen by the network during training. The issue could thus be overcome by training Phyloformer on larger trees and/or with a wider range of branch length distributions. Similar results hold under mild model misspecification, namely in the situation where the data has been generated under the WAG model of Although our network was trained only on simulations generated with the easiest set of parameters (first dataset, upper left corner) it manages to always concentrate more mass towards low RF distance values than all other methods.
Performance on alignments simulated using Evosimz We relied on alignments simulated using Evosimz  (Zaharias et al. , 2022)  to assess the performance of several phylogenetic reconstruction methods on data simulated using a sophisticated model (see section 5. 1). To test the robustness of our model we trained Phyloformer only on alignments generated with the configuration resulting in the easiest simulations among the 12, i. e. the one in which all methods perform best (corresponding to the test dataset in the upper left corner of Figure  5 ). As above, we used a training dataset of 100, 000 alignments with n = 20 and L = 200. In Figure  5  we show the violin plots of the Robinson-Foulds distances between real and predicted tree topologies, for all 12 datasets in  Zaharias et al. (2022). We evaluated 5 different methods: Phyloformer, IQ-TREE with ModelFinder (which chooses the best reconstruction model according to the Akaike information criterion)  (Nguyen et al.
Maximum Parsimony and the distance method often perform better, but the best performing methods are the maximum likelihood approach IQ-TREE, and our method Phyloformer. For Phyloformer the distribution is systematically more skewed towards low RF distance values than for IQ-TREE. Further, if we only look at the mean RF distance our method performs better than all other methods on 9 out of the 12 datasets (Table  1 ). These results show that Phyloformer can perform better than Maximum Likelihood when the data has been simulated under complex models of sequence evolution.
Speed and memory requirements on alignments simulated using Seq-Gen For IQ-TREE (SOTA), Phyloformer and PAM+NJ we evaluated the memory and time taken to analyze all 150 trees for a given number of sites and number of sequences. All methods were limited to using a single CPU core, and Phyloformer was allowed to use the GPU. Time and memory footprints were measured in triplicates on the Seq-Gen datasets, using the benchmark options in snakemake  (Mölder et al. , 2021)  on a machine with an Intel Core i7-10700 processor with 32GB of RAM and equipped with an Nvidia RTX A4000 GPU. Memory footprint for Phyloformer, which uses the GPU, were obtained with a call to torch. cuda. max_memory_allocated in pytorch  (Paszke et al. , 2019). All methods increase their footprint as the number of sequences or sites grow.
(2010)  or mutational biases (e. g. , CpG hypermutability,  Meunier et al. (2005) ) have not been included. It is unclear how the various methods would fare on data simulated with models that reproduce these processes. Overall, precisely evaluating the level of robustness of Phyloformer will still require some work, and will be a necessary step to making it usable as a routine tool. Depending on the possibility of a single network to provide robust predictions in all settings, one may need to rely on several ones or to offer the possibility to fine-tune to a specific situation. Importantly, future versions should also propose a measure of uncertainty along with the predictions  (Gal and Ghahramani, 2016; Charpentier et al. , 2022). A natural follow-up question is whether Phyloformer could perform as well as maximum likelihood methods on empirical datasets.
This question is difficult to answer with empirical data, because the true phylogeny is rarely known without a doubt. This is particularly true for gene trees, which can differ from the species tree due to gene duplication, loss, transfer, or incomplete lineage sorting  Szöllősi et al. (2015). However, in cases where the species phylogeny is known, one could count the proportion of reconstructed gene trees or subtrees that match the species phylogeny. Another way to answer this question would be to use realistic simulations, where the generating phylogeny is known, and where the model of sequence evolution has been shown to generate datasets that are realistic, i. e. , very hard to distinguish from empirical datasets. In future work, we will use generative models  (Kingma and Welling, 2014; Goodfellow et al.
, 2014; Yelmen et al. , 2021) , relying on an architecture similar to Phyloformer, to assess how realistic current simulations are and to produce more realistic ones. A more realistic simulator of sequence evolution will allow for more accurate evaluations of reconstruction methods and conceivably lead to better performances on real data of our network, retrained on such more true to life simulations. Phyloformer relies on two technical innovations to learn how to accurately reconstruct phylogenies from MSAs. Firstly, it adresses phylogenetic reconstruction as a regression problem. Previous approaches relied on a classification task between a number of competing tree topologies. The number of tree topologies grows super-exponentially with the number of leaves, which makes such an approach unable to scale to MSAs with more than a few sequences.
Secondly, through a transformer-based architecture, Phyloformer uses self-attention, both across sites and pairs in the MSA. Self-attention provides an expressive mechanism that models how each pair (resp. site) should inform the others within the MSA. This mechanism is compositional: it relies on learnable functions that each apply to individual objects (pairs or sites) and whose outputs are assembled to determine how information is shared. This compositional structure also makes the resulting function permutation-equivariant and able to process MSAs of varying sizes: Phyloformer learns how to predict the evolutionary distance between a pair of sequences given all other sequences in the MSA. The combination of these two technical innovations-the regression formulation and self-attention mechanismproduces an effective mean to jointly predict all pairwise distances across the MSA.
Acknowledgements The authors thank  Dexiong Chen ,  Martin Ruffel ,  Johanna Trost  and  Philippe Veber  for insightful discussions. This work was funded by the  Agence Nationale de la Recherche  ( ANR-20-CE45-0017 ). It was granted access to the  HPC/AI resources of IDRIS  under the allocation AD011011137R1 made by  GENCI. The taxon silhouettes in Figure 2 are modified from public domain images in the PhyloPic database.
trees varying in the number of leaves (left) or the number of sites (right). Colors as in Figure4. 6 Discussion and future workIn this manuscript we have presented Phyloformer, a new method for phylogenetic reconstruction using deep learning. We have showed on two types of simulations that it could approach or beat state-of-the-art (SOTA) maximum likelihood methods that use all the information available in the alignment (Nguyen et al. , 2015)  while being significantly faster. Our use of two simulators reduces the risk of a simulator-related bias in our conclusions. The simulations performed using the simplest model revealed two regimes: on alignments with numbers of sequences similar to the one used to train the network (20) Phyloformer lead to Robinson-Fould distances that were very close to the ones obtained by IQ-TREE while being one order of magnitude faster.
In this paper we investigate the reliability of Google's Coral Tensor Processing Units (TPUs) to both high energy atmospheric neutrons (at ChipIR) and thermal neutrons from a pulsed source (at EMMA) and from a reactor (at TENIS). We report data obtained with an overall fluence of 3. 41 × 10 12 n/cm 2 for atmospheric neutrons (equivalent to more than 30 million years of natural irradiation) and of 7. 55×10 12 n/cm 2 for thermal neutrons. We evaluate the behavior of TPUs executing elementary operations with increasing input sizes (standard convolutions or depthwise convolutions) as well as eight CNNs configurations (SSD MobileNet v2 and SSD MobileDet, trained with COCO dataset, and Inception v4 and ResNet-50, with ILSVRC2012 dataset). We found that, despite the high error rate, most neutrons-induced errors only slightly modify the convolution output and do not change the CNNs detection or classification.
ufrgs. br; Paolo Rech is also with DAUIN, Politecnico di Torino and with DII, Università di Trento; Sujit Malde, Carlo Cazzaniga, Maria Kastriotou, and Christopher Frost are with ISIS Facility, UKRI-STFC, United Kingdom; Manon Letiche is with TENIS, Institut Laue-Langevin, France. This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 886202, from The Coordenação de Aperfeiçoamento de Pessoal de Nível Superior -Brazil (CAPES) -Finance Code 001, and from the French national program Programme d'Investissements d'Avenir, IRT Nanoelec ANR-10-AIRT-05. Neutron beam time was provided by ChipIR (DOI: 10. 5286/ISIS. E. RB2000161) and ILL (TEST-3203). to execute CNNs on highly performant, costly, and power hungry devices, such as the latest GPUs or very big FPGAs.
Coupled with a good software framework (e. g. , Tensor Flow) that runs on a host device, EdgeAI devices significantly reduce the time and power consumption of the convolution, which is the most computational demanding operation of CNNs. As EdgeAI devices are likely to be used at scale and in distributed systems, it is fundamental to investigate their reliability, in particular their neutron-induced error rate. Preliminary studies showed that, despite being small, EdgeAI devices have a not negligible neutrons-or protons-induced error rate  [1] ,  [2]. In this paper, we investigate the reliability to neutrons of Google Coral TPU. Unlike previous works on EdgeAI reliability, we deeply investigate the fault model on the main elementary operations (standard and depthwise convolutions). Moreover, we compare the error rate and the prediction failures of eight CNNs configurations: SSD MobileNet v2 and SSD MobileDet, trained with COCO dataset, as well as Inception v4 and ResNet-50, trained with ILSVRC2012 dataset.
To have a broad evaluation, we test the Coral TPU with both high energy neutrons, at the ChipIR facility, and with thermal neutrons, at the EMMA facility in UK and at TENIS facility at Institut Laue-Langevin (ILL) in Grenoble, France. While the high-energy neutrons cross section of the Coral TPU is much higher than the thermal neutrons cross section, regardless of the type of neutrons, the results are consistent in the sense that depthwise convolutions are shown to have higher error rate than standard convolutions and SSD MobileDet is less reliable than SSD MobileNet V2. The rest of the paper is organized as follows. In Section II, we provide a solid background on CNNs and on the hardware and software architecture of the Coral TPU, useful to under-stand the experimentally observed behaviors.
For minimizing data transfers and storage and speed up calculations, all data that is computed and stored within the TPU is represented as 8-bit unsigned integers (UINT8). The device is capable of performing the quantization and dequantization steps for interfacing with the host floating-point representations. Since Coral TPU is simply an accelerator, it must be connected to a host device. Google provides two versions of the accelerator: one that interfaces with the host via PCIe and the other uses USB 3. 0. On our setup, we have a Raspberry Pi 4 as host, connected to the Coral USB accelerator. The software layer of the Coral TPU is based on Tensor-Flow Lite, which is a light version, optimized for embedded devices, of the TensorFlow framework developed by Google for machine learning  [10].
A. Hardware and Software Besides convolutions, we evaluate the reliability of eight neural network configurations in which we vary the network architecture, the dataset, and the training methodology. We consider neural networks that perform the two main machine learning tasks supported by Coral: image classification and object detection. In image classification, the goal is to classify the object in the image, e. g. , a dog, a car or a tree, without indicating the position. Object detection is a more complex task, as multiple objects in the image have to be located and then classified. We consider four different network architectures. Two of them, Inception V4  [16]  and ResNet-50  [17] , target image classification. Both are trained with ILSVRC  [18]  dataset and support a wide range of 1,000 different object classes.
The other two, SSDLite MobileDet  [19]  and SSD MobileNet V2  [20] , perform object detection and are trained with COCO  [21]  dataset which embraces 90 classes. The models for these NNs are based on TensorFlow Lite. In addition to these four models/configurations, we also retrain SSD MobileNet V2 with two other datasets: a subset of the COCO dataset (14 classes) and a subset of the Oxford-IIIT Pet  [22]  dataset (2 classes). Our goal is to evaluate whether and how a reduced number of objects to be detected impact the device error rate. The retraining process is done with and without the application of transfer learning technique, in which the knowledge from another machine learning model is reused in order to speed up the learning process. Considering both types and multiple sizes of convolutions, as well as the different NN configurations, we provide experimental data obtained on 16 benchmarks.
IV. EXPERIMENTAL RESULTS In this Section we present neutron experiments data obtained irradiating the TPUs with atmospheric and with low energy neutrons. We consider both Silent Data Corruptions (SDCs, i. e. , errors at the output) and Detected Unrecoverable Errors (DUEs, i. e. , crashes or hangs). We first discuss the reliability of atomic operations, i. e. standard (2D) and depthwise (3D) convolutions and then the reliability of four different neural networks that were trained with multiple datasets for a total of 8 neural networks configurations. All data is reported with 95% confidence intervals, considering a Poisson distribution.
B. Neural Networks With regards to neural networks, we test the reliability of eight different configurations by varying the network architecture, dataset and training procedure, with or without transfer learning, in which the knowledge learned by other model is reused especially to reduce the training time, but also, in most cases, the resulting NN achieves better prediction performance/accuracy). We leveraged on four NNs models that were trained and made available by Google (Inception V4, ResNet-50, SSD MobileDet, and SSD MobileNet V2). We also retrained Mo-bileNet using two different datasets (a subset of the COCO dataset and a subset of the Oxford-IIIT Pet dataset) with and without applying transfer learning techniques. By retraining the NN models, we want to evaluate: (1) how the number of object classes supported by the NN impacts its reliability, since the datasets used for retraining have much less classes than the original COCO dataset and (2) whether transfer learning has a positive effect into the detection resilience.
This behavior is in accordance with what has been observed in GPUs architectures  [11]. Transfer learning (TL) does not seem to have a significant impact on the NN cross section. This technique has shown to decrease the SDC cross section in 2-5% when compared to the analogous configuration without TL. Furthermore, the training process tend to converge much faster with this strategy and, in our case, it reduced the learning time of the NNs in around 50%. TL is a good solution when a quick re-training of the NN is needed, as it is fast but does not impact the error rate. Our results also confirm that the retraining of MobileNet with the COCO subset (14 classes) lowers the cross sections when compared to the original model trained with the total amount of 90 classes of the original COCO dataset.
The same network but trained with the Pets dataset (2 classes) have higher cross sections than the one trained with Sub-COCO, but still smaller than the one obtained for the original with the entire COCO dataset. This trend evinces that, with less classes to be considered, the detection process gets simpler and the cross section is reduced. Therefore, the training of NNs should target the real application needs and include classes of object that are really relevant to the context of the application. Finally, ResNet and Inception, which are NNs that perform image classification (not detection), have the highest DUEs cross sections. This might be related to the size of the model for these two networks which are 5 to 7 times larger than the MobileNet model. Apart from the retrained networks, which have the lowest value for DUE cross sections, the overall DUE rate is similar among the other NNs which enforces the fact that DUEs are mostly related to the hardware attributes rather than the algorithm.
Transfer learning does not seem to have a consistent impact on the criticality of the SDCs. In the case where MobileNet is retrained with the Pets dataset, the application of this technique has shown to decrease the number of critical errors by almost 3 times. On the other hand, when trained with COCO subset, it makes the NN 20% more susceptible to critical SDCs. Further studies are necessary to understand the reasons for this opposite trend. The differences, though, are not very high. Naturally, SDCs in classification NNs are considerably less critical since only a few values, the highest ones, out of 1,000 output values are indeed relevant to the outcome of the classification process. Therefore, although the SDCs are propagated to the network raw output, most of them do not influence the classification result, as confirmed by our data plot in Figure  8.
10 -9 cm 2 ] Network Configuration Neural networks Critical SDCs % -ChipIR 0. 06% Cross section [Critical SDCs % 0. 11% 4. 73% 0. 69% 0% 1% 2% 3% 4% 5% 1. 19% 0. 96% 0. 51% 1. 46% Transf Learn - - - - - - YES NO YES NO Dataset ImgNet ImgNet COCO COCO Sub-COCO Pets Neural Network Incept ResNet MobDet MobNet.
METHODS Over a 6-year collection effort, we sampled woody trees and shrubs (defined as woody plants with stems or pseudostems > 0. 5 m in height) across southern Africa (Appendix S1). We sequenced the core plant barcodes (rbcLa and matK) from sampled material, and merged these data with matching barcode sequence data in Gen-Bank from species native to the region. We then used the merged sequence matrix to reconstruct a comprehensive phylogeny of the southern African woody flora (Supplemental Methods S1). Branch lengths were calibrated in millions of years using Bayesian MCMC in BEAST v. 1. 4. 8 and 28 fossil calibrations (Table  S1 ). Voucher specimen information and GenBank accession numbers are listed in Appendix S2. For each species, we collated data on 11 separate uses: firewood and charcoal, carving, building and structural, spiritual, cultural, food, ornamental, forage and fodder, shade, chemical compounds, and medicinal, plus threat category from the IUCN Red Data List of Threatened Species (http://redlist.