dataesr/scientific-paragraphs-categorization huggingface dataset:
These include the ESMFold method for protein structure prediction  (Lin et al. 2022) , which replaces MSAs with protein language models  (Alley et al. 2019 , Chowdhury et al. 2022 , Wu et al. 2022) , and FastFold, a community effort that has implemented significant improvements including fast model-parallel training and inference  (Cheng et al. 2022 ). We 5 PDB accession code 7DQ9_A  (Wei et al. 2021)  22 expect future work to go further by disassembling OpenFold to attack problems beyond protein structure prediction. For instance, the evoformer module is a general purpose primitive for reasoning over evolutionarily related sequences. DNA and RNA sequences also exhibit a co-evolutionary signal, with efforts aimed at predicting RNA structure from MSAs fast materializing (e. g. ,  Singh et al.
B OpenProteinSet details OpenProteinSet consists of nearly 5 million unique MSAs, making it the largest publicly available MSA database. For ~400,000 of those entries, constituting the OpenFold training set, we provide additional MSAs-computed using Mgnify  (Mitchell et al. 2020) , Uni-Clust30  (Mirdita, Driesch, et al. 2017) , BFD  (Jumper et al. 2021) , and UniRef90  (Suzek et al. 2013 )-and template hits retrieved from PDB70  (Steinegger et al. 2019 ). These were generated with multiple sequence databases and alignment tools, as in the AlphaFold2 paper. JackHMMer  (Johnson et al. 2010 ) was used to search MGnify and UniRef90; HHblits-v3 was used to search BFD and Uniclust30. Templates were computed using HHSearch  (Remmert et al. 2012 ) run on the UniRef90 MSA and then realigned, if necessary, using Kalign  (Lassmann et al.
2009). As in the original AlphaFold2 procedure, we changed some of the default options for the MSA generation tools. For JackHMMer, we used -N 1 -E 0. 0001 --incE 0. 0001 --F1 0. 0005 --F2 0. 00005 --F3 0. 0000005 For HHBlits, we used -n 3 -e 0. 001 -realign_max 100000 -min_prefilter_hits 1000 -maxfilt 100000 -maxseq 1000000 To generate MSAs for the ~270,000 protein chain self-distillation set, we performed an all-against-all search on UniClust30 using HHblits-v3 with the same parameter settings as before. This yielded approximately 15 million MSAs. Using the first sequence in each cluster as a representative, we iteratively removed MSAs whose representative chains appeared in the greatest number of other MSAs until each representative chain appeared only in its own MSA. We then removed clusters whose representative sequences were longer than 1,024 residues or shorter than 200.
Finally, we removed clusters whose corresponding MSAs contained fewer than 200 sequences, leaving just 270,262 MSAs in total. Template hits were again computed using HHsearch against PDB70. To speed up the expensive training process, we generated structures for the self-distillation set using OpenFold run with AlphaFold2 weights rather than a pretrained version of OpenFold. The remainder of OpenProteinSet consists of ~4. 85 million Uniclust30 MSAs with depth >50 that were filtered from the core distillation set by this process.
We also replicated all of the stochastic training-time dataset augmentation, filtering, and resampling procedures described in the original paper. During the initial fine-tuning and subsequent predicted-TM fine-tuning phases, we manually sampled checkpoints at peaks in the validation lDDT-Cα  (Mariani et al. 2013 ). These checkpoints were added to the pool of model checkpoints used in the final model ensemble. Training was run on a cluster of 44 NVIDIA A100 GPUs, each with 40GB of DRAM. The model was trained in a data-parallel fashion, with one protein per GPU. In order to simulate as closely as possible the batch size of 128 used in training AlphaFold2, we performed threeway gradient accumulation to raise our effective batch size from 44 to 132. Supplementary Figure  1  contains additional data from the training run.
E Inference details Runtime benchmarks were performed on a single 40GB A100 GPU. Times correspond to the intensive 'model_1_ptm' config preset, which uses deep MSAs and the maximum number of templates. For proteins shorter than 1,000 residues, we take advantage of OpenFold's TorchScript tracing capability and FlashAttention. For longer proteins, both tools become unstable and so we disable them. We plan to address this shortcoming in the future. For runtime benchmarks we upgraded to PyTorch v1. 12 for its improvements to Torch-Script. AlphaFold was run with JAX v. 0. 3. 13. For reference, we include the distribution of lengths of our 132,000 PDB chains in Supplementary Figure  2.
F. 2 Inference We also introduce several inference-time optimizations to OpenFold. As previously mentioned, these features trade off memory usage for runtime, contributing to more versatile inference on chains of diverse lengths. FlashAttention: We incorporate FlashAttention  (Dao et al. 2022) , an efficient fused attention implementation that tiles computation in order to reduce data movement between different levels of GPU memory, greatly improving peak memory usage and runtime in the process. We find it to be particularly effective for short sequences with 1,000 residues or less, on which it contributes to an OpenFold speedup of up to 15% despite only being compatible with a small number of the attention modules in the network. Low-memory attention: Separately, OpenFold makes use of a recent attention algorithm that uses a novel chunking technique to perform the entire operation in constant space  (Rabe and Staats 2021).
In Al-phaFold2's case, the chunk size used in this procedure is a model-wide hyperparameter that is manually tuned. OpenFold, on the other hand, dynamically adjusts chunk size values for each module independently, taking into account the model's configuration and the current memory limitations of the system. Although the profiling runs introduced by this process incur a small computational overhead, the modules do not need to be recompiled for each run, unlike their AlphaFold2 equivalents, and said profiling runs are only necessary the first time the model is run; once computed, the optimal chunk sizes are cached and reused until conditions change. We find this to be a robust way to seamlessly improve runtimes in a variety of settings. Tensor offloading: Optionally, OpenFold can aggressively offload intermediate tensors to CPU memory, temporarily freeing additional GPU memory for memory-intensive computations at the cost of a considerable slowdown.
This feature is useful during inference on extremely long sequences that would otherwise not be computable. TorchScript tracing: Specially written PyTorch programs can be converted to Torch-Script, a JIT-compiled variant of PyTorch. We use this feature during inference to speed up parts of the Evoformer module. Although TorchScript tracing and compilation do introduce some overhead at the beginning of model inference, and lock the model to a particular sequence length similar to JAX compilation, we find that using TorchScript achieves overall speedups of up to 15%, especially on sequences shorter than 1,000 residues. This feature is particularly useful during batch inference, where sequences are grouped by length to avoid repeated re-compilations and take maximal advantage of faster inference times.
We left spiders moving along silk threads to see whether spiders dropping from a silk would either use the silk to rappel or balloon. We removed silk from the platform with a brush after each spiderling, to avoid that the presence of another spider's silk affected the frequency of aerial dispersal (see De  Meester & Bonte, 2010). After these experiments, we released the spiders back into their site of origin. We live-recorded the spiderlings' behaviours with software BORIS  (Friard & Gamba, 2016). We measured the number of occurrences and duration of ballooning, as a proxy for long-distance dispersal; rappelling, as a proxy for short-distance dispersal; tiptoeing and climbing, which we later excluded from the analysis of dispersal (see Table  S1  in Appendix 3).
Experiments were carried out using fresh water at a temperature of 15 C. We weighted each spider after the experiment to control for a possible effect of the weight on propensity for waterborne dispersal. After these experiments, we released the spiders back into their site of origin. Spider behaviours were recorded live with a camera, and later analysed with software BORIS  (Friard & Gamba, 2016). The different behaviours on the water surface were sailing (with legs,  abdomen or by raising the body), considered as a proxy of long-distance dispersal; rowing, walking and running as a proxy of short-distance dispersal; death mimicry, immobility and anchoring were quantified but not considered in the analysis (see Table  S1  in Appendix 3).
To help with interpretation, we used the probability of direction (pd), which is the probability that an effect goes in a particular direction, and the percentage of the distribution within the region of practical equivalence (ROPE)  (Makowski, Ben-Shachar, Chen, et al. , 2019; Makowski, Ben-Shachar, & Lüdecke, 2019). The thresholds beyond which the effect was considered as 'significant' (i. e. non negligible) were pd > 95% and ROPE < 2. 5%. All analyses were conducted with packages 'rstanarm'  (Goodrich et al. , 2020) , 'modelbased'  (Makowski et al. , 2020)  and 'bayestestR'  (Makowski, Ben-Shachar, & Lüdecke, 2019)  in R (R Core Team, 2020). The scripts used to generate the analyses are publicly available (see Data Accessibility Statement).
ACKNOWLEDGEMENTS The authors thank  Länsstyrelsen (Sweden)  for allowing us to sample Dolomedes and all the landowners who gave access to their properties. The authors thank  Lars Jonsson  for giving us occurrences sites of both species in Sweden and for the help in sampling some spiders. The authors thank  Gunnar Gunnarsson  and  Johan Elmberg  for giving us access to laboratory facilities at the  University of Kristianstad (Sweden). The authors thank  Karl Filipsson ,  Niclas Carlsson  and  Stina Eriksson  for giving us access to the laboratory and aquariums at the  University of Karlstad (Sweden).
The CT perfusion (CTP) is a medical exam for measuring the passage of a bolus of contrast solution through the brain on a pixelby-pixel basis. The objective is to draw "perfusion maps" (namely cerebral blood volume, cerebral blood flow and time to peak) very rapidly for ischemic lesions, and to be able to distinguish between core and penumbra regions. A precise and quick diagnosis, in a context of ischemic stroke, can determine the fate of the brain tissues and guide the intervention and treatment in emergency conditions. In this work we present UniToBrain dataset, the very first open-source dataset for CTP. It comprises a cohort of more than a hundred of patients, and it is accompanied by patients metadata and ground truth maps obtained with state-of-the-art algorithms. We also propose a novel neural networks-based algorithm, using the European library ECVL and EDDL for the image processing and developing deep learning models respectively.
DeepHealth Related Works The European Union is spending significant amount of resources properly leading the AI development and research to maintain its proper position in this emergent and challenging context. EU funds projects to deploy pilots and largescale experimental research solutions by combining the latest discovered technologies in artificial intelligence. The same effort is spent to support AI technologies like distributed high-performance computing, cloud computing and big data support. DeepHealth  [6, 13]  is an example of this effort, that is thought (and developed) as a health-focused project. CTP data gathered for UniToBrain dataset are part of a larger collection of open-access datasets  [3, 14]  developed for the DeepHealth project. The European framework of DeepHealth, called Deep-Health Toolkit, is a single entity that aims to use heterogeneous architectures, such as high-performance computing, big data and cloud computing, to provide deep learning capabilities and computer vision off-the-shelf services.
EDDL EDDL is a deep learning library that was originally developed to be generalpurpose and it meets the AI development needs inside the DeepHealth context, therefore with the aim to be used in healthcare applications. EDDL is a free and open source software library, available on online repository as a core part of the DeepHealth Toolkit. It integrates and supports the most popular and well known Artificial Neural Network (ANN) topologies, by including both convolutional and recurrent models. EDDL aims to simplify the development of hardware-specific accelerated deep learning mechanisms by providing hardware-independent tensor operations and ANN topologies components like activation functions, regularization functions, optimization methods, and multiple types of different layers. The neural network library contains both high-level tools, such as training and evaluation of a model, and low-level tools that allow finer control of each step in the training or inference loop, like individual epochs, batches, or gradients manipulation.
ECVL The European Computer Vision Library (ECVL) is the DeepHealth computer vision library, developed to be general-purpose, with a special focus on supporting healthcare applications. It provides high-level computer vision capabilities, by providing also specialized and hardware accelerated implementations of commonly used image processing algorithms in deep learning. ECVL's design revolves around the pivotal concept of Image as the centerpiece of the entire library. Special features are included in the library for medical image data manipulation. These images are often retrieved with proprietary or multi-scan formats, such as DICOM, NIfTI and others virtual slide formats. Image provides the arithmetic instruments in order to apply mathematical operations between images and scalars. More common affine image transformations, such as rotating, scaling, mirroring, and color space changes are available.
The UniToBrain Dataset The University of Turin (UniTo) released the open-access dataset UniToBrain collected within DeepHealth project  [6]. UniToBrain  [9]  is a dataset of Computed Tomography (CT) perfusion images (CTP). The dataset includes 258 patients from multiple health institutions. Perfusion data were obtained from hospital Picture archiving and communication system (PACS) of Città della Salute e della Scienza di Torino by Neuroradiology Division in Molinette Hospital, by doing a retrospective research. All the data collected within this study are retrieved with procedures complying with institutional ethical standards, the 1964 Declaration of Helsinki and its subsequent amendments, or comparable ethical standards. In addition, the requirement for written informed consent was waived because this was a retrospective study.
DeepHealth Toolkit integration The DeepHealth Toolkit is a versatile deep learning system with fully integrated image processing and computer vision capabilities that uses HPC and cloud in-frastructure to perform parallel and decentralized AI inference and learning processes. The toolkit is designed to handle large and growing datasets, by matching the nature of medical image datasets. To that end, the DeepHealth Toolkit solution aims to transparently integrate the latest parallel programming technologies to leverage the parallel performance capabilities of HPC and cloud infrastructures, by including symmetric multiprocessors (SMPs), graphics processing units (GPUs) and various other computing acceleration technologies. The toolkit also includes features that can be used for training and inference by abstracting the complexity of different computational resources and facilitating architecture developments adopted in the learning and inference phases.
The following scripts for data management are then provided along with the dataset: dicomtonpy. py: It converts the DICOM files in the dataset to NumPy arrays. These are 3D arrays, where CT slices at the same height are piled-up over the temporal acquisition. Here pyecvl is used in order to convert and resize the ground-truth images. dataloader_pyeddl. py: Dataloader for the pyeddl deep learning framework. It converts the numpy arrays in normalized tensors, which can be provided as input to standard deep learning models using the European library EDDL. The reader can look at https://github. com/EIDOSlab/UC3-UNITOBrain to have a full companion code where a U-Net model is trained over the dataset. dataloader_pytorch. py: Dataloader for the PyTorch deep learning framework. The behaviour is the same of dataloader_pyeddl.
Methodology Employing Deep Learning approaches to the problem of deconvolving timeintensity curves, offers several potential advantages over canonical algorithms. Indeed, DL allows the extraction of information and features that are relatively insensitive to noise, misalignment and variance. Gava et al. [10]  shows whether a properly trained Convolutional Neural Network (CNN), based on a U-Net-like structure, on a pre-processed dataset of CTP images, can generate clinically relevant parametric maps of CBV, CBF and time to peak TTP. Since this specific model has been originally thought for medical images segmentation  [15] , and in this case the goal is to generate parametric images, some changes to the standard model are introduced. There is evidence that this CNN model is effective in other tasks, other than image segmentation, as well  [8].
Data Preprocessing The raw pre-processing inputs are a collection of multiple CT scans and multiple targets maps for each patient, in DICOM format. DICOM is an acronym for Digital Imaging and Communications in Medicine and it is the standard for the communication and management of medical imaging information and related metadata. Each target map corresponds to the output of the perfusion scan at a certain scan height of the patient's brain. In order to synthesize a representative map of the patient's blood flow over time, CT scans are performed at different time points. Thus, given a target perfusion, we have to search between the patient CT scans for all the images acquired at the same height. All the CT scans for the same patient are ordered following their acquisition time point. Therefore CT images can be viewed as 3D tensors, with the third dimension being the time axis.
Conclusion The growing potential of machine learning-based perfusion analysis methods have the potentialities to lead to new improved acquisition protocols and reduced radiation doses. In this work we introduce UniToBrain, an open-source dataset for CTP, which allows us to evaluate a state-of-the-art DL approach for generate brain blood flow maps. We have observed that CNN models can accurately evaluate the perfusion parameters by using CT scans images only, without any other input information, while the standard decomposition-based methods for CT and MRI require AIF curves measured in the saphenous vein  [5]. This suggests that CNNs are capable of combining the information associated with both tissue and arterial signals to estimate CBV, CBF, and TTP maps. In this work we propose as well a baseline CNN model, derived from U-Net, implemented and open-sourced using the new DeepHealth Toolkit.
MATERIALS AND METHODS A more detailed description is available in Supplemental Material and Methods (SDC, http://links. lww. com/TP/ C397). As part of the clinical research project "Metabolomics in Assessing the Quality of Kidney Transplants Retained on a LifePort Perfusion Machine," 38 kidneys taken from ECDs were included. Organs were stored on HMP LifePort Kidney Transporter 1. 0 (Organ Recovery Systems) with KPS-1 (Organ Recovery System) used as a preservation solution. Approval was obtained from the Tours University Hospital (IRB ID/CPP: 16-10-40), and this clinical trial was registered with the access number NCT03024229. All patients/participants gave consent, and the appropriate institutional forms were archived. HMP parameters (temperature, flow, resistance) were recorded during kidney conservation, and a perfusate sample was collected at the end of perfusion (storage at -20 °C after centrifugation: 3000g, 10 min).
Preimplantation biopsy (targeting the renal cortex) was performed and split into 2, with one fragment embedded in paraffin after fixation for anatomopathological evaluation and the other stored at -20 °C. Serum creatinine and clinical events were recorded up to 7 d and 3 mo, respectively. IGF was characterized by a serum creatinine ≤250 μmol/L on day 7 without the need for dialysis and DGF by the necessity of dialysis within the first 7 d. Mass spectrometry chromatographic analysis was performed using an LCMS-8060 tandem mass spectrometer (Shimadzu) and the package "Method Package for Cell Culture Profiling Ver. 2" (Shimadzu). Infusion of pure substances was performed to add some molecules to the package. Perfusates were analyzed in duplicate, and native KPS-1 was injected to determine its basal composition.
With PLS-DA, a variable influence on projection (VIP) value was estimated for each variable, and features with VIP >0. 8 were considered important to the model. The MetaboAnalyst 5. 0 computational platform (www. metaboanalyst. ca/faces/home. xhtml) was used for all the analyses except for PLS, which was performed using the MixOmics package (version 1. 6. 3) in R (version 4. 0. 2).
Metabolomic Signatures and Expression of Tubular Transporters Do Not Predict IGF The present study also aimed to identify noninvasive biomarkers predictive of graft outcomes that could be measured in the preimplantation period. Reliable biomarkers would be instrumental to optimize patient management and graft outcome. Assessment of graft quality in the transplantation period is currently based on perfusion parameters, such as the restrictive index or the flow rate, and pathological evaluation of a preimplantation biopsy; however, these indicators are not sufficiently reliable to safely discard a graft. Accordingly, in our study, no indicators predicted posttransplant graft function (data not shown). Rapid determination of the perfusate metabolomic profiles in the preimplantation period would be convenient and adapted to clinical routine.
A key challenge in Artificial Intelligence (AI) has been the potential trade-off between the accuracy and comprehensibility of machine learning models, as that also relates to their safe and trusted adoption. While there has been a lot of talk about this trade-off, there is no systematic study that assesses to what extent it exists, how often it occurs, and for what types of datasets. Based on the analysis of 90 benchmark classification datasets, we find that this trade-off exists for most (69%) of the datasets, but that somewhat surprisingly for the majority of cases it is rather small while for only a few it is very large. Comprehensibility can be enhanced by adding yet another algorithmic step, that of surrogate modelling using so-called 'explainable' models. Such models can improve the accuracy-comprehensibility trade-off, especially in cases where the black box was initially better.
But why is everyone using these kinds of models? It is often claimed that they have a higher performance than more simple models, but is this always true? How often is it the case and to what extent? This trade-off between accuracy and comprehensibility is argubaly one of the important debates in Artificial Intelligence (AI) 1   [4, 5]. This trade-off can either limit the performance of AI, if accuracy is lost due to comprehensibility restrictions (for example imposed by regulators)  [6, 7] , or hurt AI adoption, if user trust is lost due to opaqueness  [8]. The Apple Card example shows that companies may use black box models to achieve higher predictive performance, but with the risk of being unable to explain their AI decisions to users or regulators. However, while there has been a lot of research mentioning this trade-off, with most claiming there is one  [5, [8] [9] [10]  and others contradicting this  [11, 12] , there is no systematic study that assesses to what extent there indeed exists a trade-off and for what types of datasets.