dataesr/scientific-paragraphs-categorization huggingface dataset:
$ pip install taurex_ggchem At which point, the plugin is automatically downloaded, compiled if needed (or prebuilt binaries used) and installed. Now GGChem becomes available and can be utilized in the input file with its own set of input keywords and retrieval parameters: On initialization, TauREx 3 searches for modules in the python environment for these plugins and adds any new keywords to be parsed in the input file. TauREx 3 does not assume the inclusion of its inbuilt classes and parameters and will perform the same search of parameters and models on itself. The plugins function like any other Python library can be installed from PyPi and git through pip install which significantly improves their ability to be distributed to other users. This ability to be installed also allows for the compilation stages of FORTRAN and C++ codes to be completely automated and transparent to the user, significantly improving their usability.
Importantly, the original author has full control of development, able to host and develop their own plugins independently of TauREx 3. The plugins for our source control forks of the FastChem  (Stock et al. 2018)  and GGchem equilibrium chemistries introduce minor or no changes to the original code. Instead, the installation stage references and compiles against the chemistry author's original FORTRAN/C++ code. Once the TauREx Python wrappers and plugin setup files have been written, an author can continue development in their language of choice; Leaving their original makefiles and overall compilation pipeline intact and allowing the plugin to benefit from their newest improvements. Development of these wrappers is also accessible. A developer can only focus on converting TauREx inputs and units to their desired input, performing the calculation, and converting them into the TauREx expected outputs and units.
As TauREx standardizes each atmospheric component, there is a guarantee that the implementation will fully function in the retrieval pipeline. Plugins also solve another problem of distribution of codes. Some of these codes may have restrictive licenses that prevent them from being fully open-source. These plugins may instead be installed directly from private repositories or distributed as compiled python code (i. e. pyc instead of. py) and binaries with licenses. A list of the available plugins are given in Table  7. 7. List of currently available plugins of TauREx 3. 1. Availability describes where a user may acquire the plugin. PyPI availability means that the plugin can be acquired using pip install. On publication refers to plugins that will be available after their relevant publication.
C. CHEMISTRY PLUGINS The plugin system has now expanded the scope of chemistry modelling to include three equilibrium chemistries: ACE  (Agúndez et al. 2012 (Agúndez et al. , 2020)) , FastChem  (Stock et al. 2018 ) and GGChem  (Woitke, P. et al. 2018). All plugins can be installed through PyPi on Windows, Linux and Mac and all are capable of being used in optimizations to retrieve elemental abundances. ACE and GGchem are all originally written in FORTRAN. To facilitate their inclusion, the numpy f2py module was used to automatically generate efficient and convenient python wrappers as well as handle their compilation during installation. FastChem is a C++ code so light wrappers were written using cython. For all of these codes, a dedicated TauREx 3 chemistry interface was written and exposed to the plugin system.
When installed through PyPI, the python packaging system handles all of the wrapping and compilation in the background. For Windows and MacOS, PyPI will bypass compilation and instead download prebuilt binaries. All chemistries present can be controlled and fit based on metallicity relative to their initial abundance (usually solar) and ratios of each metal element relative to oxygen. For the ACE plugin, only C/O and N/O ratios can be fit. The Fastchem and GGchem plugins will dynamically generate oxygen ratio fitting parameters (Al-Refaie et al. 2021) for each metal element selected by the user. For example, if the user selects C, N, S and Ti then C/O, N/O, S/O and Ti/O ratios can be fit during retrievals.
C. 1. GGchem The plugin compiles against the original GGchem FORTRAN source code during installation as well as introducing additional FORTRAN-90 glue code that directly interfaces with TauREx 3. The GGchem plugin allows for the user selection of elements. Hydrogen, helium and oxygen must be present. If the user does not specify these then they are automatically included. The initial abundance of these elements can be chosen from one of four default profiles: solar, earth, ocean or meteorite. Modification of abundances after initialization (and during retrievals) is controlled by the metallicity parameter which determines the amount of oxygen relative to the initial profile and an O_ratio parameter that determines the ratio for each metal element relative to oxygen. The oxygen ratio fitting parameters are dynamically generated  (Al-Refaie et al.
2021). The plugin also allows for equilibrium condensation, enabled at initialization by setting equilibrium_condensation=True and ions enabled by setting include_charge=True. As ion chemistry defines electron abundance, it can be seamlessly used with the H-opacity component (Section A. 2). The taurex_ggchem plugin, to the authors knowledge, presents the only true python wrapper to GGchem. All functionality of GGchem is present and keywords in the TauREx 3. 1 input file match (or closely match) a GGchem input file. The plugin can be used outside of TauREx 3 to generate chemistry profiles through a simple python interface: >>> from taurex_ggchem import GGChem >>> gg = GGChem(metallicity=1. 0, selected_elements=['H','He','C','O','N','K'], abundance_profile='earthcrust', equilibrium_condensation=True) >>> nlayers = 10 >>> temperature = np.
During a get/set/call request, the parent process will monitor the child process, if at some point the child process has died, it is assumed that a STOP command was called. The parent process will perform cleanup of hanging threads and raise a FortranStopException. On the next get/set/call request, a new process is spun up. An additional benefit to this class is that all writes to standard output in the FORTRAN code are also redirected back to TauREx 3 and logged. This is useful as FORTRAN outputs can now be hidden during retrieval. The GGchem plugin utilizes this class extensively. Multiple instances can now be created and retrievals freely run. Anytime a convergence issue is encountered, the instance is destroyed, GGchem reinitialized and an exception raised. The retrieval is informed of this and will avoid regions in the parameter space that cause it.
C. 3. FastChem The FastChem plugin, like the GGChem plugin, allows for the calculation of ion chemistry, can be used outside of TauREx 3 to generate chemistry profiles using Python: >>> from taurex_fastchem import FastChem >>> fc = FastChem(selected_elements=['H','He','C','O','N','K','e-'], with_ions=True, metallicity=1. 0) >>> nlayers = 10 >>> temperature = np. linspace(300,100,nlayers) >>> pressure = np. logspace(5,-3, nlayers) # Pa >>> fc. initialize_chemistry(nlayers,temperature,pressure) >>> fc. gases ['H', 'He', 'O', 'C', 'K', 'N', 'e-',. , 'O+', 'O-', 'O2+', 'O2-'] >>> fc. mixProfile array  ([[3. 87435866e-036, 9. 95149979e-039, 7. 62616463e-042, 1. 23490910e-045, 2. 58839801e-050, 3. 41640407e-056, 9. 40930967e-064, 9. 08433703e-074, 1. 41255491e-087, 1. 38065040e-167],. , [1. 42400626e-001, 1.
42400626e-001, 1. 42400626e-001, 1. 42400626e-001, 1. 42400626e-001, 1. 42400791e-001, 1. 42398731e-001, 1. 42398284e-001, 1. 42367067e-001, 9. 96186945e-001]]) Under the hood, the FastChem plugin generates parameter and custom elemental abundance files in a temporary directory that is then passed into the FastChem library. Doing this allows us to vary the elemental abundances through the oxygen ratio and metallicity parameters. Like GGChem, the retrieval parameters include metallicity which controls the amount of oxygen relative to the initial abundance and an O_ratio parameter that determines the ratio for each selected metal element relative to oxygen. The initial abundance can be modified by passing in FastChem abundance files through the elements_abundance_file keyword. Once more, enabling ions, FastChem computes electron abundances which can be used be the H-opacity code (Section A.
Figure10. Posteriors of ACE (blue), FastChem (red), GGChem (green), GGChem with condensation (orange) and free chemistry (purple). Each equilibrium chemical code utilizes the heavy element list (except ACE and free) and is fit against a JWST simulated transit spectra of HD 209458b using parameters given by Table1and abundances computed using FastChem (Stock et al. 2018 ) with heavy species. Priors used are given by Table3.
Figure15. Retrieved spectra for ACE (blue), FastChem (red), GGChem (green), GGChem with condensation (orange) and free chemistry (purple). Each equilibrium chemical code utilizes the heavy element list (except ACE and free) and is retrieved against simulated JWST spectra (black) computed with GGChem with heavy elements and condensation using parameters from Table1. Spectra have been binned to a lower resolution and offset for clarity.
ACKNOWLEDGEMENTS This project has received funding from the  European Research Council (ERC)  under the European Union's  Horizon 2020 research and innovation programme  (grant agreement No  758892 ,  ExoAI ) and the  European Union 's  Horizon 2020 COMPET programme  (grant agreement No  776403 , ExoplANETS A). Furthermore, we acknowledge funding by the  UK Space Agency and Science and Technology Funding Council (STFC)  grants:  ST/K502406/1 ,  ST/P000282/1 ,  ST/P002153/1 ,  ST/S002634/1 ,  ST/T001836/1  and  ST/V003380/1. The authors would like to acknowledge  P. Woitke ,  C. Helling ,  D. Kitzmann  and  J. Stock  for providing their codes under permissive open-source. AFA would also like to thank  Quratulain Jahangir  for her support. OV acknowledges the  CNRS/INSU Programme National de Planétologie (PNP)  and  CNES  for funding support.
Retrieved spectra for ACE (blue), FastChem (red), GGChem (green), GGChem with condensation (orange) and free chemistry (purple). Each equilibrium chemical code utilizes the heavy element list (except ACE and free) and is retrieved against simulated JWST spectra (black) computed with ACE using parameters from Table1. Spectra have been binned to a lower resolution and offset for clarity. ACE (C/H/O/N) FastChem GGchem GGChem-cond Free C/H/O/N input -Heavy models 0. 0175 0. 0170 0. 0165 (R p /R s ) 0. 0160 0. 0155 0. 0150 0. 0145 1 2 Wavelength ( m) 3 4 5 Figure 8.
In this work we introduce a new highly scalable deep learning architecture, Sequence UNET, that classifies and predicts variant frequency directly from protein sequence. This model learns to build representations of protein sequence features at a range of scales using a fully convolutional U-shaped compression/expansion architecture. We show that it can generalise to pathogenicity prediction, achieving comparable performance on ClinVar to methods including EVE and ESM-1b at greatly reduced computational cost. We further demonstrate its scalability by analysing the consequences of 8. 3 billion variants in 904,134 proteins detected in a large-scale proteomics analysis, showing a link between conservation and protein abundance. Sequence UNET can be run on modest hardware through an easy to use Python package.
It was consistently found among the top VEPs across a diverse set of DMS results  12, 13. It has been refined in the Bayesian Variational Autoencoder (VAE) EVE model  14  , which outperformed previous methods on human DMS data and ClinVar variants. However, these models still require MSA based training for each protein of interest, making them very computationally demanding. Protein language models, for example UniRep 15 , AminoBert  16  or ESM-1b  17  , also use unsupervised learning to capture position specific representations, which relate to a range of properties, including conservation, structural stability and secondary structure. They are trained to predict the identity of masked amino acids across many different sequences, meaning they learn general protein sequence properties. These produce sequence representations in a single forward pass but are still computationally intensive and often require top models to be trained for downstream applications, which is time consuming due to the large model size.
An intermediate approach between capturing variation in a single protein family, as EVE does, and a general protein language model is to predict per position variant frequencies for any sequence, using labelled MSA training data. This frequency defines the position's position specific scoring matrix (PSSM), summarising the cross-species diversity and conservation of the sequence. This approach balances capturing additional information contained in specific MSAs with general applicability, being able to predict rapidly from any input sequence. The link with conservation means such predictions could be used to predict deleteriousness directly or the model can be further fine-tuned using smaller scale labelled pathogenicity data. We apply this approach, presenting a fast, scalable deep learning predictor, Sequence UNET, and a corresponding python package.
It uses a fully convolutional architecture to predict protein PSSMs from sequence with optional structural input. The model is trained to directly predict variant frequency or to classify low frequency variants, as a proxy for deleteriousness, and then fine-tuned for pathogenicity prediction. It outperforms previous de-novo PSSM predictors, such as SPBuild, accurately classifies low frequency variants and achieves high VEP performance but with greater scalability. Further, our model has comparable performance on these tasks to models based on the much larger ESM-1b protein language model. These language models are much slower and require significantly more compute power as well as additional top model training time. We demonstrate the benefits of performance and scalability by rapidly calculating all possible variants for 904,134 proteins detected in a pan-genome proteomics analysis  18  , something that would be impossible or prohibitively time consuming with previous VEPs.
Since protein structure contains information that is extremely difficult to extract from sequence alone the network supports an optional graph convolutional neural network 20 (GraphCNN) module to summarise positional structural features (Fig. 1D ), which are then concatenated with the sequence input. We provide a more detailed description of the model and its inputs and outputs in the Supplementary Information. The model outputs a matrix of per position features and can therefore be trained to predict various positional properties. We demonstrate two related VEP use cases: predicting rare variants, as a proxy for deleteriousness; and directly predicting every possible variant's frequency. We trained using ProteinNet 21 , which is a large collection of protein sequence and structure information, containing data from 104,059 structures from the PDB alongside matching variant frequencies from large MSAs based on 332,283,871 sequences.
PSSM prediction and frequency classification We trained two base Sequence UNET models, optimising performance for PSSM prediction using a softmax output layer and Kullbeck-Leibler divergence loss and variant frequency classification using a sigmoid output and binary cross entropy. Hyperparameters were tuned in both modes with the same results (selected parameters in Fig. 1A , Fig. S1 ). The PSSMs predicted by the model closely resemble true results (Fig. S2A-B ) and the frequency classifier significantly separates rare and common variants (Fig. S3A-B ). We find our models PSSM results correlate more strongly to true values in the ProteinNet test set than SPBuild 24 (a state of the art de novo LSTM PSSM predictor), the amino acid propensities predicted by ESM-1b and the results from a top model using ESM-1b representations trained on ProteinNet CASP12 95% thinned data (Fig.
2A , Fig. S2C-D ). Interestingly ESM-1b logits correlate much better with raw frequencies than normalised PSSMs, potentially because they are trained to identify the most likely amino acid at a position, not differentiate between the lower frequencies that are important for PSSMs. Including structural features slightly increases performance ( = 0. 472 vs  = 0. 451). Similar results are found for frequency classification ( < 0. 01) over the ProteinNet CASP12 test set, with Sequence UNET achieving top performance equalling a top model using ESM-1b representations (Fig. 2B ). We only compared to one VEP (SIFT4G) as deleteriousness is related to but not equivalent to frequency classification, instead comparing more widely on other datasets. Different frequency thresholds lead to different classification performance (Fig.
Generalising Sequence UNET Having shown good PSSM prediction and classification performance we next sought to show Sequence UNET generalises to predicting deleterious variants and compare performance to other tools. We tested generalisation on three datasets: labelled human protein variants from ClinVar, standardised deep mutational scanning (DMS) data  25  , and a set of gold standard S. cerevisiae variant classifications  26. The model can also be fine-tuned to new tasks with additional training on external data, either refining the existing weights (fine-tuning) or replacing the final classification layer with a freshly initialised one (a top model). We trained ClinVar classification top models and fine-tuned models with and without structural features on top of the Sequence UNET frequency classification model, using a random 95%/0.
5%/4. 5% training/validation/testing split across all pathogenic and neutral variants in the ClinVar dataset that occur in proteins in ProteinNet CASP12 training data. The weights of all but the top two model layers were frozen to prevent overfitting. We also trained simple single layer CNN models to predict ClinVar pathogenicity and frequency classification, to provide a lower bound for machine learning solutions to this problem. The fine-tuned models specifically predict pathogenicity probabilities for each variant at all positions (Fig. 3A ). These predictions tend to be more similar for variants at a position, including the wild-type amino acid than the results of the frequency classification and PSSM models. This is partially because pathogenicity is related to the position's properties and importance but also suggests there might not be sufficient training data available to differentiate between different variants at one position beyond the average properties of that position.
Top models and fine-tuned models achieve comparable performance profiles to many state of the art predictors (Fig. 3B ). Performance is significantly better when only proteins with structural information are considered, suggesting this bias from the ProteinNet training data has impacted what the model learnt. A training set that was not restricted to proteins from the PDB would likely help rectify this bias. The top model performs slightly better than the finetuning approach on variants with structure but worse on those without it, independently of whether the network utilised that information. This is perhaps because the freshly initialised final weights allow the network to learn new relationships specific to structured pathogenicity whereas fine-tuning maintains more general relationships.
The fine-tuned UNET model only performs slightly worse than the much larger ESM-1v language model 27 across all variants and performs better on variants with structural data, despite being a much smaller and more manageable network. Interestingly, actually utilising structural data slightly reduces the performance of the fine-tuned models, suggesting it is less related to pathogenicity. The base Sequence UNET frequency classification models generalise less well to this task, although still with comparable performance to models such as FoldX. This suggests the frequencybased model does not fully capture deleteriousness without fine-tuning. The frequency classification model was initially trained with a range of frequency thresholds and the resulting models have different ClinVar generalisation performance (AUC0.
01 = 0. 73, AUC0. 1 = 0. 67, AUC0. 0001 = 0. 66, AUC0. 001 = 0. 65). This shows that the chosen threshold does impact performance on a given task and suggests 0. 01 is the best threshold for pathogenicity prediction, which aligns with the fact that the mean allele thousand genomes frequency for benign variants is 0. 112 and for pathogenic variants is 0. 008  28. Sequence UNET predictions would also be expected to relate to DMS results and confirmed neutral and deleterious S. cerevisiae variants. Comparing the distribution of Spearman's rank correlation values across DMS datasets (Fig. 3C ) shows the model generalises, although it performs slightly less well than top predictors on this task. A similar result is found with a ROC analysis of S. cerevisiae variants (Fig. 3D ), where Sequence UNET variations outperform FoldX but fall behind SIFT4G and ESM-1v.
The model fine-tuned on ClinVar performs best in both cases, while the top model performs relatively poorly, even falling behind simple CNN models in some cases. This further suggests that the top model may be learning something more specific to ClinVar, which may be an artefact of the dataset but could also be a real biological feature of pathogenicity or human proteins. Together this confirms that the models generalise well to other contexts, although their relative performance compared to other tools varies.
3 billion possible variants in this proteomics dataset, which took 1. 5 hours on a GPU using a batch size of 100 (6. 8 hours without batching) and 50. 9 hours using only CPU (Fig. 4B ). The additional padding required to batch different length proteins was found to have a negligible impact on predictions for an analysis of this scale, although it does impact a small number of individual results (Fig. S4 ). This also compares favourably with the ESM language model, even when using a single forward pass for all variants instead of independently masked passes for each variant as suggested  27. ESM is both a much larger model and the attention mechanism scales quadratically with protein length, whereas the convolutional design of Sequence UNET scales linearly. This means Sequence UNET is significantly faster using CPU and for larger proteins on GPU.
Small proteins are predicted at a similar rate on GPU, suggesting at this point other factors dominate. Sequence UNET also requires much less (V)RAM in all cases, making it significantly easier to deploy at scale and allowing batches of proteins to be processed simultaneously to increase efficiency. In contrast, even a batch size of 2 was prohibitive for ESM. These performance increases, combined with the Sequence UNET python package, makes large scale analyses more accessible, especially for those without high performance compute facilities. We used this large dataset of variant effect predictions for almost 1M proteins to compare protein abundance and predicted tolerance for sequence variation. Proteins that are expressed at higher abundances are generally expected to have more strongly constrained sequences than low abundance proteins  29.