dataesr/scientific-paragraphs-categorization huggingface dataset:
This effect is more important for large MSAs, especially for the PSICOV dataset for which performances increase from an average AUC200 of 138 for short sequences, to an average AUC200 of 186 for long sequences. One possible explanation is that large proteins have larger cores; as co-variations seem to be concentrated in those cores (see  [4] ), it may explain the improvement observed as sequence length increases. Note however that these are averaged behaviors over specific groups of MSA lengths. When we plot the AUC200 directly versus the sequence length L, we find linear correlations between those two variables of 0. 4, 0. 49, 0. 43, and 0. 39 for the PSICOV MSAs in the small, medium, and large groups, and for all MSAs, respectively, and of 0. 22, 0. 54, 0. 44, and 0. 26 for the NOUMENON MSAs in the small, medium, and large groups, and for all MSAs, respectively.
Amino Acid Representation 2: One Dimensional Property-Based Vectors The 20D representation of amino acids described above only accounts for their alphabetic symbols and does not consider their physico-chemical properties. An alternate representation is to consider directly one such property. With this representation in mind, we have considered AAindex, a database of numerical indices representing various physicochemical and biochemical properties of amino acids  [40, 41]. The section AAindex1 of this database contains 566 different options for representing each amino acid as one numerical value, with each option corresponding to one property of amino acids. Out of those 566 indices, the 402 first originally assembled by Tomii and Kanehisa  [45]  have been clustered into six groups using single-linkage hierarchical cluster analysis.
Those six groups, which we will refer to as clusters A, B, C, H, P, and O, mostly map with structural or physical properties of the amino acids: cluster A relates to alpha helix and turn propensities (118 indices), cluster B to beta sheet propensities (37 indices), cluster C to amino acid composition (24 indices), cluster H to hydrophobicity (149 indices), cluster P to physicochemical properties of the amino acids (46 members), and cluster O to other properties, such as the frequency of left-handed helices (28 members). The whole AAindex1 database, including the partitioning of the indices into these 6 clusters, is available at https://www. genome. jp/aaindex/. We have analyzed the performance of GaussCovar with such a 1D representation of the amino acids over the two datasets PSICOV and NOUMENON, testing each of the 402 properties of AAindex1 for which cluster information is available separately.
Each of those tests use a 1D encoding and is referred to as Gauss1D. Results are shown in Figure  4. The predictive power of GaussCovar was tested using one of 402 amino acid properties available in the database AAindex  [40, 41]. Results are presented as averages over 6 groups of AAIndex1 scales, as originally defined in  [45]. The 6 clusters A, B, C, H, P, and O correspond to indices related to alpha helix and turn propensities, beta sheet propensities, amino acid composition, hydrophobicity , physicochemical properties, and other properties, respectively. For each amino acid index, we explored the whole range of λ values between 0 and 1, with a step size of 0. 1. The corresponding 402 curves of AUC200 as a function of λ were then regrouped based on the cluster id of the index, and summarized for each cluster by simple averaging.
The conclusions from those experiments are three folds. First, the performance of GaussCovar with such a 1D representation is poor, with AUC200 around 40 for the PSICOV dataset, and around 35 for the NOUMENON dataset (those numbers should be compared to AUC200 values around 140 and around 100 for GaussCovar with the 20D amino acid representation, averaged over the PSICOV and NOUMENON datasets, respectively). GaussCovar with a 1D encoding still performs best on PSICOV, as observed for the 20D encoding. Clearly, a 1D representation decreases the differences between the amino acids, thereby reducing the performance of the method. Second, Gauss1D seems relatively insensitive to the choice of the weight of the prior λ when tested over the PSICOV dataset, while showing a clear trend of decrease in performance when the weight of the prior is increased for the tests on the NOUMENON datasets.
They have been derived from reference BLOCKs sequence alignments, using different cutoffs in the sequence identity within a BLOCK. For example, the BLOSUM62 matrix is derived from alignments with sequences that are at most 62% identical; this matrix is considered to lead to good performance for database search  [46]. Substitution matrices describe each amino acid with a set of twenty numerical values, henceforth defining a twenty-dimensional space. Swanson was the first to embed this space into a plane, using a principal component analysis (PCA) approach. Following others (see  [37]  and references therein), we expand this concept and consider projections of BLOSUM substitution matrices on different spaces with varying dimensions. In the space of dimension k for example, amino acids are assigned k "coordinates" along the k principal components of the BLOSUM matrix considered.
In Figure  5 , we show the corresponding vectors in two and three dimensions for BLOSUM30 and BLOSUM62. For both matrices, the 3D representations of hydrophobic (in blue) and hydrophilic (in red) amino acids are well separated. There is however more overlap in the projections of the BLOSUM30 matrix between hydrophobic and aromatic amino acids (in magenta), and between hydrophilic amino acids and the two small amino acids A and G (in green). We have analyzed the performance of GaussCovar using the BLOSUM62-based encoding of amino acids over the two datasets PSICOV and NOUMENON. We refer to this version of GaussCovar as BLO62. Results are shown in Figures  6  and 7. For each projection of BLOSUM62 onto spaces whose dimensions vary from 1 to 20, we have explored the whole range of λ values between 0.
1 and 0. 9, with a step size of 0. 1. The corresponding λ best are shown in Figure  6a ,b for the PSICOV and NOUMENON datasets, respectively. Not too surprisingly, the optimal λ value increases as the number of components k increases: large values of k lead to large covariance matrices for which the effects of under sampling are more important. Interestingly, while the performance of GaussCovar increases as the number of principal components k of the BLOSUM matrix considered increases between 2 and 11, the best performance is obtained for k = 11, and not for larger k values, as observed on Figure  6b ,d for the PSICOV and NOUMENON datasets, respectively. We note that larger values of k may have required larger values of λ (we stopped at 0. 9). We did not explore this option, as it is unclear whether larger contributions of the prior in the Gaussian model are meaningful.
To highlight this fact, we show the known polar amino acids (Q, R, E, K, H, N, D, T, P, and S) in red, the hydrophobic amino acids (M, V, L, I and C) in blue, and the aromatic amino acids (Y, F, and W) in magenta. Note that the two small amino acids, A and G (in green), stand out. In Figure  7 , we compare the PPV plots for GaussCovar based on BLOSUM62 vector representations of size k = 3 (green), k = 7 (blue), and k = 11 (red), to the reference of using the binarized 20D representation of amino acids (black), over the two datasets PSICOV and NOUMENON. As expected from Figure  6 , the BLOSUM-derived PPV plots get closer to the reference as the number of components increases, but never show improvements compared to this reference. As already observed for the Gauss20 and Gauss1D versions of GaussCovar, BLO62 performs better on PSICOV than on NOUMENON, highlighting again the possibilities of biases in the former.
7 )) that leads to optimal performance is plotted against the number of components for the vector representation of amino acids derived from the BLOSUM62 matrix, based on the PSICOV dataset (a), and based on the NOUMENON dataset (b). (c,d) The optimal AUC200 (over all predicted contacts) is given against the same number of components for the PSICOV dataset (c) and the NOUMENON dataset (d). The BLOSUM62 matrix, intermediate between a more permissive matrix such as BLOSUM30 (computed over BLOCKS with a low sequence identity of 30%), and a restrictive matrix such as BLOSUM90 (from BLOCKS with up to 90% sequence identity) has become a standard in protein database searches and sequence alignments (see for example references  [46, 48]  in which the performances of multiple substitution matrices were assessed).
It was therefore natural to consider it for testing GaussCovar; it is unclear however if it would lead to the best results compared to other substitution matrices. We have repeated the analyses described above for BLOSUM matrices with IDs between 30 and 100. Note that BLOSUM100 is not the identity matrix: It is computed from BLOCKS with sequence similarities up to 100%. It is used however to detect sequences with very high similarities. The corresponding best performances of GaussCovar, as measured by AUC200 computed over all contacts from the mean PPV curves over all 150 protein families for the PSICOV and NOUMENON datasets, are plotted against the BLOSUM ID in Figure  8. Overall, it is observed that performance increases as the BLOSUM ID increases, for both datasets. Their results are consistent with the fact that the 20D binarized encoding performs best, as this encoding can be seen as derived from the identity matrix, i.
Discussion In this work we have tested multiple numerical encodings of amino acids within the framework of contact predictions in proteins from a multivariate Gaussian analysis of multiple sequence alignments. We have shown that 1D encodings based on amino acid properties (from the AAindex databases), as well as higher dimension encodings computed from the principal components of BLOSUM substitution matrices, do not perform as well as a simple 20-dimensional binarized encoding in which each amino acid in represented as a vector of zeros, except at the position corresponding to its type, where the encoding is 1. We have also highlighted the importance of the prior when constructing the Gaussian model of a multiple sequence alignment, and have shown that as the dimension of the vectors representing the amino acids increases, more importance should be given to this prior in the model, even if it is based on a simple uniform distribution.
Vector Representations of Amino Acids In addition to the common 20-dimensional binarized representation of amino acids (in which an amino acid of type i is represented with a vector of 20 zeros, except at position i where the vector component is 1), we have used two different vector representations of amino acids in this work, one based on their physical properties, and one based on substitution matrices. In our first representation, each amino acid is encoded by one real index value, as defined in AAindex  [40, 41]. AAindex is a database of numerical indices representing various physicochemical and biochemical properties of amino acids and pairs of amino acids. We have used the former, and have tested the 402 properties that were available in 1996  [45]. Those are the first indices available in AAindex and accessible online at https://www.
As described above, we have considered encoding an amino acid as a single real value, as defined in one of the scales available in the AAindex database  [40, 41] , projected version of BLOSUM matrices onto spaces of varying dimensions, as well as the original binarized representation used by Baldassi et al. In step 3, the weighted empirical mean and covariance over the MSA are computed as: X = ∑ N n=1 w n X n ∑ N n=1 w n (9) and C = 1 ∑ N n=1 w n N ∑ n=1 w n (X n -X) T (X n -X) (10) The score P(i, j) between two positions i and j is computed from the Frobenius norm of the submatrix J(i, j) of side s × s, where s is the size of the vector representing amino acids. (step 6 of the algorithm). The Frobenius norm is not independent of the gauge choice (see  [25]  for a full discussion of this point).
A perfect prediction would generate a curve such that PPV(k) = 1 for all k, while a poor prediction would see PP(k) close to zero for all k. We quantify the overall quality of the prediction as the area below this curve, which we write AUC200, as it is computed over the first 200 predicted contacts. We note that the larger the AUC200 score, the better the prediction. The AUC200 score for a perfect prediction is 200. To assess the significance of the performances of GaussCovar with different amino acid encodings over a dataset, we generated a mean curve PPV, averaged over all families included in the MSA. At each contact level k, we computed a mean value and a standard deviation. To compare two such curves (say when comparing the performance of GaussCovar under certain conditions with the performance of the program PSICOV), we applied a two tailed Student's t-test at each level of contact k.
Test Datasets We have used two test sets for assessing the predictive abilities of GaussCovar under different representations of amino acids, namely the original PSICOV dataset  [26]  and the recent NOUMENON dataset  [42]. The PSICOV set is a carefully designed dataset composed of 150 protein families, originally generated as a test set for the namesake program PSICOV (Protein Sparse Inverse COVariance). It was designed as follows (for full details, see  [26] ). One hundred and fifty proteins whose structures are available in the Protein Data Bank (PDB, htt://www. pdb. org) [2]  and lengths between 75 residues and 275 residues were selected. All the structures of those proteins were of high resolution (≤ 1. 9 Å) and all are known to be biological monomers, to eliminate the risks of detecting inter-chain contacts.
The sequences of those proteins are dissimilar. For any pair of sequences, a sequence alignment using the SSEARCH package from the FASTA package (https://fasta. bioch. virginia. edu/) has a E-value greater than 10 -5 , and usually greater than 0. 1. For each sequence, multiple sequence alignments were generated using the jackhammer program from the HMMER 3. 0 package (http:hmmer. org). The sequence was checked against, and aligned with, homologues from the UNIREF100 data bank, using three iterations of jackhammer. The resulting alignments contain between 511 and 74836 sequences. Those alignments, as well as the corresponding reference PDB structures, and the results of running PSICOV on them, are available at http://bioinfadmin. cs. ucl. ac. uk/downloads/PSICOV/ as Supplementary Materials of the original PSICOV paper  [26].
The NOUMENON set is a more recent dataset that was designed to provide a bias-free test set for assessing the performance of residue contact prediction methods  [42]. It also contains 150 proteins and was derived in a similar manner that PSICOV was built. Namely, 150 sequences of proteins with known highly resolved structures (≤2 Å) were selected. The lengths of these proteins also vary between 75 residues and 275 residues. It was also ensured that their structures have at least N contacts, where N is the length of the protein. Each of those sequences was then checked against, and aligned with, homologues from the UNIREF100 data bank, using three iterations of jackhammer. The key difference with the PSICOV dataset is how the initial 150 proteins are selected. This selection is constrained such that their homologues have no, or little evolutionary relationship with proteins whose structures are available in the PDB.
Figure1. Precision Positive Values (PPV) plotted against the number of predicted contacts for the PSICOV dataset (left) and the NOUMENON dataset (right). PPV(k) is the rate of correctly predicted contacts within the first k contacts. Data for the Gaussian models were generated with GaussCovar, with a 20-dimensional binarized vector representing amino acids, with λ = 0. 8 (black curve), or λ = 0. 2 (red curve). Data for the PSICOV predictions (blue curve) on the PSICOV dataset were obtained from the data provided by the authors [26] , while the similar data on the NOUMENON dataset were generated by running directly PSICOV (see text for details). All PPV curves are arithmetic means over 150 protein families. The shaded areas represent standard deviations.
Figure2. AUC200 plotted against λ, the weight of the prior in the Gaussian model for the PSICOV dataset (left) and the NOUMENON dataset (right). AUC200, with values between 0 and 200, is a measure of performance, with higher values indicating better performance. The 150 multiple sequence alignment (MSAs) in the datasets are broken down into 3 groups, small (M e f f < 500, medium (500 ≤ M e f f < 1000), and large (1000 ≤ M e f f ).
LMDz represents the atmosphere in the Earth system model of the Institute Pierre Simon Laplace (IPSL-CM,  Dufresne et al. , 2013)  and as such has been contributing to the recent versions of the Climate Model Intercomparison Project (CMIP) established by the World Climate Research Program (https://cmip. llnl. gov/, last access: 30 October 2018). Of direct relevance for this study here is the fact that the offline version of LMDz  (Hourdin et al. , 2006b)  as well as an associated adjoint code are used in the atmospheric inversion system of  Chevallier et al. (2005) , which is the current basis for the CO 2 inversion products of the Copernicus Atmosphere Monitoring Service of the European Commission (http://atmosphere. copernicus. eu/, last access: 30 October 2018). At present, the offline model requires 44 times less CPU time than the corresponding full LMDz version.
, 2015a). For CMIP6, configuration 5B of LMDz has further evolved from  Hourdin et al. (2013) : it has a different formu-lation of the triggering assumptions and a different radiative transfer code, and it accounts for the thermodynamical effect of ice. The convective triggering is now based on evolving statistic properties on the thermal plumes by considering a thermal size distribution instead of a bulk thermal  (Rochetin et al. , 2013). The motivation behind this change was to depart from the QE hypothesis and to allow a more gradual transition between shallow and deep convection through a three-step process (appearance of clouds, crossing of the inhibition layer and deep convection triggering). In the shortwave, the code is an extension to 6 bands of the initial 2-band code that is used in LMDz5A  (Fouquart and Bonnel, 1980) , as implemented in a previous version of the European Centre for Medium-Range Weather Forecasts (ECMWF) numerical weather prediction model.
Description of the simulations We have run the two versions of the physics described above, 5A and 6A, at several resolutions for the years from 1988 to 2014. A summary of the simulations used is given on Table  1. The identification number of the LMDz code used here (that contains both physics versions) is 2791. We discard the first 10 years  (1988) (1989) (1990) (1991) (1992) (1993) (1994) (1995) (1996) (1997) (1998)  to allow enough spin-up for the tracer simulations, considering the interhemispheric exchange time of about 1 year for passive tracers  (Law et al. , 2003). The dynamics is nudged towards the 6-hourly horizontal winds from the ECMWF reanalysis  (Dee et al. , 2011)  with a relaxation time of 3 h  (Hourdin and Issartel, 2000). CO 2 , SF 6 and 222 Rn initial values are set uniformly for all model grid boxes at a value of, respectively, 350 µmol mol -1 (abbreviated as ppm), 1.
Point samples from surface sites The surface measurements of SF 6 are taken at background stations SPO, PSA, CGO, SMO, RPB, MLO, MHD, BRW, SUM and ALT from the NOAA/ESRL network (http://www. esrl. noaa. gov/gmd, NOAA ESRL GMD, 2015). The simulated mole fractions of CO 2 were compared with some of the atmospheric surface measurements that were assimilated when optimizing the surface CO 2 fluxes prescribed here. The location of these assimilated surface stations is shown in Fig. 1. As in the CAMS inverse modelling framework, we only retain early afternoon data (12:00-15:00 LST) for continuous stations under 1000 m a. s. l. and night-time data (00:00-3:00 LST) for continuous station above 1000 m a. s. l. All measurements from flasks below 1000 m a. s. l. have been kept. The reasons behind this hour selection are the failure of transport models in general to accurately represent the accumulation of tracers near the surface at night and the advection of air masses during the day by upslope winds over sunlit slopes in the afternoon  (Geels et al.
2 ppm  (Machida et al. , 2008). We selected from the CONTRAIL dataset all the CO 2 vertical profiles during the ascending and descending flights for the period 2006-2011 over the regions portrayed in Fig. 1. The regions are similar to  Niwa et al. (2011)  and have been chosen according to the number and location of the vertical profile samples. The number of hourly mean measurements at 5. 5 km a. s. l. per model grid box are shown in Fig. 2   The NOAA/ESRL Global Greenhouse Gas Reference Network Aircraft Program consists here of measurements of air samples collected every few days or months at 22 aircraft profiling sites over continental North America (shown in blue in Fig. 1 ) between altitudes of 300 and 8000 m a. s. l. In the lowest altitudes, compared to the CONTRAIL measurements that have been sampled at nearby commercial airports, these measurements are not affected by local emissions.
In this case, the land surface scheme only controls the heat and latent fluxes at the landatmosphere interface. The SF 6 and 222 Rn emissions were prescribed following the TransCom 3 protocol. The CO 2 surface fluxes have been optimized beforehand by the assimilation of surface observations in a version of LMDz close to the older model version studied here. We have compared the resulting ensemble of simulations with both assimilated and unassimilated CO 2 observations from a large dataset in different parts of the globe. This study enabled us to benchmark the effects of the resolution, land surface scheme and subgrid-scale parameterizations on CO 2 -simulated values, which is a fundamental step before implementing the recent developments in our inverse modelling system. At the surface, the comparison with the assimilated CO 2 measurements showed that the land-surface scheme and the vertical resolution have a limited impact compared to the horizontal resolution and subgrid-scale parameterizations.
, 2018). From that perspective, we quantified the impact of the model set-ups on the simulated xCO 2 convolved with the OCO-2 space-time coverage for a given year. The model ensemble spread is mainly due to the physics and exceeds 0. 5 ppm in the boreal summer high latitudes, in northern Africa and in Brazil, or locally around emission hotspots. In boreal summer, the new physics decreases the latitudinal gradient by decreasing the xCO 2 values in the high latitudes further due to less efficient vertical mixing. This may decrease the northern sink inferred by inverse modelling with the satellite data and LMDz. In austral summer, the mean xCO 2 shows large discrepancies (up to 1 ppm) over the Amazonian basin between the simulations, this region being particularly sensitive to the parameterization assumptions.
g. ,  Randall et al. , 2003). Code and data availability. The LMDz model is available from http://web. lmd. jussieu. fr/trac (last access: 30 October 2018) under the CeCILL v2 Free Software License. Supplement. The supplement related to this article is available online at: https://doi. org/10. 5194/gmd-11-4489-2018-supplement. Author contributions. MR designed, ran and analysed the simulations and prepared the paper under direction and advice of FC. AC aided in the development and preparation of the simulations. XL also aided in the preparation of simulations. XL and PB reviewed the paper. Competing interests. The authors declare that they have no conflict of interest.