dataesr/scientific-paragraphs-categorization huggingface dataset:
g. Gleckler et al. , 2008). This collection of wellestablished large-to global-scale mean climatological performance metrics provides a baseline analysis package for CMIP: the PCMDI Metrics Package (PMP;  Gleckler et al. , 2016). The Earth System Model eValuation Tool (ESMVal-Tool;  Eyring et al. , 2016)  has been developed for CMIP6 to enable routine comparisons of single or multiple models, against either predecessor versions or observations. The current priority for PMP and ESMValTool is to target selected essential climate variables (ECVs;  Gleckler et al. , 2008; Pincus et al. , 2008; Reichler and Kim, 2008) , which include rel-ative space-time root-mean square errors (RMSEs) for some cloud and radiation parameters. Among the diagnostics in the current CFMIP catalogue, the Cloud Regime Error Metric for the annual mean climatology in the present climate  (Williams and Webb, 2009)  has recently been included in ESMValTool.
All of this work contributes to the wider community efforts to improve the representation of models. The World Climate Research Program (WCRP) Grand Challenges represent areas of emphasis in climate science where there is believed to be likelihood of significant progress over the next 5-10 years. One requirement for these areas is to implement effective and measurable performance metrics, to build and strengthen collaborations between communities. The CFMIP Diagnostics Codes Catalogue will contribute to this purpose for the Clouds, Circulation and Climate Sensitivity Grand Challenge  (Bony et al. , 2015). This paper is a companion paper to the CFMIP description paper for CMIP6  (Webb et al. , 2017). In this paper we describe code repositories and introduce diagnostics which are currently available in GitHub repositories.
General description of the repositories and the catalogue The repositories are maintained and managed by the author of the associated diagnostic code. In each repository, a README file and wiki page provide an introduction to the diagnostic and list the appropriate references. Different authors use different programming languages and the codes which are currently in the repository are provided in their original languages. The code repository is structured so that a version of the code in a different language can be placed in a different directory, allowing users to select the version they prefer to use. Details of each diagnostic, such as the required input data, the outputs generated by the program and links to the source of the required observations, are also provided in the repository. For most diagnostics, the data are assumed to be in netCDF format, provided that the variables are requested by either CMIP or CFMIP.
Description of metrics, diagnostics and methodologies Here we describe the diagnostics and metrics which are currently to be found in the catalogue. For each diagnostic Table 1 summarizes its scientific target and the practical details of its application. We start with diagnostics related to all types of clouds, followed by those focussed specifically on low-level cloud. The final group consists of diagnostics targeted at understanding cloud feedbacks. In this paper the term "metric" refers to scalar quantities which can be easily compared to observations. 3. 1 Simulation of ISCCP global cloud amounts  (Klein et al. , 2013)  This code is available at https://github. com/mzelinka/ klein2013-cloud-error-metrics. This is a set of four scalar measures of model fidelity in simulating clouds, where the observational "truth" is taken to be the International Satellite Cloud Climatology Project (ISCCP) cloud fields  (Rossow and Schiffer, 1999).
The diagnostic codes are available at https://github. com/mzelinka/klein2013-cloud-error-metrics. 3. 2 Cloud Regime Error Metric (CREM;  Williams and Webb, 2009; Tsushima et al. , 2013)  This code is available at https://github. com/tsussi/ cloud-regime-error-metric. This is a set of scalar metrics which summarize the overall ability of a model to simulate the cloud radiative effects (CREs) of a set of different cloud regimes. The IS-CCP observational cloud regimes were obtained using the KMEANS clustering algorithm  (Anderberg, 1973)  and an empirical method to choose the number of clusters  (Rossow et al. , 2005). (For details, see  Williams and Tselioudis, 2007. ) Using daily mean data the cloud regime assigned to a model grid box is the observational regime which has the minimum Euclidean distance in the vector space of normalized daily mean cloud-top pressure, optical depth and cloud cover.
Zonal plots of GCM cloud and hydrometeor fraction compared with CALIPSO-GOCCP and CloudSat  (Nam and Quaas, 2012)  This code is available at https://github. com/chriscnam/ CFMIP_LidarRadar. This code produces plots of zonally averaged vertical profiles of cloud fraction and hydrometeor fraction from the active-sensor satellite observations and models. Comple- menting the ISCCP simulator referred to above, the active lidar and radar satellite simulators emulate the radiances which would be retrieved by the CALIPSO and CloudSat instruments within climate models. The active lidar and radar simulators allow a more accurate comparison of the vertical distribution of clouds and hydrometeors in climate models with the CALIPSO-GOCCP and CloudSat 2B-GeoProf data sets  (Marchand et al. , 2009). Figure  6  shows the zonally averaged cloud fraction (top row; Fig.
3. 5 A-train satellite instantaneous cloud property observations for process-oriented evaluation (CALIPSO-PARASOL;  Konsta et al. , 2015)  This code is available at https://github. com/dimitrakonsta/ process-oriented-cloud-evaluation. These 2-D histograms correlate different cloud variables from the multi-sensor A-train observations at the instantaneous timescale, and at high spatial resolution. This allows us to see how different key cloud properties vary as a function of one another  (Konsta et al. , 2012)  and to build pictures of cloud processes which are well suited for the evaluation of clouds in climate models. Specifically, the histogram shows the relationship between cloud cover from CALIPSO  (Winker et al. , 2007)  and cloud reflectance measured by PARASOL  (Parol et al. , 2004) , which is a good surrogate of the cloud optical depth.
Vertical distribution of low-level clouds This code is available at https://github. com/chriscnam/ CFMIP_LowCloudDistribution. These histograms show the frequency of occurrence of clouds below 4 km over the tropical oceans (30 • N-30 • S) in the observations and models (Fig. 8 ). This diagnostic, from  Nam et al. (2012) , identifies non-overlapped (i. e. by mid-and high-level cloud) low-level clouds within subsiding regimes. This is done by first identifying where large-scale vertical velocities at 500 and 700 hPa are greater than 10 hPa day -1 ; then distinguishing between shallow cumulus and stratocumulus regimes using the lower tropospheric stability thresh-old of 18. 55 K, as defined in  Medeiros and Stevens (2011) ; and finally testing whether lidar-defined high-and mid-level cloud covers are both less than 5 %.
Shortwave cloud radiative effect (SW CRE) and PARASOL reflectance This code is available at https://github. com/chriscnam/ CFMIP_SWCRE_Parasol. The poor vertical distributions of low-level clouds shown above lead to biases in their optical properties, which can be quantified using this diagnostic: the mean SW CRE and PARASOL reflectance broken down into different cloud fraction bins (Fig. 9 ). The comparison of shortwave cloud radiative effects from various CMIP5 models with CERES TOA fluxes as well as PARASOL reflectance above the nonoverlapped low clouds (Fig. 9 ) show that models overestimate the cloud radiative effects compared to observations, even for comparable cloud fractions and large-scale environmental conditions  (Nam et al. , 2012). 3. 7 Warm rain microphysical process diagrams  (Suzuki et al.
, 2015)  This code is available at https://github. com/kntrszk/cfodd. This diagnostic plots vertical profiles of radar reflectivity in the form of a contoured frequency diagram as a function of in-cloud optical depth (ICOD). The radar reflectivity is obtained from the CloudSat 2B-GEOPROF product (e. g. Marchand et al. , 2008)  and the cloud optical depth is obtained from the MODIS cloud product (e. g. Platnick et al. , 2003; Nakajima et al. , 2010). The diagram is constructed from the probability density function (PDF) of radar reflectivity at each ICOD bin, and shows the PDFs as the contoured frequency of radar reflectivity as a function of ICOD, which is referred to as the Contoured Frequency by Optical Depth Diagram (CFODD). The ICOD is determined by a vertical slicing of the total cloud optical thickness from MODIS into each radar bin according to the adiabatic growth assumption that provides a scaling function of the optical depth with respect to geometric height.
Sensitivity of tropical low-cloud reflection to surface temperature change at various timescales  (Brient and Schneider, 2016)  Codes for this diagnostic are available at https://github. com/ florentbrient/Cloud-variability-time-frequency and https:// github. com/florentbrient/ECS-Constraint. This diagnostic calculates covariances of time series of any cloud-related variable (e. g. low-cloud albedo α c , low-cloud fraction) and that of sea surface temperature T (robust regression slope, correlation coefficients). Brient and Schneider (2016) estimated the sensitivity of the reflectance of tropical low clouds (TLCs) to the underlying surface temperature change at intra-annual, seasonal and interannual timescales in the observations and CMIP5 models. As shown in Fig. 11 , they found that in the observations on all timescales shortwave reflection by TLC decreases robustly when the underlying surface warms.
Code is provided to calculate a set of diagnostics based upon the covariance of two time series (robust regression slope, correlation coefficients); separation of timescales is achieved by applying low-pass and highpass filters. The calculation by the code also provides the uncertainty range underlying the covariance through a stationary bootstrap procedure. An additional program is used to constrain the more likely range of ECS values by weighting models' ability to reproduce the observed covariance between TLC reflection and sea surface temperature. 3. 9 Sensitivities of low-cloud cover to estimated inversion strength and sea surface temperature  (Qu et al. , 2014)  This code is available at https://github. com/xinqu2016/ SST-and-EIS-slopes. This metric calculates the sensitivity of tropical marine low-cloud cover (LCC) to two key cloud-controlling factors, the strength of the inversion capping the atmospheric boundary layer (measured by the estimated inversion strength, EIS) and sea surface temperature (SST).
S is diagnosed from vertical gradients of humidity and temperature over warm tropical oceans at altitudes around the typical marine boundarylayer top, while D is calculated explicitly from model pressure velocity fields in the lower and middle troposphere. Both quantities were obtained from annual-mean data over tropi-cal oceans, with only 2 years giving reasonably stable results compared to the large differences in mixing rates between models, although we recommend longer time periods. Figure  14  shows the relationship of ECS with S, D and the LTMI (the sum of S and D). Observations of S and D were obtained in that study from radiosonde and reanalysis data. The ranges of D and S are similar (Fig. 13a, b ), and the LTMI explains about 50 % of the variance in total system feedback (r = 0. 70) and ECS (r = 0.
Acknowledgements. We are grateful to  Mark Ringer  and  Gill Martin  for helpful comments on the manuscript. This work was supported by the  Joint UK BEIS/Defra Met Office Hadley Centre Climate Programme  ( GA01101 ). This work was originally funded by the  European Union Seventh Framework Programme  ( FP7/2007-2013)  under grant agreement no. 244067  via the  EU Cloud Intercomparison and Process Study Evaluation Project (EUCLIPSE). Xin Qu ,  Mark D. Zelinka  and  Stephen A. Klein  are supported by the  United States Department of Energy's Regional and Global Climate Modelling Program  under the project " Identifying Robust Cloud Feedbacks in Observations and Models ". The work of  Mark D. Zelinka  and  Stephen A. Klein  was performed under the auspices of the  United States Department of Energy by Lawrence Livermore National Laboratory  under contract  DE-AC52-07NA27344.
Code availability. Each repository is linked to the CFMIP webpage and can be found on the diagnostics code page there: https:// www. earthsystemcog. org/projects/cfmip/. The following page also has links to metrics that are included in the catalogue: https: //github. com/tsussi/cfmip-diagnostics-code-repository. CMIP data are available through the PCMDI CMIP page: http://www-pcmdi. llnl. gov/projects/cmip/. CFMIP1 and CFMIP2 data can be found under CMIP3 and CMIP5, respectively. The Supplement related to this article is available online at https://doi. org/10. 5194/gmd-10-4285-2017supplement.
BTCs are determined with what we will refer to as a bulk algorithm. The core of a typical bulk algorithm includes dependencies of BTCs on the wind speed and the stability of the atmospheric surface layer (ASL) as well as the adjustment of atmospheric scalars to the standard height through the flux-profile relationships. Traditionally, bulk algorithms are developed for in situ point measurements of SSVs, with neutral drag, heat, and moisture BTCs derived from ship and buoy measurements  (Large and Pond 1981, 1982; Smith 1988; Fairall et al. 1996b Fairall et al. , 2003;; Bradley and Fairall 2007). In this sense, bulk parameterizations are most suited to applications utilizing voluntary observing ship (VOS) and buoy data only. Nevertheless, for most purposes, bulk formulas are used to compute TASFs using a variety of SSV datasets.
These datasets include SSVs measured by VOS instrumentation  (Josey et al. 1999; Gulev et al. 2007; Berry and Kent 2005, 2009) , satellite-retrieved SSVs  (Bentamy et al. 2003; Chou et al. 2004; Bourras 2006; Grodsky et al. 2009; Andersson et al. 2011; Tomita and Kubota 2011) , and SSVs derived from NWP systems, particularly those from reanalysis datasets  (Josey et al. 2013). This latter approach is used for producing global TASF climatologies and surface forcing fields for numerical experimentation with OGCMs  (Yu et al. 2008; Large and Yeager 2009; Brodeau et al. 2010). Discrepancies between state-of-the-art bulk algorithms have been repeatedly shown to lead to a relatively large spread in the computed TASFs  (Blanc 1985 (Blanc , 1986 (Blanc , 1987;; Zeng et al. 1998; Eymard et al. 1999; Brunke et al.
b. Experimental setup We use bulk formulas with 3-hourly atmospheric SSVs and daily mean SST from ERA-Interim  (Dee et al. 2011)  to compute global 3-hourly estimates of the TASFs from 1982 to 2014. Our experiments are initiated in 1982 as the SST data product used as the surface boundary condition in ERA-Interim becomes truly daily from the end of 1981. Prior to this, daily mean SSTs were derived through interpolation of monthly mean data. In the control experiment CtrlE (Table  1 ), TASFs are computed at the standard height of z 5 10 m, according to Eqs. (1) and the procedure outlined in Fig. 1. BTCs are computed using the COARE 3. 0 algorithm, which has consistently performed well in algorithm intercomparisons (e. g. ,  Brunke et al. 2003). Sensitivity experiments using the NCAR and ECMWF bulk algorithms are also performed to assess the algorithm-related TASFs uncertainties (section 4).
61. We use instantaneous values of C P and L y , calculated according to Eqs. (B4) and (B5) (appendix B, sections c and d). TASFs are set to zero over ice-covered regions. For U z , we use the absolute 10-m wind speed vector. The effect of using U z relative to the sea surface current is tested in experiment CurrE (section 3e). The 1982-2014 climatologies of wind stress and total turbulent heat flux Q T (i. e. , Q L 1 Q H ) derived in CtrlE are shown in Fig. 2. All the FORTRAN routines used to compute TASFs in this study are developed and maintained by the authors as part of the AeroBulk 1  package and are currently being implemented as the default forcing function of the upcoming major release of the NEMO 2  OGCM (version 4. 0). The approximations and assumptions studied in sections 3 and 4 are chosen to reflect many of the common parameterization issues faced by ocean and atmosphere practitioners when configuring bulk formulas for use in modeling experiments.
e. Wind vector referenced to the surface current In many applications of bulk formulas, TASFs are determined from the apparent wind speed relative to the sea surface current vector u 0 , so that in Eq. (  1 ), U z is replaced by U z 2 u 0. In sensitivity experiment CurrE (Table  1 ), we use the apparent 10-m wind vector of ERA-Interim relative to the prescribed daily u 0 from the Global Ocean Reanalysis and Simulation (GLORYS), 4  mesoscale, eddypermitting ( 1 /48 resolution) ocean reanalysis (Ferry et al. 2012;  Balmaseda et al. 2015). Note that CurrE is limited to the 1993-2013 period, which corresponds with the longest overlap in the GLORYS and ERA-Interim datasets. The bulk formulas imply that the relative error in E, Q L , and Q H is of the order of u 0 /U, while it is at least twice as large for the wind stress on account of the quadratic dependence with wind speed.
Acknowledgments. This study has been made possible by support of the  Bolin Centre for Climate Research. The computations were performed on resources provided by the  Swedish National Infrastructure for Computing (SNIC)  at the  National Supercomputer Centre at Linköping University (NSC ). All figures were created with Matplotlib (Hunter 2007). Bernard Barnier  is supported by  Centre National de la Recherche Scientifique (CNRS). Sergey K. Gulev  benefited from Grant  14-17-00697  of the  Russian Science Foundation. APPENDIX A.
The amount of data from languages spoken all over the world is rapidly increasing. Traditional manual methods in historical linguistics need to face the challenges brought by this influx of data. Automatic approaches to word comparison could provide invaluable help to pre-analyze data which can be later enhanced by experts. In this way, computational approaches can take care of the repetitive and schematic tasks leaving experts to concentrate on answering interesting questions. Here we test the potential of automatic methods to detect etymologically related words (cognates) in cross-linguistic data. Using a newly compiled database of expert cognate judgments across five different language families, we compare how well different automatic approaches distinguish related from unrelated words.
Introduction Historical linguistics is currently facing a dramatic increase in digitally available datasets  [1] [2] [3] [4] [5]. The availability of data for more and more languages and language families challenges the ways in which we traditionally compare them. The comparative method has been the core method for linguistic reconstruction for the past 200 years  [6] , and is based on manually identifying systematic phonetic correspondences between many words in pairs of languages. However, there are too few expert historical linguists to analyse the world's more than 7500 languages  [7]  and, consequently, only a small percentage of these languages have been thoroughly investigated leaving us in the dark about their history and relationships. This becomes especially evident in largely understudied linguistic areas like New Guinea, parts of South America, or the Himalayan region, and our lack of knowledge about these languages has immediate implications for our understanding of human prehistory.
Materials and Methods Materials There are few datasets available for testing the potential of cognate detection methods on language data, As such, testing algorithms run the risk of over-fitting. When developing an algorithm, one usually trains it on some datasets. If those datasets are afterwards used to also test the algorithm, the accuracy should be quite high, but we cannot tell whether the method will work on datasets apart from the ones on which the algorithm was trained. For this reason, it is important to split the available data into a training set and a test set. In our case, the training set will be used to determine the best parameters for each of the algorithms we test, while the test set will be used to carry out the actual test of cognate recovery. For this study, we took training data from existing sources  [19] , while a new test dataset was compiled from scratch.
The new test set consists of six datasets from five language families. These data were collected from different sources, including published datasets  [3, [20] [21] [22] [23] , books  [24] , and ongoing research by scholars who allowed us to use parts of their data in advance (Uralex project,  [25] ). All datasets were formatted to tabular format and semi-automatically cleaned for various kinds of errors, like misspelled phonetic transcriptions, empty word slots, or obviously erroneous cognate judgments. We further linked all languages to Glottolog  [7] , and all wordlist concepts to the Concepticon  [26]. Table  1  lists all datasets along with additional details, such as the number of words, concepts, languages, and cognate sets in the data. The diversity index given in the last column of the table is calculated by dividing the difference between cognate sets and meanings with the difference between words and meanings  [19].
This score, which ranges between 0 and 1, indicates whether large numbers of words in a given dataset are unrelated (high index) or are cognate (low index). As can be seen from the diversity indices listed in the table, our test sets have varying degrees of diversity, ranging from 0. 07 (Romance, Saenko, 2015) to 0. 57 (Uralic). As mentioned above, training data is needed for parameter estimation. The key parameter we need to estimate is the best thresholds for cognate identification in some of the methods. As training data we employed the collection of benchmark datasets for automatic cognate detection by List  [19] , which also covers six datasets from five language families. Details for this dataset (number of words, concepts, languages, cognate sets, and the diversity index) are given in Table  2.
Methods Automatic Cognate Detection. Many methods for automatic cognate detection have been proposed in the past (see Table  3  below). Unfortunately, only a few of these methods qualify as candidate methods for computer-assisted language comparison, since the majority are either (a) not able to analyse multiple languages at once, (b) have further requirements making their use more complicated  [31, 32]  e. g. require a user-specified reference phylogeny (and therefore assume that language groupings are already known), or need extensive training sets, or (c) are not freely available (see Table  3 ). We decided to take four publicly available methods as the basis of our test study, the Turchin Method, the Edit Distance Method, the SCA Method, and the LexStat Method. Additionally, we tested a modified version of the LexStat method which we call Infomap.
In this modified version of LexStat we introduced an improved partitioning method based on the Infomap algorithm for community detection  [33]. All methods are presented in more detail below. The four publicly available methods are all implemented as part of the same software package (LingPy, http://lingpy. org, [42] ), and represent different degrees of algorithmic sophistication and closeness to linguistic theory, with the Turchin Method being very simple and computationally extremely fast, and the LexStat Method being rather complex and time-consuming. For the usage of the fifth method, we wrote a small LingPy plugin which builds on the python-igraph package (http://igraph. org/python-igraph/, [43] , see details below) and is provided along with our supplementary material. Cognate Detection following Turchin et al  [44].