dataesr/scientific-paragraphs-categorization huggingface dataset:
The 24-HDR was collected from a representative sample (calibration cohort n = 36,994) from the entire EPIC cohort  22  and from which 36,972 had valid information. Individual dietary intakes of AAs were estimated by matching the dietary assessment data of the EPIC cohort with the U. S. nutrient database (National Nutrient Database for Standard Reference of the U. S. Department of Agriculture -USDA)  23  following standardized procedures  24. In brief, the U. S. nutrient database food composition data were first matched with the food list derived from the 24-HDR, which was afterwards used as a basis for the matching with the foods reported in the DQ. Specific foods and recipes that were not included in the U. S. nutrient database were decomposed in ingredients which were available in the U.
g. glycine from hydroxyproline) but is a key factor in the homeostatic control of AA concentrations through filtration, reabsorption, synthesis, degradation and urinary excretion of free AAs and peptides  37. Finally, muscles contribute largely to the homeostatic control of AA plasma concentrations since depending on its metabolism status, muscles are using free AAs from the plasma to synthesize their own proteins and liberating AAs when catabolizing proteins. To the best of our knowledge, this is the largest study to date comparing dietary intakes of AAs with circulating AA levels using two different methods to calculate AA intake (DQ and 24-HDR) and the first study to examine a dose-response dimension in humans. Moreover, great care was taken in matching the U. S. nutrients database with the EPIC food list including many different quality controls, such as independent matching by the EPIC Oxford team.
Matching European foods with the U. S. nutrient database may also have some limitations, although the U. S. nutrient database includes many foods from different regions in the world. Nevertheless, our study results demonstrated very good comparison between protein and energy intakes calculated via the U. S. nutrient database and the protein and energy intake values obtained in the "reference" EPIC Nutrient DataBase (ENDB) that was compiled by combining different country-specific food composition databases following standardised procedures (data not shown)  38. An important disadvantage of our analysis is that only one blood measurement, as well as only one measurement for the 24-HDR, were taken at baseline which may not be representative for long-term status. Another limitation is that around 60% of our study sample did not have fasting blood samples (with fasting ≥6 h since last meal).
Financial support: The coordination of EPIC is financially supported by International Agency for Research on Cancer (IARC) and also by the Department of Epidemiology and Biostatistics, Table  1. Demographic characteristics of the control subjects in five nested case-control cohorts within the EPIC study with available measures of blood concentrations of amino acids and dietary intake data measured by dietary questionnaires (DQ) (N=3768) or 24-h dietary recalls (24-HDR) (N=334).
We discuss two parallelization schemes for MagIC, an open-source, high-performance, pseudo-spectral code for the numerical solution of the magnetohydrodynamics equations in a rotating spherical shell. MagIC calculates the non-linear terms on a numerical grid in spherical coordinates, while the time step updates are performed on radial grid points with a spherical harmonic representation of the lateral directions. Several transforms are required to switch between the different representations. The established hybrid parallelization of MagIC uses message-passing interface (MPI) distribution in radius and relies on existing fast spherical transforms using OpenMP. Our new twodimensional MPI decomposition implementation also distributes the latitudes or the azimuthal wavenumbers across the available MPI tasks and compute cores.
We discuss several non-trivial algorithmic optimizations and the different data distribution layouts employed by our scheme. In particular, the two-dimensional distribution data layout yields a code that strongly scales well beyond the limit of the current one-dimensional distribution. We also show that the two-dimensional distribution implementation, although not yet fully optimized, can already be faster than the existing finely optimized hybrid parallelization when using many thousands of CPU cores. Our analysis indicates that the twodimensional distribution variant can be further optimized to also surpass the performance of the one-dimensional distribution for a few thousand cores.
While the Boussinesq approximation seems appropriate for modelling the liquid cores of terrestrial planets, the anelastic approximation is more relevant for gas planets or stars for which the density contrasts are very large. MagIC (2021) , the open-source code extensively discussed here, allows for the choice of either Boussinesq or anelastic approximation. The initial implementation by  Glatzmaier (1984) , originally geared towards modelling the solar convective zone, split into two separate codes in the 1990s. One code version was used to model the first numerical geodynamo  (Glatzmaier and Roberts, 1995)  and was later adopted and modified by  Olson et al. (1999)  to a Boussinesq code named  MAG (2021). This modified code formed the basis for an earlier version of MagIC  (Wicht, 2002).
Stel-Published by Copernicus Publications on behalf of the European Geosciences Union. lar dynamo models were carried out using a more efficient version of the original  Glatzmaier (1984)  code named ASH (anelastic spherical harmonic; see  Clune et al. , 1999) , which more recently led to the open-source code Rayleigh (2021)  (Featherstone, 2018). MagIC still mostly follows the original algorithm laid down by  Glatzmaier (1984). It is a pseudo-spectral code written in modern Fortran. Pseudo-spectral algorithms use a spectral representation to compute accurate derivatives and a grid representation for calculating non-linear terms. In MagIC, the angular representation switches between a longitude-latitude grid and a spherical harmonic representation. Chebyshev polynomials are used for the spectral representation in radius, but MagIC also offers to employ finite differences instead.
There are two main reasons. First, the large number of physical compute cores requires a large enough workload that can be fairly partitioned and distributed across these processing units. This typically puts an upper bound on the number of cores that can usefully be employed and often results in poor strong scaling. A second complication arises from the different layers of memory access and of communication between the physical cores (i. e. non-uniform memory access -NUMA -domains, sockets, nodes). Ideally, one would attempt to keep all data "local" for quick access. However, optimizing a code for properly distributing the workload while keeping a reasonable data locality can be difficult and often requires compromising one aspect in favour of another. Until recently, MagIC only offered a one-dimensional distribution of the data implemented using MPI+OpenMP (hereafter referred to as "1d-hybrid" implementation).
For calculating the non-linear terms in grid space, this code uses MPI to distribute the spherical shells between the available NUMA domains and the inherent OpenMP scheme of the open-source spherical harmonics transform library SHTns (2021)  (Schaeffer, 2013)  for computations within a NUMA domain. Since MagIC is not optimized for multithreading between NUMA domains, this configures a limitation on the maximum number of the radial grid points. This limitation could be quite severe. High-performance computing (HPC) clusters typically offer two NUMA domains per compute node, but the current trend in hardware development might lead into a further subdivision into multiple physical units (e. g. AMD EPYC and Intel CascadeLake-AP) or logical NUMA domains (e. g. sub-NUMA clustering in Intel Xeon).
Effectively, the computing power per NUMA domain has stagnated and it may even decrease in the future. This would simply prevent MagIC from taking advantage of performance gains in the foreseeable evolution of processor technology. In addition, numerical experiments in  Matsui et al. (2016)  suggest that two-or three-dimensional parallelization might be necessary for dynamo codes to achieve petascale capabilities with current hardware architecture. The first two-dimensional MPI domain decomposition in pseudo-spectral codes in spherical geometry was considered in the ASH code by  Clune et al. (1999). The layout was nevertheless still not completely two-dimensional in the sense that the maximum number of MPI ranks was still bounded by the maximum spherical harmonic degree. This limitation was released in more recent implementations by  Marti and Jackson (2016)  in the context of modelling full spheres, by  Matsui et al.
(2014)  in the geodynamo code Calypso (2021), and in the Rayleigh code. The performance benchmark by  Matsui et al. (2016) , however, revealed an important overhead for these codes compared to pseudo-spectral codes which avoid global communications such as the open-source code XSHELLS (2021) (e. g. Schaeffer et al. , 2017). Motivated by the aforementioned points, we propose in this work a two-dimensional data distribution layout for MagIC with communication-avoiding features. This required a major rethinking of data structures and communication algorithms, and it demands a re-implementation and thorough optimization of a large portion of the existing 1d-hybrid code. Due to the high complexity of the required refactoring tasks, OpenMP parallelism was dropped from the current implementation for the time being.
Implementation MagIC is a highly optimized hybrid MPI+OpenMP code under active development. The 1d-hybrid version uses MPI to distribute the N r spherical shells amongst different computing nodes, while employing OpenMP for fully using the local cores within an individual NUMA domain. Although MagIC is finely optimized, multithreading across NUMA domains is not implemented. Therefore, simulations require at least one spherical shell per NUMA domain. This restricts the maximum number of NUMA domains to N r -1, which severely limits the computational resources that can be employed for a given radial resolution. This disadvantage could increase in the future since the current trend in HPC architecture is to further subdivide the computing units into physical or even logical NUMA domains. The purpose of this section is to first familiarize the reader with the established 1d-hybrid implementation and then inhttps://doi.
org/10. 5194/gmd-14-7477-2021 Geosci. Model Dev. , 14, 7477-7495, 2021 troduce the new 2d-MPI implementation. By adding MPI parallelism in a second direction, the extension also allows distributing the computations within a shell over the NUMA domains, sockets, or even nodes of a computer cluster. Due to the high complexity of this re-implementation, our 2d-MPI version lacks any use of OpenMP. The main purpose of this work is to provide a thorough assessment of the prospects and merits of a two-dimensional data distribution, to pinpoint shortcomings, and to discuss the overall viability of a possible fully optimized and fine-tuned two-dimensional distribution implemented using OpenMP+MPI. In this section we first present the pseudocode for the sequential algorithm for MagIC in Sect. 3. 1 and then discuss the distribution of the data for both the 1d-hybrid and 2d-MPI versions in Sect.
Radial loop implementation The spherical harmonics transform (SHT) takes a substantial portion of the runtime. MagIC relies heavily on SHTns  (Schaeffer, 2013) , a highly optimized library dedicated to computing the spherical harmonics transform using OpenMP, advanced SIMD vectorization, and cache-hitting strategies  (Ishioka, 2018). SHTns has been developed to compute the steps synthesized in lines 8-10 and lines 4-6: that is, independent of the radial level r. It can handle scalar fields and is also optimized for general vector fields and vector fields with a poloidal-toroidal decomposition. SHTns is written in C but has a Fortran interface with a dedicated data layout for MagIC. Although SHTns is flexible and modular, it does not have an MPI-distributed interface. However, it offers an interface for computing the Legendre transform (lines 5 and 10 of Algorithm 1) only for an individual (m, r) pair, which is the use case in our 2d-MPI implementation.
The transposition and FFT computations are then handled separately by MagIC. Next we describe how both the 1d-hybrid and 2d-MPI version of the code use SHTns for computing the SHT. 1d-hybrid radial loop. The spherical harmonics transforms are delegated directly to SHTns (see the pseudocode in Algorithm 2). For instance, for the temperature scalar field on the numerical grid, T (θ, φ, r k ), a single call to spat_to_SH returns the fully transformed T m (r k ) for a given radial level r k. This is only possible because all (θ, φ) points are local in the 1d-hybrid version of the code. No explicit optimization or multithreading is required, as SHTns handles this internally. The number of vector transforms needed per radial loop depends on the set of equations. Dynamo models under the Boussinesq approximation of the Navier-Stokes equations considered here (Eqs.
1-4) involve four vector transforms and one scalar transform for the backward SHTs and three vector transforms for the forward SHTs. 2d-MPI radial loop. The pseudocode for our implementation is shown in Algorithm 3. Since shtns_ml is called for single modes only, the implementation cannot benefit from the efficient internal OpenMP parallelization of SHTns. The computation time for only a single mode is too small to justify the initialization and termination of a multithreading context. In addition, the larger number of SHTns calls causes some overhead. Some speed-up could possibly be gained with an OpenMP parallelization of the SHTns calls (lines 3-5 in Algorithm 3) or directly at the radial loop level (line 1 in Algorithm 3). Some preliminary tests showed promising results for the former.
https://doi. org/10. 5194/gmd-14-7477-2021 Geosci. Model Dev. , 14, 7477-7495, 2021 1d-hybrid loop. The pseudocode is shown in Algorithm 4. The transposition itself takes place in lines 1 and 20 of Algorithm 4. Inside one MPI rank, the work is parallelized with OpenMP. We have a two-level loop structure, with an outer loop over the different degrees on the MPI rank and for each a loop over all possible m ≤. A single OpenMP parallelization of the outer loop would directly compete with the MPI parallelization and would also lead to a strong load imbalance as the different harmonic degrees have different numbers of m. To avoid nested OpenMP regions (for the outer and inner loop) we use the OpenMP tasking approach instead to balance the work and to combine the parallelism of the outer and inner loop.
We create an OpenMP task for each outer loop iteration which contains the LU decomposition of the matrix and a task creation loop for the inner m loop. To minimize the overhead for OpenMP task creation, we use chunks of several m together in one task. 2d-MPI loop. The pseudocode for the loop in the sequential and the 2d-MPI implementations (see Algorithm 5) differs only in the transpositions and the number of and m modes stored locally (which consequently affects the number of linear systems and right-hand sides in each rank). The distribution of the ( , m) tuples has already been discussed in Sect. 3. 2. While the difference in the number of modes has an impact on the computation of the solution of Eq. (  17 ), this pales in comparison with the time spent in communication. In both the 1d-hybrid and 2d-MPI implementations the transposition might be performed using the P2P, A2AV, and A2AW algorithms discussed previously.
For all tests, we set N φ = 2N θ and N = 2N θ /3, meaning that the geometry is completely described by N θ and N r. All tests were performed on the Cobra cluster at the Max Planck Computing and Data Facility (MPCDF). Each node of this machine possesses two Intel Xeon Gold 6148 processors and 192 GB of main memory. Each processor has a single NUMA domain with 20 cores. The nodes are interconnected with a 100 GB s -1 Omnipath interconnect and a non-blocking fat-tree topology. The workload is managed by Slurm, and hyperthreading is always disabled. The pinning is done automatically by Slurm and packs all θ ranks within a single NUMA domain whenever 1 < θ ≤ 20. We compiled all software using the Intel Fortran Compiler 2021. 2. 0 with Intel MPI 2021. 2. 0 and Intel MKL 2020. 4. The kernels of SHTns were compiled using GCC 10.
For the 1d-hybrid implementation, we use 10 OpenMP threads per MPI task for tests with up to 1000 cores (25 nodes) and 20 threads for larger runs (this configuration resulted in the best total time). The 1d-hybrid version of the code can use at most 6000 cores (150 nodes) https://doi. org/10. 5194/gmd-14-7477-2021 Geosci. Model Dev. , 14, 7477-7495, 2021 with one radial point per socket ( r = 300 and 20 threads per MPI task). For the 2d-MPI scheme, we fix the θ -transposition implementation to A2AW (as discussed in Sect. 4. 1) and the queue lengths to 1 for θ =10, 20, or 40 and 13 otherwise. This new version of MagIC admits more combinations of θ and r and allows the use 24 000 cores (600 nodes). For some core counts, multiple values of θ and r were allowed. For example, for 6000 cores one may choose θ = 20 or θ = 40 with r = 300 or r = 150.
Conclusions and future work We described a new parallelization scheme based on a twodimensional, MPI-based data decomposition for MagIC, an open-source code for three-dimensional fluid dynamics simulations in spherical geometry, with high scientific impact in a broad range of scientific fields ranging from fundamental fluid dynamics and modelling of planetary dynamos to stellar dynamics. MagIC uses spherical surface harmonics of degree and order m for the spectral representation in longitude and latitude and Chebyshev polynomials in radius. Our newly implemented 2d-MPI scheme is compared to the previously established 1d-hybrid code version that uses MPI and OpenMP and that has been highly optimized over years. Thanks to a number of new concepts, the 2d-MPI version presented here can already compete with the hybrid paral-lelization in terms of runtime, and in addition it offers the possibility to use a significantly larger number of cores.
It opens the possibility to employ tens of thousands of CPU cores on modern HPC clusters and paves the way to using the next-generation CPU architectures. Decisive factors for its success include a communicationavoiding data distribution of the and m modes for the computation of the actual time step in MagIC and the optimization of the associated communications strategies. Compared to the existing one-dimensional data decomposition, the new two-dimensional data distribution scheme requires an additional costly transposition in the azimuthal direction. This leads to a performance penalty at low core counts which might be hard to overcome. However, for the dynamo benchmark and the setups considered here, and starting at moderate core counts of 2000, the performance of the new version is mostly limited by the not yet fully optimized main radial loop, which can be as much as 60 % slower than its counterpart in the one-dimensional data distribution.
Our results showed that a mere 10 % performance gain in the computation time of the radial loop of the 2d-MPI implementation would bridge the gap between the two code versions for 2000 cores or more, in addition to providing an extended strong scalability regime. Other sets of equations solved by MagIC, such as the anelastic equations  (Jones et al. , 2011) , may present a different cost associated with the radial loop and would require a separate investigation. Our implementation paves the way towards a future unified hybrid variant of MagIC which combines a two dimensional data distribution with the proven benefits of an additional OpenMP layer in order to further improve the computational performance across an even wider range of simulation scenarios. This future 2d-hybrid version could retain the benefits of our communication-avoiding data distribution and its improved strong scaling behaviour, while still benefiting from the performance of the finely optimized multithreaded libraries used within the radial loop.
Code availability. MagIC's source code is under active development and available for download from https://github. com/ magic-sph/magic (last access: 24 June 2021) under the GNU GPL v3 license. The 1d-hybrid version discussed here is the commit c63930b of the master branch (version 5. 10). A frozen version is archived on Zenodo (https://doi. org/10. 5281/zenodo. 5204425, Gastine et al. , 2021). The 2d-MPI version is the commit 6cabfa7 of the merge-paral branch on GitHub, branched from commit c63930b. A frozen version has also been archived on Zenodo (https://doi. org/10. 5281/zenodo. 5171780, Lago et al. , 2021).
Data availability. No data sets were used in this article. Author contributions. TG and JW developed and maintained the main branch of MagIC. TG also managed the use of external libraries (FFTW, LAPACK, SHTns, etc. ) and fine optimization of OpenMP. TD conceptualized, implemented, and tested the algorithms for the 1d-hybrid implementation. RL conceptualized, implemented, and tested the algorithms for the 2d-MPI implementation. The numerical experiments were designed and conducted by RL with the supervision of TD and TG. TD and MR supervised HPC aspects of the project. JW and TG were responsible for supervising and interpreting the geophysical aspects of the project. The original draft was written by RL. The reviewing and editing of the current draft was conducted equally by all authors.
2011). Furthermore, recent work has shown a higher diversification rate of Oreocharis than the background diversification rate for the other gesneriad lineages in China  (Roalson and Roberts 2016). The species are lithophytic (i. e. , grow on rock or stony soil) and/or terrestrial and mainly comprise rosette herbs that occupy subtropical, humid, and monsoonal forests at elevations of 50-2,500 m, but sometimes even occurring as high as 3,000 m within the HDM region. Few are known from the tropical rainforests of Hainan and southern Yunnan Provinces (Supplementary Appendix S1 available on Dryad at http://dx. doi. org/10. 5061dryad. 9cnp5hqjf). The genus has three centers of diversity: one lies within the HDM and its adjacent areas in Southwest China (average altitude ca. 4,000 m), another is in the Wuling Mountains in Central China (average altitude ca.
We then fit macroevolutionary models to investigate the spatial and diversification dynamics and compare the diversification rates among lineages from different mountain systems. We also applied diversification models to infer the effect of global temperature changes and fluctuations in the strength of East Asian monsoons during the evolutionary history of Oreocharis. regions (except Japan) during our field investigations from 2016 to 2019 (Supplementary Appendix S1 and S2 available on Dryad). These samples represented 111 species (123 populations, including varieties) out of 120 described Oreocharis species, as well as six undescribed species based on morphological characters (labeled as Oreocharis sp. nov. 1-6) (Supplementary Appendix S2 available on Dryad). In the field, we recorded geographic coordinates using a Garmin-eTrex GPS device.