dataesr/scientific-paragraphs-categorization huggingface dataset:
Workflow management When performing calculations of large systems, it is often necessary to apply more complicated workflows than are required at the small scale. Large systems take many steps to define and set up, multiple levels of theory might need to be coupled, calculations are deployed on remote supercomputers, and post-processing procedures are potentially costly in terms of processing power, memory, and disk space. To address these issues, we have developed a python package called PyBigDFT as a framework for managing DFT workflows. PyBigDFT is able to handle building complex systems or reading them from a variety of file types, performing calculations with BigDFT, and analyzing calculation results (including through the complexity reduction framework). PyBigDFT further offers interoperability with other chemistry codes for multiscale modelling.
Meeting the exascale challenge Looking ahead to the exascale era, BigDFT developments have focused on both internal code aspects (compilation and modularity, parallel performance), as well as external usability, both of which are necessary for an efficient transition to ever-evolving architectures. The compilation of the code suite relies on the splitting of the code components into modules, which are compiled by the bundler package. This package is defined from a fork of the Jhbuild program that has been conceived in the context of the GNOME developers consortium. This package lays the groundwork for developing a common infrastructure for compiling and linking together libraries for electronic structure codes, and it is employed as the basis for the ESL bundle. It can be used to install the BigDFT suite, as well as a variety of optional packages such as python modules that can be used with PyBigDFT.
Exascale machines will offer both the ability to simulate large systems as well as extraordinary capacity for high-throughput calculations. To facilitate driving thousands of calculations of large systems on those machines, we have been developing a Remote Runner capability in our PyBigDFT package. The remote runner automatically serializes data and arbitrary python closures, which are then executed through a supercomputer's queuing system. A remote function might be used to perform a BigDFT calculation, run other chemistry codes, or to perform resource heavy pre/post-processing steps. Calculations are performed in a lazy way, so that the first time a workflow is run the computationally demanding calculations are asynchronously submitted remotely, and subsequent runs of the workflow skip the calculation steps.
This makes it easy to build analysis routines on top of the data generated from large scale DFT calculations. Currently, BigDFT is being deployed on pre-exascale machines including Archer2 (Edinburgh Parallel Computing Centre), Fugaku (RIKEN Center for Computational Science), and Irene-ROME (Très Grand Centre de Calcul). We are hopeful that the ongoing developments of the BigDFT code will provide a roadmap for performing large scale QM calculations of unprecedented size. In the future, we plan to extend the scope of BigDFT sparse matrix linear algebra solvers to techniques bringing them to the exascale. The new approach will make use of lessons learned from libraries such as NTpoly, DBCSR, and EigenExa. The resulting exascale-ready library may combine multiple numerical methods, featuring a hierarchy of implementations that can be automatically adapted to current and future supercomputing platforms, and will further enhance the capabilities of the simulation software and will extend the scope of linear-scaling electronic structure methods and quantum transport to other computer codes and formalisms.
CONQUEST calculates the total energy, forces and stresses exactly, and allows structural optimization of both ions and simulation cell. MD calculations within the NVE, NVT and NPT ensembles are possible with both exact diagonalization and linear scaling  [27]. The code interfaces with LibXC to implement local density approximation (LDA) and generalized gradient approximation (GGA) functionals, with metaGGA and hybrid functionals under development. Dispersion interactions can be included using semi-empirical methods (DFT-D2/3, TS) and vdW-DF. The polarization can be calculated using Resta's approach. It is open source, and is freely available under the MIT license from https://github. com/OrderN/CONQUEST-release. We have recently applied CONQUEST to calculations in complex ferroelectric systems with up to 5000 atoms  [28] [29] [30] , investigating problems that require large simulation cells and electronic structure methods.
Development priorities CONQUEST already allows simulation of significantly larger systems than standard DFT codes, and we want to improve the efficiency and scaling of the code to enable even larger systems to be modelled. We also want to reduce the total computational time required per MD step, to enable longer timescales to be addressed. At present, exact diagonalization simulations use the ScaLAPACK solver, but an interface to ELPA has been developed and is being deployed. Alongside these efficient approaches to exact diagonalization, we are prioritizing lower scaling solvers which allow the calculation of selected eigenstates. Perhaps the best known of these in the electronic structure community is PEXSI, which is available through ELSI, though it would require some work to interface this to the CONQUEST matrix storage (which was designed specifically for high efficiency, highly parallel linear scaling calculations).
Meeting the exascale challenges The key challenge associated with the transition to exascale computing for CONQUEST is to update and adapt the code to new hardware architectures, in particular, efficient use of CPUs with many cores, and GPUs and other co-processors, while maintaining excellent parallel scaling. The code will need to become more heterogeneous, with on-node calculations distributed between CPU cores and GPUs, and a few MPI processes dedicated to inter-node communication; careful interleaving of different parts of the calculation to different hardware on a node will be key to maximizing performance. When working in exact diagonalization mode, CONQUEST currently relies on ScaLAPACK, with an interface to ELPA in deployment. We will monitor developments in the area of exascale solvers (including projects such as MAGMA), as well as implementing other solvers which scale well in parallel, and which can make efficient use of local resources.
Meeting the exascale challenges exciting envisions reaching exascale performance via two distinct strategies. The first is the implementation of novel algorithms that facilitate optimal and massive parallelization; the second is the automation of workflows to enable highly concurrent job submission and management. In GS calculations of close-packed systems, the iterative Davidson eigensolver provides only little or even no gain over the conventional diagonalization approach. To meet the challenge of performing large-scale all-electron calculations for materials science applications, we will rely on SIRIUS (https://github. com/electronic-structure/SIRIUS), a high-performance library for the LAPW basis, with asynchronous MPI and GPU support. The integration of SIRIUS in exciting enables the hand-off of the most computationally demanding parts, allowing one to perform GS calculations on systems up to an order of magnitude larger than currently possible.
The last decade has seen substantial growth in molecular and materials databases, however in general, existing data on solids lack physical validation. Therefore, the second strategy of approaching the exascale concerns high-throughput calculations. We are developing automated end-to-end workflows to leverage exciting's high precision, such to provide benchmark data for various properties and material classes. Such data are also urgently needed in view of data-quality assessment in data collections  [66]. This concerns particularly the aspect of Interoperability-the I in FAIR-when data from different sources are brought together such to employ them, for instance in machine-learning tasks. On-going work in this direction comprises various aspects that are key to reach our goals. These involve the automation of building transferable LAPW basis sets as well as the selection of optimal computational parameters.
Automation is currently built with workflows on single-and multi-task levels following the schematic shown in figure  12. The central piece labelled as excitingworkflow is a Python script put together by a user from existing components. To this extent, it makes use of (i) excitingtools  [67] , an IO-based API to exciting, that provides full control over input parameters, (ii) workflow managers such as Jobflow or ASR, and (iii) a task scheduler such as Fireworks.
Development priorities The numerical foundation on NAO basis sets lies at the core of FHI-aims, allowing to represent the electronic structure of any problem in chemistry or materials science and engineering, without shape approximations. Support for and compatibility with Gaussian-type and Slatertype orbitals is contained in the code and important for excited-state calculations and electronelectron correlation beyond DFT. Key priorities that drive the ongoing developments include: An overview of the FHI-aims code, including its core functionalities, external libraries directly coupled to the code, its integration with external workflow drivers and its integration with the 'graphical interface for materials science' (GIMS)  [76]. Start page of the graphical interface GIMS  [76]  for the FHI-aims code (print version).
In this context, we note that critical tools in high-performance computing, such as MPI libraries, compilers and numerical libraries are insufficiently standardized and can present a steep learning curve for newcomers to the field. Reducing this learning curve through tutorials, testing, dedicated code advances and infrastructure will remain an overarching priority for future FHI-aims developments, in particular targeting new accelerator models towards the exascale era (see below). • Community. FHI-aims is a code based in a large academic community, especially when it comes to a plethora of new developments that a single, small team could not shoulder. Key examples are, e. g. refined density functionals that capture dispersion interactions accurately, real-time time dependent DFT, incorporation of nuclear quantum effects both in the code and by external tools, thermal and electrical transport calculations, GW approaches, and many more.
Our strategy in FHI-aims has been to build the code around an 'MPI first' paradigm, meaning that every computational method is foremost parallelized without any a priori restriction of execution across compute nodes in even the largest supercomputers (∼200 000 cores were demonstrated already in 2011). Shared-memory parallelism within each node can be implemented through the MPI-3 standard for multicore processing systems where needed, but importantly controllable where needed from within the code. Through work for NVidia GPUs, we already have a working strategy to treat particular computational hotspots by GPU offloading  [69, 70, 78]. Figure  16  shows the impressive power that can be leveraged for large problems on even a few nodes of the preexascale computer Summit (42 Power9 cores and six NVidia V100 GPUs per node) at Oak Ridge National Laboratory, compared to many tens of nodes of the Cori Intel Haswell system (32 cores per node) at National Energy Research Center only a few years earlier.
Background and current status EST attempts to accurately approximate solutions to the electronic Schrodinger equation in order to accurately predict chemical properties and reactions. Within the field of EST, NWChem  [80]  is arguably the de facto choice for high-performance molecular EST studies. Unfortunately, NWChem was initiated over 20 years ago and computer hardware and software architecture has far surpassed the original design. With the advent of the exascale era, the decision was made to focus efforts on NWX  [81] , a complete rewrite and redesign of NWChem from the ground up. Figure  17  shows the key design points and features of NWX. The upper-left, red box lists the features anticipated to be in the version 1. 0 release (tentatively slated for late 2023). Like the original NWChem, NWX's primary niche is still high-performance EST.
Unlike the original Fortran-based NWChem, the core of NWX is written using C++17 with nearly all functionality accessible via Python bindings and targeted for heterogeneous computers. The Python bindings (and additional Python functions) allow all users to access and manipulate the software in order to run EST simulations. NWX also has the fairly unique (at least from the perspective of EST) feature of being entirely plugin based. At the heart of NWX is PluginPlay  [82] , a generic C++ framework for runtime manipulations of a program's call graph. PluginPlay views the call graph as a series of interconnected modules where each module tends to have the granularity of a function in a conventional EST package. To remain domain agnostic, PluginPlay modules necessarily have generic application programming interfaces (APIs), the SimDE layer provides additional domain-specific module APIs using familiar chemistry concepts (e.
g. molecules, basis sets, wavefunctions). The remainder of NWX is written as encapsulated modules, each of which relies on the APIs and infrastructure in SimDE. Ultimately, the reliance on modules aids in: performance-tuning, reuse of NWX modules by other EST packages, incorporation of third-party contributions into NWX, and rapid prototyping of new theories by allowing developers to dynamically, at runtime, non-invasively modify the call graph. The layered approach helps NWX obtain performance portability by providing separation of concerns through high level APIs and algorithms that have specific implementations for CUDA, HIP and SYCL (for example) underneath. OpenMP could be supported in this approach, although it is currently not as performant.
Development priorities The NWX team has taken the need to rewrite the original NWChem package as an opportunity to create our ideal next-generation EST ecosystem. In particular, not only must NWX perform well, it must also adhere to modern computer science practices, be highly customizable and extensible, easily integrate into existing scientific workflows, be developer-friendly, and be as user-friendly as possible. Historically the EST community has viewed these design points as being at odds, making this arguably a tall order. Consequently, one of the largest and most Aside from the design of NWX, our other top development priorities have been implementing: the computational infrastructure underlying NWX (i. e. PluginPlay and SimDE), electronic embedding, DFT, and domain local, pair-natural orbital (DLPNO) based implementations of second-order Moller-Plesset perturbation theory (MP2) and coupled cluster with single, double, and perturbative triple excitations [CCSD(T)].
Multiconfiguration self-consistent field (MCSCF) will also be available to facilitate modeling systems with strong-correlation. While not a development priority, we have also implemented both conventional and density-fit (DF) versions of SCF, MP2, and CCSD(T). NWX version 1. 0 will be able to compute energies for all of the aforementioned methods and gradients for SCF and DFT, at a computational cost which is reasonably competitive with existing EST packages. In addition, third-party software, such as LAMMPS  [83]  and geomeTRIC  [84] , will enable MD and optimizations. Admittedly the road to NWX version 1. 0 has been a long one fraught with challenges. Chief among these challenges has been the learning curve associated with transitioning to modern C++17 object-oriented programming, the quickly changing hardware/software landscape, and the difficulty in attracting new talent to the project.
Explicitly-correlated (-F12) implementations of DLPNO-MP2 and DLPNO-CCSD(T)  [85]  will be added to increase the fidelity of the linear-scaling methodologies. NWX is really envisioned as part of a larger software ecosystem. In the ideal ecosystem the broader EST community would develop functionality which adheres to community-wide standardized APIs. This would enable third-party software developed outside the NWX organization to be interoperable with NWX, via the module system. While SimDE proposes a set of standard APIs, modules can be made by translating between different APIs. The last NWX development priority is to leverage the SimDE APIs to grow NWX's functionality. Our initial efforts will focus on GhostFragment, a package for fragment-based methods, and EXESS  [86]  (an Exascale Computing Project offshoot of the GAMESS  [87]  package).
Meeting the exascale challenges The modular nature of NWX is one of the keys to achieving high-performance on today's machines, and to continuing to achieve high-performance on future machines. More specifically, since every major part of NWX is implemented as a module, and since the modules are only coupled at runtime, modules which are found to be bottlenecks can be rewritten without breaking the surrounding code. In practice modules are relatively fine-grained, so this allows us to, for example, focus on porting the evaluation of the exchange-correlation potential to GPUs without having to rewrite the remainder of the DFT algorithm. Of course, achieving high-performance requires porting more than just the exchange-correlation potential to GPU; however, with the separation of concerns imparted by the module system this can be done piecewise.
These details are then currently mapped to the TiledArray  [88]  back-end. As an alternative tensor model we have also developed the tensor algebra for many-body methods (TAMM)  [89]  software. The beauty of using TensorWrapper for the SimDE API is that it makes interfacing software developed directly with TiledArray or TAMM possible. The backend for the tensor is set during construction, after which point module developers have little need to worry about the backend again. The vast majority of modules take as inputs already created tensors, and create new tensors by performing fundamental tensors operations (e. g. adding, contracting, slicing, or permuting) on the input tensors. The consequence of this API is that nearly all of the complexity of the data movement is encapsulated by the tensor class (and an expression template layer).
The initial release of NWX will include a series of modules implementing several common EST methods (HF, MP2, CCSD(T), and MCSCF), as well as emerging linear-scaling variants of MP2, CCSD, and CCSD(T). The performance of these modules is currently competitive with implementations found in other EST packages, although only some of these modules will be fully exascale ready upon release. Future development priorities are focused on improving performance of existing features, implementing additional first-party modules (e. g. F12 methods and additional nuclear derivatives), and filling out NWX's feature set through third-party module support. While the PluginPlay-based framework underlying NWX is a powerful development tool, it can also be overwhelming for the user who just wants to run the software.
g. reaction barrier heights. Qbox addresses this challenge by coupling to the Software Suite for Advanced Generalized Ensemble Simulations (SSAGES)  [100]  through its client-server interface. This coupled approach has been used to explore the free energy surface of a dipeptide  [101] , catalytic reactions on a metal surface  [102] , and the conformation of gold clusters  [103]. In such coupled Qbox-SSAGES simulations, multiple instances of Qbox are 'driven' by SSAGES in order to improve the efficiency of statistical sampling (see figure  21 ). In applications to PIMD simulations, Qbox was coupled to the i-PI software [104] that implements path integral sampling and various generalized Langevin thermostats. This was used to study nuclear quantum effects in diamond and amorphous carbon  [105].
Ideally, the adoption of a single, directive-based, portable programming model such as OpenMP, OpenACC or OneAPI should make it possible to maintain a single branch of a code. This approach must however be tested to verify that the resulting performance is comparable to hand-written specialized versions of the code. As this evaluation requires considerable rewriting, it has not yet been performed on Qbox. We plan to explore an alternative way to facilitate portability. Using the polymorphism and inheritance features available in C++, the architecture-dependent, hand-optimized parts of the code may be limited to specific platform-dependent derived classes, which are selectively instantiated at run time according to the available hardware. This approach can in principle preserve the performance of hand-optimized code while limiting the amount of such specialized code.
Development priorities One of the original goals of QE was to become a hotbed for innovation. While it is hard to predict which new developments will be proposed by the scientific community in the future, one can be confident that a primary goal of QE will always be to make implementation of new algorithms, methods, and theories as easy as possible. In order to achieve such a goal while keeping the complexity of QE under control, an open and sustainable development model was devised. An increasing number of QE components are being encapsulated into reusable libraries and Fortran modules, thus making new developments easier to implement (figure  24 ). Among development priorities for the next few years, we mention in particular the implementation of new advanced functionals, notably Hubbard-corrected, Koopmans-compliant, meta-GGA and nonlocal van-der-Waals functionals.
A development priority of more technical character, but one related to some of the mentioned applicative objectives, is making QE more interoperable with other languages (notably modern object-oriented ones like python) and with other pieces of software. The effort in this field is ongoing since several years and has already produced significant results, such as structured (xml, hdf5) I/O files and various degrees of actual interoperability with other software. As a further step forward in this direction, a set of documented API's for calling QE subroutines will be introduced, making QE routines more easily accessible from other codes or directly from Python-based software packages.
Scaling beyond such limits strongly depends upon the specific calculation and in particular upon the availability of additional parallelization levels (see figure  25 ). The push towards hybrid and accelerated processors (e. g. GPGPU) of the last few years has been addressed in QE with a variety of approaches. For Nvidia GPUs, the first approach was a porting based on the CUDA Fortran extension  [131]. The result was very gratifying in terms of performance, much less so in terms of portability to other architectures and also of maintainability (due to extensive code duplications, imposed by limitations of CUDA Fortran). More recently, most NVidia-specific code has been moved to OpenACC, with no loss of performance and much reduced code duplication. An ongoing effort is under way, and almost completed, to extend the NVidia porting with OpenACC to the entire suite and not only to the most used basic components.
Concluding remarks The original motivation to develop QE was to provide a unified set of software tools for a small but very active community of experts in first-principle methodologies and simulations. The scope of QE has grown during the years and its ambition is now to cater to a wider community of scientists and engineers, working in different fields, that includes non-specialists as well. For this reason the development of new capabilities is accompanied by an intense dissemination activity, especially targeting younger scientists and aimed towards spreading knowledge and expertise in first-principle DFT calculations and in their usefulness. We consider this activity an integral part of the 'QE experience' and one of the most successful aspects of QE.