dataesr/scientific-paragraphs-categorization huggingface dataset:
In this paper, we present the VOLNA-OP2 tsunami model and implementation; a finite volume non-linear shallow water equations (NSWE) solver built on the OP2 domain specific language (DSL) for unstructured mesh computations. VOLNA-OP2 is unique among tsunami solvers in its support for several high performance computing platforms: CPUs, the Intel Xeon Phi, and GPUs. This is achieved in a way that the scientific code is kept separate from various parallel implementations, enabling easy maintainability. It has already been used in production for several years, here we discuss how it can be integrated into various workflows, such as a statistical emulator. The scalability of the code is demonstrated on three supercomputers, built with classical Xeon CPUs, the Intel Xeon Phi, and NVIDIA P100 GPUs.
In the Related Work section we discuss a number of codes currently being used in production, and as such are trusted and reliable codes, already being used as part of a workflow. Yet, the computational performance of most of these codes is "good enough"; they were written by domain scientists, and may have been tuned to one architecture or an other, but for example, GPU support is almost non-existent. In today's and tomorrow's quickly changing hardware landscape however, "future-proofing" numerical codes is of exceptional importance for continued scientific delivery. Domain scientists can not be expected to keep up with architectural advances, and spend a significant amount of time re-factoring code to new hardware. What to compute must be separated from how it is computed -indeed in a recent paper by  Lawrence et al.
(2017) , leaders in the weather community chart the ways forward, and point to Domain Specific Languages (DSLs) as a potential way to address this issue. OP2, by  Mudalige et al. (2012) , is such a DSL, embedded in C/C++ and Fortran; it has been in development since 2009: it provides an abstraction for expressing unstructured mesh computations at a high-level, and then provides automated tools to translate scientific code written once, into a range of high-performance implementations targeting multi-core CPUs, GPUs, and large heterogeneous supercomputers. The original VOLNA model  (Dutykh et al. (2011) ) was already discussed and validated in detail -it was used in production for small-scale experiments and modelling, but was inadequate for targeting large-scale scenarios and statistical analysis, therefore it was re-implemented on top of OP2; this paper describes the process, challenges and results from that work.
As the above references indicate, VOLNA-OP2 has already been key in delivering scientific results in a range of scenarios, and through the collaboration of the authors, it is now capable of efficiently supporting a number of use cases, making it a versatile tool to the community, therefore we have now publicly released it: it is freely available at github. com/reguly/volna. The rest of the paper is organised as follows: Section 2 discusses related work, Section 3 presents the OP2 library, upon which VOLNA-OP2 is built, Section 4 discusses the VOLNA simulator itself, its structure and features, Section 5 discusses performance and scalability results on CPUs and GPUs, and finally Section 6 draws conclusions.
(2010) ;  Acuña and Aoki (2009) ;  Liang et al. (2009b) , which use structured meshes and finite differences or finite volumes, it is unclear whether these are used in production, and they are not open source. Celeris  Tavakkol and Lynett (2017)  is a Boussinesq solver that uses finite volumes and a structured mesh -it is hand-coded for GPUs using graphics shaders, and its source code is available, however it can only use a single GPU. As far as we are aware, only  Tsunami-HySEA (Macías et al. (2017) ), also using finite volumes, is using GPU clusters in production -that code however only supports GPUs, and is hand-written in CUDA. Performance reported by Castro et al. (2015) on a 10M point testcase shows a strong scaling efficiency going from 1 GPU to 12 GPUs between 88% and 73% (overall 12 GPUs are 5.
88 faster than 1 GPU), and a 25× speedup with 1 GPU over an unspecified (likely single core) CPU implementation. Direct comparison to VOLNA-OP2 is not possible since Tsunami-HySEA uses (nested) structured meshes, and the multi-GPU version is not open source. The OP2 library,  Mudalige et al. (2012) , is a domain specific language embedded in C and Fortran that allows unstructured mesh algorithms to be expressed at a high level, and provides automatic parallelisation and a number of other features. It provides an abstraction that lets the domain scientist describe a mesh using a number of sets (such as quadrilaterals or vertices), connections between these sets (such as edges-to-nodes), and data defined on sets (such as x, y coordinates on vertices). Once the mesh is defined, an algorithm can be implemented as a sequence of parallel loops, each over all elements of a given set applying different "kernel functions", accessing data either directly on the iteration set, or indirectly through at most one level of indirection.
This abstraction enables the implementation of a wide range of algorithms, such as the finite volume algorithms that VOLNA uses, but it does require that for any given parallel loop, the order of execution must not affect the end result (within machine precision) -this precludes the implementation of e. g. Gauss-Seidel iterations. OP2 enables its users to write an application only once using its API, which is then automatically parallelised to utilise multicore CPUs, GPUs, and large supercomputers through the use of MPI, OpenMP and CUDA. This is done in part through a code generator that parses the parallel loop expressions and generates code around the computational kernel to facilitate parallelism and data movement, and in part through different back-end libraries that manage data, including MPI halo exchanges, or GPU memory management, as shown in Figure  1.
Parallelisation Approaches in OP2 OP2 takes full responsibility for orchestrating parallelism and data movement -from the user perspective, the code written looks and feels like sequential C code that makes calls to an external library. To utilise clusters and supercomputers, OP2 uses the Message Passing Interface (MPI) to parallelise in a distributed memory environment; once the mesh is defined by the user, OP2 automatically partitions and distributes it among the available resources. It uses the standard owner-compute approach with halo exchanges, and overlaps computations with communications. In conjunction with MPI, OP2 uses a number of shared-memory parallelisation approaches, such as CUDA and OpenMP. A key challenge in the fine-grained parallelisation of unstructured mesh algorithms is the avoidance of race conditions when data is indirectly modified.
Model, numerics, previous validation The finite volume (FV) framework is the most natural numerical method to solve the non-linear shallow water equations (NSWE), in part because of their ability to treat shocks and breaking waves. It belongs to a class of discretisation schemes that are highly efficient in the numerical solution of systems of conservation laws, which are common in compressible and incompressible fluid dynamics. Finite Volume methods are preferred over finite differences and often over finite elements because they intrinsically address conservation issues, improving their robustness: total energy, momentum and mass quantities are conserved exactly, assuming no source terms, and appropriate boundary conditions. The code was validated against the classical benchmarks in the tsunami community as described below.
Numerical model Following the needs of the target applications, the following non-dispersive NSWEs (in Cartesian coordinates) form the physical model of VOLNA: H t + ∇ • (Hv) = 0 (1) (Hv) t + ∇ • Hv ⊗ v + g 2 H 2 I 2 = gH∇d (2) Here, d (x, t) is the time-dependent bathymetry, v (x, t) is the horizontal component of the depth-averaged velocity, g is the acceleration due to gravity and H (x, t) is the total water depth. Further, I 2 is the identity matrix of order 2. The tsunami wave height or elevation of free surface η (x, t), is computed as, η (x, t) = H (x, t) -d (x, t) (3) where the sum of static bathymetry d s (x) and the dynamic seabed uplift u z (x, t) constitute the dynamic bathymetry, d (x, t) = d s (x) + u z (x, t) (4) d s is usually sourced from bathymetry datasets pertaining to the region of interest (say, global datasets like ETOPO1/GEBCO or regional bathymetries).
Code structure The structure of the code is outlined in Algorithm 1; the user inputs a configuration file (. vln), which specifies the mesh to be read in from gmsh files, as well as initial/boundary conditions of state variables, such as the bathymetry deformation starting the tsunami, which can be defined in various ways (mathematical expressions or files, or a mix of both). We use a variable timestep third-order (four stage) Runge-Kutta method for evolving the solution in time. In each iteration, events may be triggered; The original VOLNA source code was implemented in C++, utilising libraries such as  Boost, Schling (2011). This gives a very clear structure, abstracting data management, event handling and low level array operations for the higher level algorithman example is shown in Figure  5.
While this coding style was good for readability, it had its limitations in terms of performance: there was excessive amounts of data movement and certain operations could not be parallelised -indirect increments with potential race conditions in particular. Some features -such as describing the bathymetry lift with a mathematical formulawere implemented with functionality and simplicity, not performance, in mind. To better support performance and scalability, and thus allow for large-scale simulations, we have re-engineered the VOLNA code to use OP2 -the overall code structure is kept similar, but matters of data management and parallelism are now entrusted to OP2. To support parallel execution we separated the pre-processing step from the main body of the simulation: first the mesh and simulation parameters are parsed into a HDF5 data file, which can then be read in parallel by the main simulation, which also uses HDF5's parallel file I/O to write results to disk.
Performance-critical parts of the code, essentially any operations on the computational mesh, are re-implemented using OP2: they are written with an element-centric approach and grouped for maximal data reuse. Calculations that were previously a sequence of operations, each calculating all partial results for the entire mesh, now apply only to single elements (such as cells or edges), and OP2 automatically applies these computations to each element -this avoids the use of several temporaries and improves computational density. This process involves outlining the computational "kernel" to be applied at each set element (cell or edge) to a separate function, and writing a call to the OP2 library -a matching code snippet is shown in Figure  5. The workflow of VOLNA is made of a few sources of information being created and given as inputs to the code.
The first is the merged bathymetry and topography over the whole computational domain, i. e. the seafloor and land elevations, over which the flow will propagate. This is given through an unstructured triangular mesh. This is then transformed into a usable input to VOLNA via the volna2hdf5 code to generate compact HDF5 files. The mesh is also renumbered with the Gibbs-Poole-Stokmeyer algorithm to improve locality. The second is the dynamic source of the tsunami. It can be an earthquake or a landslide. To describe the temporal evolution of seabed deformation, either a function can be used, or a series of files. When a series of files is used (typically when another numerical model provides the spatio-temporal information of a complex deformation), there is a need to define the frequency of these updates in the so-called vln generic input file to VOLNA.
A recent improvement has been the ability to define these series of files for a sub-region of the computational domain, and at possibly lower resolution. Performance is better when using a function for the seabed deformation, since I/O requirements for files can generate large overheads -VOLNA-OP2 allows for describing the initial bathymetry with an input file, and then specifying relative deformations using arbitrary code that is a function of spatial coordinates and time. Similarly, one can also define initial conditions for wave elevation and velocity. The generic input file of VOLNA, includes information about the frequency of the updates in the seabed deformation, the virtual gauges where time series of outputs will be produced and possibly some options to output time series of outputs over the whole computational domain in order to create movies for instance.
Performance and Scaling on classical CPUs As the most commonly used architecture, we first evaluate performance on a classical CPUs in the Cambridge CSD3 supercomputer: dual-socket Xeon Gold 6142 CPU, with 16 cores each, supporting the AVX512 instruction set. We test a plain MPI configuration (32 processes per node), as well as a hybrid MPI+OpenMP configuration, with 2 MPI processes per node (1 per socket), and 16 OpenMP threads each, with process and thread binding enabled. We use OP2's vectorised code generation capabilities, as described in  Mudalige et al. (2016). The RK stage performs the same in both variants, however the fluxes and dT stages saw significant performance gains -the compiler did not automatically vectorise computations, it had to be forced to do so. The applyFluxes stage could not be vectorised due to a compiler issue.
Conclusions In this paper we have introduced and described the VOLNA-OP2 code; a tsunami simulator built on the OP2 library, enabling execution on CPUs, GPUs, and heterogeneous supercomputers. By building on OP2, the science code of VOLNA itself is written only once using a high-level abstraction, capturing what to compute, but not how to compute it. This approach enables OP2 to take control of the data structures and parallel execution; VOLNA is then automatically translated to use sequential execution, OpenMP, or CUDA, and by linking with the appropriate OP2 back-end library, these are then combined with MPI. This approach also future-proofs the science code: as new architectures come along, the developers of OP2 will update the back-ends and the code generators, allowing VOLNA to make use of them without further effort.
This kind of ease-of-use and portability makes VOLNA-OP2 unique between the tsunami simulation codes. Through performance scaling and analysis of the code on traditional CPU clusters, as well as GPUs and Intel's Xeon Phi, we have demonstrated that VOLNA-OP2 indeed delivers high performance on a variety of platforms and, depending on problem size, scales well to multiple nodes. We have described the key features of VOLNA, the discretisation of the underlying physical model (i. e. NSWE) in the finite volume context and the third-order Runge-Kutta timestepper, as well as the input/output features that allow the integration of the simulation step into a larger workflow; initial conditions, and bathymetry in particular, can be specified in a number of ways to minimise I/O requirements, and parallel output is used to write out simulation data on the full mesh or specified points.
facility in carrying out this work http://dx. doi. org/10. 5281/zenodo. 22558. Serge Guillas gratefully acknowledges support through the NERC grants PURE (Probability, Uncertainty and Risk in the Natural Environment) NE/J017434/1, and "A demonstration tsunami catastrophe risk model for the insurance industry" NE/L002752/1. Serge Guillas and Devaraj Gopinathan acknowledge support from the NERC project (NE/P016367/1) under the Global Challenges Research Fund: Building Resilience programme. Devaraj Gopinathan acknowledges support from the Royal Society, UK and Science and Engineering Research Board (SERB), India for the Royal Society-SERB Newton International 10 Fellowship (NF151483). Daniel Giles acknowledges support by the Irish Research Council's Postgraduate Scholarship Programme.
Acknowledgements. We would like to thank  Endre László , formerly of  PPCU ITK , who worked in the initial port of Volna to OP2. István Reguly was supported by the  János Bólyai Research Scholarship of the Hungarian Academy of Sciences. Project no. PD 124905  has been implemented with the support provided from the  National Research, Development and Innovation Fund of Hungary , financed under the PD_17 funding scheme. The authors would like to acknowledge the use of the  University of Oxford Advanced Research Computing (ARC).
Code availability. The code is available at https://github. com/reguly/volna/, and DOI: 10. 5281/zenodo. 1413124 It depends on the OP2 library, which is also available at: https://github. com/OP-DSL/OP2-Common, and depends on an MPI distribution, parallel HDF5, and a partitioner, such as ParMetis or PT-Scotch. For GPU execution, the CUDA SDK and a compatible device is required.
Methods The study was conducted in 27 centres in 14 European countries and Israel consisting of general hospitals and psychiatric specialty clinics. (Clinicaltrials. gov identifier is NCT01248195). Patients with schizophrenia or schizophreniform disorder were treated for four weeks with up to 800 mg/day amisulpride in an open design. Patients who did not meet symptomatic remission criteria at four weeks were randomized to continue amisulpride or switch to olanzapine (max 20 mg/day) during a six-week double blind phase. Patients who were not in remission at ten weeks were given clozapine (max 900 mg/day) for an additional  12  weeks in an open design. Data were analyzed using a generalised linear mixed model, with a logistic link and binomial error distribution.
Interpretation In the large majority of patients in the early stages of schizophrenia, symptomatic remission can be achieved using a simple treatment algorithm comprising the sequential administration of amisulpride and clozapine. Comments from the editor 1) Please ensure the NCT number you have provided is correct (the one cited is only for the part of the study in  Denmark). The correct NCT number is NCT01248195, this has been adjusted in the manuscript. 2) Please describe the whole study (as in the protocol); you may then state, and reference, that parts of the study have/will be published elsewhere  (eg, imaging, biomarker, psychosocial). The following information is now included in the Methods section of the manuscript: "Blood was drawn at the beginning and end of each treatment phase in order to relate proteomics, immune parameters and genetics to treatment outcome.
Methods The study was conducted in 27 centres in 14 European countries and Israel consisting of general hospitals and psychiatric specialty clinics. (Clinicaltrials. gov identifier is NCT01248195NCT01555814). Patients with schizophrenia or schizophreniform disorder were treated for four weeks with up to 800 mg/day amisulpride in an open design. Patients who did not meet symptomatic remission criteria at four weeks were randomized to continue amisulpride or switch to olanzapine (max 20 mg/day) during a six-week double blind phase. Patients who were not in remission at ten weeks were given clozapine (max 900 mg/day) for an additional 12 weeks in an open design. Data were analyzed using a generalised linear mixed model, with a logistic link and binomial error distribution. were not in remission continued to the six-week double-blind switching trial with 72 patients (77.
A randomisation table was generated by the Data Management department of the Julius Center, University Medical Center Utrecht, the Netherlands, using VB. Net with access to a SQL Server back end database. Study medication was packaged-in line with this randomisation table, using sequentially numbered kits. Randomization was performed online by a randomization website, also developed by the Julius Center, which provided the applicable kit number only. The application implemented stratification by site and gender, and applied the minimization method for randomization. The data management group was not involved in patient recruitment, which was conducted at each participating center. The complete study teams at each center as well as the participating patients were blind to treatment allocation.
1 All patients started with a four-week open label treatment with amisulpride 200-800 mg/day (Phase I). At the start of phase II, non-remitters from phase I were randomized 1:1 to double blind flexible dose treatment with olanzapine (5-20 mg/day) or amisulpride (200-800 mg/day). A randomisation table was generated by the Data Management department of the Julius Center, University Medical Center Utrecht, the Netherlands, using VB. Net with access to a SQL Server back end database. Study medication was packaged-in line with this randomisation table, using sequentially numbered kits. Randomization was performed online by a randomization website, also developed by the Julius Center, which provided the applicable kit number only. The application implemented stratification by site and gender, and applied the minimization method for randomization.
2007 , Ciudad et al. 2009 , Dunayevich et al. 2006 , Helldin et al. 2006 , Helldin et al. 2007 , Lambert et al. 2006 , San et al. 2007 , Emsley et al. 2007 , Nasrallah and Lasser 2006 , Opler et al. 2007 , van Os et al. 2006). Analyses of large databases showed that they are an achievable and realistic goal in antipsychotic drug trials  (Leucht et al. 2007a ). We will not analyse the time criterion of the remission criteria which only applies to long-term studies. Another advantage of these remission criteria is that they are easier to interpret than the mean change from baseline to endpoint of the PANSS. A symptomatic remission is also a harder outcome than a mean change of a rating scale. As remission means that key symptoms of schizophrenia are at the most only mildly present, such patients usually do not need further hospitalization.
III Finding biological predictors of treatment response The primary outcome for biological predictors will be drug response defined categorically (remission vs non-remission) as a function of biological profile (eg: presence or absence of genetic variants, proteomic profile, metabolomic profile, immunologic profile). The primary outcomes of the MRS scans are the difference between responders and non-responders in regional glutamate levels a) at baseline and b) between baseline and after one month of treatment with amisulpiride. We will addionally acquire a third MRS scan after 10 weeks of treatment (end of Phase II) to determine whether a) in patients randomised to amisulpride in phase II whether changes in glutamate take some time longer to appear, by comparing baseline and 10 week data and b) comparing second (end of phase 1) and third scans (end of phase 2) in patients randomised to amisulpride versus olanzapine in phase II whether a multireceptor agent (olanzapine) has greater effect on glutamate than a dopaminergic agent (amisulpride).
Therefore, a database lock of the eCRF of phase II (and phase I) data will occur per patient before that individual patients' treatment can be unblinded by the central study team, which has read-only access to the eCRF. After this lock, any changes to the collected data can only occur through standardised queries by the data management center, and will be appropriately documented. This procedure, 'unblinding according to protocol', needs to be performed in a very precise manner. As several data completeness checks need to be performed before the unblinding can take place, one week is scheduled in to ensure proper conduct. For this period, patients who meet remission criteria receive 1 week additional double blind medication. Once the treatment is known, they subsequently flow to the psychosocial intervention.