dataesr/scientific-paragraphs-categorization huggingface dataset:
The goal of this paper is two-fold. First, it provides a comprehensive living review dedicated to the capabilities and performance of Tinker-HP's main production methods (i. e. force fields) on Intel's architectures. Second, it is organized to present in a pedagogical fashion our code optimization efforts, outlining an optimal strategy for vectorization of Tinker-HP. Although there have been theoretical discussions of AVX-512 vectorization  [3] , and a description of the effective vectorization of a Quantum Monte-Carlo production code  [4] , such practical examples are rarely documented and we think our experience could prove useful to other software developers. The present version of the paper is organized as follows. After reviewing the specificities of the latest Intel Xeon Scalable Processors (code-named Skylake) and particularly their Intel AVX-512 vector instruction set, we present the general structure of the most CPU intensive Fortran subroutines in Tinker-HP, show their performance hotspots and propose a general strategy to vectorize the code within the Intel AVX-512 instruction set.
Definitions In this paper, we will use two versions of Tinker-HP : 1. the Release Version 1. 1, referred to as Rel. 2. the Vectorized Version 1. 1v, referred to as Vec. A third version, Release Version 1. 2 (referred to as Rel2), is mentioned in the Perspective subsection 8. 3 to give a feeling for the performance gains anticipated with newlydeveloped algorithms. Vectorization of Rel2 is in progress. It will be referred to as Vec2. We ran Tinker-HP exclusively on supercomputers under Unix/Linux Operating System (OS). These machines aggregate hundreds or even thousands of interconnected systems called computing nodes, or simply nodes, each typically having tens of CPU cores and hundreds of gigabytes of memory. On Unix/Linux computers, the code is executed by a process, which uses memory and CPU resources managed by the OS.
• -S flag produces assembly code listings. FFLAGS = -O3 -xCORE-AVX512 -qopt-zmm-usage=high -no-ipo -no-prec -d i v -shared -i n t e l -a l i g n array64byte -f a l i g n -f u n c t i o n s =64 -qopt-report -phase=vec -qopt-report =5 -S -qoverride -l i m i t s Listing 1. Flags used for the compilation of Tinker-HP with Intel Fortran compiler. The vectorization reports are of major interest, because they give precise indications on how the compiler understands the code and what decisions it makes, allowing the programmer to modify the code or give the compiler pertinent directives. Some indications are of particular interest: • the vector length, which gives the number of elements (real, integer or logical) affected by one operation • the scalar cost, which gives the number of scalar operations necessary to handle one iteration of the loop • the vector cost, which gives the same information as the scalar cost, but for vector operations • the estimated potential speedup.
Execution setup For the performance tests, calculations were performed on nodes running under the RedHat* Enterprise Linux* Server Release 7. 4 operating system, with the following configuration : • 2 × Intel Scalable Xeon 8168 processor -2,7 GHz -24 cores/processor • 192 GB of DDR4 memory, • InfiniBand* EDR interconnect. We chose 8 MS from those studied in  [1]  with increasing sizes ranging from 9 737 to 3 484 755 atoms : the Ubiquitin protein, the prototypic Dihydrofolate Reductase (DHFR), the COX-2 dimer, the Satellite Tobacco Mosaic Virus (STMV), the Ribosome full structure in polarizable water, and three water boxes (Puddle, Pond and Lake). The table  1  gives for each MS the name, the number of atoms and the number of cores used for the parallel calculations. All calculations have been made starting from an equilibrated MS with a timestep of 2fs and a RESPA integrator  [19].
Polarizable AMOEBA force field We focus here on the part of the code dedicated to the AMOEBA polarizable force field which is the most computationally challenging and gives a lower bound for Tinker-HP performance  [5, 20]. AMOEBA has been shown to have wide applicability for physical systems ranging from liquids to metals ions,  [21, 22]  including heavy metals,  [23, 24]  in gas and solution phase, to proteins  [6, 7]  and to DNA/RNA  [7]. It uses distributed atomic multipoles through quadrupole moments and a Thole damped point dipole polarization interaction model. Van der Waals interactions use the Halgren buffered 14-7 function. In this paper, we used the AMOEBA 2013 protein parametrization  [6, 7]  coupled with the 2003 water model  [5].
Hotspots The table  2  shows the profiling of Rel, running on one core for DHFR with AMOEBA (polarizable model) and with CHARMM (non-polarizable model). We give the name of the module or routine, the real CPU time spent executing it, and the vector usage percentage. All routines are sorted with higher time-consumption listed first. We can see that Rel has two kinds of hotspots : 1. NUC hotspots : these are mainly due to libraries calls, system calls and memory management operations (initialization, copy, allocation and deallocation). 2. Computational hotspots : these are mainly due to the computation of : • the matrix-vector product operation applied at each iteration of the polarization solver (tmatxb_pme2), which can be called up to 12 times at each step, depending upon the convergence method • the dipole polarization energy and forces (epolar1) • the van der Waals energy and forces for the Halgren buffered 14-7 function (ehal1) • the multipolar permanent electrostatic energy and forces (empole1 • the van der Waals energy and associated forces for the Lennard-Jones 6-12 function (elj1) • the charge-charge interaction energy and associated forces (echarge1) The routines used to build neighbor lists (vlistcell, mlistcell and clistcell ) appear.
Other widely used utility routines (image and torque) also appear. In order to raise the performances of Rel, we need to address these hotspots. Two observations guide us : 1. vmlinux is taking almost as much CPU time as the multipole polarization energy and forces computation routines. This means the process makes many system calls and performs many memory operations. 2. the vector usage percentage is strictly 0. 00 for all the computation subroutines. This confirms that these routines only use scalar operations. The first observation led us to investigate the library and system calls and, first and foremost, to work on the memory management of a process running Tinker-HP in order to reduce memory operations. The second observation forced us to rewrite the computation routines. Since using vector operations generally means using loops on arrays, the on the fly method of computation in Rel must no longer be used.
System calls When running, Tinker-HP reads a few files at the very beginning and outputs a log file which contains the simulation results and, periodically, a file containing atoms coordinates. As these input/output operations are done by the MPI-rank-0 process only, the open, read and close system calls do not really stress the Unix/Linux system, even if the simulation runs on millions of atoms. So, most of the system calls come from the MPI library and are due to two design choices in Tinker-HP : 1. As explained before, each Tinker-HP process hold a portion of the space (a domain) and maintains MPI communications with MPI processes that hold other domains nearby it, so that each process can exchange information with the others and track atoms coming in or out of its own domain. As the other processes can run on other nodes, there can be even more time spent in system calls because of network transmissions.
2. A memory region is shared between all MPI processes on a computing node, using the sharing capabilities of the MPI library. The access control of this region is implemented through system calls (semaphores, flock(). ) to synchronize processes and guarantee non overlapping when writing data. Minimizing the time spent in system calls is not so easy. We tried different distributions of the MPI processes over the nodes to favour local MPI communications, but that did not give convincing results. We also tried to improve the use of MPI framework by masking communications with computations. We used non blocking versions of MPI_SEND and MPI_RECEIVE functions, and did some calculations before calling the corresponding MPI_WAIT. The performance gain is not really noticeable for now. But improving this part of the code is a work in progress.
Memory management The figure  1  gives a simple picture of the memory layout of a Unix/Linux process. The text zone contains the code, the data zone contains initialized data. The bss zone contains uninitialized fixed size data and has itself a fixed and limited size, set at compile time. The heap zone contains allocated data and can grow or shrink, subject to allocations or de-allocations. The stack zone contains a Last-In-First-Out structure, where values are pushed and pulled at each subroutine or function call or if the processor runs out of free registers.
Process memory setup Historically, Tinker dynamically allocates and de-allocates all arrays it uses, because it was originally built to run on workstations with limited amount of memory and cores. These allocations are made by calls to the system malloc() group of functions. As a consequence, data are put in the heap section of the process, whose size is managed by the OS, allowing it to grow or shrink as needed. As Tinker-HP is a pure MPI program which distributes data and can potentially run on hundreds of different nodes, each of them with gigabytes of memory, the problem of the memory consumption is not that important. On a computing node, each core (so, each MPI process) can easily have 2 or even 4 gigabytes of memory for its own use. Still the size of some arrays are proportional to the size of the systems and therefore, the MS-size dependent data, declared when entering a subroutine, can be very large.
For example, in the listing 3, the parameter maxvlst, which represents the maximum number of neighbours in van der Waals pair list, is set to 2560. • Arrays used in a loop are listed close to each others in modules and in the same order of utilization as much as possible, to reduce the time for memory operations. For arrays declared in the subroutines (outside of modules), with shapes depending on the size of the MS, we calculated the next multiple of 16 bigger than the size, and used it as the dimension of the array. Although the setup z implies an over-consumption of memory for arrays with MS-dependent sizes, this is almost not noticeable, because we add at worst 15 elements to the size, which goes usually from thousands to millions in Tinker-HP. Almost all P-loops have been removed this way.
Vectorized loops in Tinker-HP We have 2 kinds of vectorized loops in Tinker-HP : • selection loops that select sites using various cutoffs and work on integers or logicals • compute loops that compute quantities and work on reals. We will show each of them in details, and give some insights on how we have improved the vectorization. The typical loops have been extracted from ehal1vec, which use the module MOD_vec_vdw shown in listing 3.
NUC hotspots The Real CPU Time is almost the same as for the Rel version. We can see a reduction of about 10% for the real CPU time of vmlinux. The libmkl_vml_avx512. so shared library has appeared, because we use vectorized mathematical functions and replaced all calls to the complementary error function erfc, which was in the sources of Rel, by calls to vderfc, which is a vectorized implementation in Intel MKL library.
Sequential performance We have evaluated the overall performance boost due to Intel AVX-512 by running calculations on only one core from a dedicated node. In this situation, we can easily measure the execution time of each interesting part of the code, with limited perturbation from the MPI portions of the code or the presence of other processes. We chose to measure three execution times : 1. time_vdw, which is the time taken by Van der Waals calculations. Depending on the setup, we used ehal1(vec) or elj1(vec). 2. time_elec, which is the time taken by electrostatic calculations. Depending on the setup, we used echarge1(vec) (direct charges) + reciprocal charges or empole1(vec) (direct multipoles) + reciprocal multipoles + torque(vec2). 3. time_polar, which is the time taken by polarization calculation and by far the biggest.
The boosts we obtained are significant and justify the important vectorization efforts we made to get them. The real performance gain should be estimated in a more realistic situation, where Tinker-HP is running in parallel. In this case, there could be 8 to 48 processes running on one  We focus here on the absolute performance improvements over previous published results. As vectorization did not affect the scaling of the methods, interested readers can refer to the earlier Tinker-HP software publication for a detailed analysis of scalability  [1]. Here, we ran calculations of 2 000 steps (4ps), with the core setups shown in table 1. The best performance was taken as the average of the 20 performance evaluations made by Tinker-HP during the run, after removing the first, middle and last values, which are lower because of the writing of intermediate files at these steps.
Conclusion In many ways this work represents a fundamental step in the evolution of the Tinker-HP software. First, it demonstrates that new HPC architectures can offer significant acceleration to an existing massively parallel code like Tinker-HP. A brute performance boost factor between 1. 32 and 1. 70 can be achieved on computationally intensive parts of the code, leading to an overall acceleration factor between 1. 45 and 1. 59 for AMOEBA (1. 24   simulation of molecular systems with millions of atoms. Considering that the many current calculations require a total simulation time of a several microseconds, such a speed gains represent major progress. Second, it shows that improved speed is not just available from raising the frequency of the CPU or buying more powerful computers. Large accelerations such as those reported here involve a close cooperation between the computational chemists, who write the code, and HPC specialists, who know how the CPU and the system software work.
To get these gains, we had to dig into the pieces of code that were the most CPU consuming and to rewrite them almost completely, with simplicity and efficiency in mind. But it was worth the effort. Furthermore, considering the trends observed with prior CPUs, we anticipate that vectorization will also play an important role in future architectures. Third, this work gives us a strategy and some methods to further improve the code. It can serve as a solid starting point for the future. We are now able to more easily adapt Tinker-HP to new underlying hardware or software advances. That will allow us to make the best of new technologies. Of course, optimization is far from finished as some parts of the code are not yet vectorized (for example the reciprocal space computations involved in permanent electrostatics and polarization), and other sources of possible speedups exist and will be investigated.
In particular, we have to review how we can improve the creation of neighbour lists, implement faster indexing of all the atoms (sorting indexes could be a solution) and achieve better masked MPI communications within computations. Decreasing precision is also possible in specific cases to gain performances while retaining sufficient accuracy. This paper will continue to be updated as we accumulate new data on Github, until a new version of this living document is pushed to review. The next iteration of the paper will also incorporate results on next-generation of Intel Xeon Scalable processors (codenamed Cascade Lake), and attempt to evolve towards an adaptation of the Tinker-HP code to any future architectures proposed by Intel. Future work will focus on the algorithmic boosting of our initial implementation of classical non-polarizable force fields.
Acknowledgments We want to give special thanks to David Guibert and Cedric Bourasset (Center for Excellence in Parallel Programming, Atos-Bull) for initial discussions. LHJ wishes to thank Prof Esmaïl Alikhani (Sorbonne Université) for sharing his 2x18 cores Skylake computer in the early phase of this work. LHJ also wants to thank Bernard Voillequin for fruitful discussions at the very beginning of this work. B. Voillequin is not a scientist, and not even an engineer. But he's a man of good sense and good will. That is invaluable.
We analyzed the intervalcensored time-to-event (i. e. , death) data for the three experimental groups by computing nonparametric maximum likelihood estimates (NPMLE) of the survival curves for each group, implemented in the R package interval  65. The estimation procedure takes into account uncertainty introduced by the 3-day observation period. Bootstrap with 10,000 repetitions was applied to quantify median survival time and standard deviation. Mathematical modeling. To model the expected performance of pgSIT at suppressing local Ae. aegypti populations in comparison with currently available selflimiting suppression technologies-RIDL, fsRIDL, and IIT-we simulated release schemes for each using the MGDrivE simulation framework  43  (https://marshalllab. github. io/MGDrivE/). This framework models the egg, larval, pupal, and adult mosquito life stages (both male and female adults are modeled) implementing a daily time step, overlapping generations and a mating structure in which adult males mate throughout their lifetime, while adult females mate once upon emergence, retaining the genetic material of the adult male with whom they mate for the duration of their adult lifespan.
Density-independent mortality rates for the juvenile life stages are assumed to be identical and are chosen for consistency with the population growth rate in the absence of density-dependent mortality. Additional density-dependent mortality occurs at the larval stage, the form of which is taken from Deredec et al. 66. The inheritance patterns for the pgSIT, RIDL, fsRIDL, and IIT systems are modeled within the inheritance module of the MGDrivE framework  43  , along with their impacts on adult lifespan, male mating competitiveness, and pupatory success. We implement the stochastic version of the MGDrivE framework to capture the random effects at low population sizes and the potential for population elimination. We simulated weekly releases over a period of 6 months into a randomly mixing population consisting of 10,000 adult females at equilibrium, with Ae.
Acknowledgements This work was supported in part by funding from  NIH  grants including  NIH - K22 Career Transition award  ( 5K22AI113060 ), an  NIH   Exploratory/Developmental Research Grant Award  ( 1R21AI123937 ) awarded to O. S. A. , and a  Defense Advanced Research Project Agency (DARPA)   Safe Genes Program Grant  ( HR0011-17-2-0047 ) awarded to O. S. A. and subcontracted to J. M. M. We thank  Ethan Bier  ( UCSD ),  Anthony A. James  ( UCI ), and  Craig Montell  ( UCSB ) for their insightful discussions, comments, and edits.
Experiment1. Participants. A group of 16 healthy, right-handed subjects (8 males, mean age = 24. 0 years SD = 2. 8) participated in the experiment. Subject numerosity was established based on the effect size reported in previous works addressing the effect of multisensory illusions on self and social perception  7, 38  , using the G*Power software -version 3. 1. 9. 3  39  -to achieve a power equal or above 0. 75. All of the subjects reported no history of drug abuse, no neurological and psychiatric disorders. When engaged in the experiment, the subject was naive to the actual aim of the study and gave her/his informed consent to participate to the study. The study was performed under approval (Authorization No. 10028) from the Rhône-Alpes Préfecture review board authorizing biomedical research at the Stem Cell and Brain Research Institute.
Four types of questions were presented to the subject (1) on the feeling that her/his face became the robot face (enfacement: 6 items), (2) on the feeling to be at the location of the robot (location: 5 items) and (3) on the feeling to control the robot's motion (agency: 7 items). This test was presented after each of the experimental beaming conditions (i. e. Static, Correlated). The embodiment statements are detailed in the Supplementary Material (SM2). Data analysis: The analysed variables were the Likeability score, the IIS score, the IOS score and the Enfacement, Location, Agency scores. The Likeability score was calculated as the percentage of the maximal score of the subjective scale (100). The IIS score was assessed by attributing the value 1 and 0 to the positive and negative responses respectively.
The IOS score corresponded to the value of the degree of closeness chosen by the subject. The Embodiment rating was evaluated for each category (Enfacement, Location, Agency) by averaging the percentage of the maximal score of the subjective scale (100) obtained at each item. Statistical analyses were realised with Statistica software package (TIBCO Software Inc. , CA 94304-USA). Each variable was analysed for the normal distribution using the Kolmogorov-Smirnov test and for variance homogeneity using the Levene's test. As all the variables distribution with the exception of the IIS score met the assumptions of normality and variance homogeneity at p > 0. 05, we used statistical parametric tests, including the analysis of variance (ANOVA) for Likeability, Closeness and Embodiment scores. For Likeability, we used a mixed repeated measures ANOVA design with two within-subject factors as Beaming Experience (Pre, Post) and Robot State (R-State: Static, Correlated) and one between subject factor Robot Type, identifying which robot was experienced as synchronous/correlated (R-Type: Reeti, iCub).
Usually, the chip is placed on a motherboard allowing the data to be sent to physical output terminals (USB, Bluetooth, radio, etc. ). The antenna and motherboard are the hardware part of the GNSS receiver. 3. An interface for data processing, recording and exploitation. In this study, it will be an Android device. Communication with the receiver depends on the user's choices and the specificity of the motherboard selected. Figure  1  : Hardware components of a GNSS receiver The output measurement quality is determined in the same time by the antenna and the chip. These two components work together and must be chosen in order to fully exploit their functionalities. One of the first models of low-cost chips cited in the literature is the Lea-4t in  (Takasu, Yasuda, 2009). Commercialized by U-Blox, it provides a good price/performance ratio for a single-frequency chip.
Analyse of the raw data The objective of this first test is to evaluate the performance of the low-cost receiver based on the study of raw measurements. To quantify and compare the results, we performed the same measurements with a professional Trimble R10 antenna during 15 minutes. To post-process our measurements, we used the RTKLib software which includes the conversion from UBX to Rinex format. For this experiment, the receiver was placed on a roof to provide optimal measuring conditions. The two GNSS receivers were placed side by side and observed simultaneously. The signal-to-noise ratio is used to qualify the quality of the measurement processing before and after signal amplification. A lower SNR ratio means that the signal processing quality is poorer.