# Runtime Construction of Large-Scale Spiking Neuronal Network Models on GPU Devices
## 2 Materials and Methods
### Hardware and Software of Performance Evaluation  
As a reference, we implement the proposed method for generating connections directly in GPU memory in the GPU version of the simulation code NEST. In the following, NEST GPU (_onboard_) refers to the new algorithm in which the connections are created directly in GPU memory, while NEST GPU (_offboard_) indicates the previous algorithm, which first generates the network in CPU memory and subsequently copies the network structure into the GPU as demonstrated in [9; 13]. For a quantitative comparison to other established codes, we use the CPU version of NEST [1] (version 3.3 [34]) and the GPU code generator GeNN [6] (version 4.8.0 ([[https://github.com/genn-team/genn/releases/tag/4.8.0](https://github.com/genn-team/genn/releases/tag/4.8.0)]([https://github.com/genn-team/genn/releases/tag/4.8.0](https://github.com/genn-team/genn/releases/tag/4.8.0)), accessed on 14 August 2023)).  
We evaluate the performance of the alternative codes on four systems equipped with NVIDIA GPUs of different generations and main application areas: two compute clusters, JUSUF [35] and JURECA-DC [36], both using CUDA version 11.3 and equipped with the data center GPUs V100 and A100, respectively, and two workstations with the consumer GPUs RTX 2080 Ti, with CUDA version 11.7 and RTX 4090 with CUDA version 11.4. The NEST GPU and GeNN simulations discussed in this work each employ a single GPU card, both because the novel network construction method developed for NEST GPU is limited to single-GPU simulations and also because all the simulation systems employed have enough GPU memory to simulate the models previously described using a single GPU card. The CPU simulations use a single compute node of the HPC cluster JURECA-DC and exploit its 128 cores via 8 MPI processes each running 16 threads. Table 1 shows the specifications of these three systems.  
For the network models, their specific implementations were taken from the original source for each simulator in the case of the cortical microcircuit model. In particular, both NEST ([[https://github.com/nest/nest-simulator](https://github.com/nest/nest-simulator)]([https://github.com/nest/nest-simulator](https://github.com/nest/nest-simulator)), accessed on 14 August 2023) and NEST GPU ([[https://github.com/nest/nest-gpu](https://github.com/nest/nest-gpu)]([https://github.com/nest/nest-gpu](https://github.com/nest/nest-gpu)), accessed on 14 August 2023) provide example implementations of the cortical microcircuit model inside their respective source code repositories. Additionally, GeNN provides their own implementation of the microcircuit along with the data used for their PyGeNN publication [17] in the corresponding publicly available GitHub repository ([[https://github.com/BrainsOnBoard/pygenn_paper](https://github.com/BrainsOnBoard/pygenn_paper)]([https://github.com/BrainsOnBoard/pygenn_paper](https://github.com/BrainsOnBoard/pygenn_paper)), accessed on 14 August 2023). Furthermore, to correctly compare the performance of the simulation, we adapt the existing scripts so that the overall behavior remains the same. In particular, we disabled spike recordings and we enabled optimized initialization of membrane potentials as in [26].  
The second model presented in Section 2.4 is implemented for NEST GPU (_onboard_), and different connection rules can be chosen for the simulation. See the Data Availability Statement for further details on how to access the specific model versions used in this publication.