# Runtime Construction of Large-Scale Spiking Neuronal Network Models on GPU Devices
## 2 Materials and Methods
### Simulation Phases  
On the coarse level, we divide a network simulation into two successive phases: _network construction_ and _simulation_. The _network construction_ phase encompasses all steps until the actual simulation loop starts. To assess different contributions to the network construction, we further divide this phase into stages. The consecutively executed stages in the NEST implementations (both CPU and GPU versions) follow the same pattern as follows:  
1. _Initialization_ is a setup phase in the Python script for preparing both model and simulator by importing modules, instantiating a class, or setting parameters, etc. import nestgpu
2. _Node creation_ instantiates all the neurons and devices of the model. nestgpu.Create()
3. _Node connection_ instantiates the connections among network nodes. nestgpu.Connect()
4. _Calibration_ is a preparation phase that orders the connections and initializes data structures for the spike buffers and the spike arrays just before the state propagation begins. In the CPU code, the pre-synaptic connection infrastructure is set up here. This stage can be triggered by simulating just one time step \(h\). nestgpu.Simulate(h)  
Previously, the calibration phase of NEST GPU was used to finish moving data to the GPU memory and instantiate additional data structures like the spike buffer (cf. Section 2.3). Now, as no data transfer is needed and connection sorting is carried out instead (cf. Section 2.2), the calibration phase is now conceptually closer to the operations carried out in the CPU version of NEST [37]. In GeNN, the network construction is decomposed as follows:  
1. _Model definition_ defines neurons and devices and synapses of the network model. from pygenn import genn_model model = genn_model.GeNNModel() model.add_neuron_population()  
\begin{table} \begin{tabular}{c c c} \hline \hline
**System** & **CPU** & **GPU** \\ \hline JUSUF cluster & 2\(\times\) AMD EPYC 7742, 2\(\times\) 64 cores, & NVIDIA V100 1, 1530 MHz, 16 GB \\  & 2.25 GHz & HBM2e, 5120 CUDA cores \\ \hline JURECA-DC cluster & 2\(\times\) AMD EPYC 7742, 2\(\times\) 64 cores, & NVIDIA A100 2, 1410 MHz, 40 GB \\  & 2.25 GHz & HBM2e, 6912 CUDA cores \\ \hline Workstation 1 & Intel Core i9-9900K, 8 cores, & NVIDIA RTX 2080 Ti 3, 1545 MHz, 11 GB GDDR6, 4352 CUDA cores \\ \hline Workstation 2 & Intel Core i9-10940X, 14 cores, & NVIDIA RTX 4090 4, 2520 MHz, 24 GB GDDR6X, 16384 CUDA cores \\ \hline \hline \end{tabular}  1
[FOOTNOTE:1]Footnote 1: footnotetext: Volta architecture: [[https://developer.nvidia.com/blog/inside-volta](https://developer.nvidia.com/blog/inside-volta)]([https://developer.nvidia.com/blog/inside-volta](https://developer.nvidia.com/blog/inside-volta)), accessed on 14 August 2023. 2
[FOOTNOTE:1]Footnote 1: footnotetext: Volta architecture: [[https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth](https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth)]([https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth](https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth)), accessed on 14 August 2023. 3
[FOOTNOTE:3]Footnote 3: footnotetext: Turing architecture: [[https://developer.nvidia.com/blog/nvidia-turing-architecture-in-depth](https://developer.nvidia.com/blog/nvidia-turing-architecture-in-depth)]([https://developer.nvidia.com/blog/nvidia-turing-architecture-in-depth](https://developer.nvidia.com/blog/nvidia-turing-architecture-in-depth)), accessed on 14 August 2023. 4
[FOOTNOTE:4]Footnote 4: footnotetext: Ada Lovelace architecture: [[https://www.nvidia.com/en-us/geforce/ada-lovelace-architecture](https://www.nvidia.com/en-us/geforce/ada-lovelace-architecture)]([https://www.nvidia.com/en-us/geforce/ada-lovelace-architecture](https://www.nvidia.com/en-us/geforce/ada-lovelace-architecture)), accessed on 14 August 2023. 2. _Building_ generates and compiles the simulation code. model.build()
3. _Loading_ allocates memory and instantiates the network on the GPU.