# Runtime Construction of Large-Scale Spiking Neuronal Network Models on GPU Devices
## 2 Materials and Methods  
### Creation of Connections Directly in GPU Memory  
A network model is composed of nodes, which are uniquely identifiable by index and connections between them. In NEST and NEST GPU, a node can be either a neuron or a device for stimulation or recording. Neuron models can have multiple receptor ports to allow for receptor-specific parameterization of input connections. Connections are defined in NEST GPU (and similarly in other simulators) via high-level connection routines, e.g., ngpu.Connect(sources, targets, conn_dict, syn_dict), where the connection dictionary conn_dict specifies a connection rule, e.g., one_to_one, for establishing connections between source and target nodes. The successive creation of several individual sub-networks, according to deterministic or probabilistic rules, can then lead to a complex overall network. In the rules used here, we allow autapses (self-connections) and multi-tapses (multiple connections between the same pair of nodes); see [21] for a summary of state-of-the-art connectivity concepts. The basic structure of a NEST GPU connection includes the source node index, the target node index, the receptor port index, the weight, the delay, and the synaptic group index. The synaptic group index takes non-zero values only for non-static synapses (e.g., STDP) and refers to a structure used to store the synapse type and the parameters common to all connections of that group; it should not be confused with the connection group index, which groups connections with the same delay for spike delivery. Additional parameters may be present depending on the type of synapse, which is specified by the synaptic group. The delay must be a positive multiple of the simulation time resolution and can, therefore, be represented using time-step units as a positive integer. Connections are stored in GPU memory in dynamically-allocated blocks with a fixed number of connections per block, \(B\), which can be specified by the user as a simulation kernel parameter before creating the connections. It should be chosen on the basis of a compromise. If it is chosen too small, then the total number of blocks would be high, resulting in longer execution times. Conversely, if it is chosen too large, a significant amount of memory could be wasted due to incomplete filling of the last allocated block. The default value for \(B\) used in all simulations of this study is \(10^{7}\). Each time a new connection-creation command is launched, if the last allocated block does not have sufficient free slots to store the new connections, an appropriate number of new blocks is allocated according to the following formula:  
\[N_{\text{newblocks}}=\lfloor\frac{N_{\text{conns}}+N_{\text{newconns}}+B-1}{B} \rfloor-N_{\text{blocks}} \tag{1}\]  
where \(N_{\text{newblocks}}\) is the number of new blocks that must be allocated, \(N_{\text{blocks}}\) is the old number of blocks, \(N_{\text{conns}}\) is the old number of connections, \(N_{\text{newconns}}\) is the number of connections that must be created, and \(\lfloor x\rfloor\) denotes the integer part of \(x\). The new connections are indexed contiguously as follows:  
\[i_{\text{newconns}}=0,...,N_{\text{newconns}}-1. \tag{2}\]  
To understand our parallelization strategy we introduce two hardware-related terms, CUDA threads and CUDA kernels. CUDA threads are the smallest GPU computing units, these are grouped into blocks and several blocks are present in a multiprocessor unit. Thus, a thread block refers to a physical group of computing units, each unit having a unique index in the group. CUDA kernels are functions executed on the GPU device, these concurrently exploit multiple CUDA thread blocks. Henceforth, to avoid confusion with connection blocks, we will not mention CUDA thread blocks but refer to CUDA kernels, the functions employing said thread blocks, and CUDA threads as one of the computing units used by these kernels.