# Implementation of the SoftMax Activation for Reconfigurable Neural Network Hardware Accelerators
## 3 Implementation of Neural Networks in RCE  
As previously noted, an RCE is a general design paradigm for computing systems with high parallelism and reconfiguration ability. The capabilities and characteristics of such systems essentially depend on the predefined set of PE operations, as well as the interaction between their elements. Consequently, there are many ways to implement the required algorithms. This paper explores the implementation of neural networks in an RCE that supports fine-grained reconfiguration and highly distributed computing. In accordance with our architecture, an artificial neuron can be implemented in the form of a processing elements chain (Figure 2). Separate elements perform different operations, such as storing the bias, weighting and adding each input, and applying the activation. Therefore, neurons with any number of inputs can be implemented by altering the length of this chain. Obviously, the size and complexity of an RCE processing element is highly dependent on the number and complexity of operations it supports. Thus, complex operations, such as the sigmoid or SoftMax activations, should be decomposed into simple, reusable ones. In other words, the complex operations should be implemented by a collaboration of several processing elements. The literature review allowed us to identify the following set of operations: \"signal source\" (SRC), \"signal transfer\" (TRS), \"multiply and accumulate\" (MAC), \"parametric ReLU\" (PRL) \"maximum\" (MAX), \"minimum\" (MIN), \"gate\" (GAT), \"union\" (U), \"delay\" (DEL), and \"block\" (BLK) (Figure 3). These operations are sufficient for implementing the key layers of neural networks (dense, convolution, pooling) as well as activations (sigmoid, tanh, ELU, and other) [21, 23, 24, 25]. This paper focuses on the implementation of SoftMax activation. Certain operations require an additional argument, which can be set during the configuration process and stored in the internal memory of the PE. For instance, the MAC operation treats this argument as a weight value, while the PRL employs it for its coefficient. Furthermore, each PE is configured with one of four possible signal flow directions. As will be demonstrated later, the direction setting enables the implementation of the pipeline processing in an RCE. Figure 1: Reconfigurable computing environment. By utilizing the proposed approach, the layers of a neural network can be implemented by connecting several chains (Figure 4). The values \(b1\), \(b2\), \(b3\) correspond to the neuron biases. Neuron weights are stored in the elements that are configured for the MAC operation (depicted as orange boxes). The symbol \(f\) denotes an activation function. Since all processing elements work independently, all neurons of the layer are computed in parallel, resulting in enhanced performance. As is evident from Figure 4, the proposed implementation of the neuron rotates the signal by 90 degrees. This rotation serves two purposes: to stack the neuron chains together and to compute multiple layers concurrently. The simultaneous processing of several neural network layers can be accomplished through pipelining. To process \(n\) layers, the RCE needs to be divided into \(n+1\) segments. At each moment of time, \(n\) segments implement different layers while the last segment is reconfigured to compute the next layer (Figure 5). Each segment processes signals that arrive at different points in time and passes the results to  
Figure 4: Dense layer in RCE. Figure 3: Processing element operation set. Figure 2: Implementation of a neuron with RCE processing elements. Reprinted from [23], with permission 2021 IEEE. the following segment, which is configured for the next layer of the network. At each step, the reconfiguration process is performed for the subsequent segment. Thus, the full reconfiguration of the RCE requires \(n\) steps. Through the simultaneous computation and dynamic reconfiguration, this implementation of RCE is capable of processing NNs with an arbitrary number of layers.