# Implementation of the SoftMax Activation for Reconfigurable Neural Network Hardware Accelerators
###### Abstract  
In recent decades, machine-learning algorithms have been extensively utilized to tackle various complex tasks. To achieve the high performance and efficiency of these algorithms, various hardware accelerators are used. Typically, these devices are specialized for specific neural network architectures and activation functions. However, state-of-the-art complex autonomous and mobile systems may require different algorithms for different tasks. Reconfigurable accelerators can be used to resolve this problem. They possess the capability to support diverse neural network architectures and allow for significant alterations to the implemented model at runtime. Thus, a single device can be used to address entirely different tasks. Our research focuses on dynamically reconfigurable accelerators based on reconfigurable computing environments (RCE). To implement the required neural networks on such devices, their algorithms need to be adapted to the homogeneous structure of RCE. This article proposes the first implementation of the widely used SoftMax activation for hardware accelerators based on RCE. The implementation leverages spatial distribution and incorporates several optimizations to enhance its performance. The timing simulation of the proposed implementation on FPGA shows a high throughput of 1.12 Gbps at 23 MHz. The result is comparable to counterparts lacking reconfiguration capability. However, this flexibility comes at the expense of the increased consumption of logic elements. deep neural networks; hardware accelerators; low-power systems; homogeneous structures; reconfigurable environments; parallel processing +
Footnote †: journal: _sciences_  
them are based on devices that support parallel computing: graphics processing units (GPU), field-programmable gate arrays (FPGA), and application-specific integrated circuits (ASIC) [2; 8; 9; 10; 11; 12]. GPU consists of numerous small processing units working independently of each other; consequently, it ensures high performance in machine-learning tasks [1; 9]. Nevertheless, the high power consumption of GPUs limits their use primarily to cloud, server, and desktop systems. FPGAs and ASICs are more widespread in low-power and autonomous devices. They allow one to develop a computing unit with an architecture customized for a particular task. The customizations and low-level implementation ensures the high performance and power efficiency of such accelerators. To use the benefits of the chosen platform, the required algorithms need to be adapted to its peculiarities. In addition to the specialized hardware, numerous optimizations are employed. Due to the redundancy of neural networks, some weights can be eliminated without any significant loss [13]. The high sustainability of NNs to distorted input data allows one to reduce the bit depth of both input data and parameters [14; 15; 16; 17; 18]. These approaches exhibit enhanced performance and efficiency for specific tasks. However, complex intelligent systems encounter various challenges, each of which may require an NN model with a specific architecture. Furthermore, in numerous cases, there is a lack of prior knowledge regarding the algorithms that will be required in the future. This issue is particularly relevant for remote systems with low availability, where limitations in power consumption, weight, and reliability make it challenging to carry multiple task-specific devices for any possible task. A more promising solution involves hardware accelerators with the reconfiguration ability [19; 20; 21; 22]. Such accelerators can drastically change the implemented NN model by an external signal at runtime. At the same time, reconfigurable accelerators offer comparable performance, energy efficiency, and reliability to their rigid counterparts. These features are of primary relevance to engineers, leading to numerous studies in this field. The concept of reconfigurable computing environments (RCEs) can be utilized to build hardware accelerators with such capabilities.