Circuit Design of a Seven-piecewise Linear  Activation Function

Authors: ren  cai
le  yang
sai  li
Venue: IEEE_org_ICIST_2024_Conference
Published: 2024-07-22 08:28:29

Abstract: Nonlinear activation function is a type of function that operates within artificial neural networks, introducing nonlinearity into the network, which enables the network to be applied to a variety of nonlinear models. In the design of nonlinear activation function circuits for memristive neural networks, the main approach involves using three-piecewise linear activation functions to approximate the sigmoid and tanh functions. This paper proposes a seven-piecewise linear activation function to fit the sigmoid and tanh functions. The circuit is divided into three modules: Signal sending module converts the input voltage signal into line segments with different slopes; Signal processing module is used to resolve the issue of voltage signal discontinuities. The output voltage from Signal processing module is then processed by Signal output module for final output, which is subsequently transformed into sigmoid and tanh functions. PSPICE simulation was used to verify the correctness of the design. Subsequently, the proposed seven-piecewise linear activation function was incorporated into an iris classification network. The effectiveness of the design was demonstrated by the recognition rate of the iris classification task.

Main paper: 
 i. introduction: 
 Artificial neural networks are a type of mathematical model based on algorithms that mimic the characteristics of biological neural networks to carry out distributed and parallel information processing [1] . In recent years, with the rapid advancement of technology, artificial neural networks have been applied in numerous fields of life [2] - [4] . Currently, artificial neural networks primarily operate on devices with the von Neumann architecture [5] , where memory and computation are separate. This leads to the need for frequent data access during processing, which significantly reduces efficiency and also results in higher power consumption. Against this backdrop, finding ways to break through the bottleneck of the von Neumann architecture and achieve lowpower, highly parallel computing similar to brain-like intelligence has become increasingly important [6] , [7] . In 1971, Professor Leon O. Chua from the University of California, Berkeley, proposed the concept of a memristor based on the theory of circuit symmetry [8] . It is considered to be the fourth fundamental circuit element, in addition to the resistor, capacitor, and inductor. In 2008, Strukov at HP Labs first fabricated a physical memristor based on TiO2 [9] . Due to the nanoscale size, non-volatility, and integrated memory and computation features of memristors [10] - [12] , they are considered an effective method for addressing the von Neumann bottleneck of separated memory and computation. Since then, the application of memristors in artificial neural networks has become increasingly widespread [13] .In artificial neural networks, if nonlinear activation functions are not used, the network's output will only perform a linear transformation on the input, which greatly limits the network's ability to process matters [14] . Nonlinear activation functions can introduce nonlinear factors into the network, playing a crucial role in the learning and processing of complex nonlinear matters within artificial neural networks. Therefore, it is essential to incorporate nonlinear activation functions into artificial neural networks. Commonly used nonlinear activation functions include the sigmoid function and the tanh function. Since their expressions involve exponential operations, designing circuits to implement their functional graphs is relatively difficult [15] .Binary activation functions are a common method for implementing nonlinear activation functions in circuits [16] . The principle is that when the input is greater than the threshold, the neuron turns on; otherwise, it turns off. When using binary activation functions, it is necessary to simplify the algorithm and the network, which weakens the expressive ability of the network. Additionally, some scholars have also utilized computer software to implement sigmoid and tanh functions [17] , [18] , but this method mostly relies on computers and microcontrollers for realization. Moreover, this scheme requires high-energy-consuming A/D and D/A converters, which is not conducive to on-chip implementation. In order to better implement the sigmoid and tanh functions, literature [19] , [20] proposes a three-piecewise linear activation function to approximate the sigmoid and tanh functions. However, this method cannot accurately fit the sigmoid and tanh functions. To the best of the author's knowledge, no scholar has yet proposed a piecewise linear activation function circuit design with more than three segments. This study introduces a sevenpiecewise linear activation function circuit, which, compared to the three-piecewise linear activation function, can more accurately approximate the sigmoid and tanh functions.The remainder of the paper is structured as follows: Section II introduces the fitting method. Section III presents the circuit and simulation of the seven-piecewise linear activation function fitting the sigmoid function. Section IV introduces the circuit and simulation of the seven-piecewise linear activation function fitting the tanh function. Section V constructs an iris classification network. Section VI summarizes the entire paper.In artificial neural networks, the widely used nonlinear activation functions include the sigmoid function and the tanh function. Taking the sigmoid function as an example, due to the continuously changing slope of the sigmoid function, it is difficult to design circuits to directly implement the sigmoid function curve. To this end, this paper designs a seven-piecewise linear activation function to approximate the sigmoid function. Compared to the three-piecewise linear activation function, the seven-piecewise linear activation function can better capture the characteristics of the sigmoid function. The fitting line segments are divided into seven parts: the first segment is a line segment with an output of 0, the second segment has a slope of a1, the third segment has a slope of a2, the fourth segment has a slope of a3, the fifth segment has the same slope as the third segment, the sixth segment has the same slope as the second segment, and the seventh segment is a line segment with an output of 1. , R 1 =10KΩ, R 2 = R 4 =20KΩ, R 3 =2.85KΩ, R 5 =R 6 =1KΩ,C1=20nf, its function is to eliminate the self-oscillation of OP1. V CON 1 and V CON 2 are two control voltages. The circuit workflow is as follows: 2) When V IN is greater than the threshold of T1, the subsequent circuit has voltage input. If both V CON 1 and V CON 2 are at a low level, V IN becomes a negative voltage after being amplified by OP1 and then becomes a positive voltage after passing through OP2. At this time, the slope of the output voltage U1 is the amplification factor of OP1, which is the ratio of R 4 to R 1 . This corresponds to line segment 2 with slope a1 in Fig 1 (a) .3) As V IN continues to increase, V CON 1 becomes high level, and the amplification factor of OP1 changes to the ratio of R 4 in parallel with R 1 and R 2 . The slope of the output voltage U1 increases, which corresponds to line segment 3 with slope a2 in Fig 1 (a) . When the slope of U1 changes from a1 to a2, the output voltage U1 will jump upwards. 1) When V IN is less than the T1 threshold voltage, U2 outputs 0, corresponding to the first segment of the sevenpiecewise linear activation function during this stage.2) When S3-S9 are all turned off, U1 is input from port 2 of W1 and output from port 5, and then input from port 2 of W2 and output from port 5, finally, it goes through P5 and ABM1 outputs, corresponding to the second segment of the seven-piecewise linear activation function.3) When S3 is turned on, U1 passes through the subtractor P1, subtracts the fixed voltage V 4 , and then input from port 3 of W1 and output from port 4, finally outputting from ABM1. This corresponds to the third segment of the seven-piecewise linear activation function.4) When S3, S4, and S5 are turned on, ports 3 and 4 of W2 are connected, resulting in an output of 0 from W2. U1 passes through the subtractor P2, subtracts the fixed voltage V 1 5, and then input from port 3 of W3 and output from port 4 to P5, finally outputting to ABM1. This corresponds to the fourth segment.5) When S3, S4, S5, S6, and S7 are all open, the output voltages of W2 and W4 are both 0V. U1 passes through the adder P3, adds the fixed voltage V 6 , and then input from port 3 of W5 and output from port 4 to P6, finally outputting to ABM1. This corresponds to the fifth segment.6) With S3-S9 all open, the outputs of W2, W4, and W6 are all 0. U1 passes through P4, adds the fixed voltage V 7 , and then input from port 3 of W7 and output from port 4 to P6, finally outputting to ABM1. This corresponds to the sixth segment of the seven-piecewise linear activation function.7) The operating voltage of W1-W7 is set at [-20V,20V]. As V IN continues to increase, U2 will remain at 20V. Between 0-1s, V IN is always less than 1V, T1 is disconnected, the subsequent circuit has no input, and ultimately V out is 0.Between 1-2s, V IN is greater than 1V, V IN is connected to the subsequent circuit, V T 1 is the source voltage of MOSFET T1, and the output voltage after amplification and conversion by OP1 and OP2 is:V OP 2 = - R 6 R 5 (- R 4 R 1 V T 1 ) = 2V T 1 (1)In the signal processing module, S1-S9 are all turned off, so the IN1 segment input of ABM1 is 2V T 1 , and the IN2 segment input is 0. The output of ABM1 is:EQUATION)The signal is then sent to the signal output module for output. The relationship between the circuit's output and V T 1 is as follows:EQUATIONBetween 2-3s, during this phase, V CON 1 =3V, S1 is turned on, and the output of OP2 is:EQUATIONIn the signal processing module, with S3 opened, the ports 3 and 4 of W1 are connected, and the output voltage of ABM1 is:EQUATIONThe relationship between the final circuit output and V T 1 is as follows:EQUATIONBetween 3-4s, both V CON 1 and V CON 2 are at 3V, S1 and S2 are both activated, and the output voltage of OP2 is:EQUATIONIn the signal processing module, with S3, S4, and S5 conductive, the output voltage of ABM1 is:EQUATIONThe relationship between the final circuit output and V T 1 is:EQUATIONBetween 4-5s, V CON 2 returns to 0V, the switch S2 is turned off, and the output voltage of OP2 is:EQUATIONIn the signal processing module, with S3, S4, S5, S6, and S7 all open, the output voltage of ABM1 is:EQUATIONThe final output of the circuit is:EQUATIONBetween 5-6s, V CON 1 becomes 0V, S1 is closed, and the output voltage of OP2 is:EQUATIONIn the signal processing module, with S3-S9 all open, the output voltage of ABM1 is:EQUATIONThe final output of the circuit is:EQUATIONBetween 6-7s, due to the compression of the operational amplifier, the final output of the circuit will remain at 1V. The circuit simulation of seven-piecewise linear activation function fitting the sigmoid function is shown in Fig 8 . The simulation results are consistent with the theoretical analysis, which proves the correctness of the circuit. EQUATIONFig. 9 : The circuit for the signal output module in a sevenpiecewise linear activation function circuit that fits the tanh function.The signal sending module and the signal processing module that fit the tanh function are the same as those in Fig 3 and To verify the effectiveness of this circuit in memristive neural networks, an iris classification network was constructed . The network consists of two layers, with the number of neurons in the input layer, hidden layer, and output layer being 4, 8, and 1, respectively. The initial weights of the network are between 0 and 0.25. We incorporated both the three-piecewise linear activation function and the seven-piecewise linear activation function fitting the sigmoid function into the network as activation functions. The experiment was repeated ten times, and the final results were taken as the average of the ten runs. The experimental results show that the iris recognition rate of the network using the three-piecewise linear activation function is 80.22%, while the iris recognition rate of the network using the seven-piecewise linear activation function is 94%. The experiment demonstrates that the performance of the network using the seven-piecewise linear activation function is superior to that of the network using the three-piecewise linear activation function. To test the stability of the network using the seven-piecewise linear activation function, we added 5% and 10% noise to the network. The training errors are shown in Fig 11,  which proves that the network has good robustness against noise.In order to test the impact of circuit component errors on circuit performance, we conducted a Monte Carlo analysis on the seven-piecewise linear activation function circuit for fitting the sigmoid function. We set the tolerances of R In practical circuits, temperature can affect the parameter values of electronic devices. To simulate the impact of temperature on the circuit proposed in this paper, a temperature analysis was conducted on the circuit shown in This paper constructs a seven-piecewise linear activation function circuit for memristive neural networks, which can fit the sigmoid function and the tanh function, respectively. Firstly, the circuit structures and working processes of the three modules in the seven-piecewise linear activation function circuit for fitting the sigmoid function are introduced, followed by simulations of the overall circuit. Subsequently, a bias circuit is added to the seven-piecewise linear activation function circuit that fits the sigmoid function, which enables the seven-piecewise linear activation fitting of the tanh function. Subsequently, to verify the effectiveness of the seven-piecewise linear activation function, we constructed an iris classification network. We incorporated both threepiecewise and seven-piecewise linear activation functions into the network separately. In the end, we found that the network using the seven-piecewise linear activation function achieved a higher recognition rate for iris flowers. Finally, we conducted Monte Carlo analysis and temperature analysis on the sevenpiecewise linear activation function fitting the sigmoid function. The simulation results indicate that the circuit has good stability. 
 
 Appendix:  



Reviews: review: title: title: This paper proposes a seven-piecewise linear activation function to fit the sigmoid and tanh functions. The proposed strategy is attractive for an iris classification network.
main_review: main_review: 1.Abstract section does not describe the problem and motivation clearly.
2.It is recommended to increase the comparison of results to highlight the advantages of research methods.
3.Designing nonlinear activation function circuits for memristive neural networks requires a multidisciplinary approach, integrating insights from circuit design, materials science, and neural network theory to develop effective and reliable activation function circuits for memristive neural networks. How do the authors consider this problem?
4.Grammatical errors, such as ‘In order to better implement the sigmoid and tanh functions, literature [19], [20] proposes a three-piecewise linear activation function to approximate the sigmoid and tanh functions.’ It is advisable to carefully review the entire text.
5.The ‘SEP’ in reference [1] and the ‘SEPT’ in reference [13] are inconsistent. It is advisable to carefully check.
title: title: The paper is interesting
main_review: main_review: 1. What motivated the choice of a seven-piecewise linear activation function over the commonly used three-piecewise linear functions? How does this design improve the approximation of the sigmoid and tanh functions compared to previous approaches?
2. The abstract mentions the use of the proposed activation function in an iris classification network. Could the authors provide more details on how the seven-piecewise linear activation function specifically impacted the recognition rate or overall performance compared to traditional activation functions used in similar tasks?
title: title: Accept, some mistakes should be revised
main_review: main_review: 1. The highlights of this paper should be added before the structural description in the Introduction.
2. The font size of the illustration should be smaller than the body of the paper.
3. The demonstrations of the test system in Section V is insufficient.