Abstract:
This invention relates to an artificial neural network (ANN), particularly to a neuron circuit and its activation function including the derivative. The neuron circuit capable of generating an adjustable sigmoid-like function and a good approximation of its derivative, comprises: a current generator for generating a current; a current-controlled transistor for changing an output voltage according to the current from the current generator; and at least one differential pair of transistors for generating the adjustable sigmoid-like function output and the good approximation of its derivative by the changed output voltage.

Description:
BACKGROUND OF THE INVENTION  
         [0001]    1. Field of the Invention  
           [0002]    This invention relates to an artificial neural network (ANN), particularly to a neuron component and its activation function including the derivative.  
           [0003]    2. Description of the Related Art  
           [0004]    One of the artificial neural networks&#39; most important components is the neuron, whose performance and complexity greatly affect the whole net. In many literatures, its activation function is found to be the sigmoid. Most of the analog implementations have used differential pair for generating the required function. In some cases, such as on-chip back-propagation learning, both a non-linear function and its derivative are required. Neurons that can meet the requirement have been presented. Most of these have voltage inputs and the current outputs. Since most of the applications employ current output synapses and voltage output neurons in order to enable the summation of those currents by simply connecting them together at the input of the neuron and to diverge signals from a neuron to a large quantity of synapses, these circuits a little bit inconvenient. In addition, some circuits are biased in the subthreshold region, so its driving capability is quite limited.  
         SUMMARY OF THE INVENTION  
         [0005]    Therefore, an object of the invention is to provide a novel neuron circuit, which has current input and voltage outputs and is built with strong-inversion biased transistors, thereby realizing both a neuron activation function and its derivative.  
           [0006]    Another object of the invention is to provide a novel neuron circuit, which enables threshold and gain factor adjustability and has high noise immunity.  
           [0007]    Another object of the invention is to provide a novel neuron circuit, which is very beneficial to on-chip learning neural networks.  
           [0008]    To realize the above and other objects, the invention provides a novel neuron circuit, which is very beneficial to on-chip learning neural networks. The neuron circuit comprises: a pair of transistors having a first common terminal connected to a first input voltage, a second common terminal connected to a first output, a first gate connected to a second input voltage, and a second gate connected to a third input voltage; a surface field-effect transistor having a first terminal and a third gate connected together to the first output, which is connected to a first current source, and a second terminal connected to a second current source; and a first differential pair of transistors, having a third common terminal connected to a third current source, a first gate connected to the first output, a second gate connected to a fourth input voltage, a first terminal connected to a first load to form a connecting point as a second output, and a second terminal connected to a second load; a second differential pair of transistors, having a fourth common terminal connected to the third current source, a third gate connected to the first gate of the first differential pair of transistors output, a fourth gate connected to a fifth input voltage, a third terminal connected to a third load to form a connecting point as a third output, and a fourth terminal connected to a fourth load.  
           [0009]    In the neural network, a neuron circuit design is the key point for the performance. In the invention with simple neuron circuit design, it can easily reach a requirement of errorless effect and decrease the time delay to within 1 ns. The circuit also has a large dynamic range and a high noise immunity using the adjustable threshold and gain factor. Therefore, these features make it quite fit for hardware realization besides its programmability. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0010]    The invention will become apparent by referring to the following detailed description of a preferred embodiment with reference to the accompanying drawings, wherein:  
         [0011]    [0011]FIG. 1 is a schematic diagram of a neuron circuit of the invention;  
         [0012]    [0012]FIG. 2 a  is a diagram of comparing a simulated neuron transfer function curve to a fitted sigmoid curve of FIG. 1;  
         [0013]    [0013]FIG. 2 b  is a diagram of a relative error curve of FIG. 2 a;    
         [0014]    [0014]FIG. 3 a  is a diagram of comparing a simulated derivative curve to the derivative curve of the simulated neuron of the invention;  
         [0015]    [0015]FIG. 3 b  is a diagram of a relative error curve of FIG. 3 a ; and  
         [0016]    [0016]FIG. 4 is a diagram of simulated neuron transfer function curves with different thresholds of the invention;  
         [0017]    [0017]FIG. 5 is a diagram of activation curves under different gains of the invention. 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0018]    To realize the above and other objects, the invention provides a novel neuron circuit with adjustable threshold and gain factor and high noise immunity, which is very beneficial to on-chip learning neural networks.  
         [0019]    Refer to FIG. 1, illustrating a neuron circuit of the invention. In FIG. 1, the neuron circuit comprises: a pair of transistors M 1 , M 2 , a surface field-effect transistor Ms, and at least one differential pair composed of identical transistors M 3 , M 4 .  
         [0020]    As shown in FIG. 1, a pair of transistors M 1 , M 2  having a first common terminal A and a second common terminal B, a surface field-effect transistor Ms having a first terminal S and a gate Gs concurrently connected to the output B of the pair M 1 , M 2 , and at least one differential pair composed of identical transistors M 3 , M 4 , which one gate G 3  of identical transistors M 3 , M 4  is connected to the output B of the pair M 1 , M 2  and the other D is connected to a fixed input voltage V ref2  or V ref2 +ΔV, together with the active loads to realize the actual sigmoidal shaped non-linearity.  
         [0021]    In the proposed neuron circuit schematic, as shown in FIG. 1, V out (1) outputs the sigmoidal transfer function. (V out (2)−V out (1)) realizes its approximate derivative. In the dash frame  1 , which functions as a current generator, V ref1  is carefully chosen so that both transistors M 1  and M 2  are working in their linear range. The formed linear resistor R AB  can be controlled by the gate voltage of both transistors V N  and V p . A surface field-effect transistor Ms, which is a current-controlled transistor, is connected to the output of the frame  1 , and its gate current source of the transistor Ms is used to adjust the output voltage of the neuron circuit, wherein the output voltage is the desired sigmoidal transfer function. In the case that the dash dot frame  2  outputs the desired sigmoidal transfer function while the dash dot frame  3  outputs its derivative, a simple differential pair composed of identical transistors, for example M 3  and M 4  of the frame  2 , together with the active loads, for example Acs of the frame  2 , realize the actual sigmoidal shaped non-linearity. One port of the differential pair is connected to point B and the other is connected to a fixed voltage V ref2  or V ref2 +ΔV. I ref1  and I ref2  are fixed current sources. The referenced current directions are shown as the arrows in FIG. 1.  
         [0022]    Assuming that M 3 , M 4  are operating in saturation and follow an ideal square law. V s  is the input differential voltage, i.e. V d =V B −V D , then  
                 I   d3          (     V   d     )       =       (       I   ref2     /   2     )     +     (       (     β                   V   d                             4        I   ref2       β     -     V   d   2             )     /   4     )               (   1   )                               
 
         [0023]    With  
         V   d0     ≡         2        I   ref2       β       ≤     V   d     ≤     -         2        I   ref2       β         ≡       V   d1     .                           
 
         [0024]    Here β is the transconductance parameter for transistors M 3  and M 4 . I AB =I in +I ref1 . When I in  is small, V d &gt;V d0 , V out (1) remains the low saturation voltage. As I in  increasing, V B  descends tardily and V out (1) increases slowly. When V d &lt;V 1 , V out  reaches and remains the high saturation level.  
         [0025]    To verify that it operates correctly, the proposed circuit is simulated with HSPICE (Highly Optional Simulation Program Integrated Circuit Emphasis) in the art, using level  47  transistor models for a standard 1.2 μm CMOS process. FIG. 2( a ) shows the neuron transfer function and its fitted sigmoid curve. Their relative error is not more than 3% as shown in FIG. 2( b ).  
         [0026]    Using the forward difference method, the approximate derivative voltage V deriv  is available by subtracting V out (1) from V out (2).  
           V   deriv   ≡ΔVV′   out   ≅V   out ( V   B ( I   in )− V   ref2   +ΔV )− V   out ( V   B ( I   in )− V   ref2 )= V   out (2)− V   out (1)   (2)  
         [0027]    Here ΔV is a fixed small value. FIG. 3( a ) shows the derivative found by simulation of the circuitry in FIG. 1 and the derivative of the simulated neuron transfer function. The solid line in FIG. 3( b ) shows that the relative error between them is less than 5%.  
         [0028]    The great power of an artificial neural network derives from its ability to be adapted to the unknown and changing environment. Therefore, good programmability is of fundamental importance. It is known that a sigmoid function can be given by  
           f ( X )=1/(1+exp(−α( X·W )+Θ))  
         [0029]    where X is the input matrix, W is the weight matrix, α is the gain factor and Θ is the bias weight vector. Different applications may need a difference between αand Θ. This can be realized by varying I ref1 , V N  and V p .  
         [0030]    When I ref1  changes, the bias weight also changes. As I ref1  increases, the current I in  needed to satisfy V B −V ref2&gt;V   d0  decreases, so the transfer curve shifts to the left. In the same way, when I ref1  decreases, the curve shifts to the right. FIG. 4 shows the simulated neuron transfer functions with different thresholds.  
         [0031]    When both transistors M 1  and M 2  are working in their linear range and their sizes are chosen in such a way that β 1 =β 2 , the relation between the current I AB  and V AB  can be written as  
           I   AB   =I   1   +I   2 =β 1   V   AB [( V   N   −V   T1 )−( V   p   −|V   T2 |) ]  (4)  
         [0032]    So,  
           R   AB =1/{β 1   V   AB [( V   N   −V   P )−( V   T1   +|V   T2 |)]}  (5)  
         [0033]    Equation 5 shows that the bigger (V N −V P ) is, the less R AB  is. That is, the less the slope of V B  versus I in  is. This means that V out1  increases more slowly, i.e. the gain factor is smaller.  
         [0034]    Different transfer functions with various gain factors are described in FIG. 5. Note that the saturation levels of the sigmoid remain constant for different gain values, in contrast to most implementations in the prior art. This ensures that for different gain values, the input range of synapse in subsequent layer is completely used.  
         [0035]    By utilizing the proposed neuron circuit that generates both an adjustable sigmoid-like function and a good approximation of its derivative has only a delay of within 1 ns in operation and speediness. Besides, the simple circuit composed of transistors as mentioned above, which works in the strong-inversion range, also has a large dynamic range and a high noise immunity. This is quite fit for hardware realization.  
         [0036]    Although the present invention has been described in its preferred embodiment, it is not intended to limit the invention to the precise embodiment disclosed herein. Those who are skilled in this technology can still make various alterations and modifications without departing from the scope and spirit of this invention. Therefore, the scope of the present invention shall be defined and protected by the following claims and their equivalents.