Abstract:
A voltage-mode pulse width modulation (PWM) VLSI implementation of neural networks, comprising: a voltage-pulse converter for converting an input voltage into a neuron-state pulse; a synapse multiplier, including a multiplier cell for multiplying the neuron-state pulse by an input weight voltage and an integral and summation cell for integrating and summing up the multiplied output and producing a first output voltage; and a sigmoid circuit for converting the first output voltage into a second output voltage with the non-linear activation function of neuron.

Description:
BACKGROUND OF THE INVENTION  
         [0001]    1. Field of the Invention  
           [0002]    This invention relates to a neural network, particularly to a VLSI implementation of neural networks. It shortens the convergence time and eliminates the influence of the initial value by using the GA features, thereby achieving the conservation of hardware resources.  
           [0003]    2. Description of the Related Art  
           [0004]    Because of its independence on specific model of signal processing, artificial neural networks (ANN) is now widely used in the fields of image processing and pattern recognition. Refer to FIG. 1. A multilayer neural network typically comprising an input layer, an output layer, and a hidden layer. The input layer having a row of cells  1 , . . . , z or neuron-like units (not shown) serves as an interconnect function, externally connecting a plurality of inputs x 1 , . . . , x n  to the network, wherein each input is coupled to a respective input cell. The hidden layer having a plurality of neuron cells  3 , . . . ,  4 , which is the most important part in the network and has multiple inputs and one output, is responsible for performing the specific functions of the network, wherein the output of each input cell in the input layer is coupled to each of a plurality of neuron cells in the hidden layer. For example, the specific functions can be that each neuron cell multiplies the inputs x 1 , . . . , x n  by a given weight w 1 , w 2 , . . . , w 3 , w 4  and adds the products together, as shown in  3  and  4  in the hidden layer. The output layer having at least one cell or neuron-like unit (not shown) serves as the output collection of a plurality of neuron cells  3 , . . . ,  4  in the hidden layer. For such a network, although now the majorities of neural systems are realized by software simulations, only hardware implementations can fully utilize its advantages of parallel processing and error tolerance. Among the existing 3 methods of VLSI implementation for neural systems, analog circuit has the merits of small chip area, fast speed and low power consumption, but is susceptible to process parameters and interface noises. Digital circuit has the merits of high precision and robustness, but it is area and power consuming. The digit-analog hybrid method—pulse stream technique combines both the merits of analog and digital methods. It uses digital signal to control analog multiplication, so the area of synapse analog multiplier is small and the neuron digital state signal is immune to noise. Among the various kinds of pulse modulation techniques, pulse width modulation (PWM) is the easiest for circuit implementation, so the PWM VLSI neural network is widely studied.  
           [0005]    Although the first proposed structure of 3-transistor synapse multiplier is simple, it has low precision and small linearity range because of the switching noise and working states of the transistors. Furthermore, it uses external non-linear voltages for activation function, which is not suitable for VLSI implementation because of the limited interface. A low gain 2-transistor amplifier is adopted for activation function, but its transfer characteristic is quite different from the ideal sigmoid function, so it is not suitable for ANNs which use Back Propagation (BP) learning algorithm. Although the transfer characteristic of asynchronous neuron proposed is very similar to ideal sigmoid function, it needs complex BiCMOS process. Another novel neuron circuit can easily realize both sigmoid function and its derivative, but its current input is not suitable for voltage-mode sigmoid circuit.  
         SUMMARY OF THE INVENTION  
         [0006]    Therefore, an object of the invention is to provide a new building blocks for PWM VLSI neural network, including a simple precise synapse multiplier with large linearity range and a voltages-pulse converter with high precision.  
           [0007]    Another object of the invention is to provide a CMOS voltage-mode sigmoid circuit with adjustable gain, whose transfer function is similar to ideal sigmoid function.  
           [0008]    To realize the above and other objects, the invention provides a voltage-mode pulse width modulation (PWM) VLSI implementation of neural networks, including a simple precise synapse multiplier with large linearity range and a voltages-pulse converter with high precision. Further, A CMOS voltage-mode sigmoid circuit with adjustable gain is designed, whose transfer function is similar to ideal sigmoid function, thereby constructing a 2-2-1 feedforward PWM ANN.  
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0009]    The invention will become apparent by referring to the following detailed description of a preferred embodiment with reference to the accompanying drawings, wherein:  
         [0010]    [0010]FIG. 1 is a schematic diagram of a conventional neural network;  
         [0011]    [0011]FIG. 2 illustrates a synapse multiplier of the invention;  
         [0012]    [0012]FIGS. 3 a - 3   d  illustrate the simulation results of the synapse multiplier;  
         [0013]    [0013]FIG. 4 a  illustrates a converter of the invention;  
         [0014]    [0014]FIG. 4 b  illustrates the simulation result of the converter of FIG. 4 a;    
         [0015]    [0015]FIG. 5 a  illustrates a non-linear neuron cell of the invention;  
         [0016]    [0016]FIG. 5 b  illustrates the simulation result of the non-linear neuron cell of FIG. 5 a;    
         [0017]    [0017]FIG. 6 a  is an embodiment of the invention;  
         [0018]    [0018]FIG. 6 b  is the circuit structure of FIG. 6 a ; and  
         [0019]    [0019]FIG. 6 c  is the simulation result of the circuit structure shown in FIG. 6 b.   
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0020]    In a single layer, there is no hidden neurons. Consequently, it cannot classify input patterns that are not linearly separable. However, nonlinearly separable patterns are of common occurrence. The well-known example is the XOR problem. The single layer cannot construct a straight decision line for the recognition of Exclusive-Or (XOR) logic operands. Further, the training algorithm often converges to a local minimum, which is not the optimum solution. The optimum solution would be a global minimum for a given set of examples. Hence, a 2-2-1 feedforward PWM ANN is constructed to verify a building blocks, the simulation result shows its ability to solve AND, OR and XOR problems.  
         [0021]    In this embodiment, choose pulse width modulation (PWM) for the VLSI neural network implementation because it is the easiest for circuit implementation among the various kinds of pulse modulation techniques. So, the building blocks for PWM VLSI neural networks are designed, including a simple precise synapse multiplier with large linearity range and a voltage-pulse converter with high precision. A CMOS voltage-mode sigmoid circuit with adjustable gain is designed, whose transfer function is similar to the ideal sigmoid function. The detail description is followed.  
         [0022]    Synapse Multiplier  
         [0023]    Refer to FIG. 2, showing a PWM synapse multiplier. In FIG. 2, the circuit in left dash block is multiple cell, while that in the right dash block is integral and summation cell. Vw is input weight voltage, Px is input pulse of neuron state, R ST  is reset control signal, and V OUT  is output voltage.  
         [0024]    As shown in FIG. 2, M 1  and M 2  act as a voltage-current converter, the difference of it of M 1  and M 2 &#39;s drain-source current I B  can be expressed as 
           I   B =−2κ 1 ( Vcc −2 V   TN )( Vw −½ Vcc )  (1) 
         [0025]    If V REF +V TP ≦Vw≦V REF +V TN , κ 1 =κ 2 , V TN =−V TP  and Vcc=2 V REF  exist, where V TP  and V TN  are PMOS and NMOS threshold voltages, κ 1  and κ 2  are conducting factors of M 1  and M 2  respectively.  
         [0026]    When R ST  is high, Vout can be reset to reference voltage V REF . After R ST  drops to low, I B  begins to charge capacitor C 1 , this charging process is controlled by Px via M 3 , and the charging time is equal to T WIDTH  which is the pulse width of Px. So the final result of the multiplier can be expressed as  
                 V   OUT     -       1   2        Vcc       =       -       2          κ   1     ·     T   WIDTH           C                 1              (     Vcc   -     2        V   TN         )                     (     Vw   -       1   2        Vcc       )               (   2   )                               
 
         [0027]    When M 3  is turned on, M 4  will be turned off, so V 1  still equals to V REF . This means the working states of M 1  and M 2  will not change when M 3  is on or off, so the precision of the multiplier will be improved. To reduce the feedthrough effects of Px and R ST , dummy MOSFETs M 5  and M 6  are added, whose W/L dimensions are ½ of M 3  and M 7  respectively. They can greatly reduce the switching noises at the negative input of the operational amplifier.  
         [0028]    The Spectre simulation result of the synapse multiplier under Cadence Analog Artist Environment is shown in FIG. 3 a and  3   b . The 3 curves in FIG. 3 a  represent 3 different pulse widths. When T WIDTH  is 200 ns, the input voltage linearity range of this multiplier can be ±1.5V. When T WIDTH  is 400 ns, the linearity range is only about ±0.7V because of the limited saturation voltage of the operational amplifier. According to the numerical analysis, it can be seen in FIG. 3 b  that the linearity errors of the 3 curve are below 1% when Vw is in the range from 1.8V to 3.2V. The 7 curves in FIG. 3 c  represent 7 weight voltages. They have good linearity when T WIDTH  is in the range from 0 ns to 350 ns, the numerical analysis results of 4 curves in FIG. 3 d  show their linearity errors are below 0.4%. According to FIG. 3 a , the ideal function of this synapse multiplier can be expressed as  
                   V   OUT          (   V   )       -   2.5     =       (       Vw        (   V   )       -   2.5     )     ×         T   WIDTH          (   ns   )       126               (   3   )                               
 
         [0029]    [Voltage-pulse Converter] 
         [0030]    In PWM ANN, neuron state is represented and transferred by pulse signal, so it is necessary to convert voltage to pulse. The schematic and symbol of voltage-pulse converter is shown in FIG. 4 a , where V V  is input voltage, C TRL  is controlling signal and V P  is output pulse signal.  
         [0031]    When C TRL  is high, voltage V 1  on capacitor C 1  is 0V and V P  is also low. When C TRL  drops to low, V P  will jump to high immediately and V 1  will begin to increase linearly from 0V. Once V 1  exceeds V V , the comparator will overturn, then V P  will drop to low. So the pulse width of V P  is proportional to V V .  
         [0032]    The simulation result is shown in FIG. 4 b . When V V  is in the range from 0.7V to 5V, the conversion curve maintains good linearity. But when V V  is below 0.7V, T WIDTH  is 0 ns. The cause is that when V V  is lower than MOSFET threshold voltage, the comparator can not work correctly, so V P  maintains low voltage. This problem is not important because it can be corrected by adjusting the weight learning algorithm.  
         [0033]    According to FIG. 4 a , the ideal function of this voltage-pulse converter can be expressed as 
           T   WIDTH (ns)=42.1 ×V   V ( V )  (4) 
         [0034]    [Sigmoid Circuit] 
         [0035]    To implement the non-linear activation function of neuron, a voltage-mode sigmoid function circuit with adjustable gain is designed. It is shown in FIG. 5 a , where V IN  is input voltage, V OUT  is output voltage and V CTRL  is gain-controlling voltage. Its simulation result is shown in FIG. 5 b . When V CTRL  equals 1.6V, its function can be expressed as  
               V   OUT     =     5     1   +     e       -   4          (       V   IN     -   2.5     )                     (   5   )                               
 
         [0036]    [PWM VLSI ANN Design Integration] 
         [0037]    Any feedforward PWM VLSI ANNs can be constructed by the above building blocks. For verification, a 2-2-1 feedforward ANN is designed which can solve AND, OR and XOR problems. Its network and circuit structures are shown in FIG. 6 a  and  6   b  respectively, where a 1  and a 2  are input signals, o is the output signal. C TRL1  and C TRL2  are used to control the voltage-pulse conversion in input layer and hidden layer, R SR1  and R ST2  are used to reset hidden and output neurons respectively. BP algorithm is used for training. Because the multiplication and sigmoid transformation of the circuit are not ideal, the iterative equations of the BP algorithm are adjusted according to the circuit characteristics.  
         [0038]    The simulation result is shown in FIG. 6 c , where o 1 , o 2  and o 3  are AND, OR and XOR recognition results respectively. There are 4 combinations for a 1  and a 2  during the simulation, in each combination after the calculations of hidden and output layers which take about 1.5 μs, correct recognition results are generated. To make the results easy to recognize, the idle time is added between each combination, so the whole recognition process takes about 12 μs. Ideally, it can be finished in 2 μs, which is more than 1000 times faster than the speed of the Matlab simulation on the SUM Ultra10 workstation, wherein Matlab is an integrated technical computing environment that combines numeric computation, advanced graphics and visualization, and a high-level programming language.  
         [0039]    This embodiment provides the digital-analog hybrid structure with pulse stream technique combining the merits of analog and digital techniques. That is, the structure has the advantages of a small chip area, fast speed and low power consumption in analog circuits and the advantages of high precision and robustness in digital circuit. It uses a digital signal to control analog multiplication, so the area of synapse analog multiplier is small and the neuron digital state signal is immune to noises.  
         [0040]    Although the invention has been described in its preferred embodiment, it is not intended to limit the invention to the precise embodiment disclosed herein. Those who are skilled in this technology can still make various alterations and modifications without departing from the scope and spirit of this invention. Therefore, the scope of the invention shall be defined and protected by the following claims and their equivalents.