Patent Publication Number: US-11651193-B2

Title: Operation apparatus

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2019-049035 filed on Mar. 15, 2019, the entire contents of which are incorporated herein by reference. 
     FIELD 
     Embodiments described herein relate generally to operation apparatuses. 
     BACKGROUND 
     There has been developed a technique for implementing a neuromorphic processor using a hardware-implemented neural network. In such a neuromorphic processor, a learning tool provides error data for a neural network to optimize weighting coefficients and other parameters set for the neural network. 
     The conventional neural network performs a learning process and optimizes weighting coefficients with its normal operation process halted. This allows the conventional neural network to perform the learning process with an external processor. 
     However, implementing the neuromorphic processor involves the neural network performing the operation process and the learning process in parallel. This configuration requires the neural network to perform, in parallel, a process of propagating target data to be operated, which is received from an external device, in a forward direction and a process of propagating error data for learning in a backward direction. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a structural diagram of an operation apparatus according to an embodiment; 
         FIG.  2    is a flowchart illustrating a flow of a process by the operation apparatus; 
         FIG.  3    is a structural diagram of first and second neural networks; 
         FIG.  4    is a diagram indicating input and output values to/from an n-th layer of the first and second neural networks; 
         FIG.  5    is a structural diagram of an evaluation unit; 
         FIG.  6    is a diagram illustrating input and output values to/from an m-th layer of a backward propagation neural network; 
         FIG.  7    is a diagram illustrating a signal input-output relation in a first mode; 
         FIG.  8    is a diagram illustrating a signal input-output relation in a second mode; 
         FIG.  9    is a diagram illustrating input and output values to/from an n-th updating unit; and 
         FIG.  10    is a structural diagram of an alternative embodiment of the operation apparatus. 
     
    
    
     DETAILED DESCRIPTION 
     According to an embodiment, an operation apparatus includes a first neural network, a second neural network, an evaluation circuit, and a coefficient-updating circuit. The first neural network is configured to perform an operation in a first mode, the first neural network being implemented with a hardware circuit. The second neural network is configured to perform an operation in a second mode differing from the first mode, the second neural network being implemented with a hardware circuit and having a same layer structure as a layer structure of the first neural network. The evaluation circuit is configured to evaluate an error of the operation performed by the first neural network in the first mode and evaluate an error of the operation performed by the second neural network in the second mode. The coefficient-updating circuit is configured to update, in the first mode, a plurality of coefficients set for the second neural network based on an evaluating result of the error of the operation of the first neural network, and update, in the second mode, a plurality of coefficients set for the first neural network based on an evaluating result of the error of the operation of the second neural network. 
     An operation apparatus  10  according to an embodiment will now be described with reference to the accompanying drawings. The operation apparatus  10  according to the embodiment is capable of performing an operation process and a learning process in parallel in a neural network. 
       FIG.  1    is a diagram illustrating the configuration of the operation apparatus  10  according to the embodiment. The operation apparatus  10  is hardware implemented on, for example, a semiconductor chip. The operation apparatus  10  may be a circuit formed on a substrate. The operation apparatus  10  may be a circuit formed on a plurality of semiconductor chips or a plurality of substrates. 
     The operation apparatus  10  includes a first neural network  21 , a second neural network  22 , an input unit (an input circuit)  24 , an output unit (an output circuit)  26 , an evaluation unit (an evaluation circuit)  28 , a coefficient-updating unit (a coefficient-updating circuit)  30 , and a controller (a control circuit)  32 . 
     The first neural network  21  is implemented with a hardware circuit. In the first neural network  21 , a plurality of coefficients set for each of a plurality of layers may be variable resistor elements used for resistive random access memories (ReRAMs), or may be variable capacitors, for example. 
     The second neural network  22  is implemented with a hardware circuit. The second neural network  22  has the same layer structure as that of the first neural network  21 . The number of layers, the number of values input to and output from each layer, a matrix-multiplication circuit in each layer, and an activation function circuit in each layer of the first neural network  21  are the same as those of the second neural network  22 . The values of the coefficients set for the first neural network  21  and the second neural network  22  may be different from each other. 
     The first neural network  21  and the second neural network  22  have N (N is an integer of 2 or more) layers each. The first neural network  21  and the second neural network  22  individually output a plurality of intermediate output values from each of the N layers. 
     The action of the operation apparatus  10  is switched between a first mode and a second mode. The first neural network  21  performs an operation in the first mode, and does not perform any operation in the second mode. In contrast, the second neural network  22  performs an operation in the second mode, and does not perform any operation in the first mode. 
     Coefficients set in the first neural network  21  are updated in the second mode, and are not updated in the first mode. In contrast, coefficients set in the second neural network  21  are updated in the first mode, are not updated in the second mode. 
     The input unit  24  receives a plurality of input values to be operated from another device. In the first mode, the input unit  24  provides the input values for the initial layer (first layer) of the first neural network  21 . In the second mode, the input unit  24  provides the input values for the initial layer (first layer) of the second neural network  22 . 
     In the first mode, the output unit  26  outputs, as output values, the intermediate output values output from the final layer (N-th layer) of the first neural network  21  to another device. In the second mode, the output unit  26  outputs, as output values, the intermediate output values output from the final layer (N-th layer) of the second neural network  22  to another device. 
     In the first mode, the evaluation unit  28  evaluates an error of the operation by the first neural network  21 . In the second mode, the evaluation unit  28  evaluates an error of the operation by the second neural network  22 . 
     For example, the evaluation unit  28  generates, in the first mode, a plurality of intermediate evaluation values correspondingly to each of the N layers of the second neural network  22 . The plurality of intermediate evaluation values to be generated, which correspond to each of the N layers of the second neural network  22 , are values obtained by evaluating the plurality of intermediate output values output from the corresponding layer of the N layers in the first neural network  21 . The evaluation unit  28  generates, in the second mode, a plurality of intermediate evaluation values correspondingly to each of the N layers of the first neural network  21 . The plurality of intermediate evaluation values to be generated, which correspond to each of the N layers of the first neural network  21 , are values obtained by evaluating the plurality of intermediate output values output from the corresponding layer of the N layers in the second neural network  22 . 
     The evaluation unit  28  may be implemented with a hardware circuit, or may be implemented by a processor executing a computer program. The details of the evaluation unit  28  will be further described later. 
     In the first mode, the coefficient-updating unit  30  updates the coefficients set for the second neural network  22 , based on the evaluation result of evaluating the error of the operation by the first neural network  21 . In the second mode, the coefficient-updating unit updates the coefficients set for the first neural network  21 , based on the evaluation result of evaluating the error of the operation by the second neural network  22 . 
     Specifically, the coefficient-updating unit  30  updates, in the first mode, the coefficients set for each of the N layers of the second neural network  22 , based on the intermediate evaluation values corresponding to each of the N layers of the first neural network  21 . 
     Similarly, in the second mode, the coefficient-updating unit  30  updates the coefficients set for each of the N layers of the first neural network  21 , based on the intermediate evaluation values corresponding to each of the N layers of the second neural network  22 . 
     For example, the coefficient-updating unit  30  calculates gradients of the errors of the coefficients for each of the N layers. The coefficient-updating unit  30  changes the coefficients in a direction where the gradients of the errors of the coefficients become zeros. 
     The coefficient-updating unit  30  may be implemented with a hardware circuit, or may be implemented by a processor executing a computer program. The details of the coefficient-updating unit  30  will be further described later. 
     The controller  32  switches between the first mode and the second mode alternately. The controller  32  may switch the mode, for example, for a certain period of time. The controller  32  may switch the mode each time when it receives an input value predetermined times. 
       FIG.  2    is a flowchart illustrating a flow of a process by the operation apparatus  10 . At S 11 , it is assumed that the operation apparatus  10  is set to the first mode. 
     At S 11 , the operation apparatus  10  determines whether the mode is switched. That is, the operation apparatus  10  determines whether it is switched to the second mode. 
     When the mode is not switched (No at S 11 ), the operation apparatus  10  determines, at S 12 , whether a plurality of input values is acquired. When no input values are acquired (No at S 12 ), the operation apparatus  10  returns the process to S 11 . When the input values are acquired (Yes at S 12 ), the operation apparatus  10  advances the process to S 13 . When the mode is switched (Yes at S 11 ), the operation apparatus  10  advances the process to S 16 . 
     At S 13 , the operation apparatus  10  performs the operation on the acquired input values by the first neural network  21 . As a result, the operation apparatus  10  can output a plurality of output values. At S 14 , the operation apparatus  10  evaluates an error of the operation by the first neural network  21 . At S 15 , the operation apparatus  10  updates the coefficients set for the second neural network  22 , based on the evaluation result of evaluating the error of the operation by the first neural network  21 . 
     The operation apparatus  10  returns the process to S 11  after completing S 15 . The operation apparatus  10  repeats the process from S 11  to S 15  during the first mode. 
     When the process proceeds to S 16 , the operation apparatus  10  is in a state where it is set to the second mode. At S 16 , the operation apparatus  10  determines whether the mode is switched. That is, the operation apparatus  10  determines whether it is switched to the first mode. 
     When the mode is not switched (No at S 16 ), the operation apparatus  10  determines, at S 17 , whether a plurality of input values is acquired. When no input values are acquired (No at S 17 ), the operation apparatus  10  returns the process to S 16 . When the input values are acquired (Yes at S 17 ), the operation apparatus  10  advances the process to S 18 . When the mode is switched (Yes at S 16 ), the operation apparatus  10  returns the process to S 11 . 
     At S 18 , the operation apparatus  10  performs the operation on the acquired input values by the second neural network  22 . As a result, the operation apparatus  10  can output a plurality of output values. At S 19 , the operation apparatus  10  evaluates an error of the operation by the second neural network  22 . At S 20 , the operation apparatus  10  updates the coefficients set for the first neural network  21 , based on the evaluation result of evaluating the error of the operation by the second neural network  22 . 
     The operation apparatus  10  returns the process to S 16  after completing S 20 . The operation apparatus  10  repeats the process from S 16  to S 20  during the period of the second mode. 
     By the above process, the operation apparatus  10  can perform the operation by the first neural network  21  in the first mode and perform the operation by the second neural network  22  in the second mode. Furthermore, the operation apparatus  10  can update the coefficients set for the second neural network  22  in the first mode and update the coefficients set for the first neural network  21  in the second mode. Thus, the operation apparatus  10  can perform the operation process and the learning process in the neural network in parallel. 
     The operation apparatus  10  is switched between the first mode and the second mode alternately. This enables the operation apparatus  10  to advance the update of the coefficients set for the first neural network  21  and the update of the coefficients set for the second neural network  22  alternately. Thus, the operation apparatus  10  enables both the first neural network  21  and the second neural network  22  to be adaptive. 
       FIG.  3    is a diagram illustrating the configuration of the first neural network  21  and the second neural network  22 . Each of the N layers from the first layer to the N-th layer of the first neural network  21  and the second neural network  22  performs the following process. 
     The first layer acquires input values from the input unit  24 . An x-th (x is an integer of 1 or more) input value of the input values is represented by y [0]   x . 
     Note that a superscript number in the square brackets of y denotes a layer number. [0] denotes the number of the input unit  24 . A subscript number of y denotes the order of a plurality of values input to or output from the layer. The same goes for other variables. 
     Each of the N layers outputs the intermediate output values to the subsequent layer. The number of intermediate output values output from each of the N layers may be different from each other. An x-th intermediate output value of the intermediate output values output from an n-th layer is represented by y [n]   x , where n is any integer from 1 to N. 
     Each of the second to N-th layers acquires the intermediate output values from the preceding layer. Note that the number of intermediate output values acquired by each of the second to N-th layers is the same as the number of intermediate output values output from the preceding layer. 
     Each of the N layers calculates a plurality of product-sum operation values by matrix-multiplying the intermediate output values output from the preceding layer by the set coefficients. An x-th product-sum operation value of the product-sum operation values calculated at the n-th layer is represented by v [n]   x . 
     In addition, each of the N layers calculates the intermediate output values by performing a preset activation-function operation on the product-sum operation values. Each of the N layers outputs the calculated intermediate output values. The activation function set for each of the N layers may be different from that of any other layer. 
     The N-th layer outputs the intermediate output values to the output unit  26 . The output unit  26  provides the intermediate output values output from the N-th layer for another device, as a plurality of output values output from the operation apparatus  10 . 
     Each of the N layers outputs the calculated product-sum operation values and intermediate output values to the coefficient-updating unit  30  for it to update the coefficients. The number of product-sum operation values output from each of the N layers is the same as the number of intermediate output values. 
       FIG.  4    is a diagram illustrating the input and output values to/from the n-th layer of the first neural network  21  and the second neural network  22 , as well as the set coefficients. The n-th layer of the first neural network  21  and the second neural network  22  performs the following process. 
     The n-th layer acquires I intermediate output values output from the preceding (n−1)th layer, where I is an integer of 2 or more. Note that, if n=1, the n-th layer (i.e., the first layer) acquires I input values output from the input unit  24  as the I intermediate output values. An i-th (i is an integer of 1 or more and I or less) intermediate output value of the I intermediate output values acquired by the n-th layer is represented by y [n−1]   i . 
     The n-th layer outputs J intermediate output values, where J is an integer of 2 or more. A j-th (j is an integer of 1 or more and J or less) intermediate output value of the J intermediate output values output from the n-th layer is represented by y [n]   j . 
     (I×J) coefficients placed correspondingly to a matrix of I columns and J rows are set for the n-th layer. In the (I×J) coefficients set for the n-th layer, a coefficient placed at an i-th column and a j-th row is represented by w [n]   ij . 
     The n-th layer outputs J product-sum operation values. A j-th product-sum operation value of the J product-sum operation values calculated by the n-th layer is represented by v [n]   j . 
     In such a case, the n-th layer calculates the J product-sum operation value by using a matrix multiplication illustrated in the following equation (1), where j=1 to J. 
     
       
         
           
             
               
                 
                   
                     v 
                     j 
                     
                       ⌊ 
                       n 
                       ⌋ 
                     
                   
                   = 
                   
                     
                       ∑ 
                       
                         i 
                         = 
                         1 
                       
                       I 
                     
                     
                       ( 
                       
                         
                           w 
                           ij 
                           
                             ⌊ 
                             n 
                             ⌋ 
                           
                         
                         × 
                         
                           y 
                           i 
                           
                             ⌊ 
                             
                               n 
                               - 
                               1 
                             
                             ⌋ 
                           
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     In addition, an activation function is set for the n-th layer. The activation function set for the n-th layer is represented by f [n] (⋅). 
     The n-th layer calculates the J intermediate output values by using an activation-function operation illustrated in the following equation (2), where j=1 to J.
 
 y   [n]   j   =f   [n] ( v   [n]   j )  (2)
 
     The n-th layer outputs the J calculated intermediate output values to the (n+1)th layer. Note that, when n=N, the n-th layer (i.e., the N-th layer) provides the J calculated intermediate output values for the output unit  26 . 
     The n-th layer outputs the calculated J product-sum operation values and J intermediate output values to the coefficient-updating unit  30  in order to update the coefficients. 
       FIG.  5    is a diagram illustrating a configuration of the evaluation unit  28 . The evaluation unit  28  includes an output evaluation unit (an output evaluation circuit)  36  and a backward propagation neural network  40 . 
     The output evaluation unit  36  acquires, from another device, a plurality of target values, each being a target (teacher) of the corresponding output value output from the operation apparatus  10 . An x-th target value of the target values is represented by t x . 
     In the first mode, the output evaluation unit  36  acquires the intermediate output values output from the N-th layer of the first neural network  21 . In the first mode, the output evaluation unit  36  evaluates an error for each of the intermediate output values output from the N-th layer of the first neural network  21  relative to each of the target values, and thereby generates a plurality of output evaluation values. 
     In the second mode, the output evaluation unit  36  acquires the intermediate output values output from the N-th layer of the second neural network  22 . In the second mode, the output evaluation unit  36  evaluates an error for each of the intermediate output values output from the N-th layer of the second neural network  22  relative to each of the target values, and thereby generates the output evaluation values. 
     When the output evaluation unit  36  is implemented with a processor circuit, it may generate the output evaluation values by substituting the intermediate output values output from the N-th layer of the first neural network  21  or the second neural network  22  and the target values into a loss function prepared in advance. The output evaluation unit  36  may be provided with a hardware circuit that can perform a process corresponding to the loss function. 
     The output evaluation unit  36  provides the output evaluation values for the backward propagation neural network  40 . An x-th output evaluation value of the output evaluation values is represented by e [N]   x . 
     The backward propagation neural network  40  propagates the output evaluation values output from the output evaluation unit  36  and outputs a plurality of intermediate evaluation values corresponding to each of the N layers. 
     The backward propagation neural network  40  is a neural network implemented with a hardware circuit. In the backward propagation neural network  40 , the coefficients set for each of the layers may be variable resistor elements used for resistive random access memories (ReRAMs), or may be variable capacitors, for example. 
     The backward propagation neural network  40  has (N−1) layers corresponding to the second to N-th layers of the first neural network  21  and the second neural network  22 . The backward propagation neural network  40  propagates the output evaluation values received from the output evaluation unit  36  in a direction from the N-th layer toward the second layer. 
     The N-th layer acquires the output evaluation values from the output evaluation unit  36 . An x-th output evaluation value of the output evaluation values is represented by e [N]   x . 
     An m-th layer in the second to (N−1)th layers acquires the intermediate evaluation values from the preceding (m+1)th layer. An x-th intermediate evaluation value of the intermediate evaluation values output from the m-th layer is represented by e [m]   x , where m is any integer of 2 to N. 
     Note that the number of intermediate evaluation values acquired by each of the (N−1) layers is the same as the number of intermediate output values output from the corresponding layer of the first neural network  21  and the second neural network  22 . 
     The m-th layer of the (N−1) layers calculates a plurality of operation values by matrix-multiplying the intermediate evaluate values output from the preceding (m+1)th layer by the set coefficients. 
     Each of the (N−1) layers calculates the operation values by performing a matrix multiplication in which the matrix multiplication for the corresponding layer of the first neural network  21  and the second neural network  22  is transposed. The coefficients set for each of the (N−1) layers may be independent of the coefficients set for the corresponding layer of the first neural network  21  and the second neural network  22 . For example, the coefficients set for each of the (N−1) layers of the backward propagation neural network  40  may be random values. 
     Furthermore, each of the (N−1) layers calculates the intermediate evaluation values by performing the preset function operation on the operation values. The function preset for each of the (N−1) layers may be independent of the activation function set for the corresponding layer of the first neural network  21  and the second neural network  22 . 
     The backward propagation neural network  40  outputs the intermediate evaluation values calculated by each of the (N−1) layers to the coefficient-updating unit  30  as the intermediate evaluation values corresponding to the preceding layer of the first neural network  21  and the second neural network  22 . That is, the backward propagation neural network  40  outputs the intermediate evaluation values calculated by the m-th layer as the intermediate evaluation values corresponding to the (m−1)th layer of the first neural network  21  and the second neural network  22 . 
     Furthermore, the backward propagation neural network  40  outputs the output evaluation values acquired by the N-th layer as the intermediate evaluation values corresponding to the N-th layer of the first neural network  21  and the second neural network  22 . 
       FIG.  6    is a diagram illustrating input and output values to/from the m-th layer of the backward propagation neural network  40 , as well as the set coefficients. The m-th layer of the backward propagation neural network  40  performs the following process. 
     The m-th layer acquires J output evaluation values output from the preceding (m+1)th layer. Note that, if m=N, the m-th layer (i.e., the N-th layer) acquires the J output evaluation values output from the output evaluation unit  36  as J intermediate evaluation values. A j-th intermediate evaluation value of the J intermediate evaluation values acquired by the m-th layer is represented by e [m]   j . 
     The m-th layer outputs I intermediate evaluation values. An i-th intermediate evaluation value of the I intermediate evaluation values output from the m-th layer is represented by e [m−1]   i . 
     (J×I) coefficients placed correspondingly to a matrix of J columns and I rows are set for the m-th layer. Of the (J×I) coefficients set for the m-th layer, a coefficient placed at a j-th column and an i-th row is represented by α [m]   ji . 
     The m-th layer calculates I operation values. An i-th operation value of the I operation values calculated by the m-th layer is represented by s [m−1]   i . 
     In such a case, the m-th layer calculates the I operation values by using a matrix multiplication illustrated in the following equation (3), where i=1 to I. 
     
       
         
           
             
               
                 
                   
                     s 
                     i 
                     
                       [ 
                       
                         m 
                         - 
                         1 
                       
                       ] 
                     
                   
                   = 
                   
                     
                       ∑ 
                       
                         j 
                         = 
                         1 
                       
                       J 
                     
                     
                       ( 
                       
                         
                           α 
                           ji 
                           
                             [ 
                             m 
                             ] 
                           
                         
                         × 
                         
                           e 
                           j 
                           
                             [ 
                             m 
                             ] 
                           
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
     Here, the matrix multiplication in the equation (3) is a matrix multiplication in which the matrix multiplication for the m-th layer of the first neural network  21  and the second neural network  22  is transposed. 
     In addition, an activation function is set for the m-th layer. The activation function set for the m-th layer is represented by g [m] (⋅). In such a case, the m-th layer calculates the I intermediate evaluation values by using a function operation illustrated in the following equation (4), where i=1 to I.
 
 e   [m−1]   i   =g   [m] ( s   [m−1]   i )  (4)
 
     The m-th layer outputs the I calculated intermediate evaluation values to the subsequent (m−1)th layer of the backward propagation neural network  40 . Note that, for m=2, the m layer (i.e., the second layer) does not output the I intermediate evaluation values to the subsequent layer. 
     Furthermore, the m-th layer outputs the I calculated intermediate evaluation values to the coefficient-updating unit  30  as the intermediate evaluation values corresponding to the (m−1)th layer of the first neural network  21  and the second neural network  22 . 
       FIG.  7    is a diagram illustrating a signal input-output relation within the operation apparatus  10  in the first mode. In the present embodiment, the coefficient-updating unit  30  has N, first to N-th, updating units  44 - 1  to  44 -N. 
     In the first mode, the coefficient-updating unit  30  acquires the intermediate output values output from the preceding layer of the first neural network  21 , correspondingly to each of the N, first to N-th, layers. For example, in the first mode, the n-th updating unit  44 - n  acquires the intermediate output values output from the (n−1)th layer of the first neural network  21 . 
     Note that, in the first mode, the coefficient-updating unit  30  acquires the input values output from the input unit  24  as the intermediate output values corresponding to the first (initial) layer. Specifically, in the first mode, if n=1, the n updating unit  44 - n  (i.e., the first updating unit  44 - 1 ) acquires the input values output from the input unit  24  as the intermediate output values. 
     In the first mode, the coefficient-updating unit  30  acquires the product-sum operation values calculated by the corresponding layer of the first neural network  21 , correspondingly to each of the N layers. For example, in the first mode, the n-th updating unit  44 - n  acquires the product-sum operation values calculated by the n-th layer of the first neural network  21 . 
     In the first mode, the coefficient-updating unit  30  acquires, from the evaluation unit  28 , the intermediate evaluation values calculated for the corresponding layer, correspondingly to each of the N layers. Specifically, in the first mode, the n-th updating unit  44 - n  acquires the intermediate evaluation values output from the (n−1)th layer of the backward propagation neural network  40 . Note that, in the first mode, if n=N, the n-th updating unit  44 - n  (i.e., the N-th updating unit  44 -N) acquires the intermediate evaluation values output from the output evaluation unit  36  as the intermediate evaluation values. 
     In the first mode, the coefficient-updating unit  30  updates the coefficients set for each of the N layers of the second neural network  22 , based on the intermediate output values, the product-sum operation values and the intermediate evaluation values acquired correspondingly to each of the N layers. Specifically, the n-th updating unit  44 - n  updates the coefficients set for the n-th layer of the second neural network  22 , based on the intermediate output values, the product-sum operation values and the intermediate evaluation values acquired correspondingly to the n-th layer. 
       FIG.  8    is a diagram illustrating a signal input-output relation within the operation apparatus  10  in the second mode. 
     In the second mode, the coefficient-updating unit  30  acquires the intermediate output values output from the preceding layer of the second neural network  22 , correspondingly to each of the N, first to N-th, layers. Specifically, in the second mode, the n-th updating unit  44 - n  acquires the intermediate output values output from the (n−1)th layer of the second neural network  22 . 
     Note that, in the second mode, the coefficient-updating unit  30  acquires the input values output from the input unit  24  as the intermediate output values corresponding to the first layer. For example, in the second mode, if n=1, the n-th updating unit  44 - n  (i.e., the first updating unit  44 - 1 ) acquires the input values output from the input unit  24  as the intermediate output values. 
     In the second mode, the coefficient-updating unit  30  acquires, correspondingly to each of the N layers, the product-sum operation values calculated by the corresponding layer of the second neural network  22 . Specifically, in the second mode, the n-th updating unit  44 - n  acquires the product-sum operation values calculated by the n-th layer of the second neural network  22 . 
     In the second mode, the coefficient-updating unit  30  acquires, correspondingly to each of the N layers, the intermediate evaluation values calculated for the corresponding layer from the evaluation unit  28 . Specifically, in the second mode, the n-th updating unit  44 - n  acquires the intermediate evaluation values output from the (n−1)th layer of the backward propagation neural network  40 . Note that, in the second mode, if n=N, the n-th updating unit  44 - n  (i.e., the N-th updating unit  44 -N) acquires the intermediate evaluation values output from the output evaluation unit  36  as the intermediate evaluation values. 
     In the second mode, the coefficient-updating unit  30  updates the coefficients set for each of the N layers of the first neural network  21 , based on the intermediate output values, the product-sum operation values and the intermediate evaluation values acquired correspondingly to each of the N layers. Specifically, the n-th updating unit  44 - n  updates the coefficients set for the n-th layer of the first neural network  21 , based on the intermediate output values, the product-sum operation values, and the intermediate evaluation values acquired correspondingly to the n-th layer. 
       FIG.  9    is a diagram illustrating input and output values to/from the n-th updating unit  44 - n . The n-th updating unit  44 - n  carries out the following process. 
     The n-th updating unit  44 - n  acquires the I intermediate output values y [n−1]   1 , . . . , y [n−1]   i , . . . , y [n−1]   I ). The n-th updating unit  44 - n  acquires the J product-sum operation values v [n]   1 , . . . , v [n]   j , . . . v [n]   J ) The n-th updating unit  44 - n  acquires the J intermediate evaluation values (e [n]   1 , . . . , e [n]   j , . . . , e [n]   J ). 
     The n-th updating unit  44 - n  updates the (I×J) coefficients (w [n]   11 , . . . w [n]   ij , . . . , w [n]   IJ ) set for the n-th layer of the first neural network  21  and the second neural network  22 . In the present embodiment, the n-th updating unit  44 - n  calculates the gradient of an evaluation function for evaluating an error of a coefficient, for each of the (I×J) coefficients set for the n-th layer. The n-th updating unit  44 - n  changes each of the (I×J) coefficients set for the n-th layer such that the gradient is reduced (for example, such that it becomes 0). 
     For example, among the (I×J) coefficients set for the n-th layer, a coefficient at an i-th row and a j-th column is represented by w [n] . In this case, the n-th updating unit  44 - n  changes the coefficient of the i-th row and j-th column in accordance with the following equation (5). Note that, E denotes the evaluation function, and ∂E/∂w [n]   ij  denotes the gradient of the evaluation function for the coefficient of the i-th row and j-th column.
 
 w   [n]   ij   =w   [n]   ij   −∂E/∂w   [n]   ij   (5)
 
     The n-th updating unit  44 - n  calculates the gradient (∂E/∂w [n]   ij ) by using the following equation (6),
 
∂ E/∂w   [n]   ij   =y   [n−1]   i   ×f   [n] ′( v   [n]   j )× e   [n]   j   (6)
 
where f [n] ′(⋅) is the differential function of the activation function set for the n-th layer of the first neural network  21  and the second neural network  22 .
 
     The n-th updating unit  44 - n  is able to perform an operation corresponding to the error backward propagation method to update the coefficients set for the n-th layer of the first neural network  21  and the second neural network  22 . 
       FIG.  10    is a structural diagram of an alternative embodiment of the operation apparatus  10 . In place of the evaluation unit  28  and the coefficient-updating unit  30 , the operation apparatus  10  may include an evaluation unit  28 - 1  for the first mode, an evaluation unit  28 - 2  for the second mode, a coefficient-updating unit  30 - 1  for the first mode, and a coefficient-updating unit  30 - 2  for the second mode. 
     The evaluation unit  28 - 1  for the first mode and the evaluation unit  28 - 2  for the second mode have the same configuration as that of the evaluation unit  28  illustrated in  FIG.  1   . The coefficient-updating unit  30 - 1  for the first mode and the coefficient-updating unit  30 - 2  for the second mode have the same configuration as that of the coefficient-updating unit  30  illustrated in  FIG.  1   . 
     In the first mode, the controller  32  activates the evaluation unit  28 - 1  and coefficient-updating unit  30 - 1  for the first mode and halts the action of the evaluation unit  28 - 2  and coefficient-updating unit  30 - 2  for the second mode. In the second mode, the controller  32  halts the action of the evaluation unit  28 - 1  and coefficient-updating unit  30 - 1  for the first mode and activates the evaluation unit  28 - 2  and coefficient-updating unit  30 - 2  for the second mode. 
     Such a structure of the operation apparatus  10  allows the coefficient-updating unit  30 - 2  for the second mode to be placed close to the first neural network  21 . Similarly, the operation apparatus  10  allows the coefficient-updating unit  30 - 1  for the first mode to be placed close to the second neural network  22 . Those placements in the operation apparatus allow any physical switches and other devices to be eliminated, and wires or other components for updating coefficients to be reduced. 
     As described above, the operation apparatus  10  is able to perform the operation process by the neural network and the learning process in the neural network in parallel. This enables the operation apparatus  10  to perform the learning process in real time without halting the operation process. 
     In addition, the operation apparatus  10  generates the intermediate evaluation values for learning coefficients set for each of the layers of the neural network, by using the backward propagation neural network  40 . The operation apparatus  10  produces the following effect by using the backward propagation neural network  40 . 
     In implementation of the neural network with hardware, coefficients are implemented with, for example, resistance values or capacitor capacitances. Such an implementation of the neural network with hardware may cause a learning device to spend relatively long time to update the coefficients. 
     In a conventional error backward propagation method, when updating coefficients of one layer, a learning device must have been finished updating coefficients of the immediately preceding layer. Thus, when coefficients of the hardware-implemented neural network are updated by using the conventional error backward propagation method, the learning device requires extremely long time to update the coefficients for every layers of the neural network. 
     In contrast, the operation apparatus  10  according to the present disclosure calculates the intermediate evaluation values for each of the layers by using the backward propagation neural network  40 . Thus, the operation apparatus  10  can calculate the intermediate evaluation values for each of the layers without updating the coefficients of the targeted neural network. This enables the operation apparatus  10  to rapidly perform the learning process. 
     While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the embodiments described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.