Patent Publication Number: US-2023162035-A1

Title: Storage medium, model reduction apparatus, and model reduction method

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2021-191164, filed on Nov. 25, 2021, the entire contents of which are incorporated herein by reference. 
     FIELD 
     The technique disclosed herein is related to a storage medium, a model reduction apparatus, and a model reduction method. 
     BACKGROUND 
     A machine learning model (hereafter, also simply referred to as a “model”) tends to increase in size due to, for example, evolution of the deep learning technique. As the size of a model increases, computing resources such as a memory and a processor desired for machine learning also significantly increase. Meanwhile, mobile devices and other environments that desire the deep learning technique tend to diversify. Although a huge model is desired at the start of machine learning, there may be a case where the number of parameters finally desired for inference is not many as a result of the machine learning. Accordingly, in order to address the above-described tendency, a model size-reduction technique has become widely noticed in which machine learning of a model is executed in an environment having a large amount of computing resources such as a server and the like, and the size-reduced model obtained by deleting unwanted parameters is used for inference. 
     For example, there has been proposed a method of correcting a configuration of a fuzzy inference model in which, when the fuzzy inference model is created, meaningless input and output parameters are deleted and operation time by the fuzzy inference model is decreased. According to this method, arbitrary input data is given to the fuzzy inference model, corresponding output data is calculated, a plurality of sets of pieces of pseudo data are created, and a neural network having input and output parameters common to those of the fuzzy inference model is configured. According to this method, the pseudo data is given as teacher data to determine a characteristic value of the neural network, and this neural network is used to calculate the degree of influence of each input parameter on each output parameter. According to this method, input parameters having a small degree of influence on any output parameter and output parameters influenced in a small degree by any input parameter are extracted. According to this method, the extracted input/output parameters are deleted from the input/output parameters of the fuzzy inference model, thereby to correct the fuzzy inference model. 
     Japanese Laid-open Patent Publication No. 2000-322263 is disclosed as related art. 
     SUMMARY 
     According to an aspect of the embodiments, a non-transitory computer-readable storage medium storing a model reduction program that causes at least one computer to execute a process, the process includes identifying as deletion targets a first neuron that does not connect to an input layer in a neural network; identifying as deletion targets a second neuron that does not connect to an output layer in a neural network; combining a bias of the first neuron with a bias of a third neuron connected to the first neuron on an output side; and deleting the first neuron and the second neuron from the neural network. 
     The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims. 
     It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG.  1    is a functional block diagram of a model reduction apparatus; 
         FIG.  2    is a diagram for explaining an example of an existing model size-reduction technique; 
         FIG.  3    is a diagram for explaining a problem with the existing model size-reduction technique; 
         FIG.  4    is a diagram for explaining a notation of weights between neurons; 
         FIG.  5    is a diagram illustrating an example of a parameter table; 
         FIG.  6    is a diagram for explaining identification of a deletion-target neuron and bias compensation; 
         FIG.  7    is a diagram for explaining deletion of a parameter; 
         FIG.  8    is a block diagram schematically illustrating the configuration of a computer that functions as the model reduction apparatus; 
         FIG.  9    is a flowchart illustrating an example of a model reduction process; 
         FIG.  10    is a flowchart illustrating an example of a forward weight correction process; 
         FIG.  11    is a flowchart illustrating an example of a backward weight correction process; 
         FIG.  12    is a flowchart illustrating an example of a deletion process; 
         FIG.  13    is a diagram illustrating an examples of a layer information table and a function table; 
         FIG.  14    is a diagram illustrating examples of a layer information table and a parameter table for a neural network including a convolution layer; 
         FIG.  15    is a diagram for explaining deletion of parameters in a case where a neural network including a convolution layer is a target; 
         FIG.  16    is a diagram illustrating an example of a layer configuration of the neural network; 
         FIG.  17    is a diagram for explaining the relationship between accuracy and size reduction of a model; 
         FIG.  18    is a diagram illustrating an example of layer-to-layer data sizes in a case where a reduction rate is 90%; and 
         FIG.  19    is a diagram illustrating an example of layer-to-layer data sizes in a case where a reduction rate is 98%. 
     
    
    
     DESCRIPTION OF EMBODIMENTS 
     When only parameters the degree of influence of which is small are deleted as in the related-art model size-reduction technique, useless parameters may remain due to the configuration of a network. In this case, calculation efficiency of inference by the generated model decreases. When parameters the influence of which is small are simply deleted, information useful for inference may be lost, and accuracy of the model after the deletion of the parameters may degrade. 
     In one aspect, an object of the disclosed technique is to improve an effect of size reduction of a machine learning model while suppressing degradation in accuracy of the machine learning model. 
     In the one aspect, an effect is obtained in which the effect of the size reduction of the machine learning model may be improved while suppressing degradation in the accuracy of the machine learning model. 
     Hereinafter, an example of an embodiment according to the disclosed technique will be described with reference to the drawings. 
     As illustrated in  FIG.  1   , a parameter table representing a neural network that is a machine learning model is input to a model reduction apparatus  10  according to the present embodiment. According to the present embodiment, the parameter table input to the model reduction apparatus  10  is a parameter table in which a subset of parameters is deleted by an existing model size-reduction technique. 
     An example of the existing model size-reduction technique will be described with reference to  FIG.  2   . In  FIG.  2   , circles represent neurons of a neural network, and arrows represent couplings between the neurons. These representations are similarly used in the drawings to be referred to below. Weights that are parameters of the model are set at couplings between the neurons. For example, as illustrated in  FIG.  2   , according to the existing model size-reduction technique, a threshold is applied to weights, between neurons, that are parameters of the model for which the machine learning is executed, and weights smaller than or equal to the threshold are corrected to 0. A middle section of  FIG.  2    illustrates that the weights, between neurons, represented by dashed arrows have been corrected to 0. As illustrated in a lower section of  FIG.  2   , according to the existing model size-reduction technique, a model is output in which the parameters are reduced by removing portions having the weight of 0 as unwanted parameters. 
     As illustrated in  FIG.  3   , in the case of a model reduced in size by the existing model size-reduction technique, in some cases, a neuron for which no input exists (a neuron I indicated by a thick circle in  FIG.  3   ) and a neuron that is not used for output (a neuron L indicated by a double circle in  FIG.  3   ) remain in the model. In this case, weights between neurons from the neuron without input to an output layer (weights in portions indicated by dashed arrows in  FIG.  3   ) are unwanted parameters that are not used to calculate output of the model. Similarly, weights between neurons from an input layer to neurons not used for output (weights in portions indicated by dotted arrows in  FIG.  3   ) are also unwanted parameters not used for the calculation of the output of the model. 
     Each neuron also has a bias as a parameter. For example, in a case where a value y output from a neuron is calculated by a simple linear function (y=ax+b), b is a bias term. Here, x is a value output from a neuron in a previous stage, and a is a weight between the neuron in the previous stage and a target neuron. The bias is a constant value that is obtained as a result of machine learning and does not depend on the input. In a case where a weight between a neuron (for example, I) for which input does not exist as described above and a neuron coupled, on the output side, to this neuron without input is simply deleted, a way for conveying information on the bias of the neuron without input to the neuron on the output side is lost. As a result, information useful for inference may be lost, and accuracy of the model after the size reduction may degrade. 
     Thus, according to the present embodiment, the size of a model is reduced by deleting the parameters so that the effect of the model size reduction may be improved while suppressing degradation in the accuracy of the model. Hereinafter, a functional configuration of the model reduction apparatus  10  according to the present embodiment will be described in detail. Hereinafter, as illustrated in  FIG.  4   , in a case where a neuron i in an (n−1)th layer and a neuron j in an nth layer are in a coupling relationship, a weight between the neuron i and the neuron j is represented as “w ij   (n) ”. The weight w i,j   (n)  is referred to as an output weight of the neuron i or an input weight of the neuron j. The bias of the neuron i is represented as “b i ”. The output weight is an example of a “weight on an output side” in the disclosed technique, and the input weight is an example of a “weight on an input side” in the disclosed technique. 
     As illustrated in  FIG.  1   , the model reduction apparatus  10  functionally includes a correction unit  12 , a compensation unit  14 , and a deletion unit  16 . The correction unit  12  is an example of an “identification unit” of the disclosed technique. 
     The correction unit  12  obtains parameter tables input to the model reduction apparatus  10 .  FIG.  5    illustrates examples of the parameter tables. The examples illustrated in  FIG.  5    are parameter tables of a neural network represented by a graph representation as illustrated in an upper section of  FIG.  5   . As illustrated in  FIG.  5   , the parameter tables are provided on a layer-by-layer basis. As indicated by “INPUT” in  FIG.  5   , in the parameter table of each layer, neurons of the corresponding layer correspond to respective rows. As indicated by “OUTPUT” in  FIG.  5   , neurons of a higher layer than the corresponding layer, for example, the neurons output values of which are input to the neurons of the corresponding layer correspond to respective columns. Each element of a matrix stores a weight between the neurons corresponding to the row and the column of the element. For example, each row of the parameter table stores input weights of one of the neurons corresponding to the row, and each column of the parameter table stores output weights of one of the neurons corresponding to the column. The parameter table of each layer also stores the biases of the neurons of the corresponding layer at the end column of the rows. 
     In the neural network, the correction unit  12  identifies, as deletion targets, first neurons without a coupling from the input layer and second neurons without a coupling to the output layer. Then, in the parameter tables, the correction unit  12  corrects the output weights of the first neurons to 0 and the input weights of the second neurons to 0. 
     For example, as illustrated in  FIG.  6   , the correction unit  12  sequentially identifies the first neurons that are the deletion targets by forward search from the input layer toward the output layer in the neural network. For example, the correction unit  12  searches for neurons all the input weights of which are 0 sequentially from the input layer. In an example illustrated in  FIG.  6   , based on the fact that the row of the neuron I is entirely set to 0 in a parameter table of an (n=2)th layer, the correction unit  12  determines that all the input weights of the neuron I are 0 and identifies the neuron I as the deletion target. The correction unit  12  corrects all the output weights of the neuron I, for example, all the weights of the column of the neuron I in a parameter table of an (n=3)th layer to 0. The correction unit  12  sequentially searches for neurons all the input weights of which are 0 in the forward search, thereby to identify that all the input weights of the neuron M are 0 and correct all the output weights of the neuron M to 0 (dashed arrows in  FIG.  6   ). 
     Similarly, as illustrated in  FIG.  6   , the correction unit  12  sequentially identifies the second neurons that are the deletion targets by backward search from the output layer toward the input layer in the neural network. For example, the correction unit  12  searches for neurons all the output weights of which are 0 sequentially from the output layer. In the example illustrated in  FIG.  6   , based on the fact that the column of the neuron L is entirely set to 0 in a parameter table of an (n=4)th layer, the correction unit  12  determines that all the output weights of the neuron L are 0 and identifies the neuron L as the deletion target. The correction unit  12  corrects all the input weights of the neuron L, for example, all the weights of the row of the neuron L in the parameter table of the (n=3)th layer to 0. The correction unit  12  sequentially searches for neurons all the output weights of which are 0 in the backward search, thereby to identify that all the output weights of a neuron G are 0 and correct all the input weights of the neuron G to 0 (dotted arrows in  FIG.  6   ). 
     For the first neurons identified as the deletion targets in the forward search, the correction unit  12  notifies the compensation unit  14  so that the compensation unit  14  executes a process of compensating for the biases of the identified neurons. 
     Based on the notification from the correction unit  12 , the compensation unit  14  compensates for biases of the first neurons as the deletion targets by combining the biases of the first neurons with biases of third neurons coupled to the first neurons on the output side. For example, the compensation unit  14  combines the biases by adding values obtained by multiplying the biases of the first neurons by the weights between the first neurons and the third neurons to the biases of the third neurons. 
     For example, a case is described in which the neuron I the bias of which is b I  is identified as the deletion-target first neuron. As illustrated in a one-dot chain line portion of an upper section of  FIG.  6    and a lower section of  FIG.  6   , the neuron I of the (n32 2)th layer is coupled to each of the neuron L and the neuron M of the (n=3)th layer. A bias of the neuron L is b L , a bias of the neuron M is b M , a weight between the neuron I and the neuron L is w I,L   (3) , and a weight between the neuron I and the neuron M is w I,M   (3) . In this case, the compensation unit  14  calculates b L  and b M  as described below and updates the values of the column of the bias of the rows respectively corresponding to the neuron L and the neuron M in the parameter table of the (n=3)th layer. 
     
       
      
       b 
       L 
       &lt;−b 
       L 
       +w 
       I,L 
       (3) 
       b 
       I 
       ,b 
       M 
       &lt;−b 
       M 
       +w 
       I,M 
       (3) 
       b 
       I  
      
     
     The deletion unit  16  deletes the identified deletion-target neurons from the neural network. The input weights and the output weights of the deletion-target neurons are all 0 in the parameter tables. For example, the deletion unit  16  deletes rows and columns corresponding to the weights of the deletion-target neurons in the parameter tables. For example, in a case where the neuron i in the (n−1)th layer is the deletion target, the deletion unit  16  deletes the row of the neuron i all the weights of which are 0 in the parameter table of the (n−1)th layer and the column of the neuron i all the weights of which are 0 in the parameter table of the nth layer. 
     For example, as illustrated in a left section of  FIG.  7   , it is assumed that the neuron D of the (n=2)th layer is identified as the deletion-target neuron. In this case, the weights of the row of “D” of the parameter table of the (n=2)th layer and the weights of the column of “D” of the parameter table of the (n=3)th layer are 0. As illustrated in a right section of  FIG.  7   , the deletion unit  16  deletes the row of “D” of the parameter table of the (n=2)th layer and the column of “D” of the parameter table of the (n=3)th layer. Thus, the size of the parameter tables, for example, the size of the model is reduced. The deletion unit  16  outputs the size-reduced parameter tables. 
     The model reduction apparatus  10  may be realized by using, for example, a computer  40  illustrated in  FIG.  8   . The computer  40  includes a central processing unit (CPU)  41 , a memory  42  serving as a temporary storage area, and a nonvolatile storage unit  43 . The computer  40  also includes an input/output device  44  such as an input unit, a display unit, and the like and a read/write (R/W) unit  45  that controls reading and writing of data from and to a non-temporary storage medium  49 . The computer  40  also includes a communication interface (I/F)  46  that is coupled to a network such as the Internet. The CPU  41 , the memory  42 , the storage unit  43 , the input/output device  44 , the R/W unit  45 , and the communication I/F  46  are coupled to each other via a bus  47 . 
     The storage unit  43  may be realized by using a hard disk drive (HDD), a solid-state drive (SSD), a flash memory, or the like. The storage unit  43  serving as a storage medium stores a model reduction program  50  for causing the computer  40  to function as the model reduction apparatus  10 . The model reduction program  50  includes a correction process  52 , a compensation process  54 , and a deletion process  56 . 
     The CPU  41  reads the model reduction program  50  from the storage unit  43 , loads the read model reduction program  50  on the memory  42 , and sequentially executes the processes included in the model reduction program  50 . The CPU  41  executes the correction process  52  to operate as the correction unit  12  illustrated in  FIG.  1   . The CPU  41  executes the compensation process  54  to operate as the compensation unit  14  illustrated in  FIG.  1   . The CPU  41  executes the deletion process  56  to operate as the deletion unit  16  illustrated in  FIG.  1   . Thus, the computer  40  that executes the model reduction program  50  functions as the model reduction apparatus  10 . The CPU  41  that executes the program is hardware. 
     The functions realized by the model reduction program  50  may instead be realized by, for example, a semiconductor integrated circuit, in more detail, an application-specific integrated circuit (ASIC) or the like. 
     Next, operations of the model reduction apparatus  10  according to the present embodiment will be described. When parameter tables which represents a neural network and from which a subset of parameters have been deleted by using the existing model size-reduction technique are input to the model reduction apparatus  10 , a model reduction process illustrated in  FIG.  9    is executed in the model reduction apparatus  10 . The model reduction process is an example of a method for model reduction of the disclosed technique. 
     In step S 10 , the correction unit  12  obtains parameter tables input to the model reduction apparatus  10 . Next, in step S 20 , the correction unit  12  executes a forward weight correction process, identifies the first neurons without a coupling from the input layer as the deletion targets, and corrects the output weights of the first neurons to 0 in the parameter tables. In so doing, the compensation unit  14  executes the process of compensating for the biases of the first neurons. Next, in step S 40 , the correction unit  12  executes a backward weight correction process, identifies the second neurons without a coupling to the output layer as the deletion targets, and corrects the input weights of the second neurons to 0 in the parameter tables. Next, in step S 60 , the deletion unit  16  executes a deletion process to delete the deletion-target neurons from the neural network. Hereinafter, each of the forward weight correction process, the backward weight correction process, and the deletion process will be described in detail. 
     First, the forward weight correction process will be described with reference to  FIG.  10   . 
     In step S 21 , the correction unit  12  sets a variable n that identifies a hierarchical layer to be processed in the neural network to 2. Next, in step S 22 , the correction unit  12  determines whether n exceeds N representing the number of hierarchical layers of the neural network. In a case where n does not exceed N, the process proceeds to step S 23 . 
     In step S 23 , the correction unit  12  obtains a list {c i } of the neurons all the input weights of which are 0 in the (n−1)th layer. The number of the neuron in the (n−1)th layer is represented by i, and i=1, 2, . . . , (I n−1  is the number of neurons in the (n−1)th layer). The numbers of the neurons all the input weights of which are 0 out of the neurons in the (n−1)th layer are represented by c i . For example, the correction unit  12  adds the numbers of the neurons corresponding to the rows in which all the weights are 0 to the list in the parameter table of the (n−1)th layer and obtains {c i }. 
     Next, in step S 24 , the correction unit  12  sets i to 1. Next, in step S 25 , the correction unit  12  determines whether i exceeds the maximum value C n−1  of the numbers of the neurons included in the list {c i }. In a case where i does not exceed C n−1 , the process moves to step S 26 . In step S 26 , the correction unit  12  sets j to 1. The number of the neurons in the nth layer is j, and j=1, 2, . . . , J n  (J n  is the number of neurons in the nth layer). Next, in step S 27 , the correction unit  12  determines whether j exceeds J n . In a case where j does not exceed J n , the process moves to step S 28 . 
     In step S 28 , the compensation unit  14  compensates for the bias of the ith neuron in the (n−1)th layer by combining the bias of the ith neuron in the (n−1)th layer with the bias of the jth neuron in the nth layer. For example, the compensation unit  14  calculates the bias of the jth neuron in the nth layer like b j &lt;−b j +w c_i,j   (n) b i  and updates the value in the column of the bias of the row corresponding to the jth neuron in the parameter table of the nth layer. Next, in step S 29 , the correction unit  12  deletes the output weight from the ith neuron in the (n−1)th layer to the jth neuron in the nth layer. For example, the correction unit  12  corrects the weight w c_i,j  stored in the parameter table of the nth layers to 0. Thus, both the input weight and the output weight of the ith neuron in the (n−1)th layer are 0. Although the notation “c_i” is different from c i  for the reason of notation by using subscript, c_i=c i . This similarly applies to c_j to be described later. 
     Next, in step S 30 , the correction unit  12  increments j by one, and the process returns to step S 27 . In a case where j exceeds J n  in step S 27 , the process moves to step S 31 . In step S 31 , the correction unit  12  increments i by one, and the process returns to step S 25 . In a case where i exceeds C n−1  in step S 25 , the process moves to step S 32 . In step S 32 , the correction unit  12  increments n by one, and the process returns to step S 22 . In step S 22 , in a case where n exceeds N, the forward weight correction process ends, and the processing returns to the model reduction process ( FIG.  9   ). 
     In a case where there is no coupling relationship between the ith neuron in the (n−1)th layer and the jth neuron in the nth layer, the processing in steps S 28  and S 29  described above is skipped. In a case where i is not included in the list {c i }, for example, in a case where any of the input weights of the ith neuron in the (n−1)th layer is not 0, the processing in steps S 27  to S 30  described above is skipped. Then, in step S 31  described above, the correction unit  12  may increment i by one, and the process may return to step S 25 . 
     Next, the backward weight correction process will be described with reference to  FIG.  11   . 
     In step S 41 , the correction unit  12  sets the variable n that identifies a hierarchical layer to be processed in the neural network to N−1. Next, in step S 42 , the correction unit  12  determines whether n is smaller than two. In a case where n is greater than or equal to two, the process moves to step S 43 . 
     In step S 43 , the correction unit  12  obtains a list {c j } of the neurons all the output weights of which are 0 in the nth layer. The numbers of the neurons all the output weights of which are 0 out of the neurons in the nth layer are represented by c j . For example, the correction unit  12  adds the numbers of the neurons corresponding to the columns in which all the weights are 0 to the list in the parameter table of the n+1 layer and obtains {c j }. 
     Next, in step S 44 , the correction unit  12  sets j to 1. Next, in step S 45 , the correction unit  12  determines whether j exceeds the maximum value C n  of the numbers of the neurons included in the list {C j }. In a case where j does not exceed C n , the process moves to step S 46 . In step S 46 , the correction unit  12  sets i to 1. Next, in step S 47 , the correction unit  12  determines whether i exceeds I n−1 . In a case where i does not exceed the I n−1 , the process moves to step S 49 . 
     In step S 49 , the correction unit  12  deletes the input weight from the ith neuron in the (n−1)th layer to the jth neuron in the nth layer. For example, the correction unit  12  corrects the weight w i,c_j   (n)  stored in the parameter table of the nth layers to 0. Thus, both the input weight and the output weight of the jth neuron in the nth layer are 0. 
     Next, in step S 50 , the correction unit  12  increments i by one, and the process returns to step S 47 . In a case where i exceeds I n−1  in step S 47 , the process moves to step S 51 . In step S 51 , the correction unit  12  increments j by one, and the process returns to step S 45 . In a case where j exceeds C n  in step S 45 , the process moves to step S 52 . In step S 52 , the correction unit  12  decrements n by one, and the process returns to step S 42 . In step S 42 , in a case where n becomes smaller than two, the backward weight correction process ends, and the processing returns to the model reduction process ( FIG.  9   ). 
     In a case where there is no coupling relationship between the ith neuron in the (n−1)th layer and the jth neuron in the nth layer, the processing in step S 49  described above is skipped. In a case where j is not included in the list {c j }, for example, in a case where any of the output weights of the jth neuron in the nth layer is not 0, the processing in steps S 47  to S 50  described above is skipped. Then, in step S 51  described above, the correction unit  12  may increment j by one, and the process may return to step S 45 . 
     Next, the deletion process will be described with reference to  FIG.  12   . 
     In step S 61 , the deletion unit  16  sets the variable n that identifies a hierarchical layer to be processed in the neural network to 2. Next, in step S 62 , the deletion unit  16  determines whether n exceeds N representing the number of hierarchical layers of the neural network. In a case where n does not exceed N, the process proceeds to step S 63 . 
     In step S 63 , the deletion unit  16  obtains the list {c i } of the neurons all the input weights of which are 0 out of the neurons in the (n−1)th layer. For example, the deletion unit  16  adds the numbers of the neurons corresponding to the rows in which all the weights are 0 to the list in the parameter table of the (n−1)th layer and obtains {c i }. Next, in step S 64 , the deletion unit  16  obtains a list {d i } of the neurons all the output weights of which are 0 out of the neurons in the (n−1)th layer. For example, the deletion unit  16  adds the numbers of the neurons corresponding to the columns in which all the weights are 0 to the list in the parameter table of the nth layer and obtains {d i }. 
     Next, in step S 65 , the deletion unit  16  obtains a list {e i } that includes elements which are shared between the list {c i } and the list {d i }. For example, the list {e i } stores the numbers of the neurons all the input weights and all the output weights of which are 0 out of the neurons in the (n−1)th layer. Next, in step S 66 , the deletion unit  16  obtains a difference set {f i } between the list {e i } and a list {f i } that includes all the numbers of the neurons in the (n−1)th layer. For example, the list {f i } stores the numbers of the neurons that are not deletion target out of the neurons in the (n−1)th layer. 
     Next, in step S 67 , the deletion unit  16  updates so that the weight w h,f_i   (n−1)  becomes w h,i′   (n−1)  in the parameter table of the (n−1)th layer and updates so that the weight w f_i,j   (n)  becomes w i′,j   (n)  in the parameter table of the nth layer. Here, h is the numbers (h=1, 2, . . . ) of the neurons in the (n−2)th layer, and i′ is numbers newly assigned, like 1, 2, . . . , for the numbers included in {f i }. Thus, for example, in a case where {f i }={1,3}, the third row of the parameter table of the (n−1)th layer becomes the second row of the parameter table after the deletion, and the third column of the parameter table of the nth layer is the second column of the parameter table after the deletion. For example, the rows of the parameter table of the (n−1)th layer and the columns of the parameter table of the nth layer corresponding to the neurons of the numbers included in the list {e i } are deleted. 
     Next, in step S 68 , the deletion unit  16  increments n by one, and the process returns to step S 62 . In step S 62 , in a case where n exceeds N, the deletion process ends, and the processing returns to the model reduction process ( FIG.  9   ). 
     As described above, in the neural network, the model reduction apparatus according to the present embodiment identifies, as the deletion targets, first neurons without a coupling from the input layer and second neurons without a coupling to the output layer. The model reduction apparatus compensates for the biases of the first neurons by combining the biases of the first neurons with the biases of the third neurons coupled to the first neurons on the output side. The model reduction apparatus deletes the identified deletion-target neurons from the neural network. Thus, the effect of the size reduction of the machine learning model may be improved while suppressing degradation in the accuracy of the machine learning model. 
     As the process of compensating for the biases of the first neurons, the case is described in which the values obtained by multiplying the biases of the first neurons by the weights between the first neurons and the third neurons are added to the biases of the third neurons according to the above-described embodiment. However, this is not limiting. For example, the values obtained by multiplying values obtained by applying activation functions of the first neurons to the biases of the first neurons by the weights between the first neurons and the third neurons may be added to the biases of the third neurons. In this case, the model reduction apparatus obtains, for example, a layer information table and a function table as illustrated in  FIG.  13    together with the parameter table. In an example illustrated in  FIG.  13   , in the layer information table, activation function names are associated with the layer numbers so as to define the activation function names used in the respective layers. In the function table, the activation function names and function objects used for calculation of the respective activation functions are associated with each other so as to define the activation function names and the function objects. For example, when the bias of the neuron i in the (n−1)th layer is added to the neuron j in the nth layer, the compensation unit of the model reduction apparatus obtains the activation function corresponding to the (n−1)th layer from the layer information table and the function object corresponding to this activation function from the function table. The compensation unit applies the obtained function object to f described below to update the bias b j  of the neuron j. 
         b   j   &lt;−b   j   +w   i,j   f ( b   i ) 
     The above-described embodiment may also be applied to a neural network having a configuration including a convolution layer. In this case, the model reduction apparatus obtains, for example, a layer information table and parameter tables as illustrated in  FIG.  14   . In an example illustrated in  FIG.  14   , in the layer information table, attributes of the layers are associated with the layer numbers so as to define the attributes of the respective layers. In  FIG.  14   , the attribute “conv” represents a convolution layer, and the attribute “fc” represents a fully connected layer. The parameter table of each layer has a format corresponding to the attribute of the layer. The parameter table for the fc layer is similar to the parameter table described according to the above embodiment. As elements of a matrix corresponding to the neurons, weights corresponding to the filter size applied to this layer are stored in the parameter table of the convolution layer.  FIG.  14    illustrates an example in which the filter size is 3×3. In this case, the weight corresponding to the element kth from the left and the Ith from the top of the filter between the ith neuron in the (n−1)th layer and the jth neuron in the nth layer is represented by w i,j,k,l   (n) . For example, the w 2,1,2,2   (2)  corresponds to an element indicated by a dashed line in the parameter table illustrated in  FIG.  14   . 
     In the case of the parameter table of the convolution layer, the model reduction apparatus identifies, as the neurons the input weights and the output weights of which are 0, the neurons corresponding to rows or columns in which all the weights including the weights of the elements of the filter are 0. For example, in the case of a left section of  FIG.  15   , since all the input weights of the third neuron in the (n=2)th layer are 0, the correction unit of the model reduction apparatus identifies, as the deletion target, the third neuron in the (n=2)th layer. As illustrated in a right section of  FIG.  15   , the correction unit corrects the weights in the third column that are the output weights of the third neuron in the (n=2)th layer to 0 in the parameter table of the (n=3)th layer. The deletion unit of the model reduction apparatus deletes the third row including the 3×3 elements of the filter in the parameter table of (n=2)th layer and the third column of the parameter table of (n=3)th layer that are indicated by shaded portions in the right section of  FIG.  15   . In this way, the disclosed technique may reduce the size of a model even in a neural network having a configuration including a convolution layer. In  FIG.  15   , illustration of columns in which the bias values are stored in the parameter tables is omitted. 
     The case is described where the parameter table in which a subset of parameters has been deleted by using the existing model size-reduction technique is input to the model reduction apparatus according to the above-described embodiment. However, a parameter table before the model size reduction may be input. In this case, the model reduction apparatus may also have the function of the existing model size-reduction technique. 
     An example of the relationship between the model size-reduction rate and accuracy in the case where the disclosed technique is applied is described. Here, VGG-19-BN of VGGNet having a layer configuration as illustrated in  FIG.  16    is used as the neural network, and CIFAR-10 is used as the data set.  FIG.  17    illustrates accuracy of a model in a case where the size-reduction rate is 90% and accuracy of a model in a case where the size-reduction rate is 98%. An entire data size is calculated as follows: number of input channels×number of output channels×filter size×4×2. In this calculation expression, “4” represents the amount of information held by a single floating-point type variable in bytes, and “2” is for doubling because a single weight parameter includes two pieces of information, weight information and gradient information. As illustrated in  FIG.  17   , in either case of the reduction rate, there is no change in the accuracy of the model before and after the deletion of the parameter. Thus, it is understood that the influence of the size reduction on the accuracy is suppressed. 
     Regarding the reduced data size in each layer of the neural network in the above example,  FIG.  18    illustrates a case where the reduction rate is 90%, and  FIG.  19    illustrates a case where the reduction rate is 98%. In  FIGS.  18  and  19   , “test_acc” indicates accuracy of prediction by the neural network for test data and is similar to “ACCURACY” in  FIG.  17   . Also, “train_acc” is accuracy of prediction by the neural network for training data. The term “accuracy” refers to a ratio at which a value predicted by the neural network matches a correct answer. 
     Although a form is described in which the model reduction program is stored (installed) in advance in the storage unit according to the above embodiment, this is not limiting. The program according to the disclosed technique may be provided in a form in which the program is stored in a storage medium such as a compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD)-ROM, or a Universal Serial Bus (USB) memory. 
     Regarding the above-described embodiment, the following appendices are further disclosed. 
     All examples and conditional language provided herein are intended for the pedagogical purposes of aiding the reader in understanding the invention and the concepts contributed by the inventor to further the art, and are not to be construed as limitations to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although one or more embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.