STORAGE MEDIUM, INFORMATION PROCESSING METHOD, AND INFORMATION PROCESSING DEVICE

A non-transitory computer-readable storage medium storing an information processing program that causes at least one computer to execute a process, the process includes acquiring a value that indicates a progress status of training for an input-side layer among a plurality of layers included in a machine learning model; and when the value is more than or equal to a threshold value, repeating acquiring each value for a plurality of layers that follows the input-side layer.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2021-080210, filed on May 11, 2021, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are related to a storage medium, an information processing method, and an information processing device.

BACKGROUND

Various training methods for machine learning models are being studied. For example, there is a technique of stopping update of weight information of a layer for which training of a machine learning model has progressed to some extent. In the following description, this technique will be referred to as “existing technique”. Furthermore, stopping the update of target weight information is referred to as “skip”.

FIGS. 9 and 10are diagrams for describing the existing technique.FIG. 9will be described. In an existing machine learning model, the number of layers is about160, but inFIG. 9, description will be given using layers1-0,1-1,1-2,1-3,1-4,1-5, and1-6, for convenience of description. An input-side layer is the layer1-0, and an output-side layer is the layer1-6.

In the existing technique, “Forward Propagation” and “Backward Propagation” are executed for all the layers1-0to1-6in a stage from the start of training for the machine learning model to before the training progresses to some extent, and the weight information for all the layers1-0to1-6are updated.

In the existing technique, in the stage where the training of the machine learning model has progressed to some extent, the update of the weight information for the layer in which the training has progressed is skipped in order from the input-side layer. If the update is skipped from the output-side layer, the training accuracy does not reach target accuracy, but the update is skipped from the input-side layer, the training accuracy can be improved. In the example illustrated inFIG. 9, the update of the weight information for the layers1-0,1-1,1-2is stopped.

Assuming that a total processing amount of Forward Propagation is “1”, a processing amount of Backward Propagation is “2”. For example, in a state where Backward Propagation is not performed at all, a processing speed will be tripled, which is the limit of speedup.

FIG. 10will be described. Learning for the machine learning model is performed in units of epoch. For example, one epoch includes770iterations. The processing of executing “Forward Propagation” and “Backward Propagation” once for the machine learning model and updating the weight information corresponds to one iteration.

Graphs G1and G2inFIG. 10illustrate relationships between each layer and a weight difference in iteration at different stages. The graph G1illustrates the weight difference of each layer in the case where iteration of the first time has been executed. The graph G2illustrates the weight difference of each layer in the case where iteration has been executed several hundred times after the first time. The vertical axis of graphs G1and G2is an axis corresponding to the weight difference, and the horizontal axis is an axis corresponding to the layer.

The weight difference indicates a difference between the weight information in the case where the (n−1)th iteration has been executed and the weight information in the case where the nth iteration has been executed. A layer with a large weight difference indicates that the layer has been trained. A layer with the weight difference less than a threshold value indicates that the layer has not been trained.

In the example illustrated in the graph G1, the weight difference is equal to or larger than a threshold value Th in all the layers (for example, the 0th to 158th layers), and all the layers have been trained. In the example illustrated in the graph G2, the weight differences of the input-side layers Ls1-1are less than the threshold value and have not been trained. On the other hand, the weight differences of the output-side layers Ls1-2are equal to or larger than the threshold value and have been trained.

In the existing technique, a calculation amount and a communication amount for calculating an error gradient are reduced by skipping the processing of updating the weight information of the input-side layers Ls1-1. For example, as illustrated in graph G2′, a processing amount2-1needed for normal one iteration becomes a processing amount2-2, and a processing amount2-3is reduced. In other words, the reduction effect per epoch is also the processing amount2-3. Note that, as will be described below, to specify the progress of training in each layer of the machine learning model, a norm of the weight of each layer is calculated.

U.S. Patent Application Publication No. 2020/0380365 and U.S. Patent Application Publication No. 2020/0285992 are disclosed as related art.

SUMMARY

According to an aspect of the embodiments, a non-transitory computer-readable storage medium storing an information processing program that causes at least one computer to execute a process, the process includes acquiring a value that indicates a progress status of training for an input-side layer among a plurality of layers included in a machine learning model; and when the value is more than or equal to a threshold value, repeating acquiring each value for a plurality of layers that follows the input-side layer.

DESCRIPTION OF EMBODIMENTS

The above-described existing technique has a problem that calculation cost is high.

In the existing technique, the norm of the weight of each layer is calculated in order to specify the progress of training of each layer of the machine learning model. Meanwhile, in the case of updating the weight information of all the layers regardless of the progress of training for the machine learning model, the norm calculation of the weight of each layer is not needed.

Here, the technique of updating the weight information of all the layers at all times regardless of the progress of training for the machine learning model will be referred to as “another existing technique” in order to be distinguished from the existing technique (gradient skip technique).

FIG. 11is a graph for describing a problem of the existing technique. The vertical axis ofFIG. 11is an axis corresponding to the processing amount of images (samples) per unit time (sec), and the horizontal axis is an axis corresponding to epoch. The processing amount of images per unit time is simply referred to as a “processing amount”.

Line5aillustrates the relationship between the processing amount and epoch of the another existing technique. Line5billustrates the relationship between the processing amount and epoch of the existing technique.FIG. 11illustrates an example of a case where the input-side 0th layer to 30th layer are skipped with a certain threshold value among the 150 layers of the machine learning model.

In the existing technique, the norm of the weight of the layers150is calculated for each iteration in 50 convolutional layers (hereinafter Cony layers) in the period of about 0 to 40 epochs. Therefore, as illustrated in (1) ofFIG. 11, the processing amount of the existing technique is smaller than that of the another existing technique by 500 samples/sec.

Next, in the existing technique, in the period of about 40 to 50 epochs, the weight difference of the Cony layers of 30 layers (10 layers for the Conv layers) gradually reaches the threshold value, the number of layers to be skipped gradually increases, and the norm calculation is skipped. As the calculation amount of norm decreases in this manner, the processing amount gradually increases in the existing technique, as illustrated in (2) ofFIG. 11.

Next, in the existing technique, it is possible to skip 30 layers during the transition period of 50 epochs, which reduces the calculation amount, but the norm calculation for the remaining Conv layers of 40 layers for the next skip remains. The processing amount is about 9200 samples/sec. As illustrated in (3) ofFIG. 11, the improvement is only about 100 samples/sec as compared with the another existing technique.

As described inFIG. 11, in the existing technique, the norm calculation for determining the progress of training occurs in order to determine whether the layer can be skipped, and the calculation cost is generally higher than in the another existing technique, and the processing amount may decrease on the whole.

In one aspect, an object of the present embodiment is to provide an information processing program, an information processing method, and an information processing device capable of reducing the calculation cost.

The calculation cost can be reduced.

Embodiments of an information processing program, an information processing method, and an information processing device disclosed in the present application are hereinafter described in detail with reference to the drawings. Note that the present embodiment is not limited to the embodiments.

An information processing device according to the present embodiment calculates a weight difference and specifies whether training of a target layer of a machine learning model has progressed. In the following description, among a plurality of layers included in the machine learning model, the layer for which the weight difference is to be calculated is appropriately referred to as “target layer”.

The weight difference is defined by the equation (1). The subscript “I” in the equation (1) corresponds to the number of iterations. For example, “WI+1−WI” indicates the weight difference between weight information of the (I+1)th iteration and weight information of the Ith iteration. In the equation (1), constants are preset for “LR”, “Decay”, and “mom”.

ΔWIin the equation (1) represents a difference between a weight of the previous iteration and a weight of the iteration of this time in tensor in the target layer. WIin the equation (1) represents the weight updated by the iteration of this time in tensor in the target layer. VI−1is a tensor indicating momentum. For example, the equation (2) defines a relationship between VIand VI−1.

The information processing device calculates a norm (g_weight_norm) of ΔWI, a norm (weight_norm) of WI, and a norm (momentum_norm) of VI, respectively, in order to convert the value of the equation (1) into a scalar value comparable to a threshold value. The norm (g_weight_norm) of ΔWIis calculated by the equation (3). The norm of WIis calculated by the equation (4). The norm of VIis calculated by the equation (5).

FIG. 1is a graph illustrating a tendency of evaluation indexes. The vertical axis of graph G10inFIG. 1is an axis corresponding to the weight difference, and the horizontal axis is an axis corresponding to epoch. Line10ainFIG. 10illustrates the relationship between the weight difference and epoch in a case where the 6th layer (L6) of the machine learning model is the target layer. Line10billustrates the relationship between the weight difference and epoch in a case where the 36th layer (L36) is the target layer. Line10cillustrates the relationship between the weight difference and epoch in a case where the 75th layer (L75) is the target layer. Line10dillustrates the relationship between the weight difference and epoch in a case where the 132nd layer (L132) is the target layer. When the layers are arranged from an input side to an output side of the machine learning model, the order is L6, L36, L75, and L132.

A threshold value is set for each layer, and in a case where the weight difference has reached the threshold value, calculation of the weight difference of the target layer is skipped. For example, in L6, the weight difference reaches a threshold value Th6in 10 epochs. In L36, the weight difference reaches a threshold value Th36in 22 epochs. In L75, the weight difference reaches a threshold value Th75in 35 epochs. In L132, the weight difference reaches a threshold value Th75in 46 epochs. In other words, the weight difference reaches the threshold value from the input-side layer.

Next, an example of processing of the information processing device according to the present embodiment will be described. In the information processing device according to the present embodiment, the initial target layer is only one layer. Next, the information processing device repeatedly executes the processing of calculating the weight difference for a plurality of layers as the target layers, following the layer with the weight difference having reached the threshold value, after the weight difference of the target layer has reached the threshold value. In the present embodiment, the value of the weight difference having become less than the threshold value is described that the weight difference has reached the threshold value.

FIG. 2is a table for describing processing of the information processing device according to the present embodiment. InFIG. 2, description will be given using the 0th layer (L0), the 3rd layer (L3), the 6th layer (Lb), the 9th layer (L9), and the 12th layer (L12) among the plurality of layers included in the machine learning model. Furthermore, description will be given using the 15th layer (L15), the 18th layer (L18), the 21st layer (L21), the 24th layer (L24), and the 27th layer (L27) among the plurality of layers included in the machine learning model. For example, the layers (L0,3,6,9,12,15,18,21,24, and27) corresponds to the Conv layers. In the description ofFIG. 2, calculating the weight difference is referred to as “norm calculation”.

The information processing device inputs training data into the machine learning model, executes Forward Propagation and Backward Propagation, and starts training the machine learning model. As illustrated inFIG. 2, the information processing device executes the norm calculation of the target layer for each epoch.

The norm calculation (initial norm calculation) in 1 epoch will be described. The information processing device starts the norm calculation for L0as the target layer.

The norm calculation in 2 epochs to (n−1) epochs will be described. The information processing device continues the norm calculation for L0as the target layer.

The norm calculation in n epochs will be described. n is a natural number. When specifying that the weight difference of L0has reached the threshold value, the information processing device starts the norm calculation for the three layers “L3, L6, and L9” on the output side with respect to the layer that has reached the threshold value. At the stage of n epochs, training of each layer has progressed to some extent.

FIG. 3is a graph for describing a relationship between epoch and a training status of each layer. InFIG. 3, description will be given using the weight differences of L0, L3, L6, and L9. Note that the weight differences of L3, L6, and L9are given for description, and it is assumed that the norm calculation for L3, L6, and L9has not been performed at or before (n−1) epochs.

In the example illustrated inFIG. 3, the weight difference of LU has reached the threshold value at 37 epochs. Furthermore, training of L3, L6, and L9has also progressed at 37 epochs, and in the example ofFIG. 3, the weight differences of L3, L6, and L9have also reached the threshold value. Therefore, in the case where the norm calculation is moved from L0to L3, L6, and L9, the weight differences of L3, L6, and L9will also reach the threshold value soon.

The description returns to the description ofFIG. 2. The norm calculation in (n+1) epochs will be described. The information processing device skips the norm calculation of L0. When specifying that the weight differences of L3and L6have reached the threshold value, the information processing device starts the norm calculation for “L12and L15” on the output side with respect to the layers that have reached the threshold value. The information processing device continues the norm calculation for L9.

The norm calculation in (n+2) epochs will be described. The information processing device skips the norm calculation of L0, L3, and L6. When specifying that the weight difference of L9has reached the threshold value, the information processing device starts the norm calculation for “L18” on the output side with respect to the layer that has reached the threshold value. The information processing device continues the norm calculation for L12and L15.

The norm calculation in (n+3) epochs will be described. The information processing device skips the norm calculation of L0, L3, L6, and L9When specifying that the weight difference of L18has reached the threshold value, the information processing device starts the norm calculation for “L21” on the output side with respect to the layer that has reached the threshold value.

The norm calculation in (n+4) epochs will be described. The information processing device skips the norm calculation of L0, L3, L6, and L9. The information processing device waits for stop of the norm calculation of L18because the norm calculation of the layers L12and15on the input side with respect to L18with the weight difference having reached the threshold value has not been skipped. When specifying that the weight differences of L12and L15have reached the threshold value, the information processing device starts the norm calculation for “L24and L27” on the output side with respect to the layers that have reached the threshold value. The information processing device continues the norm calculation for L21.

The norm calculation in (n+5) epochs will be described. The information processing device skips the norm calculation of L0, L3, L6, L9, L12, L15, and L18. The information processing device continues the norm calculation for L21, L24, and L27. Description of the norm calculation in (n+6) epochs is omitted.

As described above, the information processing device according to the present embodiment can narrow down the target layer for which the norm calculation is to be executed and reduce the calculation cost when training the machine learning model.

Next, one example of a configuration of the information processing device according to the present embodiment will be described.FIG. 4is a functional block diagram illustrating a configuration of the information processing device according to the present embodiment. As illustrated inFIG. 4, an information processing device100includes a communication unit110, an input unit120, a display unit130, a storage unit140, and a control unit150.

The communication unit110receives various data from an external device via a network. The communication unit110is an example of a communication device. For example, the communication unit110may also receive training data141or the like, which will be described below, from an external device.

The input unit120is an input device that inputs various types of information to the control unit150of the information processing device100. The input unit120corresponds to a keyboard, a mouse, a touch panel, and the like,

The display unit130is a display device that displays information output from the control unit150.

The storage unit140has training data141and a machine learning model142. The storage unit140corresponds to a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk drive (HDD).

The training data141is data used when training of the machine learning model142is executed. For example, the training data141has a plurality of pairs of input data and correct answer data.

The machine learning model142is model data corresponding to a neural network having a plurality of layers.

The control unit150includes a forward propagation (FP) processing unit151, a backward propagation (BP) processing unit152, and a selection unit153. The control unit150is implemented by a central processing unit (CPU), a graphics processing unit (GPU), a hard-wired logic such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA), or the like.

The FP processing unit151executes Forward Propagation for the machine learning model142, For example, the FP processing unit151inputs input data of the training data141to an input layer of the machine learning model142, and calculates an output value of the machine learning model142. The FP processing unit151outputs output value information to the BP processing unit152.

The BP processing unit152executes Backward Propagation for the machine learning model142. For example, the BP processing unit152calculates an error between an output value output from an output layer of the machine learning model142and the correct answer data of the training data141, and updates the weight information of each layer of the machine learning model142so that the error becomes small by error back propagation.

Furthermore, when receiving a notification of the target layer from the selection unit153to be described below, the BP processing unit152executes the norm calculation for the target layer among all the layers included in the machine learning model142. For example, the BP processing unit152calculates the norm of ΔWI, the norm of W, and the norm of VIon the basis of the above equations (3), (4), and (5), and outputs a calculation result of each target layer to the selection unit153.

When receiving selection of a layer for which the norm calculation is to be skipped from the selection unit153, the BP processing unit152skips the norm calculation for the target layer for which the selection is accepted. Furthermore, the BP processing unit152stops the error propagation of the target layer for which the norm calculation is to be skipped and layers on the input side with respect to the target layer.

The selection unit153selects the target layer for which the norm calculation is to be executed on the basis of the result of the norm calculation output from the BP processing unit152, and notifies the BP processing unit152of the selected target layer. The selection unit153sets only one layer as an initial target layer. For example, the selection unit153selects L0as the initial target layer and outputs the selected target layer to the BP processing unit152.

When acquiring the calculation result of the norm calculation of the target layer from the BP processing unit152, the selection unit153calculates the weight difference of the target layer on the basis of the equation (1) and determines whether the weight difference has reached the threshold value. In the case where the weight difference of the target layer has reached the threshold value, the selection unit153notifies the BP processing unit152that the norm calculation for the target layer with the weight difference having reached the threshold value is skipped.

In the case where the weight difference of the initially selected L0has reached the threshold value, the selection unit153selects a plurality of layers (L3, L6, and L9) on the output side with respect to L0as target layers and outputs the target layers to the BP processing unit152, as described inFIG. 2. The selection unit153selects the target layer so that the number of the target layers becomes “M (for example, 3)” after the weight difference of the initial target layer reaches the threshold value. When selecting the target layer, the selection unit153preferentially selects a layer with the weight difference having not reached the threshold value and closer to the input side.

By the way, the selection unit153waits for skipping L18in the case where the weight difference of L18has reached the threshold value and the weight differences of L12and L15have not reached the threshold value among the plurality of layers for which the norm calculation is performed, as described in (n+3) epochs ofFIG. 2. The selection unit153outputs that L18is to be skipped after the weight differences of L12and L15reach the threshold value to the BP processing unit152.

Next, an example of a processing procedure of the information processing device100according to the present embodiment will be described.FIG. 5is a flowchart illustrating a processing procedure of the information processing device according to the present embodiment. As illustrated inFIG. 5, the FP processing unit151and the BP processing unit152of the information processing device100start training (Forward Propagation and Backward Propagation) for the machine learning model142(step S101).

The selection unit153of the information processing device100selects input-side one layer of the machine learning model142as the target layer (step S102). The BP processing unit152executes the norm calculation of the target layer (step S103).

The selection unit153specifies whether the weight difference has reached the threshold value on the basis of the result of the norm calculation (step S104). In the case where the target layer with the weight difference having reached the threshold value is present (step S105, Yes), the selection unit153moves onto step S106. On the other hand, in the case where the target layer with the weight difference having reached the threshold value is not present (step S105, No), the selection unit153moves onto step S108.

The BP processing unit152skips the norm calculation of the target layer with the weight difference having reached the threshold value (step S106). The selection unit153selects the target layers so that the number of layers for which the norm calculation is to be executed are M (for example, three) (step S107).

In the case of terminating the training (step S108, Yes), the information processing device100terminates the processing. On the other hand, in the case of not terminating the training (step S108, No), the information processing device100proceeds to the training of the next epoch (step S109) and proceeds to step S103.

Next, effects of the information processing device100according to the present embodiment will be described. The information processing device100executes the norm calculation for only one layer as the initial target layer, Next, the information processing device100repeatedly executes the processing of calculating the weight difference for a plurality of layers as the target layers, following the layer with the weight difference having reached the threshold value, after the weight difference of the target layer has reached the threshold value. In this way, the information processing device100can reduce the calculation cost by narrowing down the target layers for which the norm calculation is to be executed.

The information processing device100can further reduce the calculation cost by skipping the norm calculation for the target layer with the weight difference having reached the threshold value.

In the case where values of a first layer and a second layer on the output side with respect to the first layer have reached the threshold value, or in the case where the calculation of the value for the first layer has been skipped (stopped) and the value of the second layer has reached the threshold value, among a plurality of layers, the information processing device100skips the calculation of the value for the second layer. Furthermore, in the case where the value of the second layer has reached the threshold value and the value of the first layer has not reached the threshold value, the information processing device100continues (waits for skipping) the calculation of the value for the second layer until the value of the first layer reaches the threshold value. As a result, it is possible to skip the layers in order from an input-side layer, and the training accuracy of the machine learning model142can be improved.

FIG. 6is graphs illustrating an effect of the information processing device according to the present embodiment. The vertical axis of graph G20inFIG. 6is an axis corresponding to the processing amount of images (samples) per unit time (sec), and the horizontal axis is an axis corresponding to epoch. The processing amount of images per unit time is simply referred to as a “processing amount”. Line5aof the graph G20illustrates the relationship between the processing amount and epoch of another existing technique. Line5billustrates the relationship between the processing amount and epoch of the existing technique. Line5cillustrates the relationship between the processing amount and epoch of the information processing device according to the present embodiment.

The vertical axis of graph G30inFIG. 6is an axis corresponding to the number of target layers for norm calculation, and the horizontal axis is an axis corresponding to epoch. The number of target layers for norm calculation is simply described as “the number of target layers”. Line6aof the graph G30illustrates the relationship between the number of target layers and epoch of the another existing technique. Line6billustrates the relationship between the number of target layers and epoch of the existing technique. Line6cillustrates the relationship between the number of target layers and epoch of the information processing device according to the present embodiment.

From 0 to about 50 epochs will be discussed. Compared with the existing technique (line6b), the number of target layers (line6c) of the information processing device100is 1/50. Thereby, the processing amount (line5c) of the information processing device100is comparable with the processing amount (line5a) of the another existing technique.

50 epochs and subsequent epochs will be discussed. In the information processing device100, the weight difference of each layer sequentially reaches the threshold value and skipping is started, the processing amount (line5c) increases due to the calculation amount for the error gradient calculation and the stop of back propagation. Even after skipping, since the total number of norm calculations is set to 3, the speed is not reduced by the processing for 37 layers and the processing amount is improved.

In the example illustrated inFIG. 6, skipping has been stopped at the 30 layers, but the processing amount further increases by skipping the subsequent layers as well.

By the way, in the machine learning model142, the mass of the number of elements of each layer is divided into four stages (four types), any layer (the last layer L36, L75, or L132of each stage) may also be selected for each mass and the norm calculation may also be performed.

FIG. 7is a diagram for describing another processing of the information processing device. In the example illustrated inFIG. 7, the machine learning model142is divided into a plurality of blocks. Each block includes Batch Norm, ReLU, convolution, and the like. The information processing device has selected layers Ib1, Ib2, Ib3, Ib4, Ib5, and Ib6as the target layers. The information processing device100may also select Batch Norm or the like as the target layer instead of the Conv layer and execute the norm calculation.

Next, an example of a hardware configuration of a computer that implements functions similar to the information processing device100described in the above embodiment will be described.FIG. 8is a diagram illustrating an example of a hardware configuration of a computer that implements functions similar to the information processing device.

As illustrated inFIG. 8, a computer200includes a CPU201that executes various types of calculation processing, an input device202that receives input of data from a user, and a display203. Furthermore, the computer200includes a communication device204that receives data from an external device and an interface device205connected with various devices. The computer200includes a RAM206that temporarily stores various types of information, and a hard disk device207. Then, each of the devices201to207is connected to a bus208.

The hard disk device207has an FP processing program207a,a BP processing program207b,and a selection program207c.The CPU201reads the FP processing program207a,the BP processing program207b,and the selection program207cand expands them in the RAM206.

The FP processing program207afunctions as an FP processing process206a.The BP processing program207bfunctions as a BP processing process206b.The selection program207cfunctions as a selection process206c.

Processing of the FP processing process206acorresponds to the processing of the FP processing unit151. Processing of the BP processing process206bcorresponds to the processing of the BP processing unit152. Processing of the selection process206ccorresponds to the processing of the selection unit153.

Note that the programs207ato207cdo not need to be stored in the hard disk device207beforehand. For example, the programs are stored in a “portable physical medium” such as a flexible disk (FD), a compact disc read only memory (CD-ROM), a digital versatile disc (DVD) disk, a magneto-optical disk, or an integrated circuit (IC) card to be inserted in the computer200. Then, the computer200may also read the programs207ato207cand execute the programs.