Patent ID: 12223694

EXAMPLE EMBODIMENTS

The preferred example embodiments of the present invention will now be described below with reference to the attached drawings.

[Device Configuration]

FIG.1is a block diagram illustrating a hardware configuration of a learning device according to example embodiments. The learning device1may be a computer, and includes a processor1and a memory2. The learning data with correct labels are inputted to the learning device1. The learning device1is a device to execute learning using knowledge distillation, and performs the learning of a student model using the learning data and a teacher model.

Specifically, the memory2stores the internal parameters forming the student model and the teacher model, as well as the temperature parameter used for knowledge distillation. The processor1performs learning by applying the learning data to the student model and the teacher model based on each parameter stored in the memory2and updating each parameter. Further, the processor1automatically updates the temperature parameter based on the information (hereinafter referred to as “estimation information”) obtained by the estimation processing by the student model and the teacher model.

[Basic Configuration]

FIG.2is a block diagram illustrating a basic functional configuration of the learning device according to the example embodiments. As illustrated, the learning device1includes an estimation unit5, an estimation unit6, and a temperature calculation unit7. The estimation unit5is an estimation unit forming the student model, and outputs the estimation information by performing estimation processing of the learning data using the temperature parameter T. On the other hand, the estimation unit6is an estimation unit forming the teacher model, and outputs the estimation information by performing the estimation processing of the learning data using the temperature parameter T. The temperature calculation unit7calculates the temperature parameter T based on the estimation information generated by the estimation unit5and the estimation information generated by the estimation unit6, and supplies the temperature parameter T to the estimation units5and6. Thus, the temperature parameter T used by the estimation units5and6are automatically updated. In an example, a logit or estimation result generated by the student model and the teacher model is used as the estimation information. In another example, as the estimation information, the loss of the estimation result of the student model with respect to the correct labels, and the loss of the estimation result of the student model with respect to the estimation result of the teacher model are used.

First Example Embodiment

Next, a description will be given of a first example embodiment. The first example embodiment updates the temperature parameter based on the loss of the estimation result generated by the student model with respect to the correct labels, and the loss of the estimation result generated by the student model with respect to the estimation result generated by the teacher model. The loss is an example of the estimation information of the present invention.

(Functional Configuration)

FIG.3is a block diagram illustrating a functional configuration of the learning device100according to the first example embodiment. The learning data is inputted to the learning device100. The learning data is a set of learning data X which shows features to be learned such as character strings and images, and its correct label Y. For example, when a DNN for determining whether an image shows a dog or a cat is learned, the learning data X is an image of a dog, and the correct label Y is a vector of binary value having “1” for the correct answer class to which the image belongs and “0” for the other incorrect answer classes, such as (Dog class 1, Cat class 0).

The learning device100roughly includes an estimation unit10a, an estimation unit10band an optimization unit30. The estimation unit10acorresponds to the DNN forming the student model, and includes a logit calculator11aand an activator12a. The student model is the model to be learned. The estimation unit10aoutputs the estimation result Ya for the inputted learning data X. Specifically, the logit calculator11aconverts the inputted learning data X to the logit ya based on the internal parameters PA. The activator12aconverts the logit ya to the estimation result Ya, which is a probability distribution, based on a predetermined function such as a softmax function shown in Equation (1).

s⁡(zi)=exp⁡(zi)∑exp⁡(zj)(1)

The activator12acalculates the estimation result Ya using the temperature parameter T, and the details thereof will be described later. The estimation result Ya is generally a vector having a value other than “0” for the incorrect answer class, such as (Dog class 0.8, Cat class 0.2). The estimation result Ya is supplied to the optimization unit30.

On the other hand, the estimation unit10bcorresponds to the DNN forming the teacher model, and includes a logit calculator11band an activator12b. The teacher model is a model that has already been learned, and is generally a highly accurate DNN having more parameters than the student model. The estimation unit10boutputs the estimation result Yb for the inputted learning data X. Specifically, the logit calculator11bconverts the inputted learning data X to the logit yb based on the internal parameters PB. The activator12bconverts the logit yb to the estimation result Yb, which is a probability distribution, based on a predetermined function such as a softmax function given as the above Equation (1). The activator12balso calculates the estimation result Yb using the temperature T, and the details thereof will be described later. Similarly to the estimation result Ya, the estimation result Yb is a vector having a value other than “0” for the incorrect answer class. The estimation result Yb is supplied to the optimization unit30.

The optimization unit30includes a loss calculator31, a weighted average calculator32, a parameter updater33and a temperature updater34. The estimation result Ya generated by the estimation unit10a, the estimation result Yb generated by the estimation unit10band the correct label Y of the learning data are inputted to the loss calculator31. The loss calculator31calculates the loss La of the estimation result Ya with respect to the correct label Y and calculates the loss Lb of the estimation result Ya with respect to the estimation result Yb, based on a predetermined function such as the categorical cross entropy given by Equation (2). “Loss” indicates how far the estimation result Ya and the correct label Y, or the estimation result Ya and the estimation result Yb are apart.
C=−Yln(Y′)  (2)

If the number of learning data is not balanced among classes, the effect of correcting the balance may be added to the losses La and Lb. Specifically, Equation (3) multiplies the loss for each class by the coefficient wifor correction. For example, the coefficient wiis calculated by Equation (4) or (5).

C=-∑i⁢⁢wi*Yi⁢log⁡(Yi′)(3)wi=1/ni(4)wi=(1/ni)/(∑j⁢1/nj)(5)

Here, niis the total number of learning data belonging to the i-th class. In another example, the sum of the estimation result Yb for each class, calculated by the teacher model for all the learning data or for mini-batches, may be used as the coefficient wi, instead of the total number niof the learning data.

The weighted average calculator32calculates the weighted average Lav of the losses La and Lb using a predetermined weight. The parameter updater33updates the internal parameters PA of the logit calculator11aso that the weighted averaged loss Lav is reduced based on a predetermined function such as a gradient descent method given by Equation (6), and supplies the updated parameters PA to the logit calculator11a.

P(n)=P(n+1)-ϵ⁢∂C∂P(6)

Thus, the internal parameters PA of the logit calculator11aare updated. It is noted that the internal parameters PB of the logit calculator11bof the estimation unit10bare not updated.

The temperature updater34updates the temperature parameter T to be used by the activator12aof the estimation unit10aand the activator12bof the estimation unit10b. Here, knowledge distillation process using a temperature T by the activators12aand12bwill be described in detail. The estimation units10aand10boutput the estimation results Ya and Yb, respectively. However, since the estimation result Yb is outputted by the learned high-accuracy teacher model, the element of the vector becomes close to the binary value, such as (Dog class 0.999, Cat class 0.001), so that the learning process is almost unchanged from the case where only the correct label Y such as (Dog class 1, Cat class 0) is used. Therefore, a hyper-parameter T called “temperature parameter” is newly introduced. The temperature parameter T is introduced into the activation function of the activator as shown in Equation (7), and generally uses T≥1.

s⁡(zi)=exp⁡(zi/T)∑exp⁡(zj/T)(7)

In this way, the estimation result Yb, which is actually (Dog class 0.999, Cat class 0.001), becomes a smooth distribution like (Dog class 0.9, Cat class 0.1). Therefore, the numerical value of “Cat class 0.1” affects the learning process, making it possible to efficiently learn the student model. That is, the temperature parameter T is introduced to adjust mismatch of the digits of the correct answer class and the incorrect answer class among the correct labels outputted by the teacher model.

Similarly to the parameter updater33optimizing the internal parameters PA of the student model using the losses, the temperature updater34sequentially optimizes the temperature parameter T using the losses. Specifically, the temperature updater34updates the temperature parameter T so that the weighted averaged loss Lay is reduced. For example, the temperature updater34may update the temperature parameter T by Equation (8).

T(n+1)=T(n)-ϵ⁢∂C∂T(8)

In Equation (8), “ε” is a constant that performs the same function as the learning rate “ε” in Equation (6). For “ε”, the same value may be used during learning, or the value may be changed according to the progress of learning as in the plateau. Further, for “ε”, the same value as in Equation (6) may be used, or a different value may be used. While Equation (8) shows a general gradient descent method, regularization techniques such as weight decay or weight upper bound may be used together, as in the case of the internal parameters P of the model. Further, the optimization method that the temperature updater34uses to update the temperature parameter T is not limited to the gradient descent method of Equation (8), and other optimization methods such as momentum, adagrad, adadelta or adam may be used.

Thus, the optimizing unit30repeats the update of the internal parameters PA of the estimation unit10aand the temperature parameter T, until the update amount becomes sufficiently small. When the update amount becomes smaller than the predetermined value, the learning ends. When the student model after learning is used for inference, the temperature parameter T is set as T=1. In other words, the temperature parameter T is set as T≥1 at the time of learning and T=1 at the time of inference.

(Processing Flow)

Next, the flow of the learning process according to the first example embodiment will be described.FIG.4is a flowchart of the learning processing according to the first example embodiment. This process is realized by the processor1shown inFIG.1, which executes a program prepared in advance.

When the learning data X is inputted to the learning device100, in the estimation unit10acorresponding to the student model, the logit calculator11acalculates the logit ya of the learning data X (step S11). Next, the activator12acalculates the estimation result Ya from the logit ya using the temperature parameter T as described above (step S12). Similarly, in the estimation unit10bcorresponding to the teacher model, the logit calculator11bcalculates the logit yb of the learning data X (step S13). Next, the activator12bcalculates the estimation result Yb from the logit yb using the temperature parameter T as described above (step S14).

Next, in the optimizing unit30, the loss calculator31calculates the loss La of the estimation result Ya with respect to the correct label Y, and the loss Lb of the estimation result Ya with respect to the estimation result Yb (step S15). Next, the weighted average calculator32calculates the weighted average Lay of the losses La and Lb (step S16).

Next, the parameter updater33updates the internal parameters PA of the logit calculator11abased on the weighted averaged loss Lay (step S17). Next, the temperature updater34updates the temperature parameter T used by the activators12aand12bbased on the weighted averaged loss Lay (step S18).

The learning device100repeats the above processing until the update amount of the internal parameters PA by the parameter updater33becomes smaller than the predetermined value, and ends the processing when the update amount becomes smaller than the predetermined value. The student model is defined by the internal parameters PA set in the logit calculator11aand the temperature parameter T set in the activator12awhen the processing ends. Then, at the time of inference, the inference processing is executed by the student model in which the temperature parameter T is set as T=1.

(Modification)

In the above-described first example embodiment, the temperature updater34updates the temperature parameter T using both the loss La of the estimation result Ya with respect to the correct label Y and the loss Lb of the estimation result Ya with respect to the estimation result Yb. Instead, the temperature updater34may update the temperature parameter T using one of the loss La of the estimation result Ya with respect to the correct label Y and the loss Lb of the estimation result Ya with respect to the estimation result Yb.

Example

In one example of the first example embodiment, the initial value of the temperature parameter T is set to “1”. The temperature updater34updates the temperature parameter T at the same timing as the parameter updater33updates the internal parameters PA of the logit calculator11a. The temperature updater34updates the temperature parameter T using the stochastic gradient descent method, and the learning rate is set as: ε=1.0×10−4.

The above process is repeated for one sample or a set of multiple samples (referred to as a “mini-batch”) of similar learning data. The repetition is ended when the update amount of the temperature parameter T or the update amount of the internal parameters PA of the logit calculator11abecomes sufficiently small, or the number of repetition reaches a predetermined number of repetitions (e.g., 100 times). Thereafter, a similar process is executed for a sample or mini-batch of another learning data, and the process is repeated until the update amount of the internal parameters PA of the logit calculator11abecomes sufficiently small. At this time, the initial value of the temperature parameter T is set to “1” every time the sample or the mini-batch is changed.

Second Example Embodiment

Next, a description will be given of a second example embodiment. The second example embodiment updates the temperature parameter based on the logits calculated by the logit calculators in the estimation units. The logit is an example of the estimation information of the present invention. The hardware configuration of the learning device200according to the second example embodiment is the same as that of the learning device100according to the first example embodiment shown inFIG.1.

(Functional Configuration)

FIG.5is a block diagram illustrating a functional configuration of the learning device200according to the second example embodiment. As can be seen in comparison withFIG.3, the learning device200has the same basic configuration as the learning device100according to the first example embodiment, and includes the estimation unit10a, the estimation unit10band the optimization unit30. However, the learning device200according to the second example embodiment includes a temperature calculator21instead of the temperature updater34according to the first example embodiment.

To the temperature calculator21, the logit ya outputted by the logit calculator11aof the estimation unit10aand the logit yb outputted by the logit calculator11bof the estimation unit10bare inputted. The temperature calculator21calculates an appropriate temperature parameter T using the logits ya and yb based on a predetermined function. As described above, the temperature parameter T has been introduced to adjust the mismatch of the digits of the correct answer class and the incorrect answer class among the correct labels outputted by the teacher model. Since the intensity of the mismatch varies with the learning data, the temperature parameter T is changed accordingly. Specifically, the temperature calculator21sets a large value to the temperature parameter T when the mismatch of the logits ya and yb is large, and sets a value close to “1” to the temperature parameter T when the mismatch is small. This makes it possible to learn any learning data equally.

The temperature calculator21supplies the calculated temperature parameter T to the activators12aand12b. The activator12acalculates the estimation result Ya based on the logit ya inputted from the logit calculator11aand the temperature parameter T, and supplies the estimation result Ya to the loss calculator31of the optimization unit30. Also, the activator12bcalculates the estimation result Yb based on the logit yb inputted from the logit calculator11band the temperature parameter T, and supplies the estimation result Yb to the loss calculator31of the optimization unit30.

In the optimization unit30, the loss calculator31calculates the loss La of the inputted estimation result Ya with respect to the correct labels and the loss Lb of the estimation result Ya with respect to the estimation result Yb, and the weighted average calculator32calculates the weighted average loss Lav of the losses La and Lb. Then, the parameter updater33updates the internal parameters PA of the logit calculator11ain the estimation unit10abased on the weighted averaged loss Lay.

If the number of learning data is not balanced among classes, the effect of correcting the balances may be added to the losses La and Lb. Specifically, Equation (3) multiplies the loss for each class by the coefficient wifor correction. For example, the coefficient wiis calculated by Equation (4) or (5) described above. Here, niis the total number of learning data belonging to the i-th class. In another example, the sum of the estimation result Yb for each class, calculated by the teacher model for all the learning data or for mini-batches, may be used as the coefficient wi, instead of the total number niof the learning data.

(Processing Flow)

Next, the flow of the learning process according to the second example embodiment will be described.FIG.6is a flowchart of the learning process according to a second example embodiment. This process is realized by the processor1shown inFIG.1, which executes a program prepared in advance.

When the learning data X is inputted to the learning device200, in the estimation unit10acorresponding to the student model, the logit calculator11acalculates the logit ya of the learning data X (step S21). In the estimation unit10bcorresponding to the teacher model, the logit calculator11bcalculates the logit yb of the learning data X (step S22). Next, the temperature calculator21determines the temperature parameter T from the logits ya and yb based on a predetermined function (step S23). The determined temperature parameter T is supplied to the activators12aand12b.

Next, the activator12acalculates the estimation result Ya from the logit ya using the temperature parameter T, and the activator12bcalculates the estimation result Yb from the logit yb using the temperature parameter T (step S24).

Next, in the optimizing unit30, the loss calculator31calculates the loss La of the estimation result Ya with respect to the correct labels Y, and the loss Lb of the estimation result Ya with respect to the estimation result Yb (step S25). Next, the weighted average calculator32calculates the weighted average Lay of the loss La and the loss Lb (step S26). Next, the parameter updater33updates the internal parameters PA of the logit calculator11abased on the weighted averaged loss Lay (step S27).

The learning device200repeats the above-described processing until the update amount of the internal parameters PA by the parameter updater33becomes smaller than the predetermined value, and ends the processing when the update amount becomes smaller than the predetermined value. The student model is defined by the internal parameters PA set in the logit calculator11aand the temperature parameter T set in the activator12awhen the processing ends. Then, at the time of inference, the inference process is executed by the student model in which the temperature parameter is set as: T=1.

(Modification)

In the second example embodiment described above, the temperature calculator21calculates the temperature parameter T using both the logit ya generated by the estimation unit10acorresponding to the student model and the logit yb generated by the estimation unit10bcorresponding to the teacher model. Instead, the temperature calculator21may calculate the temperature parameter T using one of the logit ya generated by the estimation unit10acorresponding to the student model and the logit yb generated by the estimation unit10bcorresponding to the teacher model.

Example

In one example of the second example embodiment, the temperature parameter T is set to a positive value when the value of the logit outputted by the estimation unit10acorresponding to the student model or the estimation unit10bcorresponding to the teacher model is positive, and is set to a negative value when the value of the logit outputted by the estimation unit10acorresponding to the student model or the estimation unit10bcorresponding to the teacher model is negative. For example, if the value of the logit is negative, the temperature parameter T is set as T=−5, and if the value of the logit is positive, the temperature parameter T is set as T=1. Preferably, the temperature parameter T is determined in the range from −100 to 100. For example, the temperature calculator21sets the temperature parameter T to a value from 0 to 100 when the logit outputted by the estimation unit10acorresponding to the student model is positive, and sets the temperature parameter T to a value from −100 to 0 when the logit is negative. A sigmoid function is used for the activation function of the activators12aand12b.

Third Example Embodiment

Next, a description will be given of a third example embodiment. The third example embodiment updates the temperature parameter based on the estimation result calculated by the estimation unit. The estimation result is an example of the estimation information of the present invention. The hardware configuration of the learning device300according to the third example embodiment is the same as that of the learning device100according to the first example embodiment shown inFIG.1.

(Functional Configuration)

FIG.7is a block diagram illustrating a functional configuration of the learning device300according to the third example embodiment. As can be seen in comparison withFIG.5, the learning device300includes activators22aand22bin addition to the learning device200according to the second example embodiment. The configuration of the estimation units10aand10band the optimization unit30is the same as that of the learning device200according to the second example embodiment.

The temperature calculator21calculates the temperature parameter T from the estimation results Ya and Yb supplied from the estimation units10aand10bbased on a predetermined function, and supplies the temperature parameter T to the activators22aand22b. The activator22acalculates the estimation result Y′a from the logit ya outputted by the logit calculator11abased on a predetermined function and the temperature parameter T, and supplies the estimation result Y′a to the loss calculator31of the optimization unit30. Similarly, the activator22bcalculates the estimation result Y′b from the logit yb outputted by the logit calculator11bbased on a predetermined function and the temperature parameter T, and supplies the estimation result Y′b to the loss calculator31of the optimization unit30.

The configuration of the optimizing unit30is the same as the second example embodiment. Namely, the loss calculator31calculates the loss La of the estimation result Y a with respect to the correct labels Y and the loss Lb of the estimation result Y′a with respect to the estimation result Y′b, and the weighted average calculator32calculates the weighted average loss Lab of the losses La and Lb. Then, the parameter updater33updates the internal parameters PA of the logit calculator11ain the estimation unit10abased on the weighted averaged loss Lay.

If the number of learning data is not balanced among classes, the effect of correcting the balances may be added to the losses La and Lb. Specifically, Equation (3) described above multiplies the loss for each class by the coefficient wifor correction. For example, the coefficient wiis calculated by Equation (4) or (5) described above. Here, niis the total number of learning data belonging to the i-th class. In another example, the sum of the estimation result Yb for each class, calculated by the teacher model for all the learning data or for mini-batches, may be used as the coefficient wi, instead of the total number niof the learning data.

(Processing Flow)

Next, the flow of the learning process according to the third example embodiment will be described.FIG.8is a flowchart of learning processing according to the third example embodiment. This process is realized by the processor1shown inFIG.1, which executes a program prepared in advance.

When the learning data X is inputted to the learning device300, the estimation unit10acorresponding to the student model calculates the estimation result Ya from the learning data X (step S31). When calculating the estimation result Ya, the activator12aof the estimation unit10auses the temperature parameter T=1. The estimation unit10bcorresponding to the teacher model calculates the estimation result Yb from the learning data X (step S32). When calculating the estimation result Yb, the activator12bof the estimation unit10buses the temperature parameter T=1.

Next, the temperature calculator21determines the temperature parameter T from the estimation results Ya and Yb based on a predetermined function (step S33). The determined temperature parameter T is supplied to the activators22aand22b.

Next, the activator22acalculates the estimation result Y′a from the logit ya outputted by the logit calculator11ausing the temperature parameter T. Also, the activator22bcalculates the estimation result Y′b from the logit yb outputted by the logit calculator11busing the temperature parameter T (step S34).

Next, in the optimizing unit30, the loss calculator31calculates the loss La of the estimation result Y′a with respect to the correct labels Y and the loss Lb of the estimation result Y′a with respect to the estimation result Y′b (step S35). Next, the weighted average calculator32calculates the weighted average Lav of the loss La and the loss Lb (step S36). Then, the parameter updater33updates the internal parameters PA of the logit calculator11abased on the weighted averaged loss Lav (step S37).

The learning device300repeats the above-described processing until the update amount of the internal parameters PA by the parameter updater33becomes smaller than the predetermined value, and ends the processing when the update amount becomes smaller than the predetermined value. The student model is defined by the internal parameters PA set in the logit calculator11aand the temperature parameter T set in the activator22awhen the processing ends. Then, at the time of inference, the inference processing is executed by the student model in which the temperature parameter T is set as T=1.

(Modification)

In the third example embodiment described above, the temperature calculator21calculates the temperature parameter T using both the estimation result Ya generated by the estimation unit10acorresponding to the student model and the estimation result Yb generated by the estimation unit10bcorresponding to the teacher model. Instead, the temperature calculator21may calculate the temperature parameter T using one of the estimation result Ya generated by the estimation unit10acorresponding to the student model and the estimation result Yb generated by the estimation unit10bcorresponding to the teacher model.

Examples

In a first example of the third example embodiment, the temperature calculator21sets the temperature parameter T so as to correct the magnitude balance of the probability estimate value of each class of the estimation result Yb outputted from the estimation unit10b, specifically, such that the order of the estimation result of each class is balanced. For example, the temperature calculator21sets the temperature parameter T as T=5 in the case where the maximum value/minimum value of the probability estimate is equal to or greater than 105, and sets the temperature parameter T as T=1 in other cases. Preferably, the temperature parameter T is determined in the range from 1 to 100. The softmax function is used for the activation function of the activators22aand22b.

In a second example of the third example embodiment, supposing that the value of the highest probability class is “p1” and the value of the second highest probability class is “p2” among the estimation results Yb outputted from the estimation unit10bcorresponding to the teacher model, the temperature calculator21determines the temperature parameter T based on the ratio of those values: r=p1/p2. In an example, the temperature calculator21sets T=1 when r<4, sets T=2 when 4≤r<5, and sets T=3 when r≥5. In another example, the temperature calculator21determines the temperature parameter T by a sigmoid function more continuously related to “r”. For example, the temperature calculator21sets the temperature parameter T as T=1 when “r” is sufficiently small, T=3 when “r” is sufficiently large, and T=2 when r=4, by the following Equation
T=2/(1+e(−r+4))+1

In a third example of the third example embodiment, the temperature parameter T is determined based on the entropy E of the estimation result Yb outputted from the estimation unit10bcorresponding to the teacher model. The entropy E is given by the following Equation (9), if the estimation result of the i-th class is expressed as Ybi.

E=-∑i⁢Ybi*log⁡(Ybi)(9)

The temperature parameter T is given as a monotonously decreasing function with respect to the entropy E of the estimation result Yb. In one example, the temperature parameter T is defined by Equation (10) such that T=10 when E is the minimum value and T=1 when E is the maximum value. Here, N is the number of classes that the teacher model classifies.
T=−9/log(N)*E+10  (10)

Other Example Embodiments

The first example embodiment described above may be combined with the second example embodiment or the third example embodiment. For example, when the first example embodiment and the second example embodiment are combined, the initial value of the temperature parameter T may be determined based on the logit according to the method of the second example embodiment, and then the temperature parameter T may be updated based on the loss according to the method of the first example embodiment. Also, when the first example embodiment and the third example embodiment are combined, the initial value of the temperature parameter T may be determined based on the estimation result according to the method of the third example embodiment, and then the temperature parameter T may be updated based on the loss according to the method of the first example embodiment.

A part or all of the example embodiments described above may also be described as the following supplementary notes, but not limited thereto.

(Supplementary Note 1)

A learning device comprising:a first estimation unit configured to perform estimation using a temperature parameter, based on a student model;a second estimation unit configured to perform estimation using the temperature parameter, based on a teacher model; anda temperature calculation unit configured to calculate the temperature parameter, based on estimation information generated by the first estimation unit and the second estimation unit.
(Supplementary Note 2)

The learning device according to supplementary note 1, wherein the temperature calculation unit calculates the temperature parameter based on at least one of a loss of a first estimation result generated by the first estimation unit with respect to a correct label and a loss of the first estimation result with respect to a second estimation result generated by the second estimation unit.

(Supplementary Note 3)

The learning device according to supplementary note 1, wherein the temperature calculation unit calculates the temperature parameter based on a weighted average of a loss of a first estimation result generated by the first estimation unit with respect to a correct label and a loss of the first estimation result with respect to a second estimation result generated by the second estimation unit.

(Supplementary Note 4)

The learning apparatus according to supplementary note 1, wherein the temperature calculation unit calculates the temperature parameter based on at least one of a logit generated by the first estimation unit and a logit generated by the second estimation unit.

(Supplementary Note 5)

The learning apparatus according to supplementary note 1, wherein the temperature calculation unit calculates the temperature parameter in accordance with positive and negative logits generated by the first estimation unit or the second estimation unit.

(Supplementary Note 6)

The learning apparatus according to supplementary note 1, wherein the temperature calculation unit calculates the temperature parameter based on at least one of an estimation result generated by the first estimation unit and an estimation result generated by the second estimation unit.

(Supplementary Note 7)

The learning apparatus according to supplementary note 6, wherein the temperature calculation unit calculates the temperature parameter based on entropy of an estimation result generated by the second estimation unit.

(Supplementary Note 8)

The learning apparatus according to supplementary note 6, wherein the temperature calculation unit calculates the temperature parameter as a monotonously decreasing function relating to entropy of an estimation result generated by the second estimation unit.

(Supplementary Note 9)

The learning apparatus according to supplementary note 6, wherein the temperature calculation unit calculates the temperature parameter such that magnitude balance of the estimation result of each class generated by the first estimation unit or the second estimation unit is corrected.

(Supplementary Note 10)

The learning apparatus according to supplementary note 6, wherein the temperature calculation unit calculates the temperature parameter such that the order of the estimation result of each class generated by the first estimation unit or the second estimation unit is balanced.

(Supplementary Note 11)

The learning apparatus according to supplementary note 6, wherein the temperature calculation unit calculates the temperature parameter based on a ratio of a value of a highest probability class and a second highest probability class among the estimation result of each class generated by the second estimation unit.

(Supplementary Note 12)

A learning method executed by a learning device, comprising:performing estimation using a temperature parameter, based on a student model;performing estimation using the temperature parameter, based on a teacher model; andcalculating the temperature parameter, based on estimation information generated by the first estimation unit and the second estimation unit.
(Supplementary Note 13)

A program executed by a learning device including a computer, the program causing the computer to function as:a first estimation unit configured to perform estimation using a temperature parameter, based on a student model;a second estimation unit configured to perform estimation using the temperature parameter, based on a teacher model; anda temperature calculation unit configured to calculate the temperature parameter, based on estimation information generated by the first estimation unit and the second estimation unit.

While the invention has been particularly shown and described with reference to example embodiments and examples thereof, the invention is not limited to these example embodiments and examples. It will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the claims.

DESCRIPTION OF SYMBOLS

1Processor2Memory5,6,10a,10bEstimation unit7,21Temperature calculator11a,11bLogit calculator12a,12b,22a,22bActivator30Optimization unit31Loss calculator32Weighted average calculator33Parameter updater34Temperature updater