Patent ID: 12236667

EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that when the numerals of numbers, quantities, amounts, ranges, and the like of respective elements are mentioned in the embodiment shown as follows, the present disclosure is not limited to the mentioned numerals unless specially explicitly described otherwise, or unless the disclosure is explicitly specified by the numerals theoretically. Furthermore, configurations that are described in the embodiment shown as follows are not always indispensable to the disclosure unless specially explicitly shown otherwise, or unless the disclosure is explicitly specified by the structures or the steps theoretically. In addition, in the drawings, the same reference sign or numeral will be given to the same or equivalent component, and the repetitive description thereof will be appropriately simplified or omitted.

1. OSDG (OPEN SET DOMAIN GENERALIZATION)

A learning method according to the present embodiment is a method for learning a machine learning model which extracting a feature of an input image and classifying the input image based on the feature. In particular, the learning method relates to a machine learning model aimed at solving OSDG. The following will describe OSDG.

OSDG is problems in which open-set recognition and DG are combined. That is, the input image classified into known classes and an unknown class, each of the known classes showing that the input image falls into any one of specific categories (categories of interest) and the unknown class showing that the input image does not fall into any of the specific categories. And it is necessary that the classifying is performed regardless of a domain of the input image.

FIG.1is a conceptual diagram for explaining OSDG. InFIG.1, a category indicates an object captured in image data. InFIG.1, The known classes are configured of “Dog”, “Horse”, and “Person”. And the unknown class shows that an object captured in image data is neither the “Dog”, “Horse”, and “Person”.FIG.1illustrates a plurality of samples of image data having different domains (“Photo”, “Art”, “Sketch”, etc.).

That is, inFIG.1, OSDG is a problem of classifying an object captured in image data as the “Dog,” “Horse,” or “Person” and removing image data in which other objects are captured, without depending on the domain. For example, there is a case where a user desires to recognize (is interested in) that an input image captures a “Dog,” “Horse,” or “Person” but is not interested in other objects captured in the input data. Such a case can be cited as a practical problem, for example, when a case performing object recognition based on image data.

In a case generating a machine learning model for solving OSDG as illustrated inFIG.1, learning is performed using, as a plurality of samples of training data, a plurality of samples of image data having different domains each other and each of which is with a label. Here, what is different from the conventional DG is that the classification includes the unknown class, and that the plurality of samples of training data may be with the label corresponding to the unknown class.

In OSDG, as like the conventional DG, the domain of the input image may not exist in the plurality of samples of training data. For example, inFIG.1, as the training data, image data having the domain of “Photo” (first row), “Art” (second row), or “Sketch” (third row) is illustrated. On the other hand, as the input image, image data of a cartoon-like or deformed object is illustrated. Hereinafter, the domain of the training data is also referred to as a “source domain,” and the domain of the input data is also referred to as a “target domain.”

Generally, when classifying the input image, a feature of the input image is extracted, and the input image is classified based on the position of the extracted feature in a feature space. Most of the methods for the conventional DG align the feature space by using samples of training data in multiple source domains. The difficulty of OSDG is that, even if applying the method for the conventional DG to OSDG problems, the unknown class in the target domain is not guaranteed to be properly aligned because the unknown class in source and target domains is not necessarily in the conventional DG. Thus, the methods for the conventional DG have an issue that the feature of the input data of the unknown class may be mapped into the feature space of the known classes.

A method for OSDG requires at least two mechanisms. One (first mechanism) is constructing a separable feature space for rejecting image data of the unknown class. And the other (second mechanism) is matching distributions to align the known classes in multiple domains. The method for the conventional DG has the second mechanism, so it is possible to classify the input data without depending on the domain of the input data, but as described above the first mechanism is not sufficiently implemented in the method for the conventional DG.

The learning method according to the present embodiment has the first mechanism to effectively solve OSDG. And the method for the conventional DG can be employed for the second mechanism. It is thus possible to generate a machine learning model capable of effectively solving OSDG.

2. MACHINE LEARNING MODEL

The following will describe a machine learning model in which learning is performed by the learning method according to the present embodiment.FIG.2is a block diagram illustrating a configuration example of a machine learning model10in which learning is performed by the learning method according to the present embodiment.

The machine learning model10comprises a feature extraction processing unit11and a classification processing unit12. The feature extraction processing unit11receives image data as an input and outputs a feature of the image data. The feature output by the feature extraction processing unit11is transmitted to the classification processing unit12. The classification processing unit12receives the feature as an input and outputs a classification. That is, the machine learning model10classifies the image data.

The machine learning model10has parameters defining the processing for the feature extraction processing unit11and the processing for the classification processing unit12, respectively. In other words, the learning in the machine learning model10is performed by updating the parameters.

The machine learning model10is typically implemented by a convolutional neural network. In this case, the feature extraction processing unit11is implemented by one or more convolution layers and one or more pooling layers, and the classification processing unit12is implemented by one or more affine layers. And the parameters are the weights of the filter of the convolution layer and the weights of the affine layer.

The machine learning model10may be implemented by other configurations. For example, while the feature extraction processing unit11is implemented by one or more convolution layers and one or more pooling layers, the classification processing unit12may be implemented by a support-vector machine (SVM) or a k-nearest neighbors algorithm (k-NN).

Typically, the machine learning model10is implemented by a computer program, and the processing of the machine learning model10is realized when instructions comprised in the computer program are executed by one or more computers. In this case, the parameters of the machine learning model10may be provided as a part of the computer program or may be stored into a memory and read by the one or more computers. Further, the parameters may be updated by updating the computer program or updating the memory.

3. LEARNING METHOD

The learning method according to the present embodiment calculates a loss function and learns the machine learning model10to reduce the loss function. The following will describe the learning method according to the present embodiment.

FIG.3is a flowchart illustrating the learning method according to the present embodiment.

In Step S100, by inputting a plurality of samples of training data, a set of output of the machine learning model10and a set of features for the plurality of samples of training data are acquired. After Step S100, the processing proceeds to Step S110.

In Step S110, a loss function is calculated based on the set of output and the set of features acquired in Step S100. Here, in the learning method according to the present embodiment, the loss function calculated in Step S110is characteristic to provide the first mechanism. Details of the loss function to be calculated in Step S110will be described later. After Step S110, the processing proceeds to Step S120.

In Step S120, the gradient of the loss function acquired in Step S110is calculated. In Step S120, any suitable known technique may be adopted as a method for calculating the gradient of the loss function. For example, when the machine learning model10is implemented by the convolutional neural network, the method for calculating the gradient of the loss function is typically backpropagation. After Step S120, the processing proceeds to Step S130.

In Step S130, the parameters of the machine learning model10are updated to reduce the loss function based on the gradient calculated in Step S120. That is, the parameters are updated by gradient descent. Here, hyperparameters related to the updating of the parameters may be suitably determined in accordance with an environment to which the learning method according to the present embodiment is applied. For example, the parameters may be updated by a momentum method.

In Step S140, it is determined whether or not an end condition is satisfied. The end condition is, for example, that the number of repetitions of the parameter update is equal to or greater than a predetermined value or that the loss function is equal to or less than a predetermined value.

When the end condition is satisfied (Yes in Step S140), the learning method ends. When the end condition is not satisfied (NO in Step S140), the processing returns to Step S100again, and the learning method is repeated.

The learning method illustrated inFIG.3may be realized as operations performed by one or more computers when the one or more computers execute instructions comprised in a computer program. And one or more non-transitory computer readable media encoded with the computer program may be provided for the one or more computers to execute the instructions.

4. LOSS FUNCTION

The inventors of the present disclosure have obtained the idea of employing a method of metric learning for constructing the feature space with the first mechanism. The typical loss functions for metric learning (e.g., triplet loss and contrastive loss) encourages samples with the same class to move closer to each other, and samples with different classes to be further away in the feature space. Thus, it can be expected to construct the feature space that is easily separable by class.

However, it is not obvious how to utilize samples of the unknown class for the typical loss functions for metric learning. The only thing we can say about the samples of the unknown class is that, when employing the triplet loss, the samples of the unknown class do not constitute positive pairs with the samples of the known classes. And we cannot naively determine the positive pairs including the samples of the unknown class.

A straightforward approach to exploit the samples of the unknown class is to use them only for forming negative pairs in the triplet loss. However, the inventors of the present disclosure found in an ablation study that this approach alone is not enough for OSDG because it does not explicitly move away the samples of the unknown class in the feature space.

Motivated by the above discussion, in addition to the above approach, the inventors of the present disclosure have obtained the idea of introducing a decoupling loss (first loss function) which keep the distance among the samples of the unknown class in the feature space.FIG.4is a conceptual diagram representing the feature space achieved by the first loss function. Each plane figure illustrated inFIG.4shows a feature in the feature space. It is shown here that the same plane figures are of the same class. The dotted line illustrated inFIG.4shows an example of a desirable discrimination boundary for OSDG.

As illustrated inFIG.4, both with and without the first loss function, it is possible to construct the feature space in which the features for the samples of the same known classes are closer to each other by metric learning. However, when the first loss function is not introduced, the features for the samples of the unknown class are hardly separated from the features for the samples of the known classes (see figure above). On the other hand, by introducing the first loss function, the features of samples of the unknown class are sufficiently far form the features of the samples of the known classes, and the samples of the unknown class are discriminated more accurately (see figure below).

The following will describe in detail the loss function calculated in the learning method according to the present embodiment.

In the learning method according to the present embodiment, the loss function is calculated to include the decoupling loss (first loss function) and a triplet loss (second loss function).

First, the first loss function will be described. The first loss function is represented by Ld of the following Formula. 1.

Ld=1❘"\[LeftBracketingBar]"N❘"\[RightBracketingBar]"⁢∑(xa,xn)∈Nmax⁡(0,-d⁡(f⁡(xa),f⁡(xn))+α)[Formula.1]N={(xa,xni)⁢❘"\[LeftBracketingBar]"ya∈{u},yni∈C⋃{u},d⁡(f⁡(xa),f⁡(xni))<α}i={1,2,…,K}Where x* is a sample of training data, y* is the label of x*, C is the set of the known classes, u is the unknown class, f(x*) is the feature of x*, and d(a, b) is a function measuring the distance in the feature space between a and b.

That is, xais a sample of training data with the label corresponding to the unknown class (hereinafter also referred to as “first anchor point”), xni(i={1, 2, . . . , K}) are samples of training data selected with the number of K for the first anchor point. Therefore, N is a set of combinations of a plurality of first anchor points and the samples of training data. Here, in Formula.1, xni(i={1, 2, . . . , K}) are selected on the condition that the distance to the corresponding first anchor point is less than a predetermined margin α. This condition is for preventing the selection of samples of training data that has already achieve the purpose of the first loss function and does not contribute to a change in the first loss function. It is thereby possible to improve the efficiency of the processing of learning.

Here, “i” is an additional number for distinguishing each of the samples of training data selected for the first anchor point. And the predetermined margin α defines how far the unknown class is to be kept in the feature space. The predetermined margin α may be suitably given in accordance with the environment to which the learning method according to the present embodiment is applied.

In the configuration of N, the plurality of first anchor points is selected out of the plurality of samples of training data with the label corresponding to the unknown class, and xniare selected with the number of K for each of the plurality of first anchor points. And in Formula. 1, |N| represents the element number of N.

In Formula. 1, a suitable function may be implemented as d depending on the environment to which the learning method according to the present embodiment is applied. For example, the cosine similarity is implemented as d.

As shown in Formula. 1, when calculating the first loss function, distances in the feature space (hereinafter also referred to as “first distances”) between features of the plurality of first anchor points and features of xniare calculated. And the first loss function becomes larger as the first distances become smaller than the predetermined margin. That is, by learning the machine learning model10(updating the parameters of the machine learning model10) to reduce the first loss function, it is possible to construct the feature space maintaining the distances among the samples of the unknown class.FIG.5illustrates a conceptual diagram showing the feature space when learning the machine learning model10to reduce the first loss function.

Next, the second loss function will be described. The second loss function is represented by Lt of the following Formula. 2.

Lt=1❘"\[LeftBracketingBar]"T❘"\[RightBracketingBar]"⁢∑(xa,xp,xn)∈Tmax⁡(0,d⁡(f⁡(xa),f⁡(xp))-d⁡(f⁡(xa),f⁡(xn))+α)[Formula.2]T={(xa,xp,xni)⁢❘"\[LeftBracketingBar]"ya=yp∈C,yni∈C⋃{u},ya≠yni,d⁡(f⁡(xa),f⁡(xp))<d⁡(f⁡(xa),f⁡(xni))+α}i={1,2,…,K}

As shown in Formula. 2, the second loss function is a triplet loss configured to be that a plurality of samples (hereinafter also referred to as “plurality of second anchor points”) selected out of the plurality of samples of training data with the label corresponding to any one of the known classes is implemented as the anchor of the triplet loss, and that the margin of the triplet loss is the same margin α as the predetermined margin of the first loss function. That is, in Formula.2, xais a second anchor point, xpis a positive pair of xa(a sample of training data with the same label as xa), xni(i={1, 2, . . . , K}) are negative pairs of xa(samples of training data with a different label as xa) which selected with the number of K, and T is a set of triplets.

In the configuration of T, xpmay be one or more of samples of training data selected randomly out of the plurality of samples of training data with the same label as xa. It is thus possible to execute learning without selecting as positive pairs all samples of training data with the same label of xa, then it is possible to improve the efficiency of the processing of learning.

Furthermore, xni(i={1, 2, . . . , K}) are selected with the number of K for each of the plurality of first anchor points. Incidentally, although the set of triplets shown in Formula. 2 is configured to select negative pairs by semi-hard, the selection may be modified (e.g., select negative pairs by hard). And the number of samples may not equal to that of the first loss function.

As shown in Formula. 2, when calculating the second loss function, distances in the feature space (hereinafter also referred to as “second distances”) between features of the plurality of second anchor points and features of xpare calculated. And distances in the feature space (hereinafter also referred to as “third distances”) between features of the plurality of second anchor points and features of xniare calculated. Then, the second loss function becomes larger as the second distances become larger and the third distances become smaller. In other words, the second loss function becomes larger as the distances between features of xaand xpare larger and as the distances between features of xaand xniare smaller. Note that, in Formula. 2, it is characteristic that the plurality of anchor points is selected out of the plurality of samples of training data with the label corresponding to the known classes.

That is, by learning the machine learning model10to reduce the second loss function, it is possible to satisfy the following Formula. 3. Formula. 3 means each of the plurality of second anchor points should be relatively closer to the positive pairs than the negative pairs. Therefore, it is possible to construct the feature space in which, regarding samples of the known classes, samples of the same class move closer to each other and samples of different class move further away.FIG.6illustrates a conceptual diagram showing the feature space when learning the machine learning model10to reduce the second loss function.
d(f(xa),f(xp))+α≤d(f(xa),f(xni))  [Formula. 3]

Note that, when learning the machine learning model10to just reduce the second loss function, regarding samples of unknown class, the feature space is just constructed to be that samples of the unknown class move away from samples of the known classes. Thus, learning the machine learning model10to reduce the second loss function alone, it will just construct the feature space as shown in the above ofFIG.4. Therefore, by introducing the first loss function, it is possible to construct the feature map maintaining the distances among samples of the unknown class as shown inFIG.5. In this way, it is possible to construct the feature space as shown in the bellow ofFIG.4.

As described above, the first mechanism is implemented by calculating the loss function configured to include the first loss function and the second loss function. And the conventional DG is combined in order to implement the second mechanism. That is, in the learning method according to the present embodiment, the loss function is configured as represented by L of the following Formula. 4.
L=LDG+λ*Lm
Lm=Ld+Lt[Formula. 4]

Where, LDGis a domain generalization loss related to the conventional DG. That is, by learning the machine learning model10to reduce LDG, it is possible to generate the machine learning model10being able to classify the input data without depending on the domain of the input data. The configuration of LDGis defined according to the employed method for the conventional DG. Here, as the method for the conventional DG, a suitable method (e.g., DeepAll, JiGen, MMLD) may be employed in accordance with the environment to which the learning method according to the present embodiment is adopted. Since LDGis the loss for learning the machine learning model10to classify image data on DG, LDGhas the configuration of a loss (main loss function) which become smaller as the more output matches the label of training data.

In Formula. 4, λ is a positive real number and is a hyperparameter that gives the degree of contribution of each of the first loss function and the second loss functions. λ may be suitably defined in accordance with the environment to which the learning method according to the present embodiment is applied.

5. EXPERIMENTS

The inventors of the present disclosure conduct experiments on two DG benchmark datasets: VLCS and DomainNet.FIG.7shows a table of the result of the experiments. InFIG.7, each value represents a classification accuracy (%).

Here, VLCS combines four different datasets (i.e., PASCAL VOC 2007, LabelMe, Caltech-101, Sun09) and consists of five object categories. DomainNet includes 345 object categories from six domains (i.e., sketch, real, quickdraw, painting, infograph, and clipart).FIG.8andFIG.9illustrate examples of image data of VLCS and DomainNet.

In the experiments, to set problems as OSDG, all classes are split into three subset: Ck, Csu, and Cuu. Ck is a set of the known classes in both the source domain and the target domain, Csu is a set of the unknown class in the source domain, and the Cuu is a set of the unknown class in the target domain. Ck, Csu, and Cuu are unique to each other.

In experiments on VLCS, samples of image data of |Ck|=3, |Csu|=1, and |Cuu|=1 are used. And in experiments on DomainNet, samples of image data of |Ck|=10, |Csu|=167, and |Cuu|=168 are used. Concretely on VLCS, classes of car, chair, and person are the known classes of the source domain, class of dog is the unknown class of the source domain, and class of bird is the unknown class of the target domain. And on DomainNet, the Csu and Cuu contain up to 2,000 samples to balance with the other classes.

As for baselines (i.e., LDGonly), the inventors of the present disclosure use three DG methods as shown inFIG.7: Deep All, JiGen, and MMLD. And the table shown inFIG.7shows comparisons between three experiments, a case (first stage, LDGonly) where the learning method according to the present embodiment is not applied, a case (second stage, w/Ltriplet(LDG+λ*Lt)) where the second loss function is applied, and a case (third stage, w/Lmetric(LDG+λ*Lm)) where the first loss function and the second loss function are applied.

As can be seen in the table shown inFIG.7, by applying the learning method according to the present embodiment, it is possible to improve the classification accuracy. In particular, the application of the second loss function improves the classification accuracy in all the cases. Furthermore, the application of the first loss function and the second loss function is more totally effective in improving the classification accuracy than the application of the second loss function alone. As described above, by applying the learning method according to the present embodiment, it is possible to generate the machine learning model10which can effectively solve OSDG.

6. PROCESSING APPARATUS

It is possible to configure a processing apparatus classifying the input image into the known classes and the unknown class based on the feature of the input image by using the machine learning model10. Furthermore, by using the machine learning model10which has been learned by the learning method according to the present embodiment, it is possible to configure the processing apparatus being capable of effectively solving OSDG. The following will describe the processing apparatus according to the present embodiment.FIG.10illustrates a configuration example of the processing apparatus100.

The processing apparatus100receives the input image and outputs the classification of the input image. The processing apparatus100comprises a memory110and a processor120. The processing apparatus100may be one or more computers. For example, the processing apparatus is a server (may be a virtual server) configured on a communication network (e.g., the Internet).

The memory110stores data111and a computer program112. The computer program112comprises executable instructions that, when executed by the processor120, cause the processing apparatus100to perform operations. That is, the processor120read the data111and the computer program112from the memory110and executes the instructions based on the data111.

Here, the machine learning model10which has been learned by the learning method according to the present embodiment is implemented by the computer program112. Therefore, the feature extraction processing unit11and the classification processing unit12are realized by the processor120executing the instructions of the computer program112. At this time, the learned parameters of the machine learning model10may be stored as the data111or may be stored as a part of the computer program112.

The processor120reads the computer program112related to the machine learning model10and executes the instructions of the computer program112related to the machine learning model10, thereby realizing the processing apparatus100capable of effectively solving OSDG.

7. MODIFICATION

The learning method according to the present embodiment can also be applied to a case where the learning is performed only on the feature extraction processing unit11in the machine learning model10. For example, this is the case where learning is performed only on the feature extraction processing unit11of the machine learning model10which has been learned by the method for the conventional DG.

In this case, the output of the part of the machine learning model10subject to learning is the feature of image data. And the loss function calculated in the learning method according to the present embodiment is configured to include the first loss function and the second loss function.

Accordingly, providing the feature extraction processing unit11having learned by the learning method according to the present embodiment as DG makes it possible to generate the machine learning model10capable of effectively solving OSDG. Furthermore, the classification processing unit12may be configured of SVM or k-NN and combined with the feature extraction processing unit11having learned by the learning method according to the present embodiment to generate the machine learning model10.