Patent Description:
In image identification processing using machine learning, a known technique (see Patent Literature <NUM> (PTL1)) improves the accuracy of identification of an image by adding, to a training dataset, an image whose amount of image features has a low degree of similarity to that of an existing learning image.

It should be noted that the identification processing is also referred to as inference processing. The inference processing includes detection processing as well as the identification processing.

However, in a conventional technique as disclosed in PTL <NUM>, the accuracy of inference may not be much improved, depending on the inferencer. In other words, learning efficiency may not be high, which is considered an issue.

PTL2 relates to a method for generating, deploying and managing a machine learning-based detection profile. In one embodiment, the computing device additionally receives a document that was incorrectly classified as non-sensitive data based on the MLD profile, and modifies the training data set that was used to generate the MLD profile by adding the document to the training data set as a positive example of sensitive data to generate the modified training data set.

In view of the foregoing, the present invention provides, for example, an information processing method capable of efficiently improving the accuracy of inference by an inferencer.

An information processing method according to a first aspect of the embodiments of the present invention, is an information processing method implemented by a computer, as claimed in appended claim <NUM>. According to a second aspect of the embodiments of the present invention, there is provided an information processing system, as claimed in appended claim <NUM>. According to a third and a fourth aspect of the embodiments of the present invention, there is provided a computer program and a computer-readable recording medium, as claimed in appended claims <NUM> and <NUM>, respectively.

It should be noted that a comprehensive or specific embodiment may be a system, apparatus, integrated circuit, computer program, or computer-readable recording medium, such as CD-ROM, or may be a given combination of the system, apparatus, integrated circuit, computer program, and recording medium.

The information processing method according to the present invention is capable of efficiently improving the accuracy of inference by the inferencer.

Regarding the inference processing described in Background, the inventors of the present invention found the following issues.

Enhancing a training dataset is useful to improve the performance of an inferencer that performs the inference processing using machine learning. Each piece of training data includes an image and a label that is information shown by the image. Typically, to enhance a training dataset, a new training dataset is used that includes newly added pieces of data not included in a training dataset used for training an inferencer. To add new pieces of data, given images need to be labeled. Labeling is performed by, for example, a person.

While the given images can be readily prepared, it is difficult at present to determine which data among such given images are useful to improve the accuracy of inference by the inferencer or correct a false inference made by the inferencer. Thus, to improve the accuracy of inference or correct a false inference, a vast amount of new data is added regardless of whether images can contribute to improvement in the accuracy of inference or correction of the false inference. However, to generate and add a vast amount of data, for instance, a large number of images need to be labeled, which is considered inefficient in terms of the number of steps or a period of time to be taken.

Thus, at present, the efficiency of improvement in the accuracy of inference by the inferencer is not high, which is considered an issue.

In response to the issue, in a conventional technique described as above, images having low degrees of similarity in terms of the amount of image features are selected as training data.

However, the images having low degrees of similarity in terms of the amount of image features are not necessarily useful pieces of data for an inferencer to be trained using the images. Thus, the accuracy of inference may not be much improved, depending on the inferencer. In other words, learning efficiency may still not be high, which is considered an issue.

An information processing method according to an aspect of the present invention is an information processing method implemented by a computer. The information processing method that includes: obtaining a piece of first data and a piece of second data, the piece of second data being not included in a training dataset used for training an inferencer; calculating, by using a piece of first relevant data, a first contribution representing the respective contributions of portions constituting the piece of first data to a piece of first output data output by inputting the piece of first data to the inferencer, the piece of first relevant data being obtained by inputting the piece of first data to the inferencer trained by machine learning using the training dataset; calculating, by using a piece of second relevant data, a second contribution representing the respective contributions of portions constituting the piece of second data to a piece of second output data output by inputting the piece of second data to the inferencer, the piece of second relevant data being obtained by inputting the piece of second data to the inferencer; and determining whether to add the piece of second data to the training dataset for the inferencer, according to the degree of similarity between the first contribution and the second contribution.

According to the above-described aspect, it is determined that a piece of second data having a contribution similar to that of a piece of first data will be added to the training dataset. The contribution represents an effect on the result of inference (processing performed) by the inferencer. Thus, a piece of second data similar to a piece of first data in terms of the effect on the result of inference by the inferencer can be selected. By training the inferencer by using the selected piece of second data as a piece of training data, it is more likely to suppress the inferencer from making a false inference for the piece of first data and pieces of data similar to the piece of first data. In addition, by adding a piece of data useful for the inferencer to the training dataset, it is possible to avoid randomly adding a vast amount of data. Hence, according to the above-described aspect, the accuracy of inference by the inferencer can be efficiently improved.

The piece of first data is a piece of data for which the inferencer made a false inference.

According to the above-described aspect, it is determined that a piece of second data having a contribution similar to that of a piece of falsely inferred data will be added to the training dataset. By training the inferencer by using a training dataset to which the piece of second data has been added in accordance with the determination, it is possible to suppress the inferencer from making another false inference for pieces of data similar to the piece of falsely inferred data. Hence, according to the above-described aspect, the accuracy of inference by the inferencer can be efficiently improved.

For instance, if a determination to add the piece of second data is made, the inferencer may be trained using the training dataset to which the piece of second data has been added, the training dataset being used for training the inferencer.

According to the above-described aspect, the inferencer is trained using the training dataset to which the piece of second data has been added in accordance with the determination to add the piece of second data to the training dataset. This makes it possible to suppress the inferencer from making another false inference for pieces of data similar to the piece of first data. Hence, according to the above-described aspect, the accuracy of inference by the inferencer can be efficiently improved.

For instance, regarding the determination of whether to add the piece of second data to the training dataset for the inferencer, whether to add to the training dataset for the inferencer is determined for each of pieces of second data to enable a piece of second data having a higher degree of similarity between the first contribution and the second contribution to be more preferentially selected from the pieces of second data and added to the training dataset for the inferencer, the pieces of second data each being the piece of second data.

According to the above-described aspect, the following determination can be made: a piece of second data more similar to the piece of first data in terms of the contribution is selected from the prepared pieces of second data, and the selected piece of second data is added to the training dataset. Hence, according to the above-described aspect, the accuracy of inference by the inferencer can be more efficiently improved.

For instance, regarding the determination of whether to add the piece of second data to the training dataset for the inferencer, a plurality of the first contributions that include the first contribution of each of pieces of first data may be calculated, the pieces of first data each being the piece of first data, and whether to add the piece of second data to the training dataset for the inferencer may be determined according to a representative value calculated using a plurality of degrees of similarity including degrees of similarity between the plurality of the first contributions and the second contribution.

According to the above-described aspect, if pieces of first data are present, a piece of second data to be added to the training dataset is determined using representative values calculated from the degrees of similarity calculated for each of the pieces of first data. If pieces of first data are present, a plurality of degrees of similarity are calculated. However, it is difficult to determine, by using the plurality of degrees of similarity, which piece of second data should be added to the training dataset. Thus, by using the representative values calculated from the plurality of degrees of similarity, it is possible to readily determine which piece of second data should be added to the training dataset. Hence, according to the above-described aspect, the accuracy of inference by the inferencer can be more readily improved.

For instance, the plurality of degrees of similarity may include degrees of similarity between the plurality of the first contributions and a plurality of the second contributions that include the second contribution of each of pieces of second data, the pieces of second data each being the piece of second data. Regarding calculation of the representative value, for each of the pieces of first data, a predetermined number of degrees of similarity may be selected from the plurality of degrees of similarity, and for each of the pieces of second data, the representative value may be calculated by using the predetermined number of degrees of similarity. Regarding the determination of whether to add the piece of second data to the training dataset for the inferencer, whether to add the piece of second data to the training dataset for the inferencer may be determined according to the representative values.

According to the above-described aspect, if pieces of first data are present, it is possible to suppress the determination that pieces of second data similar exclusively to a specific piece of first data in terms of the contribution will be added to the training dataset, from being made. If pieces of first data are present, in some cases, it is determined that pieces of second data similar exclusively to a specific piece of first data in terms of the contribution will be added to the training dataset. In this case, while it is possible to suppress the inferencer from making another false inference for pieces of data similar to the specific piece of first data, it may not be possible to suppress the inferencer from making another false inference for the pieces of first data other than the specific piece of first data among the pieces of first data. Thus, it may not be possible to suppress the inferencer from making a false inference, evenly among the pieces of first data. According to the above-described aspect, however, it is possible to suppress the inferencer from making a false inference, evenly among the pieces of first data. Hence, according to the above-described aspect, the accuracy of inference by the inferencer can be efficiently improved, and improvements can be made evenly among the pieces of first data.

For instance, the inferencer may be an identifier or a detector.

According to the above-described aspect, it is possible to suppress false identification in identification processing performed by the identifier or false detection in detection processing performed by the detector. Hence, according to the above-described aspect, the accuracy of identification by the identifier or the accuracy of detection by the detector can be efficiently improved.

For instance, each of the piece of first data and the piece of second data may be a piece of sensing data.

According to the above-described aspect, it is possible to efficiently improve the accuracy of inference for the piece of sensing data.

For instance, the piece of first relevant data may be the piece of first output data, and the piece of second relevant data may be the piece of second output data.

According to the above-described aspect, efficient improvement in the accuracy of inference by the inferencer can be more readily made by using the piece of first output data as the piece of first relevant data.

For instance, a presentation apparatus may present information indicating the piece of second data determined to be added to the training dataset for the inferencer.

According to the above-described aspect, information indicating the piece of second data determined to be added to the training dataset is presented. The presentation apparatus presents the piece of second data to a user according to the information and receives, from the user, a label regarding the information indicating the piece of second data. The piece of data is added to the training dataset by using the input label. Hence, by receiving, from the user, the label for the piece of data to be added, the accuracy of inference by the inferencer can be more efficiently improved.

In addition, an information processing system according to another aspect of the present invention, includes: an acquisition unit that obtains a piece of first data and a piece of second data, the piece of second data being not included in a training dataset used for training an inferencer; a computation unit (a) that calculates, by using a piece of first relevant data, a first contribution representing the respective contributions of portions constituting the piece of first data to a piece of first output data output by inputting the piece of first data to the inferencer, the piece of first relevant data being obtained by inputting the piece of first data to the inferencer trained by machine training using the training dataset and (b) that calculates, by using a piece of second relevant data, a second contribution representing the respective contributions of portions constituting the piece of second data to a piece of second output data output by inputting the piece of second data to the inferencer, the piece of second relevant data being obtained by inputting the piece of second data to the inferencer; and a determination unit that determines whether to add the piece of second data to the training dataset for the inferencer, according to the degree of similarity between the first contribution and the second contribution.

The above-described aspect provides effects similar to those obtained by implementing the information processing method set forth above.

Hereinafter, Embodiment is described in detail with reference to the drawings.

It should be noted that Embodiment described below is a comprehensive or specific example. The numerical values, shapes, materials, structural elements, arrangements and connections of the structural elements, steps, and order of the steps, and others described in Embodiment set forth below are provided as examples and are not intended to limit the present invention. Moreover, among the structural elements described in Embodiment set forth below, the structural elements not included in the independent claims, which represent superordinate concepts, are described as optional structural elements.

In Embodiment, for instance, an information processing system and an information processing method are described that enable efficient improvement in the accuracy of inference by an inferencer.

The overview of operation of recognizer <NUM> is described with reference to <FIG>.

<FIG> is a conceptual diagram illustrating processing performed by recognizer <NUM>.

As illustrated in <FIG>, when receiving a piece of input data, recognizer <NUM> performs recognition processing for the piece of input data and outputs the result of the recognition processing as a piece of output data. According to the invention, the piece of input data is an image. In examples not encompassed by the wording of the claims, instead of image, sound or text may be used as the piece of input data. If the piece of input data is an image, what is shown in the image (piece of input data) is recognized in the recognition processing.

It should be noted that recognizer <NUM> is an example of an inferencer. Another example of the inferencer is a detector. When receiving a piece of input data, the detector performs detection processing for the piece of input data and outputs the result of the detection processing as a piece of output data. If the piece of input data is an image, for instance, a specific subject shown in the image (piece of input data) is detected in the detection processing.

The piece of input data is a piece of image data to be recognized by recognizer <NUM>.

Recognizer <NUM> is a recognition model generated by machine learning training and is used for performing the recognition processing. Recognizer <NUM> has internal parameters and is trained and generated by setting an appropriate parameter so that an appropriate piece of output data is output in response to a piece of input data. Recognizer <NUM> is, for example, a mathematical model in a neural network. More specifically, a recognition model suitable for an object detection method, such as single shot multibox detector (SSD), faster region-based convolutional neural network (Faster-RCNN), or you only look once (YOLO), may be used as recognizer <NUM>.

The piece of output data is output by recognizer <NUM> and shows the result of recognition by recognizer <NUM>. Specifically, the piece of output data shows what is shown in the image, which is the piece of input data.

<FIG> illustrates an example of a training dataset used for machine learning training to generate recognizer <NUM>. Here, as an example, recognizer <NUM> obtains images showing <NUM> numbers from <NUM> to <NUM> as pieces of input data, recognizes a number shown in each image, and outputs the results of recognition.

As illustrated in <FIG>, the training dataset includes image-label pairs.

Each of the images included in the training dataset shows one of <NUM> numbers from <NUM> to <NUM>. The training dataset includes images in which each number is drawn in various patterns. For instance, thousands to tens of thousands of images are included for each number.

One label is appended to each image paired with the label and indicates the number shown in the image.

Recognizer <NUM> is generated by machining learning using the training dataset illustrated in <FIG>. Specifically, recognizer <NUM> is generated by adjusting the internal parameters so that when an image included in the training dataset is input as a piece of input data, the numerical value of a label paired with the input image is output.

<FIG> illustrates an example of the results of recognition by recognizer <NUM>. <FIG> illustrates an example of pieces of output data output by recognizer <NUM> that has received pieces of input data.

According to <FIG>, when, for instance, piece of input data <NUM> in <FIG> (i.e., image showing the number <NUM>) was input, recognizer <NUM> outputted the number <NUM> as a piece of output data. According to <FIG>, when piece of input data <NUM> in <FIG> (i.e., image showing the number <NUM>) was input, recognizer <NUM> outputted the number <NUM> as a piece of output data.

In both cases, recognizer <NUM> outputted the number shown in the input image. This means that recognition by recognizer <NUM> was true.

Meanwhile, according to <FIG>, when, for instance, piece of input data <NUM> in <FIG> (i.e., image showing the number <NUM>) was input, recognizer <NUM> outputted the number <NUM> as a piece of output data. According to <FIG>, when piece of input data <NUM> in <FIG> (i.e., image showing the number <NUM>) was input, recognizer <NUM> outputted the number <NUM> as a piece of output data.

In both cases, recognizer <NUM> outputted a number different from the number shown in the input image. This means that recognition by recognizer <NUM> was false.

When it turns out that recognizer <NUM> made false recognition, processing will be performed to prevent recognizer <NUM> from making such false recognition. Enhancing the training dataset is useful to prevent recognizer <NUM> from making such false recognition. To enhance the training dataset, a new training dataset that includes newly added pieces of data not included in the training dataset used for training recognizer <NUM> is used. Then, recognizer <NUM> is generated using the new training dataset.

However, it is difficult to determine which new piece of data should be added. For instance, to enable the number <NUM> to be output as a piece of output data when piece of input data <NUM> in <FIG> (i.e., image showing the number <NUM>) is input, a piece of data including an image that shows the number <NUM> or <NUM> and that differs from piece of input data <NUM> may be added to the training dataset. However, it is difficult to determine specifically what kind of image should be added.

In a case in which recognizer <NUM> made false recognition as described above, the information processing system in Embodiment is capable of efficiently improving the accuracy of recognition by recognizer <NUM> by appropriately determining what kind of data should be added to the training dataset to prevent recognizer <NUM> from making such false recognition in the future.

Hereinafter, the processing system in Embodiment is described.

Examples of the configuration of the processing system in Embodiment are described.

<FIG> is a block diagram illustrating a first example of the configuration of the processing system in Embodiment.

As illustrated in <FIG>, processing system 20A includes acquisition unit <NUM>, computation unit <NUM>, and determination unit <NUM>.

By means of a computer, acquisition unit <NUM> obtains a piece of first data and a piece of second data, the piece of second data being not included in the training dataset used for training the inferencer.

By means of the computer, computation unit <NUM> calculates a first contribution and a second contribution in the following manner.

According to the degree of similarity between the first contribution and the second contribution, determination unit <NUM> determines, by means of the computer, whether to add the piece of second data to the training dataset for the inferencer.

Next, the processing system in Embodiment is described in more detail.

<FIG> is a block diagram illustrating a second example of the configuration of the processing system in Embodiment.

As illustrated in <FIG>, processing system <NUM> includes controller <NUM>, accumulator <NUM>, computation unit <NUM>, determination unit <NUM>, and training unit <NUM>. Processing system <NUM> is connected to recognizer <NUM>, inputs a piece of data to recognizer <NUM>, and receives a piece of data from recognizer <NUM>. In addition, processing system <NUM> is connected to management apparatus <NUM>. Processing system <NUM> may be, for instance, a computer.

Controller <NUM> is a processing unit that inputs a piece of input data to recognizer <NUM> and receives a piece of output data from recognizer <NUM>.

Specifically, controller <NUM> inputs the piece of first data and the piece of second data to recognizer <NUM> by providing the pieces of data to at least recognizer <NUM>. Here, the piece of first data is, for example, a piece of validation data. The piece of first data is a piece of data input to recognizer <NUM> and falsely recognized by recognizer <NUM>. Hereinafter, the piece of first data is also referred to as a piece of falsely recognized data. Controller <NUM> obtains a piece of falsely recognized data, for example, in the following manner.

Controller <NUM> compares a number that is a piece of output data obtained by inputting, to recognizer <NUM>, an image showing a given number (also referred to as a validation image) as a piece of input data and a number shown in the image, which is the piece of input data. If the numbers are not the same number, controller <NUM> determines that the piece of input data is a piece of data falsely recognized by recognizer <NUM>, that is, a piece of falsely recognized data. Controller <NUM> obtains a piece of falsely recognized data in this manner.

In addition, the piece of second data is not included in the training dataset used for training recognizer <NUM> (in other words, training to generate recognizer <NUM>). Controller <NUM> obtains images accumulated in accumulator <NUM> as pieces of second data. Since the pieces of second data are candidates that may be later added to the training dataset, the pieces of second data are also referred to as pieces of candidate data.

Accumulator <NUM> is a storage device in which images are accumulated. The images accumulated in accumulator <NUM> are not labeled. Controller <NUM> obtains the images accumulated in accumulator <NUM> as the pieces of second data.

It should be noted that each of the piece of first data and piece of second data is, a piece of sensing data and, more specifically, a piece of image data obtained through sensing by a camera (by a camera capturing an image).

Computation unit <NUM> is a processing unit that calculates the respective contributions of portions constituting a piece of input data to a piece of output data output by recognizer <NUM> that has received the piece of input data. Specifically, by using a piece of first relevant data obtained by inputting a piece of first data to recognizer <NUM>, computation unit <NUM> calculates a first contribution representing the respective contributions of portions constituting the piece of first data to a piece of output data output by recognizer <NUM> that has received the piece of first data. In addition, by using a piece of second relevant data obtained by inputting a piece of second data to recognizer <NUM>, computation unit <NUM> calculates a second contribution representing the respective contributions of portions constituting the piece of second data to a piece of output data output by recognizer <NUM> that has received the piece of second data.

Here, if the piece of first data is an image, the portions constituting the piece of first data correspond to the pixels that constitute the piece of first data. In addition, the respective contributions of portions constituting the piece of first data to the piece of first output data represent the respective amounts of contribution of pixels constituting the piece of first data to recognition of a number, that is, the result of recognition of the piece of first data by recognizer <NUM>. Numerical values derived from the amounts of contribution are referred to as the degrees of contribution. The same applies to the portions constituting the piece of second data and the respective contributions of portions constituting the piece of second data to the piece of second output data.

It should be noted that a specific example of the piece of first relevant data is the piece of first output data, and a specific example of the piece of second relevant data is the piece of second output data. In addition, the piece of first relevant data (piece of second relevant data) may be output by the intermediate layer of the inferencer that has received the piece of first data (piece of second data). The intermediate layer may be a layer close to a final layer (or output layer).

Determination unit <NUM> is a processing unit and determines whether to add the piece of second data to the training dataset for recognizer <NUM>, according to the degree of similarity between the first contribution and second contribution, which have been calculated by computation unit <NUM>. When determining that the piece of second data should be added to the training dataset, determination unit <NUM> provides information indicating the piece of second data to management apparatus <NUM>.

Training unit <NUM> is a processing unit for generating recognizer <NUM>. Training unit <NUM> includes training dataset storage <NUM> and generates recognizer <NUM> by machine learning using the training dataset stored in training dataset storage <NUM>. Specifically, recognizer <NUM> is generated by adjusting the internal parameters so that when receiving, as a piece of input data, an image of the training dataset stored in training dataset storage <NUM>, recognizer <NUM> outputs, as a piece of output data, a numerical value indicated by a label appended to the image on the training dataset.

In addition, training unit <NUM> adds a new piece of training data (also referred to as a piece of additional data) to training dataset storage <NUM>. Training unit <NUM> receives an instruction to add a piece of additional data (also referred to as an addition instruction) from management apparatus <NUM>. The addition instruction includes a label that is a number shown in a piece of second data and information indicating the piece of second data. Training unit <NUM> receives the piece of second data from accumulator <NUM> according to the information included in the addition instruction and pairs the received piece of second data and the label included in the addition instruction. Training unit <NUM> then adds the piece of second data paired with the label to the training dataset. When determination unit <NUM> made a determination to add the piece of second data, a new piece of training data is added. Then, training unit <NUM> trains recognizer <NUM> by using a training dataset stored in training dataset storage <NUM> to which the new piece of training data has been added.

When the information indicating the piece of second data is provided by determination unit <NUM>, management apparatus <NUM> presents the information to user U. Here, management apparatus <NUM> corresponds to a presentation apparatus. User U identifies the number shown in the piece of second data according to the presented information and inputs the identified number to management apparatus <NUM> as a label.

Management apparatus <NUM> receives, from user U, the label indicating the number included in the piece of second data. Management apparatus <NUM> transmits, to training unit <NUM>, the addition instruction including the label and the information indicating the piece of second data. By receiving the addition instruction from management apparatus <NUM>, training unit <NUM> generates a training dataset.

<FIG> illustrates specifically how computation unit <NUM> in Embodiment calculates the degrees of contribution and the degree of similarity.

Image <NUM> in <FIG> is an example of a piece of input data input to recognizer <NUM> and, for example, a piece of data falsely recognized by recognizer <NUM>. That is, although image <NUM> shows the number <NUM>, recognizer <NUM> identified the number shown in image <NUM> as the number <NUM>, which is different from the number <NUM>.

Computation unit <NUM> obtains a piece of relevant data, which is obtained by inputting image <NUM> to recognizer <NUM>. The piece of relevant data is a piece of output data output by recognizer <NUM>. Computation unit <NUM> calculates degree of contribution <NUM> according to the piece of relevant data. Degree of contribution <NUM> represents the respective degrees of contribution of portions constituting the piece of input data to the piece of output data. For instance, degree of contribution <NUM> represents the respective degrees of contribution of pixels constituting image <NUM> to the fact that recognizer <NUM> identified the number shown in image <NUM> as a predetermined number. It should be noted that the predetermined number may be the number <NUM> output by recognizer <NUM> as the piece of output data or the number <NUM> shown in the piece of input data. Degree of contribution <NUM> is shown in gray-scale representation. Pixels having the highest degree of contribution are shown in white, and pixels having the lowest degree of contribution are shown in black. Regarding the gray pixels, the closer to white, the higher the degree of contribution.

Image <NUM> in <FIG> is an example of a piece of input data input to recognizer <NUM> and an example of an image stored in accumulator <NUM> but not included in the training dataset. Image <NUM>, which shows the number <NUM>, is not labeled.

Computation unit <NUM> obtains a piece of relevant data, which is obtained by inputting image <NUM> to recognizer <NUM>. Then, computation unit <NUM> calculates degree of contribution <NUM> according to the piece of relevant data. The piece of relevant data is also a piece of output data output by recognizer <NUM>. For instance, degree of contribution <NUM> represents the respective degrees of contribution of pixels constituting image <NUM> to the fact that recognizer <NUM> identified the number shown in image <NUM> as a predetermined number. It should be noted that the preceding predetermined number is the same as the predetermined number used when recognizer <NUM> received image <NUM> and outputted the piece of relevant data.

Computation unit <NUM> calculates the degree of similarity between degree of contribution <NUM> and degree of contribution <NUM>. The degree of similarity is calculated by a known art. As a specific example, by performing pooling, degree of contribution <NUM> and degree of contribution <NUM> are changed into vectors of a fixed dimension. Then, an inner product or cosine distance is obtained by using the vectors of the fixed dimension. In this way, the degree of similarity is calculated. It should be noted that in order to represent the degree of similarity within a predetermined numerical range, computation unit <NUM> may change a numerical range by performing an appropriate operation. Here, the degree of similarity is shown on an integer scale of <NUM> to <NUM>. Degree of similarity <NUM> denotes the lowest degree of similarity, and degree of similarity <NUM> denotes the highest degree of similarity.

<FIG> illustrates a first example of an additional data determination method implemented by determination unit <NUM> in Embodiment. <FIG> illustrates a method of determining which of images P, Q, R, and S stored in accumulator <NUM> should be added to the training dataset if, as an example, one piece of falsely recognized data is present.

Computation unit <NUM> calculates the degrees of similarity between the degree of contribution of the piece of falsely recognized data and the degrees of contribution of images P, Q, R, and S, which are pieces of candidate data stored in accumulator <NUM>. Here, the degrees of similarity to the degrees of contribution of images P, Q, R, and S are the numbers <NUM>, <NUM>, <NUM>, and <NUM>, respectively (see (a) in <FIG>).

When determining whether to add a piece of candidate data to the training dataset, determination unit <NUM> determines whether to add to enable a piece of candidate data more similar to the piece of falsely recognized data in terms of the contribution to be more preferentially selected from the pieces of candidate data stored in accumulator <NUM> and added to the training dataset.

For instance, determination unit <NUM> assigns degree of priority <NUM>, degree of priority <NUM>, and the following degrees of priority to pieces of data in descending order of the degree of similarity in terms of the degree of contribution. Here, the smaller the numerical vale of the degree of priority, the higher the precedence. Specifically, determination unit <NUM> assigns degree of priority <NUM> to image Q having the highest degree of similarity of <NUM> and degree of priority <NUM> to image P having the next highest degree of similarity of <NUM>. In the same way, determination unit <NUM> assigns degree of priority <NUM> and degree of priority <NUM> to image R and image S, respectively.

Determination unit <NUM> then selects a predetermined number of images in descending order of the degree of similarity, that is, in ascending order of the numerical value of the degree of priority and adds the selected images to the training dataset. For instance, when two images are added to the training dataset, determination unit <NUM> determines that image Q having degree of priority <NUM> and image P having degree of priority <NUM> will be added to the training dataset and used as pieces of training data. Determination unit <NUM> also determines that images R and S will not be added to the training dataset.

It should be noted that if pieces of falsely recognized data are present, images to be added to the training dataset are determined according to pieces of falsely recognized data. Two examples are described below regarding the additional data determination method in this case.

<FIG> illustrates a second example of the additional data determination method implemented by determination unit <NUM> in Embodiment.

<FIG> illustrates a method of determining which of images P, Q, R, and S, which are pieces of candidate data stored in accumulator <NUM>, should be added to the training dataset if, as an example, three pieces of falsely recognized data: piece of data A, piece of data B, and piece of data C are present.

Computation unit <NUM> calculates the degrees of similarity between the degrees of contribution of piece of data A, piece of data B, and piece of data C, which are three pieces of falsely recognized data, and the degrees of contribution of images P, Q, R, and S, which are pieces of candidate data stored in accumulator <NUM>. Here, the degrees of similarity between the degree of contribution of piece of falsely recognized data A and the degrees of contribution of images P, Q, R, and S are the numbers <NUM>, <NUM>, <NUM>, and <NUM>, respectively. The degrees of similarity between the degree of contribution of piece of falsely recognized data B and the degrees of contribution of images P, Q, R, and S are the numbers <NUM>, <NUM>, <NUM>, and <NUM>, respectively. The degrees of similarity between the degree of contribution of piece of falsely recognized data C and the degrees of contribution of images P, Q, R, and S are the numbers <NUM>, <NUM>, <NUM>, and <NUM>, respectively (see <FIG>).

Determination unit <NUM> calculates a plurality of degrees of contribution including the degree of contribution of each of pieces of falsely recognized data. According to a representative value calculated from a plurality of degrees of similarity including the degrees of similarity between the calculated plurality of degrees of contribution and the degree of contribution of a piece of candidate data, determination unit <NUM> determines whether to add the piece of candidate data to the training dataset.

For instance, the greatest value among the degrees of similarity can be used as a representative value in terms of the degree of similarity. In the example illustrated in <FIG>, the degrees of similarity between the degrees of contribution of piece of data A, piece of data B, and piece of data C, which are pieces of falsely recognized data, and the degree of contribution of image P are the numbers <NUM>, <NUM>, and <NUM>, respectively. Thus, the representative value in terms of the degree of similarity for image P is the number <NUM>, which is the greatest value among the numbers <NUM>, <NUM>, and <NUM>. Similarly, representative values in terms of the degree of similarity for images Q, R, and S are the numbers <NUM>, <NUM>, and <NUM>, respectively.

Determination unit <NUM> then determines that a predetermined number of data selected in descending order of representative value in terms of the degree of similarity will be added to the training dataset. The processing of selecting data to be added to the training dataset is the same as the processing described in the case in which one piece of falsely recognized data is present (see <FIG>). Thus, an explanation for the processing is omitted.

It should be noted that a mean value among the degrees of similarity may be used as a representative value among the degrees of similarity.

Thus, in a case in which pieces of falsely recognized data are present, which image should be added to the training dataset can be determined using the degrees of similarity to all the candidate data.

<FIG> is a third example of the additional data determination method implemented by determination unit <NUM> in Embodiment.

<FIG> illustrates a method of using a predetermined number of pieces of candidate data for each piece of falsely recognized data after the degrees of similarity illustrated in <FIG> are calculated.

Here, the plurality of degrees of similarity include the degrees of similarity between first contributions and second contributions including the second contribution of each of the pieces of second data.

Regarding the calculation of the representative value, determination unit <NUM> selects, for each of the pieces of first data, a predetermined number of degrees of similarity from the plurality of degrees of similarity and calculates a representative value for each of the pieces of second data, by using the selected predetermined number of degrees of similarity. Regarding the determination of whether to add the piece of second data to the training dataset, determination unit <NUM> determines whether to add the piece of second data to the training dataset according to the calculated representative values.

For instance, if the predetermined number is two, determination unit <NUM> selects two degrees of similarity for each of pieces of falsely recognized data A, B, and C. Regarding the method of selecting the two degrees of similarity, for instance, a method of preferentially selecting a piece of data having a higher degree of similarity may be employed. Although the foregoing selecting method is described hereinafter, another selecting method may be employed.

Regarding piece of falsely recognized data A, determination unit <NUM> selects a degree of similarity of <NUM> (similarity to image Q) and a degree of similarity of <NUM> (similarity to image P) as two relatively high degrees of similarity. Here, N/A is assigned to images R and S, which are pieces of data not selected (see <FIG>). N/A means that the degree of similarity is not taken into account.

Similarly, regarding piece of falsely recognized data B, determination unit <NUM> selects a degree of similarity of <NUM> (similarity to image S) and a degree of similarity of <NUM> (similarity to image R) as two relatively high degrees of similarity. In addition, regarding piece of falsely recognized data C, determination unit <NUM> selects a degree of similarity of <NUM> (similarity to image Q) and a degree of similarity of <NUM> (similarity to image S) as two relatively high degrees of similarity.

Determination unit <NUM> then calculates representative values in terms of the degree of similarity, by using the degrees of similarity determined as described above. Specifically, without taking into account the not-selected data, that is, the data to which N/A is assigned, determination unit <NUM> determines that the representative values in terms of the degree of similarity for images P, Q, R, and S are the numbers <NUM>, <NUM>, <NUM>, and <NUM>, respectively.

Determination unit <NUM> adds, to the training dataset, a predetermined number of data selected in descending order of representative value in terms of the degree of similarity. The processing of selecting data to be added to the training dataset is the same as the processing described in the case in which one piece of falsely recognized data is present (<FIG>). Thus, an explanation for the processing is omitted.

Thus, it is possible to suppress determination unit <NUM> from determining that pieces of candidate data similar exclusively to a specific piece of falsely recognized data in terms of the degree of contribution will be added to the training dataset. Depending on the degrees of similarity between the pieces of falsely recognized data and images P, Q, R, and S accumulated in accumulator <NUM>, pieces of candidate data similar exclusively to piece of falsely recognized data A among pieces of falsely recognized data A, B, and C in terms of the contribution, may be added to the training dataset (see <FIG>). In this case, while it is possible to suppress another false inference from being made for images similar to piece of falsely recognized data A, it may not be possible to suppress another false inference from being made for images similar to pieces of falsely recognized data B and C. Thus, as described above, by calculating representative values in terms of the degree of similarity without taking into account the not-selected data, it is possible to suppress a false inference from being made, evenly among pieces of falsely recognized data A, B, and C.

Next, a processing method implemented by the processing system having the above configuration, that is, an information processing method is described.

<FIG> is a flowchart illustrating a first example of processing performed by the processing system in Embodiment.

The processing illustrated in <FIG> is an information processing method implemented by a computer.

As illustrated in <FIG>, in step S1, a piece of first data and a piece of second data are obtained, the piece of second data being not included in the training dataset used for training the inferencer.

In step S2, a first contribution representing the respective contributions of portions constituting the piece of first data to a piece of first output data output by inputting the piece of first data to the inferencer is calculated by using a piece of first relevant data obtained by inputting the piece of first data to the inferencer trained by machine learning.

In step S3, a second contribution representing the respective contributions of portions constituting the piece of second data to a piece of second output data output by inputting the piece of second data to the inferencer is calculated by using a piece of second relevant data obtained by inputting the piece of second data to the inferencer.

In step S4, whether to add the piece of second data to the training dataset for the inferencer is determined according to the degree of similarity between the first contribution and the second contribution.

<FIG> is a flowchart illustrating the processing performed by the processing system in Embodiment.

In step S101, controller <NUM> obtains a piece of validation data, inputs the piece of obtained validation data to recognizer <NUM> as a piece of input data, and obtains a piece of output data. In addition, computation unit <NUM> obtains a piece of relevant data, which is obtained by inputting the piece of input data to recognizer <NUM>.

In step S102, controller <NUM> determines whether the piece of output data obtained in step S101 matches a label appended to the piece of validation data. If it is determined that the piece of output data and the label do not match, controller <NUM> identifies the piece of validation data input to recognizer <NUM> in step S101 as a piece of falsely recognized data.

In step S103, regarding the piece of data identified as a piece of falsely recognized data in step S102, computation unit <NUM> calculates the respective degrees of contribution of portions constituting the piece of input data to the piece of output data obtained in step S101, by using the piece of relevant data obtained in step S101.

In step S104, controller <NUM> obtains data accumulated in accumulator <NUM> as pieces of candidate data, inputs the obtained pieces of candidate data to recognizer <NUM> as pieces of input data, and obtains pieces of output data. In addition, computation unit <NUM> obtains pieces of relevant data, which are obtained by inputting the pieces of input data to recognizer <NUM>.

In step S105, regarding each of the pieces of candidate data input to recognizer <NUM> in step S104, computation unit <NUM> calculates the respective degrees of contribution of portions constituting the piece of input data to the corresponding piece of output data obtained in step S104, by using the corresponding piece of relevant data obtained in step S104.

In step S106, computation unit <NUM> calculates the degrees of similarity between the degrees of contribution calculated in step S103 and the degrees of contribution calculated in step S104.

In step S107, determination unit <NUM> selects, from the pieces of candidate data, a piece of candidate data that has a degree of contribution having a higher degree of similarity to that of the piece of falsely recognized data and determines the piece of candidate data as a piece of additional data. Determination unit <NUM> transmits information indicating the determined piece of additional data to management apparatus <NUM>, which then presents the information to user U. User U refers to the piece of additional data and inputs, to management apparatus <NUM>, a label to be appended to the piece of additional data. Management apparatus <NUM> provides the input label to training unit <NUM> by, for example, transmitting the input label to training unit <NUM> through a communication line.

In step S108, training unit <NUM> obtains the label to be appended to the piece of additional data from management apparatus <NUM> by, for example, receiving the label through the communication line.

In step S109, training unit <NUM> adds the labeled piece of additional data to the training dataset.

In step S110, training unit <NUM> generates recognizer <NUM> by machine learning that uses a training dataset including the piece of additional data added in step S109. After step S110 is performed, the processing illustrated in <FIG> ends.

By performing the processing, the processing system can efficiently improve the accuracy of recognition by recognizer <NUM>.

In Variation <NUM>, regarding an information processing method that enables efficient improvement in the accuracy of inference by an inferencer, a technique capable of improving the performance of recognizer <NUM> to at least a predetermined level is described.

The configuration of a processing system according to Variation <NUM> is the same as that of processing system <NUM> in Embodiment (see <FIG>).

Portions different between processing performed by the processing system according to Variation <NUM> and the processing performed by processing system <NUM> in Embodiment are described below. It should be noted that in the processing performed by the processing system according to Variation <NUM>, steps identical to those (see <FIG>) performed by processing system <NUM> in Embodiment are assigned the same reference symbols, and detailed explanations are omitted.

<FIG> is a flowchart illustrating processing performed by the processing system in Variation <NUM>.

Steps S101 to S110 in <FIG> are the same as those illustrated in <FIG>.

In step S111, controller <NUM> evaluates the performance of recognizer <NUM> generated in step S110. When evaluating the performance, controller <NUM> inputs pieces of validation data to recognizer <NUM> as pieces of input data and obtains pieces of output data. The percentage that a label appended in advance to a piece of validation data matches a piece of output data is calculated as the value of performance. For instance, let's assume that <NUM> pieces of validation data are input. In <NUM> of the <NUM> pieces of validation data, labels match pieces of output data, and in five of the <NUM> pieces of validation data, labels do not match pieces of output data. In this case, the value of performance is <NUM>%.

In step S112, controller <NUM> determines whether the performance evaluated in step S111 is at or above the predetermined level. Specifically, controller <NUM> compares the value of performance calculated in step S111 and a predetermined value (e.g., <NUM>%). If it is determined that the value of performance is greater than or equal to the predetermined value, controller <NUM> determines that the performance is at or above the predetermined level. If it is determined that the performance of recognizer <NUM> is at or above the predetermined level (Yes in S112), the processing illustrated in <FIG> ends. Otherwise (No in step S112), step S101 is performed again.

In this way, training data are added until the performance of recognizer <NUM> is at or above the predetermined level. Thus, the performance of recognizer <NUM> can be improved to at least the predetermined level.

It should be noted that if the determination that the value of performance is not at or above the predetermined value has been made a predetermined number of times or more in step S112, the processing may be discontinued without performing step S101. In this case, an error message indicating the discontinuation of the processing may be presented.

By performing the above processing, the processing system can improve the performance of the recognizer to at least the predetermined level, when efficiently improving the accuracy of recognition by the recognizer.

In Variation <NUM>, regarding an information processing method that enables efficient improvement in the accuracy of inference by an inferencer, a technique of repeating the processing of improving the accuracy of inference is described.

<FIG> is a flowchart illustrating processing performed by a processing system in Variation <NUM>.

The configuration of the processing system according to Variation <NUM> is the same as that of processing system <NUM> in Embodiment (see <FIG>).

Portions different between the processing performed by the processing system according to Variation <NUM> and the processing performed by processing system <NUM> in Embodiment are described below. It should be noted that in the processing performed by the processing system according to Variation <NUM>, steps identical to those (see <FIG>) performed by processing system <NUM> in Embodiment are assigned the same reference symbols, and detailed explanations are omitted.

<FIG> is a flowchart illustrating the processing performed by the processing system in Variation <NUM>.

Here, new images are repeatedly added to and accumulated in accumulator <NUM>. For instance, images obtained through sensing at intervals of minutes by an in-vehicle camera are received through a communication line and accumulated in accumulator <NUM>.

In step S121, controller <NUM> determines whether at least a predetermined number of data are accumulated in accumulator <NUM>. If controller <NUM> determines that at least a predetermined number of data are accumulated (Yes in step S121), step S122 is performed. Otherwise (No in step S121), step S121 is performed again. That is, until a predetermined number of data are accumulated in accumulator <NUM>, the processing performed by controller <NUM> does not proceed from step S121. It should be noted that the predetermined number is, for example, around <NUM>.

In step S122, controller <NUM> deletes the data accumulated in accumulator <NUM>.

After step S122, controller <NUM> performs step S121 again. In this way, in the situation in which equipment other than accumulator <NUM> accumulates data in accumulator <NUM>, every time at least a predetermined number of data are accumulated in accumulator <NUM>, processing system <NUM> performs steps S101 to S110 to add additional data to the training dataset.

By performing the above processing, the processing system can repeatedly improve the accuracy of recognition, when efficiently improving the accuracy of recognition by the recognizer.

Thus, in the information processing methods described in Embodiment and Variations <NUM> and <NUM>, it is determined that a piece of second data having a contribution similar to that of a piece of first data will be added to the training dataset. The contribution represents an effect on the result of inference (processing performed) by the inferencer. Thus, a piece of second data similar to a piece of first data in terms of the effect on the result of inference by the inferencer can be selected. By training the inferencer by using the selected piece of second data as a piece of training data, it is more likely to suppress the inferencer from making a false inference for the piece of first data and pieces of data similar to the piece of first data. In addition, by adding a piece of data useful for the inferencer to the training dataset, it is possible to avoid randomly adding a vast amount of data. Hence, according to the above-described aspect, the accuracy of inference by the inferencer can be efficiently improved.

In addition, it is determined that a piece of second data having a contribution similar to that of a piece of falsely inferred data will be added to the training dataset. By training the inferencer by using a training dataset to which the piece of second data has been added in accordance with the determination, it is possible to suppress the inferencer from making another false inference for pieces of data similar to the piece of falsely inferred data. Hence, according to the above-described aspect, the accuracy of inference by the inferencer can be efficiently improved.

The inferencer is trained using the training dataset to which the piece of second data has been added in accordance with the determination that the piece of second data will be added to the training dataset. This makes it possible to suppress the inferencer from making another false inference for pieces of data similar to the piece of first data. Hence, according to the above-described aspect, the accuracy of inference by the inferencer can be efficiently improved.

In addition, the following determination can be made: a piece of second data more similar to the piece of first data in terms of the contribution is selected from the prepared pieces of second data, and the selected piece of second data is added to the training dataset. Hence, according to the above-described aspect, the accuracy of inference by the inferencer can be more efficiently improved.

In addition, if pieces of first data are present, a piece of second data to be added to the training dataset is determined using representative values calculated from the degrees of similarity calculated for each of the pieces of first data. If pieces of first data are present, a plurality of degrees of similarity are calculated. However, it is difficult to determine, by using the plurality of degrees of similarity, which piece of second data should be added to the training dataset. Thus, by using the representative values calculated from the plurality of degrees of similarity, it is possible to readily determine which piece of second data should be added to the training dataset. Hence, according to the above-described aspect, the accuracy of inference by the inferencer can be readily improved.

If pieces of first data are present, it is possible to suppress the determination that pieces of second data similar exclusively to a specific piece of first data in terms of the contribution will be added to the training dataset, from being made. If pieces of first data are present, in some cases, it is determined that pieces of second data similar exclusively to a specific piece of first data in terms of the contribution will be added to the training dataset. In this case, while it is possible to suppress the inferencer from making another false inference for pieces of data similar to the specific piece of first data, it may not be possible to suppress the inferencer from making another false inference for the pieces of first data other than the specific piece of first data among the pieces of first data. Thus, it may not be possible to suppress the inferencer from making a false inference, evenly among the pieces of first data. According to the above-described aspect, however, it is possible to suppress the inferencer from making a false inference, evenly among the pieces of first data. Hence, according to the above-described aspect, the accuracy of inference by the inferencer can be efficiently improved, and improvements can be made evenly among the pieces of first data.

In addition, it is possible to suppress false identification in identification processing performed by the identifier or false detection in detection processing performed by the detector. Hence, according to the above-described aspect, the accuracy of identification by the identifier or the accuracy of detection by the detector can be efficiently improved.

In addition, it is possible to efficiently improve the accuracy of inference for the piece of sensing data.

In addition, efficient improvement in the accuracy of inference by the inferencer can be more readily made by using the piece of first output data as the piece of first relevant data.

In addition, information indicating the piece of second data determined to be added to the training dataset is presented. The presentation apparatus presents the piece of second data to a user according to the information and receives, from the user, a label regarding the information indicating the piece of second data. The data is added to the training dataset by using the input label. Hence, by receiving, from the user, the label for the data to be added, the accuracy of inference by the inferencer can be more efficiently improved.

It should be noted that in Embodiment and Variations <NUM> and <NUM>, each structural element may be dedicated hardware or be caused to function by running a software program suitable for the structural element. To cause each structural element to function, a program running unit, such as a CUP or processor, may read and run a software program stored in a recording medium, such as a hard disk or semiconductor memory. Here, by running a program described below, the processing systems described in Embodiment and Variations <NUM> and <NUM> are caused to function.

The program is an information processing method implemented by a computer. By running the program, the computer implements the information processing method. The information processing method includes: obtaining a piece of first data and a piece of second data, the piece of second data being not included in a training dataset used for training an inferencer; calculating, by using a piece of first relevant data, a first contribution representing the respective contributions of portions constituting the piece of first data to a piece of first output data output by inputting the piece of first data to the inferencer, the piece of first relevant data being obtained by inputting the piece of first data to the inferencer trained by machine learning using the training dataset; calculating, by using a piece of second relevant data, a second contribution representing the respective contributions of portions constituting the piece of second data to a piece of second output data output by inputting the piece of second data to the inferencer, the piece of second relevant data being obtained by inputting the piece of second data to the inferencer; and determining whether to add the piece of second data to the training dataset for the inferencer, according to the degree of similarity between the first contribution and the second contribution.

Claim 1:
An information processing method implemented by a computer, the information processing method comprising:
obtaining (S1) a piece of first data and a piece of second data, the piece of second data being not included in a training dataset, said training dataset used for training an inferencer (<NUM>), and the inferencer configured to output a recognition or detection result for one or more pieces of data, wherein each of the piece of first data and the piece of second data is a piece of sensing data comprising a piece of image data obtained through sensing by a camera;
wherein the piece of first data is a piece of data for which a false recognition or detection result is output;
calculating (S2), by using a piece of first relevant data, a first contribution, to a first recognition or detection result output, representing respective contributions of portions constituting the piece of first data, to a piece of first output data output by inputting the piece of first data to the inferencer (<NUM>), the piece of first relevant data being obtained by inputting the piece of first data to the inferencer (<NUM>) trained by machine learning using the training dataset;
calculating (S3), by using a piece of second relevant data, a second contribution, to a second recognition or detection result output, representing respective contributions of portions constituting the piece of second data, to a piece of second output data output by inputting the piece of second data to the inferencer (<NUM>), the piece of second relevant data being obtained by inputting the piece of second data to the inferencer (<NUM>); and
determining (S4) whether to add the piece of second data to the training dataset for the inferencer (<NUM>), according to a degree of similarity between the first contribution and the second contribution, wherein, if the second contribution comprises a higher degree of similarity to the first contribution, the piece of second data is added to the training data, or, if the second contribution does not comprise a higher degree of similarity to the first contribution, the piece of second data is not added to the training data (S107).