COMPUTER-READABLE RECORDING MEDIUM STORING MACHINE LEARNING PROGRAM, MACHINE LEARNING METHOD, AND INFORMATION PROCESSING DEVICE

A non-transitory computer-readable recording medium storing a machine learning program for causing a computer to execute a process, the process includes inputting moving image data that includes at least a first frame image and a second frame image to a first machine learning model trained by using training data, and training an encoder by detecting a first object and a second object from the first frame image and the second frame image, respectively, based on an inference result by the first machine learning model, determining identity between the first object and the second object that have been detected, and inputting, to the encoder, first data in a first image area that includes the first object and second data in a second image area that includes the second object, the first object and the second object having been determined to have the identity.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2022-163400, filed on Oct. 11, 2022, the entire contents of which are incorporated herein by reference.

FIELD

The embodiment discussed herein is related to a machine learning program, a machine learning method, and an information processing device.

BACKGROUND

There has been known an object tracking technique for tracking an object in a moving image using machine learning. The object tracking technique includes an object detection technique and a tracking technique, and performance of object tracking depends on performance of object detection. When it fails to obtain a sufficient amount of training data for the object detection, the performance of the object detection may be lowered due to over-training.

There has been known a label propagation method as an example of a technique for increasing the training data amount. The label propagation method propagates (copies) a data label to unlabeled data in temporal or spatial proximity using a result of the object tracking. The label propagation method generates a new label based on an inference result inferred with a confidence level equal to or higher than a predetermined value.

SUMMARY

According to an aspect of the embodiments, a non-transitory computer-readable recording medium storing a machine learning program for causing a computer to execute a process, the process includes inputting moving image data that includes at least a first frame image and a second frame image to a first machine learning model trained by using training data, and training an encoder by detecting a first object and a second object from the first frame image and the second frame image, respectively, based on an inference result by the first machine learning model, determining identity between the first object and the second object that have been detected, and inputting, to the encoder, first data in a first image area that includes the first object and second data in a second image area that includes the second object, the first object and the second object having been determined to have the identity.

DESCRIPTION OF EMBODIMENTS

Since the training data obtained by the label propagation method is generated based on the inference result inferred with the confidence level equal to or higher than the predetermined value, it may be difficult to consider a situation where there is a perturbation (e.g., influence of an occlusion, etc.) This is because the confidence level is lowered when a perturbation occurs. Therefore, even when the training data is increased by the label propagation method, it may be difficult to improve the performance of the object detection.

[A] Related Art

FIG.1is a diagram schematically illustrating an exemplary label propagation method.

As illustrated inFIG.1, in the related art, unknown unlabeled moving image data20is input to an object tracking model (not illustrated) trained by using existing training data (not illustrated). The unlabeled moving image data20includes a plurality of frame images21a,21b, and21c.

An inference process is performed on the unlabeled moving image data20using the object tracking model. As a result, an object22is detected. Boundary position information associated with the object22may be indicated by a bounding box23. The object tracking model estimates a class24of the object22. InFIG.1, the class24is an “automobile”. In the present specification, the “object” is not limited to an automobile as long as it is an object to be detected. As an example, the “object” may be an automobile, a truck, a motorcycle, a bicycle, or a person.

The bounding box23of each object22may be associated with identification information25(object identifier (ID)) for identifying identity of the object22and a confidence level26. The confidence level (confidence)26may be a weight of a determination result of the class24of the object22. It is indicated that an object detection model has made the determination with higher accuracy as a numerical value of the confidence level26is closer to 1.

InFIG.1, labeled data is newly generated using at least one of the bounding box23, the class24, the identification information25, the confidence level26, and the like, which are object tracking results.

As an example, a new label corresponding to the bounding box23(and the class24) with the confidence level26higher than a predetermined value may be generated. InFIG.1, the bounding box23, the class24, and the like may be a new label in the frame image21aand the frame image21c.

In the example illustrated inFIG.1, the object22is not detected in the frame image21bat time t. However, the object22is detected in the preceding and subsequent frame images21aand21c(at time t−1 and t+1). In this case, a bounding box27at the time t may be added as a complement based on the bounding box23or the like in the preceding and subsequent frame images21aand21c. The complementary bounding box27and the class24may be used as a new label to generate new labeled data.

Note that the complementary bounding box27may be generated in a case where the confidence level26of the object22detected in the preceding and subsequent frame images (21aand21cinFIG.1) is equal to or higher than a predetermined threshold value. Moreover, new labeled data may be generated in consideration of not only the confidence level in the object detection but also the confidence level based on a tracking algorithm.

In the related art illustrated inFIG.1, a new label is generated based on an inference result in which the confidence level26is equal to or higher than a predetermined value. On the other hand, the performance of the object detection may be rather degraded in a case where a new label is generated based on an inference result in which the confidence level26is lower than the predetermined value.

Since the confidence level26is likely to be lower when there is a “perturbation”, data when there is a “perturbation” is often not considered for generation of new labeled data. Therefore, according to the method illustrated inFIG.1, labeled data in consideration of a case where the confidence level26is lowered by a perturbation may not be generated. Note that the “perturbation” may include, for example, that a part of the object is occluded by another object, a motion blur (e.g., blurring that occurs when a moving object is captured by a camera), an angular change of the object, an influence of illuminance, and the like.

An object detection model robust to an influence of a perturbation may not be obtained even if the object detection model is trained by using the labeled data generated by the method illustrated inFIG.1as new training data. In view of the above, it is assumed that an object detection model robust to a perturbation is generated.

FIG.2is a diagram for explaining contrastive learning. The contrastive learning is a type of self-supervised learning.FIG.2exemplifies, as a simple example, a case where image data reflecting a “cat” is input as input data30and an object label “cat” is output as an output.

In the contrastive learning, two pieces of extended data31(31aand31b) are obtained from the input (input data30) through two types of data extension. For example, the data extension may be processing of making, with respect to the input data30that is an original image, changes of parallel translation, rotation, scaling, vertical inversion, horizontal inversion, brightness adjustment, and a plurality of combinations thereof.

A first feature vector33and a second feature vector34are obtained by each of the extended data31aand31bobtained by the two types of data extension being input to a contrastive learning model32.

The two pieces of extended data31aand31bare data having been subject to different changes without changing the essence of the object. Therefore, the first feature vector33and the second feature vector34match or are similar due to the unchanged essence of the object.

In the contrastive learning, machine learning of the contrastive learning model32is carried out such that a degree of matching (similarity) between the two, which are the first feature vector33(zi) and the second feature vector34(zj), becomes higher. The contrastive learning model32is an encoder. As an example, a loss function Lφ=−sim(zi, zj) may be calculated, and the parameter φ may be updated such that a value of the loss function Lφis minimized.

Hereinafter, an embodiment of techniques capable to reduce an influence of a perturbation to improve performance of object detection will be described with reference to the drawings. Note that the embodiment to be described below is merely an example, and there is no intention to exclude application of various modifications and techniques not explicitly described in the embodiment. For example, the present embodiment may be variously modified and implemented without departing from the gist thereof. Furthermore, each drawing is not intended to include only the constituent elements illustrated in the drawing, and may include another function and the like.

Hereinafter, each of the same reference signs denotes a similar part in the drawings, and thus description thereof will be omitted.

[B-1] Description of Training Process According to Embodiment

FIG.3is a diagram illustrating an outline of training of a contrastive learning model230according to the embodiment. Image data226and227of a plurality of objects223aand223brecognized as the same object by an object tracking model (to be described later) in frame images221(221aand221b) at different times of unlabeled moving image data220(moving image) are used for contrastive learning. The contrastive learning model230is trained to increase a degree of matching between a first feature vector231and a second feature vector232obtained as outputs when the respective image data226and227are output to the contrastive learning model230.

According to the method of the present embodiment, the images (226and227) of the pair of the objects (223aand223b), which are the same object reflected in the frame images221at different times, are used as data to be input to the contrastive learning model230instead of being based on two types of data extension. The training of the contrastive learning model230will be described later.

FIG.4is a diagram illustrating an exemplary training process of an object tracking model210by an information processing device1(FIG.11) according to the embodiment.FIG.5is a diagram schematically illustrating an example of training data300for object detection according to the embodiment.

The training data300(learning data) may be a data set for training of the object tracking model210. The training data300may be a moving image, and may include a plurality of frame images200a,200b, and200c(which may be collectively referred to as frame images200). The number of the frame images200is not limited to the case illustrated inFIGS.4and5.

The training data300inFIG.5includes objects201ato201e. The object201ais an automobile,201bis a motorcycle,201cis a truck,201dis an automobile, and201eis an automobile. The objects201ato201emay be collectively referred to as objects201.

Boundary position information of the respective objects201may be indicated by bounding boxes202ato202e(which may be collectively referred to as bounding boxes202). The boundary position information may include one plane coordinate of a height, a width, and a vertex of each of the bounding boxes202.

The respective objects201may be associated with classes205ato205e(which may be collectively referred to as classes205) indicating object types such as an automobile, a motorcycle, a truck, and the like.

As illustrated inFIG.4, the object tracking model210may include an object detection model212and a tracking model214. The object detection model212detects the objects201from the moving image. The tracking model214allocates the same identification information225(object ID) to the same objects201among the plurality of frame images200a,200b, and200cincluded in the moving image.

An existing method may be used as the object detection model212. As an example, an object detection method such as Regions with Convolutional Neural Network (R-CNN), You Only Look Once (YOLO), Single Shot MultiBox Detector (SSD), Deformed Convolutional Networks (DCN), End-to-End Object Detection with Transformers (DETR), or the like may be used. Therefore, detailed description will be omitted.

Various methods in existing multiple object tracking (MOT) techniques may be used as the tracking model214. As an example, feature vectors for the bounding boxes202for the objects201detected by the object detection model212are calculated. Motion prediction of the objects201is carried out using optical flow estimation and a Kalman filter. Matching of the objects201being tracked is carried out using the feature vectors and a result of the motion prediction. As a result, the same identification information225(object ID) is allocated to the objects201determined to be the same. For example, the tracking model214is a model using ByteTrack. However, the tracking model214is not limited to this case.

Parameters of the object detection model212and the tracking model214are adjusted to optimum values based on machine learning.

FIG.6illustrates an example of the unlabeled moving image data220to be input to the trained object tracking model210illustrated inFIG.4.FIG.6also illustrates results (e.g., bounding boxes222, identification information225, and confidence levels228) of label estimation by the object tracking model210(FIG.5).

The unlabeled moving image data220is an example of moving image data. The unlabeled moving image data220includes a plurality of frame images221-1to221-3. The information processing device1inputs the unlabeled moving image data220to the object tracking model210with a fixed parameter.

The information processing device1detects an object223-1(e.g., first object) from the frame image221-1by inference processing of the object tracking model210. Likewise, the information processing device1detects an object223-2from the frame image221-2and detects an object223-3from the frame image221-3by the inference processing of the object tracking model210.

The information processing device1determines identity of the plurality of detected objects223-1to223-3using the object tracking model210. For example, the information processing device1determines the identity between the object223a(e.g., first object) and the object223b(e.g., second object) based on the identification information225(e.g., object ID), which is one of inference results of the object tracking model210. The identification information225(e.g., object ID) may be information for identifying the identity of the objects223. The identity means that the objects223are the same individual.

The same identification information225(e.g., object ID) is allocated to the same objects223even among the different frame images221-1to221-3. For example, the tracking model214(e.g., tracking algorithm) of the object tracking model210links the same objects223in images at different times.

The information processing device1may estimate bounding boxes222-1to222-3as an example of the boundary position information regarding the objects223. Furthermore, the information processing device1may estimate a class (see reference sign24inFIG.1, etc.) Illustration of the class is omitted inFIG.6.

The information processing device1may also estimate confidence levels228-1to228-3. The confidence levels228-1to228-3may be weights of class determination results of the objects223. The confidence levels228-1to228-3are different among the frame images221corresponding to different times. This is due to a perturbation such as presence of an occlusion224that occludes the objects223or the like.

FIG.7is a diagram illustrating an exemplary training process of the contrastive learning model230in the information processing device1according to the embodiment.

InFIG.7, the frame image221-1and the frame image221-2are examples of a first frame image and a second frame image. The first frame image and the second frame image may be any frame images corresponding to times different from each other.

In a case where the object223-1(e.g., first object) and the object223-2(e.g., second object) are determined to be identical, the first data226and the second data227are input to the contrastive learning model230as paired images. As a result, the contrastive learning model230is trained. The contrastive learning model230is an exemplary encoding unit (e.g., encoder) to which the first data226and the second data227are input. A plurality of pairs of images may be obtained. The number of pairs may be set to a number sufficient for the contrastive learning.

The first data226is image data in a first image area including the object223-1(e.g., first object) in the unlabeled moving image data220. The second data227is image data in a second image area including the object223-2(e.g., second object) in the unlabeled moving image data220.

As an example, the first image area may be the rectangular bounding box222-1surrounding the object223-1(e.g., first object), and the second image area may be the rectangular bounding box222-2surrounding the object223-2(e.g., second object).

The first data226may be image data cut out from the frame image221-1(e.g., first frame image) according to the shape and position of the bounding box222-1. The second data227may be image data cut out from the frame image221-2(e.g., second frame image) according to the shape and position of the bounding box222-2. However, the first data226and the second data227are not limited to this case.

It is sufficient if the first data226is image data of an area including the object223-1(e.g., first object). The first data226may be the entire frame image221-1, or may be a part of the frame image221-1. In the example ofFIG.7, the first data226is a part of the frame image221-1.

Likewise, it is also sufficient if the second data227is image data of an area including the object223-2(e.g., second object). The second data227may be the entire frame image221-2, or may be a part of the frame image221-2. In the example ofFIG.7, the second data227is a part of the frame image221-2.

The first feature vector231and the second feature vector232are obtained by each of the first data226including the object223-1and the second data227including the object223-2being input to the contrastive learning model230. The first feature vector231is an exemplary first feature, and the second feature vector232is an exemplary second feature.

For example, the first data226and the second data227are image data for the same object223. However, since the corresponding times (t and t−1) are different, there are differences between the first data226and the second data227, such as an angle of the object223, presence or absence of the occlusion224, presence or absence of a motion blur, a difference in illumination, and the like.

Therefore, training similar to original contrastive learning may be carried out using the first data226and the second data227instead of extended data31aand31bobtained by data extension from the same data.

Moreover, the first data226and the second data227are generated in consideration of the differences such as the angle of the object223, the presence or absence of the occlusion224, the presence or absence of the motion blur, the difference in illumination, and the like, for example, the “perturbation”. Therefore, it becomes possible to train the contrastive learning model230in consideration of the “perturbation” by training the contrastive learning model230using the first data226and the second data227.

FIG.8is a diagram illustrating an exemplary training process of an object detection model240in the information processing device1according to the embodiment. The information processing device1newly trains the object detection model240using the trained contrastive learning model230illustrated inFIG.7. The object detection model240is an exemplary second machine learning model that detects an object from an image based on the trained contrastive learning model230.

The object detection model240may be a deep neural network (DNN) in which a hidden layer (intermediate layer) is multi-layered between an input layer and an output layer. A known object detection method may be used as the object detection model240in a similar manner to the object detection model212. Therefore, detailed description will be omitted. The object detection model240may be the object detection model212, or may be a different model.

Training image data250is prepared. As an example, the training image data250may be supervised training data including image data and a label253. The information processing device1divides the training image data250into a plurality of divided regions251(251-1,251-2, and so on) to obtain a plurality of divided images252(252-1,252-2, and so on). The plurality of divided images252(252-1,252-2, and so on) may be called patches. Each of the divided regions251may have a portion overlapping each other. The generation of the divided images252may be performed by a sliding window technique. The sliding window technique obtains a patch for each position while sliding a frame called a window.

The information processing device1inputs each of the divided images252to the trained contrastive learning model230(seeFIG.7) with a fixed parameter, and obtains a feature vector (e.g., representation vector) of each of them. As an example, the information processing device1calculates, for each frame, a feature map233represented by the feature vector of each of the divided regions251.

At a time of inference as well, the information processing device1divides the input image into the divided images252. The information processing device1inputs each of the divided images252to the trained contrastive learning model230(seeFIG.7) with the fixed parameter to generate a feature map. The information processing device1inputs the feature map233to the trained object detection model240, and infers the label253such as the class, the boundary position information (bounding box222), the confidence level228, and the like.

The information processing device1generates the feature map233robust (resistant) to a perturbation by the inference result of the contrastive learning model230trained by using the frame images221at different times. Then, the information processing device1trains the object detection model240based on the feature map233. Therefore, according to the machine learning method according to the present embodiment, it becomes possible to reduce an influence of a perturbation and to improve performance of object detection.

[B-2] Exemplary Functional Configuration of Information Processing Device1According to Embodiment

[B-2-1] Training Phase

FIG.9is a block diagram illustrating an exemplary functional configuration in a training phase by the information processing device1according to the embodiment. The information processing device1is an exemplary computer that performs a training process.

As illustrated inFIG.9, the information processing device1may illustratively include a storage unit311, an acquisition unit312, a first training execution unit313, an object detection unit314, an ID allocation unit315, an image acquisition unit316, a second training execution unit317, a third training execution unit318, and a patch generation unit319. The configuration of those312to319is an example of a control unit320.

The storage unit311is an exemplary storage area, and stores various types of data to be used by the information processing device1. The storage unit311may be implemented by, for example, a storage area included in one or both of a memory unit12and a storage device14illustrated inFIG.11to be described later.

As illustrated inFIG.9, the storage unit311may illustratively be capable of storing the training data300, the object tracking model210, the unlabeled moving image data220, the contrastive learning model230, the object detection model240, the training image data250, and the like.

Information stored in the storage unit311may be in a table format or another format. As an example, at least one of the pieces of information stored in the storage unit311may be in various formats such as a database (DB), an array, or the like.

The acquisition unit312obtains various kinds of information to be used in the information processing device1. For example, the acquisition unit312obtains the training data300from the storage unit311. The training data300(e.g., learning data) may be a data set for training of the object tracking model210.

The first training execution unit313inputs the training data300to the object tracking model210(e.g., first machine learning model) to train the object tracking model210.

The object detection unit314may input, to the trained object tracking model210, the unlabeled moving image data220including at least the frame image221-1(example of the first frame image) and the frame image221-2(example of the second frame image). The object detection unit314detects the object223-1(example of a first object) from the frame image221-1based on the inference result of the object tracking model210. Likewise, the object detection unit314detects the object223-2(example of a second object) from the frame image221-2based on the inference result of the object tracking model210.

The ID allocation unit315determines identity between the first object (object223-1, etc.) and the second object (object223-2, etc.) based on the inference result of the object tracking model210. When the object223-1and the object223-2are the same object, the ID allocation unit315allocates the same identification information225(e.g., object ID) to the object223-1and the object223-2.

When the object223-1and the object223-2are determined to be identical to each other, the image acquisition unit316obtains the first data226and the second data227. The first data226is image data in the first image area including the object223-1, and the second data227is image data in the second image area including the object223-2. The first data226may be image data cut out from the frame image221-1according to the shape and position of the bounding box222-1. The second data227may be image data cut out from the frame image221-2according to the shape and position of the bounding box222-2.

The second training execution unit317inputs the first data226and the second data227to the contrastive learning model230to train the contrastive learning model230. As an example, the second training execution unit317inputs the first data226to the contrastive learning model230to obtain the first feature vector231(example of a first feature) output from the contrastive learning model230. Likewise, the second training execution unit317inputs the second data227to the contrastive learning model230to obtain the second feature vector232(example of a second feature) output from the contrastive learning model230. The second training execution unit317adjusts a parameter of the contrastive learning model230to increase the degree of matching between the first feature vector231(zi) and the second feature vector232(zj). As an example, a loss function Lφ=−sim(zi, zj) may be calculated, and the parameter φ may be updated such that a value of the loss function Lφis minimized.

The third training execution unit318trains the object detection model240that detects an object from an image based on the trained contrastive learning model230with a fixed parameter. The third training execution unit318uses the training image data250for training of the object detection model240. The training image data250may include an image and the label253.

The training image data250may be partially or entirely in common with the training data300, or may be data different from the training data300.

The training image data250is divided into a plurality of the divided regions251by the patch generation unit319, and a plurality of the divided images252(e.g., patches) is generated. The size of the divided region251may be determined in advance according to a standard size or the like of the object.

The third training execution unit318inputs each of the divided images252to the trained contrastive learning model230(seeFIG.7) with the fixed parameter, and obtains a feature vector (e.g., representation vector) of each of them. As an example, the third training execution unit318calculates, for each frame, the feature map233represented by the feature vector of each of the divided regions251.

The third training execution unit318inputs the feature map233and the label253to the object detection model240for training. As a result, a parameter is updated in the object detection model240based on the feature for each position in the feature map233and the label253such as the bounding box222, the class, and the like as ground truth.

Since it is possible to train the object detection model240using the feature map233robust (e.g., resistant) to a perturbation, the object detection model240robust to the perturbation may be achieved.

It becomes possible to increase the data volume of the feature map233generated using a variety of data reflecting the angle of the object223, the presence or absence of the occlusion224, the presence or absence of the motion blur, the difference in illumination, and the like. Therefore, by training the object detection model240using the feature map233in which the number of data is increased, it becomes possible to suppress performance deterioration of object detection caused by over-training of the object detection model240.

FIG.10is a block diagram illustrating an exemplary functional configuration in an inference phase by the information processing device1according to the embodiment.

The information processing device1includes the storage unit311, the patch generation unit319, and an inference unit321. The patch generation unit319and the inference unit321is an example of the control unit320.

The storage unit311may include the contrastive learning model230and the object detection model240. The contrastive learning model230and the object detection model240may have been trained and may have fixed parameters.

The storage unit311may store an input image260to be subject to object detection. The storage unit311may store an inference result270obtained by inference processing.

The patch generation unit319obtains the input image260, divides the input image260into a plurality of the divided regions251, and generates a plurality of the divided images252(patches). The divided region251and the divided image252are similar except for a difference regarding whether the target image is the training image data250or the input image260.

The inference unit321inputs each of the divided images252to the trained contrastive learning model230(FIG.7) with the fixed parameter, and obtains a feature vector (representation vector) of each of them. As an example, the inference unit321calculates the feature map233represented by the feature vector of each of the divided regions251using the contrastive learning model230.

The inference unit321inputs the calculated feature map233to the object detection model240, and estimates a label based on the object detection model240. As an example, in a similar manner to the case illustrated inFIG.6, the inference unit321estimates the bounding box222that is boundary position information of the object, a class of the object, the confidence level228, and the like.

Note that a new object tracking model may be configured by inputting the estimation result of the object detection model240to the tracking model214illustrated inFIG.4.

[B-3] Exemplary Hardware Configuration of Information Processing Device1According to Embodiment

FIG.11is a block diagram illustrating an exemplary hardware (HW) configuration of a computer that implements functions of the information processing device1according to the embodiment.

As illustrated inFIG.11, the information processing device1includes a central processing unit (CPU)11, a memory unit12, a display control unit13, a storage device14, an input interface (IF)15, an external recording medium processing unit16, and a communication IF17.

The memory unit12is an exemplary storage unit, and illustratively is a read only memory (ROM), a random access memory (RAM), or the like. Programs of a basic input/output system (BIOS) and the like may be written in the ROM of the memory unit12. A software program of the memory unit12may be appropriately read and executed by the CPU11. Furthermore, the RAM of the memory unit12may be used as a temporary recording memory or a working memory.

The display control unit13is coupled to a display device131, and controls the display device131. The display device131is a liquid crystal display, an organic light-emitting diode (OLED) display, a cathode ray tube (CRT), an electronic paper display, or the like, and displays various kinds of information for an operator or the like. The display device131may be combined with an input device, and may be, for example, a touch panel.

The storage device14is a storage device having high input/output (I/O) performance, and for example, a dynamic random access memory (DRAM), a solid state drive (SSD), a storage class memory (SCM), or a hard disk drive (HDD) may be used. The storage device14may store a network configuration table101.

The input IF15may be coupled to an input device such as a mouse151and a keyboard152, and may control the input device such as the mouse151and the keyboard152. The mouse151and the keyboard152are exemplary input devices, and the operator performs various input operations through those input devices.

The external recording medium processing unit16is configured in such a manner that a recording medium160may be attached thereto. The external recording medium processing unit16is configured in such a manner that information recorded in the recording medium160may be read in a state where the recording medium160is attached thereto. In the present example, the recording medium160is portable. For example, the recording medium160is a flexible disk, an optical disk, a magnetic disk, a magneto-optical disk, a semiconductor memory, or the like.

The communication IF17is an interface that enables communication with an external device.

The CPU11is an example of a processor (e.g., computer), and is a processing device that performs various controls and calculations. The CPU11implements various functions by executing an operating system (OS) or a program loaded into the memory unit12. Note that the CPU11may be a multi-processor including a plurality of CPUs, or a multi-core processor having a plurality of CPU cores, or may have a configuration having a plurality of multi-core processors.

A device for controlling operation of the entire information processing device1is not limited to the CPU11, and may be, for example, any one of an MPU, a DSP, an ASIC, a PLD, or an FPGA. Furthermore, the device for controlling operation of the entire information processing device1may be a combination of two or more types of the CPU, MPU, DSP, ASIC, PLD, and FPGA. Note that the MPU is an abbreviation for a micro processing unit, the DSP is an abbreviation for a digital signal processor, and the ASIC is an abbreviation for an application specific integrated circuit. Furthermore, the PLD is an abbreviation for a programmable logic device, and the FPGA is an abbreviation for a field-programmable gate array.

[B-4] Exemplary Operation of Information Processing Device1According to Embodiment

[B-4-1] Training Phase

An example of operation in the training phase of the information processing device1according to the embodiment illustrated inFIG.11will be described based on a flowchart (operations S1to S6) illustrated inFIG.12.

The acquisition unit312obtains the existing training data300. The first training execution unit313trains the object tracking model210using the existing training data300(operation S1). As illustrated inFIG.4, the object tracking model210may include the object detection model212and the tracking model214.

The object detection unit314applies the trained object tracking model210to the unlabeled moving image data220(operation S2). The object detection unit314detects each of the objects223in the unlabeled moving image data220.

The ID allocation unit315determines the identity between the object223-1(e.g., first object) and the object223-2(e.g., second object) based on the inference result of the object tracking model210. When the object223-1and the object223-2are the same object, the ID allocation unit315allocates the same identification information225(e.g., object ID) to the object223-1and the object223-2. As a result of the object tracking, the image acquisition unit316cuts out a pair of images at different times having the same identification information225(e.g., object ID) (operation S3).

As an example, the paired images are the first data226and the second data227. The first data226is image data in the first image area including the object223-1, and the second data227is image data in the second image area including the object223-2.

The second training execution unit317inputs the first data226and the second data227, which are the paired images, to the contrastive learning model230to train the contrastive learning model230(operation S4).

The patch generation unit319obtains the training image data250as training data. The patch generation unit319divides the training image data250into a plurality of patches, and inputs them to the contrastive learning model230with a fixed parameter to obtain the feature map233(operation S5).

The third training execution unit318newly trains the object detection model240using the feature map233and the label253of the training image data250(operation S6). Then, the process in the training phase is terminated.

An example of operation in the inference phase of the information processing device1according to the embodiment illustrated inFIG.11will be described based on a flowchart (operations S11to S13) illustrated inFIG.13.

The control unit320receives at least one input image260(operation S11).

The patch generation unit319obtains the input image260. The patch generation unit319divides the input image260into a plurality of patches. The inference unit321inputs each of the divided patches to the trained contrastive learning model230(seeFIG.7) with the fixed parameter, and obtains a feature vector (e.g., representation vector) of each of them. As an example, the inference unit321obtains the feature map233represented by the feature vector of each of the divided regions using the contrastive learning model230(operation S12). The feature map233is similar to the feature map233except for the difference regarding whether the target image is the training image data250or the input image260.

The inference unit321inputs the calculated feature map233to the object detection model240to obtain the inference result270regarding the object detection (operation S13). Then, the process in the inference phase is terminated.

[C] First Variation

[C-1] Description of Training Process According to First Variation

FIG.14is a diagram illustrating an exemplary training process of an object detection model242by an information processing device1according to a first variation. The object detection model242is an exemplary second machine learning model that detects an object from an image based on a trained contrastive learning model230.

The process of the first variation is in common with the case of the embodiment regarding the processes illustrated inFIGS.3to7. Therefore, repetitive description will be omitted. The process of the first variation includes the process illustrated inFIG.14instead of the process of the embodiment illustrated inFIG.8.

As illustrated inFIG.14, in the first variation, a patch generation unit319of a control unit320obtains respective divided images252according to a first division resolution (e.g., high resolution), a second division resolution (e.g., medium resolution), and a third division resolution (e.g., low resolution).

As an example, the patch generation unit319divides the input training image data250into a plurality of first divided regions251aaccording to the first division resolution to obtain a plurality of first divided images252a. Furthermore, the patch generation unit319divides the input training image data250into a plurality of second divided regions251baccording to the second division resolution different from the first division resolution to obtain a plurality of second divided images252b. Moreover, the patch generation unit319divides the input training image data250into a plurality of third divided regions251caccording to the third division resolution different from the first and second division resolutions to obtain a plurality of third divided images252c.

A third training execution unit318inputs each of the first divided images252ato the trained contrastive learning model230(seeFIG.7) with a fixed parameter, and obtains a feature vector (e.g., representation vector) of each of them. As an example, the third training execution unit318calculates a first resolution feature map233arepresented by the feature vector of each of the first divided regions251a.

Likewise, the third training execution unit318inputs each of the second divided images252bto the contrastive learning model230to obtain a feature vector (e.g., representation vector) of each of them. As an example, the third training execution unit318calculates a second resolution feature map233brepresented by the feature vector of each of the second divided regions251b.

The third training execution unit318inputs each of the third divided images252cto the contrastive learning model230to obtain a feature vector (e.g., representation vector) of each of them. As an example, the third training execution unit318calculates a third resolution feature map233crepresented by the feature vector of each of the third divided regions251c.

The third training execution unit318trains the object detection model242based on the first resolution feature map233a, the second resolution feature map233b, the third resolution feature map233c, and the training image data250. As an example, the training image data250may be supervised training data including image data and a label253.

As an example, the training image data250(including the label253) may be input to an input layer of the object detection model242. Each of the outputs of the contrastive learning model230(first resolution feature map233a, second resolution feature map233b, third resolution feature map233c, etc.) may be coupled to an intermediate layer output of the object detection model242.

The object detection model242may be a deep neural network (DNN)-based object detection model. In this case, the intermediate layer outputs of the object detection model242may correspond to mutually different resolutions. As an example, in a case of a convolutional neural network (CNN)-based object detection model, object detection is carried out while gradually reducing the internal image resolution as getting closer to an output layer.

In a case where resolutions of the individual intermediate layer outputs of the object detection model242are known, the individual division resolutions such as the first division resolution, the second division resolution, the third division resolution, and the like may correspond to the resolutions of the individual intermediate layer outputs. In this case, the resolution feature map of the division resolution corresponding to the intermediate layer output resolution is coupled to the intermediate layer.

In a case where a plurality of types of objects223having different sizes is present, the object detection model242is trained by a plurality of types of feature maps (233a,233b, and233c) obtained by being divided into patches having sizes corresponding to the plurality of types of division resolutions. Therefore, it becomes possible to improve the accuracy in detecting the objects223having different sizes.

WhileFIG.14illustrates the case of the three-stage division resolution of the first to third division resolutions, the present embodiment is not limited to this case, and a division resolution of two or more stages may be sufficient. The stage of the division resolution may be set according to the number of convolution layers and pooling layers of the object detection model242.

Except for the points above, the training process of the object detection model242according to the first variation is similar to the case of the embodiment. Therefore, description regarding a software configuration and a hardware configuration of the information processing device1according to the first variation will be omitted.

[C-2] Exemplary Operation of Information Processing Device1According to First Variation

[C-2-1] Training Phase

An example of operation in a training phase of the information processing device1according to the first variation will be described based on a flowchart (operations S21to S28) illustrated inFIG.15.

Processes of operations S21to S24are similar to the processes of operations S1to S4inFIG.12, respectively. Therefore, description thereof will be omitted.

The patch generation unit319selects a patch size (operation S25). The patch size may be a side length of the divided region251. The patch size may be inversely proportional to the division resolution. The division resolution becomes lower as the patch size increases. A type of the patch size may be set in advance.

The patch generation unit319divides the training image data250into patches according to the selected patch size, inputs them to the contrastive learning model230with a fixed parameter, and obtains a feature map (e.g., first resolution feature map233a) (operation S26).

The patch generation unit319determines whether or not feature maps of all patch sizes have been obtained (operation S27). If there is a feature map of a patch size that has not been obtained yet (see NO route in operation S27), the patch generation unit319selects the next patch size (operation S25). If feature maps of all the patch sizes have been obtained (see YES route in operation S27), the process proceeds to operation S28.

The third training execution unit318inputs, to the object detection model242, the individual feature maps (233a,233b, and233c) and the label253and the image of the training image data250as training data, thereby training the object detection model242(operation S28). Then, the process in the training phase is terminated.

An example of operation in an inference phase of the information processing device1according to the first variation will be described based on a flowchart (operations S31to S35) illustrated inFIG.16.

The control unit320receives at least one input image260(operation S31).

The patch generation unit319selects a patch size (operation S32).

The patch generation unit319divides the input image260into patches according to the selected patch size, inputs them to the contrastive learning model230with the fixed parameter, and obtains a feature map (e.g., first resolution feature map233a) (operation S33).

The patch generation unit319determines whether or not feature maps of all patch sizes have been obtained (operation S34). If there is a feature map of a patch size that has not been obtained yet (see NO route in operation S34), the patch generation unit319selects the next patch size (operation S32). If feature maps of all the patch sizes have been obtained (see YES route in operation S34), the process proceeds to operation S35.

In operation S35, the inference unit321inputs the input image260and the individual feature maps (233a,233b, and233c) to the object detection model242to obtain an inference result270regarding the object detection.

[D] Second Variation

[D-1] Description of Training Process According to Second Variation

In the embodiment and the first variation, the case where the contrastive learning model230(encoder) is provided separately from the object detection models240and242has been described. However, the present embodiment is not limited to this case. In a machine learning method according to a second variation, a part of functions of an object detection model is used as a contrastive learning model (e.g., encoder).

FIG.17is a diagram illustrating a decoupled object detection model280according to the second variation. The decoupled object detection model280is an example of the object detection model, and is an example of a second machine learning model.

In the decoupled object detection model280(e.g., decoupled object detection head), a class classification feature extraction unit281and a bounding box feature extraction unit282are separated. The class classification feature extraction unit281extracts a feature for class classification of the object detection function. The bounding box feature extraction unit282extracts a feature for bounding box generation. The class classification feature extraction unit281is an exemplary class classification model for outputting a feature related to object class classification. The bounding box feature extraction unit282is an exemplary position information model that outputs a feature related to object boundary position information in a moving image.

When a feature map283is input to the decoupled object detection model280, it is divided into the class classification feature extraction unit281and the bounding box feature extraction unit282by an input unit284.

An output from the class classification feature extraction unit281is subject class classification by a class classification unit285. An output from the bounding box feature extraction unit282is input to a bounding box regression prediction unit286(regression). The bounding box regression prediction unit286calculates a position of the bounding box.

As an example, the decoupled object detection model280may be a YOLOX-based object detection model.

In the second variation, the class classification feature extraction unit281of the decoupled object detection model280is used as a contrastive learning model. The class classification feature extraction unit281is an example of the encoder.

FIG.18is a diagram illustrating an exemplary training process of the decoupled object detection model280by an information processing device1according to the second variation.

FIG.19is a block diagram illustrating an exemplary functional configuration in a training phase by the information processing device1according to the second variation.FIG.20is a block diagram illustrating an exemplary functional configuration in an inference phase by the information processing device1according to the second variation.

As illustrated inFIGS.19and20, as compared with the information processing device1according to the embodiment, a second training execution unit317may be omitted in the information processing device1according to the second variation. A third training execution unit318implements a function of the second training execution unit317. A patch generation unit319may be omitted in the information processing device1according to the second variation.

As illustrated inFIGS.18and19, an optimization unit322may be provided. As will be described later, the optimization unit322carries out machine learning of the class classification feature extraction unit281to increase a degree of matching between a value of a first element288a(e.g., first class classification feature) and a value of a second element288b(e.g., second class classification feature).

The information processing device1according to the second variation performs a process similar to the process illustrated inFIGS.3to6according to the embodiment. The information processing device1obtains a first frame image (e.g., frame image221-1) and a second frame image (e.g., frame image221-2) including objects223-1(e.g., first object) and223-2(e.g., second object), respectively, which are the mutually same object.

When the object223-1(e.g., first object) and the object223-2(e.g., second object) are determined to be identical to each other, an image acquisition unit316obtains first data226and second data227.

In the embodiment and the first variation, the case where the first data226is image data cut out from the frame image221-1according to the shape and position of the bounding box222-1has been mainly described. Likewise, the case where the second data227is image data cut out from the frame image221-2has been described.

In the second variation, the first data226may be the entire frame image221-1. The second data227may be the entire frame image221-2.

In the second variation, the third training execution unit318(also serving as the second training execution unit317) inputs each of the first data226and the second data227to the class classification feature extraction unit281that also functions as a contrastive learning model.

In a case where the objects (223-1and223-2), which are the same object, have been detected at different times (t−1 and t) as a result of object tracking, the third training execution unit318obtains the first element288a(at t−1) and the second element288b(at t) in the feature map corresponding to the positions of the objects (223-1and223-2) at the respective times. The third training execution unit318obtains the value of the first element288aand the value of the second element288b.

The value of the first element288ais an example of a first class classification feature289aobtained by inputting the first data226to the class classification model. The value of the second element288bis an example of a second class classification feature289bobtained by inputting the second data227to the class classification model.

There may be a plurality of pairs of the objects223. The number of pairs may be determined in advance.

The optimization unit322carries out machine learning of the class classification feature extraction unit281to increase a degree of matching between the first class classification feature289a(zi) and the second class classification feature289b(zj). The optimization unit322may calculate a loss function Lφ=−sim(zi, zj), and may update the parameter φ such that a value of the loss function Lφis minimized.

Since the extracted first class classification feature289a(zi) and the second class classification feature289b(zj) are features for the same object, a concept similar to that of contrastive learning may be applied.

The machine learning of the class classification by the class classification feature extraction unit281and the like, the machine learning of the boundary position information by the bounding box feature extraction unit282and the like, and the contrastive learning may be carried out in parallel. In this case, labeled moving image data may be input instead of unlabeled moving image data220.

The training may be carried out even in the case of using a part of the functions of the object detection model (decoupled object detection model280) as a contrastive learning model (encoder) as in the second variation, and the training suitable for the object detection may be carried out as compared with the case of training the contrastive learning model separately.

[D-2] Exemplary Operation of Information Processing Device1According to Second Variation

[D-2-1] Training Phase

An example of operation in the training phase of the information processing device1according to the second variation will be described based on a flowchart (operations S41to S48) illustrated inFIG.21.

Processes of operations S41and S42are similar to the processes of operations S1and S2inFIG.12, respectively. Therefore, description thereof will be omitted.

The ID allocation unit315determines the identity between the object223-1(e.g., first object) and the object223-2(e.g., second object) based on the inference result of the object tracking model210. When the object223-1and the object223-2are the same object, the ID allocation unit315allocates the same identification information225(e.g., object ID) to the object223-1and the object223-2. As a result of the object tracking, the image acquisition unit316finds a pair of the objects223-1and223-2at different times having the same identification information225(e.g., object ID) (operation S43).

The image acquisition unit316selects images including the paired objects223-1and223-2(operation S44). The paired images may be the entire frame image221-1and the entire frame image221-2.

The third training execution unit318inputs each of the selected paired images (e.g., first data226and second data227) to the object detection model (operation S45). As an example, the entire frame image221-1and the entire frame image221-2are input to the decoupled object detection model280.

The third training execution unit318identifies the first element288aand the second element288bin the feature map corresponding to the positions of the objects223-1and223-2at the respective times (operation S46).

The third training execution unit318determines whether or not the first element288aand the second element288bhave been identified for all the paired objects223determined to be identical (operation S47). If the first element288aand the second element288bhave not been identified for all the paired objects223determined to be identical (see NO route in operation S47), the third training execution unit318selects an image including other paired objects (operation S44). If the first element288aand the second element288bhave been identified for all the paired objects223determined to be identical (see YES route in operation S47), the process proceeds to operation S48.

The optimization unit322carries out machine learning of the class classification feature extraction unit281by contrastive learning to increase the degree of matching between the value of the first element288aand the value of the second element288bidentified for all the pairs (operation S48). The optimization unit322carries out machine learning of the class classification feature extraction unit281to increase a degree of matching between the first class classification feature289a(zi) and the second class classification feature289b(zj). The optimization unit322may calculate the loss function Lφ=−sim(zi, zj) for each pair, and may update the parameter φ such that the sum of the loss functions Lφfor all the pairs is minimized. Then, the process in the training phase is terminated.

An example of operation in the inference phase of the information processing device1according to the second variation will be described based on a flowchart (operations S51and S52) illustrated inFIG.22.

A control unit320receives at least one input image260(operation S51).

The input image260is input to the decoupled object detection model280, which is a trained object detection model with a fixed parameter, to obtain an inference result270(operation S52). Then, the process in the inference phase is terminated.

According to the exemplary embodiment described above, for example, the following effects may be exerted.

The control unit320inputs, to the object tracking model210trained by using the training data300, the unlabeled moving image data220including at least the first frame image (frame image221-1, etc.) and the second frame image (frame image221-2, etc.). The control unit320detects the object223-1(e.g., first object) and the object223-2(e.g., second object) from the first frame image (frame image221-1, etc.) and the second frame image (frame image221-2, etc.), respectively, based on the inference result of the object tracking model210. The control unit320determines identity between the object223-1and the object223-2having been detected. The control unit320inputs, to the contrastive learning model230, the first data226in the first image area including the object223-1and the second data227in the second image area including the object223-2determined to be identical, and trains the contrastive learning model230.

As a result, it becomes possible to reduce an influence of a perturbation caused by the occlusion224or the like to improve performance of object detection. Since the volume of the training data may be increased by the contrastive learning model230, it becomes possible to suppress deterioration in object detection performance caused by over-training or the like.

It is possible to obtain paired images for contrastive learning in consideration of differences and the like such as an angle of the object223, presence or absence of the occlusion224, presence or absence of a motion blur, a difference in illumination, and the like without passing through special data extension processing. A wide variety of training data may be obtained as compared with a method of increasing labels in a pseudo manner based on label propagation. Therefore, it becomes possible to achieve object detection robust (e.g., resistant) to the differences such as the angle of the object223, the presence or absence of the occlusion224, the presence or absence of the motion blur, the difference in illumination, and the like.

In the process of training the contrastive learning model230, the control unit320carries out the machine learning to increase the degree of matching between the first feature obtained by inputting the first data226to the contrastive learning model230and the second feature obtained by inputting the second data227to the contrastive learning model230.

As a result, the contrastive learning model230robust to the perturbation caused by the occlusion224or the like may be obtained.

The control unit320trains the object detection model240that detects an object from an image based on the trained contrastive learning model230.

It becomes possible to train the object detection model240utilizing the contrastive learning model230robust to the perturbation caused by the occlusion224or the like, which enables more robust object detection. Since volume of a variety of training data may be increased by the contrastive learning model230, over-training is suppressed.

In the process of training the object detection model240, the control unit320divides the input training image data250into a plurality of divided regions251to obtain a plurality of divided images252. The control unit320inputs the divided images252of the individual divided regions251to the contrastive learning model230, calculates a feature in each of the divided regions251, and trains the object detection model240based on the calculated result and the label253corresponding to the input training image data250.

As a result, it becomes possible to reflect the training result of the contrastive learning model230in the object detection model240without complicating the configuration.

In the process of training the object detection model242, the control unit320divides the input training image data250into a plurality of first divided regions251aaccording to the first division resolution to obtain a plurality of first divided images252a. The control unit320divides the input training image data250into a plurality of second divided regions251baccording to the second division resolution different from the first division resolution to obtain a plurality of second divided images252b. The control unit320inputs the first divided image252ain each of the first divided regions251ato the contrastive learning model230to obtain the first resolution feature map233aindicating the feature of each of the first divided regions251a. The control unit320inputs the second divided image252bin each of the second divided regions251bto the contrastive learning model230to obtain the second resolution feature map233bindicating the feature of each of the second divided regions251b. The control unit320trains the object detection model242based on the first resolution feature map233a, the second resolution feature map233b, and the training image data250.

As a result, it becomes possible to effectively improve the object detection performance even when the object to be detected appears in various scales in the image.

As the object detection model, the decoupled object detection model280is used including the position information model that outputs a feature related to boundary position information of an object in a moving image and the class classification feature extraction unit281(e.g., class classification model) for outputting a feature related to class classification of the object. In the decoupled object detection model280, the class classification feature extraction unit281is used as a contrastive learning model (e.g., encoder). The control unit320carries out the machine learning to increase the degree of matching between the first class classification feature289aobtained by inputting the first data226to the class classification feature extraction unit281and the second class classification feature289bobtained by inputting the second data227to the class classification feature extraction unit281.

As a result, it becomes possible to utilize a part of the object detection model (e.g., decoupled object detection model280) as the contrastive learning model, and to achieve processing with high compatibility between object detection and contrastive learning.

The disclosed technique is not limited to the embodiment described above, and various modifications may be made without departing from the gist of the present embodiment. Each configuration and each process of the present embodiment may be selected or omitted as needed, or may be combined as appropriate.