INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND NON-TRANSITORY COMPUTER-READABLE STORAGE MEDIUM

An information processing apparatus comprises a processing unit configured to execute, in frame periods corresponding to respective frames, an inference relating to the frame and training relating to the frame. When an inference relating to a first frame has been completed but training relating to the first frame has not been completed in a first frame period corresponding to the first frame, the processing unit executes training relating to the first frame and an inference and training relating to a second frame in a second frame period corresponding to the second frame subsequent to the first frame.

BACKGROUND

Field of the Disclosure

The present disclosure relates to techniques relating to inferencing and learning/training.

Description of the Related Art

Machine learning is one effective method used in the task for tracking an object in an image across a plurality of frames. Typically, in object tracking using machine learning, an identification model, such as a neural network or support vector machine, is used to detect the tracking target in each frame. This step is called inferencing, and the step in which the parameters of the identification model are updated on the basis of the inference result is called training.

Online training is often used in object tracking using machine learning. In “Discriminative and Robust Online Learning for Siamese Visual Tracking”, J. Zhou et al., Vol 34 No 07: AAAI-20 Technical Tracks 7 (2020), a method for object tracking using a neural network is described. The method described in “Discriminative and Robust Online Learning for Siamese Visual Tracking”, J. Zhou et al., Vol 34 No 07: AAAI-20 Technical Tracks 7 (2020) includes one or more of the parameters of the identification model being updated using the result of the inference for each frame. Accordingly, a single type of identification model can be used to track an object with various features.

However, when object tracking is actually performed, the processing is sometimes interrupted during a frame period. In Japanese Patent Laid-Open No. 2017-139725, a multi-camera system using online training to track an object is described. In the camera system described in Japanese Patent Laid-Open No. 2017-139725, when a reset is performed via a user operation, the system returns to an initial state. In Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2018-509712, a method is described for performing online training for object recognition using machine learning. In the method described in Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2018-509712, the online training is interrupted when input data corresponding to an outlier is received, and after restoration, the processing is restarted using the latest pre-interruption training result.

Desirably, the target tracking state from the interruption to after the restoration is retained via an application. For example, a surveillance camera may be installed with a function for recording still images when a specific movement of a tracking target is recognized. In such a case, the still image recording processing takes priority over the tracking processing, and thus the tracking processing is interrupted. This interrupts the tracking processing during a frame period, but desirably, the target tracking state is retained for the next frame onward.

To retain the target tracking state, it is important to retain the identification model parameter in the latest state. As the tracking target in an image is constantly changing orientation and size, the features slightly change in each frame. Due to a change in the feature, detection of the tracking target via inference for each frame may fail, causing the tracking state to be canceled. To prevent this, training needs to be performed every frame and inference needs to be performed using the learned parameter to which the latest feature has been applied.

However, with the camera system described in Japanese Patent Laid-Open No. 2017-139725, when the system returns to an initial state, online training needs to be redone in order to track the same object as before the interruption. With the method described in Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2018-509712, tracking can be restarted using the pre-interruption learned parameter, but the result of the online training in the frame period when the interruption occurred cannot be applied.

SUMMARY

The present disclosure provides techniques for completing training relating to a frame before the time when an inference relating to a frame is executed.

According to the first aspect of the present disclosure, there is provided an information processing apparatus comprising: a processing unit configured to execute, in frame periods corresponding to respective frames, an inference relating to the frame and training relating to the frame, wherein when an inference relating to a first frame has been completed but training relating to the first frame has not been completed in a first frame period corresponding to the first frame, the processing unit executes training relating to the first frame and an inference and training relating to a second frame in a second frame period corresponding to the second frame subsequent to the first frame.

According to the second aspect of the present disclosure, there is provided an information processing method executed by an information processing apparatus, comprising: executing, in frame periods corresponding to respective frames, an inference relating to the frame and training relating to the frame; and when, in the executing, an inference relating to a first frame has been completed but training relating to the first frame has not been completed in a first frame period corresponding to the first frame, training relating to the first frame and an inference and training relating to a second frame in a second frame period corresponding to the second frame subsequent to the first frame are executed.

According to the third aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing a computer program for causing a computer to function as a processing unit configured to execute, in frame periods corresponding to respective frames, an inference relating to the frame and training relating to the frame, wherein when an inference relating to a first frame has been completed but training relating to the first frame has not been completed in a first frame period corresponding to the first frame, the processing unit executes training relating to the first frame and an inference and training relating to a second frame in a second frame period corresponding to the second frame subsequent to the first frame.

DESCRIPTION OF THE EMBODIMENTS

First Embodiment

In the present embodiment, with a configuration in which, in a frame period corresponding to each frame, an inference relating to the frame and training relating to the frame are executed, when an inference relating to a second frame has been completed in a second frame period corresponding to the second frame previous to a first frame but training relating to the second frame has not been completed in the second frame period, training relating to the second frame and an inference relating to the first frame are executed in a first frame period corresponding to the first frame. In this manner, training can be completed before an inference that requires the result of the next training.

In the present embodiment described below, tracking processing that uses an identification model for identifying a tracking target in a frame to identify a tracking target in a frame and track a tracking target is used as an example of inference processing. In the present embodiment, a convolutional neural network, which is an example of a hierarchical neural network, is used as the identification model, but other types of models may be used.

A configuration example of the identification model according to the present embodiment and the data flow of the identification model will be described below using the block diagram inFIG.1. As illustrated inFIG.1, the identification model includes a first convolutional layer101and a second convolutional layer102.

The first convolutional layer101is set with a first coefficient104(weight coefficient for the first convolutional layer101) as a learned parameter. Also, the second convolutional layer102is set with a second coefficient106(weight coefficient for the second convolutional layer102) as a learned parameter. The first convolutional layer101set with the first coefficient104processes an input frame103and outputs to an intermediate output105. The second convolutional layer102set with the second coefficient106processes the intermediate output105and outputs a tracking result107indicating tracking target information such as the position and size of the tracking target included in the frame103.

The identification model runs the same for each frame that is input and executes processing (inference processing) to output a tracking result corresponding to the frame. Each frame that is input may be an image of each frame in a captured video or may be the still images captured periodically or non-periodically.

Also, in the present embodiment, the second convolutional layer102of the identification model corresponds to online training. In other words, the second coefficient106is trained (updated) each frame so that the second convolutional layer102can output a tracking result of more accurate tracking target information.

In the online training, the second coefficient106is trained (updated) using the intermediate output105input to the second convolutional layer102and the tracking result107output from the second convolutional layer102. For example, as correct data, a map is prepared in advance with a label value of “1” assigned to a tracking target region in the frame103to be input to the first convolutional layer101and a label value of “0” assigned to a region other than the tracking target region in the frame103. Also, the frame103is input into the first convolutional layer101, and the intermediate output105to be output from the first convolutional layer101and the tracking result107to be output from the second convolutional layer102are obtained. Then, the difference (error) between the tracking result107and the correct data is obtained, and the intermediate output105is used to update the second coefficient106via a known method such as backpropagation or the like. Note that this online training method is a known technique, and thus will not be described further. Also, the online training method is not limited to a specific method.

As described above, updating the second coefficient106via the online training is required to retain the tracking state of the tracking target. The feature of a moving tracking target in a frame changes in each frame. To execute inference processing for each frame, that is, the processing of the identification model illustrated inFIG.1, to continually detect the tracking target, the online training needs to be performed each frame and the second coefficient106corresponding to the changes in the feature needs to be continually obtained.

A timechart of the time allocations for executing inference processing and training processing for each frame is illustrated inFIG.2. In the first frame period, which is a processing period corresponding to the first frame, first, inference processing201using the identification model is executed for the first frame. Thereafter, training processing202(online training of the second convolutional layer102) of the identification model is executed using the result of the inference processing201to update the second coefficient106. Thereafter, in the second frame period, which is a processing period corresponding to the second frame, first, inference processing using the identification model is executed for the second frame. Thereafter, training processing (online training of the second convolutional layer102) of the identification model is executed using the result of the inference processing to update the second coefficient106. Thereafter, in a third frame period, which is a processing period corresponding to the third frame, first, inference processing using the identification model is executed for the third frame. Thereafter, training processing (online training of the second convolutional layer102) of the identification model is executed using the result of the inference processing to update the second coefficient106. In this manner, by executing the inference processing and the training processing each frame period, tracking processing to tracking a tracking target is executed while updating the second coefficient106.

A timechart according to the present embodiment is illustrated inFIG.3. In the case illustrated inFIG.3, training processing302in the first frame period is interrupted by an interrupt instruction from the user. Note that the interrupt instruction is not limited to an instruction input by a user and may be, for example, an instruction issued by a control unit such as the processor of an apparatus when a condition is satisfied or an instruction received from an external apparatus.

In this case, first, the training processing302that normally should have been executed in the first frame period is executed in the second frame period. Then, when the training processing302is completed, inference processing303using the identification model for the second frame is executed using the remaining time in the second frame period. Thereafter, training processing304of the identification model is executed using the result of the inference processing303to update the second coefficient106. As inFIG.2, in the third frame period, first, inference processing305using the identification model is executed for the third frame. Thereafter, training processing306of the identification model is executed using the result of the inference processing305to update the second coefficient106.

In this manner, since the training processing302is completed before the inference processing303for the second frame period is executed, the second coefficient106of the identification model using the inference processing303can be updated before the start of the inference processing303. In other words, when the training processing302in the first frame period is interrupted, the result of the training processing302in the first frame can be applied to the inference processing303in the second frame.

Next, an example of the hardware configuration of an information processing apparatus according to the present embodiment will be described using the block diagram inFIG.5. A personal computer (PC), a smartphone, a tablet terminal apparatus, or a similar computer apparatus can be applied as the information processing apparatus according to the present embodiment.

A CPU506executes various processing using computer programs and data stored in RAM509and ROM508. Accordingly, the CPU506performs operation control of the entire information processing apparatus and executes or controls the various types of processing described herein as processing executed by the information processing apparatus. Note that the CPU506is not limited to numbering one, and a plurality may be provided.

The RAM509includes an area for storing the computer programs and data loaded from the ROM508or a data storage unit502and an area for storing the computer programs and data received from an external apparatus via a communication unit503. Also, the RAM509includes a working area used when the CPU506, an arithmetic unit507, and an image processing unit505execute the various types of processing. The RAM509of such a configuration can provide various areas as appropriate.

Setting data of the information processing apparatus, the computer programs and data associated with activating the information processing apparatus, the computer programs and data associated with the basic operations of the information processing apparatus, and the like are stored in the ROM508. The computer programs and data stored in the ROM508are loaded on the RAM509according to control by the CPU506as appropriate and are the targets of processing by the CPU506.

An input unit501is a user interface such as a keyboard, a mouse, a touch panel, or the like. In response to a user operation, the input unit501can input various types of instruction to the CPU506, the arithmetic unit507, and the image processing unit505.

The data storage unit502is a large-capacity information storage apparatus such as a hard disk drive apparatus or the like. The data storage unit502includes an operating system (OS), the computer programs and data for causing the CPU506, the arithmetic unit507, and the image processing unit505to execute or control the various types of processing described herein as processing executed by the information processing apparatus, and the like.

The computer programs stored in the data storage unit502include computer programs for causing the CPU506and the arithmetic unit507to execute or control the operations of the identification model described above (the first convolutional layer101and the second convolutional layer102). Also, the data stored in the data storage unit502includes the parameters (the first coefficient104and the second coefficient106) of the identification model described above.

The computer programs and data stored in the data storage unit502are loaded on the RAM509according to control by the CPU506as appropriate and are the targets of processing by the CPU506, the arithmetic unit507, and the image processing unit505.

Note that the data storage unit502may be an apparatus for reading and writing the computer programs and data to/from a memory apparatus. Examples of the memory apparatus include a flexible disk, a CD-ROM, a CD-R, a DVD, a memory card, a CF card, SmartMedia, an SD card, a memory stick, an xD-picture card, a USB memory, and the like. In this case, the computer programs and data described above as being stored in the data storage unit502are stored in the memory apparatus. In this case, the data storage unit502loads the computer programs and data stored in the memory apparatus onto the RAM509via control by the CPU506.

Also, a portion of the RAM509may be used as the data storage unit502. Note that in addition to or instead of the data storage unit502, a storage apparatus of a device connected to via the communication unit503may be used.

The communication unit503is a communication OF for data communications with an external apparatus. Note that inFIG.5, the information processing apparatus includes the input unit501, the data storage unit502, and a display unit504, but these may be implemented as devices connected to the information processing apparatus via the communication unit503.

The display unit504includes a liquid crystal screen or a touch panel screen and displays the results of the processing of the CPU506, the arithmetic unit507, and the image processing unit505using images, characters, and the like. Note that the display unit504may be a projection apparatus such as a projector that projects images and characters.

Note that as in a known touch screen apparatus, the display unit504and the input unit501may be integrally formed. In this case, input for the input unit501corresponds to input via the touch screen.

When the image processing unit505receives a command from the CPU506, image processing, such as range adjustment of the pixel values of the image stored in the data storage unit502, is executed according to the command, and the image obtained via image processing is stored in the RAM509.

When the arithmetic unit507receives a command from the CPU506, the arithmetic unit507executes the arithmetic processing required for the inference processing and the training processing described above. The arithmetic unit507includes internal memory as required to execute the arithmetic processing. The various types of data illustrated inFIG.1, for example, the frame103, the first coefficient104, the second coefficient106, and the tracking result107, are all stored in the internal memory of the arithmetic unit507or the RAM509, and the arithmetic unit507reads and writes this data in the process of executing the arithmetic processing.

The input unit501, the data storage unit502, the communication unit503, the display unit504, the image processing unit505, the CPU506, the arithmetic unit507, the ROM508, and the RAM509are all connected to a system bus510.

Next, processing executed by the information processing apparatus to execute the inference processing and the training processing described above in the current frame period will be described with reference to the flowchart inFIG.4. Note that the information processing apparatus can execute the inference processing and the training processing for each frame period (each frame) by executing the processing according to the flowchart inFIG.4for each frame period (each frame).

In step S401, the CPU506determines whether or not first information indicating that the inference processing in the frame period (previous frame period) preceding the current frame period has been completed is stored in the RAM509. Also, the CPU506determines whether or not second information indicating that the training processing in the previous frame period has been completed is stored in the RAM509.

When the determination result is that the first information is stored in the RAM509and the second information is not stored in the RAM509, the CPU506determines that “in the previous frame period, the inference processing has been completed but the training processing has not been completed”, and the processing proceeds to step S402.

When the determination result is that the first information and the second information are both stored in the RAM509, the CPU506determines that “in the previous frame period, the inference processing and the training processing have been completed”, and the processing proceeds to step S404.

Also, when the determination result is that neither the first information nor the second information is stored in the RAM509, the processing proceeds to step S404. Note that when neither the first information nor the second information is stored in the RAM509and the inference processing that should have been executed in the previous frame period is restarted in the next frame period, the processing that should have been executed in the previous frame period may be discarded if it is not desirable that the delays in frame processing keep building up.

Note that when the current frame period is the initial frame period, the processing proceeds to step S404without the CPU506determining whether or not the first information and the second information are stored in the RAM509.

In step S402, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The arithmetic unit507executes the training processing for the frame (previous frame) corresponding to the previous frame period according to the operation command (operation instruction). Then, the arithmetic unit507notifies the CPU506of the completion of processing.

In step S403, the arithmetic unit507determines whether or not the training processing has been completed for the previous frame. Also, in step S403, the CPU506determines whether or not an interrupt instruction has been issued. An interrupt instruction may be issued by the CPU506in response to the user operating the input unit501or may be issued in response to the CPU506detecting that a condition has been satisfied.

When the determination result is that the training processing for the previous frame has been completed, the processing proceeds to step S404. Also, when the training processing for the previous frame has not been completed, the processing proceeds to step S402and the training processing continues. Also, when the CPU506determines that an interrupt instruction has been issued, the processing for the current frame according to the flowchart inFIG.4is ended.

In step S404, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The arithmetic unit507executes the inference processing for the frame (current frame) corresponding to the current frame period according to the operation command (operation instruction). Then, the arithmetic unit507stores the tracking result107in the RAM509and notifies the CPU506of the completion of processing.

In step S405, the arithmetic unit507determines whether or not the inference processing for the current frame has been completed (whether or not the tracking result107of the identification model with the current frame input as the frame103has been obtained). Also, in step S405, the CPU506determines whether or not an interrupt instruction has been issued.

When the determination result is that the inference processing for the current frame has been completed (that the tracking result107of the identification model with the current frame input as the frame103has been obtained), the processing proceeds to step S406.

Also, when the inference processing for the current frame has not been completed (that the tracking result107of the identification model with the current frame input as the frame103has not been obtained), the processing proceeds to step S404and the inference processing continues.

Also, when the CPU506determines that an interrupt instruction has been issued, the processing for the current frame according to the flowchart inFIG.4is ended.

In step S406, the CPU506stores (records) the first information in the RAM509. In this manner, after the processing of step S406, when the determination processing of step S401has been executed for the frame period subsequent to the current frame period (a frame period corresponding to the subsequent frame subsequent to the current frame), it is determined that the first information indicating that the inference processing has been completed in the previous frame period is stored in the RAM509.

In step S407, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The arithmetic unit507executes the training processing for the frame (current frame) corresponding to the current frame period according to the operation command (operation instruction). Then, the arithmetic unit507notifies the CPU506of the completion of processing.

In step S408, the arithmetic unit507determines whether or not the training processing has been completed for the current frame. Also, in step S408, the CPU506determines whether or not an interrupt instruction has been issued.

When the determination result is that the training processing for the current frame has been completed, the processing proceeds to step S409. Also, when the training processing for the current frame has not been completed, the processing proceeds to step S407and the training processing continues.

Also, when the CPU506determines that an interrupt instruction has been issued, the processing for the current frame according to the flowchart inFIG.4is ended.

In step S409, the CPU506stores (records) the second information in the RAM509. In this manner, after the processing of step S409, when the determination processing of step S401has been executed for the frame period subsequent to the current frame period (a frame period corresponding to the subsequent frame subsequent to the current frame), it is determined that the second information indicating that the training processing has been completed in the previous frame period is stored in the RAM509.

The processing according to the flowchart inFIG.4will now be described using the example illustrated inFIG.3. First, the processing according to the flowchart inFIG.4is executed for the first frame period.

In this example, the first frame period is the initial frame period, and thus the processing proceeds from step S401to step S404. In step S404, the arithmetic unit507executes inference processing301for the frame corresponding to the first frame period. Then, when the inference processing301is complete, the processing proceeds to step S406via step S405. In step S406, the CPU506stores the first information in the RAM509. Then, in step S407, the arithmetic unit507executes the training processing302for the frame corresponding to the first frame period, except that as an interrupt instruction has been issued during the training processing302, the processing according to the flowchart inFIG.4in the first frame period is ended.

Next, the processing according to the flowchart inFIG.4in the second frame period is executed and the first information is stored in the RAM509and the second information is not stored in the RAM509. Thus, the processing proceeds from step S401to step S402.

In step S402, the arithmetic unit507executes the training processing302for the frame corresponding to the first frame period. Then, when the training processing302is complete, the processing proceeds to step S404via step S403. In step S404, the arithmetic unit507executes the inference processing303for the frame corresponding to the second frame period. Then, when the inference processing303is complete, the processing proceeds to step S406via step S405. In step S406, the CPU506stores the first information in the RAM509. Thereafter, in step S407, the arithmetic unit507executes the training processing304for the frame corresponding to the second frame period. Then, when the training processing304is complete, the processing proceeds to step S409via step S408. In step S409, the CPU506stores the second information in the RAM509.

Next, the processing according to the flowchart inFIG.4in the third frame period is executed and the first information and the second information are stored in the RAM509. Thus, the processing proceeds from step S401to step S404.

In step S404, the arithmetic unit507executes the inference processing305for the frame corresponding to the third frame period. Then, when the inference processing305is complete, the processing proceeds to step S406via step S405. In step S406, the CPU506stores the first information in the RAM509. Thereafter, in step S407, the arithmetic unit507executes the training processing306for the frame corresponding to the third frame period. Then, when the training processing306is complete, the processing proceeds to step S409via step S408. In step S409, the CPU506stores the second information in the RAM509.

In this manner, according to the present embodiment, even when processing is interrupted by an interrupt instruction in the previous frame period, the training processing that should have been executed in the previous frame period is completed before the subsequent inference processing. Thus, the second coefficient106can be appropriately updated in the current frame period. This can prevent tracking failures caused by the second coefficient106not being updated.

Modified Example of First Embodiment

In this modified example of the first embodiment, the training processing for the previous frame and the inference processing for the current frame are executed in parallel. By using such a configuration, in addition to the effects according to the first embodiment, an increase in the processing time for the current frame period can be prevented. The differences with the first embodiment will be described below, and unless particularly mentioned, the other components are the same as in the first embodiment.

FIG.6is a block diagram illustrating a configuration example of the arithmetic unit507according to the present modified example. When an operation command (operation instruction) is received from the CPU506, a control unit601performs operational control of an inference arithmetic unit601and a training arithmetic unit602in response to the operation command (operation instruction).

The inference arithmetic unit601is an arithmetic unit that executes the inference processing described above, the training arithmetic unit602is an arithmetic unit that executes the training processing described above, and each can operate independent of one another. The inference arithmetic unit601and the training arithmetic unit602execute processing using data stored in a storage unit604and store the processing result in the storage unit604.

Note that the functional units illustrated inFIG.6may be substituted for other functional units illustrated inFIG.5. For example, a control unit603may be substituted for the CPU506, the storage unit604may be substituted for the RAM509, and the inference arithmetic unit601and the training arithmetic unit602may be substituted for the CPU506.

A timechart according to the present modified example is illustrated inFIG.7. In the case illustrated inFIG.7, training processing703in the first frame period is interrupted by an interrupt instruction.

Inference processing701executed in the first frame period corresponds to the processing (processing for obtaining the intermediate output105from the frame103) by the first convolutional layer101from among the processing included in the inference processing301described above. Inference processing702executed in the first frame period corresponds to the processing (processing for obtaining the tracking result107from the intermediate output105) by the second convolutional layer102from among the processing included in the inference processing301described above.

Inference processing704executed in the second frame period corresponds to the processing by the first convolutional layer101from among the processing included in the inference processing303described above. Inference processing705executed in the second frame period corresponds to the processing by the second convolutional layer102from among the processing included in the inference processing303described above.

Inference processing707executed in the third frame period corresponds to the processing by the first convolutional layer101from among the processing included in the inference processing305described above. Inference processing708executed in the third frame period corresponds to the processing by the second convolutional layer102from among the processing included in the inference processing305described above.

When the inference processing701and the inference processing702in the first frame period are complete, the training processing703is started, but an interrupt instruction is issued during the training processing703, causing the training processing703to be interrupted.

In the second frame period, first, the interrupted training processing703starts. Here, the inference processing704is a processing using the first convolutional layer101, and thus the first coefficient104is used but the second coefficient106is not used. In other words, the inference processing704is not affected by the training processing703and may thus be executed in parallel with the training processing703. InFIG.7, in the second frame period, the interrupted training processing703is started together with the inference processing704, with the training processing703and the inference processing704being executed in parallel. Also, the inference processing705is processing executed using the second convolutional layer102set with the second coefficient106updated by the training processing703, and thus is started after the training processing703is complete. In the present modified example, the inference arithmetic unit601and the training arithmetic unit602operate independently of one another, and thus the training processing703and the inference processing704can be executed in parallel. In the timechart illustrated inFIG.7, the amount of time required for all of the processing of the second frame period is unchanged from the first embodiment even when the training processing703is executed again in the second frame. In other words, compared to the first embodiment, the processing time of the second frame period can be decreased by an amount corresponding to the training processing703.

When the inference processing705is complete, a training processing706corresponding to the second frame period is started. Then, in the third frame period, the inference processing707and the inference processing708are executed, and when the inference processing708is complete, training processing709is executed.

Next, processing executed by the information processing apparatus to execute the inference processing and the training processing described above in the current frame period will be described with reference to the flowcharts inFIGS.8A and8B. Note that the information processing apparatus can execute the inference processing and the training processing for each frame period (each frame) by executing the processing according to the flowcharts inFIGS.8A and8Bfor each frame period (each frame).

Hereinafter, in the inference processing, processing executed using the first convolutional layer101is referred to as first inference processing, and processing executed using the second convolutional layer102is referred to as second inference processing.

In step S801, the CPU506determines whether or not fourth information indicating that the second inference processing in the frame period (previous frame period) preceding the current frame period has been completed is stored in the RAM509. Also, the CPU506determines whether or not the second information indicating that the training processing in the previous frame period has been completed is stored in the RAM509.

When the determination result is that the fourth information is stored in the RAM509and the second information is not stored in the RAM509, the CPU506determines that “in the previous frame period, the first inference processing and the second inference processing have been completed but the training processing has not been completed”. Then, the processing proceeds to step S802.

When the determination result is that the fourth information and the second information are both stored in the RAM509, the CPU506determines that “in the previous frame period, the first inference processing, the second inference processing, and the training processing have been completed”, and the processing proceeds to step S803.

Also, when the determination result is that neither the fourth information nor the second information is stored in the RAM509, the processing proceeds to step S803. Note that when neither the fourth information nor the second information is stored in the RAM509and the inference processing that should have been executed in the previous frame period is restarted in the next frame period, the processing that should have been executed in the previous frame period may be discarded if it is not desirable that the delays in frame processing keep building up.

Note that when the current frame period is the initial frame period, the processing proceeds to step S803without the CPU506determining whether or not the fourth information and the second information are stored in the RAM509.

In step S802, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The training arithmetic unit602in the arithmetic unit507executes the training processing for the frame (previous frame) corresponding to the previous frame period according to the operation command (operation instruction).

In step S803, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The inference arithmetic unit601in the arithmetic unit507executes the first inference processing for the frame (current frame) corresponding to the current frame period according to the operation command (operation instruction). Then, the arithmetic unit507stores the intermediate output105, which is the result of the first inference processing, in the RAM509and notifies the CPU506of the completion of processing.

In step S804, the arithmetic unit507determines whether or not the training processing has been completed for the previous frame and whether or not the first inference processing has been completed for the current frame. Also, in step S405, the CPU506determines whether or not an interrupt instruction has been issued.

When the determination result is that the training processing for the previous frame and the first inference processing for the current frame have not been completed, the processing proceeds to step S802. Also, when the determination result is that the training processing for the previous frame has been completed but the first inference processing for the current frame has not been completed, the processing proceeds to step S803. When the determination result is that the training processing for the previous frame and the first inference processing for the current frame have both been completed, the processing proceeds to step S805. Also, when the CPU506determines that an interrupt instruction has been issued as the result of the determination, the processing for the current frame according to the flowcharts inFIGS.8A and8Bis ended.

In step S805, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The inference arithmetic unit601in the arithmetic unit507executes the second inference processing for the frame (current frame) corresponding to the current frame period according to the operation command (operation instruction). Then, the arithmetic unit507stores the tracking result107, which is the result of the second inference processing, in the RAM509and notifies the CPU506of the completion of processing.

In step S806, the arithmetic unit507determines whether or not the second inference processing has been completed for the current frame. Also, in step S806, the CPU506determines whether or not an interrupt instruction has been issued.

When the determination result is that the second inference processing for the current frame has not been completed, the processing proceeds to step S805and the second inference processing continues. Also, when the determination result is that the second inference processing for the current frame has been completed, the processing proceeds to step S807. Also, when the CPU506determines that an interrupt instruction has been issued as the result of the determination, the processing for the current frame according to the flowcharts inFIGS.8A and8Bis ended.

In step S807, the CPU506stores the fourth information in the RAM509. In this manner, after the processing of step S807, when the determination processing of step S801has been executed for the frame period subsequent to the current frame period (a frame period corresponding to the subsequent frame subsequent to the current frame), it is determined that the fourth information indicating that the second inference processing has been completed in the previous frame period is stored in the RAM509.

In step S808, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The training arithmetic unit602in the arithmetic unit507executes the training processing for the frame (current frame) corresponding to the current frame period according to the operation command (operation instruction). Then, the arithmetic unit507notifies the CPU506of the completion of processing.

In step S809, the arithmetic unit507determines whether or not the training processing has been completed for the current frame. Also, in step S809, the CPU506determines whether or not an interrupt instruction has been issued.

When the determination result is that the training processing for the current frame has been completed, the processing proceeds to step S810. Also, when the training processing for the current frame has not been completed, the processing proceeds to step S808and the training processing continues. Also, when the CPU506determines that an interrupt instruction has been issued, the processing for the current frame according to the flowcharts inFIGS.8A and8Bis ended. In step S810, the CPU506stores the second information in the RAM509.

The processing according to the flowcharts inFIGS.8A and8Bwill now be described using the example illustrated inFIG.7. First, the processing according to the flowcharts inFIGS.8A and8Bis executed for the first frame period.

In this example, the first frame period is the initial frame period, and thus the processing proceeds from step S801to step S803. In step S803, the arithmetic unit507executes the inference processing701for the frame corresponding to the first frame period. Then, when the inference processing701is complete, the processing proceeds to step S805via step S804. In step S805, the arithmetic unit507executes the inference processing702for the frame corresponding to the first frame period. Then, when the inference processing702is complete, the processing proceeds to step S807via step S806. In step S807, the CPU506stores the fourth information in the RAM509. Then, in step S808, the arithmetic unit507executes the training processing703for the frame corresponding to the first frame period, except that as an interrupt instruction has been issued during the training processing703, the processing according to the flowcharts inFIGS.8Aand8B in the first frame period is ended.

Next, the processing according to the flowcharts inFIGS.8A and8Bin the second frame period is executed and the fourth information is stored in the RAM509and the second information is not stored in the RAM509. Thus, the processing proceeds from step S801to step S802.

In step S802, the arithmetic unit507executes the training processing703for the frame corresponding to the first frame period, and in step S803, the arithmetic unit507executes the inference processing704for the frame corresponding to the second frame period. When the inference processing704and the training processing703are complete, the processing proceeds to step S805via step S804. In step S805, the arithmetic unit507executes the inference processing705for the frame corresponding to the second frame period. When the inference processing705is complete, the processing proceeds from step S806to step S807. In step S807, the CPU506stores the fourth information in the RAM509. Thereafter, in step S808, the arithmetic unit507executes the training processing706for the frame corresponding to the second frame period. Then, when the training processing706is complete, the processing proceeds to step S810via step S809. In step S810, the CPU506stores the second information in the RAM509.

Next, the processing according to the flowcharts inFIGS.8A and8Bin the third frame period is executed and the fourth information and the second information are stored in the RAM509. Thus, the processing proceeds from step S801to step S803.

In step S803, the arithmetic unit507executes the inference processing707for the frame corresponding to the third frame period. Then, when the inference processing707is complete, the processing proceeds to step S805via step S804. In step S805, the arithmetic unit507executes the inference processing708for the frame corresponding to the third frame period. Then, when the inference processing708is complete, the processing proceeds to step S807via step S806. In step S807, the CPU506stores the fourth information in the RAM509. Thereafter, in step S808, the arithmetic unit507executes the training processing709for the frame corresponding to the third frame period. Then, when the training processing709is complete, the processing proceeds to step S810via step S809. In step S810, the CPU506stores the second information in the RAM509.

In this manner, according to the present modified example, in the inference processing for the current frame, the first inference processing that does not use the second coefficient106(is not affected by the training processing) and the training processing of the previous frame are executed in parallel, allowing the processing time of the frame period to be decreased.

Second Embodiment

The differences with the first embodiment will be described below, and unless particularly mentioned, the other components are the same as in the first embodiment. The present embodiment is an example of a method for restarting inference processing of a previous frame period when an interrupt occurs during the inference processing in the previous frame period. According to the method described below, the inference processing and the training processing can be completed before the start of the processing of the current frame, which is the processing that requires the result of the training processing of the previous frame period.

A timechart according to the present embodiment is illustrated inFIG.9. In the case illustrated inFIG.9, inference processing901in the first frame period is interrupted by an interrupt instruction.

When the inference processing901in the first frame period is interrupted, training processing902that should be executed in the first frame period is not executed in the first frame period, and the interrupted inference processing901is executed first in the second frame period. Then, when the inference processing901is complete, the training processing902is executed. When the training processing902ends, inference processing903and training processing904that are executed in the second frame period are executed, but a time-out occurs at the end timing of the second frame period, and thus the processing being executed is interrupted. In the example illustrated inFIG.9, the inference processing903is complete, but the training processing904is interrupted due to the time-out. Then, in the third frame period, first, the training processing904interrupted due to the time-out is executed, and then inference processing905and training processing906that are executed in the third frame period are executed.

In this manner, the second coefficient106can be updated by the training processing902before the inference processing903of the second frame period is executed, and the second coefficient106can be updated by the training processing904before the inference processing905of the third frame period is executed.

In other words, even when the inference processing901in the first frame period is interrupted by an instruction in response to a user operation, the result of the training processing902can be applied to the inference processing903, and the result of the training processing904can be applied to the inference processing905.

Next, processing executed by the information processing apparatus to execute the inference processing and the training processing described above in the current frame period will be described with reference to the flowcharts inFIGS.10A and10B. Note that the information processing apparatus can execute the inference processing and the training processing for each frame period (each frame) by executing the processing according to the flowcharts inFIGS.10A and10Bfor each frame period (each frame).

In step S1001, the CPU506determines whether or not the first information indicating that the inference processing in the frame period (previous frame period) preceding the current frame period has been completed is stored in the RAM509. Also, the CPU506determines whether or not the second information indicating that the training processing in the previous frame period has been completed is stored in the RAM509.

When the determination result is that the first information is stored in the RAM509and the second information is not stored in the RAM509, the CPU506determines that “in the previous frame period, the inference processing has been completed but the training processing has not been completed”, and the processing proceeds to step S1004.

When the determination result is that the first information and the second information are both stored in the RAM509, the CPU506determines that “in the previous frame period, the inference processing and the training processing have been completed”, and the processing proceeds to step S1006.

Also, when the determination result is that neither the first information nor the second information is stored in the RAM509, the processing proceeds to step S1002. Note that when neither the first information nor the second information is stored in the RAM509and the inference processing that should have been executed in the previous frame period is restarted in the next frame period, the processing that should have been executed in the previous frame period may be discarded if it is not desirable that the delays in frame processing keep building up.

Note that when the current frame period is the initial frame period, the processing proceeds to step S1006without the CPU506determining whether or not the first information and the second information are stored in the RAM509.

In step S1002, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The arithmetic unit507executes the inference processing for the frame (previous frame) corresponding to the previous frame period according to the operation command (operation instruction). Then, the arithmetic unit507stores the tracking result107in the RAM509and notifies the CPU506of the completion of processing.

In S1003, the arithmetic unit507determines whether or not the inference processing has been completed for the previous frame. Also, in step S1003, the CPU506determines whether or not an interrupt instruction has been issued.

When the determination result is that the inference processing for the previous frame has been completed, the processing proceeds to step S1004. Also, when the inference processing for the previous frame has not been completed, the processing proceeds to step S1002and the inference processing continues. Also, when the CPU506detects that an interrupt instruction has been issued, the processing for the current frame according to the flowcharts inFIGS.10A and10Bis ended.

In step S1004, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The arithmetic unit507executes the training processing for the frame (previous frame) corresponding to the previous frame period according to the operation command (operation instruction). Then, the arithmetic unit507notifies the CPU506of the completion of processing.

In step S1005, the arithmetic unit507determines whether or not the training processing has been completed for the previous frame. Also, in step S1005, the CPU506determines whether or not an interrupt instruction has been issued.

When the determination result is that the training processing for the previous frame has been completed, the processing proceeds to step S1006. Also, when the training processing for the previous frame has not been completed, the processing proceeds to step S1004and the training processing continues.

Also, when the CPU506detects that an interrupt instruction has been issued, the processing for the current frame according to the flowcharts inFIGS.10A and10Bis ended.

In step S1006, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The arithmetic unit507executes the inference processing for the frame (current frame) corresponding to the current frame period according to the operation command (operation instruction). Then, the arithmetic unit507stores the tracking result107in the RAM509and notifies the CPU506of the completion of processing.

In step S1007, the arithmetic unit507determines whether or not the inference processing has been completed for the current frame. Also, in step S1007, the CPU506determines whether or not an interrupt instruction has been issued (whether or not an interrupt instruction has been input via a user operation or whether or not the CPU506or the like has issued an interrupt instruction due to the time-out).

When the determination result is that the inference processing for the current frame has been completed, the processing proceeds to step S1008. Also, when the inference processing for the current frame has not been completed, the processing proceeds to step S1006and the inference processing continues. Also, when an interrupt instruction has been issued, the processing for the current frame according to the flowcharts inFIGS.10A and10Bis ended.

In step S1008, the CPU506stores the first information in the RAM509. In this manner, after the processing of step S1008, when the determination processing of step S1001has been executed for the frame period subsequent to the current frame period (a frame period corresponding to the subsequent frame subsequent to the current frame), it is determined that the first information indicating that the inference processing has been completed in the previous frame period is stored in the RAM509.

In step S1009, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The arithmetic unit507executes the training processing for the frame (current frame) corresponding to the current frame period according to the operation command (operation instruction). Then, the arithmetic unit507notifies the CPU506of the completion of processing.

In step S1010, the arithmetic unit507determines whether or not the training processing has been completed for the current frame. Also, in step S1010, the CPU506determines whether or not an interrupt instruction has been issued (whether or not an interrupt instruction has been input via a user operation or whether or not the CPU506or the like has issued an interrupt instruction due to the time-out).

When the determination result is that the training processing for the current frame has been completed, the processing proceeds to step S1011. Also, when the training processing for the current frame has not been completed, the processing proceeds to step S1009and the training processing continues. Also, when an interrupt instruction has been issued, the processing for the current frame according to the flowcharts inFIGS.10A and10Bis ended.

In step S1011, the CPU506stores the second information in the RAM509. In this manner, after the processing of step S1011, when the determination processing of step S1001has been executed for the frame period subsequent to the current frame period (a frame period corresponding to the subsequent frame subsequent to the current frame), it is determined that the second information indicating that the training processing has been completed in the previous frame period is stored in the RAM509.

In this manner, according to the present embodiment also, even when processing is interrupted by an interrupt instruction, the training processing that should have been executed in the previous frame period is completed before the subsequent inference processing. Thus, the second coefficient106can be appropriately updated in the current frame period. This can prevent tracking failures caused by the second coefficient106not being updated.

Modified Example of Second Embodiment

In this modified example of the second embodiment, the training processing for the previous frame and the inference processing for the current frame are executed in parallel. By using such a configuration, in addition to the effects according to the second embodiment, an increase in the processing time for the current frame period can be prevented. Also, when up to the first inference processing is completed in the previous frame, by the processing being started from the second inference processing, an increase in the processing time can be decreased. The differences with the second embodiment will be described below, and unless particularly mentioned, the other components are the same as in the second embodiment.

A timechart according to the present modified example is illustrated inFIG.11. In the case illustrated inFIG.11, inference processing1102in the first frame period is interrupted by an interrupt instruction.

Inference processing1101executed in the first frame period corresponds to the processing by the first convolutional layer101from among the processing included in the inference processing301described above. Also, the inference processing1102executed in the first frame period corresponds to the processing by the second convolutional layer102from among the processing included in the inference processing301described above.

When the inference processing1101and the inference processing1102in the first frame period are complete, training processing1103is started, but an interrupt instruction is issued during the inference processing1102, causing the inference processing1102and the training processing1103to be interrupted.

In the second frame period, the interrupted inference processing1102is executed. After the inference processing1102is complete, the training processing1103and inference processing1104(the processing by the first convolutional layer101from among the processing included in the inference processing302described above) are executed in parallel. The inference processing1104is a processing using the first convolutional layer101, and thus the first coefficient104is used but the second coefficient106is not used. In other words, the inference processing1104is not affected by the training processing1103and may thus be executed in parallel with the training processing1103. Also, inference processing1105is executed using the second convolutional layer102set with the second coefficient106updated by the training processing1103, and thus the inference processing1105is started after the training processing1103is complete. Then, when the inference processing1105is complete, training processing1106is executed. In the third frame period, the inference processing1107is executed. The inference processing1108is executed after the inference processing1107is complete. And the training processing1109is executed after the inference processing1108is complete.

Next, processing executed by the information processing apparatus to execute the inference processing and the training processing described above in the current frame period will be described with reference to the flowcharts inFIGS.12A and12B. Note that the information processing apparatus can execute the inference processing and the training processing for each frame period (each frame) by executing the processing according to the flowcharts inFIGS.12A and12Bfor each frame period (each frame).

In step S1201, the CPU506determines whether or not third information indicating that the first inference processing in the frame period (previous frame period) preceding the current frame period has been completed is stored in the RAM509. Also, the CPU506determines whether or not the fourth information indicating that the second inference processing in the previous frame period has been completed is stored in the RAM509. Also, the CPU506determines whether or not the second information indicating that the training processing in the previous frame period has been completed is stored in the RAM509.

When the determination result is that the third information is stored in the RAM509and the fourth information and the second information are not stored in the RAM509, the CPU506determines that “in the previous frame period, the first inference processing has been completed but the second inference processing and the training processing has not been completed”. Then, the processing proceeds to step S1204.

Also, when the determination result is that the third information and the fourth information are stored in the RAM509and the second information is not stored in the RAM509, the CPU506determines that “in the previous frame period, the first inference processing and the second inference processing have been completed but the training processing has not been completed”. Then, the processing proceeds to step S1206.

When the determination result is that the third information, the fourth information, and the second information are stored in the RAM509, the CPU506determines that “in the previous frame period, the first inference processing, the second inference processing, and the training processing have been completed”, and the processing proceeds to step S1207.

Also, when the determination result is that neither the third information, the fourth information, nor the second information is stored in the RAM509, the processing proceeds to step S1202. Note that when neither the third information, the fourth information, nor the second information is stored in the RAM509and the inference processing that should have been executed in the previous frame period is restarted in the next frame period, the processing that should have been executed in the previous frame period may be discarded if it is not desirable that the delays in frame processing keep building up.

Note that when the current frame period is the initial frame period, the processing proceeds to step S1207without the CPU506determining whether or not the third information, the fourth information, and the second information are stored in the RAM509.

In step S1202, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The inference arithmetic unit601in the arithmetic unit507executes the first inference processing for the frame (previous frame) corresponding to the previous frame period according to the operation command (operation instruction). Then, the arithmetic unit507stores the intermediate output105, which is the result of the first inference processing, in the RAM509and notifies the CPU506of the completion of processing.

In step S1203, the arithmetic unit507determines whether or not the first inference processing has been completed for the previous frame. Also, in step S1203, the CPU506determines whether or not an interrupt instruction has been issued.

When the determination result is that the first inference processing for the previous frame has not been completed, the processing proceeds to step S1202and the first inference processing continues. Also, when the determination result is that the first inference processing for the previous frame has been completed, the processing proceeds to step S1204. Also, when the CPU506determines that an interrupt instruction has been issued as the result of the determination, the processing for the current frame according to the flowcharts inFIGS.12A and12Bis ended.

In step S1204, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The inference arithmetic unit601in the arithmetic unit507executes the second inference processing for the previous frame according to the operation command (operation instruction). Then, the arithmetic unit507stores the tracking result107, which is the result of the second inference processing, in the RAM509and notifies the CPU506of the completion of processing.

In step S1205, the arithmetic unit507determines whether or not the second inference processing has been completed for the previous frame. Also, in step S1205, the CPU506determines whether or not an interrupt instruction has been issued.

When the determination result is that the second inference processing for the previous frame has not been completed, the processing proceeds to step S1204and the second inference processing continues. Also, when the determination result is that the second inference processing for the previous frame has been completed, the processing proceeds to step S1206. Also, when the CPU506determines that an interrupt instruction has been issued as the result of the determination, the processing for the current frame according to the flowcharts inFIGS.12A and12Bis ended.

In step S1206, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The training arithmetic unit602in the arithmetic unit507executes the training processing for the previous frame according to the operation command (operation instruction). Then, the arithmetic unit507notifies the CPU506of the completion of processing.

In step S1207, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The inference arithmetic unit601in the arithmetic unit507executes the first inference processing for the current frame according to the operation command (operation instruction). Then, the arithmetic unit507stores the intermediate output105, which is the result of the first inference processing, in the RAM509and notifies the CPU506of the completion of processing.

In step S1208, the arithmetic unit507determines whether or not the training processing has been completed for the previous frame and whether or not the first inference processing has been completed for the current frame. Also, in step S405, the CPU506determines whether or not an interrupt instruction has been issued.

When the determination result is that the training processing for the previous frame and the first inference processing for the current frame have not been completed, the processing proceeds to step S1206. Also, when the determination result is that the training processing for the previous frame has been completed but the first inference processing for the current frame has not been completed, the processing proceeds to step S1207. When the determination result is that the training processing for the previous frame and the first inference processing for the current frame have both been completed, the processing proceeds to step S1209. Also, when the CPU506determines that an interrupt instruction has been issued as the result of the determination, the processing for the current frame according to the flowcharts inFIGS.12A and12Bis ended.

In step S1209, the CPU506stores (records) the third information in the RAM509. In this manner, after the processing of step S1209, when the determination processing of step S1201has been executed for the frame period subsequent to the current frame period (a frame period corresponding to the subsequent frame subsequent to the current frame), it is determined that the third information indicating that the first inference processing has been completed in the previous frame period is stored in the RAM509.

In step S1210, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The inference arithmetic unit601in the arithmetic unit507executes the second inference processing for the current frame according to the operation command (operation instruction). Then, the arithmetic unit507stores the tracking result107, which is the result of the second inference processing, in the RAM509and notifies the CPU506of the completion of processing.

In step S1211, the arithmetic unit507determines whether or not the second inference processing has been completed for the current frame. Also, in step S1211, the CPU506determines whether or not an interrupt instruction has been issued.

When the determination result is that the second inference processing for the current frame has not been completed, the processing proceeds to step S1210and the second inference processing continues. Also, when the determination result is that the second inference processing for the current frame has been completed, the processing proceeds to step S1212. Also, when the CPU506determines that an interrupt instruction has been issued as the result of the determination, the processing for the current frame according to the flowcharts inFIGS.12A and12Bis ended.

In step S1212, the CPU506stores (records) the fourth information in the RAM509. In this manner, after the processing of step S1212, when the determination processing of step S1201has been executed for the frame period subsequent to the current frame period (a frame period corresponding to the subsequent frame subsequent to the current frame), it is determined that the fourth information indicating that the second inference processing has been completed in the previous frame period is stored in the RAM509.

In step S1213, the CPU506issues an operation command (operation instruction) to the arithmetic unit507. The training arithmetic unit602in the arithmetic unit507executes the training processing for the current frame according to the operation command (operation instruction). Then, the arithmetic unit507notifies the CPU506of the completion of processing.

In step S1214, the arithmetic unit507determines whether or not the training processing has been completed for the current frame. Also, in step S1214, the CPU506determines whether or not an interrupt instruction has been issued.

When the determination result is that the training processing for the current frame has been completed, the processing proceeds to step S1215. Also, when the training processing for the current frame has not been completed, the processing proceeds to step S1213and the training processing continues. Also, when the CPU506determines that an interrupt instruction has been issued, the processing for the current frame according to the flowcharts inFIGS.12A and12Bis ended. In step S1215, the CPU506stores the second information in the RAM509.

In this manner, according to the present embodiment and as with the second embodiment, even when processing is interrupted by an interrupt instruction, the training processing that should have been executed in the previous frame period is completed before the subsequent second inference processing. Thus, the second coefficient106can be appropriately updated in the current frame period.

Also, in the present modified example, as with the modified example of the first embodiment, the first inference processing for the current frame and the training processing for the previous frame can be executed in parallel, allowing the processing time of the frame period to be decreased.

Also, in the present modified example, as seen in the example illustrated inFIG.11, an increase in the processing time of the inference processing for the second frame period can be decreased. In the present modified example, the inference processing903described in the second embodiment is divided in the first inference processing1104and the second inference processing1105, and in the second frame period, processing is restarted from the second inference processing1105. This allows the processing time to be decreased by an amount corresponding to the first inference processing1104.

Third Embodiment

In the embodiments and the modified examples described above, the second coefficient106, which is the parameter the second inference processing requires from the training processing, is updated, but the first coefficient104, which is the parameter the first inference processing requires from the training processing, may be updated. In this case, the training processing for the previous frame needs to be completed before the first inference processing for the current frame. In the case illustrated inFIG.11, in the second frame period, the inference processing1102and the training processing1103are executed in parallel. Then, when this execution is complete, the inference processing1104and the training processing of the first coefficient104in the training processing1106are executed in parallel. Next, the inference processing1105is executed. Then, when this execution is complete, the training processing of the second coefficient106in the training processing1106is executed. In other words, execution of the training processing can be started directly after the first inference processing, and the second inference processing can be executed without waiting for the training processing to be completed. Accordingly, the second inference processing and the training processing for the same frame may be executed in parallel, for example.

Also, in an inference, separate from the inference result, post-processing for generating an ultimately output tracking result may be executed. On the other hand, when the post-processing result cannot be used by the training, the post-processing is executed for only the inference for the current frame, meaning that the post-processing may be omitted in the inference for the previous frame. Accordingly, when the inference processing for the previous frame is executed, compared to executing the inference processing for the current frame, the processing time can be decreased by an amount correspond to the post-processing.

Also, in the modified example of the second embodiment, the inference processing is split into the first inference processing and second inference processing and executed. In the case illustrated inFIG.11, in the first frame period, up to the first inference processing relating to the first frame is executed, and in the second frame period, processing is started from the second inference processing. However, the training processing may also be split into first training processing and second training processing (for example, in the training processing, the processing in the first half may be the first training processing and the processing in the second half may be the second training processing). Also, for example, when the first training processing in the first frame period has been completed, but the second training processing has not been completed, in the second frame period, first, the processing is started from the second training processing. Then, after the second training processing is complete, processing similar to that in the third frame period illustrated inFIGS.7and11may be executed. In this manner, as in the modified example of the second embodiment, the processing time can be decreased by an amount corresponding to the first training processing.

Also, the numerical values; processing timing; processing order; processing; subjects of processing; obtaining method, transmission destination, transmission source, and storage place of data (information); and the like used in the embodiments described above are examples for facilitating a detailed description, and no such limitations are intended.

Also, a part or all of the embodiments described above may be combined as appropriate. Furthermore, a part or all of the embodiments described above may be selectively used.

Other Embodiments

This application claims the benefit of Japanese Patent Application No. 2022-158760, filed Sep. 30, 2022, which is hereby incorporated by reference herein in its entirety.