Patent ID: 12243244

EXAMPLE EMBODIMENTS

In the following, example embodiments will be described with reference to the accompanying drawings.

In a case of recognizing a moving object in a captured image, when a size of the moving object is small, an error is likely to occur in an estimation of a center position of the moving object in the captured image. In the following example embodiments, the moving object in a captured image is extracted as a foreground, and a center position of the foreground is accurately estimated using a feature of a movement trajectory of the foreground and a feature of an area.

First Example Embodiment

[Hardware Configuration]

FIG.1is a block diagram illustrating a hardware configuration of an image processing apparatus according to a first example embodiment. As illustrated, the image processing apparatus100includes an input IF (InterFace)12, a processor13, a memory14, a recording medium15, and a database (DB)16.

The input IF12acquires an input image to be processed by the image processing apparatus100. For example, images capturing moving objects flying through the sky are input through the input IF12by cameras installed on the ground. The processor13is a computer such as a CPU (Central Processing Unit) and controls the entire image processing apparatus100by executing a program prepared in advance. Specifically, the processor13executes a learning process and a center estimation process to be described later.

The memory14is formed by a ROM (Read Only Memory), a RAM (Random Access Memory), or the like. The memory14stores various programs to be executed by the processor13. The memory14is also used as a working memory during executions of various processes by the processor13.

The recording medium15is a non-volatile and non-transitory recording medium such as a disk-shaped recording medium or a semiconductor memory, and is formed to be detachable from the image processing apparatus100. The recording medium15stores various programs executed by the processor13. When the image processing apparatus100executes various kinds of processes, programs recorded on the recording medium15are loaded into the memory14and executed by the processor13.

The database16stores data input from an external apparatus. Specifically, input images to be processed by the image processing apparatus100are stored. In addition to the above, the image processing apparatus100may include an input device such as a keyboard or a mouse or a display unit for allowing a user to perform instructions or inputs.

(Functional Configuration)

FIG.2is a block diagram illustrating a functional configuration of the image processing apparatus100according to the first example embodiment. As illustrated, the image processing apparatus100includes a foreground extraction unit21, a movement trajectory feature extraction unit22, an area variation feature extraction unit23, a foreground center estimation unit24, a foreground extraction model storage unit25, and a foreground center estimation model storage unit26.

The foreground extraction unit21extracts each foreground from input images. Each foreground is a portion other than a background in the input images, and is an area corresponding to a moving object to be identified. The foreground extraction model storage unit25stores a plurality of foreground extraction models for extracting foregrounds from the input images. For instance, the foreground extraction model is regarded as a model using a background subtraction technique. The foreground extraction unit21acquires a foreground extraction model from the foreground extraction model storage unit25, and extracts each foreground from the input images.

FIG.3is a diagram illustrating an example of a foreground extraction method. The foreground extraction unit21obtains a foreground extraction result by applying a plurality of foreground extraction models with respect to the input images. In the example ofFIG.3, the foreground extraction unit21applies foreground extraction models M1through M3to the input images, and outputs a foreground extraction result by voting on the obtained foreground extraction results31through33. The voting here is based on a majority. That is, the foreground extraction unit21includes each foreground included in the majority of the foreground extraction results31through33in the foreground extraction result. In an example ofFIG.3, a foreground FG1of a triangular shape included in all of the foreground extraction results31to33and a foreground FG2of a circular shape (a circle and an ellipse respectively) included in the foreground extraction results31and32are extracted and output to the movement trajectory feature extraction unit22.

The movement trajectory feature extraction unit22extracts a movement trajectory feature which is a feature of a movement trajectory of each foreground based on the foreground extraction result. Specifically, the movement trajectory feature extraction unit22first calculates a center position of each foreground included in the foreground extraction result for a plurality of adjacent frames (N frames) of the input images, and tracks each foreground. Here, the center position of each foreground is calculated based on a shape of the extracted foreground. Incidentally, in a case where the center of each foreground estimated by the foreground center estimation unit24to be described later is a final center of the foreground, the center of the foreground calculated here is a tentative center.

FIG.4Ais a diagram illustrating tracking of each foreground. In this example, three frames are used. The movement trajectory feature extraction unit22tracks the same foreground in foreground extraction results41through43obtained for a first frame through a third frame. In an example ofFIG.4A, the movement trajectory feature extraction unit22tracks the foreground FG1of the triangle in each of the foreground extraction results41through43, and generates correspondence relationships of the foreground FG1among the frames. Incidentally, the movement trajectory feature extraction unit22also tracks the foreground FG2of the circular shape in the same manner.

Next, the movement trajectory feature extraction unit22extracts the movement trajectory feature based on a tracking result of each foreground. Specifically, the movement trajectory feature extraction unit22acquires the coordinates of a center of each foreground in each frame and the movement amount of the center of the foreground between adjacent frames. Note that the center of each foreground here corresponds to the aforementioned tentative center. In the example ofFIG.4A, the movement trajectory feature extraction unit22acquires coordinates X11, X21, and X31of the center of the foreground FG1in each frame, movement amount d12of the center of the foreground FG1between the first frame and a second frame, and a movement amount d23of the center of the foreground FG1between the second frame and the third frame. After that, the movement trajectory feature extraction unit22generates the following movement trajectory feature.

Movement trajectory features: T1={X11, X21, X31, d12, d23} The movement trajectory feature extraction unit22similarly generates a movement trajectory feature T2of the foreground FG2. The obtained movement trajectory feature is output to the area variation feature extraction unit23and the foreground center estimation unit24.

The area variation feature extraction unit23extracts an area variation feature using the movement trajectory feature. The area variation feature represents a temporal variation of an area of a foreground between adjacent frames.FIG.4Bis a diagram illustrating a method for extracting the area variation feature. The area variation feature extraction unit23calculates the area of each foreground in the foreground extraction results41through43of the first frame to the third frame, and obtains the area variation feature as follows.
Area variation feature: Area1={R12,R23}
Here, an “R” indicates an area variation of the foreground, and an “R12” indicates a ratio of an area in the first frame to another area in the second frame, regarding a certain foreground. For instance, when the area in the first frame of the foreground FG1is denoted by S1, and the area in the second frame is denoted by S2, R=S1/S2is acquired. Alternatively, R=S2/S1may be acquired. Accordingly, the area variation feature extraction unit23extracts respective area variation features Area for all foregrounds included in the foreground extraction result, and outputs the extracted area variation features to the foreground center estimation unit24.

The foreground center estimation unit24estimates a center of each foreground using the movement trajectory feature and the area variation feature of the foreground. Specifically, the foreground center estimation unit24estimates the center of each foreground using a foreground center estimation model. The foreground center estimation model is stored in the foreground center estimation model storage unit26. The foreground center estimation unit24acquires the foreground center estimation model from the foreground center estimation model storage unit26, and estimates the center of each foreground using the acquired foreground center estimation model. Incidentally, the foreground center estimation model trained in advance is stored in the foreground center estimation model storage unit26.

FIG.5Aillustrates an example of a configuration of the foreground center estimation unit24. As illustrated, when the foreground center estimation unit24estimates the center of each foreground, the movement trajectory feature T extracted by the movement trajectory feature extraction unit22and the area variation feature Area extracted by the area variation feature extraction unit23are input to a foreground center estimation model51. As the foreground center estimation model, a regression model formed by a neural network or a SVM (Support Vector Machine) is used. The foreground center estimation model51estimates the center of each foreground using the input movement trajectory feature T and the area variation feature Area, and outputs coordinates of each foreground in the input images as an estimation result.

FIG.5Bshows an example of the estimation results by the foreground center estimation model. When a rectangle surrounding a foreground FG is represented by a Rect in an input image, the foreground center estimation model51outputs coordinates of a center of the foreground FG and coordinates of the rectangle Rect. Specifically, the coordinates of the center of the foreground FG are indicated by (X_center, Y_center). The coordinates of the rectangle Rect are indicated by coordinates (Rect_topx, Rect_topy) of an upper left point of the rectangle Rect and coordinates (Rect_downx, Rect_downy) of a lower right point. The foreground center estimation unit24outputs these coordinates as the estimation result.

As described above, in the present example embodiment, since the center of the foreground is estimated using the movement trajectory feature and the area variation feature of the foreground calculated based on the input images, it is possible to accurately estimate a center position of each foreground.

(Center Estimation Process)

FIG.6is a flowchart of a center estimation process performed by the image processing apparatus100. This process is realized by the processor13illustrated inFIG.1, which executes a program prepared in advance and operates as each element depicted inFIG.2.

First, the foreground extraction unit21extracts each foreground from a plurality of frames of input images (step S11). Specifically, the foreground extraction unit21acquires a plurality of foreground extraction models (N foreground extraction models) to extract foregrounds, and performs the aforementioned voting with respect to foregrounds extracted by respective foreground extraction models in order to extract each foreground. A foreground extraction result is input to the movement trajectory feature extraction unit22.

Next, the movement trajectory feature extraction unit22calculates a center position of each foreground included in the foreground extraction result of N frames (step S12), tracks a center position, and extracts the movement trajectory feature T for each foreground (step S13). The extracted movement trajectory features are output to the area variation feature extraction unit23and the foreground center estimation unit24. Next, the area variation feature extraction unit23extracts an area variation feature Area using the movement trajectory feature and the area of each foreground (step S14). The extracted area variation feature is output to the foreground center estimation unit24. After that, the foreground center estimation unit24uses the foreground center estimation model to estimate the center of each foreground based on the movement trajectory feature T and the area variation feature Area, and outputs an estimation result (step S15).

Next, the image processing apparatus100determines whether or not there is a remaining frame in the input images (step S16). When there is the remaining frame (step S16: Yes), the process goes back to step S11, and the processes of steps S11through S16are performed for a next frame set. For instance, when an object identification is performed using three adjacent frames as described above, the image processing apparatus100first performs processes of steps S11through S16using the first frame to the third frame as one frame set, and then performs processes of steps S11through S16using the second frame to the fourth frame as one frame set by shifting one frame. On the other hand, when there is no remaining frame (step S16: No), the center estimation process is terminated.

(Learning Foreground Center Estimation Model)

Next, the learning of the foreground center estimation model is explained. In the aforementioned image processing apparatus100, a foreground center estimation model that has already been trained is stored in the foreground center estimation model storage unit26, and the foreground center estimation unit24estimates a center of each foreground using the foreground center estimation model. The learning of the foreground center estimation model will be described below.

FIG.7is a block diagram illustrating a configuration of a learning apparatus200for the foreground center estimation model. The learning apparatus200for the foreground center estimation model includes a foreground extraction unit61, a movement trajectory feature extraction unit62, an area variation feature extraction unit63, a model learning unit64, a foreground extraction model storage unit65, and a foreground center estimation model storage unit66. Incidentally, the foreground extraction unit61, the movement trajectory feature extraction unit62, the area variation feature extraction unit63, and the foreground extraction model storage unit65are basically the same as the foreground extraction unit21, the movement trajectory feature extraction unit22, the area variation feature extraction unit23, and the foreground extraction model storage unit25of the image processing apparatus100illustrated inFIG.2, and operate in the same manner. Incidentally, input images for learning are input to the foreground extraction unit61.

The foreground extraction unit61extracts each foreground from the input images for learning, and outputs the extracted foregrounds to the movement trajectory feature extraction unit62. The movement trajectory feature extraction unit62extracts the movement trajectory feature based on the extracted foregrounds, and outputs the extracted movement trajectory feature to the area variation feature extraction unit63and the model learning unit64. The area variation feature extraction unit63extracts an area variation feature using the movement trajectory feature and the area of each foreground, and outputs the area variation feature to the model learning unit64.

The model learning unit64trains the foreground center estimation model.FIG.8is a block diagram illustrating a configuration of the model learning unit64. The model learning unit64includes a foreground center estimation model51and a learning unit52. Movement trajectory features and area variation features extracted from the input images for learning are input to the foreground center estimation model51. The foreground center estimation model51estimates the center of the foreground using the input movement trajectory features and area variation features, and outputs an estimation result. On the other hand, correct answer data are prepared with respect to the input images for learning. As depicted inFIG.8, the learning data are data corresponding to the estimation result, and include center coordinates of the foreground, and coordinates of an upper left point and the lower right point of a rectangle surrounding a foreground for each of foregrounds included in the input images for learning.

The learning unit52calculates a difference between the correct answer data and the estimation result output from the foreground center estimation model51for each of foregrounds, and optimizes the foreground center estimation model51so that a sum of calculated differences is taken as a loss and the loss is reduced. After that, the model learning unit64stores parameters of the foreground center estimation model51at a time when the learning is completed, in the foreground center estimation model storage unit26as the trained foreground center estimation model51.

FIG.9is a flowchart of the learning process of the foreground center estimation model. This process is realized by the processor13illustrated inFIG.1executing a program prepared in advance and operating as each element depicted inFIG.7.

First, the foreground extraction unit61extracts each foreground from a plurality of frames of input images (step S21). The foreground extraction result is input to the movement trajectory feature extraction unit62. Next, the movement trajectory feature extraction unit62calculates a center position of each foreground included in the foreground extraction result of N frames (step S22), and tracks a foreground to extract the movement trajectory feature T of each foreground (step S23). The extracted movement trajectory features are input to the area variation feature extraction unit63and the model learning unit64. Next, the area variation feature extraction unit63extracts the area variation feature Area using the movement trajectory feature and the area of each foreground (step S24). The extracted area variation features are input to the model learning unit64.

The model learning unit64estimates a center of each foreground using the movement trajectory feature T and the area variation feature Area (step S25). Next, the model learning unit64calculates a loss from the correct answer data and the estimated result by the foreground center estimation model (step S26), and optimizes the foreground center estimation model so that the loss is reduced (step S27).

Next, the learning apparatus200for the foreground center estimation model determines whether or not the learning has been completed, that is, whether or not a predetermined end condition has been provided (step S28). Until the end condition is provided, steps S21through S27are repeated using the input images for learning, and when the end condition is provided (step S28: Yes), the learning process is terminated.

Second Example Embodiment

FIG.10Ais a block diagram illustrating a configuration of an image processing apparatus according to a second example embodiment. An image processing apparatus80includes a foreground extraction unit81, a movement trajectory feature extraction unit82, an area variation feature extraction unit83, and a foreground center estimation unit84. The foreground extraction unit81extracts each foreground from input images, and generates a foreground extraction result. The movement trajectory feature extraction unit82tracks each foreground based on the foreground extraction result, and extracts the movement trajectory feature of each foreground. The area variation feature extraction unit83extracts the area variation feature indicating a temporal area variation of each foreground. The foreground center estimation unit84estimates a center of each foreground using the movement trajectory feature and the area variation feature for the foreground.

FIG.10Bis a block diagram illustrating a configuration of a learning apparatus according to the second example embodiment. A learning apparatus90includes a foreground extraction unit91, a movement trajectory feature extraction unit92, an area variation feature extraction unit93, and a learning unit94. The foreground extraction unit91extracts each foreground from the input images, and generates a foreground extraction result. The movement trajectory feature extraction unit92tracks each foreground based on the foreground extraction result, and extracts the movement trajectory feature of the foreground. The area variation feature extraction unit93extracts an area variation feature indicating the temporal area variation of each foreground. The learning unit94trains a foreground center estimation model that estimates a center of each foreground using the movement trajectory feature and the area variation feature for the foreground.

A part or all of the example embodiments described above may also be described as the following supplementary notes, but not limited thereto.

(Supplementary Note 1)

1. An image processing apparatus comprising:a foreground extraction unit configured to extract each foreground from input images, and generate a foreground extraction result;a movement trajectory feature extraction unit configured to track each foreground based on the foreground extraction result, and extract a movement trajectory feature of each foreground;an area variation feature extraction unit configured to extract an area variation feature representing a temporal area variation of each foreground; and a foreground center estimation unit configured to estimate a center of each foreground using the movement trajectory feature and the area variation feature of the foreground.
(Supplementary Note 2)2. The image processing apparatus according to claim1, whereinthe foreground extraction unit generates the foreground extraction result concerning a predetermined number of frames of the input images; andthe movement trajectory feature extraction unit extracts a movement of a tentative center of each foreground in the predetermine number of frames of the input images as the movement trajectory feature.
(Supplementary Note 3)3. The image processing apparatus according to claim2, wherein the movement trajectory feature includes position coordinates of the tentative center of each foreground and a movement amount of the tentative center of each foreground between frames with respect to each of the predetermined number of frames.
(Supplementary Note 4)4. The image processing apparatus according to any one of claims1through3, whereinthe foreground extraction unit generates the foreground extraction result with respect to the predetermined number of frames of the input images, andthe area variation feature extraction unit extracts a ratio of variation of an area of a foreground among adjacent frames based on the movement trajectory feature and the area of each foreground.
(Supplementary Note 5)5. The image processing apparatus according to any one of claims1through4, wherein the foreground center estimation unit estimates a center of each foreground using a foreground center estimation model trained in advance.
(Supplementary Note 6)6. The image processing apparatus according to claim5, wherein the foreground center estimation model corresponds to a model to which the movement trajectory feature and the area variation feature are input and which outputs center coordinates of each foreground in the input images.
(Supplementary Note 7)7. The image processing apparatus according to claim6, wherein the foreground center estimation model corresponds to the model that further outputs position coordinates of a rectangle surrounding each foreground in the input images.
(Supplementary Note 8)8. An image processing method comprising:extracting each foreground from input images, and generating a foreground extraction result;tracking each foreground based on the foreground extraction result, and extracting a movement trajectory feature of each foreground;extracting an area variation feature representing a temporal area variation of each foreground; andestimating a center of each foreground using the movement trajectory feature and the area variation feature of the foreground.
(Supplementary Note 9)9. A recording medium storing a program, the program causing a computer to perform a process comprising:extracting each foreground from input images, and generating a foreground extraction result;tracking each foreground based on the foreground extraction result, and extracting a movement trajectory feature of each foreground;extracting an area variation feature representing a temporal area variation of each foreground; andestimating a center of each foreground using the movement trajectory feature and the area variation feature of the foreground.
(Supplementary Note 10)10. A learning apparatus comprising:a foreground extraction unit configured to extract each foreground from input images, and generate a foreground extraction result;a movement trajectory feature extraction unit configured to track each foreground based on the foreground extraction result, and extract a movement trajectory feature of each foreground;an area variation feature extraction unit configured to extract an area variation feature representing a temporal area variation of each foreground; and a learning unit configured to train a foreground center estimation model that estimates a center of each foreground using the movement trajectory feature and the area variation feature.
(Supplementary Note 11)11. A learning method comprising:extracting each foreground from input images, and generate a foreground extraction result;tracking each foreground based on the foreground extraction result, and extracting a movement trajectory feature of each foreground;extracting an area variation feature representing a temporal area variation of each foreground; andtraining a foreground center estimation model that estimates a center of each foreground using the movement trajectory feature and the area variation feature.
(Supplementary Note 12)12. A recording medium storing a program, the program causing a computer to perform a process comprising:extracting each foreground from input images, and generate a foreground extraction result;tracking each foreground based on the foreground extraction result, and extracting a movement trajectory feature of each foreground;extracting an area variation feature representing a temporal area variation of each foreground; andtraining a foreground center estimation model that estimates a center of each foreground using the movement trajectory feature and the area variation feature.

While the disclosure has been described with reference to the example embodiments and examples, the disclosure is not limited to the above example embodiments and examples. It will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the claims.

DESCRIPTION OF SYMBOLS

21,61,81,91Foreground extraction unit22,62,82,92Movement trajectory feature extraction unit23,63,83,93Area variation feature extraction unit24,84Foreground center estimation unit25,65Foreground extraction model storage unit26,66Foreground center estimation model storage unit51Foreground center estimation model52,94Learning unit64Model learning unit