Object detection system and object detection method

A method detects an object in an image. The method extracts a first feature vector from a first region of an image using a first subnetwork and determines a second region of the image by processing the first feature vector with a second subnetwork. The method also extracts a second feature vector from the second region of the image using the first subnetwork and detects the object using a third subnetwork on a basis of the first feature vector and the second feature vector to produce a bounding region surrounding the object and a class of the object. The first subnetwork, the second subnetwork, and the third subnetwork form a neural network. Also, a size of the first region differs from a size of the second region.

FIELD OF THE INVENTION

This invention relates to neural networks, and more specifically to object detection systems and methods using a neural network.

BACKGROUND OF THE INVENTION

Object detection is one of the most fundamental problems in computer vision. The goal of an object detection is to detect and localize the instances of predefined object classes in the form of bounding boxes, e.g., with confidence values for given input images. An object detection problem can be converted to an object classification problem by a scanning window technique. However, the scanning window technique is inefficient because classification steps are performed for all potential image regions of various locations, scales, and aspect ratios.

The region-based convolution neural network (R-CNN) is used to perform a two-stage approach, in which a set of object proposals is generated as regions of interest (ROI) using a proposal generator and the existence of an object and the classes in the ROI are determined using a deep neural network. However, the detection accuracy of the R-CNN is insufficient for some case. Accordingly, another approach is required to further improve the object detection performance.

SUMMARY OF THE INVENTION

Some embodiments are based on recognition and appreciation of the fact that a method for detecting an object in an image includes extracting a first feature vector from a first region of an image using a first subnetwork; determining a second region of the image by processing the first feature vector with a second subnetwork, wherein a size of the first region differs from a size of the second region; extracting a second feature vector from the second region of the image using the first subnetwork; and detecting the object using a third subnetwork based on the first feature vector and the second feature vector to produce a bounding box surrounding the object and a class of the object, wherein the first subnetwork, the second subnetwork, and the third subnetwork form a neural network, wherein steps of the method are performed by a processor.

Accordingly, one embodiment discloses a non-transitory computer readable recoding medium storing thereon a program causing a computer to execute an object detection process, wherein the object detection process includes extracting a first feature vector from a first region of an image using a first subnetwork; determining a second region of the image by processing the first feature vector with a second subnetwork, wherein a size of the first region differs from a size of the second region; extracting a second feature vector from the second region of the image using the first subnetwork; and detecting the object using a third subnetwork on a basis of the first feature vector and the second feature vector to produce a bounding box surrounding the object and a class of the object, wherein the first subnetwork, the second subnetwork, and the third subnetwork form a neural network.

Another embodiment discloses an objection detection system that includes a human machine interface; a storage device including neural networks; a memory; a network interface controller connectable with a network being outside the system; an imaging interface connectable with an imaging device; and a processor configured to connect to the human machine interface, the storage device, the memory, the network interface controller and the imaging interface, wherein the processor executes instructions for detecting an object in an image using the neural networks stored in the storage device, wherein the neural networks perform steps of: extracting a first feature vector from a first region of the image using a first subnetwork; determining a second region of the image by processing the first feature vector with a second subnetwork, wherein a size of the first region differs from a size of the second region; extracting a second feature vector from the second region of the image using the first subnetwork; and detecting the object using a third subnetwork on a basis of the first feature vector and the second feature vector to produce a bounding box surrounding the object and a class of the object, wherein the first subnetwork, the second subnetwork, and the third subnetwork form a neural network.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Various embodiments of the present invention are described hereafter with reference to the figures. It would be noted that the figures are not drawn to scale elements of similar structures or functions are represented by like reference numerals throughout the figures. It should be also noted that the figures are only intended to facilitate the description of specific embodiments of the invention. They are not intended as an exhaustive description of the invention or as a limitation on the scope of the invention. In addition, an aspect described in conjunction with a particular embodiment of the invention is not necessarily limited to that embodiment and can be practiced in any other embodiments of the invention.

Some embodiments are based on recognition that an objection detection system that includes a human machine interface; a storage device including neural networks; a memory; a network interface controller connectable with a network being outside the system; an imaging interface connectable with an imaging device; and a processor configured to connect to the human machine interface, the storage device, the memory, the network interface controller and the imaging interface, wherein the processor executes instructions for detecting an object in an image using the neural networks stored in the storage device, wherein the neural networks perform steps of: extracting a first feature vector from a first region of the image using a first subnetwork; determining a second region of the image by processing the first feature vector with a second subnetwork, wherein a size of the first region differs from a size of the second region; extracting a second feature vector from the second region of the image using the first subnetwork; and detecting the object using a third subnetwork on a basis of the first feature vector and the second feature vector to produce a bounding box surrounding the object and a class of the object, wherein the first subnetwork, the second subnetwork, and the third subnetwork form a neural network.

FIG. 1shows a block diagram of an object detection system500according to some embodiments of the invention. The object detection system500includes a human machine interface (HMI)510connectable with a keyboard511and a pointing device/medium512, a processor520, a storage device530, a memory540, a network interface controller550(NIC) connectable with a network590including local area networks and internet network, a display interface560, an imaging interface570connectable with an imaging device575, a printer interface580connectable with a printing device585. The object detection system500can receive electric text/imaging documents595via the network590connected to the NIC550. The storage device530includes original images531, a filter system module532, and a neural network400. The pointing device/medium512may include modules that read programs stored on a computer readable recording medium.

For detecting an object in an image, instructions may be transmitted to the object detection system500using the keyboard511, the pointing device/medium512or via the network590connected to other computers (not shown in the figure). The object detection system500receives the instructions using the HMI510and executes the instructions for detecting an object in an image using the processor520using the neural network400stored in the storage device530. The filter system module532is operable to perform image processing to obtain predetermined formatted image from given images relevant to the instructions. The images processed by the filter system module532can be used by the neural network400for detecting objects. An object detection process using the neural network400is described below. In the following description, a glimpse region is referred as a glimpse box, a bounding box, a glimpse bounding box or a bounding box region, which is placed on a target in an image to detect the feature of the target object in the image.

Some embodiments are based on recognition that a method for detecting an object in an image include extracting a first feature vector from a first region of an image using a first subnetwork; determining a second region of the image by processing the first feature vector with a second subnetwork, wherein a size of the first region differs from a size of the second region; extracting a second feature vector from the second region of the image using the first subnetwork; and detecting the object using a third subnetwork based on the first feature vector and the second feature vector to produce a bounding box surrounding the object and a class of the object, wherein the first subnetwork, the second subnetwork, and the third subnetwork form a neural network, wherein steps of the method are performed by a processor.

FIG. 2shows a block diagram of a neural network400used in a computer-implemented object detection method for detecting the object in the image according to some embodiments. The neural network400includes a first subnetwork100, a second subnetwork200and a third subnetwork300. The neural network400is designed to detect a target object in an image by extracting features from the target object in the image. The neural network400adaptively and sequentially places glimpse boxes on the target object of the image to extract the features of the target object, in which the glimpse boxes are designed to have square shapes or rectangular with different sizes and different aspect ratios in this embodiment. However, the shape of a glimpse box may be other predetermined shapes, such as triangle, hexagonal, circle, ellipse or any polygons according to the algorithm used in the neural network400. The neural network400, which applies the sequence of placing glimpse boxes on the target object, may be referred to as an attention-based object detection (AOD) network. The neural network400allows the object detection system500to make determination of classification of the target object in the image by providing the features extracted from the glimpse boxes corresponding the target object in the image. In this case, the first subnetwork100may be a deep convolutional neural network (DCNN)100to obtain better features of target objects in the image.

In an object detection process, when an image11is provided, the whole area of the image11is processed by the deep convolutional neural network (DCNN)100to compute and preserve a set of feature maps150. Concurrently, a proposal generator10provides a proposal bounding box as a first glimpse box15to place on a target object of the image11. The first glimpse box15has parameters including positions x, y, width w and h of the proposal bounding box in the image11. The DCNN100also extracts a feature of the first glimpse box15from a region in the feature maps150, in which the region in the feature maps150corresponds to the proposal bounding box (the first glimpse box15) of the image11.

The second subnetwork200includes region of interest (ROI) pooling modules210and220, and a recurrent neural network (RNN)205. The ROI pooling module210is applied to the first feature of the first glimpse box15to generate a first feature vector230by use of the RNN205. The first feature vector230is transmitted to an Element-wise MAX310of the third subnetwork300. The second subnetwork200also generates a target vector250that is used to encode a scale-invariant translation and a log-space height/width shift regarding a second glimpse box20relative to the first glimpse box15(the anchor bounding box). The second glimpse20is obtained by using the proposed bounding box15and the target vector250. Successively, the DCNN100extracts a second feature based on the second glimpse20of the image11. The second feature of the second glimpse box20is then applied to the ROI pooling module220of the second subnetwork200to generate a second feature vector240. The second subnetwork200transmits the second feature vector240to the Element-wise MAX310of the third subnetwork300and the Element-wise MAX310preserves the second feature vector240for use of the object detection process. The third subnetwork300processes the second feature vector230and the third feature vector240and generates an object bounding box320and an object class probability330as outputs. Based on the outputs, the object detection system500makes a final decision on an object class and an object location of the target object in the image11.

In some embodiments, the proposal generator10may be arranged in the neural network400according to the design of the neural network architecture.

FIG. 3AandFIG. 3Bshow a block diagram and a flowchart for illustrating the processes of extracting the feature vectors and determining the regions in the image used in the subnetworks according to some embodiments of the invention.

In a block S0inFIG. 3A, an initial process of extracting a first feature vector using the first subnetwork100is shown. Upon the instructions given to the object detection system500, the image11is provided into a working memory of the processor520. The working memory can be an internal memory of the processor520, the memory540or the storage530connected to the processor520. The processor520may be more than one processing unit to increase the processing speed of object detections according to the system design. The DCNN100is applied to the whole area of the image11to obtain a set of feature maps150, and subsequently, the proposal generator10provides a proposal bounding box as a first glimpse region15. The first glimpse region15may be referred to as a first glimpse box15. The first subnetwork100(DCNN100) extracts a first feature of the first glimpse region15from a corresponding region of the feature maps150and transmits the first feature to the second subnetwork200. In this case, the corresponding region of the feature maps150is associated to the first glimpse region15of the image11.

Steps S1and S2inFIG. 3AandFIG. 3Bshow processes of extracting a first feature vector from the first region using the first and second subnetworks100and200. In step S1ofFIG. 3AandFIG. 3B, after receiving the first feature of the first glimpse region15from the feature maps150, the second subnetwork200generates and outputs a first feature vector. Concurrently, the second subnetwork200generates a target vector associated to a second glimpse region20. In this case, the second subnetwork200determines the second glimpse region20by processing the target vector. The sizes of the first and second glimpse regions15and20are different from each other. The sequential order of steps S1and S2may be exchanged because the second subnetwork200can generate the first feature vector and the target vector without limiting the processing order as long as the first feature of the first glimpse region15has been acquired by the second subnetwork200in step S0.

In step S3inFIG. 3AandFIG. 3B, a second feature vector is extracted from the second region using the first and second subnetwork100. After receiving a second feature of the second glimpse region20, the second subnetwork200transmits the second feature of the second glimpse region20to the third subnetwork300. It should be noted that the first feature vector has been received by the third subnetwork300in step S1.

In step S4, the third subnetwork300detects an object in the image11based on the first and second feature vectors.

FIG. 3Cshows an example of the DCNN100including convolutional layers. In the DCNN100, a set of feature maps150is computed at the last convolutional layer. The DCNN100computes so that a feature of the feature maps is associated to the first region of the image11.

Further, in some embodiments, the DCNN100may be a pre-trained network such as the AlexNet or the VGGNet to obtain an equivalent effect for detecting objects in images.

FIG. 4AandFIG. 4Bshow examples of placing glimpse boxes on target objects in images. In those examples, the object detecting process is performed by four steps of placing boxes on a target image of an image until a final detection box is placed on the target image.

FIG. 4Aincludes two images4A-1and4A-2at upper side and bottom side. The images4A-1and4A-2show an identical dog image as a target object. The image4A-1indicates an object proposal box4A, a first glimpse box4A and a second glimpse box4A on the dog image.

In the first step of detecting process of the target object, the object proposal box4A is generated as an anchor bounding box by the proposal generator10and placed on the image4A-1to surround the dog image, in which the object proposal box4A is indicated by a bright box.

In the second step, the second subnetwork200generates the first glimpse box4A and a first feature vector after receiving a feature of the object proposal box4A extracted from the feature maps150by the first subnetwork100. The first glimpse box4A, indicated by dashed lines in the image4A-1, is then placed on the image4A-1to surround the object proposal box4A, and the first feature vector is transmitted to the third subnetwork300. In this case, the size and shape of the first glimpse box4A is configured to be different from those of the object proposal box4A.

In the third step, the first subnetwork100extracts a feature of the first glimpse box4A using the feature maps150and transmits the feature of the first glimpse box4A to the second subnetwork200. The second subnetwork200generates the second glimpse box4A and a second feature vector based on the feature of the first glimpse box4A and transmits the second feature vector to the third subnetwork300. The second subnetwork200also places the second glimpse box4A on the image4A-1to surround the object proposal box4A. In this case, the area of the second glimpse box4A is configured to be narrower than that of the first glimpse box4A as indicated in the image4A-1.

In the fourth step, a feature of the second glimpse box4A is extracted from the feature maps150by the first subnetwork100and transmitted to the second subnetwork200. Successively, the second subnetwork200generates and transmits a third feature vector to the third subnetwork300.

In the final step, the third subnetwork300outputs an object class probability and an object bounding box based on the first, second and third feature vectors. The third subnetwork300determines that the target object is a dog places based on the object class probability and places the final detection box4A on the image4A-2to surround the target surrounding the dog image as indicated in the image4A-2.

FIG. 4Bincludes two images4B-1and4B-2at upper side and bottom side. The images4B-1and4B-2show bird images as target objects. The image4B-1indicates object proposal boxes4B, first glimpse boxes4B and second glimpse boxes4B on the bird images. The object detecting process is performed by four steps of placing boxes on the target image until the final detection boxes are placed on the target images. As the object detecting process performed inFIG. 4Bare identical to those described with respect toFIG. 4A, the detail descriptions are omitted. The image4B-2shows final detection boxes are placed on the bird images. It should be noted that the plural targets are correctly detected even part of one of the target objects is missing in this case.

Glimpse Box Generation

In some embodiments, a glimpse region is referred to as a glimpse box. A glimpse box G computed in time step t by the second subnetwork200is expressed by Gt∈R4. For t=0, the first glimpse box G0is provided as a proposal bounding box by the proposal generator10. The proposal bounding box is used as an anchor bounding box. For t>0, subsequent glimpse boxes Gtare dynamically determined by the first subnetwork100and the second subnetwork200by aggregating information of features of prior glimpse boxes acquired in prior process steps.

For obtaining a glimpse box Gt, the scale-invariant and height/width normalized shift parameterization is employed with the anchor bounding box. The scale-invariant translation and the log-space height/width shift parameterization provides a target vector (δx, δy, δw, δh) that indicates amounts of shift from the anchor bounding box. The target vector is expressed as follows.

(δx,δy,δw,δh)=(gx-pxpw,gy-pyph,log⁢gwpw,log⁢ghph)(1)
where (gx, gy, gw, gh) represents the center coordinate x and y, width w and height h of a glimpse box, and (px, py, pw, ph) represents the proposal bounding box.

In the first time, the target vector (δx, δy, δw, δh) is obtained from the proposal bounding box according to equation (1). In the following, a new glimpse box is obtained as (px+pwδx, py+phδy, pwexp(δw), phexp(δh)).

For each glimpse box Gt, a predetermined dimensional feature vector is extracted by applying the ROI pooling module210to the feature of the first glimpse box15. The ROI pooling module210receives the feature of the first glimpse box15as an ROI. The ROI pooling module210divides the feature of the first glimpse box15into a predetermined grid of sub-windows and then max-pools feature values of the feature of the first glimpse box15in each sub-window. The feature values pooled fed into a recurrent neural network (RNN)205having layers fc6and fc7.

FIG. 5shows an example of 3×3 grid of sub-windows901. In this case, the feature of the first glimpse box15is divided into 3×3 grids of the sub-windows according to a predetermined architecture of the recurrent neural network.

In some embodiments, the RNN200may be a stacked recurrent neural network (stacked RNN).FIG. 6shows the stacked RNN200to which the feature values are fed from the ROI pooling modules210and220. The stacked RNN200includes two layers of fc6and fc7, in which the given feature values processed along directions of the arrows as indicated inFIG. 5.

In some embodiments, the stacked RNN200may include three steps.FIG. 7shows an example of the stacked RNN200having three steps. In this case, a first glimpse box is generated by the proposal generator10, and a second glimpse box and a third glimpse box are generated through the ROI pooling modules210and220, respectively. As each step of generating glimpse boxes and glimpse vectors is identical to that described in the case ofFIG. 2, detailed descriptions are omitted.

In this case, the subnetwork300determines an object class probability and an object boundary box based on the first, second and third feature vectors.FIG. 8shows an example of the stacked RNN200including three steps according to some embodiments. The arrows indicate data processing flows in the stacked RNN200.

The number of layers fc6and fc7may be increased more than three according to the architecture design of a stacked RNN.

In some embodiments, the DCNN10may be trained using a reinforcement learning algorithm as another architecture of network to generate glimpses in order to improve the detection performance.

Reinforcement Learning and Network Training

In a process of a reinforcement learning algorithm, a reinforcement learning agent (RL agent) continually interacts with an environment by observing a state x∈χ of the environment, and then the RL agent chooses an action a∈A according to its policy π(α|x) and a probabilistic mapping from the state to actions.

Depending on the current state and the action being chosen, the state of the RL agent in the environment changes to X′ ˜P(·|x,α). The RL agent also receives a real-valued reward signal r˜R(·|x,α) that is stored as a reword. This interaction continues for predetermined finite number of steps T. An outcome resulted from the interaction in each of steps T is referred to as an episode ξ.

The RL agent is configured to maximize the sum of rewards received in all episodes, R(ξ)=Σt=1Trt. In this case, R(ξ) represents a return of ξ. The goal of the reinforcement learning is to find a policy π so that an expected return J(π)Eπ[R(ξ)] is maximized, in which the policy π is not learned based on the reword received in each of time steps T. In the reinforcement learning process, a policy gradient algorithm, which is called the REINFORCE algorithm, is employed. In this case, π is parameterized by θ. The policy gradient algorithm, in its simplest form, changes policy parameters in the direction of gradient of J(πθ) by the gradient ascent update, θi+1←θi+αi∇J(πθi) for some choice of step size αi>0.

By using the Gaussian distribution as πθ, approximate gradients are computed by

generating multiple episodes under the current policy:

The algorithm discussed above is a gradient ascent algorithm and introduced to the standard back propagation neural network training. In this embodiment, the neural network400is trained by back propagating both gradient from reinforcement learning and gradients from supervised training.

FIG. 2shows a block diagram of the second subnetwork200formed by a stacked recurrent neural network (RNN). The second subnetwork200includes a stacked recurrent neural network (RNN)205and the ROI pooling modules210and220. Recurrent connections are indicated as fc6and fc7in the figure.

The training data of the RNN205is constructed by a similar way to that of the region-based convolution neural network algorithm (R-CNN algorithm). Each proposal bounding box generated by the proposal generator10is assigned a class label c* among one background class and K foreground object classes according to the overlaps with the ground truth object bounding boxes. The background class can be any object that does not belong to the foreground classes.

Each proposal bounding box is given a bounding box target vector that encodes the scale-invariant translation and log-space height/width shift relative to the object. The bounding box target vector is not defined for the ground truth object bounding boxes, and thus they are not used for training the RNN205.

The third subnetwork300provides the final outputs that are softmax classification scores and bounding boxes for all predefined foreground classes.

During training of the second subnetwork200, ground-truth annotations are provided for all predefined foreground classes, and the Standard Back Propagation Through Time (BPTT) algorithm is used for training. In this case, the BPTT is not applied to train a glimpse generation layer (G-layer) inFIG. 2. Instead, the glimpse generation layer may be trained in the process of the policy gradient algorithm described above.

In some embodiments, the state x∈χ of the environment is an input given to a glimpse module (G-layer) inFIG. 2. The glimpse module250indicates a new glimpse region Gtat time step t. During training, multiple episodes are performed from the proposal bounding boxes in steps T. Each of all episodes starts from the identical proposal bounding box provided by the proposal generator10.

Gaussian noise is added to the current glimpse region G computed by the glimpse generation layer. In each episode, the third subnetwork300outputs class probabilities and object bounding boxes in the last time step. From each of the outputs, the neural network400computes a total reinforcement reward R(ξ)=Σt=1Trt, where each reward rtfor an episode ξ is expressed as follows:

rt={P⁡(c*)⨯IoU⁡(Bc*,Bc**)(t=T)0(otherwise)(3)
where P(c*) is the predicted probability of the true class c* and IoU is the intersection over union between a predicted bounding box for c* and a corresponding ground-truth bounding box corresponding to the predicted bounding box. Intuitively, if the glimpse bounding box after adding a Gaussian noise leads a higher class probability and a larger IoU, then a higher return is assigned to the corresponding episode. The REINFORCE algorithm updates the model such that the generated glimpses lead to higher returns.

In the following descriptions, a mean average precisions (mAP) is used as a parameter for evaluating a performance of object detections.

Some embodiments are based on recognition that a number of episodes used in training for obtaining reasonable object detection performance is eight or less than eight.

FIG. 9shows the effect of a number of episodes generated from one sample in a mini-batch according to some embodiments. As can be seen inFIG. 9, although a greater number of episodes tends to lead better performance, a case of eight episodes provides a reasonable performance result. Since the computation time and cost of computation increase with the number of episodes, the number of episodes may be chosen according to a predetermined design of neural networks.

Some embodiments are based on recognition that an architecture of a stacked RNN with an Element-wise MAX operation provides a better object detection performance compared to those of an RNN with an Element-wise MAX operation module, a stacked RNN without an Element-wise MAX operation module and an RNN without an Element-wise MAX operation module.

FIG. 10shows performance results obtained by four different architecture setups according to some embodiments. As is seen, the stacked RNN with the Element-wise max provides better detection performance than other three architectures: an RNN with an Element-wise MAX operation module, a stacked RNN without an element-wise MAX operation module and an RNN without an Element-wise MAX operation module.

FIG. 11shows demonstrate results indicating that the continuous reward operation is superior to the discrete reward operation according to some embodiments. Some embodiments are based on recognition that a continuous reward operation leads better performance than a discrete reward operation.

In each episode, the reinforcement reward rtis obtained. There are two reward operations to determine a value of the reinforcement reward. One is called the discrete reward operation, in which a reward is set to be rt=1 if the highest scoring class is a ground-truth label and otherwise the reward is set to be rt=0. In this case, there is no medium value between 0 and 1.

On the other hand, another reward operation is called the continuous reward operation, in which the reward ranges from 0 to 1. In this case, the reward is set according to equation (3) if the highest scoring class is the ground-truth label and an IoU obtained between a predicted bounding box and the ground-truth bounding box is greater than or equal to a predetermined IoU threshold value, and otherwise the reward is set to be rt=0.

Some embodiments are based on recognition that excluding background samples yields a better performance.

FIG. 12shows that a comparison between cases of including back-ground samples in the REINFORCE algorithm and excluding back-ground samples from the REINFORCE algorithm. As can be seen inFIG. 12, the case of excluding background samples yields a better performance. In this case, IOU in equation (3) is set to be unity (IOU=1) for the background samples because there is no ground-truth bounding boxes for the background samples.

FIG. 13shows results indicating that the target vector having a four-dimension vector leads better performances than the glimpse having a two-dimension vector according to some embodiments.

Some embodiments are based on recognition that a target vector having a four-dimension vector including x-shifting, y-shifting, x-scaling and y-scaling provides better performance than a glimpse having a two-dimension vector including x-shifting and y-shifting.