Hierarchical system for detecting object with parallel architecture and hierarchical method thereof

A hierarchical system for detecting an object with parallel architecture and a hierarchical method thereof is disclosed. The system includes at least one image-retrieving device retrieving at least an image and searching a plurality of obstacle position images in it. The image-retrieving device is electrically connected with an image-processing device to receive the obstacle position images transmitted by the image-retrieving device, uses parallel architecture classification to obtain at least one object image and a plurality of cropping frames thereof from the obstacle position images, synchronously separates the cropping frames to retrieve characteristic values of each cropping frame, uses convolutional neural network to simultaneously recognize the characteristic values of each cropping frame, and searches and outputs the correct cropping frame from the object image, thereby immediately detecting the object outside a vehicle and obtaining the cropping frame of the object to avoid detection error.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to a system for detecting an object outside a vehicle and a method thereof, particularly to a hierarchical system for detecting an object with parallel architecture and a hierarchical method thereof.

Description of the Related Art

With the improvement of driving safety technology, more technologies for detecting obstacles outside a vehicle are invented, such as pedestrian detection, obstacle detection etc. The technologies can apply to detection devices to detect various environments, and the detection devices are combined with a collision avoidance system, whereby a user immediately avoids collision.

The present pedestrian detection or obstacle detection uses adaptive object classification with parallel architecture to apply to various road scenes, and classifies characteristics of obstacles outside a vehicle by using histogram of oriented gradient (HOG) and support vector machine (SVM), so as to immediately choose obstacle position images and cropping frames thereof from image data. However, the adaptive object classification sometimes makes misjudgments on cropping obstacles. For example, the cropping frames of obstacles are misjudged, so that the collision avoidance system can not immediately recognize them. Alternatively, distance estimation is misjudged. The misjudgment rate maybe not high, but users still concerns about the misjudgments on driving safety.

In addition, a conventional technology uses convolutional neural network (CNN) to extract features of a whole image outside a vehicle. Generally, the structure of CNN includes two layers one is a feature extraction layer, the input of each neuron is connected to the local receptive fields of the previous layer, and extracts the local feature. Once the local features are extracted, the positional relationship between it and other features also will be determined. The other is a feature map layer, and each computing layer of the network is composed of a plurality of feature map. Every feature map is a plane, the weight of the neurons in the plane are equal. Besides, since the neurons in the same mapping plane share weight, the number of free parameters of the network is reduced. Each convolutional layer in the convolutional neural network is followed by a computing layer which is used to calculate the local average and the second extraction, the two feature extraction structures reduce the resolution, thereby analyzing feature values of the inputted image outside the vehicle to determine the correctness of obstacles. Nevertheless, when the convolutional layer extracts the features of whole image outside the vehicle, a large operation amount is required. Thus, the first layer has to cost a lot of time extracting the features, and then the second layer determines them to output the correct technical features.

As a result, the conventional technology for analyzing obstacles outside the vehicle has estimation misjudgment on cropping frames. Alternatively, the conventional technology has to cost a lot of estimation time achieving precise determination rather than immediate determination. To overcome the abovementioned problems, the present invention provides a hierarchical system for detecting an object with parallel architecture and a hierarchical method thereof.

SUMMARY OF THE INVENTION

A primary objective of the present invention is to provide a hierarchical system for detecting an object with parallel architecture and a hierarchical method thereof, which uses a hierarchical analysis of parallel architecture classification and convolutional neural network to improve the efficiency of processing images and greatly decrease the time of processing images. Relatively, the response time of a collision avoidance system is increased, so that the rates of image misjudgment and failure are decreased.

Another objective of the present invention is to provide a hierarchical system for detecting an object with parallel architecture and a hierarchical method thereof, which applies to a collision avoidance system to warn a driver before collision, thereby avoiding rear-end collision accidents, head-on collision accidents and road-related accidents, achieving immediate determination and completely ensuring the safety of a vehicle, a driver and pedestrians.

To achieve the abovementioned objectives, the present invention provides a hierarchical method for detecting object with parallel architecture, which comprises: retrieving at least an image; searching a plurality of obstacle position images of the image; using parallel architecture classification to obtain at least one object image and a plurality of cropping frames thereof from the plurality of obstacle position images; synchronously separating the plurality of cropping frames and retrieving characteristic values of each cropping frame; and using convolutional neural network to simultaneously recognize the characteristic values of each cropping frame, and searching and outputting the correct cropping frame from the object image.

The present invention also provides a hierarchical system for detecting object with parallel architecture, which comprises: at least one image-retrieving device retrieving at least an image and searching a plurality of obstacle position images of the image; and an image-processing device electrically connected with the image-retrieving device, receiving the plurality of obstacle position images transmitted by the image-retrieving device, obtaining at least one object image and a plurality of cropping frames thereof from the plurality of obstacle position images, synchronously separating the plurality of cropping frames to retrieve characteristic values of each cropping frame, simultaneously recognizing the characteristic values of each cropping frame, and searching and outputting the correct cropping frame from the object image.

The plurality of obstacle position images of the image are searched in a sliding window way. An image calculator and an image algorithm are used to crop, calculate and classify characteristic data of the plurality of obstacle position images in a parallel way. Parallel classification is used to classify the characteristic data of the plurality of obstacle position images.

Besides, the present invention uses a convolutional way of a second layer of the convolutional neural network to synchronously separate the plurality of cropping frames and retrieve the characteristic values of each cropping frame, and uses a neural network of the second layer of the convolutional neural network to recognize the characteristic values of each cropping frame.

The step of using the parallel architecture classification to obtain the object image and the plurality of cropping frames thereof is performed by an image calculator. A complexity classifier is electrically connected with the image calculator, receives the object image and the plurality of cropping frames thereof transmitted by the image calculator, and uses the convolutional way of the second layer of convolutional neural network to perform the subsequent process.

DETAILED DESCRIPTION OF THE INVENTION

The present invention combines with an autonomous emergency braking system, applies to an image system for detecting obstacles of a vehicle, such as an autonomous driving assistant system (ADS) or a parking collision avoidance system (PCAS). A hierarchical system for detecting an object with parallel architecture and a hierarchical method thereof of the present invention can decrease calculation complexity and increase precision.

Refer toFIG. 1. A hierarchical system10for detecting object with parallel architecture comprises at least one image-retrieving device12and an image-processing device14. The image-processing device14is electrically connected with the image-retrieving device12and a display16. The embodiment exemplifies one image-retrieving device12. The image-retrieving device12is a camera having a charge coupled device (CCD) and a digital signal processor (DSP). The image-processing device14further comprises an image calculator142and a complexity classifier144. The image calculator142is electrically connected with the complexity classifier144. The image-processing device14is a microcomputer on-board unit (OBU). The image calculator142can perform HOG and SVM. The complexity classifier144is a convolutional neural network classifier.

Continuing from the abovementioned description, the image-retrieving device12retrieves at least an image122from an environment outside a vehicle. The embodiment exemplifies an image122. After retrieving the image122, the image-retrieving device12searches a plurality of obstacle position images124of the image122, and then transmits the plurality of obstacle position images124to the image-processing device14, whereby the image-processing device14processes the plurality of obstacle position images124. The image calculator142uses parallel architecture classification to obtain at least one object image146and a plurality of cropping frames cropping it. The embodiment exemplifies one object image146. The image calculator142transmits the object image146and the plurality of cropping frames thereof to the complexity classifier144. The complexity classifier144uses a convolutional way of a second layer of convolutional neural network to synchronously separate the plurality of cropping frames and retrieve characteristic values of each cropping frame. Then, the complexity classifier144of the image-processing device14recognizes the characteristic values of each cropping frame, and immediately searches the correct cropping frame from the object image146. The image-processing device14outputs the correct object image146and the correct cropping frame thereof to the display16to display them.

In order to further understand how to use the hierarchical method for detecting object with parallel architecture to achieve immediate detection and reduce the rate of detection failure, refer toFIG. 1,FIG. 2andFIGS. 3a-3c. Firstly, in Step S10, the image-retrieving device12is used to retrieve at least an image122. The embodiment exemplifies an image122. Then, in Step S12, the image-retrieving device12recognizes a scene of the image122, such as sunny day, rainy day or night. Then, according to different scenes, the image-retrieving device12searches a plurality of obstacle position images124from a region of interest (ROI) of the image122in a sliding window way, such as human body images124aand124b, a vehicle image124cand a street lamp image124dinFIG. 3a. The embodiment does not limit the scene or the ROI range of the image122. The ROI mainly ranges over lower and central regions of a picture of the image122and is adjusted according to requirement of a user. In Step S14, the image-retrieving device12transmits the plurality of obstacle position images124to the image-processing device14. The image calculator142uses an image algorithm to crop, calculate and classify the plurality of obstacle position images124in a parallel way, and uses parallel classification to classify characteristic data of the plurality of obstacle position images124into the at least one object image146. After finding the object image146, the object image146is surrounded by cropping frames148, whereby the image-processing device14observes a specific position of the object image146. Parallel architecture classification formed by the image algorithm and the parallel classification easily generates a plurality of cropping frames148to surround the object image146due to classification standard. In the embodiment, the image algorithm is HOG, and the parallel classification is SVM. The amount of the object image146is adaptable according to requirement of a user. The embodiment exemplifies pedestrians, and thus two object images146are found, namely the human body images124aand124b. Since the image algorithm and the parallel classification may cause an error, the plurality of cropping frames148are easily generated to surround the two object images146when cropping the two object images146, as shown inFIG. 3b.

Then, refer toFIG. 1,FIG. 2andFIGS. 3a-3c. Although the object image146and the plurality of cropping frames thereof148are obtained, the image-processing device14cannot know the correct cropping frame of the object image146. The confused cropping frames will cause problems during recognition. Thus, the image calculator142transmits the object image146and the plurality of cropping frames to the complexity classifier144. Then, in Step S16, the complexity classifier144uses a convolutional way of a second layer of convolutional neural network to split the plurality of cropping frames148surrounding the object image146, thereby synchronously separating the plurality of cropping frames148and retrieving characteristic values of each cropping frame. In the embodiment, the second layer of convolutional neural network indicates the second stage of convolutional neural network, which is deep learning that can perform more complicated feature extract and feature map. The first layer of convolutional neural network indicates the first stage of convolutional neural network, which is shallow learning that can perform feature extract and feature map. The present invention uses the second layer to perform analysis of the second stage rather than the first layer. Then, in Step S18, the complexity classifier144uses a neural network of the second layer of the convolutional neural network to recognizes the characteristic values of each cropping frame148. For example, a user programs to recognize pedestrians, and thus the complexity classifier144sets human body-related parameters, so as to use convolutional neural network to recognize the characteristic values retrieved by the convolutional way. In this step, the complexity classifier144recognizes the characteristic values of each cropping frame148of the two object images146such as the human body images124aand124b, so as to search the correct cropping frames148a. The cropping frames148aeffectively choose the human body images124aand124b. The complexity classifier144can output the correct cropping frames148ato the display16. The display16can display the correct object images146and the cropping frames148ato warn the user. FromFIG. 3c, bottom lines L of the cropping frames148asurrounding the object images146are clearly known. The bottom line L is used to calculate a distance between either of the human body images124aor124band a vehicle driven by the user, establish an efficient collision avoidance system, and effectively dodge pedestrians in automatic driving lest an estimation error occur.

Compared with classification of convolutional neural network of a conventional technology, the hierarchical system for detecting object with parallel architecture and the hierarchical method thereof of the present invention can at least save four-fifths of recognition time. The classification of convolutional neural network of the conventional technology uses simple neural network of the first layer to determine the correctness of obstacle images. Then, the second layer is used to verify whether the results of the first layer are correct. However, when the classification of convolutional neural network of the conventional technology is used, the whole image characteristics have to be retrieved. Since the operation amount is quite large, it costs a lot of time retrieving the whole image characteristics. The present invention replaces analysis of the first layer with the parallel architecture classification, so as to save more time.

Continuing from the abovementioned paragraph, the present invention only occupies one-tenths of processing time of an obstacle recognition using HOG and parallel SVM. Compared with the obstacle recognition, the present invention greatly improves the precision of a cropping frame. For example, under a strict recognition condition, the conventional obstacle recognition may omit to crop a pedestrian to cause a serious error in recognizing the pedestrian. Alternatively, in order to avoid omitting to recognize the pedestrian, a recognition condition is relaxed. Under a loose recognition condition, the conventional obstacle recognition may generate too many cropping frames to surround the pedestrian, thereby causing an error in recognizing the pedestrian and problems with performing an autonomous driving assistant system or a collision avoidance system. For example, a practical distance between a vehicle and a pedestrian cannot be effectively calculated, so as to affect the safety of a driver and pedestrians. The hierarchical system for detecting object with parallel architecture and the hierarchical method thereof of the present invention not only decreases determination time but also achieves precise determination. Thus, an autonomous driving assistant system or a parking collision avoidance system can exactly control a vehicle to ensure safety of the driver and pedestrians.

However, pedestrians are recognized in the foregoing embodiment, but the present invention is not limited thereto. The user himself can decide to use convolutional neural network to recognize what kinds of obstacles and set parameters thereof, thereby using an autonomous driving assistant system or a collision avoidance system to set obstacles recognized. After the obstacle used as an object is found, convolutional neural network is used to determine the characteristic values of cropping frames surrounding the object, thereby finding the correct cropping frame. As a result, the autonomous driving assistant system or collision avoidance system can effectively control a distance between the vehicle and the obstacle, decrease recognition and increase response time of the user.