Apparatus, method, and computer program for human detection

An apparatus for human detection includes a processor configured to detect human regions from an image, integrate two or more human regions overlapping more than a predetermined degree into a single integrated human region among the detected human regions, input the integrated human region into a skeleton detector to determine a skeleton of an undetermined and foremost person out of one or more persons included in the integrated human region, mask a region representing the detected foremost person in the integrated human region based on the skeleton of the foremost person to modify the integrated human region, repeat detection of a skeleton and masking of a region representing a person in the modified integrated human region until a human skeleton is no longer detected, and determine the number of skeletons detected from the integrated human region as the number of persons represented in the integrated human region.

FIELD

The present invention relates to an apparatus, a method, and a computer program for detecting a human represented in an image.

BACKGROUND

Techniques for detecting a human represented in an image obtained by a camera have been researched (see Japanese Unexamined Patent Publications JP2020-98474A, JP2018-22340A, JP2016-95808A, and JP2015-222881A).

A device for determining attributes disclosed in JP2020-98474A detects the positions of control points that define a human skeleton from an image of a person taken from above, and recognizes his/her attribute from the result of detection.

An image processor disclosed in JP2018-22340A executes a process for detecting specific objects on a first region in an image to estimate the number of specific objects in the first region. The image processor further executes a regression process, which is a process for estimating the number of specific objects for each predetermined region, on a second region in the image to estimate the number of specific objects in the second region. The image processor then integrates the results of estimation.

A human detector disclosed in JP2016-95808A identifies whether sub-regions extracted from an image include a predetermined object, and determines whether to integrate the results of identification of overlapping sub-regions identified as those including a predetermined object, based on the distances in the depth direction of the sub-regions.

A monitoring device disclosed in JP2015-222881A detects a human from captured moving images to obtain positional information on a human region, and determines its regional state, which indicates the state of the person in the human region, based on the positional information. The monitoring device then sets a mask image corresponding to the regional state, and generates and outputs moving images in which the human region is changed to the mask image corresponding to the regional state. Additionally, the monitoring device obtains positional information on each person in each frame, and, when it fails to detect a human in a frame where persons overlap, obtains the positional information regarding this frame from the positional information obtained in immediately preceding frames.

SUMMARY

These techniques may fail to detect individual persons to be detected when they are represented on top of another in an image.

It is an object of the present invention to provide an apparatus for human detection that can detect individual persons even if they overlap in an image.

According to an embodiment, an apparatus for human detection is provided. The apparatus includes a processor configured to: detect human regions representing a human from an image generated by a camera, set a single integrated human region by selecting one of two or more human regions whose degree of overlap is not less than a predetermined threshold among the detected human regions or by defining a region including the two or more human regions, input the integrated human region into a skeleton detector to detect a skeleton of an undetermined and foremost person out of one or more persons included in the integrated human region, the skeleton detector having been trained to detect a human skeleton, mask a region representing the foremost person in the integrated human region based on the skeleton of the foremost person to modify the integrated human region, repeat detection of a skeleton and masking of a region representing a person in the modified integrated human region until a human skeleton is no longer detected, and determine the number of skeletons detected from the integrated human region as the number of persons represented in the integrated human region.

In the apparatus, the processor is preferably further configured to track persons detected from each of time-series past images generated by the camera earlier than the image to determine an estimated overlap region in the image assumed to represent two or more persons. In this case, when the two or more human regions whose degree of overlap is not less than the threshold are outside the estimated overlap region, the processor preferably determines that the two or more human regions represent the same person.

Additionally, when the two or more human regions whose degree of overlap is not less than the threshold are outside the estimated overlap region, the processor preferably selects one of the two or more human regions and deletes the other human regions. When the two or more human regions whose degree of overlap is not less than the threshold are inside the estimated overlap region, the processor preferably sets the integrated human region so as to include the union of the two or more human regions therein.

In addition, in the apparatus, the processor is preferably further configured to track persons detected from each of time-series past images generated by the camera earlier than the image to determine an estimated overlap region in the image assumed to represent two or more persons. In this case, the processor preferably sets the threshold for the case that the two or more human regions are inside the estimated overlap region lower than the threshold for the case that the two or more human regions are outside the estimated overlap region.

According to another embodiment, a method for human detection is provided. The method includes detecting human regions representing a human from an image generated by a camera, setting a single integrated human region by selecting one of two or more human regions whose degree of overlap is not less than a predetermined threshold among the detected human regions or by defining a region including the two or more human regions, and inputting the integrated human region into a skeleton detector to detect a skeleton of an undetermined and foremost person out of one or more persons included in the integrated human region, the skeleton detector having been trained to detect a human skeleton, masking a region representing the foremost person in the integrated human region based on the skeleton of the foremost person to modify the integrated human region; repeating detection of a skeleton and masking of a region representing a person in the modified integrated human region until a human skeleton is no longer detected; and determining the number of skeletons detected from the integrated human region as the number of persons represented in the integrated human region.

According to still another embodiment, a non-transitory recording medium that stores a computer program for human detection is provided. The computer program includes instructions causing a computer to execute a process including detecting human regions representing a human from an image generated by a camera, setting a single integrated human region by selecting one of two or more human regions whose degree of overlap is not less than a predetermined threshold among the detected human regions or by defining a region including the two or more human regions, and inputting the integrated human region into a skeleton detector to detect a skeleton of an undetermined and foremost person out of one or more persons included in the integrated human region, the skeleton detector having been trained to detect a human skeleton, masking a region representing the foremost person in the integrated human region based on the skeleton of the foremost person to modify the integrated human region; repeating detection of a skeleton and masking of a region representing a person in the modified integrated human region until a human skeleton is no longer detected; and determining the number of skeletons detected from the integrated human region as the number of persons represented in the integrated human region.

The apparatus according to the present invention has an advantageous effect of being able to detect individual persons even if they overlap in an image.

DESCRIPTION OF EMBODIMENTS

An apparatus for human detection as well as a method and a computer program therefor executed by the apparatus will now be described with reference to the attached drawings. The apparatus inputs an image into a classifier that has been trained to detect a human represented in an image, thereby detecting a region including a human (hereafter, a “human region”) in the image. When two or more human regions overlapping more than a predetermined degree are detected, the apparatus integrates these human regions into a single integrated human region. The apparatus then inputs the integrated human region into a skeleton detector that has been trained to detect a human skeleton, thereby determining a skeleton of an undetermined and foremost person out of one or more persons included in the integrated human region. Additionally, the apparatus masks pixels representing the foremost person in the integrated human region, based on the skeleton of the foremost person, and inputs the masked integrated human region into the skeleton detector again. After that, the apparatus repeats the skeleton detection process and the masking process until a human skeleton is no longer detected from the integrated human region. This enables the apparatus to detect individual persons even if they overlap in the image.

The following describes an example in which the apparatus is applied to a vehicle control system. In this example, the apparatus executes a human detection process on time-series images obtained by a camera mounted on a vehicle to detect detection targets around the vehicle. The detection targets include objects that affect travel of the vehicle10, such as other traveling vehicles around the vehicle10, humans, signposts, traffic lights, road markings including lane-dividing lines, and other objects on roads.

FIG.1schematically illustrates the configuration of a vehicle control system equipped with the apparatus for human detection.FIG.2illustrates the hardware configuration of an electronic control unit, which is an embodiment of the apparatus for human detection. In the present embodiment, a vehicle control system1, which is mounted on a vehicle10and controls the vehicle10, includes a camera2for capturing the surroundings of the vehicle10and an electronic control unit (ECU)3, which is an example of the apparatus for human detection. The camera2is connected to the ECU3via an in-vehicle network conforming to a standard, such as a controller area network, so that they can communicate with each other. The vehicle control system1may further include a storage device storing map information that represents lane-dividing lines and the positions and types of road features and that is used for autonomous driving control of the vehicle10. The vehicle control system1may further include a range sensor, such as LiDAR or radar; a GPS receiver or another receiver for determining the position of the vehicle10in conformity with a satellite positioning system; a wireless terminal for wireless communication with another device; and a navigation device for searching for a planned travel route of the vehicle10.

The camera2, which is an example of the image capturing unit, includes a two-dimensional detector constructed from an array of optoelectronic transducers, such as CCD or C-MOS, having sensitivity to visible light and a focusing optical system that forms an image of a target region on the two-dimensional detector. The camera2is mounted, for example, in the interior of the vehicle10so as to be oriented to the front of the vehicle10. The camera2captures a region in front of the vehicle10every predetermined capturing period (e.g., 1/30 to 1/10 seconds), and generates images representing this region. The images obtained by the camera2are preferably color images. The vehicle10may include multiple cameras taking pictures in different orientations or having different focal lengths.

Whenever generating an image, the camera2outputs the generated image to the ECU3via the in-vehicle network.

The ECU3controls the vehicle10. In the present embodiment, the ECU3controls the vehicle10to automatically drive it, based on objects detected from time-series images obtained by the camera2. To achieve this, the ECU3includes a communication interface21, a memory22, and a processor23.

The communication interface21, which is an example of a communication unit, includes an interface circuit for connecting the ECU3to the in-vehicle network. In other words, the communication interface21is connected to the camera2via the in-vehicle network. Whenever receiving an image from the camera2, the communication interface21passes the received image to the processor23.

The memory22, which is an example of a storage unit, includes, for example, volatile and nonvolatile semiconductor memories. The memory22stores various types of data and various parameters used in a vehicle control process, which includes a human detection process, executed by the processor23of the ECU3. Specifically, the memory22stores, for example, images received from the camera2, various parameters for defining a classifier used in the human detection process, various parameters for defining the skeleton detector, and an overlap threshold for determining integration of human regions. The memory22also stores various types of data generated in the vehicle control process for a certain period. The memory22may further store information used for travel control of the vehicle10, such as map information.

The processor23, which is an example of a control unit, includes one or more central processing units (CPUs) and a peripheral circuit thereof. The processor23may further include another operating circuit, such as a logic-arithmetic unit, an arithmetic unit, or a graphics processing unit (GPU). Whenever receiving an image from the camera2during travel of the vehicle10, the processor23executes the vehicle control process including the human detection process on the received image. The processor23then controls the vehicle10to automatically drive it, based on detected objects around the vehicle10.

FIG.3is a functional block diagram of the processor23of the ECU3, related to the vehicle control process including the human detection process. The processor23includes a detection unit31, an overlap estimation unit32, an integration unit33, a skeleton detection unit34, a masking unit35, a repetition control unit36, a driving planning unit37, and a vehicle control unit38. These units included in the processor23are, for example, functional modules implemented by a computer program executed on the processor23, or may be dedicated operating circuits provided on the processor23. Of these units in the processor23, the detection unit31, the overlap estimation unit32, the integration unit33, the skeleton detection unit34, the masking unit35, and the repetition control unit36are included in the human detection process. In the case that the vehicle10includes multiple cameras, the processor23may execute the human detection process for each camera, based on images obtained by the camera.

Whenever receiving an image from the camera2, the detection unit31inputs the latest received image into a classifier for object detection, and thereby detects an object region including a detection target represented in the image and identifies the type of the detection target. In the present embodiment, examples of the detection target include humans. Examples of the detection target may include objects that may affect travel of the vehicle10, such as other vehicles, traffic lights, road markings, and signposts. An object region including a human is referred to as a human region, as mentioned above.

As the classifier, the detection unit31uses a deep neural network (DNN) that has been trained to detect an object region including a detection target represented in an image, identify the type of the detection target. The DNN used by the detection unit31may be, for example, a DNN having a convolutional neural network (hereafter simply “CNN”) architecture, such as Single Shot MultiBox Detector (SSD) or Faster R-CNN. In this case, the classifier is trained in advance in accordance with a training technique, such as backpropagation, with a large number of training images each representing one of various types of detection targets.

Alternatively, the detection unit31may use, as the classifier, a classifier based on a machine learning technique other than a neural network, such as a support vector machine or AdaBoost. In this case, the classifier is trained in advance with a large number of training images like those mentioned above in accordance with a training technique depending on the applied machine learning technique. When such a classifier is used, the detection unit31sets windows of various sizes or aspect ratios at various positions in the image. For each window, the detection unit31calculates a feature to be inputted into the classifier (e.g., Haar-like feature or HOG feature) from the window, and inputs the calculated feature into the classifier to determine whether a detection target is represented in the window. The detection unit31then determines a window determined to represent a certain type of detection target, as an object region. In particular, the detection unit31determines a window determined to represent a human, as a human region. Multiple classifiers may be prepared for respective types of detection targets.

The detection unit31enters the positions and areas of the detected object regions in the image as well as the types of the objects included in the respective object regions, in a detected-object list. The detection unit31stores the detected-object list in the memory22.

The overlap estimation unit32tracks persons detected by the detection unit31from each of time-series past images obtained by the camera2earlier than the latest image to determine an estimated overlap region in the latest image assumed to represent two or more persons.

For example, the overlap estimation unit32applies a tracking process based on optical flow, such as the Lucas-Kanade method, to individual human regions in time-series past images obtained by the camera2, thereby tracking persons represented in the human regions. To this end, the overlap estimation unit32applies, for example, a filter for extracting characteristic points, such as SIFT or Harris operator, to a human region of interest in the latest past image, thereby extracting characteristic points from this human region. The overlap estimation unit32then identifies those points in the human regions in the preceding past images which correspond to each of the characteristic points in accordance with the applied tracking technique, thereby calculating the optical flow. Alternatively, the overlap estimation unit32may apply another tracking technique applied for tracking a moving object detected from an image to individual human regions in time-series past images, thereby tracking persons represented in the human regions.

For each person being tracked, the overlap estimation unit32uses the result of tracking through the time-series past images to predict the position and area of the human region representing the person in the latest image. Specifically, the overlap estimation unit32executes a prediction process with, for example, a Kalman filter or a particle filter, on the human regions of a person of interest in the time-series past images to predict the position and area of the human region of this person in the latest image. Alternatively, the overlap estimation unit32may extrapolate from the changing positions and areas of the human regions of a person of interest in the time-series past images to predict the position and area of the human region of this person in the latest image.

When the predicted positions and areas of human regions of two or more tracked persons in the latest image at least overlap, the overlap estimation unit32sets an estimated overlap region so as to include the two or more overlapping estimated human regions therein. For example, the overlap estimation unit32determines a circumscribed rectangular region of the union of the two or more overlapping estimated human regions or a region made by extending the circumscribed rectangular region horizontally or vertically by a predetermined number of pixels as an estimated overlap region. The overlap estimation unit32then notifies the position and area of the estimated overlap region to the integration unit33.

The integration unit33integrates two or more human regions overlapping more than a predetermined degree in the latest image obtained by the camera2into a single integrated human region. In the present embodiment, the integration unit33executes a non-maximum suppression (NMS) process on two or more overlapping human regions to determine whether to integrate the two or more human regions into a single integrated human region.

Specifically, the integration unit33calculates the degree of overlap between the two or more overlapping human regions, and compares the calculated degree of overlap with an overlap threshold. As the degree of overlap, the integration unit33calculates, for example, the ratio of the area of the overlapping region to that of the whole set of the two or more overlapping human regions, or the intersection over union (IoU). Alternatively, as the degree of overlap, the integration unit33may calculate the ratio of the area of the overlapping region to that of the largest of the two or more overlapping human regions. When the degree of overlap is not less than the overlap threshold, the integration unit33determines to integrate the two or more human regions into a single integrated human region. In contrast, when the degree of overlap is less than the overlap threshold, the integration unit33determines that the two or more human regions represent different persons, and does not integrate these human regions.

The integration unit33selects a human region (e.g., the largest object region) from among the two or more human regions determined to be integrated into a single integrated human region as the integrated human region.

The integration unit33may set the overlap threshold for the case that the two or more overlapping human regions are inside the estimated overlap region lower than the overlap threshold for the case that these human regions are not inside the estimated overlap region. This results in a relatively large human region being likely to be selected as an integrated human region in an estimated overlap region assumed to represent two or more persons. This increases the possibility that the integrated human region includes two or more persons, enabling the integration unit33to prevent failure in detecting some of persons represented on top of another in the image.

According to a modified example, the integration unit33may determine the union of the two or more human regions determined to be integrated into a single integrated human region or a circumscribed rectangular region of this union as the integrated human region. Alternatively, when the two or more human regions determined to be integrated into a single integrated human region are inside the estimated overlap region, the integration unit33determines the union of these human regions or a circumscribed rectangular region of this union as the integrated human region. In contrast, when the two or more human regions are outside the estimated overlap region, the integration unit33may determine one of the two or more human regions as the integrated human region.

FIG.4illustrates an example of the integrated human region. In an image400illustrated on the left ofFIG.4, two overlapping human regions411and412are detected. In this example, the degree of overlap between the human regions411and412is not less than the overlap threshold; thus, a single integrated human region413, which is a circumscribed rectangular region of the union of the human regions411and412, is set, as illustrated in an image401on the right.

The integration unit33notifies the position and area of the integrated human region to the skeleton detection unit34.

The skeleton detection unit34cuts out the integrated human region from the latest image obtained by the camera2. The skeleton detection unit34then inputs the cutout integrated human region into a skeleton detector that has been trained to detect a human skeleton, thereby detecting a skeleton of an undetermined and foremost person out of one or more persons included in the integrated human region. Whenever a skeleton is detected, the skeleton detection unit34increments the number of skeletons detected from the integrated human region by one.

As such a skeleton detector, the skeleton detection unit34can use a DNN having a CNN architecture. The skeleton detector is trained in advance in accordance with a training technique, such as backpropagation, with a large number of training images representing human skeletons. The skeleton detector outputs the positions of reference points for determining a skeleton, such as the head, neck, shoulders, elbows, hands, hip joints, knees, and feet, and skeletal lines representing the connection relationship between the reference points according to the structure of a human body. Specifically, for each pixel in the integrated human region, the skeleton detector calculates confidence scores indicating how likely respective types of reference points (e.g., the head and neck) are represented, and outputs a position at which one of the confidence scores is not less than a predetermined threshold as a reference point of the corresponding type. The skeleton detector then selects each type of reference point from among the detected reference points so as to maximize the degree of reliability of the selected reference points connected according to the structure of a human body, and thereby can detect the skeleton of the foremost one of undetermined persons from the integrated human region.

The skeleton detection unit34notifies the repetition control unit36of the result of determination whether a skeleton is detected. When a skeleton is detected, the skeleton detection unit34further notifies the masking unit35of skeleton information indicating the positions of the reference points and the skeletal lines of the detected skeleton.

The masking unit35modifies the integrated human region by masking a region representing the newly detected foremost person in the integrated human region, based on the skeleton information received from the skeleton detection unit34, i.e., the skeletal lines of the foremost person. In the present embodiment, the masking unit35determines a region not farther than a predetermined distance from a skeletal line as a mask region. Since the trunk of a human is thicker than the arms and the legs, the masking unit35may set the predetermined distance from a skeletal line corresponding to the trunk greater than that from a skeletal line corresponding to the arm or the leg. Additionally, since a smaller integrated human region is assumed to represent a human farther from the camera2, the masking unit35may decrease the predetermined distance from a skeletal line with the size of the integrated human region. Alternatively, the masking unit35may decrease the predetermined distance from a skeletal line with the distance between the reference points corresponding to the head and the foot.

The masking unit35masks the mask region by substituting the values of pixels included in the mask region with a constant value or with values such that the mask region will be a predetermined pattern or random noise. This results in no reference point of a skeleton being detected from the mask region. The masking unit35passes the integrated human region modified by masking the mask region to the skeleton detection unit34.

The repetition control unit36causes the skeleton detection unit34and the masking unit35to repeat processes until a human skeleton is no longer detected from the integrated human region. In the present embodiment, when receiving, from the skeleton detection unit34, the result of determination indicating that a skeleton is detected, the repetition control unit36causes the masking unit35to mask a region representing the detected person in the integrated human region, thereby modifying the integrated human region. The repetition control unit36then causes the skeleton detection unit34to execute the skeleton detection process again on the modified integrated human region outputted from the masking unit35. In contrast, when receiving, from the skeleton detection unit34, the result of determination indicating that no skeleton is detected, the repetition control unit36determines the number of skeletons detected from the integrated human region and counted by the skeleton detection unit34up to this time as the number of persons included in the integrated human region. The repetition control unit36then associates this number of persons represented in the integrated human region with those, of the persons being tracked, in human regions in the latest image whose estimated positions at least overlap the integrated human region, enabling continuous tracking of these persons.

FIG.5illustrates an example of the skeleton detection process and the masking process. The first execution of the skeleton detection process by the skeleton detection unit34on an integrated human region510in the uppermost image500leads to detection of a skeleton521of a first person, as illustrated in the second image501from the top. Then, execution of the masking process by the masking unit35yields an integrated human region511in which a mask region522centered at the skeleton521is masked, as illustrated the third image502from the top. The second execution of the skeleton detection process on the integrated human region511leads to detection of a skeleton523of a second person from the integrated human region511, as illustrated in the fourth image503from the top. In this way, alternate execution of the skeleton detection process and the masking process leads to sequential detection of all the human skeletons included in the integrated human region.

FIG.6is an operation flowchart of the human detection process executed by the processor23. Whenever receiving an image from the camera2, the processor23executes the human detection process in accordance with the operation flowchart illustrated inFIG.6.

The detection unit31of the processor23inputs an image obtained from the camera2into a classifier to detect one or more persons represented in the image. In other words, the detection unit31detects one or more human regions including a human in the image (step S101).

The overlap estimation unit32of the processor23sets an estimated overlap region so as to include two or more predicted human regions at least overlapping in the latest image, based on human regions of individual tracked persons in time-series past images (step S102).

The integration unit33of the processor23sets the overlap threshold applied to the estimated overlap region lower than the overlap threshold applied to the outside of the estimated overlap region (step S103). The integration unit33then integrates two or more overlapping human regions whose degree of overlap is not less than the overlap threshold into a single integrated human region (step S104).

The skeleton detection unit34of the processor23inputs the integrated human region into a skeleton detector to detect a skeleton of an undetected and foremost person (step S105).

The repetition control unit36of the processor23determines whether a skeleton is detected in step S105(step S106). When a skeleton is detected (Yes in step S106), the repetition control unit36causes the masking unit35of the processor23to mask a region representing the detected person in the integrated human region, based on the detected skeleton (step S107). The repetition control unit36then causes the process of step S105and the subsequent steps to be executed again on the integrated human region in which the region representing the detected person is masked.

When no skeleton is detected (No in step S106), the repetition control unit36determines the number of skeletons detected from the integrated human region up to this time as the number of persons included in the integrated human region (step S108). The processor23then terminates the human detection process.

The driving planning unit37generates one or more planned trajectories of the vehicle10in a predetermined section from the current position of the vehicle10to a predetermined distance (e.g., 500 m to 1 km) away by referring to the detected-object list so that the vehicle10will not collide with any of objects around the vehicle10. Each planned trajectory is represented, for example, as a set of target positions of the vehicle10at respective times during travel of the vehicle10through the predetermined section.

To generate a planned trajectory, the driving planning unit37tracks the detection targets entered in the detected-object list (the detection targets include a human and will be simply referred to as objects), and predicts trajectories of the tracked objects in a period from the current time until a predetermined time ahead.

For example, the driving planning unit37applies a tracking process similar to that described in relation to the overlap estimation unit32to an object region of interest in the latest image and object regions in past images obtained by the camera2, thereby tracking the object represented in these object regions.

For each object being tracked, the driving planning unit37executes viewpoint transformation, using information such as the position at which the camera2is mounted on the vehicle10, thereby transforming the image coordinates of the object into coordinates in an aerial image (“aerial-image coordinates”). To this end, the driving planning unit37can estimate the position of the detected object at the time of acquisition of each image, using the position and orientation of the vehicle10, an estimated distance to the detected object, and the direction from the vehicle10to the object at the time of acquisition of each image. The driving planning unit37can estimate the position and orientation of the vehicle10at the time of acquisition of each image, based on, for example, information indicating the current position of the vehicle10obtained from the GPS receiver (not illustrated) mounted on the vehicle10. Alternatively, whenever an image is obtained by the camera2, the driving planning unit37may detect lane-dividing lines on the right and left of the vehicle10from the image, and compare the detected lane-dividing lines with map information stored in the memory22, thereby estimating the position and orientation of the vehicle10. The driving planning unit37can identify the direction from the vehicle10to the detected object, based on the position of the object region including the object in the image and the direction of the optical axis of the camera2. Additionally, the bottom position of an object region is assumed to correspond to the position at which the object represented in this object region is in contact with the road surface. Thus the driving planning unit37can estimate the distance to the object represented in the object region, based on the direction from the camera2corresponding to the bottom of the object region and the height of the mounted position of the camera2. The driving planning unit37can estimate the predicted trajectory of the object to a predetermined time ahead by executing a prediction process with, for example, a Kalman filter or a particle filter, on time-series aerial-image coordinates in a preceding predetermined period.

The driving planning unit37sets a planned trajectory of the vehicle10, based on the predicted trajectories of the objects being tracked, so that a predicted distance between the vehicle10and any of the tracked objects will be not less than a predetermined distance until the predetermined time ahead and that the planned trajectory will follow a planned travel route to a destination. To this end, the driving planning unit37calculates the inverse of the sum of the distances to tracked objects closest to positions on the planned trajectory at respective times until the predetermined time ahead, as an evaluation function. The driving planning unit37then sets a planned trajectory in accordance with a predetermined optimization technique, such as dynamic programming or the steepest-descent method, so as to minimize the evaluation function.

The driving planning unit37may generate multiple planned trajectories. In this case, the driving planning unit37may select one of the planned trajectories such that the sum of the absolute values of acceleration of the vehicle10will be the smallest.

The driving planning unit37notifies the generated planned trajectory to the vehicle control unit38.

The vehicle control unit38controls components of the vehicle10so that the vehicle10will travel along the received planned trajectory. For example, the vehicle control unit38determines target acceleration of the vehicle10according to the received planned trajectory and the current speed of the vehicle10measured by a vehicle speed sensor (not illustrated), and sets the degree of accelerator opening or the amount of braking so that the acceleration of the vehicle10will be equal to the target acceleration. The vehicle control unit38then determines the amount of fuel injection according to the set degree of accelerator opening, and outputs a control signal depending on the amount of fuel injection to a fuel injector of the engine of the vehicle10. Alternatively, the vehicle control unit38outputs a control signal depending on the set amount of braking to the brake of the vehicle10.

When changing the direction of the vehicle10in order for the vehicle10to travel along the planned trajectory, the vehicle control unit38determines the steering angle of the vehicle10according to the planned trajectory and outputs a control signal depending on the steering angle to an actuator (not illustrated) that controls the steering wheel of the vehicle10.

FIG.7is an operation flowchart of the vehicle control process including the human detection process and executed by the processor23. Whenever receiving an image from the camera2, the processor23executes the vehicle control process in accordance with the operation flowchart illustrated inFIG.7.

The processor23detects one or more persons from an image obtained from the camera2in accordance with the operation flowchart illustrated inFIG.6(step S201). Additionally, the detection unit31of the processor23detects detection targets other than a human from the image (step S202).

The driving planning unit37of the processor23tracks the detection targets to estimate predicted trajectories of these objects. The driving planning unit37then generates a planned trajectory of the vehicle10so that it will be separated more than a predetermined distance from any of the estimated predicted trajectories (step S203). The vehicle control unit38of the processor23controls the vehicle10so that it will travel along the planned trajectory (step S204). The processor23then terminates the vehicle control process.

As has been described above, the apparatus for human detection integrates two or more of human regions detected from an image and overlapping more than a predetermined degree into a single integrated human region. The apparatus then inputs the integrated human region into a skeleton detector that has been trained to detect a human skeleton, thereby determining a skeleton of an undetermined and foremost person out of one or more persons included in the integrated human region. Additionally, the apparatus masks a region representing the foremost person in the integrated human region, based on the skeleton of the foremost person, and inputs the integrated human region, which is modified by masking, into the skeleton detector again. After that, the apparatus repeats the skeleton detection process and the masking process until a human skeleton is no longer detected from the integrated human region. This enables the apparatus to detect individual persons even if they overlap in the image.

According to a modified example, when two or more overlapping human regions whose degree of overlap is not less than the overlap threshold are outside the estimated overlap region, the integration unit33may determine that the two or more human regions represent the same person. The integration unit33may then select only one of the two or more human regions and delete the other human regions from the detected-object list. In this case, the skeleton detection unit34, the masking unit35, and the repetition control unit36may omit to execute their processes on these two or more human regions. This will result in the apparatus repeating the skeleton detection process and the masking process only for a region assumed to represent overlapping persons from the result of tracking through past images. For this reason, the apparatus can prevent a single person from being erroneously detected as multiple persons even when multiple human regions are detected for the same person. In this modified example, when two or more human regions whose degree of overlap is not less than the overlap threshold are inside the estimated overlap region, the integration unit33may set the integrated human region so as to include the union of the two or more human regions therein, as described in the embodiment. This enables the apparatus to set the integrated human region appropriately, based on whether two or more human regions whose degree of overlap is not less than the overlap threshold are inside the estimated overlap region. For this reason, the apparatus can prevent a single person from being erroneously detected as multiple persons and correctly detect multiple persons represented on top of another in an image.

The apparatus for human detection according to the embodiment or modified examples may be mounted on a device other than vehicle-mounted equipment. For example, the apparatus according to the embodiment or modified examples may be configured to detect a human from an image generated by a surveillance camera placed for taking pictures of a predetermined outdoor or indoor region at predetermined intervals. When a human is detected for a certain period, the apparatus may cause a message indicating detection of a human to appear on a display connected to the apparatus.

A computer program for achieving the functions of the units of the processor23of the apparatus for human detection according to the embodiment or modified examples may be provided in a form recorded on a computer-readable and portable medium, such as a semiconductor memory, a magnetic recording medium, or an optical recording medium.

As described above, those skilled in the art may make various modifications according to embodiments within the scope of the present invention.