Image recognizing apparatus and method

An image recognizing apparatus and method is provided for recognizing behavior of a mobile unit accurately with an image of external environment acquired during the mobile unit is moving.Behavior command output block 12 outputs behavior commands to cause the mobile unit 32 move. Local feature extraction block 16 extracts features of local areas of the image from the image of external environment acquired on the mobile unit 32 when the behavior command is output. Global feature extraction block 18 extracts feature of global area of the image using the features of local areas. Learning block 20 calculates probability models for recognizing behavior given to the mobile unit 32 based on the feature of global area of the image. After learning is finished, behavior of the mobile unit 32 may be recognized rapidly and accurately by applying the probability models to an image of external environment acquired in mobile unit 32 afresh.

TECHNICAL FIELD

The present invention relates to image recognizing apparatus and method. More specifically, the invention relates to image recognizing apparatus and method for recognizing the behavior of mobile unit using images of external environment taken when the mobile unit is moving.

BACKGROUND OF THE INVENTION

A method for detecting optical flow is well-known in the art for recognizing behavior of mobile unit using images of external environment by calculating, for example, gradient change of image density from sequence of input images.

In Japanese Patent Application Unexamined Publication (Kokai) No. 2000-171250, a method for detecting current position of a mobile unit by means of optical flow is disclosed. According to the method, optical flow of scenes of certain course are acquired every predetermined distance when the mobile unit moves along with the course beforehand. Then relationship between the optical flow and taking point is stored. After that, optical flow of the course is newly detected and undergoes matching with all of stored optical flows. The one showing maximum matching result is selected and a point associated with the selected one is recognized as current point where the mobile unit moves.

In Japanese Patent Application Unexamined Publication (Kokai) No. H11-134504, another method is disclosed. The method comprises calculating optical flow from moving images and processing the optical flow with neural network layer to recognize the behavior of the mobile unit. The method further comprises judging necessary processing based on the recognition. According to the method, detecting the approach to the obstacle is enabled with simple neural network based on moving images.

However, in order to implement the former method, many relationships between optical flow and point in a course need to be stored by moving the mobile unit in the predetermined course beforehand.

Generally, there are some problems to recognize position and behavior of the mobile unit by feature extracting based only on moving images like the latter method. For example, since relative distance between light source and a camera on the mobile unit changes continuously as the mobile unit moves, image intensity such as lightness changes so great that accurate feature extracting is difficult. In addition, since vibration of the moving mobile unit is transmitted to the camera, accuracy of feature extraction is degraded. If a recognizing system is configured to execute smoothing process over multiple frame of images to remove bad effects due to the variation of image intensity or the vibration, computing load becomes too heavy or feature extraction becomes difficult when the image moves fast due to large time variation.

It is objective of the present invention is to provide an image recognizing apparatus and method that can recognize the behavior of a mobile unit rapidly and accurately using images of external environment taken on the mobile unit in the real application.

SUMMARY OF THE INVENTION

An image recognizing apparatus according to the invention performs learning in an on-line like fashion in the real application without executing smoothing process to remove the noise effect like conventional art, and improve robustness over variation of environment by using such noise as data for feature extraction.

The image recognizing apparatus according to the invention comprises behavior command output means for outputting behavior commands to cause a mobile unit to move and local feature extraction means for extracting features of local areas within an image of external environment acquired on the mobile unit when the behavior command is output. The apparatus further comprises global feature extraction means for extracting features of global area of the image using the extracted features of the local areas and learning means for calculating probability models to recognize behavior of the mobile unit based on the extracted features of the global area.

The local feature extraction means extracts the features of the local areas within the image by utilizing image intensities obtained by applying both positive and negative component of Gabor filters on two images. Preferably, the Gabor filters are applied on one of eight different directions respectively.

The global feature extraction means combines the features of the local areas into the global feature with the use of Gaussian functions.

It is preferable that the probability models are generated utilizing expectation maximization algorithm and supervised learning with the use of neural network, but any other learning algorithm may be used.

After the probability models are generated, high-accurate behavior recognition of the mobile unit is enabled by applying the models to an image acquired afresh. Therefore, the apparatus may comprises a behavioral recognition means for applying Bayes' rule with use of the probability model on an image acquired afresh. The behavioral recognition means further calculates confidence for each of the behavior commands to recognize behavior of the mobile unit.

It is desirable that the accuracy of the behavior recognition is always greater than a certain level. Therefore, the apparatus according to the invention further comprises a behavioral assessment means for comparing the confidence with a predetermined value to assess the recognized behavior, an attention generation means for generating attentional demanding which demands to cause the probability model to be updated based on the result of the assessment, and an attentional modulation means for changing specified parameter of the global feature extraction means in response to the attentional demanding.

In this case, the learning means recalculates probability model after the parameter is changed. Then the behavior recognition means recognizes behavior of the mobile unit again with the probability models.

Other features and embodiments of the invention will be apparent for those skilled in the art by reading the following detailed description referring to the attached drawings.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Now some preferred embodiments of the invention will be described below with reference to the attached drawings.

FIG. 1is a block diagram of image recognizing apparatus10according to one embodiment of the present invention. Image recognizing apparatus10comprises a behavior command output block12, a local feature extraction block16, a global feature extraction block18, a learning block20, memory22, a behavior recognition block24, a behavior assessment block26, an attention generation block28and an attentional modulation block30.

The recognizing process of the image recognizing apparatus10comprises two parts. First part is advance learning part, where relationship is learned between behavior of a moving mobile unit and images taken by a camera equipped on the mobile unit. Second part is behavior recognizing part, where behavior of the mobile unit is recognized based on newly taken image using the knowledge learned in the advance learning part.

In the advance learning part, the behavior command output block12, the local feature extraction block16, the global feature extraction block18, the learning block20and the memory22are employed. In the behavior recognizing part, in addition to those blocks, the behavior recognition block24, the behavior assessment block26, the attention generation block28and the attentional modulation block30are employed.

At first, each block employed in the advance learning part is described.

The behavior command output block12outputs behavior commands to a mobile unit32. As used herein, term “behavior command” means a command to cause the mobile unit to go straight, to make right turn, or left turn. The behavior commands are output depending on instruction signal transmitted from external device. In alternative embodiment, the behavior command output block12may read pre-stored sequence of behavior commands and then output them. In a further alternative embodiment, the mobile unit32recognizes behavior of itself based on acquired image and determines which behavior to take next, and the behavior command output block12outputs the behavior command depending on the determination.

The behavior command is sent to mobile unit32by radio or cable broadcasting and causes it to move (for example, go straight or turn left/right). The behavior command is also supplied to global feature extraction block18and used for generation of global features described below.

Provided on the mobile unit32is an image acquisition block14such as a charge-coupled device (CCD) camera, which acquires image I(t) of external environment of the mobile unit32at time t every preset interval and supplies it to the local feature extraction block16.

The local feature extraction block16extracts feature vector of each local area of the image I(t). As used herein, the term “local area” means one of small areas into which whole image I(t) acquired by the image acquisition block14is divided such that each small area has same dimension. Each local area is composed of plurality of pixels. In the present embodiment, the local feature extraction block16calculates optical flow from two images I(t), I(t+1) consecutive in time and used the optical flow to generate feature vector for each local area (hereinafter referred to as “local feature”). The extracted local features are supplied to the global feature extraction block18.

The global feature extraction block18combines all of the local features of the image I(t) and extracts one new feature vector, which is referred to as “global feature” hereinafter. The global feature is supplied to the learning block20.

The learning block20implements learning based on the global feature and generates probability models described below. In the present embodiment, well-known expectation maximization algorithm and supervised learning with the use of neural network are used for such learning. Alternatively, other learning algorithm may be used. The probability models generated by the learning are stored in the memory22and used for recognition of behavior of the mobile unit32in the behavior recognizing part.

After the advance learning part is finished, the image recognizing apparatus10provides functionality of recognizing the behavior of the mobile unit32accurately by applying the probability models on an image acquired afresh by the image acquisition block14.

Now each block employed in the behavior recognizing part is described.

The image acquisition block14acquires image I(t) of external environment of the mobile unit32at time t for every preset interval as is noted above, and then supplies it to the behavior recognition block24this time. The behavior recognition block24applies the probability model stored in the memory22on the supplied image I(t). The block24then calculates “confidence” for each of behavior commands and recognizes the behavior of the mobile unit32based on the confidence.

The confidence is supplied to the behavioral assessment block26, which calculates logarithmic likelihood of the confidence. If the logarithmic likelihood of the confidence is larger than a specified value, no more operation is generated. If the logarithmic likelihood of the confidence is equal to or less then the predetermined value, the attention generation block28generates attentional demanding signal and supplies it to the attentional modulation block30.

Upon receiving the attentional demanding, the attentional modulation block30changes (or modulates) a specified parameter in the learning algorithm and causes the learning block20to update the probability models, which is stored in the memory22. The behavior recognition block24uses the updated probability models and recognizes the behavior of the mobile unit32again. Thus, the accuracy of behavior recognition will be expected to be more than a certain level.

While the image acquisition block14needs to be installed on the mobile unit32, it is not necessary for the image recognizing apparatus10to be installed on the mobile unit32and may be located in external place. In case of being installed on the mobile unit32, the image recognizing apparatus10may be either integral with or separate from the image acquisition block14. Communication between the image acquisition block14and the image recognizing apparatus10may be done via cable or radio transmission.

The image recognizing apparatus10may be all or partly implemented by, for example, executing on a computer program configured to execute processes noted above.

Now process in the advance learning part are described in detail with reference toFIG. 1andFIG. 2.FIG. 2is a flowchart illustrating the process of the advance learning part.

When a mobile unit32moves in accordance with behavior command from the behavior command output block12, the image acquisition block14acquires image I(t) at time t while the mobile unit32moves around in a given environment (step S42). The local feature extraction block16extracts local features from the acquired images I(t) and I(t+1) (step S44. . . S48). More specifically, Gabor filters are used to calculate image intensity El(xt,yt) by being applied to each local area image within the acquired images. A plurality of image intensities El(xt,yt) in each direction of the Gabor filter are obtained for each local area according to the following Eq. (1) (step S44).
El(xt,yt)=Img(t)·Gbri(+)+Img(t+1)·Gbri(−)  (1)
Here Gbrl(+) and Gbri(−) represent positive component and negative component of the Gabor filter respectively. Subscript “i” represents direction of the Gabor filter. In the embodiment, “i” ranges from 1 to 8. Img(t)represents a local area image within the image I(t) acquired at certain time t and Img(t+1)represents a local area image within the image I(t+1) acquired at consecutive time t+1. (xt,yt) shows coordinate of pixel in a local area at time t. Therefore, El(xt,yt) represents image intensity in direction i of the local area image.

Direction for applying Gabor filters and the number of Gabor filters are arbitrary. In the embodiment, imitating receptive field of visual sensation facility of human being, Gabor filters in eight directions extending radially in equal angles from center of whole image are employed.

The local feature extraction block16selects a direction j having the largest image intensity from the plurality of image intensities El(xt,yt) (i=1, . . . , 8) for each local area according to following Eq. (2) (step S46).
j=argmaxiEl(xt,yt)  (2)
It should be noted that selected directions j will be different between local areas.

Then the local feature extraction block16applies Gauss function to the largest image intensity Ej(xt,yt) according to following Eq. (3) to obtain local feature Ψj(xt,yt) for each local area (step S48).

Ψj⁡(xt,yt)=exp⁡(-0.5⁢(Ej⁡(xt,yt)-μj)2σj2)(3)
In Eq. (3), “μj” is an average of image intensities Ej(xt,yt). “σj” is variance of these image intensities Ej(xt,yt). Therefore, the local feature Ψj(xt,yt) means the expression of probability density distribution for image intensity Ej(xt,yt) in each local area image with regard to direction having the largest intensity. Local features Ψj(xt,yt) are calculated as many as the number of local areas. It should be noted that the direction j where local feature Ψj(xt,yt) is calculated for each local area is different.

Upon receiving the local features Ψj(xt,yt) from the local feature extraction block16and the behavior command from the behavior command output block12, the global feature extraction block18combines all local features Ψj(xt,yt) with regard to each largest direction j of the image intensity to obtain global feature ρj(χt|l) according to Eq. (4) (step S50).
ρj(χt|l)=∫χt∈2Ψj(xt,yt)dxtdyt(j=1, . . . , 8)  (4)
Here, “χt” means two-dimensional Cartesian coordinate of (xt,yt).

Calculated global feature ρj(χt|l) is distributed to one of classes according to the behavior command output by the behavior command output block12when the image I(t) is acquired, and stored in memory (step S52). Here “l” represents behavior command. In the present embodiment where three behavior commands (going straight, a left turn and a right turn) are used, l=1 corresponds to a behavior command of going straight, l=2 to a behavior command of turning left, and l=3 to a behavior command of turning right. Therefore, the global features ρjwhen the mobile unit is going straight (l=1), turning left (l=2) or turning right (l=3) are stored in different classes.

These classes are called “attention class” Ωl. As used herein, term “attention class” is for updating learning result efficiently by noticing certain feature when new feature is presented, not reflecting all of them. The attentional class is identical to the probability model of different behaviors.

It should be noted that the number of attention class is not limited to three but any number of the attention class may be employed correlating to the number of behavior commands.

Because the global feature ρj(χt|l) is calculated in association with behavior command for image acquired at time t, a lot of sets of global feature for eight directions are stored by behavior command.

FIG. 3toFIG. 5illustrate the relationship among original (acquired) image I(t), local feature Ψj(xt,yt) and global feature ρj(χt|l) in the environment where the mobile unit32runs toward obstacles.FIG. 3correspond to the case when the mobile unit32goes straight.FIG. 4correspond to the case when the mobile unit32turns left, andFIG. 5correspond to the case when the mobile unit32turns right, respectively.

Upper part (a) ofFIG. 3toFIG. 5show images of obstacles acquired at the place when the car is located in the environment. Middle part (b) ofFIG. 3toFIG. 5show graph of local feature when one direction of the Gabor filter is applied to a whole image, calculated by the Eq.(2) and Eq.(3). In this example, the whole image is divided in 77*57 local areas. Lower part (c) ofFIG. 3toFIG. 5are the Polar-shaped maps which depict the global feature ρjcalculted by Eq. (4), consisting of the eight direction of the Gabor filter. In part (c), numerals of 1 to 8 correspond to application direction of the Gabor filter (upward direction, right-upward direction, or the like).

Comparing the polar-shaped maps in part (c) ofFIG. 3toFIG. 5, feature on behavior (behavior command l) of mobile unit32may be recognized.

Back toFIG. 2, after storing the global feature ρj(χt|l) in step S52, the learning block20starts learning based on the global feature ρj(χt|l) (step S54to S58). More specifically, Expectation Maximization algorithm (EM algorithm) and supervised learning with the use of neural network are employed to obtain probability models for recognizing behavior of the mobile unit32.

The EM algorithm is an iterative algorithm for estimating parameter θ which takes maximum likelihood when observed data is viewed as incomplete data. Assuming mean of the observed data is represented as μland covariance as Σl, the parameter θ may be represented as θ(μl,Σl). EM algorithm is initiated with appropriate initial values of θ(μl,Σl). Then the parameter θ(μl,Σl) is updated one after another by iterating Expectation (E) step and Maximization (M) step alternately.

On the E step, conditional expected value φ(θ|θ(k)) is calculated according to following Eq. (5)
φ(θ|θ(k))=ΣiΣlp(ρl|Ωl;θ(k))log(p(ρl,Ωl;θ(k)))  (5)

Then on the M step, parameters μland Σlfor maximizing φ(θ|θ(k)) are calculated by following Eq. (6) and comprise a new estimated value θ(k+1).
θ(k+1)=argmaxθφ(θ,θ(k))  (6)

After repeating E steps and M steps, conditional expected value φ(θ|θ(k)) is obtained. By partial differentiating this value φ(θ|θ(k)) on θ(k)and letting a result equal to zero, parameters μland Σlmay be finally calculated. More detailed explanation will be omitted because this EM algorithm is well known in the art.

By using EM algorithm, global feature of each attention class Ωlcan be expressed by normal distribution (step S54).

The global feature extraction block18uses those calculated μland Σlin following Eq. (7) to calculate prior probabilityp(ρl|Ωl), which means the probability that global feature ρj(χt|l) belongs to class Ωlof behavior command l (step S56).

p_⁡(ρl❘Ωl)=∑m=1M⁢αm⁢⁢ρl(2⁢π)N/2⁢∑m⁢⁢ρl⁢exp⁡(-12⁢(ρl-μ)T⁢∑m⁢⁢ρl-1⁢(ρl-μ))(7)
Here, N is the number of dimensions of global feature ρj(χt|l).

Next, supervised learning with the use of neural network will be described. In this learning, conditional probability density function p(I(t)|Ωl) is calculated for image I(t) with attention class as supervising signal (step S58).

FIG. 6shows one example of hierarchical neural network used for the supervised learnin. This hierarchical neural network has three layer of node. Nodes in input layer72correspond to (original) images I(t), nodes in middle layer74to global features ρj(χt|l), and nodes in output layer76to attention classes Ωl, respectively. Although only three nodes are illustrated on the input layer72for simple illustration, there are actually as many nodes as the number of images I(t). Likewise, on the middle layer74, there are nodes as many as nodes on the input layer72. Nodes on the middle layer74correspond to nodes on the input layer72one by one. Nodes on the output layer76are generated as many as the number of attention class Ωl(three in this embodiment).

InFIG. 6, “λ” denotes synaptic weight of the hierarchical neural network. The probability that global feature ρj(χt|l) belongs to each attention class Ωlis obtained by EM algorithm, and global feature ρj(χt|l) is calculated for a set of images I(t) and I(t+1) one by one, so probability relationship (that is, “λ” inFIG. 6) between image I(t) and attention class Ωlis determined by repeating the supervised learning with attention class Ωlas supervising signal. More specifically, this probability relationship is conditional probability density function p(I(t)|Ωl). More detailed explanation will be omitted because this hierarchical neural network is well known in the art.

By such supervised learning with neural network, direct relation between image I(t) and attention class Ωl, that is, conditional probability density function p(I(t)|Ωl) may be obtained.

Process in steps S54to S58are executed every behavior command l. Therefore, in the present embodiment, prior probabilityp(ρl|Ωl) and conditional probability density function p(I(t)|Ωl) (these are called “probability model” integrally) are calculated for behavior command l=1, 2, and 3.

Probability model calculated by learning block20is stored in memory22(step S60). If advance learning is continued, “yes” is selected in step S62and a series of processes from step S42to S60is repeated, and then probability model is updated. While the mobile unit32is moving, advance learning is executed for all of images I(t) acquired in time t. When probability model is judged to be highly accurate enough to recognize behavior of mobile unit32(for example, process is completed for predetermined number of images I(t)), the process ends (step S64).

Referring toFIG. 1andFIG. 7, the behavior recognizing part will be described in detail.FIG. 7is a flow chart illustrating of process of the behavior recognizing part.

The image acquisition block14acquires two of new images at time t every preset interval (step82). Probability models (prior probabilityp(ρl|Ωl) and conditional probability density function p(I(t)|Ωl)) calculated in the advance learning part are used in following Bayes' rule and confidence p(Ωl(t)) for each attention class Ωl(l=1,2,3) is calculated (step S84). This confidence p(Ωl(t)) means the probability that the new image I(t) belongs to each attention class Ωl.

p⁡(Ωl⁡(t))=p_⁡(ρl❘Ωl⁡(t))⁢p⁡(I⁡(t)❘Ωl⁡(t))∑k⁢p_⁡(ρl❘Ωk⁡(t))⁢p⁡(I⁡(t)❘Ωk⁡(t))(8)
Then, among three calculated confidences p (Ωl(t)), p (Ω2(t)), and p(Ω3(t)), the largest one is selected (step S86).

The behavior assessment block26determines whether logarithmic likelihood of confidence p(Ωl(t)) is larger than a predetermined value K (step S88). If log p (Ωl(t))>K, behavior command l corresponding to attention class Ωlof which the confidence is largest is recognized as the behavior of the mobile unit32(step S92). Otherwise, log p(Ωl(t))≦K, the attention generation block28generates attentional demanding. The attentional modulation block30increments gaussian mixture “m” in Eq. (7) by specified value (that is, attentional modulation) (step S90). And in the learning block20, a series of process in steps S56to S60inFIG. 2is carried out again and the probability models (prior probability p−(ρ|Ωl) and conditional probability density function p(I(t)|Ωl) are updated.

The process goes back to step S84and repeats steps S84to S88. Thus, gaussian mixture m is increased until logarithmic likelihood log p(Ωl(t)) excesses the predetermined value K. In alternative embodiment, once-generated probability model may be always used without updating process.

As discussed above, the image recognizing apparatus according to the invention does not recognize the behavior of the mobile unit based only on the image. Instead, since the image recognizing apparatus first complete the learning on the relationship between the global feature extracted from the image and behavior command and then recognizes the behavior using the learning result, the apparatus can recognize the behavior of the mobile unit rapidly and accurately in real application.

In addition, in the case of the mobile unit32can not move correctly in accordance with supplied behavior command due to such as error-mounted wheels, the image recognizing apparatus may get the real moving status of the mobile unit according to the behavior recognizing process.

Now one example of the invention will be described.FIG. 8shows a block diagram of radio control car (hereinafter referred to as “RC car”)100that carried image recognizing apparatus10of the present invention. Blocks with same numerals as blocks inFIG. 1have similar function. In addition to the image recognizing apparatus10, on the RC car100is provided an image input camera114to acquire images, a steering control block132and a motor control block134for controlling the behavior of the RC car100according to the behavior command, and receiver136and transmitter138for communicating with outside device.

The receiver136receives behavior command signal from the outside device and supplies it to the behavior command output block12. The RC car100makes going straight, left turn, or right turn in accordance with the behavior command. The transmitter138transmits behavior of RC car100recognized by the behavior recognition block24to the outside device.

After completing advance learning part, the result of behavior recognition for 24 frames of image is described as follows.

FIG. 9is an exemplar illustrating change of logarithmic likelihood log p (Ωl(t)) of confidence for an attention class. Horizontal axis expresses the number of gaussian mixture m of Eq. (7) and vertical axis expresses logarithmic likelihood log p(Ωl(t)) of confidence. It can be understood that the logarithmic likelihood becomes saturated when the number of Gaussian mixture m reaches about 50. Larger logarithmic likelihood for an attention class for each image means that the image recognizing apparatus10recognizes the probability as sufficiently high that the RC car100takes the behavior corresponding to that attention class.

FIGS. 10show graphs of logarithmic likelihood of confidence with 24 frames of image according to the Eq. (7) and (8) with applied number of Gaussian mixtures m=20 when the RC car100goes straight and then turns left/right.FIGS. 11show the same with m=50. InFIG. 10andFIGS. 11, upper graphs (a) show logarithmic likelihood of confidence for attention class for behavior command l=1, middle graphs (b) show logarithmic likelihood of confidence for attention class for behavior command l=2, and lowest graphs (c) shows logarithmic likelihood of confidence for attention class for behavior command l=3. Horizontal axis of each graph shows 24 frames of image on which behavioral recognition is performed. First 8 frames (frame1-8) correspond to the images acquired during behavior command l=2 is given to the RC car100(when the RC car turns left), middle 8 frames (frame9-16) to the images acquired during behavior command l=1 is given to the RC car100(when the RC car goes straight), and last8framess (frame17-24) corresponds to the images acquired during behavior command l=3 is given to the RC car100(when the RC car turns right).

Referring toFIG. 10, in graph (a), middle eight frames (when going straight) show largest logarithmic likelihood. In graph (b), first eight frames (when turning left) show largest logarithmic likelihood, and in graph (c) last eight images (when going right) show largest logarithmic likelihood. However, variation of logarithmic likelihood between frames is so large that recognition of behavior is not sufficient.

Referring toFIG. 11, similar toFIG. 10, frames corresponding to behavior command show largest logarithmic likelihood. However, comparing toFIG. 10, the variation of logarithmic likelihood between frames becomes small and connection is smoother. These advantages are accomplished by incrementing the number of gaussian mixture by attentional modulation.

As described above, the image recognizing apparatus according to the invention improves the reliability by repeating learning in attention class generated in bottom-up fashion in the advance learning part. In the behavior recognizing part, recognizing accuracy of behavior is improved because learning result is updated until logarithmic likelihood of confidence excesses the predetermined value.

According to the invention, instead of recognizing the behavior of mobile unit based only on images, the image recognizing apparatus pre-learned the relationship between images and behavior commands and determines the behavior using the learning result. Therefore, it can recognize the behavior of the mobile unit rapidly and accurately in real application.

Although it has been described in details in terms of specific embodiment according to the invention, it is not intended to limit the invention to such specific embodiment. Those skilled in the art will appreciate that various modifications can be made without departing from the scope of the invention.