Apparatus and method of controlling mobile terminal based on analysis of user's face

An apparatus and method of controlling a mobile terminal by detecting a face or an eye in an input image are provided. The method includes performing face recognition on an input image facing and being captured by an image input unit equipped on the front face of the mobile terminal; determining, based on the face recognition, user state information that includes whether a user exists, a direction of the user's face, a distance from the mobile terminal, and/or a position of the user's face; and performing a predetermined function of the mobile terminal according to the user state information. According to the method, functions of the mobile terminal may be controlled without direct inputs from the user.

PRIORITY

This application claims priority under 35 U.S.C. §119(a) to Korean Patent Applications filed in the Korean Intellectual Property Office on May 2, 2012 and assigned Serial No. 10-2012-0046548, and on Apr. 25, 2013 and assigned Serial No. 10-2013-0046000, the entire disclosures of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to an image processing apparatus and method, and more particularly, to an apparatus and method of controlling a mobile terminal based on analysis of a user's face in an input image.

2. Description of the Related Art

Mobile terminals have increasingly expanded their services and additional functions, among which a camera function is considered a key function. To increase usefulness of the mobile terminal and satisfy different desires of users, combinations of the mobile terminal having a camera with their services and additional functions are required. As an example of the combinations, there is a technology to recognize faces from images inputted through the camera. The face recognition technology has been found to be very important in many application fields, such as expression studies, driver's drowsiness detection, or identification check.

Among the face recognition technologies, there is one method where three or more of features, such as a face, eyes, a mouth, a jaw, eye brows, a facial contour, etc., are identified in an image captured from the camera, a rotational direction of the image is determined with respect to a reference direction, and the image is displayed in the determined rotational direction. In addition, there is a technology to display information by recognizing the user's facial contour and positions of his/her eyes, jaw, and mouth, calculating a rotation angle of a line that links both eyes, and determining a display direction of the information. In this regard, the face recognition procedure requires detecting correct positions of many facial components or features in the image, such as the face, eyes, mouth, jaw, etc.

As such, since most conventional technologies are based on detection of the user's face together with at least one facial component, such as the eyes, mouth, jaw, etc., they may not obtain normal face recognition results nor correctly perform functions corresponding to the face recognition results from an image having a face being obscured by any other object or having only a part of the face being captured due to a particular viewing angle.

Furthermore, for a face recognition technology, a correct eye detection technique needs to be done for correct face recognition. However, conventional technologies only focus on detection of eye positions in a face image and do not consider the situation where the face detection has failed. Thus, a need exists for correctly detecting the eye positions even in the situation where the face detection has failed.

Furthermore, in order for a user to perform a desired operation of the mobile terminal, hardware interfaces, such as keypads or touchscreens have to be used. If it is possible for the mobile terminal to be controlled through eye detection without direct inputs from the user, it guarantees better quality of user experience.

SUMMARY OF THE INVENTION

The present invention has been made to address at least the problems and disadvantages described above and to provide at least the advantages described below. Accordingly, an aspect of the present invention provides an apparatus and method of controlling a mobile terminal by detecting a face or an eye in an input image without direct input from a user.

An aspect of the present invention also provides an apparatus and method of controlling a mobile terminal through eye detection even if faces are not detected in an input image.

Another aspect of the present invention provides an apparatus and method of performing face detection even on a partial face in an input image.

In accordance with an aspect of the present invention, an apparatus for controlling a mobile terminal based on analysis of a user's face is provided, the apparatus including an image input unit; an image processor configured to perform, upon reception of an input image through the image input unit, face detection or eye detection on the input image; and a controller configured to determine user state information based on face recognition results or eye recognition results from the image processor, and control to perform a predetermined function of the mobile terminal according to the user state information.

In accordance with another aspect of the present invention, a method of controlling a function based on analysis of a user's face in a mobile terminal is provided, the method including driving, upon occurrence of a predetermined event, an image input unit; performing, upon reception of an input image through the image input unit, face detection on the input image; performing eye detection on the input image if the face detection fails; determining user state information based on face recognition results or eye recognition results; and performing a predetermined function of the mobile terminal according to the user state information.

Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. In this regard, the embodiments of the present invention may have different forms and should not be construed as being limited to the descriptions set forth herein. Accordingly, the embodiments are merely described below, by referring to the figures, to explain aspects of the present invention.

The present invention provides a method of controlling a mobile terminal by detecting a face or an eye in an input image. The method includes performing face recognition on an input image facing and being captured by an image input unit equipped on a front face of the mobile terminal; determining, based on the face recognition, user state information that includes whether a user exists, a direction of the user's face, a distance from the mobile terminal, and/or a position of the user's face; and performing a predetermined function of the mobile terminal according to the user state information. According to the method, functions of the mobile terminal may be controlled without direct inputs from the user.

Features and operations of the mobile terminal will now be described in connection withFIG. 1.FIG. 1is a block diagram of the mobile terminal, according to an embodiment of the present invention. The mobile terminal may be a smartphone, a cell phone, a game console, a television, a display device, a vehicle head unit, a notebook, a laptop, a tablet Personal Computer (PC), a Personal Media Player (PMP), a Personal Digital Assistant (PDA), or the like.

Referring toFIG. 1, the mobile terminal mainly includes an image input unit100, an image processor105, a controller120, a memory130, a display unit140, and a communication unit150. Other components such as a microphone may be further included in the mobile terminal, but the following description will only describe the components in connection with an embodiment of the present invention.

The image input unit100equipped on the front face of the mobile terminal faces a user and serves to capture the face of the user. The image input unit100may be, e.g., a camera including a lens system, an image sensor, a flash, etc. The user may capture a video image or a still image through the image input unit100, and an image whose front is captured or in which the user is captured is delivered to the image processor105in an image signal. In processing an image frame that constitutes a still image or a video image, the image signal output by the image sensor comprised of a plurality of pixels arranged in an M×N matrix corresponds to a set of voltages, i.e., a set of pixel values, output from the pixels. The image signal output from the image input unit100represents a single frame, i.e., a still image. The frame is comprised of M×N pixels. The image sensor may be a Charge-Coupled Device (CCD) image sensor, Complementary Metal-Oxide Semiconductor (CMOS) image sensor, etc.

An image captured by the image input unit100is, in general, displayed on the display unit140for preview, but in the embodiment of the present invention where a specific function or a specific application of the mobile terminal is intelligently controlled by using face detection results or eye detection results, while the specific function or the specific application is being activated by the user, the input image may not be displayed on the display unit140. Thus, even though the user does not recognize himself/herself being captured, an intuitive input for controlling the mobile terminal may be made by the user adjusting a distance between the mobile terminal and the user, an eye size, or the like.

The image processor105performs face recognition on the captured image by processing the image in frame units. The face recognition may be applied not only to still images like a photograph but also to video images. For example, in the case of videos, the videos are comprised of consecutive still image frames and so the still image frames may be used in the face recognition.

The image processor105extracts and recognizes a face image if the face image is included in the input image, or performs recognition by detecting eyes if the face image is not included in the input image. To do so, the image processor105mainly includes a face detector110and an eye detector115. Operations of the face detector110will be described in detail later.

The face recognition is performed based on a face recognition algorithm contained in the mobile terminal, and uses a method of identifying the face by using contour information about the entire face of an object or uses a face extraction technology that uses color and/or texture of facial skin, templates, and/or the like. For example, the image processor105may perform face learning with numerous face images, and may detect a face image from input image frames based on accumulated face learning data.

Furthermore, eye detection is based on an eye recognition algorithm, and in general, an eye is detected after face detection in a restricted part of a detected face area. Using both eyes in the input area may also be used to detect the eye. In particular, in an embodiment of the present invention, if a face is not detected in the input image, a method of performing eye detection is used within the entire input image or in a predetermined area of interest.

The memory130stores an operating system of the mobile terminal, various applications, information, data, or files input to the mobile terminal, and information, data, or files created in the mobile terminal. For example, the memory130stores images such as photos, videos, or the like, and the face recognition algorithm to recognize the face of an object, which is captured by the image input unit100, and data to be used in the face recognition. For example, the memory130may store information about faces to be used in the face detection, and facial feature information of each user, which is registered in advance to be used in the face recognition. The face information stored in the memory130to be used in the face recognition may be learned and updated by repetitive face recognition.

The display unit140displays images based on image signals input from the controller120and delivers received user input data to the controller120. The display unit140includes a display part, such as a Liquid Crystal Display (LCD), Organic Light Emitting Diodes (OLEDs), or LEDs, and a touch panel arranged under or on top of the display part. The touch panel detects user inputs. When a user input means (e.g., a finger, a stylus pen, etc.) presses the surface of the display part, the touch panel outputs a detection signal (or a touch detection signal) that has information about input positions (or coordinates) and/or input states (e.g., mouse down, mouse up, mouse movement, etc.). For example, the user runs applications related to various executable items by touching the items displayed on a screen (i.e., the surface) of the display unit140. The display unit140is a means for receiving user inputs and for outputting screens related to applications, such as a camera application or a video communication application. In an embodiment of the present invention, the display unit140uses the touchscreen as an example, but the display unit140may also be configured only with the display part.

The communication unit150transmits messages, data, and/or files created by the controller120through a wired line connection or wirelessly, or delivers messages, data, and/or files received through the wired line connection or wirelessly to the controller120.

The sensor unit160detects a position, a direction, or movement of the mobile terminal, and may be implemented with an inertial sensor, such as an accelerometer, a gyroscope, a shock sensor, or a tilt sensor, an altimeter, a gravity sensor, a geomagnetic sensor, or a combination of them. The present invention is not limited thereto, but the sensor unit160may also be implemented with a different type of sensor that may detect rotational direction, movement direction, or tilting level.

The speaker170outputs a sound signal (or voice data) input from the controller120to the air, and specifically, outputs the sound signal by making stereo sounds to the left or the right according to a face direction under control of the controller120. Alternately, the sound signal is output with different volume according to an eye size under control of the controller120.

The controller120is a Central Processing Unit (CPU), which controls general operations of the mobile terminal and serves to implement the method of controlling functions of the mobile terminal based on analysis of the user's face. The controller120detects various user inputs received not only through the display unit140in the touch screen but also through the image input unit100. The user input may include different forms of information entered into the image input unit100, such as touches, user gestures, pupil movements, etc. The controller120controls predetermined operations or functions to be performed in the mobile terminal in response to the user inputs.

Specifically, the controller120estimates a size, a position, and an angle of the face or the eye using the face and/or eye detection results from the image processor105. Accordingly, the controller120determines the user state information based on the face and/or eye detection results. The user state information includes at least one of the user's presence or absence, a direction of the user's face, a relative distance between the mobile terminal and the user, and/or a position of the user's face.

Specifically, the controller120determines the user's presence or absence based on whether the face or the eye has been detected in the input image, and estimates the relative distance between the mobile terminal and the user according to a percentage of the face or the eye in the screen, i.e., the face size or the eye size. The controller120also determines the direction and the position of the user's face by using an angle and position at which the face is detected e.g., among 0°, 90°, 180°, and 270°. The controller120may further determine the direction and the position of the user's face according to the eye position and angle even when the face has not been detected in the input image.

Subsequently, the controller120controls to perform a predetermined function according to the user state information. The predetermined function according to the user state information may be classified and described in the following embodiments.

In a first embodiment of the present invention, if a predetermined time has elapsed while the user is using the mobile terminal, the controller120turns the display unit140off and activates a screen lock mode. When the screen lock mode is activated, the controller120drives the image input unit100to capture what is in front of the mobile terminal, determines the user's presence or absence based on the face or eye detection results provided from the image processor105, and controls to unlock the screen if it is determined that the user exists ahead of or in front of the mobile terminal. After that, even if the predetermined time has passed, the controller120keeps the display unit140on as long as the user's face is detected in the input image. In this case, no additional input, such as a particular key, a password, and/or a lock pattern is required for unlocking the screen, thereby increasing user convenience. Otherwise, if it is determined that the user does not exist ahead of the mobile terminal, the controller120maintains the screen lock mode.

As such, according to the first embodiment of the present invention, the controller120controls extension of screen lighting time and screen lock. For example, even when a user leaves his/her seat while watching a video, the video continues to be played until the user selects to pause the video. However, in the embodiment of the present invention, if it is determined that the user does not exist because a face or an eye has not been detected in the input image, the controller120may pause the video without need for a separate key input.

In a second embodiment of the present invention, the controller120drives the image input unit100upon detection of rotation of the mobile terminal through the sensor unit160. Then, the image processor105determines the direction of the user's face using face and/or eye detection results provided. The controller120thus determines a direction in which to display the screen according to the direction of the user's face. In other words, even though the mobile terminal has been rotated, the direction in which to display the screen is determined preferentially based on the direction of the user's face and so the user may always see the screen in his/her direction. As such, according to the second embodiment of the present invention, a screen rotation function may be intelligently controlled.

In a third embodiment of the present invention, the controller120makes stereo sounds to the left or the right through the speaker170according to the direction of the user's face.

In a fourth embodiment of the present invention, in making video calls or playing a video, the controller120changes sound outputs according to the distance between the mobile terminal and the user estimated from the face size or the eye size.

According to the foregoing embodiments, while the user is using the mobile terminal, the user may control the mobile terminal intuitively without direct inputs and the mobile terminal controls itself intelligently.

FIG. 2is a flowchart of operations of the mobile terminal, according to an embodiment of the present invention.

Referring toFIG. 2, if a predetermined event occurs, in step200, the controller120drives the image input unit100, in step205. The predetermined event includes both changes in hardware and software of the mobile terminal, such as rotation or shaking of the mobile terminal, a lighting change, incoming calls, reception of text messages, and notifications made in smartphone applications.

Then, in step210, the image processor105discards initial image frames input through the image input unit100and performs face recognition on subsequent input images. The initial images inputted after driving the image input unit100may be dark and out of focus because camera parameters including exposure, focus, white balance or the like have not been completely set for the surrounding environment, and it is thus desirable to skip at least one of the initial input images and use image frames after completion of the automatic settings.

If, in step215, face recognition using the face recognition algorithm on the input image is successful, the method proceeds to step225. Otherwise, in step215, if the face recognition fails, the image processor105performs eye detection and determines if the eye detection is successful. If the eye detection fails, the method returns to step210where images are received again by the image input unit100. If, in step220, the eye detection is successful, or after successful face detection in step215, the controller120analyzes the user state information based on the face detection results and/or the eye detection results provided from the image processor105, in step225. Specifically, the controller120determines, upon detection of a face or an eye in the input image, that the user exists, and analyzes a position and direction of the user's face looking at the mobile terminal, and a relative distance from the mobile terminal.

Next, in step230, a function of the mobile terminal is controlled according to the analyzed user state information. For example, when the predetermined event, such as reception of a text message, occurs, to notify the user of the reception of the text message through the screen, the user needs to be looking at the screen. Thus, the controller120displays the text message on the display unit140if the face or the eye is detected.

FIG. 3is a flowchart of a detailed description of the steps for performing the face recognition or the eye recognition, represented by a reference numeral240ofFIG. 2. That is,FIG. 3shows a detailed procedure of the face detection and the eye detection ofFIG. 2, andFIGS. 4 and 5will be referred to for better understanding of the present invention. In general, since the mobile terminal is often used at one of 0°, 90°, 180°, and 270° angles, the user's face in the input image may be positioned at any of 0°, 90°, 180°, and 270° angles, as inFIG. 4, as represented by reference numerals400,410,415, and420, respectively.

Referring toFIG. 3, after receiving an input image in step300, the image processor105performs steps305to320to determine if a face is detected at any of 0°, 90°, 180°, and 270° angles. If the face is detected at any of 0°, 90°, 180°, and 270° angles, the controller120determines a direction of the face based on the face detection results from the image processor105, in step355.

Otherwise, if the face is not detected in any of steps305to320, the image processor105performs steps325to340to determine if an eye is detected at any of 0°, 90°, 180°, and 270° angles. For example, even though the image input unit100is driven and starts capturing images, a screen of a currently running application is output instead of a preview screen and thus, the user is not aware that he/she is being captured and ends up being located out of a proper capture range of the image input unit100. Therefore, as illustrated inFIG. 5, only a part of the face500,510, or520may be captured.

In this case, even if the face detection in the input image has failed, it is possible to detect an eye505,515, or525. In this regard, as shown inFIG. 5, an image520having both eyes525may be captured, or images500and510having a single eye505or515may be captured. The present invention provides a method of performing eye detection on images having both eyes as well as a single eye.

The eye detection method will be described below. Upon detection of an eye at any of 0°, 90°, 180°, and 270° angles, the controller120determines the top or bottom direction of the face, in step350. Then, in step355, the face direction is determined, and in step360, whether to rotate the screen according to the determined face direction is determined. For example, if the user's face is detected at an angle of 180°, which is represented by reference numeral415inFIG. 4, the user is positioned upside down on the mobile terminal and the screen of the mobile terminal is seen upside down from the user. According to an embodiment of the present invention, the screen of the mobile terminal may be rotated in the direction in which the user's face was detected. That is, the screen of the mobile terminal is rotated and displayed in the direction of the user's face.

On the other hand, eye detection is performed using the following method that uses skin color, eye templates, or the like. In particular, the embodiment of the present invention provides a method of performing eye detection on input images having both eyes as well as a single eye.

Specifically, the image processor105performs eye detection on the entire input image or on a predetermined area of interest. In this regard, to slow down the eye detection speed, local eye detection may be performed based on a position where an eye is detected, which is obtained from the previous image frame. Furthermore, skin color may be used to perform the eye detection, which will be described with reference toFIG. 6.

The image processor105detects one or more eye candidate areas with an eye recognition algorithm. For example, the eye candidate areas are detected by extracting closed curves that may be eye candidates through a process of e.g., digitizing the input image, or by using a characteristic in that the color of an eye, even for Caucasians having eyes without black pupils, tends to come in charcoal in the input image. By doing so, one or more eye candidate areas may be detected compared with general eye detection performed on a detected face area, and some of the eye candidate areas may be far from an actual eye area.

Thus, in the embodiment of the present invention, in case the one or more eye candidate areas are detected through the eye recognition algorithm, an eye detection method that uses skin color is provided to eliminate incorrectly detected eye areas from among the detected eye candidate areas.

To implement the method, the image processor105calculates percentages of skin color areas and non-skin color areas by using values of Y, Cb, Cr of the detected one or more eye candidate areas, and performs eye detection using the percentages. Ranges of the values of Y, Cb, Cr corresponding to the skin color are determined based on predetermined thresholds Ty, B1, B2, R1, R2, and pixel values corresponding to Y, Cb, Cr that satisfy the following Equation (1) are determined to be in a skin color area.
(Y≧Ty)
and
(B1<Cb<B2)
and
(R1<Cr<R2)  Equation (1)

The predetermined thresholds may vary depending on the surrounding environment, such as ethnic groups, lighting conditions and/or the like.

Calculation of the percentages of skin color areas and non-skin color areas are illustrated inFIG. 6. Referring toFIG. 6, the eye detector115in the image processor105detects the eye candidate areas, and divides the detected eye candidate areas into three areas600,605, and610in the vertical direction and into four areas in the horizontal direction. Then, the image processor105calculates a proportion S of the skin color in areas600and610, and calculates a proportion NS of the non-skin color in shaded parts of the middle area605, i.e., ¼ to ¾ parts615of the area605. Pixel values corresponding to the skin color are determined using values of Y, Cb, Cr of the detected eye candidate areas with the requirements represented by Equation (1).

Once the proportions of the skin color S and the non-skin color NS in the eye candidate areas are obtained as described above, the proportions of the skin color S and the non-skin color NS are compared with predetermined thresholds Ts and Tn, respectively. If the comparison satisfies the following Equation (2), the eye detection is determined to be successful, or else, it is determined to be an incorrect detection.
(S>Ts)
and
(NS>Tn)  Equation (2)

Thus, one of the one or more eye candidate areas that satisfies Equation (2) is finally determined as where the eye is detected. Using the method of eliminating incorrectly detected eye candidate areas facilitates easier eye detection in images having not only both eyes but also a single eye. Division of the eye candidate areas in horizontal and vertical directions is not limited to that as inFIG. 6.

For example, the image processor105may collect normal eye images and images frequently mistaken for eye images, perform eye learning on the collected images, and detect the eye in an input image based on eye learning data accumulated through the eye learning.

According to the embodiments of the present invention, even though the face is not detected as a whole, eye detection may indicate the user's presence, and direction of the user's face looking at the mobile terminal is estimated by sequentially performing eye detection on the input image at 0, 90, 180, and 270 degrees. Face detection is first performed in four directions (top, bottom, left, and right), and if a face is found, the face detection procedure is used to determine the direction of the user's face, or else if a face is not found, the eye detection procedure begins. Similar to the face detection, eye detection is performed in four directions and reduces an amount of calculation by using rotated images that have been used in the face detection. If the eye detection fails in any of the four directions, it is repeated until an eye is detected in a particular direction. The eye detection in the particular direction is followed by a procedure of determining the top and bottom direction of the face.

Unlike face detection, the eye detection procedure itself does not bring out information about the top and bottom direction of the face and thus an additional procedure to determine the top and bottom direction of the face is required even if a final eye area is determined from among the eye candidate areas. For example, in determining the eye direction, an image of surrounding areas of the final eye area is normalized for a detected single eye or both eyes, and the top and bottom of the face are determined using the normalized image. For example, the top and bottom of the face may be determined by detecting positions of the eye brows as well as eyes.

In the embodiments described above, a face is detected by scanning with learned information about the face in a particular size in an input image. However, in capturing an image with a camera, only a part of the face may appear at the edge of the input image due to a viewing angle or distance between the camera and the face.

FIG. 7illustrates possible face positions in input images. Referring toFIG. 7, there may be a face700existing at the upper left corner, a face705existing at the upper (top) edge, a face710existing at the upper right corner, a face715existing at the left edge, a face720existing in the center of the image, a face725existing at the right edge, a face730existing at the lower left corner, a face735existing at the lower (bottom) edge, or a face740existing at the lower right corner in an input image. That is, partial faces may exist at the upper, lower, left, right edges or at each corner of an input image.

In those cases where partial faces700to740exist (except720having the face in the center), the face is not detected. Since a face is generally detected by scanning an input image using a learned face image in a particular size, the partial face is not recognized from the input image. In the conventional technology, if a left face detector and a right face detector are separately used to detect a partial face, the input image has to contain a left or right face. In this case, the face detection rate noticeably drops because of low compatibility with the known face detector. Furthermore, it may be difficult to detect the parts of the face other than the left and right parts.

Therefore, in an embodiment of the present invention, if an image including a partial face is obtained, face detection will be performed in the following method.

FIG. 8illustrates an image frame explaining a scanning method using a face determination area.

Referring toFIG. 8, a course830along which to scan an image frame800using a face determination area810from the upper left corner A to the lower right corner I is illustrated. The face determination area810has a predetermined size and may be a pre-trained face detector. Referring toFIG. 8, in the image frame800including a plurality of face images, a face image that exists near an edge of the image frame800may be partially cut. In other words, partial faces are likely to appear at the edge of the image frame800.

Considering this, the present invention provides a method of scanning the image frame800with the face determination area810partially deviating from the edge of the image frame800.

FIGS. 9A and 9Billustrate movements of the face determination area810in a different scaled image frame.

Referring toFIGS. 9A and 9B, the image frame800ofFIG. 9Amay be gradually scaled down to the image frame800ofFIG. 9B. Since the face determination area810has a predetermined size, a face as small as the face determination area810contained in the image frame800ofFIG. 9Amay be detected with the face determination area810, while a face as big as the face determination area810contained in the image frame800ofFIG. 9Bmay be detected with the face determination area810. The scale of each image frame800may be gradually changed.

FIG. 10illustrates a positional relationship between the face determination area810at the edge of the image frame800and the image frame800.

Referring toFIG. 10, a shaded area (hereinafter referred to as an additional area) surrounding the image frame800corresponds to an area in need to be corrected for face detection. A width and a height of the additional area are determined on the basis of the size of the face determination area810. For example, assuming that the width of the face determination area810is a and the height is b, an initial position A of the face determination area810is the upper left corner of the image frame800where the center of the face determination area810falls on the vertex of the upper left corner. At the initial position A, a width of the additional area surrounding the left edge of the image frame800is determined such that one half of the width a/2 of the face determination area810is included in the image frame800while the other half of the width a/2 is included in the shaded area.

Similarly, a width of the additional area surrounding the left edge of the image frame800may be determined when the face determination area810is moved from the upper left corner in the horizontal direction and lies at the upper right corner C. Although, in the embodiment of the present invention, the width of the additional area is determined to be a/2 at the left and right edges, widths of the additional area at the left and right edges may vary based on settings and may also be determined to be different from each other.

A height of the shaded area surrounding the upper and lower edges of the image frame800is determined such that only one half of the height b/2 of the face determination area810is included in the image frame800. Although, in the embodiment of the present invention, the height of the additional area is determined to be b/2 at the left and right edges, heights of the additional area at the upper and lower edges may vary based on settings and may also be determined to be different from each other.

Meanwhile, after the face determination area810is moved from the initial position at the upper left corner A, to the upper right corner C, the face determination area810is moved again to the left edge. The center of the face determination area810is moved accordingly to a position indicated by920. In this position, the upper third of b of the face determination area810overlaps the face determination area810at the upper left corner A. For scanning, the face determination area810is moved until it reaches a final position, i.e., the lower right corner I.

FIG. 11Aillustrates the image frame800in which a partial face exists at the right edge, andFIG. 11Billustrates face images included in the face determination area at different scanning positions.

Referring toFIG. 11A, if the face determination area lies at the right edge, part of the face determination area becomes empty. Different parts of a face, as shown inFIG. 11B, may be included at different scanning positions of the face determination areas1010,1030,1040, and1060ofFIG. 11A. Before performing face detection, each part1012,1032,1042, or1062within respective face determination areas1010,1030,1040, or1060is assumed to have a partial face and actually consists of corresponding pixels near the right edge of the image frame800.

Each face determination area1010,1030,1040, or1060may be represented as shown inFIG. 11Bas the right edge of the image frame800is scanned using the face determination areas1010,1030,1040, or1060as shown inFIG. 11A. Each face determination area1010,1030,1040, or1060includes an area (also called ‘a part’)1012,1032,1042, or1062which is filled with pixels and expected to have a partial face (hereinafter called an expected area), and an empty area (shaded area)1015,1035,1045, or1065.

In embodiments of the present invention, compensation of the empty areas1015,1035,1045, or1065of each face determination area1010,1030,1040, or1060is provided.

FIG. 11Cillustrates images compensated by a mirroring method, according to embodiments of the present invention.

Referring toFIG. 11C, the empty area1015,1035,1045, or1065ofFIG. 11Bis filled with pieces of pixel information of the partial face area1012,1032,1042, or1062with a mirroring method. Accordingly, as shown inFIG. 11C, an opposite area1115,1135,1145, or1165of each partial face area1012,1032,1042, or1062is filled with the same pixel information as the partial face area1012,1032,1042, or1062. Alternatively, other methods, such as Modified Census Transform (MCT), Local Binary Pattern (LBP), etc., may be used to fill the opposite area1115,1135,1145, or1165. As the face determination area is compensated as described above after scanning the image frame800in the horizontal or vertical direction, compensated images may be obtained as shown inFIG. 11C. Among the compensated images, there may be an image from which to detect a face. Specifically, if a compensated image in which main facial features, such as the eyes, nose, mouth is arranged is obtained by mirroring, face detection from the compensated image1060may be successfully performed. Therefore, compared with the conventional face detection method, the present invention performs successful face detection even on a partial face.

The mirroring method will be described in detail with reference toFIG. 12. Referring toFIG. 12, in the case where the face determination area lies across the edge of the image frame800, an area a1210deviating from the edge is filled with the same image as a shaded area a′1200of the face determination area, i.e., the area a1210is compensated by mirroring. Similarly, an area b1230is compensated by mirroring the same image as a shaded area b′1220.

The image compensation method is performed using the mirroring method for the left/right edge of the image frame800as in the foregoing embodiment. In another embodiment, the mirroring method may be applied to the upper/lower edge and the corner of the image frame800.

FIG. 13illustrates the image frame800in which a partial face lies at the lower edge or at the lower right corner, according to an embodiment of the present invention.

Referring toFIG. 13, where part of the face determination area810lies at the lower edge of the image frame800, an area deviating from the lower edge (a shaded area)1300appears empty. Similarly, if part of the face determination area810lies at the lower right corner, there is an empty area1310included within the face determination area810. For the upper and lower edges or the corners, an embodiment of the present invention provides a method to compensate an empty area of the face determination area810using a predetermined template. In another embodiment of the present invention, both the mirroring method and the template matching method may be employed to compensate the empty area of the face determination area810at the corner. Specifically, for the face determination area810at the lower right corner, as shown inFIG. 13, the empty area1310is filled with pixels that correspond to an eye part by mirroring the face determination area810with respect to the right edge of the image frame810, and then is filled with a part of a template image by attaching the part, which is a facial part below the nose of the template image by means of the template matching method for an area below the right corner of the image frame810.

FIG. 14illustrates predetermined face templates1400and1410. It is possible to compensate a facial area whose upper or lower part is cut away by compensating an empty area of the face determination area810using the predetermined face templates1400and1410. As such, in an embodiment of the present invention, by applying either the mirroring method or the template matching method, or both, a partial face that lies at the edge may be detected, thus improving the face detection performance.

FIG. 15is a flowchart of an image compensating method, according to an embodiment of the present invention.

Referring toFIG. 15, the controller120receives an image frame from the image input unit100or reads out an image frame from the memory130, in step1500. Although the embodiment of the invention is applied to an image frame, the embodiment may be applied to each of sequentially input or read-out image frames. The embodiments of the present invention may also be applied to an image captured by a camera on the spot while a video call between two persons, or a multilateral video conference, is performed, or in any other application mode. Furthermore, although it is illustrated inFIG. 15that the controller120controls the face detection operation, the face detection operation may, of course, be performed by the face detector110.

In step1505, the controller120determines whether an entire face is detected from an image frame. The entire face detection may be performed using a common face detection method in which face extraction technologies that use contour of a face, facial skin color and/or texture, templates, etc. may be employed. For example, the controller120may perform face learning with numerous face images, and may detect an entire face image from input image frames based on accumulated face learning data.

If the entire face is detected, the controller120stores facial information detected in step1540. The facial information may be used to control, e.g., a predetermined function of a mobile terminal.

Otherwise, if the entire face is not detected, whether an object cut out from the image frame is detected is determined in step1510. For example, if edge detection finds a discontinued edge line, it may be determined that there is a broken object. In this case, the broken object may be a partial face. Here, to detect a face, compensation should be performed for the additional area surrounding the edge of the image frame800. If the broken object is not detected in step1510, the face detection process ends.

If the broken object is detected in step1510, the controller120determines whether mirroring is possible in step1515. For example, the controller120may determine whether the face determination area lies at the left or right edge of the image frame800, whether there is a symmetry axis for mirroring in the face determination area, such as a vertical axis based on the nose, and/or whether there is a part cut away from the symmetry axis.

If mirroring is possible, the controller120performs mirroring in the same way as described in connection withFIG. 12, in step1520. Otherwise, if mirroring is not possible, the controller120performs template matching in step1525. Subsequently, the controller120performs face detection on the face determination area in step1530that includes an area compensated by mirroring or template matching Such face detection on the face determination area is performed by applying a common face detection algorithm. Then, the controller120determines whether a face is detected from the face determination area, in step1535. If facial information is included in the face determination area, the controller120detects the face and stores the facial information, in step1540. Otherwise, if a face is not detected, the controller120ends the face detection process. To scan the image frame with the face determination area in the horizontal or vertical direction, steps1510to1530should be repetitively performed.

Depending on the size of a partial face and the extent to which the partial face appears in the screen, a step of adaptively determining validity of the partial face may further be applied. The step of determining validity of a partial face may be required, because if only the partial face including the mouth exists around the corner and even if the partial face is detected as the face, the face information might not be appropriate or sufficient to be used to control a function of the mobile terminal. In other words, the step of determining validity of a partial face may be performed to lessen the probability of wrong detection, taking into account vulnerability of the information of the partial face, and the step may be optionally added or excluded depending on applications.

FIG. 16is a flowchart of an image compensating method, according to another embodiment of the present invention. Detailed descriptions of steps1600to1610will be omitted because they are the same as steps1500to1510ofFIG. 15.

In step1615, the controller120determines whether the face determination area lies at the edge of an image. If the face determination area lies at the edge of the image, mirroring is performed in step1620and face detection is performed on the face determination area that includes an area compensated by mirroring in step1625. If the face determination area does not lie at the edge in step1615, face detection is performed in step1630. If a face is detected in step1630, the detected face information is stored in step1650. If a face is not detected in step1630, template matching is further performed in step1635. Subsequently, the controller120performs face detection on the face determination area that includes an area compensated by the template matching, in step1640. By applying both mirroring and template matching for a corner of the image frame800, efficient partial face detection may be performed. Detailed descriptions of steps1645and1650will also be omitted because they are the same as steps1535and1540ofFIG. 15. Although, inFIG. 16, mirroring is followed by template matching, in other embodiments only one of mirroring or template matching may be applied or template matching may be followed by mirroring.

FIG. 17is a flowchart of an image compensating method, according to another embodiment of the present invention.

Referring toFIG. 17, detailed descriptions of steps1700and1705will be omitted because they are the same as steps1500and1505ofFIG. 15.

In step1710, the controller120determines whether part of the face determination area deviates from the left or right edge of an image frame. If part of the face determination area deviates from the left or right edge of an image frame, the controller120performs mirroring in step1715in the same way as performed inFIG. 12. Otherwise, if part of the face determination area does not deviate from left or right edge of an image frame, the controller120determines whether the face determination area deviates from a corner, e.g., the upper left corner or lower left corner of the image in step1720. If part of the face determination area deviates from the corner, the controller120performs mirroring followed by template matching, in step1725. Otherwise, if part of the face determination area does not deviate from the corner, e.g., the upper left corner or lower left corner of the image, the controller120determines whether part of the face determination area deviates from the top or bottom of the image in step1730. If part of the face determination area deviates from the top or bottom of the image, the controller120performs template matching. If part of the face determination area does not deviate from the top or bottom of the image, the process returns to step1705. Detailed descriptions of steps1740to1750will be omitted because they are the same as steps1530to1540ofFIG. 15.

The foregoing descriptions have taken examples of using face detection or eye detection information in determining the direction of the user's face, but sensor information from the sensor unit160may also be used. For example, once the sensor unit160detects rotation of the mobile terminal, the rotation information, such as rotational direction, movement direction, and tilting level of the mobile terminal may be used. In this case, face detection may be performed on the input image not in all four directions as inFIG. 4, but in a particular direction with respect to the rotational direction of the mobile terminal. In this regard, if the mobile terminal is rotated while the user is looking at the screen thereof, face analysis results show that the direction of the user's face is not toward the front of the screen of the mobile terminal, and thus the display direction of the screen may need to be rotated to fit to the user's gaze. Otherwise, if the user does not rotate the mobile terminal but a face detection result shows that the direction of the user's face has been changed, the screen display direction may be changed to correspond to the direction of the user's face.

According to the present invention, the mobile terminal is controlled by intelligently controlling hardware and software through analysis of user states, such as the user's presence or absence, a distance between the mobile terminal and the user, and/or the like, rather than being controlled according to settings that the user establishes in advance, thereby increasing user convenience.

According to the present invention, face detection may be replaced by eye detection when the face detection fails, and eye detection information is used to determine the user's presence or absence, based on which lighting time and on/off of the screen of the display unit may be controlled and a function to pause the screen in video playback may also be provided to the user.

Furthermore, user convenience in using the mobile terminal may be increased by controlling the screen rotation intelligently by estimating the direction of the user's face toward the mobile terminal according to an improved method of controlling the mobile terminal compared with the conventional method, and providing services to intelligently control the volume of the mobile terminal according to a change in the eye size. Thus, the user may use the mobile terminal more conveniently and intuitively, thereby increasing the quality of user experience.

In addition, according to the present invention, a partial face, even appearing near an edge of an image, may be detected, thereby significantly reducing the possibility of false recognition, and thus providing the user with smoother functions that correspond to the face detection results.

It will be appreciated that the embodiments of the present invention may be implemented in a form of hardware, software, or a combination of hardware and software. The software may be stored as program instructions or computer readable codes executable on the processor on a computer-readable medium. Examples of the computer readable recording medium include magnetic storage media (e.g., ROM, floppy disks, hard disks, etc.), and optical recording media (e.g., CD-ROMs, or DVDs). The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. This media can be read by the computer, stored in the memory, and executed by the processor. The memory may be an example of the computer readable recording medium suitable for storing a program or programs having instructions that implement the embodiments of the present invention. Numerous modifications and adaptations will be readily apparent to those skilled in this art without departing from the spirit and scope of the present invention.

The mobile terminal may receive and store the program from a program provider by wired or wireless connections thereto. The program provider may include a program having instructions to perform the method of controlling the mobile terminal based on analysis of the user's face, a memory for storing information required for the method of controlling the mobile terminal based on analysis of the user's face, a communication unit for communicating by wire or wirelessly with the mobile terminal, and a controller for receiving a request from the mobile terminal and delivering corresponding programs to the mobile terminal.