Mobile terminal releasing a locked state and method for controlling the same

Disclosed are a mobile terminal and a method for controlling the same. The mobile terminal includes: a terminal body; a camera; a sensor configured to sense a raising gesture of the terminal body; and a controller configured to: activate the camera in response to the raising gesture; detect a face within an image received via the camera; and release a locked state of the mobile terminal based on a face recognition of the detected face.

CROSS-REFERENCE TO RELATED APPLICATION

Pursuant to 35 U.S.C. § 119(a), this application claims the benefit of earlier filing date and right of priority to Korean Patent Application No. 10-2017-0075950, filed on Jun. 15, 2017, the contents of which are hereby incorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This specification relates to a mobile terminal capable of recognizing a gesture through a camera, and a method for controlling the same.

2. Background of the Invention

Mobile terminals have become increasingly more functional. Examples of such functions may include data and voice communications, capturing images and video through a camera, recording audio, playing music files through a speaker system, and displaying images and video on a display unit. Some mobile terminals additionally provide functions such as playing an electronic game, or executing a function of multimedia players. Especially, recent mobile terminals may receive multicast signal for providing visual content such as broadcasts, videos, or television programs.

As it becomes multifunctional, a mobile terminal can be allowed to capture still images or moving images, play music or video files, play games, receive broadcast and the like, so as to be implemented as an integrated multimedia player.

For minimized battery consumption, the mobile terminal converts a display unit into an off state when a predetermined time lapses, or maintains an Always On Display (AOD) state to provide only preset information. In this case, a power button should be clicked in order to wake up the display unit in an off state or an AOD state, and/or an additional operation to release a locked state preset for protection of privacy should be performed. Especially, when a user has a difficulty in using both hands, user's inconvenience is caused.

SUMMARY OF THE INVENTION

Therefore, an aspect of the detailed description is to provide a mobile terminal capable of easily performing a waking-up operation and releasing a locked state, through a user's intentional gesture to use the mobile terminal, and a method for controlling the same.

Another aspect of the detailed description is to provide a mobile terminal capable of logging a use's emotional state and a health state when a locked state of the mobile terminal is released, and a method for controlling the same.

Another aspect of the detailed description is to provide a mobile terminal capable of inputting a specific control command through a gesture together with a user's authentication, and a method for controlling the same.

To achieve these and other advantages and in accordance with the purpose of this specification, as embodied and broadly described herein, there is provided a mobile terminal, comprising: a terminal body; a camera; a sensor configured to sense a raising gesture of the terminal body; and a controller configured to: activate the camera in response to the raising gesture; detect a face within an image received via the camera; and release a locked state of the mobile terminal based on a face recognition of the detected face.

In an embodiment, wherein: the raising gesture corresponds to a movement of the mobile terminal which exceeds a threshold angle change and exceeds a threshold distance change within a reference time; the controller is further configured to change the mobile terminal from a first mode to a second mode in response to the raising gesture; and the face recognition is performed while the mobile terminal is in the second mode.

In an embodiment, the mobile terminal further comprises a touch screen, wherein the controller is further configured to activate the camera in response to the raising gesture when a degree of a first touch input received via the touch screen satisfies a predetermined condition.

In an embodiment, wherein the controller is further configured to: perform fingerprint recognition based on a second touch input to the touch screen while the face is being detected within the image received via the camera; and release the locked state based on a fingerprint recognition of the second touch input in addition to the face recognition of the detected face.

In an embodiment, the mobile terminal further comprises a touch screen, wherein the controller is further configured to control the touch screen to display a lock screen when the face is detected within the image received via the camera.

In an embodiment, wherein the controller is further configured to deactivate the camera when no face is detected within the image received via the camera within a preset time; and control the touch screen to display an updated notification related to an event at the mobile terminal.

In an embodiment, wherein: an indicator indicating that the face recognition is being performed is displayed on the lock screen; and the lock screen is changed to a lock release screen when the face recognition is successful.

In an embodiment, wherein: a notification indicating that the face recognition is successful is displayed on the lock release screen; and the controller is further configured to control the touch screen to display a lock release state setting menu in response to a touch input to the displayed notification.

In an embodiment, wherein the lock release screen is an execution screen of a preset application.

In an embodiment, the mobile terminal further comprises a touch screen, wherein the controller is further configured to: recognize a facial expression of the detected face during the face recognition of the detected face; and control the touch screen to display an execution screen of a particular application mapped with the recognized facial expression when the locked state is released.

In an embodiment, the mobile terminal further comprises a touch screen, wherein the controller is further configured to: recognize a facial expression of the detected face during the face recognition of the detected face; identify an emotion associated with a change in the recognized facial expression; and control the touch screen to display an image mapped with the identified emotion when the locked state is released.

In an embodiment, the mobile terminal further comprises a touch screen, wherein the controller is further configured to: determine biometric information of a user based on the detected face; and store the biometric information and control the touch screen to display a lock release screen when the locked state is released, wherein information related to the biometric information is displayed on the lock release screen for a predetermined time.

In an embodiment, the mobile terminal further comprises a touch screen, wherein the controller is further configured to control the touch screen to display: feedback information on a lock screen when the face recognition of the detected face is unsuccessful; and a screen related to another authentication method using the lock screen.

In an embodiment, wherein the raising gesture is sensed following an event at the mobile terminal, and the controller is further configured to: recognize a face gesture after the face recognition of the detected face is successful; and execute a particular action related to the event, wherein the particular action is mapped to the recognized face gesture.

In an embodiment, wherein the raising gesture is sensed following an event at the mobile terminal, and the controller is further configured to: cease output of notifications related to the event when a lowering gesture of the terminal body is sensed after the face recognition of the detected face is successful.

To achieve these and other advantages and in accordance with the purpose of this specification, as embodied and broadly described herein, there is provided a mobile terminal, comprising: a terminal body; a touch screen; a camera configured to maintain a constant activated state in a low-power mode; a sensor configured to sense a raising gesture of the terminal body, wherein the raising gesture corresponds to a movement of the mobile terminal which exceeds a threshold angle change and exceeds a threshold distance change within a reference time; and a controller configured to: detect a face within an image received via the camera in response to the raising gesture; control the touch screen to display a lock screen when the face is detected within the image received via the camera; and release a locked state of the mobile terminal based on a face recognition of the detected face.

In an embodiment, wherein: an indicator indicating that the face recognition is being performed is displayed on the lock screen; and the lock screen is changed to a lock release screen when the face recognition is successful.

DETAILED DESCRIPTION OF THE INVENTION

The mobile communication module112can transmit and/or receive wireless signals to and from one or more network entities. Typical examples of a network entity include a base station, an external mobile terminal, a server, and the like. Such network entities form part of a mobile communication network, which is constructed according to technical standards or communication methods for mobile communications (for example, Global System for Mobile Communication (GSM), Code Division Multi Access (CDMA), Wideband CDMA (WCDMA), High Speed Downlink Packet access (HSDPA), Long Term Evolution (LTE), and the like).

A magnetic sensor143indicates a sensor configured to detect an object approaching a predetermined surface or an object which exists nearby, and a position and a direction of the object, using a force of a magnetic field. That is, the magnetic sensor143indicates a sensor configured to measure a size and a direction of a peripheral magnetic field or a line of magnetic force. In the present invention, a plurality of 3-axis magnetic sensors143a,143bare provided at the mobile terminal100to more precisely sense a position and a direction of an object which generates a magnetic field.

For this, the plurality of 3-axis magnetic sensors143a,143bmay be independent from each other, and may be spaced from each other in different directions. The controller180may execute a differentiated operation based on a size of a magnetic field measured by the plurality of 3-axis magnetic sensors143a,143b. More specifically, the controller180may detect a position, a direction, an angle, etc. of an object which generates a magnetic field, based on a size of a magnetic field measured by the plurality of 3-axis magnetic sensors143a,143b.

A third camera121cmay be further provided on a front surface of the terminal body. In this case, the third camera121cmay be disposed near the first camera121a, or near the opposite side to the first camera121a(i.e., the optical output unit154).

The third camera121cmay be provided with a wide angle lens. In this case, the third camera121cmay support a wider viewing angle than the first camera121a. In a general angle mode, the first camera121amay be activated for capturing. And in a wide angle mode, the third camera121cmay be activated for capturing. An image captured by the third camera121cmay include a larger number of subjects than an image captured by the first camera121a. However, in this case, distortion occurs toward a peripheral part of a frame.

The third camera121cmay be activated together with or independently from the first camera121awhen the first camera121ais activated. If the third camera121cis activated together when the second camera121bformed on a rear surface of the terminal body is activated, an image captured by the third camera121cmay be used to compensate for a white balance reference value of an image captured by the first camera121a.

When a pick-up gesture to lift the terminal body is sensed while the terminal body is in a sleep mode, the aforementioned mobile terminal100according to an embodiment of the present invention activates the camera121provided at the terminal body in the sleep mode, thereby detecting a face which exists within a view angle range. Then, the mobile terminal100may release a locked state of the terminal body by performing a user's face recognition based on the detected face.

FIG. 2is a representative flowchart for explaining an operation of the mobile terminal according to the present invention.

Referring toFIG. 2, the mobile terminal100according to the present invention is operated in a sleep mode (S10). The sleep mode means a state where the controller180not only receives a wakeup signal and senses occurrence of a pickup gesture to be explained later, but also stops power supply to other components of the mobile terminal. In the sleep mode, the display unit151maintains an off state or an Always On Display (AOD) state to output only preset information (e.g., time information, etc.) with a low power.

In the sleep mode, the controller180may sense occurrence of a pickup gesture through the sensing unit140of the mobile terminal100(S20). More specifically, if occurrence of a pickup gesture is sensed through a combination of at least two such as a geomagnetic sensor, a gyroscope sensor and an acceleration sensor, the sensing unit transmits a wakeup signal to wake some components of the mobile terminal while an off state or an AOD state of the display unit151is maintained, to the controller180.

Hereinafter, such a wakeup signal may be referred to as a ‘limited wakeup’ signal in order to be differentiated from a wakeup signal for converting the display unit151into an on state. Therefore, unless otherwise specified, the ‘wakeup signal’ disclosed hereinafter means a signal to wake up a preset specific component (e.g., the camera) through power supply, while an off state of the display unit151is maintained.

Accordingly, if the aforementioned pickup gesture occurs, the controller180may perform only a limited operation in a sleep mode, i.e., while an off state or an AOD state of the display unit is maintained. In this case, battery consumption is relatively less than that when the sleep mode is released. This may reduce battery consumption due to frequent release of the sleep mode.

The pickup gesture may mean a movement to lift the terminal body within more than a preset angle range and by more than a preset distance. Further, the pickup gesture may mean not only a direct pickup gesture to lift the mobile terminal, but also a tilt gesture applied to the terminal body to incline the mobile terminal by a predetermined angle, and a pocket-out gesture to take the mobile terminal out of a pocket, a bag, etc. Hereinafter, a method to sense a pickup gesture will be explained in more detail with reference to the attached drawings.

In response to the occurrence of the pickup gesture, the controller180may activate the camera121in the sleep mode to detect a face which exists within a view angle range (S30). That is, the pickup gesture may be a limited wakeup signal for activating only the camera while maintaining the sleep mode of the mobile terminal.

For this, a face may be detected based on an image acquired by various types of image sensors. For instance, in case of a facial image, image information on a user's face may be acquired by a Complementary MOS (CMOS) image sensor, a Charge-Coupled Device (CCD) image sensor, an IR image sensor, etc. Then, a wakeup signal for converting the display unit to an on state may be transmitted to the controller180based on the acquired image information.

More specifically, if a face which is towards a front side based on the camera disposed on a front surface of the terminal body and configured to recognize a face has been detected, the controller180may determine a face detection. Further, the controller180may determine that a user's face has been detected only when the user's face is within a predetermined angle based on the camera151.

Here, various face detection algorithms may be used. For instance, this can be achieved by learning a matching template through a forward image.

As another example, the controller180may determine a face detection when a user's face is positioned within a predetermined distance from the camera121. For this, a distance between the camera121and the user's face spaced apart from the camera121may be measured through an additional sensor. For instance, the number of pixels of a face image according to a resolution is counted, and a user's face may be detected only when the number of the pixels exceeds a threshold value. Alternatively, a face may be recognized only when a size of a face image is within a preset range.

If a plurality of faces are detected through the camera121, only one face which satisfies a preset condition may be detected. For instance, a face image having a shortest separating distance from the camera121, a face image having a highest resolution, or a face image having a largest size may be detected from a plurality of obtained face images.

Once a specific face is detected, the display unit151is converted into an on state, and a lock screen may be displayed on the display unit151(S40). As a result, the sleep mode is completely released. On the other hand, if a face detection fails, the mobile terminal returns to the initial sleep mode. As a result, the camera121is converted into an inactivated state.

Continuously, the controller180performs a user's face recognition based on the detected face (S50). For this, it is assumed that a screen locking method of the mobile terminal is preset as a face recognition method.

The controller180compares a user's face pre-registered to the mobile terminal with the detected face. If it is determined that the user's face matches the detected face, the controller180releases a locked state. A user needs not perform any additional operation to release a locked state of the mobile terminal, and has only to maintain a gripped state of the mobile terminal for a predetermined time.

Then, a face recognition is successful after a predetermined time, the locked state of the mobile terminal is released (S60). As a result, the displayed lock screen is immediately converted into a lock release screen, for instance, a home screen. On the other hand, a face recognition fails, the controller180may induce an input of another recognition means through the display unit151. For instance, a screen requiring another screen locking method such as a pattern input, a PIN input and a fingerprint input may be output to the display unit151.

As aforementioned, in an embodiment of the present invention, a locked state of the mobile terminal such as a sleep mode may be released without an additional operation, through a user's gesture to intentionally lift the mobile terminal which is in the sleep mode. Further, since the sleep mode is completely released when a face is detected, battery consumption due to frequent release of the sleep mode is reduced. Further, since the user's gesture to intentionally lift the mobile terminal can be performed by one hand, there is not any difficulty due to a control using two hands which is appropriate for a locked state releasing input.

Hereinafter, a method of sensing a pickup gesture will be explained in more detail with reference toFIGS. 3A to 3E.

As aforementioned, the pickup gesture may be sensed through a sensor activated in a sleep mode of the mobile terminal, e.g., a geomagnetic sensor, a gyroscope sensor, an acceleration sensor, etc. Such sensors do not greatly influence on battery consumption due to their low battery consumption, even if they always maintain an activated state.

As aforementioned, the pickup gesture may be largely categorized into a direct pickup gesture to lift the mobile terminal (1), and a tilt gesture to incline the terminal body by a predetermined angle and a pocket-out gesture to take the mobile terminal out of a pocket, a bag, etc (2).

FIG. 3Ashows an example of a pickup gesture to lift the terminal body of the mobile terminal100.

In determining occurrence of such a pickup gesture, a state where the terminal body disposed on an object which satisfies a horizontal condition does not move for a predetermined time may be set as an initial condition.

The satisfying a horizontal condition may mean that a gradient of the mobile terminal which is disposed on an object, sensed through sensors, is within a range of about −15°˜+25° in an X-axis and about −25°˜+25° in a Y-axis. Further, if the mobile terminal which is within such a range has not moved for more than a predetermined time, e.g., about 2 seconds˜5 seconds, and if there is a subsequent gesture to lift the mobile terminal, it may be determined as occurrence of a pickup gesture. Here, the time for which the mobile terminal has not moved means a standby time for determining occurrence of a pickup gesture.

If an operation to lift the terminal body within more than a preset angle range and by more than a preset distance is sensed within a reference time, the controller180may determine that a pickup gesture has occurred. For instance, if the mobile terminal is upward moved by more than 3 cm at a speed of 30 cm/sec with satisfying a viewable angle, the controller180may determine that a pickup gesture has occurred.

As shown inFIGS. 3B to 3E, the viewable angle may mean that a gradient of the mobile terminal is within a range of about −40°˜+70° in an X-axis and about −40°˜+40° in a Y-axis. That is, the viewable angle means a range of a preset angle inclined for a user's face to be towards a front surface of the camera121. The viewable angle is a gradient of the mobile terminal required for face detection, and is applied to all of a direct pickup gesture, a tilt gesture and a pocket-out gesture.

As another example, in case of a tilt gesture, if the proximity sensor of the mobile terminal is not blocked by an object as a condition for prevention of a gesture mal-recognition, and if a gripped state of the terminal body is maintained for a predetermined time, the controller180may determine occurrence of a pickup gesture. This is a case to perform a pickup gesture while a user grips the mobile terminal which is in a sleep mode.

The context that the gripped state of the terminal body is maintained for a predetermined time may mean that a gradient of the mobile terminal sensed to be within a range of ±20° in an X-axis and a Y-axis is maintained for about more than 2 seconds.

Further, if a gripped state of the terminal body is maintained for a predetermined time, if a gradient of the mobile terminal is being inclined by about more than 35° at a proper speed towards a user's face based on a user's hand which has gripped the mobile terminal, and if a viewable angle is satisfied, the controller180may determine that a pickup gesture has occurred in the gripped state of the mobile terminal. Here, the proper speed may be about 120°/second, for instance.

As another example, in case of a pocket-out gesture, an additional limiting condition for prevention of a gesture mal-recognition may include a condition which does not satisfy a horizontal state of the mobile terminal while the proximity sensor is blocked. In this state, if there is not an object which blocks the proximity sensor and if a viewable angle is satisfied according to a pocket-out gesture, the controller180may determine that a pickup gesture has occurred.

A user may change the aforementioned viewable angle through an input. For instance, the user may prevent another person's face from being detected together through the camera121, by making a preset angle range more steep.

For instance, if a wakeup signal generated as the pickup gesture occurs is transmitted to the controller180in a first sleep mode of the mobile terminal, the first sleep mode may be converted into a second sleep mode for sensing a face.

The first sleep mode may mean an operation state of the mobile terminal which has a smaller amount of battery consumption than in the second sleep mode. In the first sleep mode, the controller180may not only receive a wakeup signal, but also stop power supply to other components. Alternatively, the first and second sleep modes may mean that an off state of the display unit151is maintained.

For instance, if no face is detected within a predetermined time in the second sleep mode, the controller180may immediately convert the second sleep mode into the first sleep mode without an additional input, in order to minimize battery consumption.

FIGS. 4A and 4Bare conceptual views for explaining different examples of an additional limiting condition to enhance a recognition rate of a pickup gesture, in the mobile terminal according to an embodiment of the present invention.

When a pickup gesture is performed, a touch for gripping the mobile terminal at least occurs on a front surface or a side surface of the terminal body. As an additional limiting condition to enhance a recognition rate of a pickup gesture, may be used the touch screen151including a touch sensor disposed on a front surface of the terminal body, and/or an edge or a side frame disposed on a side surface of the terminal body and having a grip sensor for sensing a touch pressure. Although not shown, if a grip sensor is provided on a rear surface of the terminal body, whether a user has gripped the mobile terminal or not may be recognized through the grip sensor provided on the rear surface of the terminal body.

FIG. 4Ashows a case to determine occurrence of a pickup gesture, in a condition that a touch401having a pressure exceeding a reference pressure is applied to the touch screen151.

More specifically, the controller180may determine whether to activate or deactivate the camera121, based on a touch degree of a touch input applied to the touch screen151when a pickup gesture occurs or after a pickup gesture occurs. For instance, if a touch input having a pressure exceeding a reference pressure is applied to the touch screen151when a pickup gesture occurs or after a pickup gesture occurs, the controller180may determine that a user's intentional pickup gesture has occurred. This may prevent an operation to detect a face due to a malfunction.

InFIG. 4A, it is assumed that a touch input having a pressure exceeding a reference pressure is applied to a region on the touch screen151. However, it is also possible to determine whether a user intentionally grips the mobile terminal or not, by combining a touch sensed on the touch screen151with sensing values sensed by a grip sensor disposed on a side surface and a rear surface of the mobile terminal, for a more natural pickup gesture.

For instance, the controller180may perform a fingerprint recognition based on a touch input applied to the touch screen151while a face detection is performed through the activated camera121. For this, if a touch input having a pressure exceeding a reference pressure is applied to a region on the touch screen151, a fingerprint sensor may be activated based on the region.

In this case, the controller180may release the locked state based on a result of the aforementioned face recognition and the fingerprint recognition. Since user's two recognitions (authentications) for releasing the locked state are simultaneously performed, a user needs not perform an additional input with more enhanced security.

FIG. 4Bshows that a gripped state of the mobile terminal by a user's one hand is sensed by a grip sensor provided on a side surface and/or a rear surface of the terminal body.

In an embodiment, if the aforementioned pickup gesture occurs as a user's grip is sensed by a grip sensor even if a horizontal state is not maintained for a predetermined time, the controller180may determine that a pickup gesture for activating the camera has occurred.

If there occurs a tilt gesture as a user's grip is sensed by the grip sensor even if the proximity sensor is temporarily blocked, the controller180may determine that a pickup gesture for activating the camera has occurred.

In case of a pocket out gesture, occurrence of a pickup gesture is determined by checking whether the mobile terminal has fallen from a pocket, a bag, etc. through the grip sensor. This may prevent activation of the camera due to a malfunction.

FIGS. 5A and 5Bshow an operation to detect a face by activating the camera as occurrence of a pickup gesture is sensed.

If the camera121of the mobile terminal is activated as a pickup gesture occurs, the controller180controls a face image to be detected based on a face detection algorithm stored in the mobile terminal. If a surrounding environment has an illumination of 20 lx or more based on a front of a face, the face is detected with a recognition rate of about 95% or more.

Here, the controller180may control a face which is positioned within a predetermined distance from the camera121, e.g., 20 cm˜40 cm, to be detected. Further, when a plurality of faces are detected, the controller180may control only one face which satisfies a preset condition, to be detected. While a face detection is being performed, the camera121is in an activated state but the display unit151is in an off state. That is, a user cannot visually check whether the camera121is in an activated or inactivated state.

While a face detection is being performed, the mobile terminal should not move. However, during a face detection, a slight movement of the mobile terminal up and down or right and left may be allowable. Exceptionally, if a slight movement has a rapid speed or is repeated, a face detection may fail. While a face detection is being performed, a user's face should be towards a front side, and eyes, a nose, a mouth and a face outline should be within a range of a view angle of the camera121. As shown inFIGS. 5A and 5B, a face detection has a high recognition rate within a range of ±10° in right and left directions of the mobile terminal, and within a range of ±10° in upper and lower directions of the mobile terminal. Thus, when the mobile terminal moves out of such a range, a face detection may fail.

It takes about 1 second for the camera to be activated to detect a face. And the display unit151is automatically turned on when a face detection is successful. If the mobile terminal is not in a locked state, a home screen is immediately output to the display unit151. On the other hand, if the mobile terminal is in a locked state, the display unit151is turned on according to a face detection, and a lock screen is output.

Although not shown, even while a face detection is being performed through a camera activation, a user may wake up a sleep mode at any time by pushing a power button or a home button. In this case, a face detection processor may be stopped, and a locked state may be released by a preset method to release the locked state (e.g., a pattern input or a PIN input).

FIGS. 6A˜6C and9show various screen changes related to a face detection and a face recognition, in the mobile terminal according to an embodiment of the present invention.

FIG. 6Ashows that the camera is activated as a pickup gesture occurs and a face detection fails, when the mobile terminal is in an AOD state as a sleep mode (601). In the present invention, while a face detection is being performed as a pickup gesture occurs and the camera is activated, the display unit151has no screen change.

However, if a face detection fails within a predetermined time, a command for notification update may be performed on the AOD screen601, as shown inFIG. 6A. More specifically, if a face detection fails within a predetermined time, the controller180may convert the camera121into a deactivated state, and may control the touch screen151to display a notification informing an event occurrence in an updated manner.

Even if an event occurs, a notification may be output at preset time intervals, or a notification may be delayed until a sleep mode is released, according to a type of the event. In some cases, a user may not check a previous notification of an occurred event. Accordingly, in the present invention, the controller180may output a refresh notification with respect to information stored in an AOD state (e.g., a message arrival), without turning on the display unit151, under an assumption that a pickup gesture has resulted from a user's intention to check a time or an event occurrence, not a user's intention to use the mobile terminal.

Accordingly, as shown inFIG. 6A, a highlighting effect may be output to the notification icon601einforming a message arrival, or a preset notification such as vibrations and a sound may be output thereto. Although not shown, notifications of other occurred events may be displayed on the AOD screen601. Then, after a predetermined time lapses, the mobile terminal is converted into a sleep mode.

In the aforementioned embodiment, since a user needs not frequently release a sleep mode in order to check simple information or an event, battery consumption may be more reduced.

FIG. 6Bshows an example of a screen change when both of a face detection and a face recognition succeed.

As shown inFIG. 6B, when a face detection is successful, the display unit151is turned on and a lock screen602is output. In this case, an indicator611indicating that a user's face recognition is being performed, e.g., ‘Face recognition will be performed.’ is displayed on the lock screen602. If a face recognition is successful while a user maintains a grip state for a predetermined time, the lock screen602is converted into a lock release screen603.

As another example, if a face recognition is successful, as shown inFIG. 6C, notification information612indicating a successful face recognition may be output to a lock release screen, e.g., a home screen604.

In an embodiment, if a touch input is applied to the output region of the notification information612before the notification information612disappears, a setting menu620related to maintenance of a lock released state of the mobile terminal may be output to one region of a lock release screen604″, e.g., a lower end.

The setting menu620may include a first menu icon to set whether to maintain a lock released state or not, a second menu icon to set a face recognition time interval for extending a lock released state, etc. The setting menu620may disappear when a preset touch gesture (e.g., a downward drag input) is applied to the output region of the setting menu620.

FIG. 9shows an example to process when a face recognition fails. As shown inFIG. 9, a face recognition screen801is output to the display unit151. Then, if a face recognition fails, a screen related to another recognition means is provided to the display unit151together with feedback information812. For instance, as shown inFIG. 9, a screen804requiring for input of a pin number may be output.

Although not shown, a user may preset a type of a lock release screen displayed when a face recognition is successful. For instance, the lock release screen output when a face detection and a face recognition are successful may be an execution screen of a preset application, e.g., an execution screen of a call application.

In the aforementioned embodiments, the camera is activated as a pickup gesture occurs. However, in an embodiment, the camera121of the mobile terminal may always maintain an activated state in a low power mode (e.g., Always-On Camera). In this case, if an operation time to activate the camera does not exist when a pickup gesture occurs, a face recognition is performed based on a detected face or a previously detected face, at the same time when the pickup gesture occurs. As a result, time taken to release a locked state is more reduced.

As another example, various operations based on a facial expression change during a face recognition will be explained in more detail with reference toFIGS. 7A˜7C and8A˜8B. In the following examples, various analysis results may be provided as a facial expression change is recognized from a face image obtained through the camera121.

A face lock setting screen701of the mobile terminal shown inFIGS. 7A and 7Bis an enhanced face recognition menu, and may provide a face quick app setting menu712. Through the face quick app setting menu712, executions of different applications may be mapped with user's various registered facial expressions (702). For instance, during a facial recognition, an execution of a first application may be mapped with a winking expression, and an execution of a second application may be mapped with a blinking expression. And an execution of a third application may be mapped with a smiling expression, and an execution of a fourth application may be mapped with a yawning expression. Such mapping information may be stored in the memory170of the mobile terminal100through a user's input, etc.

As another example, various operations such as terminating a specific application or activating a specific configuration of the mobile terminal (e.g., activation of a recording function), as well as executing an application may be mapped with a specific facial expression.

More specifically, if a preset facial expression is sensed during or after the user's face recognition, the controller180may control a lock screen corresponding to a lock state or a lock release screen corresponding to a lock released state, to be converted into an execution screen of an application mapped with the sensed facial expression. In the former case, an execution screen of a mapped application is immediately output to a lock screen. On the other hand, in the latter case, a lock release screen such as a home screen is output, and then the execution screen of the mapped application is output.

The former case is shown inFIG. 7C. A lock screen721, displayed when a face recognition is successful and a preset facial expression is sensed, is immediately converted into an execution screen of an application mapped with a preset facial expression, e.g., a specific shopping mall web page screen722.

FIGS. 8A and 8Bshow examples to log-in an emotional state or a health state through a facial expression change which can be analyzed during a facial recognition, not a preset facial expression.

For this, an image recognition may be used in the present invention. The image recognition is a technology to analyze an object disposed on an image. As an example of the image recognition, an emotion recognition API is an algorithm to analyze a facial expression and an emotional state. In the present invention, a facial expression change may be analyzed by various emotion recognition algorithms including an emotion recognition API, through a server, etc. stored in the memory170of the mobile terminal or interworked with the memory170.

For instance, the controller180may recognize a user's emotional state based on a facial expression change during the user's face recognition (801). Then, as shown inFIG. 8A, if the locked state is released, the controller180may provide an image mapped with the recognized emotional state, e.g., a crying image811, to a lock release screen802. Such an emotional state is continuously logged to be stored and collected in the memory170, etc., in the form of data.

As another utilization, when a user uploads a text to an SNS, etc. or chats with an acquaintance on messenger, the image811corresponding to the user's emotional state may be recommended to be uploaded together or to be transmitted. Further, recommendation information for recovering an emotional state to a normal level may be provided based on a history of continuously logged-in facial expression changes. Alternatively, an application may be driven for attempt to dialogue with the mobile terminal, in an extended utilizable manner.

As another example, the controller180may acquire user's bio information based on a facial image analyzed during a user's face recognition. Then, when a locked state is released, the controller180may log-in the acquired user's bio information, and may provide corresponding screen information, e.g., a heart rate measuring screen803to a lock release screen for a predetermined time. For instance, as shown inFIG. 8B, a heart rate may be measured based on a micro change degree of a face color, by using a face heart rate measuring algorithm implemented based on a face color change when a heart beats during a face recognition. For this, the display unit151may be turned on when a face detection is successful, and at the same time, a heart rate measuring application may be driven.

With such a configuration, a user may measure health-related data during a face recognition, without measuring a heart rate by executing an additional application. Further, a heart rate measured whenever a locked state of the mobile terminal is released is accumulatively logged, and may be used as an index to continuously manage a health.

Further, since it takes a relatively short time to perform a face recognition, screen information to continue a heart rate measurement may be output to a lock release screen, or a screen to additionally measure health-related data may be output to the lock release screen.

Hereinafter, examples to input a control command according to another embodiment related to a face recognition will be explained in more detail with reference toFIGS. 10 and 11.

The aforementioned pickup gesture may be performed in response to occurrence of an event from one or more applications installed on the mobile terminal. For instance, as shown inFIG. 10, even if the mobile terminal is in a sleep mode, a pickup gesture may occur while the sleep mode is released, as a specific event such as a call reception occurs.

In this case, as aforementioned, a face is detected as the pickup gesture occurs, and a face recognition is performed based on the detected face. In the present invention, there is provided a novel type of user interface which determines a subsequent face gesture as an input of a control command, by considering a pickup gesture according to an event occurrence. More specifically, the controller180may perform a user authentication based on a detected face, and may process an event based on a control command corresponding to a subsequent face gesture.

Referring toFIG. 10, as a call is received, a call receiving screen1001is output to the display unit151. Then, if a pickup gesture is performed, the camera is activated on a background of the mobile terminal to perform a face detection and a face recognition. Once the face recognition is successful, a face gesture as well as a button input is used as a user interface to input a control command.

Accordingly, if a user rotates the recognized face to the right by a predetermined angle, in correspondence to a position of a hang up button on the call receiving screen1001, instead of pressing the hang up button, the controller180recognizes such a face gesture through the camera121to perform a hang up command. As a result, the call receiving screen1001may be converted into a call recording screen1003as shown inFIG. 10, or a screen (not shown) informing a missed call.

On the other hand, if a user rotates the recognized face to the left by a predetermined angle, in correspondence to a position of a call button on the call receiving screen1001, instead of pressing the call button, the controller180recognizes such a face gesture through the camera121to perform a call command. As a result, a call is connected and a call screen1002is output. In this case, a speaker may be activated together with the call connection. That is, another person's voice may be output through the speaker such that the user may perform a call while maintaining a grip state of the mobile terminal by one hand.

As another example, if a subsequent put-down gesture is sensed when a face recognition is performed in response to an occurred event, the controller180sets a notification corresponding to the event not to be output. The put-down gesture means an operation to put the mobile terminal down after a pickup gesture occurs.

For instance, if there occurs a gesture to put down the mobile terminal by a predetermined angle at a proper speed within a proper time, the gesture may be determined as a put-down gesture. Further, in case of putting the mobile terminal on a floor in an overturning posture such that the touch screen151is towards a lower side, it may be determined that a put-down gesture has occurred. Referring toFIG. 10, if a put-down gesture occurs over the call receiving screen1001, even if a call is additionally received from the same person, a notification (e.g., vibrations, a phone ring tone, etc.) may not be output.

FIG. 11shows an input of a control command to change an image sorting through a face recognition, to an execution screen of a gallery application.

Referring toFIG. 11, in a state that a list1101of a plurality of thumbnail images is output without a specific sorting reference as a gallery application is executed, a user's pickup gesture may occur, and then a face detection and a face recognition may be performed. As a result, the list1101may be changed into a list1102of photo images including the recognized face, according to a subsequent face gesture (e.g., a nodding gesture).

As aforementioned, the mobile terminal and the method for controlling the same according to the present invention can provide a UX to perform a wakeup function and a lock release function of the mobile terminal, through a user's gesture to lift the mobile terminal without a specific manipulation. Further, a user's emotional state, health state, etc. may be checked and logged based on a facial expression change recognized during a face recognition. Further, since a control command is generated based on a subsequent face gesture together with a face recognition, a new input method can be provided.

The foregoing embodiments are merely exemplary and are not to be considered as limiting the present disclosure. The present teachings can be readily applied to other types of methods and apparatuses. This description is intended to be illustrative, and not to limit the scope of the claims. Many alternatives, modifications, and variations will be apparent to those skilled in the art. The features, structures, methods, and other characteristics of the exemplary embodiments described herein may be combined in various ways to obtain additional and/or alternative exemplary embodiments.