Patent Description:
Recent advances in communication represented by smartphones, information processing technology, camera imaging technology and semiconductor technology have been remarkable, and with the development and expansion of such environments, technology for performing operations more smoothly than operations in portable terminals of a related art has been developed.

For example, an advanced man-machine interface system in which a voice recognition function is installed in a smartphone, and a smartphone is operated by voice or the like has also been introduced. Further, a touch pad that enables an intuitive operation by an icon is generally installed in smartphones. The reason why the operation by the touch pad is common is that an operation is stable, and it can be manufactured with inexpensive parts.

However, even though operability is excellent, in the case of the operation by voice, it is likely to be affected by a surrounding noise or other people's voice, and in the case of the operation by the touch pad, it is likely that a hand slips so that another nearby icon is operated.

As a technique related to prevention of an erroneous operation, for example, a technique of setting a line of sight effective range for detecting whether or not it is within a line of sight range effective for an operation by detecting a user's line of sight, disabling an operation when a key is operated, and the user's line of sight is not within the effective range, and preventing an erroneous operation is disclosed in Patent Document <NUM>.

<CIT> discloses a method and system for assisting a user when interacting with a graphical user interface which combines gaze based input with gesture based user commands.

The erroneous operation of the key can be prevented using the technique disclosed in Patent Document <NUM>. However, such a technique is likely to impair the operability of smartphones and is difficult to apply to smartphones.

It is an object of the present invention to provide a technique of preventing an erroneous operation without impairing the operability in a portable terminal such as a smartphone.

A representative portable terminal according to the present invention includes a touch pad that displays an icon and detects a touch, a camera that images a direction in which light of a display of the touch pad travels, a detecting unit that detects a point of view to which an eye of a face imaged by the camera is turned, an information management unit that outputs a position of the point of view detected by the detecting unit on the touch pad, a cursor display unit that performs control such that a cursor is displayed on the touch pad according to the position of the point of view on the touch pad output from the information management unit, an input condition management unit that determines whether or not a preset input condition is satisfied for an input by a touch to a preset special region of the touch pad and an input of the position of the point of view on the touch pad output from the information management unit, and an application program executing unit that activates an application program corresponding to the icon according to the determination of the input condition management unit.

According to the present invention, it is to provide a technique of preventing an erroneous operation without impairing the operability in a portable terminal such as a smartphone.

Hereinafter, exemplary embodiment of the present invention will be described with reference to the appended drawings.

<FIG> is a diagram illustrating an example of a smartphone <NUM> and an operation thereof. In the example of <FIG>, a user <NUM> who operates the smartphone <NUM> is holding the smartphone <NUM> with one hand <NUM> since one hand is holding onto a strap <NUM> in a transportation system such as a train or bus, and so one hand is occupied.

The user <NUM> looks at the smartphone <NUM> with eyes <NUM> with the face facing the smartphone <NUM> in order to operate the smartphone <NUM>. The smartphone <NUM> includes a touch pad (display screen) <NUM>, a camera <NUM>, a camera <NUM>, a side button <NUM>, and a microphone <NUM>, and the user <NUM> looks at the touch pad <NUM>.

The camera <NUM> and the camera <NUM> image the face of the user <NUM>, especially, eyes <NUM> of the user <NUM>. Then, a distance between the face and the touch pad <NUM> is calculated by imaging with the two cameras, the line of sight is specified from a positional relation of the white of the eyes <NUM> and the apple of the eye, and a point of view in the touch pad <NUM> is obtained, but it will be further described later.

In the example of the smartphone <NUM> illustrated in <FIG>, two cameras are disposed, but one camera may be disposed if the distance between the face and the touch pad <NUM> can be detected by another sensor. For example, instead of the camera <NUM> on the same surface as the touch pad <NUM>, an infrared or ultrasonic distance sensor may be disposed at the position of the camera <NUM> to measure the distance from the face of the user <NUM>.

Further, in a case in which it is difficult to identify the positional relation between the white of the eye of eyes <NUM> and the apple of the eye and specify the point of view, for example, since the user <NUM> is wearing sunglasses or the like, other inputs may be used instead of it. However, in order to determine whether or not the face of the user <NUM> is facing the touch pad <NUM>, a contour of the face, for example, a contour <NUM> including hair and the ears may be identified.

<FIG> is a diagram illustrating an example in which icons <NUM> are displayed on a smartphone <NUM>. The example in <FIG> is a display called a home screen or a standby screen, and the icons <NUM> are displayed on the touch pad <NUM>. Further, in a case in which any one of icons <NUM>-<NUM> to <NUM>-<NUM> is not specified, these are represented by an icon <NUM>, and hereinafter, the same applies to other reference numerals.

Each of the icons <NUM>-<NUM> to <NUM>-<NUM> is an icon which is set in association with an application program in advance, and the application program corresponding to the icon <NUM> selected among them is executed. A cursor <NUM> is displayed at the point of the view position of the user <NUM>. That is, it indicates a cursor position at which the point of view is fed back to the user is illustrated.

First, the line of sight of the user <NUM> vaguely captures a point on touch pad <NUM>. In a case in which there is an incoming call for the smartphone <NUM>, an activation icon of an application program that automatically gives a response indicating that it is unable to transition to a reception state is an icon <NUM>-<NUM>, the user <NUM> moves the line of sight to align the cursor <NUM> with the icon <NUM>-<NUM>.

Then, the smartphone <NUM> specifies that there is the point of view on the icon <NUM>-<NUM>, and activates the application program corresponding to the icon <NUM>-<NUM> if a switch-on operation of the side button <NUM> or a so-called "decision" operation is detected. Accordingly, it is possible to select and activate the application with a one-hand operation.

In <FIG>, the user <NUM>'s action of holding onto the strap <NUM> is used as an example, but there are many cases in which the operation by only one hand is required such as a case in which the user is carrying luggage, holding materials with a hand, walking while holding a hand with a child, or has an injured one hand, and in these cases, the operation described above is effective.

Further, if it is specified that the point of view has moved to the icon <NUM>-<NUM>, the smartphone <NUM> may perform a highlight display by changing a display color of the icon <NUM>-<NUM>, a display size, or a display shape in addition to moving the cursor <NUM> onto the icon <NUM>-<NUM>. Accordingly, the user <NUM> can confirm that the icon <NUM>-<NUM> is selected more reliably.

Also, if it is specified that the point of view has moved onto one of the icons <NUM> without displaying the cursor <NUM>, the smartphone <NUM> may change the display color of the icon <NUM>, the display size, or the display shape.

Further, in the example of <FIG>, the shape of the cursor <NUM> is indicated by an arrow, but in order to emphasize a meaning indicating a part at which the line of sight stays, for example, a target mark in which a circle and a cross are combined may be used or the like.

In the above description, an example in which the application program is activated in the standby screen has been described, but the process of the operation described above can also be used for an operation while the application program is being executed. For example, the icon <NUM> illustrated in <FIG> may be a number key for inputting a telephone number, and a number may be input when the icon <NUM> is selected while an application program related to a telephone is being executed.

Also, for example, in a case in which an automatic answering function is used, it may be required to input a number according to a synthesized voice message. In other words, a state in which numbers can be input from the touch pad and a state in which a synthesized voice message output from the smartphone is heard may overlap.

On the other hand, in the smartphone <NUM>, since the icon <NUM> which is the number key is selected by the point of view of the user <NUM> by the cameras <NUM> and <NUM>, even when the ear touches the touch pad <NUM> to hear a message, the icon <NUM> which is the number key is not selected, and a situation in which an unintentional touch pad input is received does not occur.

Further, a situation in which an unintentional input caused by an unconscious touch to the touch pad <NUM> is received does not occur in the middle of an action of putting the smartphone <NUM> into a pocket in the standby screen or while the application program is being executed. This is because the selection of the icon <NUM> by the point of view is not performed, further, the switch-on operation of the side button <NUM> is not performed, and an intended condition of decision (execution) of the user is not satisfied.

As described above, even though one type of operation occurs by mistake as one instruction such as decision of an application program or the number key is input by a combination of two types of operations, it does not lead to an erroneous instruction.

The point here is that great effects such as erroneous input recognition prevention and usability improvement by only one-hand operation can be obtained by the relatively simple software process if the camera function that is a standard built-in function of a smartphone such as user face recognition and line of sight or point of view detection is provided.

An example of executing other application programs in the smartphone <NUM> will be described. <FIG> is a diagram illustrating an example in which a text is displayed on a smartphone <NUM>. The text includes a sequence of words, and for example, the user <NUM> read words <NUM> in the order of a word <NUM>-<NUM>, a word <NUM>-<NUM>, a word <NUM>-<NUM>, and a word <NUM>-<NUM> while moving the point of view.

Here, in a case in which the user <NUM> desires to copy the word <NUM>-<NUM>, the point of view of the user <NUM> is at the word <NUM>-<NUM>, and the smartphone <NUM> identifies the point of view. Here, the microphone <NUM> receives an operation by voice as the side button <NUM> is switched on. The operation by voice is, for example, "copy.

On the other hand, the smartphone <NUM> copies the word <NUM>-<NUM> and pastes it on a clipboard. Also, the operation by voice may be another type of operation or may be an operation for text editing.

As described above, even though one type of operation occurs by mistake as two instructions such as selection and copy of the word <NUM>-<NUM> are input by a combination of three types of operations, it does not lead to an erroneous instruction.

Next, an example of a tablet terminal is described. <FIG> is a diagram illustrating an example of a display of a tablet terminal <NUM>. The tablet terminal <NUM> includes a touch pad (display screen) <NUM>, a camera <NUM>, a camera <NUM>, a decision button <NUM>, and a microphone <NUM>, and the user looks at the touch pad <NUM>.

These correspond to the touch pad <NUM>, the camera <NUM>, the camera <NUM>, the side button <NUM>, and the microphone <NUM> of the smartphone <NUM> described above, a cursor <NUM> corresponds to the cursor <NUM>, activation of an application program with the displayed icon <NUM> or input of the number while an application program is being executed are similar to those described above, and a process for words with the displayed text is also similar to that described above.

However, in the activation of the application program with the displayed icon <NUM>, instead of pushing the decision button <NUM>, the microphone <NUM> may receive "execute" as an operation by voice.

The instructions by such operations can also be used for information search. In the example illustrated in <FIG>, in photograph information in which three persons, that is, a person <NUM>-<NUM>, a person <NUM>-<NUM>, and a person <NUM>-<NUM> are subjects, in a case in which the user desires to know person information of the person <NUM>-<NUM>, when the user turns the line of sight to the display of the person <NUM>-<NUM>, and the cursor <NUM> is moved to the display of the person <NUM>-<NUM>, the user pushes the decision button <NUM> or say "search for information" to the microphone <NUM>.

On the other hand, when the user's point of view is specified to move the cursor <NUM>, and the decision button <NUM> is pushed or an operation by voice such as "search for information" is received by the microphone <NUM>, the tablet terminal <NUM> performs face recognition on an image at a position at which the point of view is specified, that is, a position at which the cursor <NUM> is being displayed, searches information stored in the tablet terminal <NUM> or information on the Internet, and output found results by voice synthesis or displays the found results on the touch pad <NUM>.

When the user's point of view is specified to move the cursor <NUM>, an operation in which the decision button <NUM> is pushed is received, and an operation by voice such as "search for information" is received by the microphone <NUM>, the tablet terminal <NUM> may perform face recognition.

An information search target may be map information. <FIG> is a diagram illustrating an example in which map information is displayed on the tablet terminal <NUM>. In the tablet terminal <NUM> illustrated in <FIG>, a down button <NUM> and an up button <NUM> are added to the tablet terminal <NUM> illustrated in <FIG>. The down button <NUM> and the up button <NUM> are used to adjust a sound output volume but are also used to adjust image zooming.

First, the user performs an operation so that an "Africa map" is displayed, and in <FIG>, the Africa map is displayed on the touch pad <NUM>. The operation for displaying the "Africa map" may be performed when a voice "Africa map" is input to the microphone <NUM> or may be performed when the decision button <NUM> is pushed together or a specific region of the touch pad <NUM> is touched together with this voice.

Then, if the user aligns the point of view with a part of the Africa map and pushes the up button <NUM>, the tablet terminal <NUM> specifies the point of view from the image of the user obtained by the camera <NUM> and the camera <NUM> in accordance with the operation of the user operation, displays the cursor <NUM> according to the position of the point of view, and zooms up the map centering on the cursor <NUM> which is the position of the point of view as illustrated in <FIG> if it is detected that the up button <NUM> is pushed.

Accordingly, a part which the user desires to see in order to confirm a further detailed position is zoomed up, and thus the user can observe a part which the user desires to know in detail.

Further, if it is detected that the down button <NUM> is pushed instead of the up button <NUM>, the tablet terminal <NUM> zooms down the map centering on the cursor <NUM> which is the position of the point of view. Further, when it is detected that the up button <NUM> or the down button <NUM> is pushed and an operation by voice such as "zoom up" or "zoom down" is received the microphone <NUM>, the zoom up or zoom down process may be performed on the map.

Further, in the display illustrated in <FIG>, if it is detected that the decision button <NUM> is pushed, the tablet terminal <NUM> specifies a country indicated by the cursor <NUM> that is the position of the point of view on the map, searches for information related to the specified country from a storage device in the tablet terminal <NUM> or the Internet, and displays the searched information on the touch pad <NUM> as illustrated in <FIG>.

Here, when it is detected that the decision button <NUM> is pushed and an operation by voice such as "detailed information" is received by the microphone <NUM>, the tablet terminal <NUM> may execute the search process and the display process.

In the information search based on the map information, map display and search for surrounding information according to the position of the tablet terminal <NUM> may be enabled based on a global positioning system (GPS). If a map application program is activated, the tablet terminal <NUM> displays a map centered on a current position based on the GPS.

<FIG> is a diagram illustrating an example in which map information is displayed on the tablet terminal <NUM>. A touch pad <NUM>, a camera <NUM>, a camera <NUM>, a decision button <NUM>, a down button <NUM>, an up button <NUM>, a cursor <NUM>, and a microphone <NUM> are similar to those described above. A four-direction arrow for scrolling the map and a current position mark <NUM> are also displayed on the touch pad <NUM>.

If the point of view is aligned with an up arrow <NUM>, and the decision button <NUM> is pushed in a state in which the current position mark <NUM> is displayed at the center of the map, the map is scrolled down, and a display screen illustrated in <FIG> is obtained. In <FIG>, the current position mark <NUM> is also moved down as the map is scrolled down.

In the display screen illustrated in <FIG>, if the point of view of the user is aligned with a department store <NUM>, the cursor <NUM> is displayed to indicate the department store <NUM>, and if the decision button <NUM> is pushed, a display screen illustrated in <FIG> is displayed. Accordingly, the user can obtain in advance information of each floor of the department store to visit.

A calibration and distance measurement for specifying the point of view will be described below with reference to <FIG> and <FIG>. If the calibration is started, as illustrated in <FIG>, first, a mark <NUM>-<NUM> flashing at the center as in a display screen <NUM>-<NUM> is displayed in a display screen <NUM> of the touch pad <NUM> or the touch pad <NUM>.

A captured image <NUM>-<NUM> by the camera <NUM>, the camera <NUM>, the camera <NUM>, or the camera <NUM> is acquired for the display screen <NUM>-<NUM>. Here, as will be described later with reference to <FIG>, a distance to a subject <NUM>, that is, the user may be measured using a parallax of the two cameras <NUM> and <NUM> or the two cameras <NUM> and <NUM>. It is desirable for a magnification of the camera in the captured image <NUM>-<NUM> to be a degree at which the upper body of the subject <NUM> fits.

Then, a mark <NUM>-<NUM> flashing in the lower left corner is displayed as in a display screen <NUM>-<NUM>. At this time, a message such as "pay attention to flashing shape" may be output as a synthesized voice. Further, the magnification of the camera is increased, an eye part <NUM> of the subject <NUM> is closed up, and a captured image <NUM>-<NUM> is acquired.

Then, a mark <NUM>-<NUM> flashing in the upper right corner is displayed as in a display screen <NUM>-<NUM>. A captured image <NUM>-<NUM> in which the eye part <NUM> of the subject <NUM> is closed up with the same magnification of the camera from which the captured image <NUM>-<NUM> is acquired, and a correlation of eye patterns of the captured image <NUM>-<NUM> is acquired and the captured image <NUM>-<NUM> is obtained.

That is, as the line of sight is turned to a position far from the center of the display screen <NUM>, a position <NUM> of a pupil or an iris in a contour <NUM> of the eye is displaced away from the center of the contour <NUM> of the eye. A ratio of a displacement amount of a captured image newly obtained to specify the point of view actually after the calibration from end points of the display screen may be calculated as the position of the point of view from a ratio relative to a reference of displacement on the basis of a displacement between a position <NUM>-<NUM> of a pupil or an iris to the contour <NUM>-<NUM> of the eye and a position <NUM>-<NUM> of a pupil or an iris to the contour <NUM>-<NUM> of the eye when the end points of the display screen <NUM> (the mark <NUM>-<NUM> and the mark <NUM>-<NUM>) are viewed.

Also, the angle of the line of sight between the captured image <NUM>-<NUM> and the captured image <NUM>-<NUM> may be calculated from the distance between the mark <NUM>-<NUM> and the mark <NUM>-<NUM> and the distance between the smartphone <NUM> (or the tablet terminal <NUM>) and the user (the subject <NUM>) measured when the captured image <NUM>-<NUM> is acquired, the ratio of the angle of the line of sight may be calculated from the ratio of the displacement amount of the newly obtained captured image relative to the reference of the displacement, and the position of the point of view may be calculated from the angle of the line of sight and the distance when the newly captured image is acquired.

Further, in the description using <FIG>, the process for one eye has been described, but the process may be performed for each of both eyes, and then averaging or the like may be performed. Also, the mark <NUM>-<NUM> and the mark <NUM>-<NUM> may be displayed without displaying the mark <NUM>-<NUM>. In addition to the display of the mark <NUM>-<NUM> and the mark <NUM>-<NUM> and the acquisition of the captured image <NUM>-<NUM> and the captured image <NUM>-<NUM>, a mark may be displayed even on the upper left and lower right of the display screen <NUM>, and the respective captured images may be acquired.

<FIG> is a diagram illustrating an example of distance measurement by two cameras. An example in which a point <NUM> which is a center position of both eyes of the subject <NUM> is detected by face recognition, and distance measurement is performed using the detected point <NUM> will be described. Further, to simplify the description, the distance measurement in a state in which the user who is the subject <NUM> is facing the front of one of the two cameras.

In <FIG>, a positional relation in which a portable terminal <NUM> images the subject <NUM> is illustrated in an xy plane as seen from above a z axis. The portable terminal <NUM> is the smartphone <NUM> or the tablet terminal <NUM> and includes a lens <NUM>, a lens <NUM>, an imaging element <NUM>, and an imaging element <NUM>.

In the imaging element <NUM>, the subject <NUM> including the point <NUM> is imaged through the lens <NUM>. Here, the point <NUM> is a point <NUM> on the imaging element <NUM>. The subject <NUM> including the point <NUM> is also imaged on the imaging element <NUM> through the lens <NUM>. Here, the point <NUM> is a point <NUM> on the imaging element <NUM>.

Then, a difference in position between a point <NUM> in the imaging element <NUM> which corresponds to the point <NUM> in the imaging element <NUM> and the point <NUM> corresponds to a parallax <NUM>. As described above, if one subject <NUM> is imaged with two cameras, there is a difference in image between the imaging element <NUM> and the imaging element <NUM> of the respective cameras.

Further, a large triangle formed by the point <NUM>, a point <NUM>, and a point <NUM> and a small triangle formed by the point <NUM>, the point <NUM>, and the point <NUM> have a similarity relation due to optical properties. Therefore, when a length <NUM> from the portable terminal <NUM> to the subject <NUM> is indicated by a distance L, a length <NUM> between lenses is indicated by a distance D, a focal distance <NUM> is indicated by a distance F, and the parallax <NUM> is indicated by a distance S, these distances have a relation of L : D = F : S.

If the distance L is indicated by L = D × F/S, and, for example, D = <NUM>, F = <NUM>, and S = <NUM> are set, L = <NUM> is obtained from this relation. In the smartphone <NUM> or the tablet terminal <NUM> which is the portable terminal <NUM>, the relation between the position of the lens of each of the two cameras and the position of the imaging element is fixed, and thus the distance between the subject <NUM> and the portable terminal <NUM> can be calculated as described above.

As described above, the point of view and the distance can be calculated if the eye of the user who is the subject can be imaged, but since there may be a big difference in how to open the eyes or how to calculate the difference between the user and the portable terminal depending on an individual difference of the user, there are cases in which it is difficult to calculate the point of view and the distance with the contour and the pupil of the eye. In this regard, an example using a glasses-type device will be described below.

<FIG> is a diagram illustrating examples of a glasses-type device <NUM> and a portable terminal <NUM>. The glasses-type device <NUM> has a shape of glasses and is fixed to the face of the user's through contact with the nose and the ears and includes a camera <NUM>-<NUM> and a camera <NUM>-<NUM>. The camera <NUM> is located near the eye of the user, and thus it is easy to image the contour and the pupil of the eye. Further, a single camera <NUM> may be disposed.

The distance between the camera <NUM> and the eye of the user is almost constant depending on the shape of the glasses-type device <NUM> although there are individual differences in the shape of the face of the user, and may be set as a fixed value when the shape of the glasses-type device <NUM> is designed.

The portable terminal <NUM> is the smartphone <NUM> or the tablet terminal <NUM> and is connected to the glasses-type device <NUM> with a connection code <NUM>, and the captured image data of the camera <NUM> is obtained from the glasses-type device <NUM>. Although <FIG> illustrates an example of a wired connection using the connection code <NUM>, the portable terminal <NUM> and the glasses-type device <NUM> may be wirelessly connected without the connection code <NUM>.

The portable terminal <NUM> includes a touch pad (display screen) <NUM>, an up button <NUM>, a down button <NUM>, and a decision button <NUM>, and operations thereof are similar to those described above, and two cameras may not be disposed. A transparent or colored lens is inserted into a rim <NUM> of the glasses-type device <NUM> so that the user can see the touch pad <NUM>.

A liquid crystal screen may be inserted into one or both of the rim <NUM>-<NUM> and the rim <NUM>-<NUM> of the glasses-type device <NUM>. The portable terminal <NUM> may output image data to the glasses-type device <NUM> via the connection code <NUM> in order to display an icon or the like on the LCD screen.

In a case in which the liquid crystal screen is inserted into the glasses-type device <NUM>, the portable terminal <NUM> calculates the point of view of the user to the icon or the like displayed on the liquid crystal screen. Further, the portable terminal <NUM> in this case may not include the touch pad <NUM> or may include a touch pad that is smaller in size or lower in resolution than the touch pad <NUM>.

Either or both of temples <NUM>-<NUM> and <NUM>-<NUM> of the glasses-type device <NUM> may include an earphone. The portable terminal <NUM> may output synthesized voice data to the glasses-type device <NUM> via the connection code <NUM> in order to convey information to the user by sound using an earphone. The glasses-type device <NUM> may include a headphone instead of the earphone. In order to provide the headphone, a headband that touches the head of the user may be provided.

Also, the temple <NUM> may include a microphone like a hands-free phone. To obtain voice data from the microphone, the portable terminal <NUM> may input voice data from the glasses-type device <NUM> via the connection code <NUM>.

Further, the glasses-type device <NUM> may include a liquid crystal screen and may include the up button <NUM>, the down button <NUM>, and the decision button <NUM>, and the glasses-type device <NUM> executes the process of the portable terminal <NUM>. In this case, the glasses-type device <NUM> can operate independently, and thus the connection code <NUM> is not necessary.

In the above description, the camera is used to specify the point of view of the user, but the camera need not be necessarily used. For example, in a case in which the smartphone is grabbed by the user's hand, the user's finger may unintentionally touch the touch pad and an input not intended by the user is performed, whereas the operation illustrated in <FIG> may be detected without using the camera.

<FIG> is a diagram illustrating an example of an operation in which a camera is not used. The smartphone <NUM> includes a touch pad <NUM>, a side button <NUM>, and a microphone <NUM>. A camera may be disposed but not used. The smartphone <NUM> executes a process corresponding to the detected operation by detecting the touch position on the touch pad <NUM> by one hand <NUM> instead of the position of the point of view and detecting the pushing of the side button <NUM> by one hand <NUM>.

Further, the smartphone <NUM> may detect the touch position on the touch pad <NUM> with one hand <NUM>, and detect the operation with the voice on the microphone <NUM> instead of detecting the side button <NUM> being pushed. Accordingly, detection of other operations can be performed in addition to the detection of the operation on the touch pad <NUM>, so that the possibility of erroneous detection of an operation not intended by the user can be reduced.

In the above description, operations other than the operation on the touch pad are detected, but an example of reducing the possibility of erroneous detection of an operation not intended by the user in the detection of the operation on the touch pad will be described. <FIG> is a diagram illustrating an example of a tablet terminal <NUM> that detects touches of a plurality of fingers. The tablet terminal <NUM> includes a touch pad (display screen) <NUM> and a camera <NUM>.

The camera <NUM> is installed to be tilted toward the touch pad <NUM> so that a hand <NUM> that operates the touch pad <NUM> can be imaged together with the fingers, and an imaging optical axis of the camera <NUM> is tilted in a direction of the touch pad <NUM>, and an imaging range includes the periphery of the touch pad <NUM> like an angle of field <NUM>.

The tablet terminal <NUM> detects a touch of a first finger (for example, the index finger) of the hand <NUM>, and moves the cursor if the movement of the touched finger is detected in that state. Then, if a touch by a second finger (for example, the thumb) of the hand <NUM> within a range of an execution (decision) button in the touch pad <NUM> is detected, the operation corresponding to the position of the cursor that is the position at which the first finger is detected is executed or decided.

<FIG> is a diagram illustrating an example of a screen of the tablet terminal <NUM>. An icon <NUM>-<NUM>, an icon <NUM>-<NUM>, and an icon <NUM>-<NUM> are displayed on the touch pad <NUM>, and each icon <NUM> corresponds to an application program or an instruction for an application program.

An icon <NUM> is an execution (decision) button. For example, if the touch to the icon <NUM> by the second finger is detected in a state in which the touch to the icon <NUM>-<NUM> by the first finger is detected, the tablet terminal <NUM> activates the application program corresponding to the icon <NUM>-<NUM> or performs an operation based on an instruction.

The icon <NUM> may not be displayed on the touch pad <NUM>. In this case, the range of the execution (decision) button is a region other than the icon <NUM>. Also, if it is detected that the first finger is continuously touching any one icon <NUM> in a state in which the icon <NUM> is not displayed for a preset time or more, the icon <NUM> may be displayed, and the touch to the icon <NUM> by the second finger may be detected.

The finger type touching the touch pad <NUM> is recognized by analyzing the image captured by the camera <NUM>. In a case in which the touch of the second finger is recognized as the touch of the thumb by the finger type recognition, it may be detected as the so-called left click of the mouse, and an application program corresponding to the icon <NUM> may be activated.

Further, in a case in which the touch of the second finger is recognized as the touch of the middle finger, it may be detected as the so-called right click of the mouse, and a menu of an application program corresponding to the icon <NUM> may be displayed or information of the application program may be displayed. Further, in a case in which the touch of the first finger is performed with the middle finger, the touch of the second finger may be detected as the left click of the mouse when the touch of the second finger is performed with the index finger and may be detected as the right click of the mouse when the touch of the second is performed with the ring finger.

To make the user aware of the finger type recognition result, the area around the touch part of the touch pad <NUM> may be displayed in a specific color corresponding to the recognized finger type. Then, the display of the color around the touch part may be regarded as a cursor. Accordingly, it is possible to reduce the possibility of an erroneous instruction caused by erroneous finger type recognition.

Further, the image of the hand <NUM> captured by the camera <NUM> may be displayed on the touch pad <NUM> to be superimposed on the icon <NUM> and the icon <NUM>.

Then, an example in which the point of view is detected by the camera, but the touch pad is used instead of the side button <NUM> illustrated in <FIG> and the decision button <NUM> illustrated in <FIG> will be described. <FIG> is a diagram illustrating an example of a smartphone <NUM> in which a part of the touch pad <NUM> is a special region.

The smartphone <NUM> includes a touch pad <NUM>, a camera <NUM>, and a camera <NUM>. The camera <NUM> and the camera <NUM> are similar to those described above. The smartphone <NUM> may include a side button and a microphone, but they are not used here. The smartphone <NUM> may be the tablet terminal <NUM>.

In order to use the touch pad <NUM> instead of the side button <NUM> and the decision button <NUM>, a special region <NUM> that replaces the side button <NUM> and the decision button <NUM> is set in the touch pad <NUM>. The special region <NUM> is, for example, a region that is touched with the thumb when the smartphone <NUM> is held with one hand.

In a case in which the user grips the smartphone <NUM> with one hand, the gripping position differs depending on the user's habit, and the gripping position differs depending on whether the user is right-handed or left-handed. In this regard, a setting that one special region <NUM> of special regions <NUM>-<NUM> to <NUM>-<NUM> illustrated in <FIG> is selected is received after the smartphone <NUM> is powered on or reset.

In <FIG>, for example, the special regions <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> are for cases in which it is held with the left hand, and the special regions <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> are for cases in which it is held with the right hand. It is desirable that the special region <NUM> touched with the hand holding the smartphone <NUM> be a region that touches the edge of the touch pad <NUM>.

<FIG> is an example in which a setting that the special region <NUM>-<NUM> is selected is received. A position designation by the point of view is valid for the touch pad <NUM> excluding the special region <NUM>-<NUM>, and the special region <NUM>-<NUM> is set to be dedicated for touch. For this reason, the icon <NUM> is not placed in the special region <NUM>-<NUM>.

Then, as described above, the point of view of the user is specified based on the captured images of the camera <NUM> and the camera <NUM>, and the cursor <NUM> is displayed. The position of the cursor <NUM> which is the position of the point of view can be moved from the icon <NUM>-<NUM> to the icon <NUM>-<NUM>.

If it is detected that the special region <NUM>-<NUM> is touched by the thumb of one hand <NUM> at a position at which the cursor <NUM> is positioned over the icon <NUM>-<NUM>, an application corresponding to the icon <NUM>-<NUM> is activated, or an instruction is sent to the application.

Further, a character string representing "decision", "touch" or the like, or a special icon may be displayed in the special region <NUM>-<NUM> so that the user can understand that it is the special region.

As a plurality of operations such as a first operation by the point of view or the first finger described above and a second operation by a button or voice are detected, the validity of the operation is confirmed, and an operation according to the user's intention is realized. The operation for confirming the validity is not limited to the operation described above.

For example, if the icon selection state by the point of view continues for a few seconds as the detection of the operation using the camera, a message such as "Please close only the right eye for three seconds if you like" may be presented to the user at the time of execution or decision, and if such an action is detected, it may transition to an operation of execution or decision.

Also, if the user is wearing the sunglasses and an input other than the eye is necessary, for example, in order to confirm it with a lip shape, a message such as "Please make a mouth shape when you pronounce 'O' if you like" may be presented to the user, and it may be determined whether such an action is detected.

The sensor of the smartphone <NUM> or the tablet terminal <NUM> for detecting the operation is similar to that described above, but the detected operation may be given a priority as an input condition. For example, a priority may be set to the following information in advance, and an arbitrary combination may be made:.

An input condition management unit to be described later selects an input condition with a high priority or an input condition with a low priority according to a surrounding environment. For example, if it is determined that the surrounding environment is a dark environment, the input condition in which the camera is used is not selected, and the switch input and the touch information to the touch pad with a low priority may be selected.

<FIG> is a diagram illustrating an example of a first hardware configuration of the portable terminal <NUM>. The portable terminal <NUM> is the smartphone <NUM> or the tablet terminal <NUM> as described above. A camera unit <NUM> corresponds to the cameras <NUM> and <NUM>, the cameras <NUM> and <NUM>, or the like, a touch pad display unit <NUM> and a touch pad sensing unit <NUM> correspond to the touch pad <NUM>, the touch pad <NUM>, or the like.

A microphone unit <NUM> corresponds to the microphone <NUM> or the microphone <NUM>, and a switch input unit <NUM> corresponds to the side button <NUM>, the decision button <NUM>, or the like. Further, hardware such as the original phone function of the smartphone <NUM> is omitted because it does not relate to the description of the present embodiment. Further, the activation of the application program will be described in particular.

The camera unit <NUM> outputs image information obtained by imaging of two or more cameras to a face recognizing unit <NUM>. Further, the face recognizing unit <NUM> recognizes the face and extracts features of the face parts. The face recognizing unit <NUM> may calculate a distance up to the face and perform an operation of outputting a message to the user in a case in which the face is unable to be recognized in the front or the eye is unable to be recognized.

A point of view detecting unit <NUM> receives information of the eye part and the distance to the face from the face recognizing unit <NUM>, and calculates the position of the point of view. Here, it may be determined whether or not the point of view is turned to the touch pad display unit <NUM>. Then, since the point of view has a property of generating fine movement continuously, a low pass filter unit <NUM> cuts fibrillation information and outputs the information of the point of view to a point of view information management unit <NUM>.

The point of view information management unit <NUM> outputs information indicating a location at which there is a point of view on the screen of the touch pad display unit <NUM> to a cursor display <NUM> and displays the cursor for the user via the touch pad display unit <NUM>. Further, as described above, icon highlight display or the like may be used instead of the cursor.

On the other hand, as at least one or more input combined with the point of view information, there is a voice input which is input from the microphone unit <NUM> and decoded via a voice recognizing unit <NUM>, a switch input from the switch input unit <NUM> such as the side button, or a touch input of the finger which is input from the touch pad sensing unit <NUM> or the like.

By aligning these inputs with the point of view input from the point of view information management unit <NUM>, an input condition management unit <NUM> causes an application program executing unit <NUM> to activate the application program specified by the input. As described above, when the input condition management unit <NUM> determines that a plurality of inputs satisfy a predetermined condition, the input condition management unit <NUM> outputs an instruction to the application program executing unit <NUM>.

The input condition management unit <NUM> may have correspondence information between the display position of the icon and the identification information of the application program corresponding to the icon and output the identification information corresponding to the point of view input from the point of view information management unit <NUM> to the application program executing unit <NUM>.

Alternatively, the application program executing unit <NUM> may have the correspondence information of the display position of the icon and the identification information of the application program corresponding to the icon, and the input condition management unit <NUM> may transfer the input of the point of view from the point of view information management unit <NUM> to the application program executing unit <NUM> if it is determined that a predetermined condition is satisfied.

The input condition management unit <NUM> may have correspondence information of a region of the touch pad <NUM> or the touch pad <NUM> or the like and an instruction to be given to the application program and output an instruction according to the input of the touch position from the touch pad sensing unit <NUM> to the application program executing unit <NUM>.

The input condition management unit <NUM> may have a plurality of pieces of information of the special region <NUM> as the region of the touch pad <NUM> or the touch pad <NUM> or the like, have information of the selected special region <NUM>-<NUM>, and have a region of the decision button of the icon <NUM>.

In the above description, the input condition management unit <NUM> is described as an independent control unit, but it may be a part of the application program executing unit <NUM>. Finally, through the application program executing unit <NUM>, a speaker unit <NUM> may transmit information to the user by voice, or a command may be sent to a communication control unit <NUM> for communication.

Further, as described above with reference to <FIG>, since there are individual differences between the information of the eye imaged by the camera and the point of view, a calibration management unit <NUM> manages a relation of the information of the eye imaged by the camera and the point of view and output it to the point of view information management unit <NUM>, and the point of view information management unit <NUM> specifies the point of view based on the relation.

Through the units from the camera unit <NUM> to the point of view information management unit <NUM>, the finger may be recognized instead of the face, and the position and the type of the finger may be specified. The input condition management unit <NUM> may use the information of the position and the type of the finger from the point of view information management unit <NUM> as an input and may forward the input from the touch pad sensing unit <NUM> to the point of view information management unit <NUM> and cause the point of view information management unit <NUM> to display the cursor at the touch position.

<FIG> is a diagram illustrating an example of a second hardware configuration of the portable terminal <NUM>. Each unit of the hardware described with reference to <FIG> may be realized by software. The example of the hardware configuration illustrated in <FIG> is an example of a hardware configuration for realization in the software. Further, the hardware configuration of the portable terminal <NUM> illustrated in <FIG> may be a hardware configuration of a general smartphone or tablet terminal.

A processor <NUM> executes a program stored in a memory <NUM>, exchanges data with each unit illustrated in <FIG>, and processes the data. The memory <NUM> is, for example, a random-access memory (RAM) and stores a program and data.

A storage <NUM> is, for example, a flash memory, and stores a program and data. A program and data may be transferred between the memory <NUM> and the storage <NUM>. A communication unit <NUM> performs communication via a wired connection using the connection code <NUM>, or a wireless telephone line or wireless local area network (LAN).

The communication unit <NUM> performs transmission and reception under the control of the processor <NUM> that executes the application program stored in the memory <NUM>. Also, the program and the data stored in the memory <NUM> or the storage <NUM> may be received by the communication unit <NUM>.

An input unit <NUM> stores an input signal from a microphone <NUM> or the like in the memory <NUM> or the storage <NUM> as data. The microphone <NUM> is the microphone <NUM> illustrated in <FIG> or the microphone <NUM> illustrated in <FIG> and inputs a voice signal. A camera <NUM> and a camera <NUM> are the camera <NUM> and the camera <NUM> illustrated in <FIG> or the camera <NUM> and the camera <NUM> illustrated in <FIG>, respectively, and input the imaged video signal.

An output unit <NUM> converts the data stored in the memory <NUM> or the storage <NUM> into a signal and outputs it to a speaker <NUM> or the like. A touch pad <NUM> is the touch pad <NUM> illustrated in <FIG> or the touch pad <NUM> illustrated in <FIG> or the like, and if a touch is detected, the touch pad <NUM> outputs the touch information to the input unit <NUM>, and displays a screen according to the video signal input from the output unit.

<FIG> is a diagram illustrating an example of a process of the portable terminal <NUM> illustrated in <FIG> and is an example of a flowchart of a program which is stored in the memory <NUM> and executed by the processor <NUM>. If the portable terminal <NUM> is powered on or released from sleep, the processor <NUM> starts the process.

First, the processor <NUM> determines whether or not the calibration is incomplete (step <NUM>). For this determination, for example, a flag indicating whether or not the calibration is incomplete may be stored in the storage <NUM>, and the processor <NUM> may read and determine the flag. When the portable terminal <NUM> is manufactured and reset, the flag is set to incomplete.

In a case in which it is determined that it is incomplete in step <NUM>, the processor <NUM> executes the calibration (step <NUM>), and if it is determined that it is not incomplete, step <NUM> is skipped. Processing content of the calibration in step <NUM> is similar to that described above with reference to <FIG> and <FIG>, and the cameras <NUM> and <NUM> of the portable terminal <NUM> are enabled to image the user. Further, the flag is set to complete.

The processor <NUM> displays the home screen called the standby screen or the like (step <NUM>). As illustrated in <FIG>, the home screen includes icons respectively corresponding to a plurality of application programs, and the application program is activated by operating the icon. At this point, if the camera unit <NUM> or the cameras <NUM> and <NUM> of the portable terminal <NUM> are not enabled, the camera unit or the cameras are enabled.

The processor <NUM> controls the cameras <NUM> and <NUM> such that the face is imaged and recognized (step <NUM>). In the face recognition, the process for recognizing the face facing the cameras <NUM> and <NUM> is executed, and when the face is recognized, the position of the pupil or the iris in the contour of the eye is recognized.

when the face can be recognized in the front as a result of executing the process for recognizing the face in step <NUM>, the face is regarded as facing the portable terminal <NUM>, and the processor <NUM> proceeds to step <NUM>, and when the face is unable to be recognized or when the face can be recognized as facing sideways, the face is regarded as not facing the portable terminal <NUM>, and the processor <NUM> proceeds to step <NUM>.

In step <NUM>, the processor <NUM> outputs a message for urging the user's attention or a message for causing the user to turn the face. This message may be output from the speaker <NUM>.

In step <NUM>, the processor <NUM> calculates the point of view from the relation between the position of the pupil or the iris in the contour of the eye recognized in step <NUM> and the position of the pupil or the iris in the contour of the eye recorded in the calibration of step <NUM>. The distance to the face may be calculated from the parallax of the camera <NUM> and the camera <NUM> and used to calculate the point of view.

The processor <NUM> applies the low pass filter to prevent the erroneous detection caused by a small fluctuation in the calculated point of view (step <NUM>). For example, the point of view may be calculated from images captured at a preset time interval, and if a preset number of points of views are calculated, the positions of the points of views may be averaged.

The processor <NUM> displays the cursor at the position corresponding to the point of view to which the low pass filter is applied (step <NUM>). In a case in which the position corresponding to the point of view coincides with the position of the icon on the home screen displayed in step <NUM>, the icon at the coinciding position may be highlighted.

The processor <NUM> determines whether or not there is an input for selecting the icon designated by the point of view (step <NUM>). This selection input is, for example, an input by the side button <NUM> in <FIG>. If it is determined that there is no input, the processor <NUM> returns to step <NUM> because the user is likely to just see the icon designated by the point of view.

If it is determined that there is a selection input, the processor <NUM> determines whether or not the selection input satisfies an input condition (step <NUM>). This input condition is similar to that described above as the input condition with a priority and is set in advance. For example, if it is determined that the input condition is not satisfied although there is an input by the side button <NUM>, the processor <NUM> proceeds to step <NUM>.

In step <NUM>, the processor <NUM> outputs a message indicating that the input condition is not satisfied or a message indicating that the input is not valid. This message may be output from the speaker <NUM>. Further, in step <NUM>, it may be determined whether it is executed with the determination result of step <NUM> or with the determination result of step <NUM>.

If it is determined in step <NUM> that the input condition is satisfied, the processor <NUM> executes the application program corresponding to the icon designated by the point of view (step <NUM>). A message serving as the execution result of the application program may be output from the speaker <NUM>, or an error message by execution may be output from the speaker <NUM>.

After the application program is executed in step <NUM> or after the message is output in step <NUM>, the processor <NUM> determines whether there is an operation before a preset time elapses. If it is determined that the preset time elapses, the processor <NUM> causes the portable terminal <NUM> to transition to, for example, the sleep state and ends the process of the flowchart.

If it is determined that there is an operation before the preset time elapses, the processor <NUM> returns to step <NUM>. Further, the determination of whether there is an operation in step <NUM> is not limited to an independent determination and may be combined with step <NUM> or step <NUM>.

Further, in the above description using <FIG>, the example of using the point of view has been described, but other input conditions may be used as described above, and the operation of each unit described with reference to <FIG> may be realized by a program.

As described above, the portable terminal operates by determining a plurality of input conditions for the input by the operation of the user, and thus the possibility of erroneous operation caused by the erroneous operation of the user can be reduced.

Claim 1:
A method of operating a portable terminal (<NUM>), the portable terminal (<NUM>) comprising:
a touch pad (<NUM>) that is configured to display one or more icons (<NUM>) in a first region and to detect a touch in a second region, wherein a part of the touch pad (<NUM>) is preset as the second region (<NUM>) dedicated for touch and the one or more icons (<NUM>) are not displayed in the second region (<NUM>) of the touch pad (<NUM>);
the method comprising:
imaging, by a camera (<NUM>, <NUM>) of the portable terminal (<NUM>), a face of a user of the portable terminal (<NUM>);
detecting, by a detecting unit of the portable terminal (<NUM>), a point of view on the touch pad (<NUM>) to which an eye of the face of the user imaged by the camera (<NUM>, <NUM>) is turned, wherein the point of view is not detected in the preset second region (<NUM>) of the touch pad (<NUM>);
outputting, by an information management unit of the portable terminal (<NUM>), a position of the point of view detected by the detecting unit;
performing, by a cursor display unit of the portable terminal (<NUM>), control such that a cursor is displayed on the touch pad (<NUM>) according to the position of the point of view on the touch pad (<NUM>) output from the information management unit;
determining, by an input condition management unit of the portable terminal (<NUM>), whether or not a preset input condition is satisfied for an input by a touch to the preset second region (<NUM>) of the touch pad (<NUM>) and an input of the position of the point of view on the touch pad (<NUM>) output from the information management unit; and
activating, by an application program executing unit of the portable terminal (<NUM>), an application program corresponding to an icon of the one or more icons (<NUM>) of the position of the point of view on the touch pad (<NUM>) from the information management unit according to the determination of the input condition management unit for the inputting to the second region (<NUM>) and the inputting of the position of the point of view on the first region.