Method for tracking a user's eye to control an indicator on a touch screen and electronic device thereof

A method includes detecting a movement of an eye pupil in an eye tracking mode, moving an indicator on the screen according to the movement of the eye pupil, detecting a touch on a touch area, and in response to detecting the touch on the touch area, selecting an object on which the indicator is located, among the plurality of the objects on the screen. An electronic device includes a display configured to display displaying a plurality of objects on a screen, and a processor configured to detect a movement of an eye pupil through a camera in an eye tracking mode, move an indicator on the screen according to the movement of the eye pupil, detect a touch on the electronic device, and in response to detecting the touch, select an object on which the indicator is located, among the plurality of the objects on the screen.

PRIORITY

The present application is related to and claims priority under 35 U.S.C. §119 to an application number 10-2014-0031132 filed in the Korean Intellectual Property Office on Mar. 17, 2014 the contents of which are incorporated herein by reference.

TECHNICAL FIELD

Various exemplary embodiments of the present disclosure relate to a method of processing an input in an electronic device including a camera and a display, and the electronic device.

BACKGROUND

In an operation of an electronic device, a camera included in the electronic device can detect an input such as a designated movement and designated gesture of a user, and can handle it by using an instruction designated in the electronic device in response to the input. When the camera included in the electronic device detects a designated input, a pointer which indicates a designated position of a display in response to the input can be displayed at a corresponding position of the display.

SUMMARY

Although a display screen size of an electronic device is increased, since a weight of the electronic device becomes lighter, the electronic device can be used by one hand. If the electronic device is used by one hand, there may be a restriction when a designated position of a display is touched by a designated input means (e.g., a thumb). There is a need to provide a method capable of performing an input for an entirety of the display when the electronic device is used in a state of being gripped with one hand.

To address the above-discussed deficiencies, it is a primary object to provide a method of operating an electronic device that may include displaying a specific position in a display, and handling a touch detected in the display as a touch of the specific position.

According to various exemplary embodiments of the present disclosure, an electronic device may include a display for displaying a specific position and for detecting a touch, and a processor for providing control to display the specific position in the display and for handling the touch detected in the display as a touch of the specific position.

A method for operating an electronic device includes displaying a plurality of objects on a screen, detecting a movement of an eye pupil through a camera in an eye tracking mode, moving an indicator on the screen according to the movement of the eye pupil, detecting a touch on a touch area, and in response to detecting the touch on the touch area, selecting an object on which the indicator is located, among the plurality of the objects on the screen.

An electronic device includes a display configured to display displaying a plurality of objects on a screen, and a processor configured to detect a movement of an eye pupil through a camera in an eye tracking mode, move an indicator on the screen according to the movement of the eye pupil, detect a touch on the electronic device, and in response to detecting the touch, select an object on which the indicator is located, among the plurality of the objects on the screen.

DETAILED DESCRIPTION

FIGS. 1 through 10, discussed below, and the various embodiments used to describe the principles of the present disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably arranged electronic devices. Hereinafter, various exemplary embodiments of the present disclosure are described with reference to the accompanying drawings.

While the various exemplary embodiments of the present disclosure are susceptible to various modifications and alternative forms, a specific embodiment thereof has been shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that it is not intended to limit the various exemplary embodiments of the present disclosure to the particular form disclosed, but, on the contrary, the various exemplary embodiments of the present disclosure are to cover all modifications and/or equivalents and alternatives falling within the spirit and scope of the various exemplary embodiments of the present disclosure as defined by the appended claims. Like reference numerals denote like constitutional elements throughout the drawings.

In various exemplary embodiments of the present disclosure, an expression “or” includes any and all combinations of words enumerated together. For example, “A or B” may include A or B, or may include both of A and B.

Although expressions used in various exemplary embodiments of the present disclosure such as “1st”, “2nd”, “first”, “second” may be used to express various constitutional elements of the various exemplary embodiments, it is not intended to limit the corresponding constitutional elements. In addition, the above expressions may be used to distinguish one constitutional element from another constitutional element. In addition, when a constitutional element is mentioned as being “connected” to or “accessing” another constitutional element, this may mean that it is directly connected to or accessing the other constitutional element, but it is to be understood that there are no intervening constitutional elements present. On the other hand, when a constitutional element is mentioned as being “directly connected” to or “directly accessing” another constitutional element, it is to be understood that there are no intervening constitutional elements present.

Terms used in various exemplary embodiments of the present disclosure are for the purpose of describing particular exemplary embodiments only and are not intended to be limiting of the various exemplary embodiments of the present disclosure. A singular expression includes a plural expression unless there is a contextually distinctive difference therebetween.

According to various exemplary embodiments, the electronic device may be a device including a communication function. For example, the electronic device may include at least one of a smart phone including a communication function, a tablet Personal Computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a Personal Digital Assistant (PDA), a Portable Multimedia Player (PMP), a MPEG-1 Audio Layer 3 (MP3) player, a mobile medical device, a camera, a wearable device (e.g., a Head-Mounted-Device (HMD) such as electronic glasses, electronic clothes, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, or a smart watch), a television (TV), a Digital Video Disk (DVD) player, an audio, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, a TV box (e.g., Samsung HomeSync™, Apple TV™, or Google TV™), a game console, an electronic dictionary, an electronic key, a camcorder, medical devices (e.g., Magnetic Resonance Angiography (MRA), Magnetic Resonance Imaging (MRI), Computed Tomography (CT), imaging equipment, ultrasonic instrument, etc.), a navigation device, a Global Positioning System (GPS) receiver, an Event Data Recorder (EDR), a Flight Data Recorder (FDR), a car infotainment device, an electronic equipment for ship (e.g., a vessel navigation device, a gyro compass, etc.), avionics, a security device, an industrial or domestic robot, a furniture or a part of building/constructions including a communication function, an electronic board, an electronic signature input device, a projector, and various measurement machines (e.g., water supply, electricity, gas, propagation measurement machine, etc.). The electronic device according to the present disclosure may be one or more combinations of the aforementioned various devices. In addition, it is apparent those ordinarily skilled in the art that the electronic device according to the present disclosure is not limited to the aforementioned devices.

Hereinafter, an electronic device according to various exemplary embodiments will be described with reference to the accompanying drawings. The term ‘user’ used in the various exemplary embodiments may refer to a person who uses the electronic device or a device which uses the electronic device (e.g., an Artificial Intelligence (AI) electronic device).

FIG. 1illustrates a network environment including an electronic device according to various exemplary embodiments.

Referring toFIG. 1, an electronic device101can include a bus110, a processor120, a memory130, an input/output interface140, a display150, a communication interface160, a gesture processing module180, a microphone, and a speaker.

The bus110can be a circuit for connecting the aforementioned constitutional elements to each other and for delivering a communication signal (e.g., a control message) between the aforementioned constitutional elements.

The processor120can receive an instruction from the aforementioned different constitutional elements (e.g., the memory130, the input/output interface140, the display150, the communication interface160, or the gesture processing module180), for example, via the bus110, and thus can interpret the received instruction and execute arithmetic or data processing according to the interpreted instruction.

The memory130can store an instruction or data received from the processor120or different constitutional elements (e.g., the input/output interface140, the display150, the communication interface160, or the gesture processing module180) or generated by the processor120or the different constitutional elements. The memory130can include programming modules such as a kernel131, a middleware132, an Application Programming Interface (API)133, an application134, and the like. Each of the aforementioned programming modules can consist of software, firmware, or hardware entities or can consist of at least two or more combinations thereof.

The kernel131can control or manage the remaining other programming modules, for example, system resources (e.g., the bus110, the processor120, the memory130, etc.) used to execute an operation or function implemented in the middleware132, the API133, or the application134. In addition, the kernel131can provide a controllable or manageable interface by accessing individual constitutional elements of the electronic device101in the middleware132, the API133, or the application134.

The middleware132can perform a mediation role so that the API133or the application134communicates with the kernel131to exchange data. In addition, regarding task requests received from the application134, for example, the middleware132can perform a control (e.g., scheduling or load balancing) for the task requests by using a method of assigning a priority capable of using a system resource (e.g., the bus110, the processor120, the memory130, etc.) of the electronic device101to at least one of the applications134.

The API133can include at least one interface or function (e.g., instruction) for file control, window control, video processing, character control, and the like, as an interface capable of controlling a function provided by the application134in the kernel131or the middleware132.

The application134can be an application related to an information exchange between the electronic device101and an external electronic device (e.g., an electronic device102, an electronic device103, or an electronic device104). The application related to the information exchange can include, for example, a notification relay application for relaying specific information to the external electronic device or a device management application for managing the external electronic device. According to various exemplary embodiments, the application134can include an application additionally designated according to attribute information (e.g., an electronic device type) of the external electronic device (e.g., the electronic device102, the electronic device103, or the electronic device104).

The input/output interface140can relay an instruction or data input from a user via a sensor (e.g., an acceleration sensor, a gyro sensor) or an input/output device (e.g., a keyboard, or a touch screen) to the processor120, the memory130, the communication interface160, or the gesture processing module180, for example, via the bus110. For example, the input/output interface140can provide data regarding a user's touch input via the touch screen to the processor120. In addition, the input/output interface140can output an instruction or data received from the processor120, the memory130, the communication interface160, or the gesture processing module180to an output device (e.g., a speaker or a display), for example, via the bus110. For example, the input/output interface140can output audio data provided by using the processor120to the user via the speaker.

The display150can display a variety of information (e.g., multimedia data or text data) to the user. The display150can consist of a touch screen for inputting an instruction by touching or proximity touching an input means to the display. The touch screen can include a touch panel so that an input function and a display function can be performed simultaneously. The touch panel can include a Liquid-Crystal Display (LCD) or an Active-Matrix Organic Light-Emitting Diode (AM-OLED), and can be implemented in a flexible, transparent, or wearable manner. The touch panel can recognize a touch input by using at least one of an electrostatic type, a pressure-sensitive type, and an ultrasonic type. The touch panel can further include a controller. In case of the electrostatic type, the touch panel can recognize not only a direct touch but also a proximity touch. The proximity touch is also expressed as a non-contact touch or hovering. The touch penal can further include a tactile layer to provide the user with a tactile reaction. The touch screen can include a hologram. The hologram can use an interference of light and show a stereoscopic image in the air. In addition, the touch screen can further include a control circuit for controlling the touch panel or the hologram. Unless otherwise specified in particular in the following description, the touch screen and the touch panel can be represented by the display150. A device area for displaying a variety of information can be designated as a screen in the display150. The electronic device101can use at least some areas of the touch panel as an input area (or a touch area). If the input area which operates on a touch basis is configured to the display150, the electronic device101can determine the input area distinctively at a designated position in the touch panel of the touch screen, and can display the configured input area to the display150.

The communication interface160can connect a communication between the electronic device101and an external device (e.g., the electronic device102, the electronic device103, the electronic device104, or the server106). For example, the communication interface160can support a network communication162(e.g., Internet, Local Area Network (LAN), Wide Area Network (WAN), telecommunication network, cellular network, satellite network, Plain Old Telephone Service (POTS), etc.) and a near distance communication164(e.g., Wireless Fidelity (WiFi), Bluetooth (BT), Near Field Communication (NFC), or wired communication (e.g., Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), Recommended Standard (RS)-232, POTS, etc.). According to one exemplary embodiment, a protocol for a communication between the electronic device101and the external device (e.g., a near distance communication protocol, a network communication protocol, or a wired communication protocol) can be supported in at least one of the API133and the middleware132. Each of the electronic devices102,103, and104can be a device which is the same (e.g., the same type) as the electronic device101or can be a different (e.g., a different type) device.

One or more cameras170can be included at a designated position of a front portion of the electronic device101. The camera170can confirm a user's eye pupil movement, and can determine a designated position of the display150gazed by the eye pupil by using a database in association with the eye pupil movement. The electronic device101can display a pointer which indicates an object displayed on the display150at the designated position of the display150gazed by the eye pupil.

The gesture processing module180can process at least a part of information acquired from other constitutional elements (e.g., the processor120, the memory130, the input/output interface140, the communication interface60, etc.), and can provide it to the user in various manners. For example, the gesture processing module180can control at least a part of functions of the electronic device101so that the electronic device101interworks with other electronic devices (e.g., the electronic device102, the electronic device103, and the electronic device104, or the server106) by using the processor120or independently.

The gesture processing module180can display a specific position in the display, and can handle a touch detected in the display as a touch of the specific position. The gesture processing module180can determine a coordinate associated with the display as the specific position based on an eye pupil or a movement of the eye pupil. The gesture processing module180can be a device for displaying at least one of a pointer, an identifier, and an indicator at the specific position or for displaying a state where an object associated with the specific position is selected. The gesture processing module180can handle the detected touch as a touch at a coordinate associated with the specific position. The gesture processing module180can handle a touch detected at the specific position or at a portion of a display other than the specific position as the detected touch. The gesture processing module180can handle a touch detected in an input area configured in at least a portion of the display as the touch of the specific position. The gesture processing module180can determine a single touch or a touch including a drag as the detected touch. The gesture processing module180can configure a layer for displaying a specific position and a layer for inputting a touch as different layers. The gesture processing module180can handle a touch detected in a touch panel operatively connected with the display as the touch of the specific position. Instead of handling the touch detected at the position of the display in which the detected touch occurs, the gesture processing module180can handle it as the touch of the specific position.

The gesture processing module180can display a pointer display area and an input area to the display by distinguishing the areas. A pointer can be displayed on the pointer display area, and a touch detected in the input area can be handled as a touch at a position at which the pointer is displayed.

The gesture processing module180can determine the input area at a designated position of the display which can be easily touched by a designated finger if the electronic device is gripped by one hand. The gesture processing module180can display the input area based on setup information including at least one of position, shape, and size information of the input area. When the input area is displayed, the gesture processing module180can determine at least one of a position, shape, and size of the input area by touching or dragging the display in which the input area is displayed. The gesture processing module180can change a position of an input area previously displayed on the display based on a designated input. If a pointer is located in the input area, the gesture processing module180can release the display of the input area. The gesture processing module180can display the input area to a layer different from the layer to which the pointer area is displayed. The gesture processing module180can detect from the input area a gesture designated to execute at least one function by touching the display. If the touch is detected from the pointer display area, the gesture processing module180can release the display of the pointer display area and the input area. The gesture processing module180can additionally determine the input area in a touch panel operatively coupled to the display to which the pointer display area is displayed. The gesture processing module180can display a pointer based on a user's eye pupil movement. The pointer can be displayed in an overlapping manner on at least one of icons displayed on the display.

The gesture processing module180can include a processor for displaying a pointer to a display, for detecting a touch in an input area independent of the display, and for handling a touch detected in the input area as a touch of a position at which the pointer is displayed. Herein, the pointer displayed on the display can be moved based on an eye pupil movement. The gesture processing module180can determine a touch panel formed independently in a rear portion of the display as the input area. An operation of the gesture processing module180can be performed by at least one processor120, or can be performed by the gesture processing module180under the control of the processor120.

FIG. 2is a block diagram of a gesture processing module of an electronic device according to various exemplary embodiments. Referring toFIG. 2, the gesture processing module180can include one or more of a sensing module210, a determining module220, and a providing module230.

The sensing module210can sense various gestures which are input to the electronic device101. According to one exemplary embodiment, the sensing module210can detect a user's eye pupil movement by using the camera170included in the electronic device101. The sensing module210can detect an input of an operation designated in the electronic device101, such as a touch, a touch-and-hold, or a touch-and-drag, to a display to which a touch input is achieved by using one or more designated input means.

The determining module220can determine an operation associated with the various gestures which are input to the electronic device101. The determining module220can further determine a touch which is input to the display150of the electronic device101as a touch of a specific position (e.g., a pointer displayed on the display150in association with an eye pupil or a movement of the eye pupil) of the display150by using setup information of an eye pupil sensing mode. The determining module220can determine an input area according to a gesture which is input to the display150at a time of entering the eye pupil sensing mode.

The providing module230can perform a function associated with one or more objects selected in the display150by referring to the gesture which is input to the electronic device101. The providing module230can display a pointer which indicates an object displayed on the display150at a designated position of the display150gazed by the user's eye pupil sensed by using the camera170. The providing module230can provide a function of an object indicated by a pointer displayed on the display150in association with the gesture which is input to the input area displayed on the display150.

FIG. 3AandFIG. 3Billustrate an operation of selecting an object by processing a user's eye pupil in an electronic device according to various exemplary embodiments.

When the electronic device101enters an eye pupil sensing mode, the electronic device101can display a designated pointer at a position of an eye gaze of a user of the display150of the electronic device101by sensing a user's eye pupil through the camera170. The electronic device101can display information regarding a movement of the user's eye pupil through the camera170by matching to a virtual coordinate configured in the display150, and can move a pointer displayed on the display150according to the movement of the eye pupil. An operation of moving the pointer displayed on the display150according to the movement of the eye pupil in the electronic device101can use an eye pupil tracking (or eye tracking) technique which is used in general. Herein, the pointer displayed on the display150based on the eye pupil can be used in the meaning of an identifier or an indicator which represents a specific position of the display150and which is used in the electronic device101. Further, when the electronic device101displays the pointer to the display150, it can be used in a sense that an object displayed at the specific position of the display150is determined in a designated state (e.g., a selected state or an activation state).

Hereinafter, various exemplary embodiments of the present disclosure are described with reference toFIG. 3A.

The electronic device101can move a pointer303according to an eye gaze of a user at a designated position of the display150by tracking an eye pupil movement through the camera170, and a user can touch a designated area of the display150to perform a function of the electronic device101in association with an object indicated by the pointer303displayed on the display150based on a touch operation.

The display150of the electronic device101can perform an operation of an input device for controlling the pointer303on the display150. For example, in the eye pupil sensing mode of the electronic device101, an area of the pointer303which indicates the designated position can be displayed on the display150based on the movement of the eye pupil, and a gesture which is input to the display150by a touch can be handled as being an input to the pointer of the display150. Herein, the display150can be defined as an input area when a touch detected in the display150is handled as an input of the pointer displayed on the display150.

When the electronic device101determines an area of the pointer303or an input area to the display150in the eye pupil sensing mode, it can be determined in an additional layer (e.g., a second layer) which is different from a layer (e.g., a first layer) in which one or more objects displayed on the display150are located. The electronic device101can display the input area in the additional layer, and can handle a gesture of a designated position (e.g., an area300for detecting a touch) of the input area as a gesture for a designated object of the first layer indicated by a pointer displayed on the display150other than a gesture for the first layer displayed at a position at which the touch is detected.

According to one exemplary embodiment, upon sensing a touch input (e.g., a single touch) detected in the display150(e.g., the designated position300of the display150as illustrated inFIG. 3A), the electronic device101can handle it as that a single touch is performed on an object (or a target) indicated by the pointer303displayed on the designated position of the display150.

According to one exemplary embodiment, when the electronic device101is gripped by one hand of a user, in order to perform an input at the designated position of the display150, which is not easily touched by a designated finger (e.g., a thumb or an index finger), in the display150of the electronic device101, a pointer which indicates a specific position can be displayed on the display150, and a touch input detected in the display150can be handled as being input to the pointer displayed on the display150.

The electronic device101can determine an input area for detecting a touch by limiting to the designated area of the display150in the eye pupil sensing mode in which the touch detected in the display150is handled to be performed at the pointer displayed on the display150. According to one exemplary embodiment, when the electronic device101determines the input area of the display150, if the electronic device101is gripped by a right hand, an area (e.g., a part of a right lower area of the display150) in which the display150can be easily touched by a thumb can be determined as the input area. If the electronic device101detects a touch in the input area determined in the right lower portion of the display150, the detected touch can be handled as the touch detected at the pointer displayed on the display150.

The electronic device101can determine the input area at a position (e.g., the position300at which the touch is detected in the display150ofFIG. 3A) which can be easily touched by a finger (e.g., a thumb or an index finger) as the input area. The input area for detecting a touch gesture which handles a sensed touch as a touch of the pointer of the display150can be displayed based on setup information (e.g., setup information of the eye pupil sensing mode) or may not be displayed. Although it is described hereinafter that the determined input area is displayed in the display150of the electronic device101, apparently, the present disclosure is not limited that the input area must to be displayed.

According to various exemplary embodiments, the electronic device101can detect a movement of the user's eye pupil through the camera170, and can move a pointer at a position of the display150according to an eye gaze of the user. The electronic device101can move the pointer displayed on the display150in association with the movement of the user's eye pupil, and can detect a touch input within the input area marked on the display150. When the electronic device101detects a touch within the input area displayed on the display150, the touch input within the input area can be handled as a touch on an object pointed by the pointer, instead of a touch on its physical touch point.

According to the aforementioned exemplary embodiment, when a touch operation of the display150is performed by using a hand which grips the electronic device101in a state where the electronic device101is gripped by one hand of the user, there can be an area which cannot be easily touched by a designated finger (e.g., a thumb or an index finger) of the hand which grips the electronic device101. The electronic device101can provide a function of easily performing an operation associated with a touch in the area which cannot be easily touched by the designated finger (e.g., the thumb or the index finger) of the hand which grips the electronic device101, by using a pointer which moves based on a movement of an eye pupil and which is displayed on the display150and an input area displayed on the display150which operates based on a touch.

According to various exemplary embodiments, the input area of the display150may not be explicitly distinctive in the electronic device101. For example, if an entire area of the display150is determined as the input area. In this case, the pointer can be displayed on the input area (e.g., the display150) and thus a touch detected in a portion of the input area can be handled as a touch detected at the pointer.

According to various exemplary embodiments, the electronic device101can divide at least a portion of the area of the display150into a pointer display area which is an area for displaying a pointer based on a movement of an eye pupil (e.g., a designated area of the display150for performing a function of detecting the movement of the eye pupil and displaying the pointer at a position determined as being gazed by the eye pupil) and an input area which detects a touch input handled as being input at a position of a pointer currently displayed on the pointer display area of the display150. According to one exemplary embodiment, the electronic device101can determine the input area in the display150, and can determine the remaining areas other than the determined input area as the pointer display area. Alternatively, the remaining areas other than the determined area of the pointer or a part of the remaining areas can be determined as the input area. In various exemplary embodiments of the present disclosure described below, the pointer can move based on the movement of the eye pupil in a state where the electronic device101is gripped by one hand (e.g., the eye pupil sensing mode) as described above, and the touch area can detect a touch in association with the pointer driven by the movement of eye pupils.

Hereinafter, various exemplary embodiments of the present disclosure are described with reference toFIG. 3B.

A user can touch on an input area301of the display150to perform a function of the electronic device101to select an object indicated by a pointer displayed on the display150.

The electronic device101can select one of images (e.g., an image311) displayed in an album program and can display it on a designated area of the display150. Upon entering an eye pupil sensing mode, the electronic device101can display an input area310to the designated area of the display150. The electronic device101can sense a movement of a user's eye pupil through the camera170, and can display a pointer at a position of the display150according to an eye gaze. The electronic device101can sense a user's eye pupil which gazes an object (i.e., the image311) displayed on the display150, and can display a pointer313which indicates a position gazed by the user's eye in the object (e.g., the image311) displayed on the display150. The electronic device101can input a gesture for selecting the object (e.g., the image311) displayed on the display150to the input area301of the display150and thus can select an object indicated by the pointer displayed on the display150. According to one exemplary embodiment, by performing a gesture (e.g., a single touch) for executing a designated object among objects displayed on the display150of the electronic device101, the object (e.g., the image311) indicated by the pointer31can be executed (e.g., displayed at a designated area of the display150) in the display150.

According to another exemplary embodiment, upon entering the eye pupil sensing mode, the electronic device101can distinguish the input area301of the display150. The electronic device101can sense a user's eye pupil which gazes a designated object (e.g., the image311) of the display150through the camera170, and can display the pointer313which indicates the designated object (e.g., the image311) of the display150. The electronic device101can select the object (e.g., the image311) indicated by the pointer313by performing a gesture (e.g., a touch or a touch-and-hold for more than a designated time), which selects the designated object to move among objects displayed on the display150, in the input area301of the display150. The electronic device101can sense a movement of the user's eye pupil through the camera170in a state of selecting the designated object (e.g., the image311), and can move the pointer displayed on the display150of the electronic device101or the object (e.g., the image311) selected by the pointer according to the movement of the user's eye gaze.

According to one exemplary embodiment, the electronic device101can sense that a user's eye gaze sensed through the camera170moves to a camera folder317of an album program displayed on the display150, and can display (see321) a pointer319and the selected object (e.g., the image311) at a designated position of the camera folder317of the display150gazed by the user's eye pupil. The electronic device101can release a touch on an object selected through a touch or through an input of a touch-and-hold gesture to perform a gesture associated with an completion of moving to a position to which the object is moved by releasing a touch on an input means in a touch and touch-hold state in the input area310of the display150, and can move the selected and moved object (e.g., the image311) to the camera folder317indicated by the pointer319moved through eye pupil sensing.

FIG. 4illustrates the operation of entering an eye pupil sensing mode in an electronic device according to various exemplary embodiments.

The electronic device101can execute the eye pupil sensing mode by using a method designated by a user in a setup mode of the eye pupil sensing mode. According to one exemplary embodiment, the electronic device101can display an icon401in a notification area by using a designated method and is displayed on the display150. If the icon401indicating the eye pupil sensing mode is selected, the electronic device101can display an input area to a designated area of the display150. The electronic device101can handle a gesture which is input to an input area displayed on the display150as a gesture which is input to a pointer displayed on the display150. According to one exemplary embodiment, if the electronic device101selects an icon (e.g., the icon401) associated with an eye pupil sensing mode in a state where a user's eye pupil gazes the camera170, the input area can be displayed on the designated area of the display150, and a pointer which indicates the designated position of the display150can be displayed by sensing a user's eye gaze.

FIG. 5AtoFIG. 5Eillustrate the setup operation of the eye pupil sensing mode in an electronic device according to various exemplary embodiments.

The electronic device101can perform an operation of a pointer displayed on the display150and can perform a setup for an input area through a setup mode of the eye pupil sensing mode. The input area of the display150of the electronic device101can be stored in setup information of the setup mode of the eye pupil sensing mode and thus the input area determined based on the setup information can be displayed on the display150when the electronic device101enters the eye pupil sensing mode. Further, the electronic device101can determine the input area of the display150at a time of entering the eye pupil sensing mode. Furthermore, the electronic device101can change a position of the input area of the display150or a shape of the input area according to a user's selection during an operation of the eye pupil sensing mode.

Hereinafter, various exemplary embodiments of the present disclosure are described with reference toFIG. 5A.

The electronic device101can determine the shape and position of the input area of the display150by setting up the eye pupil sensing mode. According to one exemplary embodiment, the electronic device101can sense a user's eye pupil which gazes the display150of the electronic device101in the setup of the eye pupil sensing mode and can determine whether to use an eye pupil tracking mode which displays a pointer at a designated position of the display150gazed by the eye pupil. The electronic device101can provide a menu501for determining whether to move a pointer displayed on the display150by the movement of the eye pupil, and can provide a menu503capable of selecting On/Off. If On is selected, the electronic device101can display the pointer at the designated position of the display150gazed by the eye pupil.

The electronic device101can provide a menu505for determining the shape of the input area of the display150upon entering an eye pupil mode in the setup of the eye pupil sensing mode. The electronic device101can provide a menu capable of determining the shape of the input area of the display150in various shapes such as a triangle, a square, a circle, and a manual, and can provide a menu capable of determining a size.

The electronic device101can provide a menu509for determining a method of ending the eye pupil mode in the setup of the eye pupil sensing mode. According to one exemplary embodiment, the electronic device101can be configured to end the eye pupil sensing mode when an external portion of the input area of the display150is touched by using one or more input means in a state of the eye pupil sensing mode. When the external portion of the input area of the display150is touched and thus the eye pupil sensing mode ends, the electronic device101can provide a menu511to determine whether to display a notification for notifying the end of the eye pupil sensing mode or to end it without having to display the notification.

The electronic device101can provide a menu513for determining the input area on the display150in the setup of the eye pupil sensing mode. According to one exemplary embodiment, the electronic device101can provide a menu515capable of determining a position of an input area displayed at a designated position of the display150, and can provide a menu517capable of determining an input area to be displayed on the display150if a shape of the input area is determined manually.

Hereinafter, various exemplary embodiments of the present disclosure are described with reference toFIG. 5B.

The electronic device101can determine a shape of an input area in which an input for an object indicated by a pointer on the display150is acquired upon entering an eye pupil sensing mode. According to one exemplary embodiment, the electronic device101can determine a square-shape input area521shown inFIG. 5Bthrough a setup of the eye pupil sensing mode instead of being limited to a circular-shape input area of the display150(e.g., the input area301ofFIG. 3Aor the input area301of theFIG. 3B). The electronic device101can determine a width and/or a height of the square-shape input area of the display150through the setup, and can determine a designated position at which the input area is displayed on the display150. The electronic device101can display two or more input areas at the designated position of the display150without being limited to display one input area521to the display150. When the electronic device101determines the input area in the setup mode, a coordinate range can be input based on a coordinate system configured to the display150of the electronic device101, and an input coordinate area can be determined as the input area.

When the electronic device101displays the determined input area to the display150, the input area can be displayed on a layer different from a layer currently displayed on the display150. According to one exemplary embodiment (FIG. 3Bcan be referred herein), when the electronic device101displays an album program (seeFIG. 3B) to the display150, the electronic device101can display the input area in a layer different from a layer in which the album program is displayed. The electronic device101can display the input area at a designated position of the layer currently displayed on the display150based on setup information of the eye pupil sensing mode. According to one exemplary embodiment, when the album program is currently displayed on the display150, the electronic device101can split the layer in which the album program is displayed, and can display the input area in the split area.

Hereinafter, various exemplary embodiments of the present disclosure are described with reference toFIG. 5C.

When the electronic device101determines a shape of the input area of the display150upon entering an eye pupil sensing mode, without being limited to a designated shape such as a circle, a triangle, a square, etc., a user can use an input means (e.g., a finger, etc.) to perform an input to the display150to determine the input area. According to one exemplary embodiment, the electronic device101can determine an area531which is drawn in a state of touching and touching-and-holding the display150as shown inFIG. 5Cas an input area531. The electronic device101can display the input area531to a designated area of the display150when executing the eye pupil sensing mode by storing the input area531determined as described above in the setup of the eye pupil sensing mode, or can determine the input area at a time of executing the eye pupil sensing mode.

Hereinafter, various exemplary embodiments of the present disclosure are described with reference toFIGS. 5D and 5E.

Referring toFIG. 5D, instead of determining a designated area of the display150, the electronic device101can use an entire area of the display150as an area for displaying a pointer which is responsive to a movement of a user's eye pupil. The electronic device101can receive an input which functions at a pointer on the display150by using an input device included in the electronic device101in addition to the display150.

Referring toFIG. 5E, when gripped by one hand of a user, the electronic device101can include a touch input device (e.g., a touchpad, track pad, or touch sensor541) at a designated position of the electronic device101(e.g., a position designated at a rear portion of the electronic device) in which a designated finger (e.g., an index finger) can be easily moved. The electronic device101can determine the input device (e.g., the touchpad or touch sensor541) operatively coupled to the display150included in the electronic device101as an input area. According to one exemplary embodiment, the electronic device101can display an input area at a designated position of the display150of the electronic device101upon entering an eye pupil sensing mode. The electronic device101can release the display of the input area of the display150upon sensing a gesture which is input to the touchpad541operatively coupled to a touch screen and capable of performing a touch input, and can handle the gesture which is input to the touchpad541as a gesture which is input to a pointer on the display150.FIG. 6illustrates an operation of determining a position of an input area of a display in an electronic device according to various exemplary embodiments.

The electronic device101can include one or more grip sensors at a designated position outside the electronic device101. The grip sensor included at the designated position outside the electronic device101can determine whether the user grips the electronic device101by using a left hand or a right hand. According to one exemplary embodiment, the electronic device101can include a first grip sensor601and a second grip sensor603at a designated position of the electronic device101as shown inFIG. 6. When it is sensed that the electronic device101is gripped by a user's hand, an area of the hand which grips the first grip sensor601and the second grip sensor603can be detected. If an area in contact with the first grip sensor601by the hand is greater than an area in contact with the second grip sensor603, the electronic device101can determine that the electronic device101is gripped by the left hand of the user. If it is determined that the electronic device101is gripped by the left hand of the user, an input area605of the display150in the eye pupil sensing mode can be displayed at a position (e.g., a left or right to a center) designated as a left hand mode with respect to a center of the display150.

FIG. 7illustrates the operation of moving an input area of a display in an electronic device according to various exemplary embodiments.

The electronic device101can change a position of an input area on the display150in a state of an eye pupil sensing mode. According to one exemplary embodiment, the electronic device101can enter a change mode of the input area by inputting a gesture (e.g., a double touch and a touch-and-hold) designated in an input area710of the display150. The electronic device101can sense an input means for dragging the input area701of the display150, and can sense an operation of releasing the input means by a touch at the designated position of the display150. The electronic device101can move the input area701of the display150to a position of an input area703according to a drag operation of the input means. The electronic device101can change a form of the input area of the display150and/or a shape of the input area not only by moving the position of the input area of the display150but also by performing a designated operation stored in setup information of an eye pupil sensing area.

According to various exemplary embodiments, the electronic device101can change the position of the input area of the display150. According to one exemplary embodiment, the electronic device101can determine the position at which the input area is displayed through the setup mode of the eye pupil sensing mode, and can display the input area at the designated position of the display150based on the input area's position stored in the setup information upon entering the eye pupil sensing mode.

According to one exemplary embodiment, if the electronic device101is gripped by one hand in a state of being vertically positioned, as illustrated inFIG. 7, the display150can be touched by an index finger or the display150can be touched by a thumb. If the eye pupil sensing mode is used in a state where the electronic device101is gripped with one hand, an input area can be displayed at a position of the display150at which a thumb or an index finger cannot be easily moved according to a size of a user's hand or a grip state (or a grip condition) of the electronic device101. The electronic device101can change the position of the input area of the display150by using a designated input (e.g., a touch-and-drag). According to various exemplary embodiments, an operation of changing the input area of the display150described with reference toFIG. 7can also be performed in the eye pupil mode in a state where the electronic device101is located vertically, instead of being limited to the eye pupil sensing mode in a state where the electronic device101is located horizontally.

FIGS. 8A and 8Billustrate operations based on a position of a pointer displayed on a display in an electronic device according to various exemplary embodiments.

The electronic device101can release a display of an input area of the display150when a user's eye gaze is located at the input area of the display150, and can perform an input to the display150by using a typical touch operation used in the display150.

According toFIG. 8A, the electronic device101can display a pointer at a designated position of the display150gazed by an eye pupil of the user through the camera170. The electronic device101can display a pointer at a designated object (e.g., an Internet icon803) displayed on the display150by sensing a movement of the eye pupil. The electronic device101can sense that a pointer801displayed on the display150is moved in an input area802as shown inFIG. 8A.

Referring toFIG. 8B, the electronic device101can release a display of the input area802displayed on the display150when the pointer801is located in the input area (e.g., the input area802displayed inFIG. 8Adisplayed on the display150. The electronic device101can release the display of the input area by releasing a display of a layer for displaying the input area to the display150when the input area is displayed on a layer different from a layer for displaying a content, or can stop a function of the input area in a state where the display of the layer is not released. The electronic device101can release the display of the input area or stop the function of the input area when the input area is displayed at a designated position of the layer for displaying the content. When the electronic device101releases the display of the input area displayed on the display150, similarly to a typical touch screen, a touch input can be performed in the designated area of the display150in which the input area is located. According to one exemplary embodiment, the electronic device101can display a connected Internet program to the display150in association with the Internet icon803upon sensing a gesture for touching a designated object (e.g., the Internet icon803) displayed on the display150by using an input means.

The electronic device101can display the input area of which the display is released when it is sensed that the user's eye gaze is deviated from the designated input area of the display150through the camera170, and can recover the stopped function when a function of the input area is stopped.

FIG. 9AandFIG. 9Bare flowcharts illustrating an operation of performing a function of an electronic device by detecting a movement of an eye pupil in an electronic device according to various exemplary embodiments.

Hereinafter, various exemplary embodiments of the present disclosure are described with reference toFIG. 9A.

In operation901, the electronic device101can enter an eye pupil sensing mode capable of performing a function associated with at least a part of objects displayed on the display150through a movement of a user's eye pupil and a designated gesture. The electronic device101can display a pointer at a designated position of the display150gazed by the user's eye pupil through the camera170, and can display at the designated position of the display150the designated input area capable of handling a designated gesture which is sensed and input at the pointer.

In operation903, the electronic device101can display the pointer at the designated position of the display150gazed by the user's eye pupil. The electronic device101can sense a movement of the user's eye pupil through the camera170, and can determine the designated position of the display150gazed by the eye pupil as information regarding a coordinate in a virtual coordinate system configured in the display150. The electronic device101can display a pointer which indicates an object displayed on the display150in a designated coordinate of the display150associated with the eye gaze.

In operation905, the electronic device101can handle a gesture which is input to the input area of the display150as being input at the pointer displayed on the display150. According to one exemplary embodiment, the electronic device101can execute a function associated with an object indicated by the pointer displayed on the display150when a gesture for executing an object (e.g., a single touch) is input to the input area of the display150. The electronic device101can perform the gesture which is input to the input area as described above in the object indicated by the pointer displayed on the display150.

When the electronic device101performs operation905, the exemplary embodiment ofFIG. 9Acan end.

Hereinafter, various exemplary embodiments of the present disclosure are described with reference toFIG. 9B.

In operation911, the electronic device101can display the input area capable of inputting a touch gesture in a designated area of the display150upon entering an eye pupil sensing mode. When the input area locates on the display150, the electronic device101can display an input area having a designated shape at the designated position of the display150according to setup information of the eye pupil sensing mode. Alternatively, the electronic device101can determine the input area by inputting a designated gesture (e.g., a touch and a touch-and-drag) to the display150upon entering the eye pupil sensing mode.

In operation913, the electronic device101can display a pointer which indicates an object at the designated position of the display150gazed by a user's eye through the camera170. According to one exemplary embodiment, the electronic device101can sense a movement of a user's eye pupil through the camera170, and can display the pointer which indicates the object displayed on the display150at the designated position of the display150in association with the eye gaze.

In operation915, the electronic device101can handle a gesture which is input to the input area of the display150as a gesture which is input to the pointer displayed on the display150. When the electronic device101handles the gesture which is input to the input area of the display150as the gesture which is input to the pointer displayed on the display150, a function associated with the gesture of the electronic device101coupled to the object indicated by the pointer can be performed.

When the electronic device101performs operation915, the exemplary embodiment ofFIG. 9Bcan end.

According to various exemplary embodiments, a method of operating an electronic device can include displaying a pointer in a touch screen of the electronic device and determining an input area at a part of a position of the touch screen, moving the pointer based on a movement of a user's eye pupil, and recognizing an input detected in the input area as an input by the pointer.

According to various exemplary embodiments, the input area can be located on the touch screen based on setup information of the eye pupil sensing mode. According to various exemplary embodiments, the input area can be determined by dragging in the touch screen in a touch and touch-and-hold state at a time of entering the eye pupil sensing mode. According to various exemplary embodiments, a position of the input area can be changed to a designated position of the touch screen. According to various exemplary embodiments, the method can further include releasing a function of the input area if the pointer is located in the input area. According to various exemplary embodiments, the method can further include starting the function of the input area if the pointer is moved from the input area to an external portion of the input area. According to various exemplary embodiments, the input area can be displayed in a layer different from a layer including one or more objects in the touch screen. According to various exemplary embodiments, the moving of the pointer can be determined based on a coordinate for a position of the touch screen determined to as position gazed by the eye pupil by sensing a movement of the eye pupil. According to various exemplary embodiments, the input can be determined by a designated gesture which is input by touching the input area. According to various exemplary embodiments, the method can further include releasing the eye pupil sensing mode upon sensing a touch input at the external portion of the input area of the touch screen. According to various exemplary embodiments, the input area can be determined to an area configured in a touch panel operatively coupled to the touch screen.

According to various exemplary embodiments, a method of operating an electronic device can include displaying a pointer in a display of the electronic device and determining an input area at a designated area of a touch panel, moving the pointer based on a movement of a user's eye pupil, and recognizing an input detected in the input area as an input by the pointer.

According to various exemplary embodiments, a method of operating an electronic device can include displaying a pointer display area and an input area by distinguishing the area, displaying a pointer to the pointer display area, and upon detection of a touch in the input area, handling that the touch is input at a position at which the pointer is displayed.

According to various exemplary embodiments, if the electronic device is gripped by one hand, a position of the input area can be determined to a designated position of a display which can be easily touched by a designated finger. According to various exemplary embodiments, the input area can be displayed based on setup information including at least one of position, shape, and size information of the input area. According to various exemplary embodiments, the displaying of the input area can further include determining at least one of a position, shape, and size of the input area by touching or dragging a display in which the input area is displayed when the input area is displayed. According to various exemplary embodiments, a position of the input area can be changed in the display based on a designated input. According to various exemplary embodiments, the method can further include releasing a display of the input area if the pointer is located in the input area. According to various exemplary embodiments, the input area can be displayed on a layer different from the layer to which the pointer area is displayed. According to various exemplary embodiments, a case where a touch is detected in the input area can be a case where a gesture designated to execute at least one function by touching the display is detected in the input area. According to various exemplary embodiments, the method can further include, if the touch is detected in the pointer display area, releasing the display of the pointer display area and the input area. According to various exemplary embodiments, the input area can be additionally determined in a touch panel operatively coupled to the display to which the pointer display area is displayed. According to various exemplary embodiments, the pointer can be displayed based on a user's eye pupil movement, and can be displayed in an overlapping manner on at least one of icons displayed on the display.

According to various exemplary embodiments, a method of operating an electronic device can include displaying a pointer to a display, detecting a touch in an input area independent of the display, and handling a touch detected in the input area as a touch of a position at which the pointer is displayed. According to various exemplary embodiments, the input area can be a touch panel formed independently in a rear portion of the display.

FIG. 10is a block diagram of an electronic device according to various exemplary embodiments.

For example, an electronic device1000can constitute all or some parts of the electronic device101ofFIG. 1. Referring toFIG. 10, the electronic device1000can include one or more processors1010, a Subscriber Identification Module (SIM) card1014, a memory1020, a communication module1030, a sensor module1040, an input module1050, a display1060, an interface1070, an audio module1080, a camera module1091, a power management module1095, a battery1096, an indicator1097, and a motor1098.

In addition, the processor1010(e.g., the processor120) can include one or more Application Processors (APs)1011or one or more Communication Processors (CPs)1013. Although it is illustrated inFIG. 10that the AP1011and the CP1013are included in the processor1010, the AP1011and the CP1013can be included respectively in different Integrated Circuit (IC) packages. According to one exemplary embodiment, the AP1011and the CP1013can be included in one IC package.

The AP1011can control a plurality of hardware or software constitutional elements connected to the AP1011by driving an operating system or an application program, and can process a variety of data including multimedia data and can perform an arithmetic operation. The AP1011can be implemented, for example, with a System on Chip (SoC). According to one exemplary embodiment, the processor1010can further include a Graphic Processing Unit (GPU, not shown).

The CP1013can perform a function of managing a data link and changing a communication protocol in communication between different electronic devices (e.g., the electronic device102, the electronic device104, or the server106) connected through a network with the electronic device1000(e.g., the electronic device101) including hardware elements. The CP1013can be implemented with an SoC. According to one exemplary embodiment, the CP1013can perform at least a part of a multimedia control function. For example, the CP1013can identify and authenticate an electronic device in a communication network by using a subscriber identity module (e.g., the SIM card1014). In this case, the CP1013can provide a service including a voice call, a video call, a text message, or packet data to a user.

Further, the CP1013can control data transmission/reception of the communication module1030. Although constitutional elements such as the CP1013, power management module1095, the memory1020, and the like are illustrated as separate constitutional elements with respect to the AP1011inFIG. 10, according to one exemplary embodiment, the AP1011can also be implemented such that at least one part (e.g., the CP1013) of the aforementioned constitutional elements is included.

According to one exemplary embodiment, the AP1011or the CP1013can load an instruction or data, which is received from each non-volatile memory connected thereto or at least one of different constitutional elements, to a volatile memory and can process the instruction or data. In addition, the AP1011or the CP1013can store data, which is received from at least one of different constitutional elements or generated by at least one of different constitutional elements, into the non-volatile memory.

The SIM card1014can be a card which includes a SIM, and can be inserted to a slot formed at a specific location of the electronic device. The SIM card1014can include unique identification information (e.g., an Integrated Circuit Card IDentifier (ICCID)) or subscriber information (e.g., an International Mobile Subscriber Identity (IMSI)).

The memory1020(e.g., the memory130) can include an internal memory1022or an external memory1024. The internal memory1022can include, for example, at least one of a volatile memory (e.g., a Dynamic RAM (DRAM), a Static RAM (SRAM), a Synchronous Dynamic RAM (SDRAM), etc.) or a non-volatile memory (e.g., a One Time Programmable ROM (OTPROM), a Programmable ROM (PROM), an Erasable and Programmable ROM (EPROM), an Electrically Erasable and Programmable ROM (EEPROM), a Mask ROM, a Flash ROM, a NAND flash memory, a NOR flash memory, etc.). According to one exemplary embodiment, the internal memory1022can have a form of a Solid State Drive (SSD). The external memory1024can be a flash drive, and can further include, for example, Compact Flash (CF), Secure Digital (SD), Micro Secure Digital (Micro-SD), Mini Secure digital (Mini-SD), extreme Digital (xD), memory stick, and the like. The external memory1024can be operatively coupled to the electronic device1000via various interfaces. According to one exemplary embodiment, the electronic device1000can further include a storage unit (or a storage medium) such as a hard drive.

The communication module1030(e.g., the communication interface160) can include a wireless communication module1031or a Radio Frequency (RF) module1034. The wireless communication module1031can include, for example, a WiFi1033, a BlueTooth (BT)1035, a Global Positioning System (GPS)1037, or a Near Field Communication (NFC)1039. For example, the wireless communication module1031can provide a wireless communication function by using a radio frequency. Additionally or alternatively, the wireless communication module1031can include a network interface (e.g., a LAN card), modem, or the like for connecting the electronic device1000to a network (e.g., Internet, LAN, WAN, telecommunication network, cellular network, satellite network, POTS, etc.).

The RF module1034can serve to transmit/receive data, for example, to transmit/receive an RF signal. Although not shown, the RF module1034can include, for example, a transceiver, a Power Amp Module (PAM), a frequency filter, a Low Noise Amplifier (LNA), and the like. The RF module1034can further include a component for transmitting/receiving a radio wave on a free space in a wireless communication, for example, a conductor, a conducting wire, and the like.

The sensor module1040can measure a physical quantity or detect an operation state of the electronic device1000, and thus can convert the measured or detected information into an electric signal. The sensor module1040can include, for example, at least one of a gesture sensor1040A, a gyro sensor1040B, a pressure sensor1040C, a magnetic sensor1040D, an acceleration sensor1040E, a grip sensor1040F, a proximity sensor1040G, a color sensor1040H (e.g., a Red, Green, Blue (RGB) sensor), a bio sensor1040I, a temperature/humidity sensor1040J, an illumination sensor1040K, and an Ultra Violet (UV) sensor1040M. Additionally or alternatively, the sensor module1040can include, for example, an E-node sensor (not shown), an ElectroMyoGraphy (EMG) sensor (not shown), an ElectroEncephaloGram (EEG) sensor (not shown), an ElectroCardioGram (ECG) sensor (not shown), a fingerprint sensor (not shown), and the like. The sensor module1040can further include a control circuit for controlling at least one or more sensors included therein.

The input module1050can include a touch panel1052, a (digital) pen sensor1054, a key1056, or an ultrasonic input unit1058. The touch panel1052can recognize a touch input, for example, by using at least one of an electrostatic type, a pressure-sensitive type, and an ultrasonic type. The touch panel1052can further include a control circuit. In case of the electrostatic type, not only a physical contact but also a proximity recognition is possible. The touch panel1052can further include a tactile layer. In this case, the touch panel1052can provide the user with a tactile reaction.

The (digital) pen sensor1054can be implemented, for example, by using the same or similar method of receiving a touch input of the user or by using an additional sheet for recognition. The key1056can be, for example, a physical button, an optical key, a keypad, or a touch key. The ultrasonic input unit1058is a device by which a user equipment detects a sound wave through a microphone (e.g., a microphone1088) by using a pen which generates an ultrasonic signal, and is a device capable of radio recognition. According to one exemplary embodiment, the electronic device1000can use the communication module1030to receive a user input from an external device (e.g., a network, a computer, or a server) connected thereto.

The display1060(e.g., the display150) can include a panel1062, a hologram1064, or a projector1066. The panel1062can be, for example, a Liquid-Crystal Display (LCD), an Active-Matrix Organic Light-Emitting Diode (AM-OLED), and the like. The panel1062can be implemented, for example, in a flexible, transparent, or wearable manner. The panel1062can be constructed as one module with the touch panel1052. The hologram1064can use an interference of light and show a stereoscopic image in the air. The projector1066can display an image by projecting a light beam onto a screen. The screen can be located, for example, inside or outside the electronic device1000. According to one exemplary embodiment, the display1060can further include a control circuit for controlling the panel1062, the hologram1064, or the projector1066.

The interface1070can include, for example, a High-Definition Multimedia Interface (HDMI)1072, a Universal Serial Bus (USB)1074, an optical communication interface1076, or a D-subminiature (D-sub)1078. The interface1070can be included, for example, in the communication interface160ofFIG. 1. Additionally or alternatively, the interface1070can include, for example, a Mobile High-definition Link (MHL) (not shown), a Secure Digital (SD)/Multi-Media Card (MMC) (not shown) or Infrared Data Association (IrDA) (not shown).

The audio module1080can bilaterally convert a sound and electric signal. At least some constitutional elements of the audio module1080can be included in, for example, the input/output interface140ofFIG. 1. The audio module1080can convert sound information which is input or output, for example, through a speaker1082, a receiver1084, an earphone1086, the microphone1088, and the like.

The camera module1091is a device for image and video capturing, and according to one exemplary embodiment, can include one or more image sensors (e.g., a front sensor or a rear sensor), a lens (not shown), an Image Signal Processor (ISP) (not shown), or a flash (not shown, e.g., LED or xenon lamp).

The power management module1095can manage a power of the electronic device1000. Although not shown, the power management module1095can include, for example, a Power Management Integrated Circuit (PMIC), a charger Integrated Circuit (IC), or a battery fuel gauge.

The PMIC can be placed, for example, inside an IC or SoC semiconductor. Charging can be classified into wired charging and wireless charging. The charger IC can charge a battery, and can avoid an over-voltage or over-current flow from a charger. According to one exemplary embodiment, the charger IC can further include a charger IC for at least one of the wired charging and the wireless charging. The wireless charging can be classified into, for example, a magnetic resonance type, a magnetic induction type, and an electromagnetic type. An additional circuit for the wireless charging, for example, a coil loop, a resonant circuit, a rectifier, and the like, can be added.

The battery gauge can measure, for example, a residual quantity of the battery1096and a voltage, current, and temperature during charging. The battery1096can store or generate an electricity, and can supply a power to the electronic device1000by using the stored or generated electricity. For example, the battery1096can include a rechargeable battery or a solar battery.

The indicator1097can indicate a specific state, for example, a booting state, a message state, a charging state, and the like, of the electronic device1000or a part thereof (e.g., the AP1011). The motor1098can convert an electric signal into a mechanical vibration. Although not shown, the electronic device1000can include a processing unit (e.g., a GPU) for supporting a mobile TV. The processing unit for supporting the mobile TV can process media data according to a protocol of, for example, Digital Multimedia Broadcasting (DMB), Digital Video Broadcasting (DVB), media flow, and the like.

According to various exemplary embodiments, a method of operating an electronic device, the method comprising displaying a specific position in a display; and handling a touch detected in the display as a touch of the specific position. Wherein the specific position is a coordinate associated with the display based on an eye pupil and a movement of the eye pupil. Wherein the displaying of the specific position includes displaying at least one of a pointer, an identifier, and an indicator at the specific position or displaying a state where an object associated with the specific position is selected. Wherein the handling includes handling the detected touch as a touch on a coordinate associated with the specific position. Wherein the detected touch is a touch detected in a portion of the display other than the specific position. Wherein the detected touch is a touch detected in an input area configured in at least a portion of the display. Wherein the detected touch is a single touch or a touch including a drag. Wherein a layer configured to display the specific position is a layer different from a layer configured to input the touch. Further comprising handling a touch which is input to a touch panel operatively coupled to the display as a touch of the specific position. Wherein the detected touch is handled not as a touch of a position at which the detected touch occurs but as a touch of the specific position.

Each of the aforementioned constitutional element of the electronic device according to various exemplary embodiment of the present disclosure can consist of one or more components, and names of corresponding constitutional elements can vary depending on a type of electronic device. The electronic device according to various exemplary embodiments of the present disclosure can include at least one of the aforementioned constitutional elements. Some of the constitutional elements can be omitted, or additional other constitutional elements can be further included. In addition, some of the constitutional elements of the electronic device according to various exemplary embodiments of the present disclosure can be combined and constructed as one entity, so as to equally perform functions of corresponding constitutional elements before combination.

According to various exemplary embodiments, an electronic device comprising a display configured to display a specific position and configured to detect a touch, and a processor configured to provide control to display the specific position in the display, and configured to handle the touch detected in the display as a touch of the specific position. Wherein the processor determines the specific position to a coordinate associated with the display based on an eye pupil and a movement of the eye pupil. Wherein the processor displays at least one of a pointer, an identifier, and an indicator at the specific position or displays a state where an object associated with the specific position is selected. Wherein the processor handles the detected touch as a touch on a coordinate associated with the specific position. Wherein the processor handles a touch detected in a portion of the display other than the specific position as the detected touch. Wherein the processor handles a touch detected in an input area configured in at least a portion of the display as the touch of the specific position. Wherein the processor determines a single touch or a touch including a drag as the detected touch. Wherein the processor configures a layer configured to display the specific position and a layer configured to input the touch as different layers. Wherein the processor handles a touch which is input to a touch panel operatively coupled to the display as a touch of the specific position. Wherein the processor handles the detected touch not as a touch of a position at which the detected touch occurs but as a touch of the specific position.

The electronic device can move a pointer displayed on a display in association with an eye gaze by sensing a user's eye pupil, and can handle this as a gesture which is input to a pointer by touching a designated position of the display. Therefore, the electronic device can perform a function associated with an object displayed at a designated position of the display which cannot be touched by a designated finger of a hand which grips the electronic device.

Various exemplary embodiments of the present disclosure can be performed by at least one of one or more programs included in a memory of the electronic device101, one or more processors (e.g., the processor120), and the gesture processing module180.

According to various exemplary embodiments, at least a part of an apparatus and method according to various exemplary embodiments disclosed in the claims and/or specification of the present disclosure can be implemented in a form (e.g., a module) including two or more combination of hardware, software, and firmware. The “module” can be a minimum unit of an integrally constituted component or can be a part thereof. The “module” can be a minimum unit for performing one or more functions or can be a part thereof. The “module” can be mechanically or electrically implemented. When implemented in software, computer-interpretable storage medium (or a computer-readable storage medium) for storing one or more programs (or programming modules) can be provided. For example, the software can be implemented with an instruction stored in a computer-readable storage medium in a programming module form. The one or more programs include instructions for allowing the electronic device to execute the methods based on the embodiments disclosed in the claims and/or specification of the present disclosure. If the instruction is executed by one or more processors (e.g., the processor210), the one or more processors can perform a function corresponding to the instruction. The computer-readable storage media can be, for example, the memory130. At least some parts of the programming module can be implemented (e.g., executed), for example, by the processor120. At least some parts of the programming module can include modules, programs, routines, sets of instructions, processes, and the like, for performing one or more functions.

The program (i.e., the software module or software) can be stored in a random access memory, a non-volatile memory including a flash memory, a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a magnetic disc storage device, a Compact Disc-ROM (CD-ROM), Digital Versatile Discs (DVDs) or other forms of optical storage devices, and a magnetic cassette. Alternatively, the program can be stored in a memory configured in combination of all or some of these storage media. In addition, the configured memory can be plural in number.

Further, the program can be stored in an attachable storage device capable of accessing the electronic device through a communication network such as the Internet, an Intranet, a Local Area Network (LAN), a Wide LAN (WLAN), or a Storage Area Network (SAN) or a communication network configured by combining the networks. The storage device can access an electronic device via an external port. Further, an additional storage device can access a portable electronic device on a communication network. The aforementioned hardware device can be configured to operate as one or more software modules to perform the operation for various exemplary embodiments of the present disclosure, and the other way around is also possible.

The module or programming module according to various exemplary embodiments of the present disclosure can further include at least one or more constitutional elements among the aforementioned constitutional elements, or may omit some of them, or may further include additional other constitutional elements. Operations performed by a module, programming module, or other constitutional elements according to various exemplary embodiments of the present disclosure may be executed in a sequential, parallel, repetitive, or heuristic manner. In addition, some of the operations may be executed in a different order or may be omitted, or other operations may be added.