Patent Description:
Conventionally, as gaze tracking technology using a camera has developed, a technology has emerged which provides information customized to the interests of users by means of tracking the gaze of a user so as to enlarge or reduce an object which is displayed on a screen and being focused on, or to display related information around the object being focused on.

For example, there may be a technology that tracks the gaze of a user and enlarges or reduces a predetermined portion when the user gazes at the predetermined portion on the screen.

As another example, information associated with each player may be displayed when a user gazes at players on a playground, or information associated with ancient artifacts may be displayed in a historical site. The conventional technology provides a user with detailed information by enlarging a predetermined area or displaying related text information when a user focuses on the predetermined area. However, when an image displayed on a screen or photographed in an environment where the user actually gazes at an object through a camera attached onto a face, or the like has a low resolution, or the image is dark due to an insufficient amount of ambient light, or the image is unclear due to insufficient exposure, or a distance to a desired object such as a sign or the like is distant, the conventional technology has difficulty in providing a user with the desired visual information even though it enlarges the image.

Also, a technology that displays information associated with a target object through texts may provide detailed information which does not appear in visual information but may not provide detailed visual information such as an appearance, a color, or the like of an actual object. Document <CIT> discloses enhancing the experience of a user wearing a see-through, near eye mixed reality display device. Document <CIT> discloses rendering an image based on a relative position and orientation of a display in relation to a viewer's head. Document <CIT> discloses a device that estimates a user's degree of interest. Document <CIT> discloses a method for improving visibility of each object by differently displaying each object from the background when providing an augmented reality service.

The present invention has been made to at least partially solve, reduce, or remove at least one of the problems and/or disadvantages related to the conventional art, and to provide at least the advantages described below. Accordingly, an aspect of the present invention is to improve an information providing method that enlarges an existing image and displays text information, and to provide a high-visibility image by tracking a gaze of a user, determining a focus object, reproducing an image or replacing the image with a related image or the like, so as to enable the user to readily identify the object and to provide the user with plenty of visual information.

In accordance with an aspect of the present invention, there is provided a method of improving visibility using gaze tracking, the method including detecting a gaze of a user by means of a camera; determining a focus object at which the user gazes from among at least one object viewed through a display unit of an electronic device or an object in an extended line of the gaze of the user in a transparent display or the like; and displaying, on the display unit, an image with high visibility that corresponds to the focus object and has a higher visibility than the focus object. For example, the image with higher visbility (i.e. a high-visibility image corresponding to the focus object) may have at least one of higher resolution, higher definition, and higher brightness than a, particularly previously displayed, image corresponding to the focus object (e.g. a retrieved or previously photographed image).

In accordance with another aspect of the present invention, there is provided an electronic device, including a camera that photographs an eye of a user; a display unit that displays at least one object; and a controller that detects a gaze of a user from the photographed eye of the user, determines a focus object at which the user gazes from among the at least one object, and displays, through the display unit, an image with high visibility that corresponds to the focus object and has a higher visibility than the focus object.

In accordance with another aspect of the present invention, there is provided a machine-readable storage medium having a program recorded thereon, which when executed, performs a method of improving visibility using gaze tracking, the method including detecting a gaze of a user by means of a camera; determining a focus object at which the user gazes from among at least one object viewed through a display unit of an electronic device; and displaying, on the display unit, an image with high visibility that corresponds to the focus object and has higher visibility than the focus object.

In accordance with another aspect of the present invention, there is provided an electronic device including a machine-readable storage medium having recorded thereon a program, which when executed, performs a method of improving visibility using gaze tracking, the method including detecting a gaze of a user by means of a camera; determining a focus object at which the user gazes from among at least one object viewed through a display unit of an electronic device; and displaying, on the display unit, an image with high visibility that corresponds to the focus object and has higher visibility than the focus object.

In accordance with another aspect of the present invention, there is provided a wearable electronic device including means for positioning the wearable electronic device on a user's body; at least one camera that photographs an eye of the user; at least one display unit that displays at least one object; and a controller configured to detect a gaze from the photographed eye of the user, determine a focus object at which the user gazes from among the at least one object, and display, through the display unit, an image with high visibility that corresponds to the focus object and has higher visibility than the focus object.

The above and other aspects, features, and advantages of the present invention will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:.

The present invention may have various modifications and various embodiments, among which specific embodiments will now be described more fully with reference to the accompanying drawings. However, it should be understood that there is no intent to limit the present invention to the specific embodiments, but on the contrary, the present invention covers all modifications, equivalents, and alternatives falling within the scope of the description herein.

Although terms including an ordinal number such as first, second, etc. can be used for describing various elements, the elements are not restricted by the terms. For example, without departing from the scope of the present invention, a first structural element may be referred to as a second structural element. Similarly, the second structural element also may be referred to as the first structural element.

The terms used in this application are for the purpose of describing particular embodiments only and are not intended to limit the invention. In the description, it should be understood that the terms "include" or "have" indicate existences of a feature, a number, a step, an operation, a structural element, parts, or a combination thereof, and do not previously exclude the existences or probability of addition of one or more another features, numeral, steps, operations, structural elements, parts, or combinations thereof.

Unless defined differently, all terms used herein, which include technical terminologies or scientific terminologies, have the same meaning as understood by a person skilled in the art to which the present invention belongs. Such terms as those defined in a generally used dictionary are to be interpreted to have the meanings equal to the contextual meanings in the relevant field of art, and are not to be interpreted to have ideal or excessively formal meanings unless clearly defined in the present specification.

The electronic device of the present invention may be a certain device, and may be referred to as a portable device, a mobile terminal, a communication terminal, a portable communication terminal, a portable and mobile terminal and so on.

For example, the electronic device may be a smartphone, a portable phone, a game player, a TV, a display unit, a heads-up display unit for a vehicle, a notebook computer, a laptop computer, a tablet Personal Computer (PC), a Personal Media Player (PMP), a Personal Digital Assistant (PDA), or the like. An electronic device may be embodied as a pocket-sized portable communication terminal or an attachable communication terminal including a wireless communication function. Also, the electronic device may be a flexible device, a flexible display device, a transparent display device, or the like.

An electronic device according to the embodiments of the present invention may be a portable terminal, and a few component elements may be omitted or changed in a representative configuration of the electronic device as necessary.

<FIG> is a block diagram illustrating a first electronic device according to the first embodiment of the present invention, <FIG> is a front side perspective view of the first electronic device according to the first embodiment of the present invention (the front side in this application referring to the side facing the user in normal use), and <FIG> is a back side perspective view of the first electronic device according to the first embodiment of the present invention (the back side in this application referring to a side opposite to the side facing the user in normal use).

Referring to <FIG>, the first electronic device <NUM> may be connected to an external electronic device through a communication module <NUM>, a connector, or the like. The external electronic device may include a Bluetooth communication unit, a Near Field Communication (NFC) unit, a WiFi Direct communication unit and a wireless Access Point (AP), which are wirelessly connectable. Further, the first electronic device <NUM> may be connected with another portable terminal or another electronic device, for example, one of a portable phone, a smart phone, a tablet Personal Computer (PC), a desktop PC, and a server in a wired or wireless manner.

Referring to <FIG>, the first electronic device <NUM> includes at least one touch screen <NUM>. Also, the first electronic device <NUM> includes a controller <NUM>, a communication module <NUM>, a camera module <NUM>, a input/output module <NUM>, a first sensor module <NUM>, and a storage unit <NUM>.

The controller <NUM> may include a CPU <NUM>, a ROM <NUM> in which a control program for controlling the first electronic device <NUM> is stored, and a RAM <NUM> which stores signals or data input from outside of the first electronic device <NUM> or is used as a storage region for operations performed by the first electronic device <NUM>. The CPU <NUM> may include a single core, a dual core, a triple core, or a quadruple core. The CPU <NUM>, the ROM <NUM>, and the RAM <NUM> may be connected to each other through an internal bus.

Also, the controller <NUM> controls the communication module <NUM>, the camera module <NUM>, the input/output module <NUM>, the sensor module <NUM>, the storage unit <NUM>, and the touch screen <NUM>.

The controller <NUM> detects a user input being input using input means (e.g. a user's finger), which touches or approaches an object on the touch screen <NUM> or is located close to the object in a state where a plurality of objects or items are displayed on the touch screen <NUM>. The controller <NUM> identifies an object corresponding to a position on the touch screen <NUM> where the user input is generated. The user input through the touch screen <NUM> includes one of a direct touch input that directly touches the object and a hovering input corresponding to an indirect touch input that approaches the object within a preset recognition range but does not directly touch the object. According to the present invention, a user input includes a gesture input through the camera module <NUM>, a switch/button input through the button <NUM>, a voice input through the microphone <NUM>, or the like, in addition to the user input through the touch screen <NUM>.

An object or item (or a function item) is displayed or may be displayed on the touch screen <NUM> of the first electronic device <NUM>. For example, the object or item indicates at least one of an image, an application, a menu, a document, a widget, a picture, a video, an E-mail, a Short Message Service (SMS) message, and a Multimedia Message Service (MMS) message, and may be selected, executed, deleted, canceled, stored, or changed by a user input means. The term "item" has an inclusive meaning and includes a button, an icon (or a short-cut icon), a thumbnail image, and a folder storing at least one object in the portable terminal. Further, the item may be displayed in the form of an image, text and /or the like.

The short-cut icon is an image displayed on the touch screen <NUM> of the first electronic device <NUM> for use in a rapid execution of each application, or a voice call, an address, a menu, or the like which are basically provided in the first electronic device <NUM>, and executes a corresponding application when a command or a selection for the execution of the short-cut icon is input.

When a user input event is generated with respect to a preset item or object, or based on a predetermined scheme, the first controller <NUM> executes a predetermined program operation corresponding to the user input event.

The first electronic device <NUM> includes at least one of a mobile communication module <NUM>, a wireless LAN module <NUM>, and a short range communication module <NUM> according to the performance thereof.

The mobile communication module <NUM> enables the first electronic device <NUM> to be connected with an external electronic device through mobile communication by using one antenna or a plurality of antennas under a control of the controller <NUM>. The mobile communication module <NUM> transmits/receives, through a communication network provided by a communication company, a wireless signal for voice call, video call, an SMS, or an MMS to/from a portable phone, a smart phone, a tablet PC, or another device, having a phone number that may be input into the first electronic device <NUM>.

The sub-communication module <NUM> includes at least one of the wireless LAN module <NUM> and the short range communication module <NUM>. For example, the sub-communication module <NUM> may include only the wireless LAN module <NUM> or only the short range communication module <NUM>. Alternatively, the sub-communication module <NUM> may also include both the wireless LAN module <NUM> and the short range communication module <NUM>. The sub-communication module <NUM> may also include a plurality of short range communication modules.

The wireless LAN module <NUM> may be Internet-connected in a place where a wireless Access Point (AP) is installed according to a control of the controller <NUM>. The wireless LAN module <NUM> supports a wireless LAN standard (IEEE802.11x) of the Institute of Electrical and Electronics Engineers (IEEE). The short range communication module <NUM> wirelessly performs short range communication between the first electronic device <NUM> and an external electronic device, under a control of the controller <NUM>. The short range communication scheme may include a Bluetooth communication scheme, an Infrared Data Association (IrDA) scheme, a Wi-Fi Direct communication scheme, a Near Field Communication (NFC) scheme, or the like.

The camera module <NUM> includes at least one of the first camera <NUM> and the second camera <NUM> for photographing a still image or a video according to a control of the controller <NUM>. Also, the camera module <NUM> may further include a flash <NUM> that provides an auxiliary light source required for photographing a subject. The first camera <NUM> may be disposed on a front side of the first electronic device <NUM>, and the second camera <NUM> may be disposed on a back side of the first electronic device <NUM>.

Each of the first and second cameras <NUM> and <NUM> may include a lens system, an image sensor, and the like. Each of the first and second cameras <NUM> and <NUM> converts an optical signal input (or photographed) through the lens system to an electrical image signal (or a digital image) and outputs the converted electrical image signal to the controller <NUM>. Then, the user photographs a video or a still image through the first and second cameras <NUM> and <NUM>.

The input/output module <NUM> includes at least one button <NUM>, at least one microphone <NUM>, at least one speaker <NUM>, and a connector.

The button <NUM> may be formed on the front side, a lateral side, or the back side of a housing (or a case) of the electronic device <NUM>, and may include at least one of a power/lock button, a volume button, a menu button, a home button, a back button, and a search button.

The microphone <NUM> receives a voice or a sound, and generates an electrical signal according to a control of the controller <NUM>.

The speaker <NUM> outputs sounds corresponding to various signals or data (for example, wireless data, broadcasting data, digital audio data, digital video data and the like) to the outside of the first electronic device <NUM>, under a control of the controller <NUM>. A connector may be used as an interface for connecting the first electronic device <NUM> with an external electronic device or an electric power source. The controller <NUM> may transmit data stored in the storage unit <NUM> of the first electronic device <NUM> to an external electronic device or may receive data from an external electronic device, through a cable connected to a connector by wire.

The sensor module <NUM> includes at least one sensor that detects a state of the first electronic device <NUM> or an ambient environment state. For example, the sensor module <NUM> may include at least one of a proximity sensor for detecting whether or not a user approaches the first electronic device <NUM>, an illuminance sensor for detecting an amount of light surrounding the first electronic device <NUM>, a motion sensor for detecting a movement of the first electronic device <NUM> (for example, a rotation, acceleration or vibration of the first electronic device <NUM>), a gyroscope for detecting a rotational movement of the first electronic device <NUM>, an accelerometer for detecting an acceleration movement of the first electronic device <NUM>, a geo-magnetic sensor for detecting a point of a compass of the first electronic device <NUM> by using a magnetic field of the Earth, a gravity sensor for detecting the direction of gravity's pull, an altimeter for measuring atmospheric pressure so as to detect an altitude, a GPS module <NUM>, and the like.

Also, the first sensor module <NUM> includes a first biometric sensor <NUM> and a second biometric sensor <NUM>.

Referring to <FIG>, the first biometric sensor <NUM> is disposed on the front side of the first electronic device <NUM>, and the first biometric sensor <NUM> includes a first infrared ray light source <NUM> and a first infrared ray camera <NUM>. The first infrared ray light source <NUM> outputs infrared rays, and the first infrared ray camera <NUM> detects infrared rays reflected from the subject. For example, the first infrared ray light source <NUM> may be formed of an LED array in a matrix structure.

For example, the first infrared ray camera may include a filter that passes an infrared light and blocks light in a wavelength band other than that of the infrared light, a lens system that focuses the infrared light having passed the filter, and an image sensor that converts an optical image formed by the lens system to an electric image signal. For example, the image sensor may be formed in a PD array of a matrix structure.

Referring to <FIG>, the second biometric sensor <NUM> is disposed on the back side of the first electronic device <NUM>, and the second biometric sensor <NUM> has a configuration identical to the first biometric sensor <NUM> and includes a second infrared ray light source <NUM> and a second infrared ray camera <NUM>.

The GPS module <NUM> receives electromagnetic waves from a plurality of GPS satellites in the Earth's orbit and calculates a position of the first electronic device <NUM> by using a time of arrival from the GPS satellites to the first electronic device <NUM>.

Also, the storage unit <NUM> stores signals or data input/output according to operations of the communication module <NUM>, the camera module <NUM>, the input/output module <NUM>, the sensor module <NUM>, and the touch screen <NUM>, based on a control of the controller <NUM>. The storage unit <NUM> stores a control program and applications for a control of the first electronic device <NUM> or the controller <NUM>.

Also, the storage unit <NUM> may store various applications such as a navigation, video calls, games, time-based alert applications, or the like; images to provide a Graphical User Interface (GUI) related to the applications; a database or data related to user information, documents and a visibility improvement method using gaze tracking; background images (such as a menu screen, an idle screen, or the like) or operating programs for operating the first electronic device <NUM>; images taken by the camera module <NUM>, and the like.

Also, the storage unit <NUM> stores a program and related data for implementing a visibility improvement method using gaze tracking according to the present invention.

The storage unit <NUM> is a machine (e.g., a computer)-readable medium, and the term "machine-readable medium" may be defined as a medium for providing data to the machine such that the machine may perform a specific function. The storage unit <NUM> may include a non-volatile medium and a volatile medium. All of these media should be tangible entities so that commands transferred by the media may be detected by a physical instrument through which the machine reads the commands.

The computer readable storage medium includes, but is not limited to, at least one of a floppy disk, a flexible disk, a hard disk, a magnetic tape, a Compact Disc Read-Only Memory (CD-ROM), an optical disk, a punch card, a paper tape, a RAM, a Programmable Read-Only Memory (PROM), an Erasable PROM (EPROM), and a Flash-EPROM.

Furthermore, the first electronic device <NUM> includes at least one touch screen <NUM> providing a user with user graphical interfaces corresponding to various services (for example, a phone call, data transmission, broadcasting, and photographing).

The touch screen <NUM> receives at least one user input through a user's body, for example, a finger, or an input unit, for example, a stylus pen, an electronic pen, and the like.

The touch screen <NUM> may receive successive motions in one touch (that is, a drag input). The term "touch" used in the present specification is not limited to a contact between the touch screen <NUM> and a finger or an input unit, and may include a non-contact (for example, a case where a user input means is located within a recognition distance (for example, <NUM>) where the user input means may be detected without a direct contact with the touch screen <NUM>).

The touch screen <NUM> may be embodied as a resistive type, a capacitive type, an infrared type, an acoustic wave type, an ElectroMagnetic Resonance (EMR) type, or a combination thereof.

Referring to <FIG> and <FIG>, the touch screen <NUM> is disposed in the center of the front side <NUM> of the first electronic device <NUM>. The touch screen <NUM> may have a large size to occupy most of the front side <NUM> of the first electronic device <NUM>.

The touch keys such as the home button 161a, the menu button 161b, the back button 161c and the like, mechanical keys, or a combination thereof may be arranged at a lower portion of the touch screen <NUM>. Further, the touch keys may be formed as a part of the touch screen <NUM>.

The main home screen is displayed on the touch screen <NUM> through the home button 161a.

The menu button 161b provides a connection menu which may be displayed on the touch screen <NUM>. The connection menu includes a widget addition menu, a background changing menu, a search menu, an editing menu, a configuration setup menu and the like.

The back button 161c may be used for displaying a screen that was executed immediately before a currently executed screen, or for terminating the most recently used application.

The first camera <NUM>, an illuminance sensor 170a, a proximity sensor 170b, and the first biometric sensor <NUM> may be disposed on an upper portion (i.e., the upper portion in normal use) of the front side <NUM> of the first electronic device <NUM>. The second camera <NUM>, the flash <NUM>, the speaker <NUM>, and the second biometric sensor <NUM> may be disposed on the back side <NUM> of the first electronic device <NUM>.

The lateral side <NUM> of the first electronic device <NUM> may include, for example, a power/lock button, a volume button 161e including a volume up button 161f and a volume down button <NUM>, one or a plurality of microphones <NUM>, and the like.

Also, a connector may be formed on a lower end of the lateral side of the first electronic device <NUM>. A plurality of electrodes are formed on the connector, and may be connected by wire to an external device.

The present invention provides a visibility improvement method using gaze tracking, and the visibility improvement method using gaze tracking may be applied to an object on a general touch screen, a surrounding object viewed through a transparent touch screen, an augmented reality, a Head Mounted Display (HMD) device, and the like.

The controller <NUM> controls general operations of the first electronic device <NUM>, and the controller <NUM> controls other component elements in the first electronic device <NUM> so as to execute a visibility improvement method using gaze tracking according to the present disclosure.

<FIG> is a perspective view of a second electronic device viewed from the external side according to a second embodiment of the present invention (the external side in this application referring to the side opposite to the side facing the user in normal use), and <FIG> is a perspective view of the second electronic device viewed from the internal side (the internal side in this application referring to the side facing the user in normal use). The second electronic device <NUM> according to the second embodiment is a Head Mounted Display (HMD).

The second electronic device <NUM> is shaped like a pair of glasses, corresponds to a portable terminal (or a portable communication terminal), and includes a housing <NUM> and a circuit contained in the housing <NUM>.

The housing <NUM> includes a front frame <NUM> in which first and second windows <NUM>, (e.g. panes or eyeglass lenses), and <NUM>, respectively, corresponding to a left eye and a right eye are fixed, and first and second temple frames <NUM> and <NUM> that are folded to the inside or spread to the outside through first and second hinges. Hereinafter, a left eye and a right eye may be referred to as a first eye and a second eye, and the first and second windows <NUM> and <NUM> may referred to as first and second window panels, respectively.

A third camera <NUM> is disposed on the internal side of the front frame <NUM> (i.e. the frame to be arranged on the front of a user's head), a fourth camera <NUM> is disposed on the external side of the front frame <NUM>. The third and fourth cameras <NUM> and <NUM> are disposed on a portion of the front frame <NUM> between the first and second window <NUM> and <NUM> (that is, a bridge of general glasses).

A microphone <NUM> is disposed on the external side of the front frame <NUM>, and a touch sensor <NUM> is disposed on the external side of the front frame <NUM>.

A first aperture <NUM> through which a first projected light <NUM> output from a first projector <NUM> disposed in the housing <NUM> is output to the outside of the front frame <NUM>, is disposed on the internal side of the front frame <NUM>. Also, a second aperture <NUM> through which a second projected light <NUM> output from a second projector <NUM> disposed in the housing <NUM> is output to the outside of the front frame <NUM>, is disposed on the internal side of the front frame <NUM>.

At least one button <NUM> is disposed on the external side of the first temple frame <NUM>.

At least one speaker <NUM> is disposed on the internal side of the first and second temple frames <NUM> and <NUM>.

The first projector <NUM> outputs the first projected light <NUM> for forming a first virtual image, the first projected light <NUM> output from the first projector <NUM> is focused and reflected by the first window <NUM>, and the focused and reflected first projected light <NUM> forms the first virtual image on a retina of a left eye <NUM> of the user. In this example, the term "focus" refers to collecting light, and has a meaning including convergence that collects light at a point, reducing a beam spot of light, or the like. Desirably, the reflected first incident light <NUM> is collected in a crystalline lens or a pupil of the left eye <NUM>.

The second projector <NUM> outputs the second projected light <NUM> for forming a second virtual image, the second projected light <NUM> output from the second projector <NUM> is focused and reflected by the second window <NUM>, and the focused and reflected second projected light <NUM> forms the second virtual image on a retina of a right eye <NUM> of the user. Desirably, the reflected second incident light <NUM> is collected in a crystalline lens or a pupil of the right eye <NUM>. Although the present example sets forth the case of two projectors, a single projector may be used.

In the following description, the first virtual image and the second virtual image are identical except for being displayed in the left eye and the right eye, respectively. However, the present invention is not limited thereto, and may display only one of the first virtual image and the second virtual image.

<FIG> is a diagram of a circuit configuration of the second electronic device. A circuit configuration of the second electronic device <NUM> is similar to the circuit configuration of the first electronic device <NUM> and thus, duplicate descriptions will be omitted.

The second electronic device <NUM> includes an input/output module <NUM>, a storage unit <NUM>, a sensor module <NUM>, a communication module <NUM>, the touch sensor <NUM>, the third and fourth cameras <NUM> and <NUM>, the first and second projectors <NUM> and <NUM>, the first and second windows <NUM> and <NUM>, and a controller <NUM>.

The input/output module <NUM> includes at least one button <NUM>, at least one microphone <NUM>, at least one speaker <NUM>, a connector, a keypad, or a combination thereof, as a means of receiving a user input, informing a user of information, receiving data from the outside, and/or outputting data to the outside.

The microphone <NUM> may be disposed on the external side of the front frame <NUM>. In the present description, the terms "signals" and "data" may be interchangeable, and also the terms "data" and "data signals" may be interchangeable.

The button <NUM> is provided for receiving a user input, and is used for turning the second electronic device <NUM> on or off, selecting and/or searching for a menu item or an item, or the like. The button <NUM> may be disposed on the external side of the first temple frame <NUM>.

The sensor module <NUM> includes at least one sensor that detects a state of the second electronic device <NUM> or an ambient environment state.

The communication module <NUM> may be a wired, wireless, or wired/wireless communication unit, wiredly or wirelessly transmits data from the controller <NUM> to an external electronic device through an external communication network or the air, or wiredly or wirelessly receives data from the external electronic device through an external communication network or the air and transfer the received data to the controller <NUM>.

The touch sensor <NUM> transmits a signal corresponding to at least one touch input, to the controller <NUM>. The controller <NUM> recognizes from the touch input information, user input information such as selection or movement of a menu item or an item, a handwritten input, or the like, and executes a function corresponding to the user input information (connection of a call, photographing with a camera, writing/viewing a message, transmitting data, or the like).

Each of the third and fourth cameras <NUM> and <NUM> may include a lens system and an image sensor, and may additionally include a flash, and the like. Each of the third and fourth cameras <NUM> and <NUM> converts light input (or photographed) through a corresponding lens system into an electronic image signal, and outputs the image signal to the controller <NUM>. A user may photograph a still image or record a video through the cameras <NUM> and <NUM>.

The controller <NUM> processes an image input from the cameras <NUM> and <NUM>, an image stored in the storage unit <NUM>, or an image formed by the controller <NUM> using data stored in the storage unit <NUM>, and outputs an image converted to be appropriate for screen output properties (a size, a definition, a resolution, or the like) of the first projector <NUM> and/or second projector <NUM> to the outside through the first projector <NUM> and/or the second projector <NUM>.

The controller <NUM> provides images corresponding to various services (for example, phone call, data transmission, broadcasting, photographing, and the like) to the user through the first projector <NUM> and/or the second projector <NUM>.

The user may view an image in which a surrounding landscape image obtained by an ambient light and a virtual image obtained by a light input from the projector <NUM> and <NUM> overlap, and the virtual image may be viewed by the user as if a transparent layer is layered on the surrounding landscape.

The controller <NUM> controls general operations of the second electronic device <NUM>, and controls other component elements in the second electronic device <NUM> so as to execute a visibility improvement method using gaze.

The controller <NUM> displays, to a user through the first projector <NUM> and/or the second projector <NUM>, image data stored in the storage unit <NUM>, or received through the communication module <NUM>, or formed by the controller <NUM> based on data stored in the storage unit <NUM> according to a user command, selection of a menu item or an icon, or event information, input through the sensor module <NUM>, the input/output module <NUM>, the third camera <NUM> or fourth camera <NUM>, or the touch sensor <NUM>,. In this example, the image may be a still image or a video.

The storage unit <NUM> stores signals or data according to a control of the controller <NUM>. The storage unit <NUM> may store a control program and applications for a control of the second electronic device <NUM> or the controller <NUM>.

Hereinafter, a visibility improvement method using gaze tracking will be described with reference to the first electronic device.

<FIG> is a flowchart illustrating a visibility improvement method using gaze tracking according to the first embodiment of the present invention.

As a user detection step, the controller <NUM> detects a user through the first camera <NUM> or the first biometric sensor <NUM> in step S110. The controller <NUM> determines whether a subject included in the image photographed by the first camera <NUM> includes a face.

For example, the facial recognition is generally executed based on a facial recognition method, and a facial recognition technology using a facial contour line, a color and/or texture of facial skin, a template, and/or the like stored in the storage unit <NUM> may be used. For example, the controller <NUM> learns faces through facial images of a plurality of users, and may recognize a face from an input image based on the face learning. The face learning information is stored in the storage unit <NUM>.

Also, the controller <NUM> may determine whether a user registered in advance with the first electronic device <NUM> is detected from the image input from the first camera <NUM>. The registration of the user may be executed by storing information used for detecting the user in a database of the storage unit <NUM>. The information used for detecting a registered user may include a facial image of the user, features points of the facial image of the user (or referred to as a feature image, a feature pattern, or the like), and/or the like. The feature point may be an edge, a corner, an image pattern, or a contour line.

For example, the controller <NUM> may determine whether a user in an input image is identical to a registered user by matching feature points of a registered user image and feature points detected from the input image.

The controller <NUM> determines whether a subject included in the image photographed by the first camera <NUM> includes a body part.

The body part may include an eye, an iris, a face, a blood vessel (e.g., a vein), and the like.

Referring to <FIG>, an eye <NUM> of a user includes a pupil <NUM>, an iris <NUM>, and a sclera <NUM>. The first infrared ray light source <NUM> emits infrared rays into the eye <NUM> of the user, and the first infrared ray camera <NUM> photographs the eye <NUM> of the user so as to output an eye image.

For example, eye recognition is executed based on a general eye recognition method, and eye recognition technology based on an eye contour line, a template, and/or the like stored in the storage unit <NUM> may be used. For example, the controller <NUM> learns eyes through eye images of a plurality of users, and may recognize an eye from an input image based on the eye learning. The eye learning information is stored in the storage unit <NUM>.

Also, the controller <NUM> compares iris information of a user stored in the storage unit <NUM> (e.g., an iris image, feature point information, and/or the like) and an input image, so as to recognize an iris of the user, and to determine whether the user in the input image is identical to a registered user.

As described above, the controller <NUM> may also detect a user through the third camera <NUM>.

As a gaze detection step, the controller <NUM> detects a gaze of the user from the input image in step S120. The controller <NUM> may detect the gaze of the user from the input image based on general eye tracking technology.

Referring to <FIG>, the controller <NUM> detects a gaze (a direction of a gaze) of the user from a pose (a position and/or a direction) of the pupil <NUM>.

As a focusing state determination step, the controller <NUM> determines whether the user focuses on (in this application, this may mean, for example, that the user steadily gazes at) an object in step S130. For example, the controller <NUM> may determine whether the user focuses on an object based on a facial expression in an input image, a gazing time, or a change in the pupil <NUM>.

<FIG> illustrate a method of determining whether the user focuses on an object based on a facial expression.

<FIG> illustrates a facial expression when a user does not focus on an object, and displays two eyes <NUM>, two eyebrows <NUM>, corrugators muscles <NUM> (corrugators supercilii muscles) between the two eyebrows <NUM> in a glabella portion <NUM>, and procerus muscle <NUM> between the two eyes <NUM> in the glabella portion <NUM>.

<FIG> illustrates a facial expression when a user focuses on an object. When a user focuses on an object, the user generally frowns and whether the user focuses on an object may be determined based on a change in muscles of the glabella portion <NUM>. For example, when a user focuses on an object, corrugators muscles move and thus, a distance between the two eyebrows <NUM> becomes narrow. Accordingly, wrinkles <NUM> appear in the glabella portion <NUM> and the nose is drawn upward due to a movement of procerus muscle <NUM>, and thus wrinkles <NUM> appears on the bridge of the nose.

Also, the controller <NUM> determines whether the user focuses on an object based on the gazing time, and when a gaze of a user is detected, sets a timer including a predetermined time limit (that is, a threshold time). For example, the time limit of the timer may be <NUM> second. The timer counts down from <NUM> to <NUM> seconds and notifies the controller <NUM> of the expiration of the timer when a remaining time of the timer is <NUM> seconds, or the controller <NUM> may sense the expiration of the timer. Alternatively, the timer may count time from <NUM> seconds and notifies the controller <NUM> of the expiration of the timer when the elapsed time is <NUM> second, or the controller <NUM> senses the expiration of the timer. When the gaze does not change during the time limit of the timer, the controller <NUM> determines that the user focuses on an object.

Alternatively, the controller <NUM> determines whether the user focuses on an object based on a change in the pupil <NUM> of the user. That is, whether the user focuses on an object is determined based on the phenomenon in which the pupil <NUM> is enlarged while the user focuses on an object.

The controller <NUM> executes step S140 when the user focuses on an object, and periodically or continuously monitors whether the user focuses on an object while the user is not focusing on any object, in particular, until it is detected that the user focuses on an object.

As a focus object/area detection step, the controller <NUM> determines an area where a gaze of the user intersects an image surface (that is, a focus area), and detects an object placed in the focus area in step S140.

<FIG> is a diagram illustrating a method of detecting a focus object according to the first embodiment of the present invention. The present embodiment describes the case in which the touch screen <NUM> is a non-transparent touch screen.

An image surface <NUM> is a screen or a surface of the touch screen <NUM>, and an area in which the gaze <NUM> of the user's eye <NUM> intersects the image surface <NUM> is a focus area <NUM>. The controller <NUM> determines an object <NUM> in the focus area <NUM>.

<FIG> is a diagram illustrating a method of detecting a focus object according to the second embodiment of the present invention. The present embodiment describes the case in which the touch screen <NUM> is a non-transparent touch screen.

A surrounding landscape (e.g. surrounding the device and/or user) limited by an angle of view <NUM> of the second camera <NUM>, that is, an image of a surrounding landscape photographed by the second camera <NUM>, is an image surface <NUM>, and an area in which a gaze <NUM> of a user's eye <NUM> intersects the image surface <NUM> is a focus area <NUM>.

<FIG> is a diagram illustrating a method of detecting a focus object according to a third embodiment of the present invention. The present embodiment describes the case in which the second electronic device <NUM> detects a focus object. The focus object includes a person, an animal, a plant, an object, a building, a mountain, or the like included in a surrounding landscape.

A surrounding landscape <NUM> limited by an angle of view of a fourth camera <NUM>, that is, an image of a surrounding landscape area <NUM> photographed by the fourth camera <NUM>, is an image surface <NUM>, and an area in which a gaze <NUM> of a user's eye <NUM> intersects the image surface <NUM> is a focus area <NUM>. The controller <NUM> determines that a tree image <NUM> included in the focus area <NUM> which corresponds to a tree <NUM> in a surrounding landscape <NUM> is the focus object. That is, an area or an object in an extended line of a gaze of the user in or through the transparent window <NUM> and <NUM> may be a focus area or a focus object. In other words, an area or an object in the extended line of a gaze of the user in or through a transparent display such as a transparent touch screen <NUM>, a transparent window <NUM> and <NUM>, or the like, may be the focus area or the focus object.

Referring back to <FIG>, as an image search and display step, the controller <NUM> searches for an image with high visibility corresponding to a focus object, and displays the retrieved image in step S150.

The image with high visibility includes a high-resolution (or high-definition) image, an image with enhanced visibility, a vector graphics image, an image in a short-distance, an image photographed when a great amount of light is provided (that is, an image photographed in a state in which an ambient illuminance is high), or the like. The image with high visibility has a higher resolution, definition, or brightness than the focus object, based on an identical size.

The controller <NUM> may search the storage unit <NUM> for an image with high visibility corresponding to a focus object having low visibility and, for example, the controller <NUM> determines an image with high visibility stored in the storage unit <NUM> through an image matching process.

Alternatively, the controller <NUM> may search an external server for an image with high visibility. For example, the controller <NUM> transmits, to a server, an image including a focus object having low visibility (or the image and location information of the focus object) and/or information associated with the focus object, and receives an image with high visibility retrieved from the server. The information associated with the focus object may include location information of the first electronic device <NUM> at the time of photographing the focus object, pose (compass direction or direction) information of the first electronic device <NUM>, image source information (image-related program, a web address from which an image is downloaded, or the like), and/or the like.

<FIG> illustrate a method of displaying an image with high visibility according to the first embodiment of the present invention.

Referring to <FIG>, the eye <NUM> of the user focuses on a thumbnail image <NUM> displayed on the touch screen <NUM> of the first electronic device <NUM>, and is captured by first camera <NUM>.

<FIG> is a comparison example of the present invention, and illustrates an enlarged image <NUM> of a thumbnail image having low resolution.

Referring to <FIG>, the controller <NUM> determines related information of the thumbnail image <NUM> (that is, location information of an original image), and displays an enlarged original image <NUM> on the touch screen <NUM>.

As shown in <FIG>, when the thumbnail image having low resolution is enlarged, the unclear image <NUM> having low definition is displayed. Conversely, as shown in <FIG>, when an original image associated with a thumbnail image is used, the clear image <NUM> having high definition is displayed.

<FIG> illustrate a method of displaying an image with high visibility according to the second embodiment of the present invention.

Referring to <FIG>, an image <NUM> obtained by photographing first through third boxes <NUM>, <NUM>, and <NUM> placed on a table is displayed on the touch screen <NUM> of the first electronic device <NUM>. The controller <NUM> determines a focus area <NUM> including the first box <NUM> based on a gaze of a user.

Referring to <FIG>, the first controller <NUM> transmits the focus area image <NUM> to a server.

Referring to <FIG>, the first controller <NUM> receives, from a server, an image with high visibility <NUM> associated with the first box included in the focus area, and displays the image with high visibility 830a on the touch screen <NUM>.

<FIG> illustrate a method of displaying an image with high visibility according to the third embodiment of the present invention.

Referring to <FIG>, an image <NUM> obtained by photographing a bird, having low visibility, is displayed on the touch screen <NUM> of the first electronic device <NUM>. The controller <NUM> determines the image <NUM> having low visibility as a focus object, based on the gaze of a user.

Referring to <FIG>, when an image with high visibility corresponding to a focus object is not retrieved, the controller <NUM> generates an image having high definition through image filtering or applying a visual effect, and displays the generated the high-definition image <NUM> on the touch screen <NUM>.

Also, a parameter of the camera used for photography is adjustable. For example, when an amount of light around the bird is excessive or insufficient, an object focused on is not captured well due to a camera aperture or a white balance value. In this example, to display an image with high visibility, the object focused on is checked and the parameter of the camera is adjusted based on the object.

<FIG> illustrate a method of displaying an image with high visibility according to a fourth embodiment of the present invention.

Referring to <FIG>, a user focuses on a character <NUM> displayed on the touch screen <NUM> of the first electronic device <NUM>, and the controller <NUM> determines a focus area <NUM> based on a gaze of the user, and determines the character <NUM> having low visibility due to its size, to be a focus object.

<FIG> is a comparison example of the present invention, and illustrates that an image <NUM> which is difficult to identify is generated when a raster graphics image or a bitmap image representing a small character is enlarged.

Referring to <FIG>, the controller <NUM> generally recognizes a character which is a focus object in a bitmap image through a character recognition method, searches the storage unit <NUM> for a vector image corresponding to the focus object, and enlarges the retrieved vector image <NUM> for display. Also, when the character which is the focus object is a vector image, the character may be displayed by increasing the size of the character.

<FIG> is a block diagram illustrating a configuration of a server.

A server <NUM> includes a communication module <NUM>, a storage unit <NUM>, and a controller <NUM>.

The communication module <NUM> executes wired or wireless communication with the first electronic device <NUM> or the second electronic device <NUM>.

The storage unit <NUM> includes a program including instructions that instruct the first electronic device <NUM> or the second electronic device <NUM> to execute a method of displaying an image with high visibility, update information of the program, an image database, a street panorama database, a two-dimensional (2D) map, and/or the like.

The street panorama database includes street view images mapped to a plurality of different locations. The street panorama database is a map database, and stores an actual three-dimensional (3D) map formed of actual images obtained by photographing cities or streets through a camera. Here, the actual 3D map refers to a 3D map embodied by photographing an actual street using a vehicle, an airplane, or the like and using the photographed actual image. Such an actual 3D map is obtained by photographing a city or a street using a stereo camera mounted on a vehicle or the like. Therefore, it may obtain not only three dimension coordinates (x, y and z coordinates) of objects included in the photographed image, but also depth information corresponding to the distance between the camera used in the photographing and the objects. In addition, such a 3D map may be implemented by photographing a plurality of 2D images for a wide area using an airplane, extracting depth information in an overlapping area between two neighboring 2D images among the photographed images, and performing 3D modeling through 3D mapping. In addition, each of the objects included in the actual 3D map has a plurality of pieces of 3D information and depth information items. For example, each of a plurality of pixels expressing each object may have 3D information and depth information. As a result, the actual 3D map may differentiate not only a position of a specific building but also the contour of the building, for example, the front, rear and side views of the specific building, and may also differentiate respective floors of the building. On the contrary, an existing 2D map is different from the above-described 3D in that since only GPS information is used from the 2D map, only a single location information item may be provided for a specific building but the front and back sides or respective floors of the building may not be differentiated in detail.

The controller <NUM> transmits a program or update information to the first electronic device <NUM> or the second electronic device <NUM> automatically, or in response to a request from the first electronic device <NUM> or the second electronic device <NUM>.

Also, the controller <NUM> matches a focus object image (and/or related information) received from the first electronic device <NUM> or the second electronic device <NUM> and an image stored in the image database or the street panorama database, and transmits a matched image with high visibility to the first electronic device <NUM> or the second electronic device <NUM>. The controller <NUM> improves the matching speed and accuracy based on the information associated with the focus object received from the first electronic device <NUM> or the second electronic device <NUM>. Each database may store an image, and also store location information corresponding to the image, including a location, a pose (a compass direction), or the like, image source information (such as an image-related program, and a web-address from which an image is downloaded), and/or the like.

While the above embodiments have been described using the first electronic device <NUM> as an example, the embodiments of the present invention may be applied to the first electronic device <NUM> or the second electronic device <NUM>, in the same or similar manner. Hereinafter, other examples of a method of displaying an image with high visibility using the second electronic device <NUM> will be provided.

<FIG>, <FIG> illustrate a method of displaying an image with high visibility according to a fifth embodiment of the present invention.

When a plurality of focus objects exists in a direction of a gaze of a user, the controller <NUM> may virtually move a viewpoint of the user based on a command of the user. The movement of the viewpoint corresponds to enlarging a focus object for display in the described examples, or may correspond to reducing the focus object.

Referring to <FIG>, when a first focus object <NUM> and a second focus object <NUM> exist in a direction <NUM> of a gaze of a user's eye <NUM>, although the controller <NUM> displays an image with high visibility associated with the first focus object <NUM> that is closest to the user, the controller <NUM> may move a virtual viewpoint back and forth along the direction of the gaze based on a command of the user.

<FIG> illustrates that the user views the first focus object <NUM> and the second focus object <NUM> through the second window <NUM> of the second electronic device <NUM>.

Referring to <FIG> and <FIG>, an enlarged first focus object <NUM> (that is, an image with high visibility) at a first virtual viewpoint <NUM> between the user and the first focus object <NUM> is displayed through the second window <NUM>. In this manner, the enlarged first focus object <NUM> is displayed through the second window <NUM>.

Referring to <FIG> and <FIG>, an enlarged second focus object <NUM> (that is, an image with high visibility) at a second virtual viewpoint <NUM> between the first focus object <NUM> and the second focus object <NUM> is displayed on the second window <NUM>.

The controller <NUM> transmits an image of a focus object and related information (e.g., location information of the second electronic device <NUM>, pose (or compass direction or direction) information of the second electronic device <NUM>, and/or the like) to the server <NUM>, and receives and displays an image with high visibility of a focus object received from the server <NUM>. The virtual viewpoint may be switched by a command of the user (a voice, a gesture, or a touch). Also, a provided image may be switched by a change in a facial expression of the user (providing an image of a distant object when focus is maintained).

<FIG> illustrate a change of a focus object.

When an image with high visibility is displayed on the second window <NUM>, an image placed in a location different from the intention of a user may be displayed due to an error in an angle of a detected gaze. When the image with high visibility is displayed, a movement of the gaze of a user is detected and a focus object is changed so as to display an image that meets the intention of the user.

Referring to <FIG>, even though the gaze <NUM> of a user is actually directed towards a second object <NUM>, it is mistakenly determined that a gaze <NUM> of the user is directed towards a first object <NUM> due to an error in focus object determination.

Referring to <FIG>, the controller <NUM> mistakenly measures that the user focuses on the first object <NUM>, and displays an image with high visibility <NUM> of the first object <NUM> in the center of the second window <NUM>.

Subsequently, the controller <NUM> tracks a gaze of the user, determines that a gaze <NUM> of the user is directed towards an image with high visibility <NUM> of the second object <NUM>, and changes the focus object.

Referring to <FIG>, the controller <NUM> changes the focus object, and displays the image with high visibility <NUM> of the second object <NUM> in the center of the second window <NUM>.

Although the present embodiment tracks a gaze of a user, and corrects a provided object image based on an object that the user focuses on in the provided image, detailed adjustments of the provided image may be available based on a command of the user (a gesture, a voice, a touch, or the like), in addition to sensing a gaze of a user.

<FIG> illustrate a method of displaying an image with high visibility according to a sixth embodiment of the present invention.

<FIG> illustrates that the user <NUM> gazes at a dark focus object <NUM> through the second window <NUM> of the second electronic device <NUM> during a dark night.

Referring to <FIG>, a brightly enlarged image <NUM> of the focus object <NUM> (that is, an image with high visibility) is displayed on the second window <NUM>.

The controller <NUM> transmits an image of the focus object <NUM> and related information (e.g., location information of the second electronic device <NUM> and pose (or compass direction or direction) information of the second electronic device <NUM>, and/or the like) to the server <NUM>, and displays the image with high visibility <NUM> of the focus object <NUM> received from the server <NUM>.

In the above described examples, after displaying an image with high visibility to a user, returning to a previous screen may be executed by cancelling a focusing state. For example, when a user no longer gazes at a corresponding object, such as when a gaze of a user changes, when a user no longer frowns, or the like, the controllers <NUM> and <NUM> may display a screen which was displayed immediately before displaying an image with high visibility. In the case of a transparent touch screen or the second electronic device <NUM>, the controllers <NUM> and <NUM> may interrupt the display of an image with high visibility.

<FIG> illustrate a method of interrupting the display of an image with high visibility according to an embodiment of the present invention.

<FIG> illustrates an enlarged webpage <NUM> which is an image with high visibility and is displayed on the touch screen <NUM> of the first electronic device <NUM>.

Referring to <FIG>, when a user <NUM> sighs <NUM> and makes a sound such as 'Whew', the controller <NUM> displays a screen which was displayed immediately before displaying an image with high visibility, that is, a webpage <NUM> before being enlarged. The controller <NUM> determines whether a focusing state is cancelled using a fricative sound created when the user's breath passes through the microphone <NUM> shown in <FIG>.

Although the controllers <NUM> and <NUM> determine whether a user focuses on (or steadily gazes at) an object in the above described example, whether the user focuses on an object may be determined based on a command directly from the user, in addition to the determination based on a facial expression and a gazing time. It may be regarded as determining whether a user command is received instead of determining whether the user focuses on an object.

<FIG> is a flowchart illustrating a visibility improvement method using gaze tracking according to a second embodiment of the present invention.

As a user detection step, the controller <NUM> detects a user through the first camera <NUM> or the first biometric sensor <NUM> in step S210. The controller <NUM> determines whether a subject included in an image photographed by the first camera <NUM> or the first biometric sensor <NUM> includes a face or a body part.

As a gaze detection step, the controller <NUM> detects the gaze of the user from the input image in step S220.

As a user command determination step, the controller <NUM> determines whether a user command is received in step S230.

The command may be provided through any means that transfers user intentions such as a voice, a touch, an input through a physical key/button, a user gesture, a motion of the first electronic device <NUM>, a facial expression of a user, a breath of a user, a brainwave, and/or the like,.

For example, the user may input a voice command such as "enlarge", "reduce", "brighten", or the like, and the controller <NUM> receives the voice command through the microphone <NUM> to enlarge, reduce, or brighten an image with high visibility for display. For reducing, the controller <NUM> displays an image with high visibility by reducing the image with high visibility to be smaller than a corresponding object, or reduces a currently displayed image with high visibility and displays the reduced image. For enlarging, the controller <NUM> additionally enlarges a currently displayed image with high visibility and displays the enlarged image.

For example, the controller <NUM> may display a button for instructing the display of an image with high visibility on the touch screen <NUM>, or may map "enlarge" and "reduce" functions to the volume up button 161f and the volume down button <NUM> which are physical keys.

For example, the controller <NUM> may monitor whether a gesture set in advance is detected through the fourth camera <NUM>. Examples of the gesture include, for example, assigning a focus object through a motion that indicates an object viewed through the window <NUM> and <NUM> with a finger, enlarging or reducing a focus object through a pinching motion of a finger (a motion of closing or opening fingers), and moving a virtual viewpoint through a motion that moves a finger back and forth (a motion that moves a finger to be close or distant to/from the second electronic device <NUM>). For example, the user makes a motion that moves the first electronic device <NUM> back and forth, or a motion that tilts the first electronic device <NUM>. The controller <NUM> may sense a movement of the first electronic device <NUM> through the first sensor module <NUM>. The controller <NUM> enlarges an image with high visibility and displays the enlarged image when the first electronic device <NUM> moves or is tilted in the forward direction (that is, a direction facing the user), and reduces an image with high visibility and displays the reduced image when the first electronic device <NUM> moves or is tilted in the backward direction (that is, a direction distant from the user).

For example, the first controller <NUM> may determine a degree of enlargement of the image with high visibility based on a degree of a frown on the user's face. Alternatively, when an image with high visibility is already displayed, the first controller <NUM> additionally enlarges the image with high visibility when the user further frowns, and reduces the image with high visibility when the user no longer frowns.

A virtual viewpoint may be determined based on a degree of a frown on a face. In the case where an image with high visibility is already displayed, when the user further frowns, the second controller <NUM> moves the virtual viewpoint to be closer or more distant than before.

When a user command is received, the controller <NUM> executes step S240, and when a user command is not received, the controller <NUM> periodically or continuously monitors whether a user command is received or not.

As a focus object/area detection step, the controller <NUM> determines an area (that is, a focus area) where a gaze of a user intersects an image surface, and detects an object that is located in the focus area in step S240.

As an image search and display step, the first controller <NUM> searches for an image with high visibility corresponding to a focus object, and displays the retrieved image in step S250. In this example, the controller <NUM> displays the retrieved image based on the user command of step S230.

<FIG> is a flowchart illustrating a visibility improvement method using gaze tracking according to the third embodiment of the present invention.

As a user command receiving step, the controller <NUM> receives a user command through any means that transfers a user intention such as a voice, a touch, an input through a physical key/button, a user gesture, a motion of the first electronic device <NUM>, a facial expression of a user, a breath of a user, a brainwave, or the like, in step S310.

As a user detection step, the controller <NUM> detects a user through the first camera <NUM> or the first biometric sensor <NUM> in step S320.

As a gaze detection step, the controller <NUM> detects a gaze of the user from an input image, in step S330.

As a focus object/area detection step, the controller <NUM> determines an area (that is, a focus area) where the gaze of the user intersects an image surface, and detects an object located on the focus area, in step S340.

As an image search and display step, the controller <NUM> searches for an image with high visibility corresponding to a focus object, and displays the retrieved image in step <NUM>. In this example, the first controller <NUM> displays the retrieved image based on the user command of step S310.

Although the touch screen has been illustrated as a representative example of the display unit displaying a screen in the above-described embodiments, a general display unit or a transparent display, e.g. a Liquid Crystal Display (LCD), an Organic Light Emitting Diode (OLED), and a Light Emitting Diode (LED), which do not have a touch detection function, may also be used instead of the touch screen.

Also, the first window <NUM> and the second window <NUM> of the second electronic device <NUM> may be a display unit(s) or a transparent display.

It will be appreciated that the embodiments of the present invention may be implemented in software, hardware, or a combination thereof. Any such software may be stored, for example, in a volatile or non-volatile storage device such as a ROM, a memory such as a RAM, a memory chip, a memory device, or an IC, or a optical or magnetic recordable and machine (e.g., computer) readable medium such as a CD, a DVD, a magnetic disk, or a magnetic tape, regardless of its ability to be erased or its ability to be re-recorded. It is appreciated that the storage unit included in the electronic device is one example of the machine-readable storage media suitable for storing a program or programs including commands for implementing various embodiments of the present invention. Accordingly, the present invention includes a program that includes a code for implementing an apparatus or a method defined in any claim in the present specification and a machine-readable storage medium that stores such a program. Further, the program may be electronically transferred by a predetermined medium such as a communication signal transferred through a wired or wireless connection, and the present invention appropriately includes equivalents of the program.

Claim 1:
A method of improving visibility using gaze tracking, the method comprising:
displaying a first image on a display of an electronic device, wherein the first image is a first view of a three-dimensional, 3D, map including at least one of a front of a building, a rear of the building, or a side of the building;
while the first image is displayed on the display of the electronic device, detecting a gaze of a user by means of a camera of the electronic device;
determining a focus object based on a gazing time of the gaze of the user;
receiving, from a server, a second image with high visibility than the first image including the focus object, wherein the second image is a second view of the 3D map;
displaying, on the display of the electronic device, the second image with high visibility received from the server, wherein the second image with high visibility has at least one of higher resolution, higher definition, or higher brightness than at least one of a corresponding resolution, corresponding definition, or corresponding brightness of the first image including the focus object; and
while the second image is displayed on the display of the electronic device, moving a virtual viewpoint of the user based on the gaze of the user.