Patent Description:
Augmented reality (AR) glasses are a growing market. Current AR glasses only project the content at a fixed distance (usually infinity). Even though there are optical methods to change the focal plane, suitable methods and systems to track the user focal distance in a portable head-mounted AR system do not exist. That is, a current optical head-mounted display draws information at a fixed distance, while a user's focal distance varies. Therefore, if the eye does not focus at the focal distance, images will blur and may produce dizziness. <NPL> describes features, functionality and methods used in eye typing. <NPL>, describes technical considerations for using eye movements as an input medium. <NPL>, describes eye tracking technology. <NPL> describes eye movement based interaction. <CIT> describes a system to determine the gaze of a user. <CIT> describes a system including a wearable device having at least one optical sensor configured to capture optical data corresponding to a view area and one or more eye tracking sensors configured to detect eye movement. <NPL> describes gaze control technology. <CIT> describes a near-eye display system. <CIT> describes a system for and method of projecting augmentation imagery in a head-mounted display. <CIT> describes an optical system for a head-worn computer.

This disclosure provides a system and method for controlling a display system based on information from focal point tracking or gaze tracking, or both, such as for use a head mounted display system.

According to an embodiment of the present disclosure, a method for displaying an image by an electronic device is provided. The method comprises determining, by an electronic device, a gaze and a focal point of an eye of the user; based on information associated with at least one of the gaze and the focal point of the eye of the user, controlling, by the electronic device, a focal point of an adjustable lens; presenting, through a display screen of the electronic device, an image of an object based on at least one of the gaze and the focal point of the eye of the user; controlling the focal point of the adjustable lens to vary a focus of the image of the object presented through the display screen based on at least one of the gaze and the focal point of the eye of the user; and creating a simultaneous localization and mapping (SLAM) image using the determined focal point of the eye of the user.

According to another embodiment of the present disclosure, an electronic device for displaying an image is provided. The electronic device comprises a display screen; and at least one processor coupled to the display screen, wherein the at least one processor is configured to, determine a gaze and a focal point of an eye of the user, and based on at least one of the gaze and the focal point of the eye of the user, control a focal point of an adjustable lens (<NUM>); present, through the display screen (<NUM>), an image of an object based on at least one of the gaze and the focal point of the eye of the user, wherein the at least one processor (<NUM>) is configured to control the focal point of the adjustable lens to vary a focus of the image of the object presented through the display screen (<NUM>) based on at least one of the gaze and the focal point of the eye of the user; and create a simultaneous localization and mapping (SLAM) image using the determined focal point of the eye of the user.

According an example of the present disclosure outside of the scope of the claims, a non-transitory computer readable medium is provided. The non-transitory computer readable medium, stores instructions, when executed by at least one processor , to cause the at least one processor to perform operations of controlling at least one of an operation of the electronic device, or a focal point of an adjustable lens based on at least one of a gaze of the user or a focal point of the eye of the user, and presenting, using a display screen of the electronic device, an image of an object based on at least one of the gaze of the user or the focal point of the eye.

For a more complete understanding of this disclosure and its advantages, reference is now made to the following description, taken in conjunction with the accompanying drawings, in which:.

The description which follows describes a number of embodiments. The embodiments are only embodiments of the invention where the device and/or method is in accordance with the appended claims. The reader will appreciate that features of the embodiments which do not fall within the scope of the invention may nevertheless be incorporated in embodiments of the invention which do fall within the scope of the invention. For this reason, the description of the embodiments which are absent these features are retained in order to provide useful background information to the reader.

As used herein, the terms "have," "may have," "include," "may include," "can have," or "can include" a feature (e.g., a number, function, operation, or a component such as a part) indicate the existence of the feature and do not exclude the existence of other features.

As used herein, the terms "A or B," "at least one of A and/or B," or "one or more of A and/or B" may include all possible combinations of A and B. For example, "A or B," "at least one of A and B," "at least one of A or B" may indicate all of (<NUM>) including at least one A, (<NUM>) including at least one B, or (<NUM>) including at least one A and at least one B.

As used herein, the terms "first" and "second" may modify various components regardless of importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate different user devices from each other regardless of the order or importance of the devices. For example, a first component may be denoted a second component, and vice versa without departing from the scope of the present disclosure.

It will be understood that when an element (e.g., a first element) is referred to as being (operatively or communicatively) "coupled with/to," or "connected with/to" another element (e.g., a second element), it can be coupled or connected with/to the other element directly or via a third element. In contrast, it will be understood that when an element (e.g., a first element) is referred to as being "directly coupled with/to" or "directly connected with/to" another element (e.g., a second element), no other element (e.g., a third element) intervenes between the element and the other element.

As used herein, the terms "configured (or set) to" may be interchangeably used with the terms "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of" depending on circumstances. The term "configured (or set) to" does not essentially mean "specifically designed in hardware to. " Rather, the term "configured to" may mean that a device can perform an operation together with another device or parts.

For example, the term "processor configured (or set) to perform A, B, and C" may mean a generic-purpose processor (e.g., a CPU or application processor) that may perform the operations by executing one or more software programs stored in a memory device or a dedicated processor (e.g., an embedded processor) for performing the operations.

The terms as used herein are provided merely to describe some embodiments thereof, but not to limit the scope of other embodiments of the present disclosure. It is to be understood that the singular forms "a," "'an," and "the" include plural references unless the context clearly dictates otherwise. All terms including technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of the present disclosure belong. In some cases, the terms defined herein may be interpreted to exclude embodiments of the present disclosure.

For example, examples of the electronic device according to embodiments of the present disclosure may include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a PDA (personal digital assistant), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (e.g., smart glasses, a head-mounted device (HMD), electronic clothes, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch).

According to embodiments of the present disclosure, the electronic device can be a smart home appliance. Examples of the smart home appliance can include at least one of a television, a digital video disk (DVD) player, an audio player, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washer, a drier, an air cleaner, a set-top box, a home automation control panel, a security control panel, a TV box (e.g., Samsung HomeSync™ Apple TV™ or Google TV™ , a gaming console (Xbox™ PlayStation™, an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.

According to certain embodiments of the present disclosure, examples of the electronic device can include at least one of various medical devices (e.g., diverse portable medical measuring devices (a blood sugar measuring device, a heartbeat measuring device, or a body temperature measuring device), a magnetic resource angiography (MRA) device, a magnetic resource imaging (MRI) device, a computed tomography (CT) device, an imaging device, or an ultrasonic device), a navigation device, a global positioning system (GPS) receiver, an event data recorder (EDR), a flight data recorder (FDR), an automotive infotainment device, an sailing electronic device (e.g., a sailing navigation device or a gyro compass), avionics, security devices, vehicular head units, industrial or home robots, automatic teller's machines (ATMs), point of sales (POS) devices, or Internet of Things devices (e.g., a bulb, various sensors, an electric or gas meter, a sprinkler, a fire alarm, a thermostat, a street light, a toaster, fitness equipment, a hot water tank, a heater, or a boiler).

According to certain embodiments of the disclosure, the electronic device can be at least one of a part of a piece of furniture or building/structure, an electronic board, an electronic signature receiving device, a projector, or various measurement devices (e.g., devices for measuring water, electricity, gas, or electromagnetic waves).

According to embodiments of the present disclosure, the electronic device is one or a combination of the above-listed devices. According to embodiments of the present disclosure, the electronic device is a flexible electronic device. The electronic device disclosed herein is not limited to the above-listed devices, and can include new electronic devices depending on the development of technology.

As used herein, the term "user" may denote a human or another device (e.g., an artificial intelligent electronic device) using the electronic device.

<FIG>, discussed below, and the various embodiments used to describe the principles of this disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of this disclosure can be implemented in any suitably arranged wireless communication system.

<FIG> illustrates an example network environment <NUM> according to various embodiments of the present disclosure. The embodiment of the network environment <NUM> shown in <FIG> is for illustration only. Other embodiments of the network environment <NUM> could be used without departing from the scope of this disclosure.

According to an embodiment of the present disclosure, an electronic device <NUM> is included in a network environment <NUM>. The electronic device <NUM> can include at least one of a bus <NUM>, a processor <NUM>, a memory <NUM>, an input/output (IO) interface <NUM>, a display <NUM>, a communication interface <NUM>, or sensors <NUM>. In some embodiments, the electronic device <NUM> can exclude at least one of the components or can add another component.

The bus <NUM> includes a circuit for connecting the components <NUM> to <NUM> with one another and transferring communications (e.g., control messages and/or data) between the components.

The processor <NUM> includes one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor <NUM> is able to perform control on at least one of the other components of the electronic device <NUM>, and/or perform an operation or data processing relating to communication.

For example, the processor <NUM> can receive a plurality of frames captured by the camera during a capture event. The processor <NUM> can identify a salient region in each of the plurality of frames. The processor <NUM> can determine a reference frame from the plurality of frames based on the identified salient regions. The processor <NUM> can fuse non-reference frames with the determined reference frame into a completed frame. The processor <NUM> can operate the display to display the completed frame.

The memory <NUM> can include a volatile and/or non-volatile memory. For example, the memory <NUM> can store commands or data related to at least one other component of the electronic device <NUM>. In various embodiments, the memory <NUM> can store spatial map data that can include mapping information of a real environment such as the interior of an office building, mall, house, amusement park, neighborhood or any other real world or virtual world mapping information utilized by an application <NUM> on the electronic device <NUM>. According to an embodiment of the present disclosure, the memory <NUM> stores software and/or a program <NUM>. The program <NUM> includes, e.g., a kernel <NUM>, middleware <NUM>, an application programming interface (API) <NUM>, and/or an application program (or "application") <NUM>. At least a portion of the kernel <NUM>, middleware <NUM>, or API <NUM> can be denoted an operating system (OS).

For example, the kernel <NUM> can control or manage system resources (e.g., the bus <NUM>, processor <NUM>, or a memory <NUM>) used to perform operations or functions implemented in other programs (e.g., the middleware <NUM>, API <NUM>, or application program <NUM>). The kernel <NUM> provides an interface that allows the middleware <NUM>, the API <NUM>, or the application <NUM> to access the individual components of the electronic device <NUM> to control or manage the system resources.

The middleware <NUM> can function as a relay to allow the API <NUM> or the application <NUM> to communicate data with the kernel <NUM>, for example. A plurality of applications <NUM> can be provided. The middleware <NUM> is able to control work requests received from the applications <NUM>, e.g., by allocating the priority of using the system resources of the electronic device <NUM> (e.g., the bus <NUM>, the processor <NUM>, or the memory <NUM>) to at least one of the plurality of applications <NUM>.

The API <NUM> is an interface allowing the application <NUM> to control functions provided from the kernel <NUM> or the middleware <NUM>. For example, the API <NUM> includes at least one interface or function (e.g., a command) for filing control, window control, image processing, or text control.

The IO interface <NUM> serve as an interface that can, e.g., transfer commands or data input from a user or other external devices to other component(s) of the electronic device <NUM>. Further, the IO interface <NUM> can output commands or data received from other component(s) of the electronic device <NUM> to the user or the other external device.

The display <NUM> includes, e.g., a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or a microelectromechanical systems (MEMS) display, or an electronic paper display. The display <NUM> is able to display, e.g., various contents (e.g., text, images, videos, icons, or symbols) to the user. The display <NUM> can include a touchscreen and may receive, e.g., a touch, gesture, proximity or hovering input using an electronic pen or a body portion of the user.

For example, the communication interface <NUM> is able to set up communication between the electronic device <NUM> and an external electronic device (e.g., a first electronic device <NUM>, a second external electronic device <NUM>, or a server <NUM>). For example, the communication interface <NUM> can be connected with the network <NUM> or <NUM> through wireless or wired communication to communicate with the external electronic device. The communication interface <NUM> can be a wired or wireless transceiver or any other component for transmitting and receiving signals, such as video feeds or video streams.

Electronic device <NUM> further includes one or more sensors <NUM> that can meter a physical quantity or detect an activation state of the electronic device <NUM> and convert metered or detected information into an electrical signal. For example, sensor <NUM> can include one or more buttons for touch input, a camera, a gesture sensor, a gyroscope or gyro sensor, an air pressure sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a depth or distance sensor, a grip sensor, a proximity sensor, a color sensor (e.g., a red green blue (RGB) sensor), a bio-physical sensor, a temperature sensor, a humidity sensor, an illumination sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an IR sensor, an ultrasound sensor, an iris sensor, a fingerprint sensor, etc. The sensor(s) <NUM> can further include a control circuit for controlling at least one of the sensors included therein. Any of these sensor(s) <NUM> can be located within the electronic device <NUM>. A camera sensor <NUM> can capture a plurality of frames for a single image to be combined by the processor <NUM>.

The first external electronic device <NUM> or the second external electronic device <NUM> can be a wearable device or an electronic device <NUM>-mountable wearable device (e.g., an optical head mounted display (HMD)). When the electronic device <NUM> is mounted in a HMD (e.g., the electronic device <NUM>), the electronic device <NUM> is able to detect the mounting in the HMD and operate in an augmented reality mode. In certain embodiments, the electronic device <NUM> is able to detect the mounting in the HMD and operate in an augmented reality mode. When the electronic device <NUM> is mounted in the electronic device <NUM> (e.g., the HMD), the electronic device <NUM> can communicate with the electronic device <NUM> through the communication interface <NUM>. The electronic device <NUM> can be directly connected with the electronic device <NUM> to communicate with the electronic device <NUM> without involving a separate network.

The wireless communication is able to use at least one of, e.g., long term evolution (LTE), long term evolution-advanced (LTE-A), 5th generation wireless system (<NUM>), mm-wave or <NUM> wireless communication, Wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), or global system for mobile communication (GSM), as a cellular communication protocol. The wired connection can include at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard <NUM> (RS-<NUM>), or plain old telephone service (POTS).

The network <NUM> includes at least one of communication networks. Examples of communication include a computer network (e.g., local area network (LAN) or wide area network (WAN)), Internet, or a telephone network.

The first and second external electronic devices <NUM> and <NUM> and server <NUM> each can be a device of the same or a different type from the electronic device <NUM>. According to certain embodiments of the present disclosure, the server <NUM> includes a group of one or more servers. According to certain embodiments of the present disclosure, all or some of operations executed on the electronic device <NUM> can be executed on another or multiple other electronic devices (e.g., the electronic devices <NUM> and <NUM> or server <NUM>). According to certain embodiments of the present disclosure, when the electronic device <NUM> should perform some function or service automatically or at a request, the electronic device <NUM>, instead of executing the function or service on its own or additionally, can request another device (e.g., electronic devices <NUM> and <NUM> or server <NUM>) to perform at least some functions associated therewith. The other electronic device (e.g., electronic devices <NUM> and <NUM> or server <NUM>) is able to execute the requested functions or additional functions and transfer a result of the execution to the electronic device <NUM>. The electronic device <NUM> can provide a requested function or service by processing the received result as it is or additionally. To that end, a cloud computing, distributed computing, or client-server computing technique can be used, for example.

Although <FIG> shows that the electronic device <NUM> includes the communication interface <NUM> to communicate with the external electronic device <NUM> or server <NUM> via the network <NUM>, the electronic device <NUM> can be independently operated without a separate communication function, according to an embodiment of the present disclosure.

The server <NUM> can support to drive the electronic device <NUM> by performing at least one of operations (or functions) implemented on the electronic device <NUM>. For example, the server <NUM> can include a processing module or processor that may support the processor <NUM> implemented in the electronic device <NUM>.

For example, the electronic device <NUM> can include an event processing module, such as within processor <NUM>. The event processing module can process at least part of information obtained from other elements (e.g., the processor <NUM>, the memory <NUM>, the input/output interface <NUM>, or the communication interface <NUM>) and can provide the same to the user in various manners. The server event processing module can include at least one of the components of the event processing module and perform (or instead perform) at least one of the operations (or functions) conducted by the event processing module.

For example, according to an embodiment of the present disclosure, the event processing module processes information related to an event, which is generated while the electronic device <NUM> is mounted in a wearable device (e.g., the electronic device <NUM>) to function as a display apparatus and to operate in the augmented reality mode, to fit the augmented reality mode and display the processed information. When the event generated while operating in the augmented reality mode is an event related to running an application, the event processing module can block the running of the application or process the application to operate as a background application or process. Additional information on the event processing module <NUM> may be provided through <FIG> described below.

The event processing module can be separate from the processor <NUM> or at least a portion of the event processing module can be included or implemented in the processor <NUM> or at least one other module, or the overall function of the event processing module can be included or implemented in the processor <NUM> shown or another processor. The event processing module can perform operations according to embodiments of the present disclosure in interoperation with at least one program <NUM> stored in the memory <NUM>.

<FIG> illustrates an example electronic device <NUM> according to various embodiments of the present disclosure. The embodiment of the electronic device <NUM> shown in <FIG> is for illustration only. Other embodiments of electronic device <NUM> could be used without departing from the scope of this disclosure. The electronic device <NUM> depicted in <FIG> can be configured the same as, or similar to, any of electronic devices <NUM>, <NUM>, or <NUM>.

<FIG> is a block diagram illustrating an example configuration of an electronic device according to an embodiment of the present disclosure. Referring to <FIG>, the electronic device <NUM> according to an embodiment of the present disclosure can be an electronic device <NUM> having at least one display. In the following description, the electronic device <NUM> can be a device primarily performing a display function or can denote a normal electronic device including at least one display. For example, the electronic device <NUM> can be an electronic device (e.g., a smartphone) having a touchscreen <NUM>.

According to certain embodiments, the electronic device <NUM> can include at least one of a touchscreen <NUM>, a controller <NUM>, a storage unit <NUM>, or a communication unit <NUM>. The touchscreen <NUM> can include a display panel <NUM> and/or a touch panel <NUM>. The controller <NUM> can include at least one of an augmented reality mode processing unit <NUM>, an event determining unit <NUM>, an event information processing unit <NUM>, or an application controller <NUM>.

For example, when the electronic device <NUM> is mounted in a wearable device <NUM>, the electronic device <NUM> can operate, e.g., as an HMD, and run an augmented reality mode. Further, according to an embodiment of the present disclosure, even when the electronic device <NUM> is not mounted in the wearable device <NUM>, the electronic device <NUM> can run the augmented reality mode according to the user's settings or run an augmented reality mode related application. In the following embodiment, although the electronic device <NUM> is set to be mounted in the wearable device <NUM> to run the augmented reality mode, embodiments of the present disclosure are not limited thereto.

According to certain embodiments, when the electronic device <NUM> operates in the augmented reality mode (e.g., the electronic device <NUM> is mounted in the wearable device <NUM> to operate in a head mounted theater (HMT) mode), two screens corresponding to the user's eyes (left and right eye) can be displayed through the display panel <NUM>.

According to certain embodiments, when the electronic device <NUM> is operated in the augmented reality mode, the controller <NUM> can control the processing of information related to an event generated while operating in the augmented reality mode to fit in the augmented reality mode and display the processed information. According to certain embodiments, when the event generated while operating in the augmented reality mode is an event related to running an application, the controller <NUM> can block the running of the application or process the application to operate as a background process or application.

More specifically, according to an embodiment of the present disclosure, the controller <NUM> can include at least one of an augmented reality mode processing unit <NUM>, an event determining unit <NUM>, an event information processing unit <NUM>, or an application controller <NUM> to perform functions according to various embodiments of the present disclosure. An embodiment of the present disclosure can be implemented to perform various operations or functions as described below using at least one component of the electronic device <NUM> (e.g., the touchscreen <NUM>, controller <NUM>, or storage unit <NUM>).

According to certain embodiments, when the electronic device <NUM> is mounted in the wearable device <NUM> or the augmented reality mode is run according to the user's setting or as an augmented reality mode-related application runs, the augmented reality mode processing unit <NUM> can process various functions related to the operation of the augmented reality mode. The augmented reality mode processing unit <NUM> can load at least one augmented reality program <NUM> stored in the storage unit <NUM> to perform various functions.

The event detecting unit <NUM> determines or detects that an event is generated while operated in the augmented reality mode by the augmented reality mode processing unit <NUM>. Further, the event detecting unit <NUM> can determine whether there is information to be displayed on the display screen in relation with an event generated while operating in the augmented reality mode. Further, the event detecting unit <NUM> can determine that an application is to be run in relation with an event generated while operating in the augmented reality mode. Various embodiments of an application related to the type of event are described below.

The event information processing unit <NUM> can process the event-related information to be displayed on the display screen to fit the augmented reality mode when there is information to be displayed in relation with an event occurring while operating in the augmented reality mode depending on the result of determination by the event detecting unit <NUM>. Various methods for processing the event-related information can apply. For example, when a three-dimensional (3D) image is implemented in the augmented reality mode, the electronic device <NUM> converts the event-related information to fit the 3D image. For example, event-related information being displayed in two dimensions (2D) can be converted into left and right eye information corresponding to the 3D image, and the converted information can then be synthesized and displayed on the display screen of the augmented reality mode being currently run.

When it is determined by the event detecting unit <NUM> that there is an application to be run in relation with the event occurring while operating in the augmented reality mode, the application controller <NUM> performs control to block the running of the application related to the event. According to certain embodiments, when it is determined by the event detecting unit <NUM> that there is an application to be run in relation with the event occurring while operating in the augmented reality mode, the application controller <NUM> can perform control so that the application is run in the background so as not to influence the running or screen display of the application corresponding to the augmented reality mode when the event-related application runs.

The storage unit <NUM> can store an augmented reality program <NUM>. The augmented reality program <NUM> can be an application related to the augmented reality mode operation of the electronic device <NUM>. The storage unit <NUM> can also store the event-related information <NUM>. The event detecting unit <NUM> can reference the event-related information <NUM> stored in the storage unit <NUM> in order to determine whether the occurring event is to be displayed on the screen or to identify information on the application to be run in relation with the occurring event.

The wearable device <NUM> can be an electronic device including at least one function of the electronic device <NUM> shown in <FIG>, and the wearable device <NUM> can be a wearable stand to which the electronic device <NUM> can be mounted. In case the wearable device <NUM> is an electronic device, when the electronic device <NUM> is mounted on the wearable device <NUM>, various functions can be provided through the communication unit <NUM> of the electronic device <NUM>. For example, when the electronic device <NUM> is mounted on the wearable device <NUM>, the electronic device <NUM> can detect whether to be mounted on the wearable device <NUM> for communication with the wearable device <NUM> and can determine whether to operate in the augmented reality mode (or an HMT mode).

According to certain embodiments, upon failure to automatically determine whether the electronic device <NUM> is mounted when the communication unit <NUM> is mounted on the wearable device <NUM>, the user can apply various embodiments of the present disclosure by running the augmented reality program <NUM> or selecting the augmented reality mode (or, the HMT mode). According to an embodiment of the present disclosure, when the wearable device <NUM> functions with or as part the electronic device <NUM>, the wearable device can be implemented to automatically determine whether the electronic device <NUM> is mounted on the wearable device <NUM> and enable the running mode of the electronic device <NUM> to automatically switch to the augmented reality mode (or the HMT mode).

At least some functions of the controller <NUM> shown in <FIG> can be included in the event processing module <NUM> or processor <NUM> of the electronic device <NUM> shown in <FIG>. The touchscreen <NUM> or display panel <NUM> shown in <FIG> can correspond to the display <NUM> of <FIG>. The storage unit <NUM> shown in <FIG> can correspond to the memory <NUM> of <FIG>.

Although in <FIG> the touchscreen <NUM> includes the display panel <NUM> and the touch panel <NUM>, according to an embodiment of the present disclosure, the display panel <NUM> or the touch panel <NUM> may also be provided as a separate panel rather than being combined in a single touchscreen <NUM>. Further, according to an embodiment of the present disclosure, the electronic device <NUM> can include the display panel <NUM>, but exclude the touch panel <NUM>.

According to certain embodiments, the electronic device <NUM> can be denoted as a first device (or a first electronic device), and the wearable device <NUM> may be denoted as a second device (or a second electronic device) for ease of description.

According to certain embodiments, an electronic device can comprise a display unit displaying on a screen corresponding to an augmented reality mode and a controller performing control that detects an interrupt according to an occurrence of at least one event, that varies event-related information related to the event in a form corresponding to the augmented reality mode, and that displays the varied event-related information on the display screen that corresponds to the augmented reality mode.

According to certain embodiments, the event can include any one or more selected from among a call reception event, a message reception event, an alarm notification, a scheduler notification, a wireless fidelity (Wi-Fi) connection, a WiFi disconnection, a low battery notification, a data permission or use restriction notification, a no application response notification, or an abnormal application termination notification.

According to certain embodiments, the electronic device further comprises a storage unit configured for storing the event-related information when the event is not an event to be displayed in the augmented reality mode, wherein the controller can perform control to display the event-related information stored in the storage unit when the electronic device switches from the virtual reality mode into an augmented reality mode or a see-through (non-augmented reality) mode. According to certain embodiments, the electronic device can further comprise a storage unit that stores information regarding at least one event to be displayed in the augmented reality mode. According to certain embodiments, the event can include an instant message reception notification event. According to certain embodiments, when the event is an event related to running at least one application, the controller can perform control that blocks running of the application according to occurrence of the event. According to certain embodiments, the controller can perform control to run the blocked application when a screen mode of the electronic device switches from a virtual reality mode into an augmented reality mode or a see-through (non-augmented reality) mode. According to certain embodiments, when the event is an event related to running at least one application, the controller can perform control that enables the application, according to the occurrence of the event, to be run on a background of a screen of the augmented reality mode. According to certain embodiments, when the electronic device is connected with a wearable device, the controller can perform control to run the augmented reality mode. According to certain embodiments, the controller can enable the event-related information to be arranged and processed to be displayed in a three dimensional (3D) space of the augmented reality mode screen being displayed on a current display screen. According to certain embodiments, the electronic device <NUM> can include additional sensors such as one or more red, green, blue (RGB) cameras, dynamic vision sensor (DVS) cameras, <NUM> degree cameras, or a combination thereof.

<FIG> is a block diagram illustrating a program module according to an embodiment of the present disclosure. The embodiment illustrated in <FIG> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure. In the example shown in <FIG>, although an augmented reality (AR) system is depicted, at least some embodiments of the present disclosure apply equally to a virtual reality (VR) and the augmented reality (AR). Referring to <FIG>, the program module can include a system operating system (e.g., an OS) <NUM>, a framework <NUM>, and an application(s) <NUM>.

The system operating system <NUM> can include at least one system resource manager or at least one device driver. The system resource manager can perform, for example, control, allocation, or recovery of the system resources. The system resource manager may include at least one manager, such as a process manager, a memory manager, or a file system manager. The device driver may include at least one driver, such as, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a Wi-Fi driver, an audio driver, or an inter-process communication (IPC) driver.

According to certain embodiments, the framework <NUM> (e.g., middleware) can provide, for example, functions commonly required by an application or provide the application with various functions through an application programming interface (API) to allow the application to efficiently use limited system resources inside the electronic device.

The AR framework included in the framework <NUM> can control functions related to augmented reality mode operations on the electronic device. For example, when running an augmented reality mode operation, the AR framework <NUM> can control at least one AR application <NUM>, which is related to augmented reality, among applications <NUM> so as to provide the augmented reality mode on the electronic device.

The application(s) <NUM> can include a plurality of applications and can include at least one AR application <NUM> running in the augmented-reality mode and at least one normal application <NUM> running in a non-augmented-reality mode.

The application(s) <NUM> can further include an AR control application <NUM>. An operation of the at least one AR application <NUM> and/or at least one normal application <NUM> can be controlled by the AR control application <NUM>.

When at least one event occurs while the electronic device operates in the augmented reality mode, the system operating system <NUM> can notify the framework <NUM>, for example the AR framework, of an occurrence of an event.

The framework <NUM> can then control the running of the normal application <NUM> so that event-related information can be displayed on the screen for the event occurring in the non-augmented reality mode, but not in the augmented reality mode. When there is an application to be run in relation with the event occurring in the normal mode, the framework <NUM> can perform or provide control to run at least one normal application <NUM>.

According to certain embodiments, when an event occurs while operating in the augmented reality mode, the framework <NUM>, for example the AR framework, can block the operation of at least one normal application <NUM> to display the information related to the occurring event. The framework <NUM> can provide the event occurring, while operating in the augmented reality mode, to the AR control application <NUM>.

The AR control application <NUM> can process the information related to the event occurring while operating in the augmented reality mode to fit within the operation of the augmented reality mode. For example, a 2D, planar event-related information can be processed into 3D information.

The AR control application <NUM> can control at least one AR application <NUM> currently running and can perform control to synthesize the processed event-related information for display on the screen being run by the AR application <NUM> and display the result of the event related information thereon.

According to certain embodiments, when an event occurs while operating in the augmented reality mode, the framework <NUM> can perform control to block the running of at least one normal application <NUM> related to the occurring event.

According to certain embodiments, when an event occurs while operating in the augmented reality mode, the framework <NUM> can perform control to temporarily block the running of at least one normal application <NUM> related to the occurring event, and then when the augmented reality mode terminates, the framework <NUM> can perform control to run the blocked normal application <NUM>.

According to certain embodiments, when an event occurs while operating in the augmented reality mode, the framework <NUM> can control the running of at least one normal application <NUM> related to the occurring event so that the at least one normal application <NUM> related to the event operates in the background so as not to influence the screen used by the AR application <NUM> currently running.

Embodiments described in connection with <FIG> are examples for implementing an embodiment of the present disclosure in the form of a program, and embodiments of the present disclosure are not limited thereto and rather can be implemented in other various forms. Further, while the embodiment described in connection with <FIG> references AR, it can be applied to other scenarios such as mixed reality, or virtual reality etc. Collectively the various reality scenarios can be referenced herein as extended reality (XR).

Various examples of aspects of a user interface (UI) for XR scenarios. It should be noted that aspects of XR UIs disclosed herein are merely examples of XR UIs and are not intended to be limiting.

There are different types of display elements that can be used in XR scenarios. For example, displayed elements are either tied directly to the real world or tied loosely to the XR display space. In-world elements are elements that move in relation to the real or virtual environment itself (i.e., move in relation to the environment itself). Depending on the object, in-world elements may not necessarily move in relation to the user's head when wearing a head mounted display (HMD).

Heads up display (HUD) elements are elements wherein users can make small head movements to gaze or look directly at various application (app) elements without moving the HUD elements container or UI panel in the display view. HUD elements can be a status bar or UI by which information is visually displayed to the user as part of the display.

<FIG>, <FIG> illustrate examples of a head mounted display (HMD) for use in augmented reality, mixed reality, or virtual reality according to an embodiment of this disclosure. The embodiments of the HMDs shown in <FIG> are for illustration only and other configurations could be used without departing from the scope of the present disclosure.

The HMD can generate an augmented reality environment in which a real-world environment is rendered with augmented information. The HMD can be monocular or binocular and can be an opaque, transparent, semi-transparent or reflective device. For example, the HMD can be a monocular electronic device <NUM> having a transparent screen <NUM>. A user is able to see through the screen <NUM> as well as able to see images rendered, projected or displayed on the screen <NUM>. The images may be projected onto the screen <NUM>, generated or rendered by the screen <NUM> or reflected on the screen <NUM>. In certain embodiments, the HMD is a monocular electronic device <NUM> having an opaque or non-see through display <NUM>. The non-see through display <NUM> can be a liquid crystal display (LCD), a Light emitting diode (LED), active-matrix organic light emitting diode (AMOLED), or the like. The non-see through display <NUM> can be configured to render images for viewing by the user. In certain embodiments, the HMD can be a binocular electronic device <NUM> having a transparent screen <NUM>. The transparent screen <NUM> can be a single contiguous screen, such as adapted to be viewed by, or traverse across, both eyes of the user. The transparent screen <NUM> also can be two transparent screens in when one screen is disposed corresponding to a respective eye of the user. The user is able to see through the screen <NUM> as well as able to see images rendered, projected or displayed on the screen <NUM>. The images may be projected onto the screen <NUM>, generated or rendered by the screen <NUM> or reflected on the screen <NUM>. In certain embodiments, the HMD is a binocular electronic device <NUM> having an opaque or non-see through display <NUM>. The HMD can include a camera or camera input configured to capture real-world information and display, via the non-see through display <NUM>, real-world information. The non-see through display <NUM> can be an LCD, LED, AMOLED, or the like. The non-see through display <NUM> can be configured to render images for viewing by the user. The real-world information captured by the camera can be rendered as a video image on the display with augmented information.

Embodiments of the present disclosure utilize focal point tracking, such as disclosed in <CIT>, naming Masak Suzuki, Sergio Perdices-Gonzalez, and Pranav Mistry as inventors, and entitled "SYSTEM AND METHOD FOR TRACKING A FOCAL POINT FOR A HEAD MOUNTED DEVICE".

Embodiments of the present disclosure relate to focal point tracking or gaze tracking, or both, for use in augmented reality (AR) systems. In the recent years, a significant increase in interest in Augmented Reality (AR) glasses has been experienced. Because a see-through AR HMD is compact and lightweight, demand is expected to continue and increase. A significant issue with the current technology is that AR HMDs may draw an extremely blurred image in some cases. The blurred image may cause nausea, dizziness or generally ill feelings in the user of the HMD. Additionally, gaze too long at an electronic display can cause vision concerns.

Embodiments of the present disclosure provide a system and method that can determine a coordinate, such as an X,Y position, to which a user is looking, and in conjunction with a focal point of the eye of the user, the real depth of the object (distance from user eye to the object). Embodiments of the present disclosure enable the processing circuitry to distinguish what object is the user looking at when there are multiple objects in the same line of sight at different distances.

<FIG> illustrates an electronic device having focal point tracking circuitry and gaze tracking circuitry according to embodiments of the present disclosure. The embodiment of the electronic device shown in <FIG> is for explanation only and other illustrations could be used without departing from the scope of the present disclosure. The electronic device <NUM> can be the same as, or similar to, one of the electronic devices <NUM>, <NUM>, <NUM>, <NUM> or <NUM>.

In certain embodiments, the electronic device <NUM> includes modules, or circuitry, for focal point tracking and circuitry for gaze tracking. A switch <NUM>, which can be an adaptive switch, is configured to switch between the focal point tracking circuitry and gaze tracking circuitry.

The gaze tracking circuitry can include a light emitting diode (LED) circuit <NUM> and a first camera <NUM>, which is coupled to a gaze tracking processor <NUM> through a first switch <NUM>. The first switch <NUM> can be configured to be responsive to commands from the switch <NUM> or included in the switch <NUM>.

The focal point tracking circuitry can include an LED circuit <NUM> and a second camera <NUM>, which is coupled to a focal point tracking processor <NUM> through a first switch <NUM>. The first switch <NUM> can be configured to be responsive to commands from the switch <NUM> or included in the switch <NUM>.

The gaze tracking processor <NUM> and the focal point tracking processor <NUM> each are able to drive images on a display unit. For example, the gaze tracking processor <NUM> can be used, at least in part, to operate one or more displays in response to gaze tracking functions outlined herein. Alternatively, the focal point tracking processor <NUM> can be used, at least in part, to vary focal points of objects rendered on the display based on a determined focal point of the eye of the user.

<FIG> illustrates an electronic device having circuitry for focal point tracking and gaze tracking according to embodiments of the present disclosure. The embodiment of the electronic device shown in <FIG> is for explanation only and other illustrations could be used without departing from the scope of the present disclosure. The electronic device <NUM> can be the same as, or similar to, one of the electronic devices <NUM>, <NUM>, <NUM>, <NUM> or <NUM>.

In certain embodiments, the electronic device <NUM> includes a processor system <NUM> configured to perform both focal point tracking and gaze tracking. For example, the processor system <NUM> can include one or more processors to perform focal point tracking <NUM> and gaze tracking <NUM>. In certain embodiments, processor system <NUM> includes a single processor to perform focal point tracking <NUM> and gaze tracking <NUM>. In certain embodiments, the processor system <NUM> is configured to control the switch <NUM> to switch between the focal point tracking circuitry and gaze tracking circuitry.

The gaze tracking circuitry can include a LED circuit <NUM> and a first camera <NUM>, which is coupled to a gaze tracking processor <NUM> through a first switch <NUM>. The first switch <NUM> can be configured to be responsive to commands from the switch <NUM> or included in the switch <NUM>.

The processor system <NUM> is able to drive images on a display unit. For example, the processor system <NUM> can operate one or more displays in response to gaze tracking functions outlined herein. Alternatively, the processor system <NUM> can vary focal points of objects rendered on the display based on a determined focal point of the eye of the user.

<FIG> illustrates a focal point estimation system according to embodiments of the present disclosure. The embodiment of the focal point estimation system (FPES) <NUM> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure. The FPES <NUM> can be used with an HMD. The FPES <NUM> can be included as a component of an HMD. The FPES <NUM> can be removably coupled to the HMD. The HMD can be configured as one of the HMD's <NUM>, <NUM>, <NUM> or <NUM>.

The FPES <NUM> is positioned in relation to the user's eye <NUM>. The FPES <NUM> is able to emit a light towards the user's eye <NUM> and detect a reflected light from the user's eye <NUM>. That is, the FPES <NUM> can emit a light towards the user's eye <NUM> while the user, being a wearer of the HMD, is looking at an object placed in front of the HMD, wherein the reflected light is reflected by an anterior surface of the eye of the user and inner lens of the eye of the user. In one example, the FPES includes OLED display <NUM>, a lens assembly <NUM>, a reflective interface <NUM>, a processing system <NUM>, an infrared light source <NUM>, and an infrared camera <NUM>.

The OLED display <NUM> displays images, such as images to augment a reality view of the user when wearing the HMD. In certain embodiments, the OLED display <NUM> can be integrated into the FPES <NUM>, such as when the FPES <NUM> is part of, or comprises, the HMD. In certain embodiments, the OLED display <NUM> is part of the HMD and interacts with the FPES <NUM>. For example, the FPES <NUM> can drive command signals to control an image rendered by the OLED display <NUM> and the OLED display <NUM> can render images based on estimations performed by the FPES <NUM>. That is, the OLE display is configured to present an image of the object at a 2nd distance based on the focal point of the lens unit to create a perception for the user that the image is placed at the 1st distance.

The lens assembly <NUM> can include a single lens or a plurality of lenses. The lens assembly <NUM> can be a part of the OLED display <NUM> or coupled to the OLED display <NUM>. For example, when the FPES is included with the HMD, the lens assembly <NUM> can be disposed in proximity to or over a display surface of the OLED display <NUM>. In certain embodiments, when the FPES <NUM> is coupled to the HMD, the lens assembly <NUM> may be included as part of the HMD, may be included as part of the OLED display <NUM>, or may be configured to removably couple to the HMD or OLED display <NUM>, such that the lens assembly <NUM> is disposed in proximity to or over a display surface of the OLED display <NUM>. The lens assembly <NUM> is configured to adjust to vary a focal point of the lens assembly <NUM>.

The reflective interface <NUM> is a transparent, semi-transparent, or opaque material. The reflective interface <NUM> includes a reflective surface. For example, the reflective interface <NUM> can be a transparent mirror. In certain embodiments, the reflective interface <NUM> is integrated as part of the FPES <NUM>. In certain embodiments, the reflective interface <NUM> is part of the HMD <NUM> to which the FPES <NUM> is coupled. The reflective interface <NUM> is configured to reflect light from the infrared light source <NUM> towards the eye <NUM> and reflect light from the eye <NUM> towards the infrared camera <NUM>.

The infrared light source <NUM> emits an infrared light towards the reflective interface <NUM>. The infrared light is emitted at a radiant intensity sufficiently low to be safe for the eye <NUM>. A standard IEC-<NUM> describes a safe level of infrared's intensity, for example, a radiant intensity of <NUM> watt per steradian (W/sr). In certain embodiments, the infrared light source <NUM> emits the infrared light at or below <NUM> W/sr. In certain embodiments, the infrared light source <NUM> emits the infrared light at or below <NUM> W/sr. In certain embodiments, the infrared light source <NUM> includes a switcher configured to enable the infrared light source <NUM> to emit the infrared light at or below <NUM> W/sr. It is noted that a first reflection point PS1 corresponds to light reflected from an anterior (or outer) surface of the cornea; a second reflection point PS2 corresponds to light reflected from an posterior (or inner) surface of the cornea; a third reflection point PS3 corresponds to light reflected from an anterior (or outer) surface of the lens; and fourth reflection point PS2 corresponds to light reflected from an posterior (or inner) surface of the lens. In certain embodiments, the illuminance of the light source <NUM> is <NUM>% while the illuminance of PS1 is <NUM>% with a magnification of <NUM>; the illuminance of PS2 is <NUM>% with a magnification of <NUM>, the illuminance of PS3 is <NUM>% with a magnification of <NUM>; and the illuminance of PS4 is <NUM>% with a magnification of <NUM>. It is further noted that the image at PS4 is inverted (i.e., flipped) from the image at the light source.

The infrared camera <NUM> is configured to capture an image of the eye <NUM>. The infrared camera <NUM> can detect light reflected from the eye <NUM>. In certain embodiments, the infrared camera <NUM> can be, or can include, a light sensor configured to detect a reflected light. The reflected light can be reflected by an anterior surface of the eye of the user and inner lens of the eye of the user. The reflected light can be further reflected by reflective interface <NUM>. The infrared camera <NUM> transmits signals corresponding to the detected or captures images to the processing system <NUM>.

The processing system <NUM> can include one or more processors configured to control operations of the FPES <NUM>, the HMD, or a combination thereof. The processing system <NUM> can include a memory to store instructions for operating the processing system <NUM>, operating the FPES <NUM>, operating the HMD, or a combination thereof. The memory also can store data captured by the FPES <NUM>, such as via the infrared camera <NUM>. The processing system <NUM> receives signals from the infrared camera <NUM> corresponding to the images of the eye <NUM>. The processing system <NUM> analyzes a reflected light pattern on the eye <NUM> and estimates a corresponding focal point for the eye <NUM>. For example, the processing system <NUM> can estimate a focal point of the eye <NUM> using a Purkinje-Sanson Image estimation method. The processing system <NUM> analyzes a reflection pattern in the image of the eye <NUM> as captured by infrared camera <NUM>. The processing system <NUM> identifies reflection points corresponding to the reflection point PS1, the second reflection point PS2, the third reflection point PS3 and the fourth, inverted reflection point PS4. It is again noted that the first reflection point PS1 corresponds to light reflected from an anterior (or outer) surface of the cornea; the second reflection point PS2 corresponds to light reflected from an posterior (or inner) surface of the cornea; the third reflection point PS3 corresponds to light reflected from an anterior (or outer) surface of the lens; and the fourth reflection point PS2 corresponds to light reflected from an posterior (or inner) surface of the lens. The processing system <NUM> calculates, measures, or otherwise determines a distance between PS1 and PS3. The measurements vary based on eye rotation and an anterior surface curvature of the eye of the user and the inner lens of the eye of the user while the user is looking at the object. Based on the distance determination, the processing system <NUM> adjusts a focal point of the lens assembly <NUM>. As such, the processing system <NUM> is configured to adjust the focal point of the lens unit in response to a 1st distance of the object from the HMD, wherein the 1st distance is determined based on position of the reflected light.

In response to the adjustment of the focal point of the lens assembly <NUM>, the OLED display <NUM> is able to present an image of an object at a second distance based on the focal point of the lens assembly <NUM>. Therefore, the FPES <NUM> is able to create a perception for the user that the image is at a first distance, which is different than the second distance.

In certain embodiments, the FPES <NUM> is configured to adjust for different interpupillary distance (IPD). The FPES <NUM> can automatically, i.e., without user intervention, mechanically move an illumination LED using camera feedback loop to initially adjust and track user changes in HMD positioning during use and to adjust for different eye positions for different users. The FPES <NUM> can automatically, i.e., without user intervention, mechanically move the half-mirror (reflective interface <NUM>), using camera feedback loop to initially adjust and track user changes in HMD positioning during use and to adjust for different eye positions for different users. In certain embodiments, the FPES <NUM> includes a Multi-LED: array (1D) or matrix (2D). The FPES <NUM> can perform feedback loop testing of different LED's and camera tracking of PS3 <NUM>. The FPES <NUM> can perform feedback loop optimization for once it is locked at the optimal LED can skip the loop or just track neighbor LEDs. In certain embodiments, the FPES <NUM> is configured to initially calibrate to a respective user and, thereafter, adjust a focal point based on user eye movements and adjustments in focus. In certain embodiments, the FPES <NUM> is configured to recalibrate in response to HMD movements of the user. Additionally or alternatively, in certain embodiments, the FPES <NUM> can be configured to adjust for different interpupillary distances (IPDs) based on manual user input/adjustment. In certain embodiments, feedback for the manual user input/adjustment can be provided to the user visually, haptically, and/or acoustically, etc..

In certain embodiments, the FPES <NUM> is configured to communicate with an external device. For example, the FPES <NUM> can be connected to a smartphone <NUM> to provide focal point estimation or gaze point tracking in conjunction with functionality in the smartphone <NUM>. The FPES <NUM> can provide notifications, graphs or data, or health monitoring information <NUM> to the user via the smartphone <NUM>.

<FIG> illustrates an electronic device having a display on-off auto changer (DAC) according to embodiments of the present disclosure. The embodiment of the extended reality electronic device shown in <FIG> is for explanation only and other illustrations could be used without departing from the scope of the present disclosure. In certain extended reality electronic device <NUM> is configured to create an augmented reality experience for the user. In certain extended reality electronic device <NUM> is configured to create a virtual reality experience for the user. The electronic device <NUM> can be the same as, or similar to, one of the electronic devices <NUM>, <NUM>, <NUM>, <NUM> or <NUM> and can include circuitry as described with respect to electronic device <NUM>, electronic device <NUM> or FPES <NUM>.

The electronic device <NUM> is positioned in relation to the user's eye <NUM>. The electronic device <NUM> is able to determine a user's gaze and focal point. The electronic device <NUM> includes a processor <NUM> configured to perform gaze tracking, focal point tracking, or a combination thereof. The electronic device <NUM> also includes an adaptive switch <NUM> configured to operate to control command signals from the processor <NUM> to the display <NUM>. In certain embodiments, the adaptive switch <NUM> includes processing circuitry configured to operate in response to determinations regarding the gaze of the user. In certain embodiments, the adaptive switch <NUM> operates in response to command signals received from the processor <NUM>.

In certain embodiments, the display <NUM> is included in the electronic device. For example, the display <NUM> can be OLED display <NUM>. In certain embodiments, the display <NUM> is a display on an external device. For example, the display <NUM> can be multiple displays on different external devices, as shown in <FIG>, such that the adaptive switch <NUM> engages different displays on different devices in response to command signals received from the processor <NUM>.

The adaptive switch <NUM> operates the display <NUM> as a function of the gaze of the user. When the electronic device <NUM> determines that the user is looking at a near object, such as at the display <NUM>, the adaptive switch <NUM> turns the display <NUM> ON. When the electronic device <NUM> determines that the user is looking at a far object, that is, that the gaze of the user is beyond a predetermined threshold distance, the adaptive switch <NUM> turns the display <NUM> OFF. Persons of ordinary skill in the art will recognize that, in the ON state, the display <NUM> is able to render images or video as a function of the process being executed, while, in the OFF state, the display <NUM> is functionally off such that no power is flowing to the display screen or the display <NUM> receives no images, data, or video for rendering on the display screen.

<FIG> illustrates a display auto-ON / auto-OFF process according to embodiments of the present disclosure. <FIG> does not limit the scope of this disclosure to any particular embodiments. While the auto-ON / auto-OFF process <NUM> depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. For ease of explanation, the auto-ON / auto-OFF process <NUM> is described with respect to processing system <NUM> of the electronic device <NUM> of <FIG>. However, the auto-ON / auto-OFF process <NUM> can be used with any other suitable system.

In block <NUM>, a determination is made as to whether the user is looking at a near object, such as at the display <NUM>. For example, the processor <NUM> can determine whether a gaze, or focal point, or both, of the eye of the user is at a distance corresponding to a position of the display <NUM>.

When the gaze or focal point of the eye of the user corresponds to a near proximity, such as at the distance corresponding to a position of the display <NUM>, the display <NUM> is turned ON in block <NUM>. For example, in response to the processor <NUM> determining that the gaze of the eye of the user corresponds to the distance corresponding to the position of the display <NUM>, the processor <NUM> causes the adaptive switch to turn ON the display <NUM>. In certain embodiments, the adaptive switch <NUM> operates automatically, namely without user intervention, in response to the determination that the gaze of the eye of the user corresponds to the distance corresponding to the position of the display <NUM>. For example, the adaptive switch <NUM> can detect a flag or condition state in the processor <NUM> when the processor <NUM> determines the gaze of the eye of the user.

When the gaze or focal point of the eye of the user does not correspond to a near proximity, such as at the distance exceeding a preset or predetermined distance, which could correspond to a distance of the display <NUM>, the display <NUM> is turned OFF in block <NUM>. For example, in response to the processor <NUM> determining that the gaze of the eye of the user exceeds the preset or predetermined distance, the processor <NUM> causes the adaptive switch to turn OFF the display <NUM>. In certain embodiments, the adaptive switch <NUM> operates automatically, namely without user intervention, in response to the determination that the gaze of the eye of the user exceeds the preset or predetermined distance.

<FIG> illustrates an example processing system according to embodiments of the present disclosure. The embodiment of the processing system <NUM> shown in <FIG> is for illustration only and other embodiments could be used without departing from the present disclosure. The processing system <NUM> can be configured the same as processor <NUM> or configured to perform the functions of processor <NUM>. For example, the processor <NUM> can be implemented as a multiple processor system, such as certain embodiments of the processing system <NUM>.

In certain embodiments, the processing system <NUM> includes one or more processors configured as a focal point estimator <NUM>. In certain embodiments, the processing system <NUM> includes a single processor configured to control the operations of a FPES as well as perform the functions of the focal point estimator <NUM>. The focal point estimator <NUM> is configured to estimate a focal point according to one or more of the methods outlined herein above.

In certain embodiments, the processing system <NUM> includes a focal point database <NUM>. The focal point database <NUM> can be stored in the memory of the processing system <NUM>. The processing system <NUM> can store captured images in focal point database <NUM> as well as pupil and spot detection information. The focal point database <NUM> can include a plurality of eye rotation, anterior surface and inner lens curvature data of the user looking at a particular object placed at different positions during a calibration mode. Additionally, the processing system <NUM> can retrieve focal point estimation information from the focal point database <NUM>.

The processing system <NUM> is configured to receive data signals from infrared camera <NUM>. The processing system <NUM> stores the received data signals in the focal point database <NUM> for recall and use in configuring the FPES <NUM>. In certain embodiments, the camera system operates responsive to a user input <NUM>, such as via a touch input, button, or other control input for measurement (i.e., a measurement button). For example, when a user pushes the measurement button, the FPES changes a display focal point to a user's focal point. The processing system <NUM> drives the operation of, such as by issuing command signals, the light source <NUM> and the lens assembly <NUM>.

The processing system <NUM> also includes a display ON/OFF selector <NUM>. The display ON/OFF selector <NUM> drives the operation of, such as by issuing command signals or operating adaptive switch, the OLED display <NUM>. In the example, the display ON/OFF selector <NUM> drives the operation of OLED display <NUM>; however, embodiments herein apply equally to the display ON/OFF selector <NUM> driving the operation of the display <NUM>. For example, the display ON/OFF selector <NUM> can determine a whether a gaze, or focal point, or both, of the eye of the user is at a distance corresponding to a position of the OLED display <NUM>. When the gaze or focal point of the eye of the user corresponds to a near proximity, such as at the distance corresponding to a position of the OLED display <NUM>, the display ON/OFF selector <NUM> turns ON the OLED display <NUM>. When the gaze or focal point of the eye of the user does not correspond to a near proximity, the display ON/OFF selector <NUM> turns OFF the OLED display <NUM> (or display <NUM>).

In certain embodiments, the display ON/OFF selector <NUM> is configured as a semi-automatic display changer. That is, when a user pushes a button, such as user input <NUM>, the display ON/OFF selector <NUM> turns OFF or ON the OLED display <NUM>.

<FIG>, and <FIG> illustrate operation of external devices according to embodiments of the present disclosure. The examples of the control of the external devices shown in <FIG>, and <FIG> are for illustration only and other examples could be used without departing from the scope of the present disclosure.

In the example shown in <FIG>, a user <NUM> is looking at a first display 815a. A HMD, such as FPES <NUM> or electronic device <NUM>, is configured to perform gaze tracking and focal point tracking as described herein. To perform gaze tracking, a determination is made regarding X and Y positions (e.g., horizontal and vertical) of where the user is looking, while to perform focal point tracking, a determination is made regarding the Z position (e.g., depth) of where the user is looking. As such, in some embodiments, the "gaze <NUM>" can further depend on the focal point tracking (depth information) as well. The HMD determines that the gaze <NUM> of the user <NUM> is directed at the first display 815a and not at the second display 815b nor at the third display 815c. In response to determining that the gaze <NUM> of the user <NUM> is upon the first display 815a, the HMD automatically operates the first display 815a and turns OFF the second display 815b and turns OFF the third display 815c. That is, without user intervention, the HMD turns ON the first display 815a and turns OFF the second display 815b and turns OFF the third display 815c.

In the example shown in <FIG>, a user <NUM> is looking at a first personal computer (PC) <NUM>. The user <NUM> may desire to input text into the first PC <NUM>. A HMD, such as FPES <NUM> or electronic device <NUM>, is configured to perform gaze tracking and focal point tracking as described herein. Again, it is noted that, to perform gaze tracking, a determination is made regarding X and Y positions (e.g., horizontal and vertical) of where the user is looking, while to perform focal point tracking, a determination is made regarding the Z position (e.g., depth) of where the user is looking. As such, in some embodiments, the "gaze <NUM>" can further depend on the focal point tracking (depth information) as well. The HMD determines that the gaze <NUM> of the user <NUM> is directed at the first PC <NUM> and not at the second PC <NUM> nor at the third PC <NUM>. In response to determining that the gaze <NUM> of the user <NUM> is upon the first PC <NUM>, the HMD automatically couples the input device <NUM> to the PC <NUM> such that text input via the input device <NUM>, including adjacent mouse <NUM>, is received, displayed, or executed by the first PC <NUM>. That is, without user intervention, the HMD turns ON the first PC <NUM>, couples the first PC <NUM> to input device <NUM> and mouse <NUM>, turns OFF the second PC <NUM> and turns OFF the third PC <NUM>.

In the example shown in <FIG>, a user <NUM> is looking at a PC <NUM>. The user <NUM> may desire to operate the PC <NUM> or view information on PC <NUM>. A HMD, such as FPES <NUM> or electronic device <NUM>, is configured to perform gaze tracking and focal point tracking as described herein. Again, it is noted that, to perform gaze tracking, a determination is made regarding X and Y positions (e.g., horizontal and vertical) of where the user is looking, while to perform focal point tracking, a determination is made regarding the Z position (e.g., depth) of where the user is looking. As such, in some embodiments, the "gaze <NUM>" can further depend on the focal point tracking (depth information) as well. The HMD determines that the gaze <NUM> of the user <NUM> is directed at the PC <NUM> and not at the phone <NUM> nor at the television <NUM>. In response to determining that the gaze <NUM> of the user <NUM> is upon the first PC <NUM>, the HMD automatically operates the first PC <NUM>. That is, without user intervention, the HMD turns ON the first PC <NUM>. Additionally, the HMD does not engage, turns OFF, or places in a sleep mode, the phone <NUM> and the television <NUM>.

Accordingly, the HMD, such as via an adaptive switch and display ON/OFF selector, is able to vary which external device to operate and which display on the respective external devices to operate. The HMD is able to, in response to determining a gaze <NUM> of the user <NUM>, or focal point of the eye of the user <NUM>, select one of a number of external devices to operate. Additionally, the HMD is able to, in response to determining a gaze <NUM> of the user <NUM>, or focal point of the eye of the user <NUM>, turn ON a display of the selected external device and turn OFF the displays of the respective non-selected external devices.

In certain embodiments, the HMD, such as FPES <NUM> or electronic devices <NUM>, <NUM> or <NUM>, is configured to provide an eye health awareness notification to the user. Continuous tracking/viewing of objects at extended periods of too-close focus can negatively affect the health of the eye. Ophthalmologists, optometrists and other eye professionals note a seeming link between myopia, also called nearsightedness, and "near work"-visual activities that take place at a distance of about <NUM> centimeters (<NUM> inches) from the eye-such as reading a book. Staring at a computer screen qualifies as well, though monitors usually are around <NUM> centimeters (<NUM> inches) away. For example, when a user keeps looking at a near object, or near focal or gaze point, for a long time, an HMD in accordance with the present disclosure can be configured to provide an alert to the user. The HMD can generate the alert via OLED display <NUM>, display <NUM>, or via smartphone <NUM>.

The HMD, such as FPES <NUM>, or electronic devices <NUM>, <NUM> or <NUM>, is configured to generate a human centered Simultaneous Localization and Mapping (SLAM). SLAM is a technology constructing or updating a map of an unknown environment with a camera. The HMD is able to capture private texture, such as faces of people, letters, and symbols. The HMD camera <NUM> (or cameras <NUM> or <NUM>) creates SLAM using the focal points of the eye of the user. As such, the HMD camera <NUM> generates the SLAM images by observing a user, not an outside environment. This method also allows the HMD to focus only on the surfaces to which the user pays attention. This method also allows the HMD to reduce significantly the amount of data, mapping and computation.

<FIG> illustrates a Simultaneous Localization and Mapping (SLAM) system. The embodiment of the SLAM system <NUM> shown in <FIG> is for illustration only and other embodiments could be used without departing from the present disclosure. The SLAM system <NUM> can be configured the same as processor <NUM> or configured to perform the functions of processor <NUM>.

SLAM is the computational problem of constructing or updating a map of an unknown environment while simultaneously keeping track of an agent's location within it. SLAM algorithms are tailored to the available resources, hence not aimed at perfection, but at operational compliance. Published approaches of SLAM are employed in self-driving cars, unmanned aerial vehicles, autonomous underwater vehicles, planetary rovers, newer domestic robots and even inside the human body.

The SLAM system <NUM> includes an output focal plane, which is a depth where the image of the display is projected. As such, a SLAM system <NUM> tracking system is coupled with a display output system. In certain embodiments, the display includes an optical system that adjusts a lens (mechanically, acoustically or electrically). In certain embodiments, the SLAM system <NUM> display image may be digitally placed by processing digitally the content to be in focus at a certain distance(s).

The SLAM system <NUM> can choose from multiple options regarding where the content is placed in space. For example:.

○ User driven: The SLAM system <NUM> can project information at the depth that the user is focusing. The SLAM system <NUM> can guide the user to focus in a certain plane by adjusting the focal plane close (but not exactly) to the user focus and keep moving it until the user reaches target depth.

○ Environment driven: In an augmented reality scenario, the SLAM system <NUM> needs to overlay digital content at the depth of the targeted object or plane. The SLAM system <NUM> can use the user eye focal point to recognize or calculate where the real object is placed in depth relative to the user.

In certain embodiments, the SLAM system <NUM> includes a SLAM generator <NUM>. The SLAM generator <NUM> can be implemented as a single processor or a multiple processor system. In certain embodiments, the SLAM generator <NUM> is implemented as part of another processor system. The focal point estimator <NUM> is configured to estimate a focal point according to one or more of the methods outlined herein above.

In certain embodiments, the SLAM system <NUM> includes a focal point database <NUM>. The focal point database <NUM> can be stored in the memory of the processing system <NUM>. The SLAM system <NUM> can store captured images in focal point database <NUM> as well as pupil and spot detection information. The focal point database <NUM> can include a plurality of eye rotation, anterior surface and inner lens curvature data of the user looking at a particular object placed at different positions during a calibration mode. Additionally, the SLAM system <NUM> can retrieve focal point estimation information from the focal point database <NUM>.

The SLAM system <NUM> is configured to receive data signals from infrared camera <NUM>. The SLAM system <NUM> stores the received data signals in the focal point database <NUM> for recall and use in configuring the FPES <NUM>. In certain embodiments, the camera system operates responsive to a user input <NUM>, such as via a touch input, button, or other control input for measurement (i.e., a measurement button). For example, when a user pushes the measurement button, the FPES changes a display focal point to a user's focal point. The SLAM system <NUM> drives the operation of, such as by issuing command signals, the light source <NUM> and the lens assembly <NUM>.

The SLAM system <NUM> also includes a display ON/OFF selector <NUM>. The display ON/OFF selector <NUM> drives the operation of, such as by issuing command signals or operating adaptive switch, the OLED display <NUM>. In the example, the display ON/OFF selector <NUM> drives the operation of OLED display <NUM>; however, embodiments herein apply equally to the display ON/OFF selector <NUM> driving the operation of the display <NUM>. For example, the display ON/OFF selector <NUM> can determine a whether a gaze, or focal point, or both, of the eye of the user is at a distance corresponding to a position of the OLED display <NUM>. When the gaze or focal point of the eye of the user corresponds to a near proximity, such as at the distance corresponding to a position of the OLED display <NUM>, the display ON/OFF selector <NUM> turns ON the OLED display <NUM>. When the gaze or focal point of the eye of the user does not correspond to a near proximity, the display ON/OFF selector <NUM> turns OFF the OLED display <NUM> (or display <NUM>).

The SLAM generator <NUM> uses a data from focal point estimator <NUM>. In certain embodiments, when a user pushes a button, such as via user input <NUM>, the SLAM generator <NUM> generates a SLAM.

In certain embodiments, the HMD, such as FPES <NUM>, or electronic devices <NUM>, <NUM> or <NUM>, is configured to generate depth information. Certain embodiments provide an application for a mono-eye person's aid. A mono-eye person is not able to perceive depth information. Certain embodiments of the present disclosure provide depth information on a display. Examples include an overlapped color layer for depth, sound related to depth, and the like. In certain embodiments, a depth information generator uses focal point data. When a user pushes a button, the HMD provides depth information to the user.

<FIG> illustrates a process for Simultaneous Localization and Mapping (SLAM) according to embodiments of the present disclosure. While the SLAM process <NUM> depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. For ease of explanation, the SLAM process <NUM> is described with respect to the SLAM system <NUM> of <FIG>. However, the SLAM process <NUM> can be used with any other suitable system.

In some instances, embodiments of the present disclosure provide for two depths. A point cloud from SLAM (Depth map) is generated based on the user gaze (x,y or θ,ϕ) and focal point (z or r). This provides a point in the 3D space. This process can be assisted with inertial measurement unit (IMU) data or mapped together with red-green-blue (RGB) camera, black and white (B&W) camera, or the like.

In block <NUM>, within a tracking thread <NUM>, to generate depth information, visual odometry is performed. The visual odometry is a process of determining the position and orientation of the system by analyzing the associated camera images, such as a new frame <NUM> and a last frame <NUM>. In block <NUM>, a keyframe is identified and sent to keyframe queue <NUM>. In a mapping thread <NUM>, an epipolar search, in block <NUM>, is performed on a selected keyframe in which new landmarks may be added. A map <NUM> is generated from the output of the epipolar search, in block <NUM>. The map <NUM> is optimized and updated in block <NUM>. Additionally, in the recognition thread <NUM>, a determination is made as to whether a duplicate object is found in block <NUM>. If a duplicate object is found, a determination is made as to whether the duplicate is a new object in block <NUM>. If the object is new, the object is added to an object database <NUM>. If the object is not new, an instance is added to the database <NUM> regarding the "not new" object. Thereafter, the database <NUM> is also used to generate and update the map <NUM>.

<FIG> illustrates a depth information system according to embodiments of the present disclosure. The embodiment of the depth information system <NUM> shown in <FIG> is for illustration only and other embodiments could be used without departing from the present disclosure. The depth information system <NUM> can be configured the same as processor <NUM> or configured to perform the functions of processor <NUM>.

In certain embodiments, the depth information system <NUM> includes a depth generator <NUM>. The depth generator <NUM> can be implemented as a single processor or a multiple processor system. In certain embodiments, the depth generator <NUM> is implemented as part of another processor system. The depth information system <NUM> also includes focal point estimator <NUM> configured to estimate a focal point according to one or more of the methods outlined herein above.

In certain embodiments, the depth information system <NUM> includes a focal point database <NUM>. The focal point database <NUM> can be stored in the memory of the processing system <NUM>. The depth information system <NUM> can store captured images in focal point database <NUM> as well as pupil and spot detection information. The focal point database <NUM> can include a plurality of eye rotation, anterior surface and inner lens curvature data of the user looking at a particular object placed at different positions during a calibration mode. Additionally, the depth information system <NUM> can retrieve focal point estimation information from the focal point database <NUM>.

The depth information system <NUM> is configured to receive data signals from infrared camera <NUM>. The SLAM system <NUM> stores the received data signals in the focal point database <NUM> for recall and use in configuring the FPES <NUM>. In certain embodiments, the camera system operates responsive to a user input <NUM>, such as via a touch input, button, or other control input for measurement (i.e., a measurement button). For example, when a user pushes the measurement button, the FPES changes a display focal point to a user's focal point. The depth information system <NUM> drives the operation of, such as by issuing command signals, the light source <NUM> and the lens assembly <NUM>.

The depth generator <NUM> generates depth information for display on OLED display <NUM> or display <NUM>. The depth information can be generated as described herein above with respect to the SLAM in <FIG> and <FIG>. In certain embodiments, the depth generator <NUM> uses a data from focal point estimator <NUM>. In certain embodiments, when a user pushes a button, such as via user input <NUM>, the depth generator <NUM> generates the depth information. The user inputs can include one or more of: a voice command, an action, a gesture, or blink pattern to trigger the depth aid. For example, the camera <NUM> can detect a blinking of the eye of the user. In response, the depth generator <NUM> generates depth information for display by the OLED display <NUM> or display <NUM>. For example, processor <NUM> or processor <NUM> can cause the depth generator <NUM> to generate the depth information. In certain embodiments, the information captured by the camera <NUM> is detected by the depth generator <NUM>, which, in response thereto, generates the depth information. Thereafter, the depth information is displayed by the OLED display <NUM> or display <NUM>.

<FIG> illustrates a camera focus example according to the present disclosure. The camera focus example shown in <FIG> is for illustration only. Other examples could be used without departing from the scope of the present disclosure.

In the example shown in <FIG>, a user <NUM> is trying to capture an image of an object <NUM> using camera <NUM>. The object is on a first side of a glass <NUM> and the camera is on the other side of the glass <NUM>. The user <NUM> attempts to set a focal point of camera <NUM> through the glass <NUM>. In some situations, the camera <NUM> will focus on glass <NUM> while the user <NUM> wants to take a picture of the object <NUM>.

<FIG> illustrates a camera assisted focus system according to embodiments of the present disclosure. The embodiment of the camera assisted focus system (CAFS) <NUM> shown in <FIG> is for illustration only. Other embodiments could be used without departing from the scope of the present disclosure. The CAFS <NUM> can be configured the same as, or similar to, the FPES700 of <FIG>.

The CAFS <NUM> is positioned in relation to the user's eye <NUM>. The CAFS <NUM> is able to emit a light towards the user's eye <NUM> and detect a reflected light from the user's eye <NUM>. That is, the CAFS <NUM> can emit a light towards the user's eye <NUM> while the user, being a wearer of the HMD, is looking at an object placed in front of the HMD, wherein the reflected light is reflected by anterior surface of eye of the user and inner lens of the eye of the user. The CAFS <NUM> includes OLED display <NUM>, a lens assembly <NUM>, a reflective interface <NUM>, a processing system <NUM>, an infrared light source <NUM>, and an infrared camera <NUM>. The CAFS <NUM> also includes, or is coupled to, camera <NUM>.

The OLED display <NUM> displays images, such as images to augment a reality view of the user when wearing the HMD. In certain embodiments, the OLED display <NUM> can be integrated into the CAFS <NUM>, such as when the CAPS <NUM> is part of, or comprises, the HMD. In certain embodiments, the OLED display <NUM> is part of the HMD and interacts with the CAFS <NUM>. For example, the CAFS <NUM> can drive command signals to control an image rendered by the OLED display <NUM> and the OLED display <NUM> can render images based on estimations performed by the CAPS <NUM>. That is, the OLE display is configured to present an image of the object at a 2nd distance based on the focal point of the lens unit to create a perception for the user that the image is placed at the 1st distance.

The lens assembly <NUM> can include a single lens or a plurality of lenses. The lens assembly <NUM> can be a part of the OLED display <NUM> or coupled to the OLED display <NUM>. For example, when the CAPS <NUM> is included with the HMD, the lens assembly <NUM> can be disposed in proximity to or over a display surface of the OLED display <NUM>. In certain embodiments, when the CAPS <NUM> is coupled to the HMD, the lens assembly <NUM> may be included as part of the HMD, may be included as part of the OLED display <NUM>, or may be configured to removably couple to the HMD or OLED display <NUM>, such that the lens assembly <NUM> is disposed in proximity to or over a display surface of the OLED display <NUM>. The lens assembly <NUM> is configured to adjust to vary a focal point of the lens assembly <NUM>.

The reflective interface <NUM> is a transparent, semi-transparent, or opaque material. The reflective interface <NUM> includes a reflective surface. For example, the reflective interface <NUM> can be a transparent mirror. In certain embodiments, the reflective interface <NUM> is integrated as part of the CAFS <NUM>. In certain embodiments, the reflective interface <NUM> is part of the HMD to which the CAPS <NUM> is coupled. The reflective interface <NUM> is configured to reflect light from the infrared light source <NUM> towards the eye <NUM> and reflect light from the eye <NUM> towards the infrared camera <NUM>.

The infrared light source <NUM> emits an infrared light towards the reflective interface <NUM>. The infrared light is emitted at a radiant intensity sufficiently low to be safe for the eye <NUM>. A standard IEC-<NUM> describes a safe level of infrared's intensity. For example, a radiant intensity of <NUM> watt per steradian (W/sr). In certain embodiments, the infrared light source <NUM> emits the infrared light at or below <NUM> W/sr. In certain embodiments, the infrared light source <NUM> emits the infrared light at or below <NUM> W/sr. In certain embodiments, the infrared light source <NUM> includes a switcher configured to enable the infrared light source <NUM> to emit the infrared light at or below <NUM> W/sr. It is again noted that the first reflection point PS1 corresponds to light reflected from an anterior (or outer) surface of the cornea; the second reflection point PS2 corresponds to light reflected from an posterior (or inner) surface of the cornea; the third reflection point PS3 corresponds to light reflected from an anterior (or outer) surface of the lens; and the fourth reflection point PS2 corresponds to light reflected from an posterior (or inner) surface of the lens. In certain embodiments, the illuminance of the light source <NUM> is <NUM>% while the illuminance of PS1 is <NUM>% with a magnification of <NUM>; the illuminance of PS2 is <NUM>% with a magnification of <NUM>, the illuminance of PS3 is <NUM>% with a magnification of <NUM>; and the illuminance of PS4 is <NUM>% with a magnification of <NUM>. It is further noted that the image at PS4 is inverted (i.e., flipped) from the image at the light source.

The infrared camera <NUM> is configured to capture an image of the eye <NUM>. The infrared camera <NUM> can detect light reflected from the eye <NUM>. In certain embodiments, the infrared camera <NUM> can be, or can include, a light sensor configured to detect a reflected light. The reflected light can be reflected by anterior surface of eye of the user and inner lens of the eye of the user. The reflected light can be further reflected by reflective interface <NUM>. The infrared camera <NUM> transmits signals corresponding to the detected or captures images to the processing system <NUM>.

The processing system <NUM> can include one or more processors configured to control operations of the CAFS <NUM>, the HMD, or a combination thereof. The processing system <NUM> can include a memory to store instructions for operating the processing system <NUM>, operating the CAFS <NUM>, operating the HMD, or a combination thereof. The memory also can store data captured by the CAFS <NUM>, such as via the infrared camera <NUM>. The processing system <NUM> receives signals from the infrared camera <NUM> corresponding to the images of the eye <NUM>. The processing system <NUM> analyzes a reflected light pattern on the eye <NUM> and estimates a corresponding focal point for the eye <NUM>. For example, the processing system <NUM> can estimate a focal point of the eye <NUM> using a Purkinje-Sanson Image estimation method. The processing system <NUM> analyzes a reflection pattern in the image of the eye <NUM> as captured by infrared camera <NUM>. The processing system <NUM> identifies reflection points corresponding to the reflection point PS1, the second reflection point PS2, the third reflection point PS3 and the fourth, inverted reflection point PS4. It is again noted that the first reflection point PS1 corresponds to light reflected from an anterior (or outer) surface of the cornea; the second reflection point PS2 corresponds to light reflected from an posterior (or inner) surface of the cornea; the third reflection point PS3 corresponds to light reflected from an anterior (or outer) surface of the lens; and the fourth reflection point PS2 corresponds to light reflected from an posterior (or inner) surface of the lens. The processing system <NUM> calculates, measures, or otherwise determines a distance between PS1 and PS3. The measurements vary based on eye rotation and anterior surface curvature of the eye of the user and the inner lens of the eye of the user while the user is looking at the object. Based on the distance determination, the processing system <NUM> adjusts a focal point of the lens assembly <NUM>. As such, the processing system <NUM> is configured to adjust the focal point of the lens unit in response to a 1st distance of the object from the HMD, wherein the 1st distance is determined based on position of the reflected light.

In response to the adjustment of the focal point of the lens assembly <NUM>, the OLED display <NUM> is able to present an image of an object at a second distance based on the focal point of the lens assembly <NUM>. Therefore, the CAFS <NUM> is able to create a perception for the user that the image is at a first distance, which is different than the second distance.

In certain embodiments, the CAFS <NUM> is configured to adjust for different interpupillary distance (IPD). The CAPS <NUM> can automatically, i.e., without user intervention, mechanically move an illumination LED using camera feedback loop to initially adjust and track user changes in HMD positioning during use and to adjust for different eye positions for different users. The CAFS <NUM> can automatically, i.e., without user intervention, mechanically move the half-mirror (reflective interface <NUM>), using camera feedback loop to initially adjust and track user changes in HMD positioning during use and to adjust for different eye positions for different users. In certain embodiments, the CAFS <NUM> includes a Multi-LED: array (1D) or matrix (2D). The CAFS <NUM> can perform feedback loop testing of different LED's and camera tracking of PS3 <NUM>. The CAPS <NUM> can perform feedback loop optimization for once it is locked at the optimal LED can skip the loop or just track neighbor LEDs. In certain embodiments, the CAPS <NUM> is configured to initially calibrate to a respective user and, thereafter, adjust a focal point based on user eye movements and adjustments in focus. In certain embodiments, the CAFS <NUM> is configured to recalibrate in response to HMD movements of the user.

In certain embodiments, the CAFS <NUM> is configured to communicate with an external device. For example, the CAFS <NUM> can be connected to a smartphone <NUM> to provide focal point estimation or gaze point tracking in conjunction with functionality in the smartphone <NUM>. The CAFS <NUM> can provide notifications, graphs or data, or health monitoring information <NUM> to the user via the smartphone <NUM>.

In certain embodiments of the present disclosure, a focal point of the CAPS <NUM> is assisted by the camera <NUM>. The camera <NUM> can be a visible ray camera. The camera <NUM> is able to emit a visible ray upon an object enabling the user and the CAFS <NUM> to focus at the focal point of the object.

<FIG> illustrates a semi-automatic user lens power estimator according to the present disclosure. The semi-automatic user lens power estimator <NUM> shown in <FIG> is for illustration only. Other examples could be used without departing from the scope of the present disclosure. The semi-automatic user lens power estimator <NUM> can be included in an HMD as described herein, such as being part of FPES <NUM>, electronic device <NUM>, electronic device <NUM>, of CAFS <NUM>.

In certain embodiments, the HMD includes a mobile eye-lens power estimator <NUM>. The semi-automatic user lens power estimator <NUM> can obtain not only user's lens power but also obtain a degradation over time if a user is always wearing this device. The semi-automatic user lens power estimator <NUM> includes a light analyzer <NUM>, a database <NUM>, a selector <NUM> and a focal point estimator <NUM>. The light analyzer <NUM> determines an amount of light emitted by the HMD and compares the determined light value to values in database <NUM>. In certain embodiments, the light analyzer <NUM> operates in response to a user input via input <NUM>.

<FIG> illustrates an adaptive selector according to embodiments of the present disclosure. adaptive selector <NUM> shown in <FIG> is for illustration only. Other examples could be used without departing from the scope of the present disclosure.

In certain embodiments, the adaptive selector <NUM> includes an eye motion database <NUM> and a reflected light database <NUM>. The adaptive selector <NUM> can compare values detected regarding the user's eye to values stored in the eye motion database <NUM> and the reflected light database <NUM> to determine if degradation of the HMD display has occurred.

Certain embodiments of the present disclosure provide enhanced driver safety. Although embodiments of the present disclosure are not limited to self-driving, recently there has been an increase interest in self-driving cars. The self-driving cars still require operator attention and operator supervision. For example, TESLA vehicles require an operator to maintain their hands on the steering wheel to try to ensure operator attention on the road. Embodiments of the present disclosure provide for tracking of the focal point of the operator. Tracking the user focal point is key to distinguish lack of attention to the road. Near focal point means the driver is focusing on the inside of the vehicle, such as when interacting with the entertainment system, checking the phone, and the like. Embodiments of the present disclosure can determine whether the driver's focus is on objects within the vehicle or on objects outside the vehicle. In certain embodiments, the electronic device is able to detect a speed of the vehicle, uses surrounding sensors to determine motion and proximity to objects and vehicle lanes, and uses a focal point estimator to determine the operator's focal point.

Claim 1:
An electronic device (<NUM>) comprising:
a display screen (<NUM>); and
at least one processor (<NUM>) coupled to the display screen, wherein the at least one processor is configured to:
determine a gaze and a focal point of an eye of the user, and
based on at least one of the gaze and the focal point of the eye of the user, control a focal point of an adjustable lens (<NUM>);
present, through the display screen (<NUM>), an image of an object based on at least one of the gaze and the focal point of the eye of the user,
wherein the at least one processor (<NUM>) is configured to control the focal point of the adjustable lens to vary a focus of the image of the object presented through the display screen (<NUM>) based on at least one of the gaze and the focal point of the eye of the user; and
create a simultaneous localization and mapping (SLAM) image using the determined focal point of the eye of the user.