Patent Description:
Device manufacturers are continually challenged to provide compelling services and applications to consumers. One area of development has been providing more immersive experiences through augmented reality and electronic displays (e.g., near-eye displays, head-worn displays, etc.). For example, in augmented reality, virtual graphics (i.e., visual representations of information) are overlaid on the physical world and presented to users on a display. These augmented reality user interfaces are then presented to users over a variety of displays, from the aforementioned head-worn display (e.g., glasses) to hand-held displays (e.g., a mobile phone or device). In some cases, the overlay of representations of information over the physical world can create potential visual miscues (e.g., focus mismatches). These visual miscues can create a poor user experience by causing, for instance, eye fatigue. Accordingly, device manufactures face significant technical challenges to reducing or eliminating the visual miscues or their impacts on the user.

<CIT> discloses a system which integrates virtual information with real world images presented on a display, such as a head-mounted display of a wearable computer. The system modifies how the virtual information is presented to alter whether the virtual information is more or less visible relative to the real world images. The modification may be made dynamically, such as in response to a change in the user's context, or user's eye focus on the display, or a user command. The virtual information may be modified in a number of ways, such as adjusting the transparency of the information, modifying the color of the virtual information, enclosing the information in borders, and changing the location of the virtual information on the display. Through these techniques, the system provides the information to the user in a way that minimizes distraction of the user's view of the real world images.

<CIT> discloses an electronic device which may include a microdisplay in which a displayed image element may be selected by gazing upon it. An eye gaze detector may determine what the user is looking at at any given instance of time if the user looks at it for sufficient time. Once an image element is identified as being selected by being gazed upon, the screen display may be altered to change the appearance of the selected image element and the unselected image elements. For example, the selected image element may be brought into focus and other image elements may be blurred.

<CIT> discloses a display which is placed in an optical pathway extending from an entrance pupil of a person's eye to a real-world scene beyond the eye. The display includes at least one <NUM>-D added-image source that is addressable to produce a light pattern corresponding to a virtual object. The source is situated to direct the light pattern toward the person's eye to superimpose the virtual object on an image of the real-world scene as perceived by the eye via the optical pathway. An active-optical element is situated between the eye and the added-image source at a location that is optically conjugate to the entrance pupil and at which the active-optical element forms an intermediate image of the light pattern from the added-image source. The active-optical element has variable optical power and is addressable to change its optical power to produce a corresponding change in perceived distance at which the intermediate image is formed, as an added image to the real-world scene, relative to the eye.

<CIT> discloses a display apparatus that displays an image based on image data and distance data concerning each pixel. The display apparatus has a detector, a display data generator. The detector detects a visual range from a viewer's eye to the viewer's point of regard. The generator produces display data based on the image data, the distance data and the visual range. The display apparatus further has an image display, projector and controller. The image display displays an image based on the display data. The projector projects the displayed image as a virtual image. The controller controls a projection distance of the virtual image so that the virtual image is projected onto the point of regard.

<CIT> discloses a method for overlaying first and second images in a common focal plane of a viewer comprises forming the first image and guiding the first and second images along an axis to a pupil of the viewer. The method further comprises adjustably diverging the first and second images at an adaptive diverging optic to bring the first image into focus at the common focal plane, and, adjustably converging the second image at an adaptive converging optic to bring the second image into focus at the common focal plane.

<CIT> discloses an optical system that has an aperture through which virtual and real-world images are viewable along a viewing axis. The optical system may be incorporated into a head-mounted display (HMD). By modulating the length of the optical path along an optical axis within the optical system, the virtual image may appear to be at different distances away from the HMD wearer. The wearable computer of the HMD may be used to control the length of the optical path. The length of the optical path may be modulated using, for example, a piezoelectric actuator or stepper motor. By determining the distance to an object with respect to the HMD using a range-finder or autofocus camera, the virtual images may be controlled to appear at various distances and locations in relation to the target object and/or HMD wearer.

A method, apparatus, and computer program product are therefore provided for determining the representations of displayed information. In an embodiment, the method, apparatus, and computer program product determines a representation of information (e.g., the visual or rendering characteristics of the representation) based on the focus distance of a user (e.g., a distance associated with where the user is looking or where the user's attention is focused in the field of view provided on the display). By way of example, the representation of information may be blurred, shadowed, colored, etc. based on whether the representation aligns with the determined focus distance. In this way, the various example embodiments of the present disclosure can reduce potential visual miscues and user eye fatigue, thereby improving the user experience associated with various displays.

Other
aspects, features, and advantages are readily apparent from the following detailed description, simply by illustrating a number of particular embodiments and implementations, including the best mode contemplated for carrying out the invention. The disclosure provides other and different embodiments, and several details can be modified in various obvious respects, all without departing from the scope of the claims. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.

The embodiments of the disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings:.

Examples of a method, apparatus, and computer program product for determining representations of displayed information based on focus distance are disclosed. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It is apparent, however, to one skilled in the art that the embodiments of the disclosure may be practiced without these specific details or with an equivalent arrangement. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the embodiments of the disclosure.

<FIG> is a perspective view of a display embodied by a pair of glasses with a see-through display, according to at least one example embodiment. As discussed previously, see-through displays and other electronic displays may be used to present a mixture of virtual information and physical real-world information. In other words, a see-through display enables a presentation of virtual data (e.g., visual representations of the data) while enabling the user to view information, objects, scenes, etc. through the display. For example, augmented reality applications may provide graphical overlays over live scenes to present representations of information to enhance or supplement the scene viewable through the display. As shown in <FIG>, a display <NUM> is embodied as a pair of head-worn glasses with a see-through display. In the illustrated example, a user is viewing a real-world object <NUM> (e.g., a sphere) through the display <NUM>. In at least one example embodiment, the display <NUM> includes two lenses representing respective subdisplays 105a and 105b to provide a binocular view of the object <NUM>. Through each subdisplay 105a and 105b, the object <NUM> is visible. In this case, additional information (e.g., representations 107a and 107b of smiley faces, also collectively referred to as representations <NUM>) is also presented as overlays on the object <NUM> to provide an augmented reality display.

Embodiments of a see-through display includes, for instance, the glasses depicted <FIG>. However, the various embodiments of the method, apparatus, and computer program product described herein also are applicable to any embodiment of see-through displays including, for instance, heads-up display (HUD) units, goggles, visors, windshields, windows, and the like. Typically, see-through displays like the display <NUM> have been implemented with a fixed point of focus for presenting the representations of the overlaid information. This can cause conflicts or visual miscues when the fixed focus of the display is set but other depth cues (e.g., vergence, shadows, etc.) cause the user to perceive the object <NUM> and the representations 107a and 107b at different depths. For example, in binocular vision, looking at the object <NUM> at a distance will automatically cause vergence and accommodation in the eye. Vergence, for instance, is the movement of both eyes to move the object <NUM> of attention into the fovea of the retinas. Accommodation, for instance, is the process by which the eye changes optical power to create a clear foveal image in focus, much like focusing a camera lens.

Accordingly, a conflict or visual miscue is the vergence-accommodation mismatch (e.g., a focus mismatch), where the eye accommodates or focuses to a different depth than the expected depth for accommodation. This can cause fatigue or discomfort in the eye. In a fixed-focus system, this problem is compounded because the eye generally will try to accommodate at a fixed focus, regardless of other depth cues.

<FIG> is a perspective view of a see-through display illustrating a visual miscue, according to at least one example embodiment. Although <FIG> illustrates the visual miscue with respect to a see-through display, similar visual miscues may exist in other types of displays including, e.g., embedded displays. In addition, depending on the rendering system employed by the see-through display, the display need not have the same components described below. For example, depending on the renderer <NUM> that is used for the display, a lightguide <NUM> may or may not be present. As shown in this example, <FIG> depicts one subdisplay 105a (e.g., one lens of the glasses of the display <NUM>) of the display <NUM> from a top view. As shown from the top view, the object distance <NUM> (e.g., a perceived distance from the user's eye <NUM> to the object <NUM>) and the representational distance <NUM> (e.g., a perceived distance from the user's eye <NUM> to the representation 107a) do not coincide when the subdisplay 105a is operating in a fixed focus mode. For example, when operating in a fixed focus mode, the subdisplay 105a may project (e.g., via a renderer <NUM>) the representation 107a through a lightguide <NUM> (e.g., the lens) to be perceived by the user at the representational distance <NUM> (e.g., typically set at infinity for a fixed focus mode). However, in this example, the representational distance <NUM> (e.g., infinity) conflicts with the perceived object distance <NUM> (e.g., a finite distance). Accordingly, because representation 107a is intended to be displayed on the object <NUM>, the difference between accommodating at an infinite distance for the representation 107a versus accommodating at a finite distance for the object <NUM> can create a visual miscue or conflict in the user's eye.

To address at least these challenges, the various embodiments of the method, the apparatus, and the computer program product described herein introduce the capability to determine how representations <NUM> are presented in the display <NUM> based on a focus distance of the user. In at least one example embodiment, the representations <NUM> are presented so that they correspond to the focus distance of the user. By way of example, the focus distance represents the distance to the point from the user's eye <NUM> on which the user is focusing or accommodating. The various embodiments of the present disclosure enable determination of how representations are to be presented in the display <NUM> based on optical techniques, non-optical techniques, or a combination thereof. By way of example, the representations are determined so that visual miscues or conflicts can be reduced or eliminated through the optical and non-optical techniques.

Optical techniques are based on determining a focus distance of a user, determining focal point settings based on the focus distance, and then configuring one or more dynamic focus optical elements with the determined focal point settings. In at least one example embodiment, the focus distance is determined based on gaze tracking information. By way of example, a gaze tracker can measure where the visual axis of each eye is pointing. The gaze tracker can then calculate an intersection point of the visual axes to determine a convergence distance of the eyes. In at least one example embodiment of the gaze tracker, the convergence distance is then used as the focus distance or focus point of each eye. It is contemplated that the other means, including non-optical means, can be used to determine the focus distance of the eye.

In addition or alternatively, the focus distance can be determined through user interface interaction by a user (e.g., selecting a specific point in the user's field of view of display with an input device to indicate the focus distance). At least one example embodiment of the present disclosure uses gaze tracking to determine the focus of the user and displays the representations <NUM> of information on each lens of a near eye display so that the representations <NUM> properly correspond to the focus distance of the user. For example, if the user is focusing on a virtual object that should be rendered at a distance of <NUM> feet, gaze tracking can be used to detect the user's focus on this distance, and the focal point settings of optics of the display are changed dynamically to result in a focus of <NUM> feet. In at least one example embodiment, as the focus distance of the user changes, the focal point settings of the dynamic focus optical components of the display can also be dynamically change to focus the optics to the distance of the object under the user's gaze or attention.

<FIG> depicts at least one example embodiment of a display <NUM> that employs dynamic focus optical components to represent a determined focus distance for representations <NUM>. More specifically, the display <NUM> includes two dynamic focus optical components 121a and 121b whose focal point settings can be dynamically changed to alter their focus. It is contemplated that the dynamic focus optical components 121a and 121b can use technologies such as fluidics, electrooptics, or any other dynamic focusing technology. For example, fluidics-based dynamic focus components may include focusing elements whose focal point settings or focus can be changed by fluidic injection or deflation of the focusing elements. Electrooptic-based dynamic focus components employ materials whose optical properties (e.g., birefringence) can be changed in response to varying of an electric field. The change in optical properties can then be used to alter the focal point settings or focus of the electrooptic-based dynamic focus components. One advantage of such dynamic focus optical components is the capability to support continuous focus over a range of distances. Another example includes a lens-system with focusing capability based on piezoelectric movement of its lenses. The examples of focusing technologies described herein are provided as examples and are not intended to limit the use of other technologies or means for achieving dynamic focus.

As shown in <FIG>, the display <NUM> is a see-through display with one dynamic focus optical component 121a positioned between a viewing location (e.g., a user's eye <NUM>) and a lightguide <NUM> through which the representations <NUM> are presented. A second dynamic focus optical component 121b is positioned between the lightguide <NUM> and the information that is being viewed through the lightguide <NUM> or see-through display. In this way, the focal point settings of for correcting the focus of the representations <NUM> can be independently controlled from the focal point settings for ensuring that information viewed through the display <NUM>. In at least one example embodiment, the information viewed through the display <NUM> may be other representations <NUM> or other objects. In this way, multiple displays <NUM> can be layered to provide more complex control of focus control of both representations <NUM> and information viewed through the display.

In an example embodiment that is not claimed, the display may be a non-see-through display that presents representations <NUM> of data without overlaying the representations <NUM> on a see-through view to the physical world or other information. In this example, the display would be opaque and employ a dynamic focus optical element in front of the display to alter the focal point settings or focus for viewing representations <NUM> on the display. The descriptions of the configuration and numbers of dynamic focus optical elements, lightguides, displays, and the like are provided as examples and are not intended to be limiting. It is contemplated that any number of the components described in the various embodiments can be combined or used in any combination.

<FIG> depicts at least one example embodiment of a display <NUM> that provides an optical technique for dynamic focus based on multiple focal planes. As shown, the display <NUM> includes three lightguides 127a-127c (e.g., exit pupil expanders (EPEs)) configured to display representations <NUM> of data at respective focal point settings or focus distances 129a-129c. In this example, each lightguide 127a-127d is associated with a fixed but different focal point setting or focal plane (e.g., close focal plane 129a, middle focal plane 129b, and infinite focal plane 129c). Depending on the desired focus distance, the renderer <NUM> can select which of the lightguides 127a-127c has a focal point setting closest to the desired focus distance. The renderer <NUM> can then present the representations <NUM> through the selected lightguide or focal plane. In at least one example embodiment, the lightguides 127a-127c are curved to enable closer focus distance matching between the representations <NUM> and data (e.g., an image source) seen through the display <NUM>. By way of example, the curved lightguides 127a-127c can be stacked cylindrically or spherically shaped EPEs for multiple virtual image distances. Although the example of <FIG> is described with respect to three lightguides 127a-127c providing three focal planes 129a-129c, in at least one example embodiment, the display <NUM> can be configured with any number of lightguides or focal planes depending on, for instance, how fine a granularity is desired for the focal point settings between each discrete focal plane.

As noted above, in at least one example embodiment, non-optical techniques can be used in addition to or, in examples not claimed, in place of the optical techniques described above to determine how the representations <NUM> of data can be presented to reduce or avoid visual miscues or conflicts. For example, a display (e.g., the display <NUM>, the display <NUM>, or the display <NUM>) can determine or generate representations <NUM> to create a sense of depth and focus based on (<NUM>) the focus distance of a user, (<NUM>) whether the representation <NUM> is a subject of interest to the user, or (<NUM>) a combination thereof. In at least one example embodiment, the display <NUM> determines the focus distance of the user and then determines the representations <NUM> to present based on the focus distance. The display <NUM> can, for instance, render representations <NUM> of data out of focus when they are not subject of the gaze or focus of the user and should be fuzzy. In at least one example embodiment, in addition to blurring or defocusing a representation, other rendering characteristics (e.g., shadow, vergence, color, etc.) can be varied based on the focus distance.

In at least one example embodiment, the various embodiments of the method, apparatus, and computer program product of the present disclosure can be enhanced with depth sensing information. For example, the display <NUM> may include a forward facing depth sensing camera or other similar technology to detect the depth and geometry of physical objects in the view of the user. In this case, the display <NUM> can detect the distance of a given physical object in focus and make sure that any representations <NUM> of data associated with the given physical object are location at the proper focal distance and that the focus is adjusted accordingly.

The processes described herein for determining representations of displayed information based on focus distance may be advantageously implemented via software, hardware, firmware or a combination of software and/or firmware and/or hardware. For example, the processes described herein, may be advantageously implemented via processor(s), Digital Signal Processing (DSP) chip, an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Arrays (FPGAs), etc. Such exemplary hardware for performing the described functions is detailed below.

<FIG> is a block diagram of an apparatus <NUM> for determining representations of displayed information based on focus distance, according to at least one example embodiment of the present disclosure. In at least one example embodiment, the apparatus <NUM> is associated with or incorporated in the display <NUM>, the display <NUM>, and/or the display <NUM> described previously with respect to <FIG>. However, it is contemplated that other devices or equipment can deploy all or a portion of the illustrated hardware and components of apparatus <NUM>. In at least one example embodiment, apparatus <NUM> is programmed (e.g., via computer program code or instructions) to determine representations of displayed information based on focus distance as described herein and includes a communication mechanism such as a bus <NUM> for passing information between other internal and external components of the apparatus <NUM>. Information (also called data) is represented as a physical expression of a measurable phenomenon, typically electric voltages, but including, in other embodiments, such phenomena as magnetic, electromagnetic, pressure, chemical, biological, molecular, atomic, subatomic and quantum interactions. For example, north and south magnetic fields, or a zero and non-zero electric voltage, represent two states (<NUM>, <NUM>) of a binary digit (bit). Other phenomena can represent digits of a higher base. A superposition of multiple simultaneous quantum states before measurement represents a quantum bit (qubit). A sequence of one or more digits constitutes digital data that is used to represent a number or code for a character. In at least one example embodiment, information called analog data is represented by a near continuum of measurable values within a particular range. Apparatus <NUM>, or a portion thereof, constitutes a means for performing one or more steps of determining representations of displayed information based on focus distance as described with respect the various embodiments of the method, apparatus, and computer program product discussed herein.

A processor (or multiple processors) <NUM> performs a set of operations on information as specified by computer program code related to determining representations of displayed information based on focus distance. The computer program code is a set of instructions or statements providing instructions for the operation of the processor and/or the computer system to perform specified functions. The code, for example, may be written in a computer programming language that is compiled into a native instruction set of the processor. The code may also be written directly using the native instruction set (e.g., machine language). The set of operations include bringing information in from the bus <NUM> and placing information on the bus <NUM>. The set of operations also typically include comparing two or more units of information, shifting positions of units of information, and combining two or more units of information, such as by addition or multiplication or logical operations like OR, exclusive OR (XOR), and AND. Each operation of the set of operations that can be performed by the processor is represented to the processor by information called instructions, such as an operation code of one or more digits. A sequence of operations to be executed by the processor <NUM>, such as a sequence of operation codes, constitute processor instructions, also called computer system instructions or, simply, computer instructions. Processors may be implemented as mechanical, electrical, magnetic, optical, chemical or quantum components, among others, alone or in combination.

Apparatus <NUM> also includes a memory <NUM> coupled to bus <NUM>. The memory <NUM>, such as a random access memory (RAM) or any other dynamic storage device, stores information including processor instructions for determining representations of displayed information based on focus distance. Dynamic memory allows information stored therein to be changed by the apparatus <NUM>. RAM allows a unit of information stored at a location called a memory address to be stored and retrieved independently of information at neighboring addresses. The memory <NUM> is also used by the processor <NUM> to store temporary values during execution of processor instructions. The apparatus <NUM> also includes a read only memory (ROM) <NUM> or any other static storage device coupled to the bus <NUM> for storing static information, including instructions, that is not changed by the apparatus <NUM>. Some memory is composed of volatile storage that loses the information stored thereon when power is lost. Also coupled to bus <NUM> is a non-volatile (persistent) storage device <NUM>, such as a magnetic disk, optical disk or flash card, for storing information, including instructions, that persists even when the apparatus <NUM> is turned off or otherwise loses power.

Information, including instructions for determining representations of displayed information based on focus distance, is provided to the bus <NUM> for use by the processor from an external input device <NUM>, such as a keyboard containing alphanumeric keys operated by a human user, or a camera/sensor <NUM>. A camera/sensor <NUM> detects conditions in its vicinity (e.g., depth information) and transforms those detections into physical expression compatible with the measurable phenomenon used to represent information in apparatus <NUM>. Examples of sensors <NUM> include, for instance, location sensors (e.g., GPS location receivers), position sensors (e.g., compass, gyroscope, accelerometer), environmental sensors (e.g., depth sensors, barometer, temperature sensor, light sensor, microphone), gaze tracking sensors, and the like.

Other external devices coupled to bus <NUM>, used primarily for interacting with humans, include a display device <NUM>, such as a near eye display, head worn display, cathode ray tube (CRT), a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, a plasma screen, or a printer for presenting text or images, and a pointing device <NUM>, such as a mouse, a trackball, cursor direction keys, or a motion sensor, for controlling a position of a small cursor image presented on the display <NUM> and issuing commands associated with graphical elements presented on the display <NUM>. In at least one example embodiment, the commands include, for instance, indicating a focus distance, a subject of interest, and the like. In at least one example embodiment, for example, in embodiments in which the apparatus <NUM> performs all functions automatically without human input, one or more of external input device <NUM>, display device <NUM> and pointing device <NUM> is omitted.

Examples of ASICs include graphics accelerator cards for generating images for display <NUM>, cryptographic boards for encrypting and decrypting messages sent over a network, speech recognition, and interfaces to special external devices, such as robotic arms and medical scanning equipment that repeatedly perform some complex sequence of operations that are more efficiently implemented in hardware.

Apparatus <NUM> also includes one or more instances of a communications interface <NUM> coupled to bus <NUM>. Communication interface <NUM> provides a one-way or two-way communication coupling to a variety of external devices that operate with their own processors, such as external displays. In general, the coupling is with a network link <NUM> that is connected to a local network <NUM> to which a variety of external devices with their own processors are connected. For example, communications interface <NUM> may be a local area network (LAN) card to provide a data communication connection to a compatible LAN, such as Ethernet. For wireless links, the communications interface <NUM> sends or receives or both sends and receives electrical, acoustic or electromagnetic signals, including infrared and optical signals, that carry information streams, such as digital data. For example, in wireless handheld devices, such as mobile telephones like cell phones, the communications interface <NUM> includes a radio band electromagnetic transmitter and receiver called a radio transceiver. In at least one example embodiment, the communications interface <NUM> enables connection to the local network <NUM>, Internet service provider <NUM>, and/or the Internet <NUM> for determining representations of displayed information based on focus distance.

The term "computer-readable medium" as used herein refers to any medium that participates in providing information to processor <NUM>, including instructions for execution. Such a medium may take many forms, including, but not limited to computer-readable storage medium (e.g., non-volatile media, volatile media), and transmission media. Non-transitory media, such as non-volatile media, include, for example, optical or magnetic disks, such as storage device <NUM>. Transmission media include, for example, twisted pair cables, coaxial cables, copper wire, fiber optic cables, and carrier waves that travel through space without wires or cables, such as acoustic waves and electromagnetic waves, including radio, optical and infrared waves. Forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, CDRW, DVD, any other optical medium, punch cards, paper tape, optical mark sheets, any other physical medium with patterns of holes or other optically recognizable indicia, a RAM, a PROM, an EPROM, a FLASH-EPROM, an EEPROM, a flash memory, any other memory chip or cartridge, a carrier wave, or any other medium from which a computer can read. The term computer-readable storage medium is used herein to refer to any computer-readable medium except transmission media.

Logic encoded in one or more tangible media includes one or both of processor instructions on a computer-readable storage media and special purpose hardware, such as ASIC <NUM>.

Network link <NUM> typically provides information communication using transmission media through one or more networks to other devices that use or process the information. For example, network link <NUM> may provide a connection through local network <NUM> to a host computer <NUM> or to equipment <NUM> operated by an Internet Service Provider (ISP). ISP equipment <NUM> in turn provides data communication services through the public, world-wide packet-switching communication network of networks referred to as the Internet <NUM>.

A computer called a server host <NUM> connected to the Internet hosts a process that provides a service in response to information received over the Internet. For example, server host <NUM> hosts a process that provides information for presentation at display <NUM>. It is contemplated that the components of apparatus <NUM> can be deployed in various configurations within other devices or components.

At least one embodiment of the present disclosure is related to the use of apparatus <NUM> for implementing some or all of the techniques described herein. According to at least one example embodiment of the disclosure, those techniques are performed by apparatus <NUM> in response to processor <NUM> executing one or more sequences of one or more processor instructions contained in memory <NUM>. Such instructions, also called computer instructions, software and program code, may be read into memory <NUM> from another computer-readable medium such as storage device <NUM> or network link <NUM>. Execution of the sequences of instructions contained in memory <NUM> causes processor <NUM> to perform one or more of the method steps described herein. In alternative embodiments, hardware, such as ASIC <NUM>, may be used in place of or in combination with software to implement the disclosure. Thus, embodiments of the disclosure are not limited to any specific combination of hardware and software, unless otherwise explicitly stated herein.

The signals transmitted over network link <NUM> and other networks through communications interface <NUM>, carry information to and from apparatus <NUM>. Apparatus <NUM> can send and receive information, including program code, through the networks <NUM>, <NUM> among others, through network link <NUM> and communications interface <NUM>. In an example using the Internet <NUM>, a server host <NUM> transmits program code for a particular application, requested by a message sent from apparatus <NUM>, through Internet <NUM>, ISP equipment <NUM>, local network <NUM> and communications interface <NUM>. The received code may be executed by processor <NUM> as it is received, or may be stored in memory <NUM> or in storage device <NUM> or any other non-volatile storage for later execution, or both. In this manner, apparatus <NUM> may obtain application program code in the form of signals on a carrier wave.

Various forms of computer readable media may be involved in carrying one or more sequence of instructions or data or both to processor <NUM> for execution. For example, instructions and data may initially be carried on a magnetic disk of a remote computer such as host <NUM>. The remote computer loads the instructions and data into its dynamic memory and sends the instructions and data over a telephone line using a modem. A communications interface <NUM> receives the instructions and data carried in the infrared signal and places information representing the instructions and data onto bus <NUM>. Bus <NUM> carries the information to memory <NUM> from which processor <NUM> retrieves and executes the instructions using some of the data sent with the instructions. The instructions and data received in memory <NUM> may optionally be stored on storage device <NUM>, either before or after execution by the processor <NUM>.

<FIG> is a block diagram of operations for determining representations of displayed information based on focus distance, according to at least one example embodiment of the present disclosure. In at least one example embodiment, the apparatus <NUM> and/or its components (e.g., processor <NUM>, display <NUM>, camera/sensors <NUM>) of <FIG> perform and/or provide means for performing any of the operations described in the process <NUM> of <FIG>. In addition or alternatively, a chip set including a processor and a memory as shown in <FIG> and/or a mobile terminal as shown in <FIG> may include means for performing any of the operations of the process <NUM>. It also is noted that the operations <NUM>-<NUM> of <FIG> are provided as examples of at least one embodiment of the present disclosure. Moreover, the ordering of the operations <NUM>-<NUM> can be changed and some of the operations <NUM>-<NUM> may be combined. For example, operation <NUM> may or may not be performed or may be combined with operation <NUM> or any of the other operations <NUM> or <NUM>.

As noted previously, potential visual miscues and conflicts (e.g., focus mismatches) and/or their impact on a user can be reduced or eliminated by optical and/or non-optical techniques. The method, apparatus, and computer program product for performing the operations of the process <NUM> relate to non-optical techniques for manipulating or determining the displayed representations <NUM> of data on the display <NUM>. In operation <NUM>, the apparatus <NUM> performs and includes means (e.g., a processor <NUM>, camera/sensors <NUM>, input device <NUM>, pointing device <NUM>, etc.) for determining a focus distance of a user. By way of example, the focus distance represents the distance to a point in a display's (e.g., displays <NUM>, <NUM>, <NUM>, and/or <NUM>) field of view this is the subject of the user's attention.

In at least one example embodiment, the point in the field of view and the focus distance are determined using gaze tracking information. Accordingly, the apparatus <NUM> may be configured with means (e.g., camera/sensors <NUM>) to determine the point of attention by tracking the gaze of the user and to determine the focus distance based on the gaze tracking information. In at least one example embodiment, the apparatus <NUM> is configured with means (e.g., processor <NUM>, memory <NUM>, camera/sensors <NUM>) to maintain a depth buffer of information, data and/or objects (e.g., both physical and virtual) present in at least one scene within a field of view of a display <NUM>. For example, the apparatus <NUM> may include means such as a forward facing depth sensing camera to create the depth buffer. The gaze tracking information can then, for instance, be matched against the depth buffer to determine the focus distance. In at least one example embodiment, the apparatus <NUM> may be configured with means (e.g., processor <NUM>, input device <NUM>, pointing device <NUM>, camera/sensors <NUM>) to determine the point in the display's field of view that is of interest to the user based on user interaction, input, and/or sensed contextual information. For example, in addition to or instead of the gaze tracking information, the apparatus <NUM> may determine what point in the field of view is selected (e.g., via input device <NUM>, pointing device <NUM>) by the user. In another example, the apparatus <NUM> may process sensed contextual information (e.g., accelerometer data, compass data, gyroscope data, etc.) to determine a direction or mode of movement for indicating a point of attention. This point can then be compared against the depth buffer to determine a focus distance.

After determining the focus distance of the user, the apparatus <NUM> performs and is configured with means (e.g., processor <NUM>) for determining a representation of data that is to be presented in the display <NUM> based on the focus distance (operation <NUM>). In at least one example embodiment, determining the representation includes, for instance, determining the visual characteristics of the representation that reduces or eliminates potential visual miscues or conflicts (e.g., focus mismatches) that may contribute to eye fatigue and/or a poor user experience when viewing the display <NUM>.

In at least one example embodiment, the apparatus <NUM> may be configured to determine the representation based on other parameters in addition or as an alternate to focus distance. For example, the apparatus <NUM> may be configured with means (e.g., processor <NUM>) to determine the representation based on a representational distance associated with the data. The representational distance is, for instance, the distance in the field of view or scene where the representation <NUM> should be presented. For example, in an example where the representation <NUM> augments a real world object viewable in the display <NUM>, the representational distance might correspond to the distance of the object. Based on this representational distance, the apparatus <NUM> may be configured with means (e.g., processor <NUM>) to apply various rendering characteristics that are a function (e.g., linear or non-linear) of the representational distance.

In at least one example embodiment, the display <NUM> is configured with means (e.g., dynamic focus optical components 121a and 121b) to optically adjust focus or focal point settings. In these embodiments, the apparatus <NUM> may be configured with means (e.g., processor <NUM>) determine the representations <NUM> based, at least in part, on the focal points settings of the dynamic focus optical components. For example, if a blurring effect is already created by the optical focal point settings, the representations need not include as much, if any, blurring effect when compared to displays <NUM> without dynamic focus optical components. In other cases, the representations <NUM> may be determined with additional effects to add or enhance, for instance, depth or focus effects on the display <NUM>.

In at least one example embodiment, the apparatus <NUM> may be configured with means (e.g., processor <NUM>) to determine a difference of the representational distance from the focus distance. In other words, the visual appearance of the representation <NUM> may depend on the how far (e.g., in either the foreground or the background) the representational distance is from the determined focus distance. In this way, the apparatus <NUM> may be configured with means (e.g., processor <NUM>) to determine a degree of at least one rendering characteristics to apply to the representation <NUM> based on the representational distance from the focus distance. For example, the rendering characteristics may include blurring, shadowing, vergence (e.g., for binocular displays), and the like. Representations <NUM> that are farther away from the focus distance may be rendered with more blur, or left/right images for a binocular display may be rendered with vergence settings appropriate for the distance. It is contemplated that any type of rendering characteristics (e.g., color, saturation, size, etc.) may be varied based on the representational distance.

After determining the representation <NUM>, the apparatus <NUM> performs and is configured with means (e.g., processor <NUM>, display <NUM>) to cause a presentation of the representation <NUM> on a display (operation <NUM>). Although various embodiments of the method, apparatus, and computer program product described herein are discussed with respect to a binocular head-worn see-through display, it is contemplated that the various embodiments are applicable to presenting representation <NUM> on any type of display where visual miscues can occur. For example, other displays include, monocular displays where only one eye may suffer from accommodation mismatches, and the like. In addition, the various embodiments may apply to displays of completely virtual information (e.g., with no live view).

As shown in operation <NUM>, the apparatus <NUM> can perform and be configured with means (e.g., processor <NUM>, camera/sensors <NUM>) to determine a change in the focus distance and then to cause an updating of the representation based on the change. In at least one example embodiment, the apparatus <NUM> may monitor the focus distance for change in substantially real-time, continuously, periodically, according to a schedule, on demand, etc. In this way, as a user changes his/her gaze or focus, the apparatus <NUM> can dynamically adjust the representations <NUM> to match with the new focus distance.

<FIG> is a block diagram of operations for determining representations of displayed information based on determining a subject of interest, according to at least one example embodiment of the present disclosure. In at least one example embodiment, the apparatus <NUM> and/or its components (e.g., processor <NUM>, display <NUM>, camera/sensors <NUM>) of <FIG> perform and/or provide means for performing any of the operations described in the process <NUM> of <FIG>. In addition or alternatively, a chip set including a processor and a memory as shown in <FIG> and/or a mobile terminal as shown in <FIG> may include means for performing any of the operations of the process <NUM>.

As shown in operation <NUM>, the apparatus <NUM> may perform and be configured with means (e.g., processor <NUM>, camera/sensors <NUM>) to determine a subject of interest within a user's field of view on a display <NUM> (e.g., what information or object presented in the display <NUM> is of interest to the user). Similar to determining the focus distance, gaze tracking or user interactions/inputs may be used to determine the subject of interest. In at least one example embodiment, the apparatus <NUM> may be configured with means (e.g., processor <NUM>, camera/sensors <NUM>) to determine the subject of interest based on whether the user is looking at a representation <NUM>. In at least one example embodiment, where multiple representations <NUM>, information, or objects are perceived at the approximately the same focus distance, the apparatus <NUM> may further determine which item in the focal plane has the user's interest (e.g., depending on the accuracy of the gaze tracking or user interaction information).

In operation <NUM>, the apparatus <NUM> may perform and be configured with means (e.g., processor <NUM>) to determine the representation based on the subject of interest. For example, when the user looks at a representation <NUM>, the representation <NUM> may have one appearance (e.g., bright and in focus). In a scenario where the user looks away from the representation <NUM> to another item in the same focal plane, the representation may have another appearance (e.g., dark and in focus). In a scenario where the user looks away from the representation <NUM> to another item in a different focal plane or distance, the representation may have yet another appearance (e.g., dark and out of focus).

<FIG> is a user's view through a display, according to at least one example embodiment of the present disclosure. In at least one example embodiment, the apparatus <NUM> may include means for determining the representations <NUM> of data to present the display <NUM> based on the focus distance of the user. As shown, a user is viewing an object <NUM> through the display <NUM>, which is a see-through binocular display comprising a subdisplay 105a corresponding to the left lens of the display <NUM> and a subdisplay 105b corresponding to the right lens of the display <NUM>. Accordingly, the apparatus may include means (e.g., processor <NUM>, display <NUM>) for generating a binocular user interface presented in the display <NUM>.

In this example, the apparatus <NUM> has determined the focus distance of the user as focus distance <NUM> corresponding to the object <NUM>. As described with respect to <FIG>, the apparatus <NUM> has presented augmenting representations 503a and 503b for each respective subdisplay 105a and 105b as overlays on the object <NUM> at the determined focus distance <NUM>. As shown, the apparatus <NUM> is also presenting representations 505a and 505b of a virtual object <NUM> located at a representational distance <NUM>, representations 511a and 511b of a virtual object <NUM> location at a representational distance <NUM>.

As illustrated in <FIG>, the difference between the representational distance <NUM> of virtual object <NUM> from the focus distance <NUM> is greater than the difference between the representational distance <NUM> of the virtual object <NUM> from the focus distance <NUM>. Accordingly, the apparatus <NUM> is configured with means (e.g., processor <NUM>) to determine the representations 505a and 505b of the virtual object <NUM> to have more blurring effect than the representations 511a and 511b of the virtual object <NUM>. In addition, because of the binocular display the representations 503a-503b, 505a-505b, and 51la-<NUM> lb are determined so that vergence of each representation pair is appropriate for the determined focus distance. In at least one example embodiment, the apparatus <NUM> may determine the blurring effect and vergence separately or in combination for the representations.

<FIG> is a block diagram of operations for determining focal point settings for dynamic focus optical components of display, according to at least one example embodiment of the present disclosure. In at least one example embodiment, the apparatus <NUM> and/or its components (e.g., processor <NUM>, display <NUM>, camera/sensors <NUM>) of <FIG> perform and/or provide means for performing any of the operations described in the process <NUM> of <FIG>. In addition or alternatively, a chip set including a processor and a memory as shown in <FIG> and/or a mobile terminal as shown in <FIG> may include means for performing any of the operations of the process <NUM>. It also is noted that the operations <NUM>-<NUM> of <FIG> are provided as examples of at least one embodiment of the present disclosure.

As noted previously, potential visual miscues and conflicts (e.g., focus mismatches) and/or their potential impacts on the user can be reduced or eliminated by optical and/or non optical techniques. The method, apparatus, and computer program product for performing the operations of the process <NUM> relate to optical techniques for determining focal point settings for dynamic focus optical components <NUM> of a display <NUM> to reduce or eliminate visual miscues or conflicts. Operation <NUM> is analogous to the focus distance determination operations described with respect to operation <NUM> of <FIG>. For example, in operation <NUM>, the apparatus <NUM> performs and includes means (e.g., a processor <NUM>, camera/sensors <NUM>, input device <NUM>, pointing device <NUM>, etc.) for determining a focus distance of a user. By way of example, the focus distance represents the distance to a point in a display's (e.g., displays <NUM>, <NUM>, <NUM>, and/or <NUM>) field of view that is the subject of the user's attention.

In at least one example embodiment, the point in the field of view and the focus distance are determined using gaze tracking information. Accordingly, the apparatus <NUM> may be configured with means (e.g., camera/sensors <NUM>) to determine the point of attention by tracking the gaze of the user and to determine the focus distance based on the gaze tracking information. In at least one example embodiment, the apparatus <NUM> is configured with means (e.g., processor <NUM>, memory <NUM>, camera/sensors <NUM>) to maintain a depth buffer of information, data and/or objects (e.g., both physical and virtual) present in at least one scene within a field of view of a display <NUM>. For example, the apparatus <NUM> may include means such as a forward facing depth sensing camera to create the depth buffer. The depth sensing camera or other similar sensors are, for instance, means for determining a depth, a geometry or a combination thereof of the representations <NUM> and the information, objects, etc. viewed through display <NUM>. For example, the depth buffer can store z-axis values for pixels or points identified in the field of view of the display <NUM>.

The depth and geometry information can be stored in the depth buffer or otherwise associated with the depth buffer. In this way, the gaze tracking information, for instance, can be matched against the depth buffer to determine the focus distance. In at least one example embodiment, the apparatus can be configured with means (e.g., processor <NUM>, memory <NUM>, storage device <NUM>) to store the depth buffer locally at the apparatus <NUM>. In addition or alternatively, the apparatus <NUM> may be configured to include means (e.g., communication interface <NUM>) to store the depth buffer and related information remotely in, for instance, the server <NUM>, host <NUM>, etc..

In at least one example embodiment, the apparatus <NUM> may be configured with means (e.g., processor <NUM>, input device <NUM>, pointing device <NUM>, camera/sensors <NUM>) to determine the point in the display's field of view that is of interest to the user based on user interaction, input, and/or sensed contextual information. For example, in addition to or instead of the gaze tracking information, the apparatus <NUM> may determine what point in the field of view is selected (e.g., via input device <NUM>, pointing device <NUM>) by the user. In another example, the apparatus <NUM> may process sensed contextual information (e.g., accelerometer data, compass data, gyroscope data, etc.) to determine a direction or mode of movement for indicating a point of attention. This point can then be compared against the depth buffer to determine a focus distance.

In operation <NUM>, the apparatus <NUM> may perform and be configured with means (e.g., processor <NUM>) for determining at least one focal point setting for one or more dynamic focus optical components <NUM> of the display <NUM> based on the focus distance. In at least one example embodiment, the parameters associated with the at least one focal point setting may depend on the type of dynamic focusing system employed by the display <NUM>. As described with respect to <FIG>, one type of dynamic focus optical component is a continuous focus system based on technologies such as fluidics or electrooptics. For fluidics-based system, the apparatus <NUM> may be configured with means (e.g., processor <NUM>) to determine parameters or focal point settings associated with fluid inflation or deflation to achieve a desired focal point. For electrooptics based system, the apparatus <NUM> may be configured to include means (e.g., processor <NUM>) for determining parameters for creating an electric field to alter the optical properties of the electrooptics system.

<FIG> describes a dynamic focusing system based on a display with multiple focal planes. For this type of system, the apparatus <NUM> may be configured to include means (e.g., processor <NUM>) determine focal point settings to indicate which of the focal planes has a focal point most similar to the determined focus distance. It is contemplated that the discussion of the above optical systems is for illustration and not intended to restrict the dynamic focusing systems to which the various embodiments of the method, apparatus, and computer program product apply.

In at least one example embodiment, the apparatus <NUM> may be configured with means (e.g., processor <NUM>, camera/sensors <NUM>) to determine the at least one focal point setting based on a focus mismatch between representations <NUM> of data presented on the display <NUM> and information view through the display <NUM>. By way of example, the apparatus <NUM> determines a depth for presenting a representation <NUM> on the display <NUM> and another depth for viewing information through the display. Based on these two depths, the apparatus <NUM> can determine whether there is a potential focus mismatch or other visual miscue and then determine the at least one focal point setting to cause a correction of the focus mismatch.

In at least one example embodiment, wherein the display <NUM> includes at least two dynamic focus optical components <NUM>, the apparatus <NUM> may be configured with means (e.g., processor <NUM>, camera/sensors <NUM>) to determine a focus mismatch by determining a deviation of the perceived depth of the representation, the information viewed through the display, or a combination thereof resulting from a first set of the focal points settings configured on one of the dynamic focus optical components <NUM>. The apparatus <NUM> can then determine another set of focal point settings for the other dynamic focus optical component <NUM> based on the deviation. For instance, the second or other set of focal point settings can be applied to the second or other dynamic focus optical elements to correct any deviations or miscues between representations <NUM> presented in the display <NUM> and information viewed through the display. Additional discussion of the process of focus correction using optical components is provided below with respect to <FIG>.

In at least one example embodiment, in addition to optical focus adjustments, the apparatus may be configured with means (e.g., processor <NUM>) for determining at least one vergence setting for the one or more dynamic focus optical components based on the focus distance. In at least one example embodiment, vergence refers to the process of rotating of the eyes around a vertical axis to provide for binocular vision. For example, objects closer to the eyes typically require greater inward rotation of the eyes, whereas for objects that are farther out towards infinity, the eyes are more parallel. Accordingly, the apparatus <NUM> may determine how to physically configure the dynamic focus optical components <NUM> to approximate the appropriate level of vergence for a given focus distance. In at least one example embodiment, the at least one vergence setting includes a tilt setting for the one or more dynamic focus optical elements. An illustration of the tilt vergence setting for binocular optical components is provided with respect to <FIG> and <FIG> below. Enabling adjustment of focus and vergence settings as described in the various embodiments enables the apparatus <NUM> to reduce or eliminate potential visual miscues that can lead to eye fatigue.

In at least one example embodiment, the apparatus <NUM> can be configured with means (e.g., processor <NUM>, camera/sensors <NUM>) to combine use of both optical and non-optical techniques for determining focus or other visual miscue correction. Accordingly, in operation <NUM>, the apparatus <NUM> may perform and be configured with means (e.g., processor <NUM>) to determine a representation <NUM> based, at least in part, on the focal points settings of the dynamic focus optical components (operation <NUM>). For example, if a blurring effect is already created by the optical focal point settings, the representations need not include as much, if any, blurring effect when compared to displays <NUM> without dynamic focus optical components. In other cases, the representations <NUM> may be determined with additional effects to add or enhance, for instance, depth or focus effects on the display <NUM> with a given focal point setting.

As shown in operation <NUM>, the apparatus <NUM> can perform and be configured with means (e.g., processor <NUM>, camera/sensors <NUM>) to determine a change in the focus distance and then to cause an updating of the at least one focal point settings for the dynamic focus optical components <NUM> based on the change. In at least one example embodiment, the apparatus <NUM> may monitor the focus distance for change in substantially real-time, continuously, periodically, according to a schedule, on demand, etc. In this way, as a user changes his/her gaze or focus, the apparatus <NUM> can dynamically adjust the focus of the optical components to match with the new focus distance.

<FIG> are perspective views of a display providing focus correction using dynamic focus optical components, according to at least one example embodiment of the present disclosure. As discussed with respect to <FIG> above, a typical near-eye see-through display <NUM> presents representations <NUM> (e.g., a virtual image) of data over a physical world view at a fixed focus. This can lead to a focus mismatch between the representations <NUM> which are typically fixed at focus distance of infinity and real objects or information viewed through display. As shown in <FIG>, in at least one example embodiment, a lens <NUM> is provided between the eye <NUM> and the lightguide <NUM>. By way of example, the single lens <NUM> has the effect of bringing the virtual image (e.g., the representation <NUM>) closer. In the example that is not claimed of a display <NUM> that is not see-through, a single lens can effectively change the focus distance of the virtual images or representations <NUM> presented on the display.

However, in the case of a see-through display <NUM>, the perceived depth of the image of the object <NUM> viewed through the display is also brought closer, therefore maintaining a potential focus mismatch. In the embodiment of <FIG>, a second lens <NUM> is positioned between the lightguide <NUM> and the object <NUM> to effectively move the perceived depth of the object <NUM> to its actual depth. Accordingly, a single lens can be effective in changing a focus distance of representations <NUM> or images on the display <NUM> when the display is opaque or non-see-through. On the other hand, a dual lens system can be effective in correcting visual miscues and focus mismatches when the display <NUM> presents real objects (e.g., object <NUM>) mixed with virtual objects (e.g., representations <NUM>).

In at least one example embodiment, when the dual lens system of <FIG> is configured with dynamic focus optical components <NUM> as lenses, the system can offer greater flexibility in mixing virtual images with information viewed through the display. As discussed with respect to operation <NUM> of <FIG>, the focal point settings of the two lenses can be adjusted to reconcile focus mismatches. For example, the focal point settings of the first lens <NUM> can be adjusted to present representations <NUM> of data at a focus distance determined by the user. Then a deviation of the perceived depth of information viewed through the display <NUM> can be used to determine the focal point settings of the second lens <NUM>. In at least one example embodiment, the focal point settings of the second lens <NUM> is determined so that it will correct any deviation of the perceived distance to move the perceived distance to the intended or actual depth of the information when viewed through the display <NUM>.

<FIG> depicts a binocular display <NUM> that includes dynamic focus optical elements 707a and 707b corresponding to the left and right eyes 709a and 709b of a user, according to at least one example embodiment. In addition to accommodation or focus conflicts, vergence can affect eye fatigue when not aligned with an appropriate focus distance. In at least one example embodiment, the dynamic focus optical elements 707a and 707b are means for optically adjusting convergence. As shown in <FIG>, when viewing an object <NUM> (particularly when the object <NUM> is close to the display <NUM>), the eyes 709a and 709b typically have to rotate inwards to bring the object <NUM> within the visual area (e.g., the foveal area) of the retinas and provide for a coherent binocular view of the object <NUM>. In the example of <FIG>, the subdisplays 713a and 713b that house the respective dynamic focus optical elements 707a and 707b include means for physically rotating in order to adjust for convergence.

<FIG> depicts a binocular display <NUM> that can adjust for convergence by changing an angle at which light is projected onto the subdisplays 717a and 717b housing respective dynamic focus elements 719a and 719b, according to at least one example embodiment. For example, instead of physically rotating the subdisplays 717a and 717b, the display <NUM> may include means for determining an angle α that represents the angle the eyes 709a and 709b should be rotated inwards to converge on the object <NUM>. The display <NUM> then may include means (e.g., rendering engines 721a and 721b) to alter the angle of light projected into the subdisplays 717a and 717b to match the angle α. In this way, the subdisplays 717a and 717b need not physically rotate as described with respect to <FIG> above.

<FIG> illustrates a chip set or chip <NUM> upon which an embodiment of the disclosure may be implemented. Chip set <NUM> is programmed to determine representations of displayed information based on focus distance as described herein and includes, for instance, the processor and memory components described with respect to <FIG> incorporated in one or more physical packages (e.g., chips). By way of example, a physical package includes an arrangement of one or more materials, components, and/or wires on a structural assembly (e.g., a baseboard) to provide one or more characteristics such as physical strength, conservation of size, and/or limitation of electrical interaction. It is contemplated that in at least one example embodiment, the chip set <NUM> can be implemented in a single chip. It is further contemplated that in at least one example embodiment, the chip set or chip <NUM> can be implemented as a single "system on a chip. " It is further contemplated that in at least one example embodiment, a separate ASIC would not be used, for example, and that all relevant functions as disclosed herein would be performed by a processor or processors. Chip set or chip <NUM>, or a portion thereof, constitutes a means for performing one or more steps of providing user interface navigation information associated with the availability of functions. Chip set or chip <NUM>, or a portion thereof, constitutes a means for performing one or more steps of determining representations of displayed information based on focus distance.

In at least one example embodiment, the chip set or chip <NUM> includes a communication mechanism such as a bus <NUM> for passing information among the components of the chip set <NUM>. Similarly, an ASIC <NUM> can be configured to performed specialized functions not easily performed by a more general purpose processor. Other specialized components to aid in performing the inventive functions described herein may include one or more field programmable gate arrays (FPGA), one or more controllers, or one or more other special-purpose computer chips.

In at least one example embodiment, the chip set or chip <NUM> includes merely one or more processors and some software and/or firmware supporting and/or relating to and/or for the one or more processors.

The processor <NUM> and accompanying components have connectivity to the memory <NUM> via the bus <NUM>. The memory <NUM> includes both dynamic memory (e.g., RAM, magnetic disk, writable optical disk, etc.) and static memory (e.g., ROM, CD-ROM, etc.) for storing executable instructions that when executed perform the inventive steps described herein to determine representations of displayed information based on focus distance. The memory <NUM> also stores the data associated with or generated by the execution of the inventive steps.

<FIG> is a diagram of exemplary components of a mobile terminal (e.g., handset) for communications, which is capable of operating in the system of <FIG>, according to at least one example embodiment. In at least one example embodiment, mobile terminal <NUM>, or a portion thereof, constitutes a means for performing one or more steps of determining representations of displayed information based on focus distance. Generally, a radio receiver is often defined in terms of front-end and back-end characteristics. The front-end of the receiver encompasses all of the Radio Frequency (RF) circuitry whereas the back-end encompasses all of the base-band processing circuitry. As used in this application, the term "circuitry" refers to both: (<NUM>) hardware-only implementations (such as implementations in only analog and/or digital circuitry), and (<NUM>) to combinations of circuitry and software (and/or firmware) (such as, if applicable to the particular context, to a combination of processor(s), including digital signal processor(s), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions). As a further example, as used in this application and if applicable to the particular context, the term "circuitry" would also cover an implementation of merely a processor (or multiple processors) and its (or their) accompanying software/or firmware. The term "circuitry" would also cover if applicable to the particular context, for example, a baseband integrated circuit or applications processor integrated circuit in a mobile phone or a similar integrated circuit in a cellular network device or other network devices.

Pertinent internal components of the telephone include a Main Control Unit (MCU) <NUM>, a Digital Signal Processor (DSP) <NUM>, and a receiver/transmitter unit including a microphone gain control unit and a speaker gain control unit. A main display unit <NUM> provides a display to the user in support of various applications and mobile terminal functions that perform or support the steps of determining representations of displayed information based on focus distance. The display <NUM> includes display circuitry configured to display at least a portion of a user interface of the mobile terminal (e.g., mobile telephone). Additionally, the display <NUM> and display circuitry are configured to facilitate user control of at least some functions of the mobile terminal. An audio function circuitry <NUM> includes a microphone <NUM> and microphone amplifier that amplifies the speech signal output from the microphone <NUM>. The amplified speech signal output from the microphone <NUM> is fed to a coder/decoder (CODEC) <NUM>.

In use, a user of mobile terminal <NUM> speaks into the microphone <NUM> and his or her voice along with any detected background noise is converted into an analog voltage. In at least one example embodiment, the processed voice signals are encoded, by units not separately shown, using a cellular transmission protocol such as enhanced data rates for global evolution (EDGE), general packet radio service (GPRS), global system for mobile communications (GSM), Internet protocol multimedia subsystem (IMS), universal mobile telecommunications system (UMTS), etc., as well as any other suitable wireless medium, e.g., microwave access (WiMAX), Long Term Evolution (LTE) networks, code division multiple access (CDMA), wideband code division multiple access (WCDMA), wireless fidelity (WiFi), satellite, and the like, or any combination thereof.

In order to prepare the signal for transmission, an up converter <NUM> combines the sine wave output from the modulator <NUM> with another sine wave generated by a synthesizer <NUM> to achieve the desired frequency of transmission. The signals may be forwarded from there to a remote telephone which may be another cellular telephone, any other mobile phone or a land-line connected to a Public Switched Telephone Network (PSTN), or other telephony networks.

Voice signals transmitted to the mobile terminal <NUM> are received via antenna <NUM> and immediately amplified by a low noise amplifier (LNA) <NUM>. A Digital to Analog Converter (DAC) <NUM> converts the signal and the resulting output is transmitted to the user through the speaker <NUM>, all under control of a Main Control Unit (MCU) <NUM> which can be implemented as a Central Processing Unit (CPU).

The MCU <NUM> receives various signals including input signals from the keyboard <NUM>. The keyboard <NUM> and/or the MCU <NUM> in combination with other user input components (e.g., the microphone <NUM>) comprise a user interface circuitry for managing user input. The MCU <NUM> runs a user interface software to facilitate user control of at least some functions of the mobile terminal <NUM> to determine representations of displayed information based on focus distance. The MCU <NUM> also delivers a display command and a switch command to the display <NUM> and to the speech output switching controller, respectively. Further, the MCU <NUM> exchanges information with the DSP <NUM> and can access an optionally incorporated SIM card <NUM> and a memory <NUM>. In addition, the MCU <NUM> executes various control functions required of the terminal. The DSP <NUM> may, depending upon the implementation, perform any of a variety of conventional digital processing functions on the voice signals. Additionally, DSP <NUM> determines the background noise level of the local environment from the signals detected by microphone <NUM> and sets the gain of microphone <NUM> to a level selected to compensate for the natural tendency of the user of the mobile terminal <NUM>.

The software module could reside in RAM memory, flash memory, registers, or any other form of writable storage medium known in the art. The memory device <NUM> may be, but not limited to, a single memory, CD, DVD, ROM, RAM, EEPROM, optical storage, magnetic disk storage, flash memory storage, or any other non-volatile storage medium capable of storing digital data.

An optionally incorporated SIM card <NUM> carries, for instance, important information, such as the cellular phone number, the carrier supplying service, subscription details, and security information. The SIM card <NUM> serves primarily to identify the mobile terminal <NUM> on a radio network. The card <NUM> also contains a memory for storing a personal telephone number registry, text messages, and user specific mobile terminal settings.

Further, one or more camera sensors <NUM> may be incorporated onto the mobile station <NUM> wherein the one or more camera sensors may be placed at one or more locations on the mobile station. Generally, the camera sensors may be utilized to capture, record, and cause to store one or more still and/or moving images (e.g., videos, movies, etc.) which also may comprise audio recordings.

Claim 1:
An apparatus comprising:
a see-through display (105a; 105b; <NUM>; 713a; 713b; 717a; 717b);
a first optical component (121a; <NUM>);
and a second optical component (121b; <NUM>); wherein each of the first optical component and the second optical component is a dynamic focus component whose focal point settings can be dynamically changed to alter their focus; and the apparatus further comprises
at least one processor and at least one memory configured to:
determine a focus distance of a user, wherein the focus distance corresponds with a distance of the object from the see-through display to a point in the user's field of view through the see-through display that is a point of attention of the user, wherein an object (<NUM>) is at the point of attention;
determine a representation (<NUM>; <NUM>; <NUM>) of data;
cause presentation of the representation on the see-through display;
adjust focus of the first optical component that is positioned between the see-through display and the user such that a focus distance of the representation (<NUM>; <NUM>; <NUM>) is reduced; and
adjust focus of the second optical component that is positioned between the see-through display and the object such that a perceived distance of the object moves to the actual distance of the object from the see-through display when viewed through the see-through display, and such that the representation is presented as overlaid on the object;
characterised in that
at least one of the optical components comprises a fluidics-based dynamic focus component; and/or
wherein at least one of the optical components comprises a lens-system with focusing capability based on piezoelectric movement of at least one lens.