Patent Description:
A head-up display (HUD) system generates a virtual image in front of a driver of a vehicle and provides a variety of information to the driver by displaying the information in the virtual image. The information provided to the driver may include, for example, navigation information and dashboard information such as a vehicle velocity, a fuel level, and an engine revolution per minute (RPM). The driver may more easily recognize the information displayed in front without turning his or her gaze during driving, and thus, driving safety may improve. In addition to the navigation information and the dashboard information, the HUD system may also provide the driver with, for example, a lane indicator, a construction indicator, an accident indicator, or a pedestrian detection indicator using augmented reality (AR), to assist with driving when a field of view is poor and/or inadequate.

<CIT> discloses an image rendering method and apparatus. The image rendering apparatus may determine which one of a two-dimensional (2D) display area that displays a 2D image and a three-dimensional (3D) display area that displays a 3D image includes a current pixel, may perform a 2D rendering operation at a position of the current pixel when the current pixel is included in the 2D display area, and may perform a 3D rendering operation at a position of the current pixel when the current pixel is included in the 3D display area.

<CIT> discloses systems and methods for predictive visual rendering. A method of displaying information to a user includes: tracking the user's eye to obtain information about the time-varying physiology of the user's eye; correlating the information about the time-varying physiology of the user's eye to information about the user's field of view; predicting where the user will look at a future time based on the correlation; and displaying to the user's eye, at the future time, information related to one or more objects in the user's field of view based on the prediction.

<CIT> discloses an apparatus and method for predicting an eye position includes a storer configured to store detected position information of an eye of user during a sample time interval, a calculator configured to calculate a weighted average value of a variation of the detected position information, and a predictor configured to generate prediction position information of the eye of user at a target time based on the weighted average value, and the calculator is configured to apply a weight to the variation of the detected position information such that the weight increases as the target time is approached during the sample time interval.

<CIT> discloses a method and apparatus for controlling stereoscopic 3D image in vehicle. The method of controlling a vehicle display apparatus includes generating a user image as information on a user sitting on a seat corresponding to the display apparatus, extracting a midpoint between two eyes of the user from the user image and monitoring a position of the midpoint, and changing an output mode or an output area of the display apparatus in correspondence with the monitored position of the midpoint.

The invention is what is claimed in the independent claims.

One or more example embodiments may address at least the above problems and/or disadvantages and other disadvantages not described above. Also, the example embodiments are not required to overcome the disadvantages described above, and an example embodiment may not overcome any of the problems described above.

According to an aspect of the disclosure, there is provided a method of controlling a head-up display (HUD), the method comprising: performing eye tracking of an eye of a user in a captured image; identifying an eye tracking status based on a result of the eye tracking; identifying a rendering mode for an HUD image to be one of a two-dimensional (2D) rendering mode and a three-dimensional (3D) rendering mode based on the eye tracking status; and rendering the HUD image in the identified rendering mode.

The identifying the eye tracking status comprises classifying the eye tracking status as one of a stable status and an unstable status based on whether eye coordinates are present in the result of the eye tracking or based on a rate of change of the eye coordinates.

The identifying the rendering mode comprises: identifying the rendering mode to be the 3D rendering mode based on the eye tracking status being classified as the stable status; and identifying the rendering mode to be the 2D rendering mode based on the eye tracking status being classified as the unstable status.

The eye tracking status is classified as the stable status based on the eye coordinates being included in the result of the eye tracking and a speed of change of the eye coordinates is less than a reference value.

The reference value may correspond to a system processing rate.

The eye tracking status is classified as the unstable status based on the eye coordinates being included in the result of the eye tracking and a speed of change of the eye coordinates is greater than a reference value, or may be based on the eye coordinates not being included in the result of the eye tracking.

The HUD image may be rendered based on a first source image for a first viewpoint and a second source image for a second viewpoint.

Based on the identified rendering mode being the 2D rendering, the rendering the HUD image mat comprise rendering the HUD image by setting the first viewpoint and the second viewpoint equally as a single viewpoint.

The rendering the HUD image may comprise: setting, based on the result of the eye tracking including current eye coordinates of both eyes and a speed of change of the current eye coordinates being greater than a reference value, the first viewpoint and the second viewpoint equally as a center viewpoint of the current eye coordinates; and setting, based on the result of the eye tracking not including the current eye coordinates, the first viewpoint and the second viewpoint equally as a center viewpoint of previous eye coordinates.

Based on the identified rendering mode being the 3D rendering mode, the rendering the HUD image may comprise rendering the HUD image by setting the first viewpoint and the second viewpoint as different viewpoints.

Based on the rendering mode being identified, the rendering mode may be switched from the 3D rendering mode to the 2D rendering mode or from the 2D rendering mode to the 3D rendering mode during a buffer time corresponding to a plurality of frames.

The HUD image may be rendered based on a first source image for a first viewpoint and a second source image for a second viewpoint, and wherein, based on the rendering mode being switched from the 2D rendering mode to the 3D rendering mode, the rendering the HUD image comprises rendering the HUD image while gradually changing the first viewpoint and the second viewpoint to a single viewpoint used in the 2D rendering mode over the buffer time.

According to another aspect of the disclosure, there is provided a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method.

According to another aspect of the disclosure, there is provided an apparatus for controlling a head-up display (HUD), the apparatus comprising: a memory configured to store one or more instructions; and a processor configured to execute the one or more instructions to: perform eye tracking of an eye of a user in a captured image, identify an eye tracking status based on a result of the eye tracking, identify a rendering mode for an HUD image to be one of a two-dimensional (2D) rendering mode and a three-dimensional (3D) rendering mode based on the eye tracking status, and render the HUD image in the identified rendering mode.

The processor is further configured to classify the eye tracking status as one of a stable status and an unstable status based on whether eye coordinates are present in the result of the eye tracking and based on a speed of position change with respect to the eye coordinates.

The processor is further configured to: identify the rendering mode to be the 3D rendering mode based on the eye tracking status being classified as the stable status; and identify the rendering mode to be the 2D rendering mode based on the eye tracking status being classified as the unstable status.

According to another aspect of the disclosure, there is provided a head-up display (HUD) device comprising: an eye tracking camera configured to capture an image including a user; a processor configured to perform eye tracking on the captured image, identify an eye tracking status based on a result of the eye tracking, identify a rendering mode for an HUD image to be one of a two-dimensional (2D) rendering mode and a three-dimensional (3D) rendering mode based on the eye tracking status, and render the HUD image in the identified rendering mode; and a display device configured to provide the HUD image to the user using augmented reality (AR).

The processor is further configured to, based on the rendering mode being identified, switch the rendering mode from the 3D rendering mode to the 2D rendering mode or from the 2D rendering mode to the 3D rendering mode during a buffer time corresponding to a plurality of frames.

The above and/or other aspects will be more apparent by describing certain example embodiments with reference to the accompanying drawings, in which:.

Hereinafter, example embodiments will be described in detail with reference to the accompanying drawings. However, various alterations and modifications may be made to the example embodiments. Here, the example embodiments are not construed as limited to the disclosure. The example embodiments should be understood to include all changes, equivalents, and replacements within the idea and the technical scope of the disclosure.

The terminology used herein is for the purpose of describing particular example embodiments only and is not to be limiting of the example embodiments.

Unless otherwise defined, all terms including technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which example embodiments belong. It will be further understood that terms, such as those defined in commonly-used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

When describing the example embodiments with reference to the accompanying drawings, like reference numerals refer to like constituent elements and a repeated description related thereto will be omitted. In the description of example embodiments, detailed description of well-known related structures or functions will be omitted when it is deemed that such description will cause ambiguous interpretation of the disclosure.

Also, in the description of the components, terms such as first, second, A, B, (a), (b) or the like may be used herein when describing components of the disclosure. These terms are used only for the purpose of discriminating one constituent element from another constituent element, and the nature, the sequences, or the orders of the constituent elements are not limited by the terms. When one constituent element is described as being "connected", "coupled", or "attached" to another constituent element, it should be understood that one constituent element can be connected or attached directly to another constituent element, and an intervening constituent element can also be "connected", "coupled", or "attached" to the constituent elements.

The same name may be used to describe an element included in the example embodiments described above and an element having a common function. Unless otherwise mentioned, the descriptions on the example embodiments may be applicable to the following example embodiments and thus, duplicated descriptions will be omitted for conciseness.

<FIG> illustrates a head-up display (HUD) device according to an example embodiment. Referring to <FIG>, an HUD device <NUM> includes an HUD control apparatus <NUM>, a display device <NUM>, an eye tracking camera <NUM>, and a translucent optical device <NUM>. The HUD device <NUM> may be mounted on a vehicle (for example, a car or an airplane) to provide an HUD image to a user (for example, a driver, a pilot, and the like). The HUD device <NUM> may provide the HUD image using augmented reality (AR). For example, contents provided through an AR HUD may include dashboard information, navigation information, a lane indicator, a construction indicator, an accident indicator, a pedestrian detection indicator, and the like. AR may be applied to an HUD, a transmissive head-mounted display (HMD), and the like. Hereinafter, an HUD will be described. However, the following description may also apply to an HMD or other display devices.

The display device <NUM> may include a light source, a display panel, a three-dimensional (3D) optical layer, and an optical element. The optical element may include a cata-dioptric system. Light corresponding to an HUD image may be provided by the display panel and the light source of the display device <NUM>, and the cata-dioptric system may reflect the light corresponding to the HUD image toward the translucent optical device <NUM>. In this case, the cata-dioptric system may refract the light corresponding to the HUD image to enlarge the HUD image. A light-emitting diode (LED) or a laser may be used as the light source.

A virtual screen <NUM> may be formed by the light corresponding to the HUD image output by the display device <NUM>. A portion of the light output by the display device <NUM> may be reflected by the translucent optical device <NUM> positioned in front of the user and viewable by the user. The translucent optical device <NUM> may be a windshield of the car or airplane, or a combiner provided separately from the windshield for the purpose of reflecting an HUD image. The user views light passing through the front of the translucent optical device <NUM>, and a portion of the light reflected by the translucent optical device <NUM> among the light radiated by the display device <NUM> at the same time. Thus, a real object and a virtual object may overlap each other and be provided to the user as AR content. For example, the real object may be an object in the surrounding environment visible through the translucent optical device <NUM>.

The display device <NUM> may display the virtual object at a position corresponding to the real object. For example, traveling direction information of the vehicle, lane information, hazard information, and the like may be displayed through the HUD as virtual objects at positions corresponding to real objects. A position on the background or the surrounding environment at which a virtual object is to be displayed may be referred to as a target position. The HUD control apparatus <NUM> may display the virtual object at the target position using a transformation relationship between a coordinate system of the eye tracking camera <NUM> and a coordinate system of the virtual screen <NUM>, 3D information on the background, and eye position information.

For example, the 3D information on the background may be obtained through a camera or a 3D sensor provided to face ahead of the vehicle. Eye positions of the user may be obtained through the eye tracking camera <NUM> provided to face the user. The eye tracking camera <NUM> may capture the user and generate a user image including the user (for example, the face of the user), and the HUD control apparatus <NUM> may obtain the eye positions by performing eye tracking on the user image. The HUD control apparatus <NUM> may generate the HUD image for displaying the virtual object at intersecting points where lines connecting the eye positions of the user and the target position intersect with the virtual screen <NUM>, and the display device <NUM> may represent the virtual object at the target position by displaying the HUD image.

The display device <NUM> may provide a 3D image through the 3D optical layer. The HUD control apparatus <NUM> may generate a first source image (for example, a left image) for a first viewpoint (for example, the left eye) and a second source image (for example, a right image) for a second viewpoint (for example, the right eye), and render the HUD image based on the eye positions tracked by the eye tracking camera <NUM>, the first source image and the second source image. Here, a viewpoint may correspond to a viewing position (for example, a position of an eye of a viewer). An operation of rendering the HUD image may include determining pixel values of the HUD image so that the first source image may be viewed at the first viewpoint and the second source image may be viewed at the second viewpoint. Hereinafter, an example of using two viewpoints for a 3D image will be described. However, embodiments of the disclosure are not limited to the description below, and as such, according to another example embodiment, two or more viewpoints may be used for a light field.

The display device <NUM> may display the HUD image generated as described above. The display device <NUM> may display the HUD image on the display panel. The HUD image may pass through the 3D optical layer and be provided to the user. In this case, different images corresponding to the first source image and the second source image may be provided to both eyes of the user. For each of the first source image and the second source image, the HUD control apparatus <NUM> may render the HUD image so that the virtual object may be displayed at the intersecting point where a line connecting each eye position of the user and the target position intersects with the virtual screen <NUM>.

<FIG> illustrates a path of light in relation to an HUD device according to an example embodiment. Referring to <FIG>, an HUD device <NUM> includes a display device <NUM> and mirrors <NUM> and <NUM>. The display device <NUM> may correspond to the display device <NUM> of <FIG>. The display device <NUM> may include a display panel and a light source and provide light corresponding to an HUD image through the display panel and the light source. For example, the light source may include a backlight unit (BLU).

Light corresponding to the HUD image output by the display device <NUM> may be reflected by the mirrors <NUM> and <NUM> and projected onto a windshield <NUM>. At least one of the mirrors <NUM> and <NUM> may correspond to an aspheric surface mirror, and adjust a path of the light corresponding to the HUD image to enlarge the HUD image. A user may view a virtual image corresponding to the HUD image on a virtual screen <NUM> through light reflected by the windshield <NUM> toward an eye box <NUM>.

In this way, the HUD system <NUM> may display information on the virtual screen <NUM> provided in front of the user through projection. In order to provide AR information through an HUD, the virtual screen <NUM> on which the HUD image is viewed may be implemented with a wide field of view (FOV). If the size of an image to be represented is not large enough or the FOV is not wide enough, it may be difficult to represent information on an object or background in front of a vehicle using AR.

The display device <NUM> and the mirrors <NUM> and <NUM> may be mounted in a dashboard of the vehicle. The display device <NUM> and the mirrors <NUM> and <NUM> may be designed to provide an FOV wide enough to implement AR. For example, the BLU of the display device <NUM> may optimize an output angle of the light output from the LED using a secondary lens array, and compensate for a shortfall in the output angle using a side reflector. In this case, the diffusion angles of a diffuser plate and a polarized diffuser plate may be maintained at small values, so that a decrease in the efficiency of the BLU may be prevented. Accordingly, it is possible to achieve compact BLU volume, wide FOV, uniformity, improved side brightness, and the like.

<FIG> illustrates a structure of a display device according to an example embodiment. Referring to <FIG>, a display device <NUM> includes a light source <NUM>, a diffuser <NUM>, a display panel <NUM>, and a 3D optical layer <NUM>. The light source <NUM> may correspond to a BLU. According to an example embodiment, the light source may include a white LED, a red/green/blue (RGB) LED, or an RGB laser. If an aspheric mirror is used as an enlarging and reflecting mirror, any of the white LED, the RGB LED, and the RGB laser may be used. However, if a holographic mirror is used, the RGB LED or the RGB laser may be used depending on recording characteristics. The diffuser <NUM> may be implemented in the form of a film, and light uniformity between the light source <NUM> and the display panel <NUM> may be provided through the diffuser <NUM>. According to an example embodiment, the diffuser <NUM> may be formed directly on the display panel <NUM>. According to another example embodiment, the diffuser <NUM> may be spaced apart from the display panel <NUM>.

The display panel <NUM> may include a liquid crystal display (LCD) panel, or a spatial light modulator (SLM) such as a digital light processor (DLP) and liquid crystal on silicon (LCoS). The 3D optical layer <NUM> may be any one of a parallax barrier, a lenticular lens, and a directional backlight unit. The display panel <NUM> may display an HUD image, and the 3D optical layer <NUM> may control a path of the light corresponding to the HUD image. For example, the 3D optical layer <NUM> may give directivity to the light corresponding to the HUD image so that images of different viewpoints may be provided to both eyes of the user.

<FIG> illustrates 3D AR according to an example embodiment. Referring to <FIG>, a virtual object <NUM> is displayed at an intersecting point where a line connecting an eye position <NUM> of the user and a target position <NUM> intersects with a virtual screen <NUM>. The eye position <NUM> may be tracked through an eye tracking camera <NUM>. In this case, a scene <NUM> may be viewed at the eye position <NUM>. The scene <NUM> includes the virtual object <NUM> and a real object <NUM>. The virtual object <NUM> may be accurately displayed at the target position <NUM> through a relationship between a coordinate system of the eye tracking camera <NUM> and a coordinate system of the virtual screen <NUM>, 3D information on the background, and information on the eye position <NUM>.

This process may be performed for each of the eyes of the user. For example, a first source image (for example, a left image) may be generated so that the virtual object <NUM> may be displayed at an intersecting point where a line connecting a first viewpoint (for example, the left eye) and the target position <NUM> intersects with the virtual screen <NUM>, and a second source image (for example, a right image) may be generated so that the virtual object <NUM> may be displayed at an intersecting point where a line connecting a second viewpoint (for example, the right eye) and the target position <NUM> intersects with the virtual screen <NUM>. Thereafter, the scene <NUM> may be implemented as a 3D AR HUD by rendering the HUD image based on the first source image and the second source image.

According to an example embodiment, the 3D HUD may represent the virtual object <NUM> at various depths in response to a change in the position of the user, and as such, the 3D HUD may more accurately display the virtual object <NUM> at the target position <NUM> than a 2D HUD. However, in order to stably provide such a 3D HUD, continuous tracking of the eye position <NUM> may be necessary and the virtual object <NUM> may be displayed on the virtual screen <NUM> based on the tracked eye position <NUM>.

In an example scenario, the eye position <NUM> may not be tracked due to an environmental factor, such as low illuminance or because the eye is covered. Moreover, an appropriate HUD image corresponding to the current eye position <NUM> may not be generated due to a systemic factor such as a system delay. In this example, a deterioration in the quality of the 3D HUD, such as crosstalk observed in the image as an image for the left eye is provided to the right eye, may occur. In this case, driving information may be stably provided by providing a 2D HUD instead of the low-quality 3D HUD. According to example embodiments, 2D rendering or 3D rendering may be selectively performed based on a current circumstance associated with eye tracking, whereby the HUD stability may be improved.

<FIG> illustrates a process of generating an HUD image according to an example embodiment. Operations <NUM> to <NUM> described below may be performed on a current frame of a user image. Referring to <FIG>, in operation <NUM>, an HUD control apparatus performs eye tracking. For example, the HUD control apparatus may generate a user image using an eye tracking camera and perform eye tracking on the user image. The HUD control apparatus may generate an eye tracking result while performing eye tracking. If eye tracking is successful, the eye tracking result may include eye coordinates. If eye tracking fails, the eye tracking result may not include eye coordinates. Instead, the eye tracking results may include information indicating that eye tracking has failed. The eye coordinates may include 3D coordinate values of each of the eyes.

In operation <NUM>, the HUD control apparatus determines an eye tracking status. For example, the HUD control apparatus may classify the eye tracking status as one of a stable status and an unstable status based on whether the eye tracking result complies with a 3D rendering condition. Here, the 3D rendering condition may be defined based on the presence of eye coordinates and a rate of change of the eye coordinates. As described above, in order to maintain the quality of a 3D HUD image, eye coordinates must be identified, and system performance for tracking a change in the eye coordinates is required.

For example, if eye coordinates are absent, or if there are eye coordinates but the eye coordinates change so severely that the rendering performance of the system is incapable of coping with the change, crosstalk is likely to be observed in a 3D HUD image. Accordingly, in a first state in which the eye tracking result includes the eye coordinates and a speed of position change with respect to the eye coordinates is less than a threshold, the eye tracking status may be classified as the stable state. In this case, the threshold may correspond to a system processing rate. In addition, in a second state in which the eye tracking result includes the eye coordinates and the speed of position change with respect to the eye coordinates is greater than the threshold, or in a third state in which the eye tracking result does not include the eye coordinates, the eye tracking status may be classified as the unstable status.

In operation <NUM>, the HUD control apparatus determines a rendering mode. Here, the determined rendering mode may be used to render an HUD image corresponding to a current frame of the user image. The rendering mode may include a 2D rendering mode and a 3D rendering mode. The HUD control apparatus may determine the rendering mode for an HUD image to be one of the 2D rendering mode and the 3D rendering mode based on the eye tracking status. For example, if the eye tracking status is classified as the stable status, the HUD control apparatus may determine the rendering mode to be the 3D rendering mode. Conversely, if the eye tracking status is classified as the unstable status, the HUD control apparatus may determine the rendering mode to be the 2D rendering mode.

The HUD control apparatus may render the HUD image so that the same HUD image is provided to both eyes of the user in the 2D rendering mode, or may render the HUD image so that different images are provided to both eyes of the user in the 3D rendering mode. For example, the HUD control apparatus may generate a first source image (for example, a left image) for a first viewpoint (for example, the left eye) and a second source image (for example, a right image) for a second viewpoint (for example, the right eye), and render the HUD image so that the first source image may be provided to the first viewpoint and the second source image may be provided to the second viewpoint. If the rendering mode is determined to be the 2D rendering mode, the HUD control apparatus may render the HUD image by setting the first viewpoint and the second viewpoint equally as a single viewpoint. Conversely, if the rendering mode is determined to be the 3D rendering mode, the HUD control apparatus may render the HUD image by setting the first viewpoint and the second viewpoint as different viewpoints.

The 2D rendering mode may include a tracking 2D rendering mode and a fixed 2D rendering mode. As described above, in the second state in which the eye tracking result includes the eye coordinates and the speed of position change with respect to the eye coordinates is greater than the threshold, or in the third state in which the eye tracking result does not include the eye coordinates, the eye tracking status may be classified as the unstable status. In the case of the second state, since the eye coordinates are present, the tracking 2D rendering mode may be performed using the eye coordinates. For example, if the eye tracking result includes current eye coordinates of both eyes and a speed of position change with respect to the current eye coordinates is greater than the threshold, the first viewpoint of the first source image and the second viewpoint of the second source image may set equally as a center viewpoint of the current eye coordinates. On the other hand, in the case of the third state, since eye coordinates are absent, the fixed 2D rendering mode may be performed using eye coordinates previously obtained. For example, if the eye tracking result does not include the current eye coordinates, the first viewpoint and the second viewpoint may be set equally as a center viewpoint of previous eye coordinates recently used.

In operation <NUM>, the HUD control apparatus renders the HUD image in the determined rendering mode. The HUD image may be displayed by a display device and provided to the user through a 3D optical layer. If 3D rendering is performed, the HUD image may traverse through the 3D optical layer such that images of different viewpoints may be provided to both eyes of the user. Even if 2D rendering is performed, the HUD image may be provided to the user through the 3D optical layer. However, in this case, unlike 3D rendering, an image of the same viewpoint may be provided to both eyes of the user. After operation <NUM> is performed on the current frame as described above, operations <NUM> to <NUM> may be performed on a subsequent frame. This process may be performed for each frame of the user image.

<FIG> illustrates eye tracking status according to an example embodiment. Referring to <FIG>, eye coordinates at positions marked with "X" may be obtained for each frame in a user image <NUM> by performing eye tracking on the user image <NUM>. In addition, since the eye coordinates for each frame do not change greatly, an eye tracking status of the user image <NUM> may be classified as a stable status.

For a user image <NUM> as well, eye coordinates at positions marked with "X" may be obtained for each frame through eye tracking. However, since the eye coordinates for each frame of the user image <NUM> change greatly, an eye tracking status of the user image <NUM> may be classified as an unstable status. For example, when a vehicle drives over a speed bump, drives on an uneven road, or takes a sharp curve, the eye positions may quickly change as shown in the user image <NUM>.

A user image <NUM> does not have "X" marks corresponding to eye positions each of the frames illustrating a case in which eye tracking fails. For example, eye tracking may fail as in the user image <NUM> due to an environmental factor such as low illuminance or occlusion.

If the eye tracking status is classified as a stable status as in the user image <NUM>, the HUD image may be rendered through a 3D rendering mode. If the eye tracking status is classified as an unstable status as in the user images <NUM> and <NUM>, the HUD image may be rendered through a 2D rendering mode. If eye coordinates are present as in the user image <NUM>, a tracking 2D rendering mode may be performed. If eye coordinates are absent as in the user image <NUM>, a fixed 2D rendering mode may be performed.

<FIG> illustrates eye movements in a viewing space according to an example embodiment. Referring to <FIG>, a viewing space <NUM> includes a first viewing space Si in which a first source image is viewed and a second viewing space S<NUM> in which a second source image is viewed. An eye position <NUM> is a position of a first viewpoint (for example, the left eye) at a time t<NUM>, and an eye position <NUM> is a position of a second viewpoint (for example, the right eye) at the time t<NUM>. A difference between the time t<NUM> and a time t<NUM> may correspond to a time difference between two consecutive frames. A user may view the first source image through the first viewpoint of the eye position <NUM> and view the second source image through the second viewpoint of the eye position <NUM>. The first viewing space Si and the second viewing space S<NUM> may be divided through a borderline <NUM>. An HUD control apparatus may adjust the borderline <NUM> in response to changes in the eye positions <NUM> and <NUM>, so that the eye position <NUM> may stay in the first viewing space Si and the eye position <NUM> may stay in the second viewing space S<NUM>.

The eye position <NUM> is the position of the first viewpoint at the time t<NUM>, and an eye position <NUM> is a position of the first viewpoint at the time t<NUM>. Further, the eye position <NUM> is the position of the second viewpoint at the time t<NUM>, and an eye position <NUM> is a position of the second viewpoint at the time t<NUM>. Thus, a speed of change (or speed of movement) of the eye positions <NUM> and <NUM> (or the eye coordinates) may be defined as Ve. In addition, a speed of adjustment (or speed of movement) of the borderline <NUM> may be defined as Vt. Ve and Vt may correspond to a variation of the eye positions <NUM> and <NUM> and a variation of the borderline <NUM> during the time difference between the two consecutive frames. Since system processing such as updating an HUD image is required to adjust the borderline <NUM>, the maximum value of Vt may be limited by the system processing rate. If Ve is greater than the maximum value of Vt, the eye position <NUM> of the first viewpoint may be in the second viewing space S<NUM>, and the eye position <NUM> of the second viewpoint may be in the first viewing space S<NUM>, for example, as shown in <FIG>. Accordingly, crosstalk may be observed.

A threshold may be set based on the system processing rate. The threshold may be a speed of the borderline <NUM> that is adjustable to the maximum based on the system processing rate. For example, the threshold may be set to <NUM> millimeters per second (mm/s). In this case, if a speed of change of the eye positions <NUM> and <NUM> (or the eye coordinates) is greater than the threshold in a frame of a user image, an eye tracking status for the frame may be determined to be an unstable status. Accordingly, an HUD image corresponding to the frame may be rendered through a 2D rendering mode. In detail, since eye coordinates are present, a tracking 2D rendering mode may be used.

<FIG> illustrates a process of switching a rendering mode according to an example embodiment. For example, the switching of the rendering mode may include switching from a 2D rendering mode to a 3D rendering mode and switching from a 3D rendering mode to a 2D rendering mode. In another example, the switching of the rendering mode may include switching from one of a tracking 2D rendering mode, a fixed 2D rendering mode, and a 3D rendering mode to another of the tracking 2D rendering mode, the fixed 2D rendering mode, and the 3D rendering mod. While the rendering mode is switched, a change in viewpoint may occur in an HUD image and cause a user to feel uncomfortable when viewing the HUD image. According to example embodiments, to reduce such discomfort, switching of the rendering mode may be performed for a predetermined time.

Referring to <FIG>, in operation <NUM>, the HUD control apparatus determines whether to switch the rendering mode. For example, in operation <NUM>, after the rendering mode is determined in operation <NUM> of <FIG>, it is determined as to whether to switch the rendering mode based on the determination of operation <NUM>. For example, if the rendering mode is determined to be a 3D rendering mode in operation <NUM> for iteration on a previous frame and the rendering mode is determined to be a 2D rendering mode in operation <NUM> for iteration on a current frame, the rendering mode is to be switched. In this case, operation <NUM> may be performed.

In operation <NUM>, the HUD control apparatus performs a switching operation during a buffer time. The buffer time may correspond to a plurality of frames. For example, if the frame rate of the HUD image is <NUM> frames per second (fps), the buffer time may correspond to <NUM> sec = <NUM> frames. If the rendering mode is to be switched from the 2D rendering mode to the 3D rendering mode, the HUD control apparatus may render the HUD image while gradually changing a first viewpoint and a second viewpoint to a single viewpoint used in the 2D rendering mode over the buffer time. The switching operation will be described further with reference to <FIG>.

If the rendering mode is determined to be a 3D rendering mode in operation <NUM> for iteration on a previous frame and the rendering mode is maintained to be the 3D rendering mode in operation <NUM> for iteration on a current frame, the rendering mode is not to be switched. In this case, the HUD image corresponding to the current frame may be rendered in the 3D rendering mode through operation <NUM> of <FIG>.

<FIG> illustrates buffer viewpoints and buffer source images for switching a rendering mode according to an example embodiment. <FIG> shows an example of switching a rendering mode from a 3D rendering mode to a 2D rendering mode. However, the disclosure is not limited thereto, and the example of <FIG> and the following description may also apply to another type of switching process according to another example embodiment. Referring to <FIG>, a first viewpoint <NUM> (for example, the left eye), a second viewpoint <NUM> (for example, the right eye), and a center viewpoint <NUM> are shown in a viewing space <NUM>. The center viewpoint <NUM> may be in the middle of the first viewpoint <NUM> and the second viewpoint <NUM>. Buffer viewpoints may be between the first viewpoint <NUM> and the center viewpoint <NUM> and between the second viewpoint <NUM> and the center viewpoint <NUM>. The number of buffer viewpoints may correspond to a buffer time. For example, if the buffer time corresponds to <NUM> frames, <NUM> buffer viewpoints may be between the first viewpoint <NUM> and the center viewpoint <NUM>, and <NUM> buffer viewpoints may be between the second viewpoint <NUM> and the center viewpoint <NUM>.

A first source image <NUM> may correspond to the first viewpoint <NUM>, a second source image <NUM> may correspond to the second viewpoint <NUM>, and a center source image <NUM> may correspond to the center viewpoint <NUM>. In addition, there may be buffer source images corresponding to the buffer viewpoints. In a 3D rendering mode, a 3D HUD may be provided by rendering the HUD image based on the first source image <NUM> and the second source image <NUM>. Further, in a 2D rendering mode, a 2D HUD may be provided by rendering the HUD image based on the center source image <NUM>. If the rendering mode is switched from the 3D rendering mode to the 2D rendering mode, a 3D HUD image may be rendered based on the first source image <NUM> and the second source image <NUM> at a time t<NUM>, buffer HUD images may be rendered based on buffer source images from a time t<NUM> to a time tB-<NUM>, and a 2D HUD image may be rendered based on the center source image <NUM> at a time tB. B may correspond to a buffer time.

The buffer source images may be generated based on an interpolation operation based on the first source image <NUM>, the second source image <NUM>, and the center source image <NUM>. For example, center source images corresponding to the buffer viewpoints between the first viewpoint <NUM> and the center viewpoint <NUM> may be generated through an interpolation operation using the first source image <NUM> and the center source image <NUM>, and center source images corresponding to the buffer viewpoints between the second viewpoint <NUM> and the center viewpoint <NUM> may be generated through an interpolation operation using the second source image <NUM> and the center source image <NUM>.

<FIG> illustrates images in a 3D rendering mode according to an example embodiment. Referring to <FIG>, a first source image <NUM> includes a virtual object <NUM> at a position corresponding to a first viewpoint, and a second source image <NUM> includes a virtual object <NUM> at a position corresponding to a second viewpoint. In <FIG>, the first viewpoint and the second viewpoint are different viewpoints. The first source image <NUM> is generated for displaying the virtual object <NUM> at an intersecting point where a line connecting the first viewpoint and a target position intersects with a virtual screen, and the second source image <NUM> is generated for displaying the virtual object <NUM> at an intersecting point where a line connecting the second viewpoint and the target position intersects with the virtual screen. An HUD image <NUM> may be generated through 3D rendering based on the first source image <NUM> and the second source image <NUM>, such that a user viewing the HUD image <NUM> may experience the virtual object <NUM> in 3D.

<FIG> illustrates images in a 2D rendering mode according to an example embodiment. Unlike the example of <FIG>, the first viewpoint and the second viewpoint in <FIG> correspond to the same viewpoint. Thus, a virtual object <NUM> of a first source image <NUM> and a virtual object <NUM> of a second source image <NUM> are located at the same position. An HUD image <NUM> may be generated through 2D rendering based on the first source image <NUM> and the second source image <NUM>.

For example, the first source image <NUM> and the second source image <NUM> of <FIG> may correspond to the first source image <NUM> and the second source image <NUM> of <FIG>. Further, the first source image <NUM> and the second source image <NUM> of <FIG> may correspond to the center source image <NUM> of <FIG>. In addition, the buffer source images of <FIG> may be generated through an interpolation operation based on the first source image <NUM>, the second source image <NUM>, and the first source image <NUM> (or the second source image <NUM>).

<FIG> and <FIG> illustrate a process of tracking eyes using a tracking region according to an example embodiment. The process of tracking the eyes may include using a tracking region described below. Referring to <FIG>, operations <NUM> and <NUM> are operations performed on a first frame F1 of a user image. In operation <NUM>, an HUD control apparatus performs eye detection on an entire region of an image of the first frame F1. For example, the HUD control apparatus may determine a detection region corresponding to eyes by scanning the entire image. In operation <NUM>, the HUD control apparatus determines a tracking region. The HUD control apparatus may determine the tracking region based on the detection region. For example, the size of the tracking region may be determined based on the size of the detection region, and the position of the tracking region may be determined to include the detection region at the center thereof.

Next, operations <NUM> to <NUM> are operations performed on a second frame F2 of the user image. In operation <NUM>, the HUD control apparatus performs eye tracking based on the tracking region. For example, the HUD control apparatus may detect the eyes within the tracking region by scanning the tracking region, rather than scanning an entire region of an image of the second frame F2. Such eye detection using the tracking region may be referred to as eye tracking. In operation <NUM>, the HUD control apparatus determines whether there are eyes in the tracking region. If the eyes are in the tracking region, the HUD control apparatus updates the tracking region, in operation <NUM>. In the same manner as the tracking region is determined based on the detection region of the first frame F1, the tracking region may be updated based on a detection region of the second frame F2.

Next, operations <NUM> to <NUM> are operations performed on a third frame F3 of the user image. In response to the determination of operation <NUM> that there are no eyes in the tracking region, the HUD control apparatus terminates a tracking mode and returns to a detection mode to perform eye detection in an entire region of an image of the third frame F3, in operation <NUM>. If the tracking region is updated in operation <NUM> in response to the determination of operation <NUM> that there are eyes in the tracking region, the HUD control apparatus performs eye tracking based on the updated tracking region, in operation <NUM>. The details of operations <NUM> to <NUM> are the same as those of operations <NUM> to <NUM>. As described above, if there are eyes in a tracking region, the tracking region may be updated and a tracking mode may be maintained. However, if there are no eyes in the tracking region, a detection mode may be activated again to scan the entire image.

Referring to <FIG>, frames F1, F2, and F3 of a user image are shown. The HUD control apparatus determines a detection region in the frame F1, and determines a first tracking region <NUM> based on the detection region. The eyes in the frame F2 may be at an upper right position than the eyes in the frame F1. The HUD control apparatus detects the eyes from a first tracking region <NUM> in the frame F2. Since the eyes are detected in the first tracking region <NUM>, the HUD control apparatus updates a tracking region based on a detection region within the first tracking region <NUM> in the frame F2. Accordingly, a second tracking region <NUM> is determined. In the same manner as in the frame F2, the eyes may be detected from the second tracking region <NUM> in the frame F3, and a third tracking region <NUM> may be determined by updating the tracking region. In this way, in response to the determination that the eyes are included in each tracking region, the HUD control apparatus may track the eyes without scanning the entire image.

<FIG> illustrates a process of generating an HUD image based on eye tracking according to an example embodiment. Referring to <FIG>, in operation <NUM>, an HUD control apparatus determines an eye tracking status. In operation <NUM>, the HUD control apparatus determines whether the eye tracking status is a stable status. According to the example embodiments described with reference to <FIG> and <FIG>, eye tracking may be performed using a tracking region.

If the eye tracking status corresponds to the stable status, the HUD control apparatus performs 3D rendering, in operation <NUM>. If the eye tracking status does not correspond to the stable status, the HUD control apparatus performs 2D rendering, in operation <NUM>. For example, if the eye positions are determined in the tracking region through eye tracking, but a speed of change in the eye positions is faster than a threshold, tracking 2D rendering may be performed, in operation <NUM>. When the eye positions are not determined in the tracking region through eye tracking, fixed 2D rendering may be performed, in operation <NUM>. In this case, a tracking mode for eye tracking may be canceled, and a detection mode may be activated again.

While 2D rendering is performed through operation <NUM>, the HUD control apparatus checks whether the status changes, in operation <NUM>. For example, a case in which the status changes may include a case in which the speed of change in the eye positions is reduced below the threshold while tracking 2D rendering is performed, and a case in which the speed of change in the eye positions is less than the threshold although the eyes are detected outside the tracking region while fixed 2D rendering is performed. If the status changes, the HUD control apparatus switches the rendering mode from a 2D rendering mode to a 3D rendering mode during a buffer time, in operation <NUM>. Then, in operation <NUM>, the HUD control apparatus performs 3D rendering.

Similarly, while 3D rendering is performed through operation <NUM>, the HUD control apparatus checks whether the status changes, in operation <NUM>. For example, a case in which the status changes may include a case in which the speed of change in the eye positions exceeds the threshold, and a case in which the eyes are not detected in a tracking region. If the status changes, the HUD control apparatus switches the rendering mode from the 3D rendering mode to the 2D rendering mode during a buffer time, in operation <NUM>. Then, in operation <NUM>, the HUD control apparatus performs 2D rendering.

<FIG> illustrates a method of controlling an HUD considering an eye tracking status according to an example embodiment, and further describes the example of <FIG>. Referring to <FIG>, in operation <NUM>, an HUD control apparatus generates an eye tracking result by performing eye tracking on a user image. In operation <NUM>, the HUD control apparatus determines an eye tracking status related to a change in eye positions based on the eye tracking result. In operation <NUM>, the HUD control apparatus determines a rendering mode for an HUD image to be one of a 2D rendering mode and a 3D rendering mode based on the eye tracking status. In operation <NUM>, the HUD control apparatus renders the HUD image in the determined rendering mode. In addition, the description provided with reference to <FIG>, <FIG>, and <FIG> may apply to the method of controlling an HUD, and thus, a detailed description will be omitted for conciseness.

<FIG> illustrates a configuration of an HUD control apparatus according to an example embodiment. Referring to <FIG>, an HUD control apparatus <NUM> includes a processor <NUM> and a memory <NUM>. The memory <NUM> is connected to the processor <NUM>, and may store instructions executable by the processor <NUM>, data to be computed by the processor <NUM>, or data processed by the processor <NUM>. The memory <NUM> may include a non-transitory computer-readable medium (for example, a high-speed random access memory) and/or a non-volatile computer-readable medium (for example, a disk storage device, a flash memory device, or another non-volatile solid-state memory device). However, the disclosure is not limited thereto, and according to another example embodiment, the memory <NUM> may be storage devices configured to store data, information and/or instructions.

The processor <NUM> may execute instructions to perform the operations described with reference to <FIG>, and <FIG>. For example, the processor <NUM> may generate an eye tracking result by performing eye tracking on the user image, determine an eye tracking status related to a change in eye positions based on the eye tracking result, determine a rendering mode for an HUD image to be one of a 2D rendering mode and a 3D rendering mode based on the eye tracking status, and render the HUD image in the determined rendering mode. In addition, the description provided with reference to <FIG>, and <FIG> may apply to the HUD control apparatus <NUM>, and thus, a detailed description will be omitted for conciseness.

<FIG> illustrates a configuration of an electronic device according to an example embodiment. Referring to <FIG>, an electronic device <NUM> may acquire a user image, track eyes from the acquired user image, and provide an AR HUD image based on an eye tracking status. The electronic device <NUM> may structurally and/or functionally include the HUD device <NUM> of <FIG>, the HUD control apparatus <NUM> of <FIG> and/or the HUD control apparatus <NUM> of <FIG>. For instance, the HUD device <NUM> of <FIG>, the HUD control apparatus <NUM> of <FIG> and/or the HUD control apparatus <NUM> of <FIG> may be implemented as the electronic device <NUM>.

The electronic device <NUM> may include a processor <NUM>, a memory <NUM>, a camera <NUM>, a storage device <NUM>, an input device <NUM>, an output device <NUM>, and a network interface <NUM>. However, the disclosure is not limited thereto, and as such, according to another example embodiment, the electronic device <NUM> may include other components or one or more of the components illustrated in <FIG> may be omitted from the electronic device <NUM>. The processor <NUM>, the memory <NUM>, the camera <NUM>, the storage device <NUM>, the input device <NUM>, the output device <NUM>, and the network interface <NUM> may communicate with each other through a communication bus <NUM>. For example, the electronic device <NUM> may be implemented as part of a means of transportation such as a car or an airplane.

The processor <NUM> executes instructions or functions to be executed in the electronic device <NUM>. For example, the processor <NUM> may process the instructions stored in the memory <NUM> or the storage device <NUM>. The processor <NUM> may perform the operations described through <FIG>.

The memory <NUM> stores a variety of data for providing an HUD image. The memory <NUM> may include a computer-readable storage medium or a computer-readable storage device. The memory <NUM> may store instructions to be executed by the processor <NUM> and may store related information while software and/or an application is executed by the electronic device <NUM>.

The camera <NUM> may capture a photo and/or a video. For example, the camera <NUM> may capture a user image including a user (for example, the face of the user). In detail, the camera <NUM> may include the eye tracking camera <NUM> of <FIG>. The camera <NUM> may provide a 3D image including depth information related to objects.

The storage device <NUM> includes a computer-readable storage medium or computer-readable storage device. The memory <NUM> may store a variety of data for providing an HUD image. The storage device <NUM> may store a more quantity of information than the memory <NUM> for a long time. For example, the storage device <NUM> may include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, or other non-volatile memories known in the art.

The input device <NUM> may receive an input from the user in traditional input manners through a keyboard and a mouse, and in new input manners such as a touch input, a voice input, and an image input. For example, the input device <NUM> may include a keyboard, a mouse, a touch screen, a microphone, or any other device that detects the input from the user and transmits the detected input to the electronic device <NUM>.

The output device <NUM> may provide an output of the electronic device <NUM> to the user through a visual, auditory, or haptic channel. The output device <NUM> may include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that provides the output to the user. In detail, the output device <NUM> may include the display device <NUM> of <FIG>. The network interface <NUM> may communicate with an external device through a wired or wireless network.

Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher-level code that may be executed by the computer using an interpreter. The devices may be configured to act as software modules in order to perform the operations of the above-described examples, or vice versa.

The software may include a computer program, a piece of code, an instruction, or some combination thereof, to independently or uniformly instruct or configure the processing device to operate as desired. The software also may be distributed over network-coupled computer systems so that the software is stored and executed in a distributed fashion. The software and data may be stored by non-transitory computer-readable recording mediums.

Claim 1:
A method of controlling a head-up display, HUD, (<NUM>), the method comprising:
performing (<NUM>) eye tracking of an eye of a user in a captured image;
characterized by:
identifying (<NUM>) an eye tracking status based on a result of the eye tracking
identifying (<NUM>) a rendering mode for an HUD image to be one of a two-dimensional, 2D, rendering mode and a three-dimensional, 3D, rendering mode based on the eye tracking status; and
rendering (<NUM>) the HUD image in the identified rendering mode,
wherein the identifying (<NUM>) the eye tracking status comprises classifying the eye tracking status as one of a stable status and an unstable status based on whether eye coordinates are present in the result of the eye tracking and based on a speed of position change with respect to the eye coordinates,
wherein the identifying (<NUM>) the rendering mode comprises:
identifying the rendering mode to be the 3D rendering mode based on the eye tracking status being classified as the stable status; and
identifying the rendering mode to be the 2D rendering mode based on the eye tracking status being classified as the unstable status,
wherein the eye tracking status is classified as the stable status based on the eye coordinates being included in the result of the eye tracking and a speed of position change with respect to the eye coordinates is less than a threshold, and
wherein the eye tracking status is classified as the unstable status based on the eye coordinates being included in the result of the eye tracking and the speed of position change with respect to the eye coordinates is greater than the threshold.