Patent Description:
A head-up display (HUD) system may generate a virtual image in front of a driver and display information in conjunction with the display of the virtual image to provide information to the driver. The information provided to the driver may include navigation information and dashboard information such as a vehicle speed, a fuel level, and an engine speed as measured in revolutions per minute (RPM). The driver may easily acquire the displayed information without needing to move a gaze during a driving, and thus a driving stability may be enhanced. In addition to the dashboard information and the navigation information, the HUD system may provide the driver with information on lane marking, construction marking, traffic accident marking, and/or warning signs for pedestrians based on an augmented reality (AR) technique.

<CIT> discloses a method and an apparatus for controlling an image-forming device of a head-up display in a vehicle.

It is the object of the present invention to provide an improved method for providing an accurate parameter for a virtual screen.

The above and/or other aspects will be more apparent by describing certain exemplary embodiments with reference to the accompanying drawings, in which:.

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. Exemplary embodiments are described below in order to explain the present disclosure by referring to the figures.

The following structural or functional descriptions are exemplary to merely describe the exemplary embodiments, and the scope of the exemplary embodiments is not limited to the descriptions provided in the present specification. Various changes and modifications can be made thereto by persons having ordinary skill in the art.

Although terms of "first" or "second" are used to explain various components, the components are not limited to the terms. These terms should be used only to distinguish one component from another component. For example, a "first" component may be referred to as a "second" component, or similarly, and the "second" component may be referred to as the "first" component within the scope of the right, according to the concept of the present disclosure.

As used herein, the singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the terms "comprises," "includes," "comprising," and/or "including," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components or a combination thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

Unless otherwise defined herein, all terms used herein including technical or scientific terms have the same meanings as those generally understood by one of ordinary skill in the art. Terms defined in dictionaries generally used should be construed to have meanings matching with contextual meanings in the related art and are not to be construed as an ideal or excessively formal meaning unless otherwise defined herein.

<FIG> is a diagram illustrating an augmented reality (AR) system, according to an exemplary embodiment. An AR may be a technology for overlapping a virtual object and a real world to be viewed by a user. The AR technology may be applied to, for example, a head-up display (HUD) and a transmissive head-mounted display (HMD). An AR system <NUM> may be implemented by, for example, the HUD.

Referring to <FIG>, the AR system <NUM> may include an AR optical system <NUM>, a transparent optical element <NUM>, a virtual screen <NUM>, and a viewpoint tracking camera <NUM>. The AR optical system <NUM> may include a light source, a display panel, and at least one optical element. The display panel and the light source may be used to provide light that corresponds to an AR image. The at least one optical element may reflect the light that corresponds to the AR image toward the transparent optical element <NUM>. For example, a light emitting diode (LED) or a laser may be used as the light source.

The light that corresponds to the AR image may be provided by the AR optical system <NUM> to form the virtual screen <NUM>. In addition, a portion of the light provided by the AR optical system <NUM> may be reflected by the transparent optical element <NUM>, which is positioned in front of a user and provided to the user. The transparent optical element <NUM> may be, for example, a windshield of a vehicle or an aircraft, or a combiner that is provided separate from the windshield and configured to reflect the AR image. The user may simultaneously view light coming from a source in front of the transparent optical element <NUM> and the portion of the light provided from the AR optical system <NUM> to be reflected by the transparent optical element <NUM>. In this example, a real object that overlaps a virtual object may be viewed by the user.

The AR system <NUM> may display the virtual object at a position that corresponds to the real object. For example, information on a vehicle driving direction, lane information, and obstacle information may be displayed at the position that corresponds to the real object on the HUD as the virtual object. Hereinafter, a position at which a virtual object is to be displayed may be referred to as a target position. To accurately display the virtual object at the target position, three-dimensional (3D) background information, a parameter of the virtual screen <NUM>, and user viewpoint information may be required. The AR system <NUM> may estimate a viewpoint of the user to display the virtual object at an intersection between the virtual screen <NUM> and a line connecting the viewpoint and the target position.

The 3D background information may be acquired via a 3D sensor (not shown) or a camera (not shown) facing a front of a vehicle. In addition, the viewpoint of the user may be acquired via the viewpoint tracking camera <NUM> that faces the user. The viewpoint tracking camera <NUM> may be an image sensor that includes a plurality of pixels and is configured to capture a color image or a gray scale image. A designed value may be used as the parameter of the virtual screen <NUM>. When basic driving information such as a current speed and a time of arrival is to be displayed, matching between the real object and the virtual object may not need to be performed, so that the designed value is used as the parameter of the virtual screen <NUM>. When the designed value is used, there may be a limit to the accuracy of the displaying of the virtual object at the target position because the designed value may be inaccurate. Even when the designed value is accurate, a parameter of the virtual screen <NUM> may be changed while the AR system <NUM> is installed or used.

An estimated value of the parameter of the virtual screen <NUM> may be used instead of the designed value. The parameter of the virtual screen <NUM> may include a size parameter and a transformation parameter. The size parameter may correspond to a virtual pattern interval of the virtual screen, a pixel interval of the virtual screen, or an overall size of the virtual screen. The transformation parameter may correspond to a pose, an orientation, or relative coordinates. The transformation parameter of the virtual screen <NUM> may be a transformation parameter of the virtual screen <NUM> relative to the viewpoint tracking camera <NUM>. By using the transformation parameter of the virtual screen <NUM>, coordinates of the virtual screen <NUM> may be transformed into coordinates of the viewpoint tracking camera <NUM>.

The AR system <NUM> may render the AR image by using an estimated parameter of the virtual screen <NUM> such that the virtual object is accurately displayed at the target position. The parameter of the virtual screen <NUM> may be estimated in a process of producing the AR system <NUM>, in an installation process of the AR system <NUM>, or in a process of utilizing the AR system <NUM>. For example, since the parameter of the virtual screen <NUM> may be changed in the process of utilizing the AR system <NUM>, an update may be continuously conducted via a process of estimating the parameter of the virtual screen <NUM>.

<FIG> is a diagram illustrating a calibration result, according to an exemplary embodiment. Referring to <FIG>, a virtual screen <NUM> and a background <NUM> may be overlapped to be provided to a user. When a parameter of the virtual screen <NUM> is inaccurate, a virtual object that is not matched to a real object may be provided to the user as shown in an image <NUM>. An estimation apparatus <NUM> may estimate the parameter of the virtual screen <NUM> and provide the estimated parameter of the virtual screen <NUM> to an AR system. The AR system may render an AR image by using the estimated parameter of the virtual screen <NUM>. In this manner, the virtual object that is matched to the real object may be provided to the user as shown in an image <NUM>.

The estimation apparatus <NUM> may capture a virtual image of the virtual screen <NUM> reflected by a reflector by using a viewpoint tracking camera to estimate the parameter of the virtual screen <NUM>. When the virtual screen <NUM> is directly captured using the viewpoint tracking camera, the parameter of the virtual screen <NUM> and a position of the viewpoint tracking camera may be unclear so that a measured value is not directly analyzed. The estimation apparatus <NUM> may additionally capture a known calibration pattern when capturing the virtual screen <NUM> via the reflector. In this manner, the estimation apparatus <NUM> may estimate the position of the viewpoint tracking camera from the calibration pattern and estimate the parameter of the virtual screen <NUM> based on the estimated position of the viewpoint tracking camera.

<FIG> is a flowchart illustrating a method of estimating a parameter of a virtual screen, according to an exemplary embodiment. Referring to <FIG>, in operation <NUM>, an estimation apparatus may acquire a plurality of captured images. In operation <NUM>, the estimation apparatus may detect a virtual pattern and a physical pattern from the plurality of captured images. In operation <NUM>, the estimation apparatus may estimate positions of virtual cameras. The plurality of captured images may be acquired by using a viewpoint tracking camera and a reflector. The virtual camera may be a result obtained by reflecting the viewpoint tracking camera to the reflector. In this aspect, in order to distinguish from the virtual camera, the viewpoint tracking camera may also be referred to as a physical camera. The plurality of captured images may be captured by adjusting a reflection angle of the reflector. Thus, a respective one from among a plurality of virtual cameras may be estimated for each reflection angle of the reflector.

A physical pattern displayed on a physical plane and a virtual pattern displayed on a virtual screen is used in a parameter estimating process. The estimation apparatus is aware of a size parameter of the physical pattern. The estimation apparatus determines positions of virtual cameras based on the physical pattern for which the size parameter is known and determines a size parameter of the virtual screen such that differences between the determined positions and positions of the virtual cameras estimated based on the virtual pattern are minimized.

Operations <NUM>, <NUM>, and <NUM> will be further described with reference to <FIG>, <FIG>, and <FIG>.

<FIG> is a diagram illustrating a system for estimating a parameter of a virtual screen. Referring to <FIG>, an estimation system <NUM> includes a physical camera <NUM>, a reflector <NUM>, virtual cameras <NUM>, a physical pattern <NUM>, a virtual pattern <NUM>, and an estimation apparatus <NUM>. The physical pattern <NUM> is present on a physical plane. The virtual pattern <NUM> is present on a virtual screen. The estimation apparatus <NUM> is aware of a size parameter of the physical pattern <NUM>. The physical pattern <NUM> may be a pattern of a first type, and the virtual pattern <NUM> may be a pattern of a second type that is different from the first type. Although <FIG> illustrates that the physical pattern <NUM> is a chessboard pattern and the virtual pattern <NUM> is a circular grid pattern, various calibration patterns may also be used as the physical pattern <NUM> and the virtual pattern <NUM> in addition to the chessboard pattern and the circular grid pattern.

The reflector <NUM> may reflect the physical pattern <NUM> and the virtual pattern <NUM>. The physical camera <NUM> captures the reflector <NUM> reflecting the physical pattern <NUM> and the virtual pattern <NUM>. Although what is actually captured by the physical camera <NUM> is a virtual image of the physical pattern <NUM> and a virtual image of the virtual pattern <NUM>, for ease of description, the virtual image of the physical pattern <NUM> may also be referred to as the physical pattern <NUM> and the virtual image of the virtual pattern <NUM> may also be referred to as the virtual pattern <NUM>. The physical camera <NUM> captures the reflector <NUM> while a reflection angle of the reflector <NUM> is adjusted. The physical camera <NUM> may acquire captured images by capturing a reflector <NUM> at a first angle, a reflector <NUM> at a second angle, and a reflector <NUM> at a third angle.

<FIG> is a diagram illustrating a plurality of captured images, according to an exemplary embodiment. Referring to <FIG>, in response to a reflection angle of a reflector being adjusted, a physical pattern and a virtual pattern may be captured at different angles. For example, a captured image <NUM> may be acquired via the reflector <NUM> of <FIG>. Likewise, captured images <NUM> and <NUM> may be acquired via the reflectors <NUM> and <NUM> of <FIG>, respectively.

Referring back to <FIG>, the estimation apparatus <NUM> may acquire a plurality of captured images by using the physical camera <NUM>. The estimation apparatus <NUM> may extract the physical pattern <NUM> and the virtual pattern <NUM> from each of the captured images. The estimation apparatus <NUM> may obtain a position of an intersection and a position of a calibration pattern from each of the captured images.

The estimation apparatus <NUM> may estimate the virtual cameras <NUM> that correspond to the physical camera <NUM> for each reflection angle of the reflector <NUM>. The virtual cameras <NUM> may correspond to virtual images of the physical camera <NUM> obtained via the reflector <NUM>. A virtual camera <NUM> may be estimated based on the reflector <NUM>. A virtual camera <NUM> may be estimated based on the reflector <NUM>. A virtual camera <NUM> may be estimated based on the reflector <NUM>.

As described above, the estimation apparatus <NUM> may determine positions of the virtual cameras <NUM> based on the physical pattern <NUM> for which a size parameter is known, and determine a size parameter of a virtual screen such that differences between the determined positions and positions of the virtual cameras <NUM> estimated based on the virtual pattern <NUM> are minimized. A process of estimating the positions of the virtual cameras <NUM> will be described below with reference to <FIG>.

<FIG> is a diagram illustrating a process of estimating positions of virtual cameras, according to an exemplary embodiment. An estimation apparatus may extract intersections from a physical pattern in a captured image. An intersection X on a physical pattern in a real world may correspond to coordinates (x, y, <NUM>) on the physical pattern in the captured image. A relationship between the intersection X and a point x obtained by projecting the intersection X onto an image plane that corresponds to the captured image may be expressed as shown in Equation <NUM>.

In Equation <NUM>, R denotes a rotation parameter of a pattern, t denotes a translation parameter of the pattern, i denotes an index of a captured image, and j denotes an index of an intersection in the captured image. Also, in Equation <NUM>, K denotes an intrinsic parameter of a camera and U denotes a calibration function. The intrinsic parameter K may be expressed by Equation <NUM>. The calibration function U may be expressed by Equation <NUM>.

In Equation <NUM>, fx denotes a focal length of the camera with respect to an x axis and fy denotes a focal length of the camera with respect to a y axis, and cx and cy denote coordinate values of a principal point. cx and cy may correspond to an x-axis value and a y-axis value of coordinates on which a principal axis of the camera or a z-axis of coordinates of the camera meets an image plane.

A distortion may occur in a camera that uses a lens and thus, the distortion may need to be corrected. In an example, a radial distortion and a tangential distortion may be taken into consideration based on Equation <NUM>. In Equation <NUM>, k<NUM> and k<NUM> denote parameters associated with the radial distortion, and k<NUM> and k<NUM> denote parameters associated with the tangential distortion in Equation <NUM> where r is <MAT>.

Equation <NUM> may be defined based on Equations <NUM>, <NUM>, and <NUM>.

If Equation <NUM> is referred to as <IMG>, a camera parameter may be obtained based on an optimization scheme for minimizing <IMG>. For example, the camera parameter may be obtained by minimizing a difference between the point x on a captured image and a point obtained by projecting real coordinates X on the captured image by using a function P. In Equation <NUM>, n denotes a number of captured images and m denotes a number of intersections in a captured image.

A rotation parameter R and a translation parameter t associated with intersections in each captured image may be obtained by using Equation <NUM>. The rotation parameter R and the translation parameter t may represent a pose of the physical pattern relative to the physical camera. A transformation parameter T may be represented on homogeneous coordinates based on the rotation parameter R and the translation parameter t as shown in Equation <NUM>.

Since the physical pattern is in a fixed position and the virtual camera moves, transformation parameters <MAT> of the virtual cameras relative to the physical pattern may be expressed as shown in Equation <NUM>.

In Equation <NUM>, <NUM> (i.e., a lowercase letter L) denotes an index of a virtual camera and has a value ranging from <NUM> to n. In <FIG>, <MAT> denotes a transformation parameter of a first virtual camera relative to a physical pattern, <MAT> denotes a transformation parameter of a second virtual camera relative to a physical pattern, O denotes reference coordinates of the physical pattern, and <MAT> through <MAT> denote relative coordinates of the first virtual camera through an nth virtual camera relative to the reference coordinates O. Among the first virtual camera through the nth virtual camera, a reference camera may be determined. A transformation parameter of a virtual camera relative to the reference camera may be expressed as shown in Equation <NUM>.

For example, the first virtual camera, which corresponds to the relative coordinates <MAT>, may be set to be the reference camera. In this example, <MAT> of <FIG> denotes a transformation parameter of the second virtual camera relative to the first virtual camera.

Transformation parameters of virtual cameras relative to a virtual pattern are determined via a process similar to that described with reference to <FIG>. In this manner, a transformation parameter of a virtual camera relative to a reference camera is determined. Hereinafter, a transformation parameter of a virtual camera relative to a reference camera determined based on a physical pattern may also be referred to as a first transformation parameter, and a transformation parameter of a virtual camera relative to a reference camera determined based on a virtual pattern may also be referred to as a second transformation parameter.

The estimation apparatus may extract intersections from a virtual pattern in a captured image. An accurate value of an intersection on a virtual pattern of a virtual screen may be unknown until a size parameter of the virtual screen is known. When a spacing of the virtual pattern is d, a horizontal length of the virtual pattern is w, a vertical length of the virtual pattern is h, a start position of the virtual pattern is (bx, by), and a pixel size of the virtual pattern is e, the intersection Y may be defined as shown in Equation <NUM>.

In Equation <NUM>, i denotes an index of a captured image, j denotes an index of an intersection, W denotes an xth ordinal position of Yij, and H denotes a yth ordinal position of Yij. W and H may be defined as shown in Equations <NUM> and <NUM>. In Equation <NUM>, values other than e may be defined in advance. <MAT> <MAT>.

A perspective-n-point (PnP) method may be used to estimate a pose of the virtual pattern relative to the virtual camera. Poses of the virtual pattern relative to the virtual cameras may be defined as R'i and t'i, i being an index of a captured image or a virtual pattern that appears in the captured image. The poses may correspond to coordinates of the virtual pattern and thus, may be transformed into poses of the virtual cameras relative to the virtual pattern as shown in Equation <NUM>.

In Equation <NUM>, <NUM> denotes an index of a virtual camera. As described above with reference to Equation <NUM>, a transformation parameter on homogeneous coordinates may be expressed as shown in Equation <NUM>.

Further, as described above with reference to Equation <NUM>, a reference camera may be determined from the virtual cameras and a transformation parameter of the virtual camera relative to the reference camera may be expressed as shown in Equation <NUM>.

Transformation parameters T̂ of the virtual cameras relative to the physical pattern and first transformation parameters <MAT> of the virtual cameras relative to the reference camera may be determined by using Equations <NUM> and <NUM>. Transformation parameters T̂' of the virtual cameras relative to the virtual pattern and second transformation parameters <MAT> of the virtual cameras relative to the reference camera may be determined by using Equations <NUM> and <NUM>. The estimation apparatus may determine the first transformation parameters <MAT> based on the transformation parameters T̂ and determine the second transformation parameters <MAT> based on the transformation parameters T̂'.

Referring back to <FIG>, the estimation apparatus may estimate a size parameter of the virtual screen in operation <NUM>. The estimation apparatus may estimate the size parameter of the virtual screen based on the first transformation parameters and the second transformation parameters. For example, the estimation apparatus may determine relative coordinates of the first transformation parameters and relative coordinates of the second transformation parameters. The relative coordinates of the second transformation parameters may vary based on the size parameter of the virtual screen. Since a position of the virtual camera using the physical pattern is the same as a position of the virtual camera using the virtual pattern, the size parameter of the virtual screen may be estimated therefrom.

The estimation apparatus may determine a value that minimizes a difference between each respective one of the first transformation parameters and each corresponding one of the second transformation parameters to be the size parameter, which may be expressed as shown in Equation <NUM> below.

In Equation <NUM>, <MAT> denotes a first transformation parameter, <MAT> denotes a second transformation parameter, n denotes a number of captured images, and e denotes a size parameter. The estimation apparatus may estimate the size parameter by using Equation <NUM>. In response to the size parameter of the virtual screen being estimated, a pose of the virtual screen relative to the virtual cameras may also be estimated.

Operation <NUM> will be further described below with reference to <FIG>, <FIG>, <FIG>, and <FIG>.

<FIG> is a flowchart illustrating a method of estimating a size parameter of a virtual screen by using transformation parameters, according to an exemplary embodiment. Referring to <FIG>, in operation <NUM>, an estimation apparatus may set a candidate size parameter. The candidate size parameter may be set to be a predetermined value. In operation <NUM>, the estimation apparatus may determine second transformation parameters based on the candidate size parameter. In operation <NUM>, the estimation apparatus may determine differences between respective first transformation parameters and corresponding ones of the second transformation parameters as determined based on the candidate size parameter. In operation <NUM>, the estimation apparatus may compare each of the determined differences to a threshold. The threshold may be set in advance. The estimation apparatus may estimate the candidate size parameter to be a size parameter based on a comparison result of operation <NUM>. When the determined differences are greater than the threshold, the estimation apparatus may change the candidate size parameter in operation <NUM> and then perform operation <NUM> again. When the determined differences are less than the threshold, the estimation apparatus may estimate the candidate size parameter to be the size parameter in operation <NUM>.

<FIG> is a diagram illustrating relative coordinates of a physical pattern and relative coordinates of a virtual pattern, according to an exemplary embodiment. A first transformation parameter may be represented by relative coordinates <NUM> of a physical pattern. A second transformation parameter may be represented by relative coordinates <NUM> of a virtual pattern. Distributions of the relative coordinates <NUM> of the physical pattern and the relative coordinates <NUM> of the virtual pattern may vary based on whether a size parameter is calibrated. A position of a virtual camera based on the physical pattern may be the same as a position of a virtual camera based on the virtual pattern. When the size parameter is calibrated, the relative coordinates <NUM> of the physical pattern and the relative coordinates <NUM> of the virtual pattern may indicate positions that correspond to each other. A distribution <NUM> may be obtained when the size parameter is not calibrated. A distribution <NUM> may be obtained when the size parameter is calibrated.

<FIG> is a diagram illustrating relative coordinates for each size parameter, according to an exemplary embodiment, and <FIG> is a graph illustrating a difference between relative coordinates of size parameters, according to an exemplary embodiment.

<FIG> illustrates relative coordinates of a physical pattern and relative coordinates of a virtual pattern obtained when a virtual screen has size parameters of K1, K2, K3, K4, K5, and K6. In <FIG>, the relative coordinates of the physical pattern are indicated on solid lines and the relative coordinates of the virtual pattern are indicated on dashed lines. Since a size parameter of the physical pattern is known, the relative coordinates of the physical pattern may be fixed. In addition, positions of the relative coordinates of the virtual pattern may be changed in response to a change in a size parameter of the virtual screen. For example, in response to a change in a size parameter of the virtual screen, the positions of the relative coordinates of the virtual pattern may approach the relative coordinates of the physical pattern and then, separate farther from the relative coordinates of the physical pattern.

In the graph of <FIG>, an x-axis represents a size parameter of a virtual screen and a y-axis represents a difference between relative coordinates of a physical pattern and relative coordinates of a virtual pattern. The estimation apparatus may calculate a first value that represents the relative coordinates of a physical pattern and a second value that represents the relative coordinates of the virtual pattern. The estimation apparatus may determine a difference between the first value and the second value to be the difference between a respective one of the relative coordinates of the physical pattern and a corresponding one of the relative coordinates of the virtual pattern. For example, a representative value of relative coordinates may be calculated to be an average value of reference coordinates of the relative coordinates.

Referring to <FIG>, a difference value of the relative coordinates of the physical pattern and the relative coordinates of the virtual pattern may be minimized when a size parameter of a virtual screen is K4. In the example of <FIG>, when the size parameter of the virtual screen is K4, the relative coordinates of the physical pattern and the relative coordinates of the virtual pattern may be at corresponding positions and thus, the differences between the relative coordinates of the physical pattern and the relative coordinates of the virtual pattern may be minimized. The estimation apparatus may estimate a candidate size parameter that minimizes the difference value to be the size parameter while adjusting the candidate size parameter. Further, the estimation apparatus may estimate a candidate size parameter that reduces the difference value to be less than a threshold to be the size parameter.

Referring back to <FIG>, in operation <NUM>, the estimation apparatus may estimate a transformation parameter of the virtual screen. In the following description, a transformation parameter of the virtual screen may be, for example, a transformation parameter of the virtual screen relative to a physical camera. The transformation parameter of the virtual screen may correspond to a pose, an orientation, or relative coordinates.

In an example, the estimation apparatus may estimate the transformation parameter of the virtual screen based on the size parameter estimated in operation <NUM>. When calibration of the size parameter is completed, the virtual pattern may function identically to the physical pattern and thus, the transformation parameter of the virtual screen may be estimated based on the virtual pattern. The estimation apparatus may estimate geometric relationships between patterns and reflectors based on captured images and calculate a projection error between the patterns and virtual images of the patterns in the captured images based on the estimated geometric relationships. The estimation apparatus may estimate the transformation parameter of the virtual screen relative to the physical camera such that the projection error is minimized. In this aspect, the pattern may be, for example, a physical pattern or a virtual pattern. The present example will be further described below with reference to <FIG>, <FIG>, and <FIG>.

In another example, the estimation apparatus may estimate a transformation parameter of the virtual screen relative to the physical camera based on transformation relationships of the physical camera, virtual cameras, and the virtual screen. When the estimation apparatus is aware of a transformation parameter of the physical camera relative to the virtual cameras and transformation parameters of the virtual cameras relative to the virtual screen, the estimation apparatus may estimate the transformation parameter of the virtual screen relative to the physical camera based on these transformation parameters.

When the size parameter of the virtual screen is estimated in operation <NUM> of <FIG>, a pose of the virtual screen relative to the virtual cameras may also be estimated. The estimation apparatus may estimate a transformation parameter of the virtual screen relative to the virtual cameras based on the first transformation parameters and the second transformation parameters in operation <NUM> of <FIG>. Further, the estimation apparatus may estimate a transformation parameter of the physical camera relative to the virtual cameras based on a geometric relationship between the physical pattern and a reflector, or estimate a transformation parameter of the physical camera relative to the virtual cameras by using a reflector that includes the physical pattern. The present example will be further described below with reference to <FIG>.

<FIG> is a diagram illustrating virtual images generated by a reflector, according to an exemplary embodiment. An estimation apparatus may estimate positions of virtual images based on captured images. The virtual images may be obtained by reflecting a physical pattern to a reflector or by reflecting a virtual pattern to the reflector. A number of virtual images may correspond to a number of captured images.

<FIG> is a diagram illustrating a geometric relationship of a pattern, a reflector, and a virtual image, according to an exemplary embodiment. <FIG> illustrates a geometric relationship of an estimated pattern <NUM>, a reflector <NUM>, and a virtual image <NUM>. The pattern <NUM> may be, for example, a physical pattern or a virtual pattern. An estimation apparatus may project a feature point of the pattern <NUM> on the virtual image <NUM> based on a candidate transformation parameter of a virtual screen. As the candidate transformation parameter of the virtual screen converges to true, a distance between the projected feature point and a corresponding point of the virtual image <NUM> may decrease. The estimation apparatus may verify whether the distance between the feature point and the corresponding point decreases while adjusting the candidate transformation parameter.

The estimation apparatus may be aware of a pose between a physical camera <NUM> and the virtual image <NUM> based on an extrinsic parameter of the physical camera <NUM>. The estimation apparatus may determine positions of the physical camera <NUM> and the virtual image <NUM>. The position of the pattern <NUM> may be determined based on an initial value of the candidate transformation parameter. The position of the reflector <NUM> may be determined in a middle between the pattern <NUM> and the virtual image <NUM>. An angle between the reflector <NUM> and the pattern <NUM> may be the same as an angle between the reflector <NUM> and the virtual image <NUM>. In this manner, the geometric relationship of the pattern <NUM>, the reflector <NUM>, and the virtual image <NUM> may be estimated.

The estimation apparatus may calculate a projection error between the pattern <NUM> and the virtual image <NUM> based on the geometric relationship of the pattern <NUM>, the reflector <NUM>, and the virtual image <NUM>. A normal vector nmi obtained via normalization of the pattern <NUM> may be expressed as shown in Equation <NUM>.

In Equation <NUM>, n denotes a normal vector of the pattern <NUM> and ni denotes a normal vector of the virtual image <NUM>. When a feature point Xj of the pattern <NUM> is moved toward the normal vector nmi in a normal direction of the reflector <NUM> by a distance dij, the feature point Xj may be projected on a position of Xij. dij may be expressed as shown in Equation <NUM>.

Further, a projection of the aforementioned feature point may be expressed as shown in Equation <NUM>.

A projection error may be a Euclidean distance from a feature point Xij of an ith virtual image, for example, the virtual image <NUM> that corresponds to tij(Xj) onto which the feature point Xj of the pattern <NUM> is projected. An average error Em obtained by projecting the feature point Xj to all virtual images may be expressed as shown in Equation <NUM>.

The estimation apparatus may determine a candidate transformation parameter that minimizes Em to be a transformation parameter. For example, when a value of Em which is obtained based on a value changed from an initial value of the candidate transformation parameter is less than a value of Em which is obtained based on the initial value, the candidate transformation parameter may be updated with a different value. The estimation apparatus may obtain a final transformation parameter by repetitively performing this process until the candidate transformation parameter remains unchanged.

<FIG> is a diagram illustrating a process of estimating a transformation parameter of a virtual screen relative to a physical camera by using a pattern, according to an exemplary embodiment. Referring to <FIG>, a candidate transformation parameter having minimum differences between X̃j and X<NUM>J, X<NUM>J, and X<NUM>J obtained based on a pattern <NUM>, reflectors <NUM>, <NUM>, and <NUM>, and virtual images <NUM>, <NUM>, and <NUM> may be determined to be a transformation parameter.

When the pattern <NUM> is a virtual pattern, the determined transformation parameter may be a transformation parameter of a virtual screen relative to a physical camera <NUM>. When the pattern <NUM> is a physical pattern, the determined transformation parameter may be a transformation parameter of a physical plane relative to the physical camera <NUM>. An estimation apparatus may estimate a transformation parameter of the physical camera <NUM> relative to virtual cameras by using the transformation parameter of the physical plane relative to the physical camera <NUM>. The estimation apparatus may estimate the transformation parameter of the virtual screen relative to the physical camera <NUM> based on the transformation parameter of the physical camera <NUM> relative to the virtual cameras and a transformation parameter of the virtual cameras relative to the virtual screen.

<FIG> is a diagram illustrating a conversion relationship of a physical camera, a virtual screen, and virtual cameras, according to an exemplary embodiment. Referring to <FIG>, an estimation apparatus may estimate a transformation parameter Tps of a virtual screen relative to a physical camera based on a transformation parameter Tpv of the physical camera relative to virtual cameras and a transformation parameter Tvs of the virtual cameras relative to the virtual screen.

In an example, as described above with reference to <FIG>, <FIG>, and <FIG>, the estimation apparatus may estimate a transformation parameter of the physical camera relative to the virtual cameras based on a geometric relationship between a physical pattern and a reflector. Further, in operation <NUM> of <FIG>, the estimation apparatus may estimate a transformation parameter of the virtual screen relative to the virtual cameras based on the first transformation parameters and the second transformation parameters.

In another example, the estimation apparatus may estimate a transformation parameter of the physical camera relative to the virtual cameras by using a reflector that includes a physical pattern. In addition, in operation <NUM> of <FIG>, the estimation apparatus may estimate a transformation parameter of the virtual screen relative to the virtual cameras based on the first transformation parameters and the second transformation parameters.

<FIG> is a diagram illustrating a reflector that includes a physical pattern, according to an exemplary embodiment. Referring to <FIG>, a reflector <NUM> may include a reflection area <NUM> and a physical pattern <NUM>. The reflector <NUM> may reflect a virtual pattern by using the reflection area <NUM>. An estimation apparatus may be aware of a size parameter of the physical pattern <NUM>. The estimation apparatus may estimate a transformation parameter of the reflector <NUM> relative to a physical camera based on the physical pattern <NUM>. Further, the estimation apparatus may estimate a geometric relationship between the reflector <NUM> and the physical camera and a geometric relationship between the reflector <NUM> and a virtual camera based on the transformation parameter of the reflector <NUM> relative to the physical camera. The estimation apparatus may estimate a transformation parameter of the physical camera relative to virtual cameras based on the geometric relationship between the reflector <NUM> and the physical camera and the geometric relationship between the reflector <NUM> and the virtual camera.

<FIG> is a diagram illustrating an estimation system that uses a head-mounted display, according to an exemplary embodiment. A head-mounted display <NUM> may be a device such as smart glasses that is attachable to a head of a user so as to provide an AR environment. The head-mounted display <NUM> may overlap a real object with a virtual object and display the overlapped real object in front of the user. An estimation apparatus may estimate a transformation parameter of a virtual screen relative to a physical camera <NUM> by using a physical pattern <NUM> and a virtual pattern <NUM>. The description of <FIG> and <FIG> may also be applied to an operation of the estimation apparatus.

<FIG> is a diagram illustrating an estimation system that uses a plurality of physical cameras, according to an exemplary embodiment. In the foregoing examples, a reflector may be used to acquire a plurality of captured images. Referring to <FIG>, a plurality of captured images may be acquired by using a plurality of physical cameras <NUM> instead of the reflector. An estimation apparatus may estimate a transformation parameter of a virtual screen relative to at least one of a plurality of physical cameras <NUM> by using a physical pattern <NUM> and a virtual pattern <NUM>. The description of <FIG> and <FIG> may also be applied to an operation of the estimation apparatus.

<FIG> is a block diagram illustrating an apparatus for estimating a parameter of a virtual screen, according to an exemplary embodiment. Referring to <FIG>, an estimation apparatus <NUM> may include a camera <NUM>, an AR optical system <NUM>, a processor <NUM>, and a memory <NUM>. The camera <NUM> may be an image sensor that includes a plurality of pixels and that is configured to capture a color image or a gray scale image. A captured image of the camera <NUM> may be provided to at least one of the processor <NUM> and the memory <NUM>. The AR optical system <NUM> may include a light source, a display panel, and at least one optical element. The display panel and the light source may be used to provide light that corresponds to an AR image. The at least one optical element may reflect the light that corresponds to the AR image toward a transparent optical element. For example, an LED or a laser may be used as the light source. The light that corresponds to the AR image may be provided by the AR optical system <NUM> to form a virtual screen <NUM>. Further, a portion of the light provided by the AR optical system <NUM> may be reflected by the transparent optical element, which is positioned in front of a user, and thereby provided to the user.

The memory <NUM> may include an instruction to be read by the processor <NUM>. When the instruction is executed in the processor <NUM>, the processor <NUM> may perform an operation to estimate a parameter of a virtual screen. The processor <NUM> may acquire, from a physical camera, a plurality of captured images that represent a reflector reflecting a virtual pattern displayed on the virtual screen and a physical pattern displayed on a physical plane, estimate virtual cameras that correspond to the physical camera for each reflection angle of the reflector, determine first transformation parameters of the virtual cameras relative to a reference camera of the virtual cameras based on transformation parameters of the virtual cameras relative to the physical pattern, determine second transformation parameters of the virtual cameras relative to the reference camera based on transformation parameters of the virtual cameras relative to the virtual pattern, and estimate a size parameter of the virtual screen based on the first transformation parameters and the second transformation parameters. The description of <FIG> and <FIG> may also be applied to the estimation apparatus <NUM>.

<FIG> is a flowchart illustrating a method of estimating a parameter of a virtual screen, according to an exemplary embodiment. Referring to <FIG>, in operation <NUM>, an estimation apparatus may acquire, from a physical camera, a plurality of captured images that represent a reflector reflecting a virtual pattern displayed on a virtual screen and a physical pattern displayed on a physical plane. In operation <NUM>, the estimation apparatus may estimate virtual cameras that correspond to the physical camera for each reflection angle of the reflector. In operation <NUM>, the estimation apparatus may determine first transformation parameters of the virtual cameras relative to a reference camera of the virtual cameras based on transformation parameters of the virtual cameras relative to the physical pattern. In operation <NUM>, the estimation apparatus may determine second transformation parameters of the virtual cameras relative to the reference camera based on transformation parameters of the virtual cameras relative to the virtual pattern. In operation <NUM>, the estimation apparatus may estimate a size parameter of the virtual screen based on the first transformation parameters and the second transformation parameters. In operation <NUM>, the estimation apparatus may estimate a transformation parameter of the virtual screen relative to the physical camera based on the estimated size parameter. The description of <FIG> may also be applied to the method of estimating the parameter of the virtual screen.

<FIG> is a flowchart illustrating a method of estimating a parameter of a virtual screen, according to an exemplary embodiment. Referring to <FIG>, in operation <NUM>, an estimation apparatus may acquire, from a physical camera, a plurality of captured images that represent a reflector reflecting a virtual pattern displayed on a virtual screen and a physical pattern displayed on a physical plane. In operation <NUM>, the estimation apparatus may estimate virtual cameras that correspond to the physical camera for each reflection angle of the reflector. In operation <NUM>, the estimation apparatus may estimate a transformation parameter of the physical camera relative to the virtual cameras based on the plurality of captured images. In operation <NUM>, the estimation apparatus may determine first transformation parameters of the virtual cameras relative to a reference camera of the virtual cameras based on transformation parameters of the virtual cameras relative to the physical pattern. In operation <NUM>, the estimation apparatus may determine second transformation parameters of the virtual cameras relative to the reference camera based on transformation parameters of the virtual cameras relative to the virtual pattern. In operation <NUM>, the estimation apparatus may estimate a size parameter of the virtual screen and transformation parameters of the virtual cameras relative to the virtual screen based on the first transformation parameters and the second transformation parameters. In operation <NUM>, the estimation apparatus may estimate a transformation parameter of the virtual screen relative to the physical camera based on the transformation parameters of the virtual cameras relative to the virtual screen and a transformation parameter of the physical camera relative to the virtual cameras. The description of <FIG> may also be applied to the method of estimating the parameter of the virtual screen.

<FIG> is a flowchart illustrating a method of estimating a parameter of a virtual screen, according to an exemplary embodiment. Referring to <FIG>, in operation <NUM>, an estimation apparatus may acquire, from a physical camera, a plurality of captured images that represent a reflector that includes a physical pattern and reflects a virtual pattern displayed on a virtual screen. In operation <NUM>, the estimation apparatus may estimate virtual cameras that correspond to the physical camera for each reflection angle of the reflector. In operation <NUM>, the estimation apparatus may estimate a transformation parameter of the physical camera relative to the virtual cameras based on a geometric relationship between the physical pattern and the reflector. In operation <NUM>, the estimation apparatus may determine first transformation parameters of the virtual cameras relative to a reference camera of the virtual cameras based on transformation parameters of the virtual cameras relative to the physical pattern. In operation <NUM>, the estimation apparatus may determine second transformation parameters of the virtual cameras relative to the reference camera based on transformation parameters of the virtual cameras relative to the virtual pattern. In operation <NUM>, the estimation apparatus may estimate a size parameter of the virtual screen and transformation parameters of the virtual cameras relative to the virtual screen based on the first transformation parameters and the second transformation parameters. In operation <NUM>, the estimation apparatus may estimate a transformation parameter of the virtual screen relative to the physical camera based on the transformation parameters of the virtual cameras relative to the virtual screen and a transformation parameter of the physical camera relative to the virtual cameras. The description of <FIG> may also be applied to the method of estimating the parameter of the virtual screen.

The units and/or modules described herein may be implemented using hardware components and/or software components. For example, the hardware components may include microphones, amplifiers, band-pass filters, audio to digital convertors, and processing devices. A processing device may be implemented by using one or more hardware device configured to carry out and/or execute program code by performing arithmetical, logical, and input/output operations. The processing device(s) may include a processor, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a field programmable array, a programmable logic unit, a microprocessor, and/or any other device capable of responding to and executing instructions in a defined manner. The processing device may run an operating system (OS) and one or more software applications that run on the OS. The processing device also may access, store, manipulate, process, and create data in response to execution of the software. For purpose of simplicity, the description of a processing device is used as singular; however, one skilled in the art will appreciate that a processing device may include multiple processing elements and multiple types of processing elements. For example, a processing device may include multiple processors or a processor and a controller. In addition, different processing configurations are possible, such as a configuration that implements parallel processors.

Claim 1:
A method of estimating a size parameter of a virtual screen (<NUM>), the method comprising
acquiring (<NUM>), from a physical camera (<NUM>), a plurality of captured images (<NUM> - <NUM>) that represent a reflector (<NUM>) reflecting a virtual pattern (<NUM>) displayed on a virtual screen (<NUM>) and a physical pattern (<NUM>) displayed on a physical plane, wherein each of the plurality of captured images is acquired by adjusting a reflection angle of the reflector;
determining (<NUM>) at least one first transformation parameter for each virtual camera from among a plurality of virtual cameras (<NUM>) relative to a reference camera of the plurality of virtual cameras (<NUM>) based on a rotation parameter R and a translation parameter t associated with intersections of the physical pattern in each captured image of the plurality of captured images, wherein each virtual camera from among the plurality of virtual cameras (<NUM>) corresponds to the physical camera (<NUM>) for a respective reflection angle of the reflector (<NUM>), wherein a size parameter of the physical pattern is known;
determining (<NUM>) at least one second transformation parameter for each respective one from among the plurality of virtual cameras (<NUM>) relative to the reference camera based on a rotation parameter R and a translation parameter t associated with intersections of the virtual pattern in each captured image of the plurality of captured images; and
estimating (<NUM>) a size parameter of the virtual screen (<NUM>) based on each of the at least one first transformation parameter and each of the at least one second transformation parameter, wherein the estimating the size parameter comprises estimating a value that corresponds to a minimum difference between a respective one from among the at least one first transformation parameter and a corresponding one from among the at least one second transformation parameter.