Patent Description:
In recent years, virtual reality technology is widely used in the fields such as gaming, engineering and military, etc. In order to experience the virtual reality environment, a user needs to view the displayed images through the display apparatus disposed at such as, but not limited a head-mounted device (HMD) wear by the user. The displayed images may be generated from the cameras disposed at the HMD. However, if the depth information of the images retrieved by the cameras or the baseline between the cameras is unknown, the 3D reconstruction cannot be performed accurately. Under such a condition, the user is not able to experience the accurate 3D environment.

<NPL>, discloses a method for calibrating an optical see-through Head Mounted Display (HMD) using techniques usually applied to camera calibration (photogrammetry). Using a camera placed inside the HMD to take pictures simultaneously of a tracked object and features in the HMD display, established camera calibration techniques are used to recover both the intrinsic and extrinsic properties of the HMD (width, height, focal length, optic centre and principal ray of the display).

Accordingly, what is needed is a 3D reconstruction method, a 3D reconstruction apparatus and a non-transitory computer readable storage medium thereof to address the above issues.

An aspect of the present disclosure is to provide a three dimensional (3D) reconstruction method used in a 3D reconstruction apparatus that includes the steps outlined below. A plurality of positioning signals are received by a signal receiver disposed at a head mounted device (HMD) at a first time spot and a second time spot to determine a HMD displacement vector and a HMD rotation amount according to the positioning signals. A first image at the first time spot and a second image at the second time spot are retrieved by a first camera disposed at the HMD to determine a first camera rotation amount according to the first image and the second image. A relative rotation amount and a relative displacement vector between the HMD and the first camera are calculated. A first camera displacement vector of the first camera is calculated according to the HMD displacement vector, the HMD rotation amount, the relative rotation amount and the relative displacement vector. Depth information of the first image and the second image is obtained based on the first camera displacement vector and the first camera rotation amount. 3D reconstruction is performed according to images retrieved by the first camera and the depth information.

Another aspect of the present disclosure is to provide a 3D reconstruction apparatus used in a HMD that includes a storage module, a signal receiver, a first camera and a processing module. The storage module is configured to store a plurality of computer-executable instructions. The processing module is electrically coupled to the storage module and the signal receiver and is configured to retrieve and execute the computer-executable instructions to perform a reconstruction method when the computer-executable instructions are executed. The reconstruction method includes the steps outlined below. A plurality of positioning signals are received by the signal receiver at a first time spot and a second time spot to determine a HMD displacement vector and a HMD rotation amount according to the positioning signals. A first image at the first time spot and a second image at the second time spot are retrieved by the first camera to determine a first camera rotation amount according to the first image and the second image. A relative rotation amount and a relative displacement vector between the HMD and the first camera are calculated. A first camera displacement vector of the first camera is calculated according to the HMD displacement vector, the HMD rotation amount, the relative rotation amount and the relative displacement vector. Depth information of the first image and the second image is obtained based on the first camera displacement vector and the first camera rotation amount. 3D reconstruction is performed according to images retrieved by the first camera and the depth information.

Yet another aspect of the present disclosure is to provide a non-transitory computer readable storage medium that stores a computer program comprising a plurality of computer-executable instructions to perform a 3D reconstruction method used in a 3D reconstruction apparatus used in a head-mounted device, the 3D reconstruction apparatus at least includes a storage module, a signal receiver, a first camera and a processing module electrically coupled to the storage module, the signal receiver and the first camera and configured to retrieve and execute the computer-executable instructions to perform the 3D reconstruction method when the computer-executable instructions are executed. The 3D reconstruction method includes the steps outlined below. A plurality of positioning signals are received by the signal receiver disposed at a head mounted device (HMD) at a first time spot and a second time spot to determine a HMD displacement vector and a HMD rotation amount according to the positioning signals. A first image at the first time spot and a second image at the second time spot are retrieved by the first camera disposed at the HMD to determine a first camera rotation amount according to the first image and the second image. A relative rotation amount and a relative displacement vector between the HMD and the first camera are calculated. A first camera displacement vector of the first camera is calculated according to the HMD displacement vector, the HMD rotation amount, the relative rotation amount and the relative displacement vector. Depth information of the first image and the second image is obtained based on the first camera displacement vector and the first camera rotation amount. 3D reconstruction is performed according to images retrieved by the first camera and the depth information.

These and other features, aspects, and advantages of the present disclosure will become better understood with reference to the following description and appended claims.

It is to be understood that both the foregoing general description and the following detailed description are by examples, and are intended to provide further explanation of the disclosure as claimed.

The disclosure can be more fully understood by reading the following detailed description of the embodiment, with reference made to the accompanying drawings as follows:.

Reference will now be made in detail to the present embodiments of the disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.

It will be understood that, in the description herein and throughout the claims that follow, when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may be present. Moreover, "electrically connect" or "connect" can further refer to the interoperation or interaction between two or more elements.

It will be understood that, in the description herein and throughout the claims that follow, although the terms "first," "second," etc. may be used to describe various elements, these elements should not be limited by these terms. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the embodiments.

It will be understood that, in the description herein and throughout the claims that follow, unless otherwise defined, all terms (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

Any element in a claim that does not explicitly state "means for" performing a specified function, or "step for" performing a specific function, is not to be interpreted as a "means" or "step" clause as specified in <NUM> U. § <NUM>(f). In particular, the use of "step of" in the claims herein is not intended to invoke the provisions of <NUM> U. § <NUM>(f).

<FIG> is a block diagram of a 3D reconstruction apparatus <NUM> in an embodiment of the present invention. In an embodiment, the 3D reconstruction apparatus <NUM> is used in a head-mounted device (HMD, not illustrated in <FIG>). More specifically, the components of the 3D reconstruction apparatus <NUM> are disposed at various positions of the HMD.

The 3D reconstruction apparatus <NUM> includes a storage module <NUM>, a signal receiver <NUM>, a first camera <NUM> and a processing module <NUM>.

In an embodiment, the storage unit <NUM> can be such as, but not limited to CD ROM, RAM, ROM, floppy disk, hard disk or optic magnetic disk. The storage unit <NUM> is configured to store a plurality of computer-executable instructions <NUM>.

The signal receiver <NUM> is disposed at the HMD and is configured to receive positioning signals <NUM> from a plurality of signal positioning apparatus <NUM>. In an embodiment, the signal positioning apparatus <NUM> are, for example, lighthouses that deliver the positioning signals <NUM> in the form of such as, but not limited to electromagnetic signals or supersonic signals. As a result, by keeping receiving the positioning signals <NUM>, the motion of the HMD can be determined.

The first camera <NUM> is configured to retrieve images at different time spots. In an embodiment, the first camera <NUM> is a camera with either charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS) image sensors.

The processing module <NUM> is electrically coupled to the storage unit <NUM>. In an embodiment, the processing module <NUM> is configured to retrieve and execute the computer-executable instructions <NUM> to operate the function of the 3D reconstruction apparatus <NUM> accordingly.

More specifically, by receiving the images retrieved by the first camera <NUM>, the processing module <NUM> is able to perform 3D reconstruction accordingly.

Reference is now made to <FIG>, <FIG> at the same time. The detail of the function of the 3D reconstruction apparatus <NUM> is described in the following paragraphs in accompany with <FIG>, <FIG>.

<FIG> is a flow chart of a 3D reconstruction method <NUM> in an embodiment of the present invention. The 3D reconstruction method <NUM> can be used in the 3D reconstruction apparatus <NUM> illustrated in <FIG>.

<FIG> is a diagram illustrating the motion of a HMD <NUM> that the 3D reconstruction apparatus <NUM> locates and the motion of the first camera <NUM> in an embodiment of the present invention. As described above, the first camera <NUM> is disposed at the HMD <NUM> as well.

<FIG> is a diagram illustrating a decomposed motion of the first camera <NUM> in <FIG> without illustrating the HMD <NUM> in an embodiment of the present invention.

The 3D reconstruction method <NUM> includes the steps outlined below (The steps are not recited in the sequence in which the steps are performed. That is, unless the sequence of the steps is expressly indicated, the sequence of the steps is interchangeable, and all or part of the steps may be simultaneously, partially simultaneously, or sequentially performed).

In step <NUM>, the positioning signals <NUM> are received from the positioning apparatus <NUM> by the signal receiver <NUM> at a first time spot and a second time spot to determine a HMD displacement vector to and a HMD rotation amount R<NUM> according to the positioning signals <NUM>.

In an embodiment, the signal receiver <NUM> may include a plurality of receiver units (not illustrated) disposed on various locations on the HMD <NUM>, in which a geometry center of the HMD <NUM> and a HMD coordinate-system centered at the geometry center can be calculated based on the distribution of the locations of the receiver units. In detail, the displacement vector and the rotation amount of the HMD coordinate-system can be determined by some well-known algorithms provided with the positioning signals <NUM> received by the plurality of receiver units. As well known, such algorithms are often formulated in the form to solve the perspective-n-points problem.

When the HMD <NUM> moves from a first position P1 corresponding to the first time spot to a second position P2 corresponding to the second time spot, as illustrated in <FIG>, the motion of the HMD <NUM> can be determined by the signal receiver <NUM> according to the positioning signals <NUM>, in which the motion can be expressed as the HMD displacement vector to and the HMD rotation amount R<NUM>. In an embodiment, the HMD displacement vector t<NUM> is a 3D vector and the HMD rotation amount R<NUM> is expressed as a 3D rotation amount, a quaternion or a rotation matrix.

In step <NUM>, a first image I1 and a second image I2 are retrieved by the first camera <NUM> respectively at the first time spot and the second time spot to determine a first camera rotation amount R<NUM> according to the first image I1 and the second image I2.

In an embodiment, the first camera rotation amount R<NUM> can be obtained by using known algorithms related to such as, but not limited to epipolar geometry based on the first image I1 and the second image I2, e.g. by calculating the difference of orientations of the objects within the first image I1 and the second image I2.

In step <NUM>, a relative rotation amount R<NUM> and a relative displacement vector t<NUM> between the HMD <NUM> and the first camera <NUM> are calculated by using Euclidean transformation matrix.

In an embodiment, though the first camera <NUM> is disposed at the HMD <NUM>, the position and the orientation of the first camera <NUM> are different from the position and the orientation of the HMD <NUM> coordinate-system determined according to the distribution of the location of receiver units. The relative displacement vector t<NUM> and the relative rotation amount R<NUM> exists between the first camera <NUM> and the HMD <NUM>.

In an embodiment, a motion between a HMD orientation of the HMD <NUM> at the first time spot (i.e. when the HDM <NUM> is at the first position P1) to a camera orientation of the first camera <NUM> at the second time spot (i.e. when the HMD <NUM> is at the second position P2) is equivalent to a first path and a second path. The first path is to rotate the HMD orientation of the HMD <NUM> at the first time spot (i.e. when the HMD <NUM> is at the first position P1) for the HMD rotation amount R<NUM> first and for the relative rotation amount R<NUM> subsequently. The second path is to rotate the HMD orientation at the first time spot for the relative rotation amount R<NUM> first and for the first camera rotation amount R<NUM> subsequently.

By applying Euclidean transformation matrix to describe the first path and the second path, the relation described above can be expressed as: <MAT>.

The right side of the equal sign in equation <NUM> means transforming XHMD (a coordinate of a point X in the HMD coordinate system) from the HMD coordinate-system at the first time spot to the camera coordinate-system of the first camera <NUM> at the first time spot, and then transforming to the camera coordinate-system of the first camera <NUM> at the second time spot. The left side of equation <NUM> means transforming XHMD from the HMD coordinate-system at the first time spot to the HMD coordinate-system at the second time spot, and then transforming to the camera coordinate-system at the second time spot.

We conclude that the equation <NUM> holds for any XHMD in 3D-space, so the Euclidean transformation matrix in both sides of the equal sign in the equation <NUM> are the same, i.e.<MAT>.

Multiplying the matrices in both sides of the equal sign in the equation <NUM> results in:
<MAT>.

Equalizing the corresponding entries of the matrices in both sides of the equal sign in equation <NUM>, we obtain:
<MAT><MAT>.

As a result, since the HMD rotation amount R<NUM> and the first camera rotation amount R<NUM> are known, the relative rotation amount R<NUM> can be solved by using spectral decomposition method and numerical optimization based on the equation <NUM>.

In an embodiment, the motion of the HMD <NUM> between the first time spot and the second time spot is decomposed to the HMD displacement corresponding to the HMD displacement vector to and the HMD rotation corresponding to the HMD rotation amount R<NUM>.

Similarly, the motion of the first camera <NUM> between the first time spot and the second time spot can be decomposed to a first displacement identical to the HMD displacement vector to that displaces to a temporary position P3 and a second displacement corresponding to a displacement vector t<NUM>, as illustrated in <FIG>.

Further, as shown in <FIG>, when both of the first camera <NUM> and the HMD <NUM> are displaced according to the HMD displacement vector to without rotation, the displacement vector between the first camera <NUM> and the HMD <NUM> under such a condition is t<NUM>. When both of the first camera <NUM> and the HMD <NUM> are displaced and rotated to the position corresponding to the second time spot, the displacement vector between the first camera <NUM> and the HMD <NUM> can be expressed as R<NUM>-<NUM>t<NUM> in the HMD coordinate-system at the second time spot before rotation.

This is because the rotation from "the HMD coordinate-system at the second time spot after rotation" to "the HMD coordinate-system at the second time spot before rotation" is R<NUM>-<NUM> (i.e. the inverse matrix of Ro). Moreover, the displacement vector between the first camera <NUM> and the HMD <NUM> is presented as t<NUM> in "the HMD coordinate-system at the second time spot after rotation" (if the relative position between HMD and camera remains the same during the period in which translation and rotation occurs). Thus the displacement vector between the first camera <NUM> and the HMD <NUM> from "the camera at the second time spot before rotation" to "the camera at the second time spot after rotation" is represented as R<NUM>-<NUM>t<NUM> in "the HMD coordinate-system at the second time spot before rotation".

Further, the displacement vector t<NUM> can be expressed as R<NUM>-<NUM>t<NUM> - t<NUM> in "the HMD coordinate-system at the first time spot". In other words, the HMD coordinate-system at the first time spot has the same orientation as the HMD coordinate-system at the second time spot without rotation, and so any vector has the same coordinate-representation in these two coordinate-systems at different time spots.

As a result, the sum of the displacement vector to from the position of the first camera <NUM> at the first time spot to the temporary position P3 and the displacement vector t<NUM> from the temporary position P3 to the position of the first camera <NUM> at the second time spot equals to the first camera displacement vector t<NUM>, in which such a relation can be expressed as:
<MAT>.

Further, the sum of the displacement vector t<NUM> from the position of the HMD <NUM> at the second time spot without rotation to the temporary position P3 and the displacement vector t<NUM> from the temporary position P3 to the position of the first camera <NUM> at the second time spot equals to the displacement vector R<NUM>-<NUM>t<NUM>, in which such a relation can be expressed as:
<MAT>.

By equations <NUM> and <NUM>, the formula can be further expressed as:
<MAT>.

However, the first camera displacement vector t<NUM> described in equations <NUM> and <NUM> are substantially under "the HMD coordinate-system at the first time spot". When the first camera displacement vector t<NUM> is under "the camera coordinate-system at the first time spot", a rotation R<NUM> is applied to equation <NUM> to obtain:
<MAT>.

By substituting equation <NUM> back to equation <NUM>, the following result is obtained:
<MAT>.

By rearranging the equation <NUM>, the following result is obtained:
<MAT>.

The relative displacement vector t<NUM> can thus be solved by using equation <NUM>.

In step <NUM>, the first camera displacement vector t<NUM> of the first camera <NUM> is calculated according to the HMD displacement vector t<NUM>, the HMD rotation amount Ro, the relative rotation amount R<NUM> between the first camera <NUM> and the HMD <NUM> and the relative displacement vector t<NUM> between the first camera <NUM> and the HMD <NUM>.

More specifically, by substituting t<NUM> back in equation <NUM>, the first camera displacement vector t<NUM> can be obtained.

In step <NUM>, depth information of the first image I1 and the second image I2 is obtained based on the first camera displacement vector t<NUM> and the first camera rotation amount R<NUM>.

More specifically, since the first camera displacement vector t<NUM> and the first camera rotation amount R<NUM> are obtained, the real distance of the objects moving between the first image I1 and the second image I2 can be determined. The depth information can thus be obtained.

In step <NUM>, 3D reconstruction is performed according to images, e.g. the first image I1 and the second image I2 retrieved by the first camera <NUM>, and the depth information.

In some approaches, an additional object that has a known size is required to appear in the images taken by the camera for the 3D reconstruction apparatus to obtain the depth information. However, when such an additional object is absent or if such an additional object deforms, the 3D reconstruction apparatus can not obtain the accurate depth information and is not able to perform 3D reconstruction accurately.

The 3D reconstruction apparatus <NUM> and the 3D reconstruction method <NUM> in the present invention can establish the depth information according to the information from the positioning signals <NUM> received by the signal receiver <NUM> and the images retrieved by the first camera <NUM>. Consequently, the 3D reconstruction can be performed accurately based on the accurate depth information.

It is appreciated that the embodiment is described by using a single camera scheme. In other embodiments, the method described above can be applied to a multi-camera scheme.

Reference is now made to <FIG> at the same time. <FIG> is a block diagram of a 3D reconstruction apparatus <NUM> in an embodiment of the present invention. <FIG> is a diagram illustrating the motion of a HMD <NUM> that the 3D reconstruction apparatus <NUM> locates in an embodiment of the present invention.

The 3D reconstruction apparatus <NUM> in <FIG> includes components identical to the 3D reconstruction apparatus <NUM> illustrated in <FIG>. As a result, the detail of the identical components is not described herein. However, the 3D reconstruction apparatus <NUM> further includes a second camera <NUM>.

As illustrated in <FIG>, both the first camera <NUM> and the second camera <NUM> are disposed at the HMD <NUM>. In an embodiment, the first camera <NUM> is a left camera and the second camera <NUM> is a right camera. However, in other embodiments, the first camera <NUM> can be the right camera and the second camera <NUM> can be the left camera.

In an embodiment, the depth information can be obtained by using either the first camera <NUM>, as illustrated in <FIG>, or the second camera <NUM> based on the method described in <FIG>.

Further, a third image I3 is retrieved by the first camera <NUM> and a fourth image I4 is retrieved by the second camera <NUM> at the same time spot. A disparity between the third image I3 and the fourth image I4 can be determined accordingly.

Claim 1:
A three dimensional, 3D, reconstruction method (<NUM>) used in a 3D reconstruction apparatus (<NUM>) comprising:
receiving (<NUM>) a plurality of positioning signals (<NUM>) by a signal receiver (<NUM>) disposed at a head mounted device, HMD, (<NUM>) at a first time spot and a second time spot to determine a HMD displacement vector and a HMD rotation amount according to the positioning signals;
retrieving (<NUM>) a first image at the first time spot and a second image at the second time spot by a first camera (<NUM>) disposed at the HMD (<NUM>) to determine a first camera rotation amount according to the first image and the second image;
calculating (<NUM>) a relative rotation amount and a relative displacement vector between the HMD (<NUM>) and the first camera (<NUM>);
calculating (<NUM>) a first camera displacement vector of the first camera (<NUM>) according to the HMD displacement vector, the HMD rotation amount, the relative rotation amount and the relative displacement vector;
obtaining (<NUM>) depth information of the first image and the second image based on the first camera displacement vector and the first camera rotation amount; and
performing (<NUM>) 3D reconstruction according to images retrieved by the first camera (<NUM>) and the depth information.