Patent ID: 12205328

DETAILED DESCRIPTION

Reference will now be made in detail to the present embodiments of the disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.

Reference is made toFIG.1, which is a schematic diagram illustrating a tracking system100according to an embodiment of this disclosure. As shown inFIG.1, the tracking system100includes a camera120, a first trackable device141and a tracking station160located in a spatial area SA. For example, the spatial area SA as shown inFIG.1can be a filming studio or a conference room in the real world, but the disclosure is not limited thereto. In some other embodiments, the spatial area SA can also be a specific area at an outdoor space (not shown in figures).

In some embodiments, the camera120is able to film a video or capture an image about a real object OBJ in the real world. For example, the real object OBJ is a fighting actor and the camera120is able to film a video about the fighting actor and merge the real object OBJ as a character in an immersive scenario. The immersive scenario may further include a virtual background (e.g., outer space) and some virtual objects (e.g., spaceships and aliens). To make sure the immersive scenario look real, it is important to track a position and an orientation of the camera120, such that a view point of the video captured by the camera120can be determined precisely.

In some embodiments, the tracking system100includes a first trackable device141and a tracking station160. As shown inFIG.1, the first trackable device141is physically attached to the camera120. The tracking station160is disposed at a fixed point in the spatial area SA. For example, the tracking station160is disposed at a corner near a ceiling of the room shown inFIG.1. When the camera120moves, the first trackable device141correspondingly moves along with the camera120. The tracking station160is capable of tracking the first trackable device141, such that the tracking station160is able to acknowledge a rough position about the camera120(according to the tracked position of the first trackable device141). In this case, the tracking station160is not able to track an exact position of the camera120, because there is an offset distance still existed between the first trackable device141and the camera120.

In some embodiments, the tracking station160may emit some optical tracking signals, and the first trackable device141may include optical sensors (not shown) for sensing the optical tracking signals from the tracking station160, so as to track a spatial relationship between the first trackable device141and the tracking station160. However, the disclosure is not limited to this optical sensing manner to track the first trackable device141. In some other embodiments, the tracking station160can utilize a computer vision to track a feature pattern on the first trackable device141.

Based on functions of the tracking station160and the first trackable device141, the tracking system100can track a reference center C141of the first trackable device141. Because the first trackable device141is physically attached to the camera120, in some cases, the reference center C141is recognized as the position of the camera120. However, as shown inFIG.1, there is an offset distance DIS between the reference center C141of the first trackable device141and an optical center C120of the camera120.

If the reference center C141of the first trackable device141is assumed to be the view point of the camera120, the video filmed by the camera120will be assumed to capture from a wrong view point (i.e., the reference center C141) slightly shifted from a real view point (i.e., the optical center C120). It is desired to acknowledge the offset distance DIS between the reference center C141and the optical center C120to calibrate aforesaid shifting. The optical center C120of the camera120is located inside the camera120and affected by lens, pixel sensors and optical components of the camera120. It is hard to determine a precise position of the optical center C120of the camera120. Therefore, the offset distance DIS can't be measured directly.

In some embodiments, the tracking system100provides a manner to measure the offset distance DIS, so as to track and calibrate the camera120precisely. Reference is further made toFIG.2, which is a schematic diagram illustrating the tracking system100with a function to track the optical center C120of the camera120during a calibration process according to an embodiment of this disclosure.

As shown inFIG.2, the tracking system100further includes a calibration chart180, a second trackable device142and a processing unit190. The second trackable device142is physically attached to the calibration chart180. The tracking station160is capable of tracking the second trackable device142, so as to track a reference center C142of the second trackable device142.

In some embodiments, the processing unit190is communicated with the camera120, the tracking station160, the first trackable device141and the second trackable device142. The processing unit190can be a central processing unit (CPU), a graphic processing unit (GPU), a processor and/or an application-specific integrated circuit (ASIC). In some embodiments, the processing unit190can be implemented in a stand-alone computer or a stand-alone server, but the disclosure is not limited thereto. In some other embodiments, the processing unit190can be integrated in the camera120or the tracking station160.

In some embodiments, the tracking station160may emit some optical tracking signals, and the second trackable device142may include optical sensors (not shown) for sensing the optical tracking signals from the tracking station160, so as to track a spatial relationship between the second trackable device142and the tracking station160. However, the disclosure is not limited to this optical sensing manner to track the second trackable device142. In some other embodiments, the tracking station160can utilize a computer vision to track the second trackable device142.

The calibration chart180includes a feature pattern182and a mounting socket184. As shown inFIG.2, the feature pattern182in some embodiments can be a chessboard pattern, which includes white blocks and dark blocks with predetermined sizes and arranged in predetermined gaps. The chessboard pattern is helpful to perform a geometric camera calibration on the camera120. For example, the geometric camera calibration can be executed based on a pinhole camera model for calibration.

The second trackable device142is attached on the mounting socket184and located with a mechanical arrangement relative to the feature pattern182. As shown inFIG.2, when the second trackable device142is mounted on the mounting socket184, a reference center C142of the second trackable device142will hold at a fixed position limited by the mounting socket184. In this case, the reference center C142of the second trackable device142will has a predetermined relationship relative to a reference center C182of the feature pattern182on the calibration chart180. The mechanical arrangement can be directly measured and manually designed. For example, by forming the mounting socket184at a desired position, the reference center C142can be placed at 30 cm left and 10 cm under relative to the reference center C182of the feature pattern182in the mechanical arrangement. Aforesaid spatial relationship between the second trackable device142and the feature pattern182can be described as a fifth rotation-translation matrix RT5.

In some embodiments, during the calibration process, the camera120is triggered to capture at least one image IMG involving the calibration chart180. Based on the image IMG and tracking results of the first/second trackable devices141and142, the processing unit190is able to calculate the offset distance between the reference center C141of the first trackable device141and the optical center C120of the camera120.

Reference is further made toFIG.3andFIG.4.FIG.3is a flow chart illustrating a control method300aperformed by the tracking system100inFIG.2during the calibration process.FIG.4is a schematic diagram illustrating the tracking system100during the calibration process according to an embodiment of this disclosure.

As shown inFIG.2,FIG.3andFIG.4, step S310is executed to trigger the camera120to capture the image IMG (as shown inFIG.2) involving the calibration chart180.

Step S312is executed to track pose data of the first trackable device141by the tracking station160, further to generate a first rotation-translation matrix RT1, between the first trackable device141and the tracking station160, according to the pose data of the first trackable device141. The first rotation-translation matrix RT1is configured to describe a rotational relationship and a positional relationship between a coordinate system O141of the first trackable device141and a coordinate system O160of the tracking station160. In some embodiments, the first rotation-translation matrix RT1defines as a rotation-translation matrix RTfirst trackable devicetracking stationfor transforming a vector originally in the coordinate system O141into an equivalent vector in the coordinate system O160.

Step S314is executed to track pose data of the second trackable device142by the tracking station160, further to generate a second rotation-translation matrix RT2, between the second trackable device142and the tracking station160, according to the pose data of the second trackable device142. The second rotation-translation matrix RT2is configured to describe a rotational relationship and a positional relationship between a coordinate system O142of the second trackable device142and the coordinate system O160of the tracking station160. In some embodiments, the second rotation-translation matrix RT2defines as a rotation-translation matrix RTtracking stationsecond trackable devicefor transforming a vector originally in the coordinate system O160into an equivalent vector in the coordinate system O142.

Step S316is executed to provide a fifth rotation-translation matrix RT5between the calibration chart180and the second trackable device142. In some embodiments, the fifth rotation-translation matrix RT5is derived according to spatial relationship between the second trackable device142and the feature pattern182. Since the second trackable device142is mounted on the mounting socket184disposed on the calibration chart180, a location of the mounting socket184can be designed to achieve desirable values of the fifth rotation-translation matrix RT5. The fifth rotation-translation matrix RT5can be directly measured and manually designed by adjusting the location of the mounting socket184. In some embodiments, the fifth rotation-translation matrix RT5defines as a rotation-translation matrix RTsecond trackable devicecalibration chartfor transforming a vector originally in the coordinate system O142into an equivalent vector in the coordinate system O182.

Step S320is executed to generate a third rotation-translation matrix RT3between a camera coordinate system O120of the camera120and a coordinate system O182of the feature pattern182on the calibration chart180according to the calibration chart180appeared in the image IMG. In some embodiments as shown inFIG.3.

In some embodiments, during the step S320, the processing unit190utilizes a computer vision algorithm to generate the third rotation-translation matrix RT3, between the camera coordinate system O120of the camera120and the coordinate system O182of the feature pattern182on the calibration chart180, according to the calibration chart180appeared in the image IMG In some embodiments, the computer vision algorithm in step S320is performed to detect a relative movement between the camera120and the calibration chart180. For example, when the feature pattern182appears to be smaller in the image IMG, the third rotation-translation matrix RT3generated by the processing unit190will indicate that the camera coordinate system O120is moved to a position far from the coordinate system O182. In another example, when the feature pattern182appears to be bigger in the image IMG, the third rotation-translation matrix RT3generated by the processing unit190will indicate that the camera coordinate system O120is moved to a position close to the coordinate system O182. In another example, when one edge of the feature pattern182appears to be bigger than an opposite edge of the feature pattern182in the image IMG, the third rotation-translation matrix RT3generated by the processing unit190will indicate a rotation angle between the camera coordinate system O120is close to the coordinate system O182. In some embodiments, the third rotation-translation matrix RT3defines as a rotation-translation matrix RTcalibration chartcamerafor transforming a vector originally in the coordinate system O182into an equivalent vector in the camera coordinate system O120.

It is noticed that the camera coordinate system O120is originated at the optical center C120of the camera120. It is hard to directly measure the position of the optical center C120in a mechanical way, because the optical center C120is a theoretical point inside the camera120. In this case, a fourth rotation-translation matrix RT4between the camera coordinate system O120and the coordinate system O141of the first trackable device141can't be measured directly according to the physical connection between the first trackable device141and the camera120. As shown inFIG.3andFIG.4, step S330is executed by the processing unit190to calculate the rotation-translation matrix RT4, between the camera coordinate system O120and the coordinate system O141of the first trackable device141, according to the rotation-translation matrices RT1, RT2, RT3and RT5.

In some embodiments, the fourth rotation-translation matrix RT4defines as a rotation-translation matrix RTcamerafirst trackable devicefor transforming a vector originally in the camera coordinate system O120into an equivalent vector in the coordinate system O141(of the first trackable device141). The fourth rotation-translation matrix RT4can be calculated as below:

RT⁢4=RTfirst⁢trackable⁢devicecamera=RTcalibration⁢chartcamera×RTsecond⁢trackable⁢devicecalibration⁢chart×RTtracking⁢stationsecond⁢trackable⁢device×RTfirst⁢trackable⁢devicetracking⁢station=RT⁢3×RT⁢5×RT⁢2×RT⁢1

As shown above, the fourth rotation-translation matrix RT4is calculated by the processing unit190according to a product of the third rotation-translation matrix RT3, the fifth rotation-translation matrix RT5, the second rotation-translation matrix RT2and the first rotation-translation matrix RT1.

In some embodiments, the tracking system100and the control method300ais able to calculate the fourth rotation-translation matrix RT4during the calibration process. The fourth rotation-translation matrix RT4is configured to describe a rotational relationship and a positional relationship between the camera coordinate system O120and the first trackable device141. Because the first trackable device141is physically attached to the camera120at a fixed position, the fourth rotation-translation matrix RT4known in S330will remain stable. The camera coordinate system O120is originated at the optical center C120of the camera120.

In this case, the tracking system100is able to track a precise position of the optical center C120of the camera120and the camera coordinate system O120, by tracking the first trackable device141(and the coordinate system O141) and applying the fourth rotation-translation matrix RT4onto the tracking result of the first trackable device141(and the coordinate system O141).

In step S340, as shown inFIG.2,FIG.3andFIG.4, the processing unit190can store the fourth rotation-translation matrix RT4. In some embodiments, the fourth rotation-translation matrix RT4can be stored in digital data storage (not shown in figures), such as a memory, a hard drive, a cache, a flash memory or any similar data storage. After the calibration process, the stored fourth rotation-translation matrix RT4is utilized to track the optical center C120of the camera120.

In aforesaid embodiments shown inFIG.3, the control method300atriggers the camera to capture one image IMG for calculating the fourth rotation-translation matrix RT4. However, the disclosure is not limited to capture one image IMG.

In some embodiments, the camera120requires some calibration parameters (e.g., intrinsic parameters and/or distortion parameters) to adjust an image frame of the camera120. In other words, based on the intrinsic parameters and/or the distortion parameters, the camera120can convert optical sensing results of pixel sensors into a frame of the image IMG accordingly. Reference is further made toFIG.5, which is another flow chart illustrating a control method300bperformed by the tracking system100inFIG.2during the calibration process. In the embodiments shown inFIG.5, the control method300bis able to generate a fourth rotation-translation matrix RT4(for tracking the camera120) and also measure intrinsic parameters and/or distortion parameters of the camera120.

The control method300bin the embodiments shown inFIG.5is similar to the control method300ainFIG.3. One difference between the control methods300aand300bis that, as shown inFIG.2andFIG.5, during step S310, the control method300btriggers the camera to capture N images IMGa˜IMGn involving the calibration chart180. N is a positive integer larger than 1. For example, if N=5, the camera120can capture 5 different images involving the calibration chart180. The control method300bfurther includes step S315.

Reference is further made toFIG.6AandFIG.6B, which are schematic diagrams illustrating the camera120capturing the images IMGa˜IMGn at different positions relative to the calibration chart180during the calibration process according to an embodiment of this disclosure.

As shown inFIG.6A, the camera120is located at one position relative to the calibration chart180, and the camera120capture one image IMGa; as shown inFIG.6B, the camera120is located at another position relatve to the calibration chart180, and the camera120capture another image IMGn.FIG.6AandFIG.6Billustrates two images IMGa and IMGn captured from two positions. However, the disclosure is not limited thereto. The camera120can moved to more different positions to capture more images.

In step S315, according to the feature pattern182of the calibration chart180appeared in the images IMGa˜IMGn, the processing unit190can perform a geometric camera calibration to the camera120, to generate intrinsic parameters and distortion parameters. In some embodiments, the geometric camera calibration is a process of estimating the intrinsic parameters and the distortion parameters of a camera model (e.g., a pinhole camera model) approximating the camera120that produced a given photograph (i.e., the feature pattern182shown inFIG.2). It is noticed that the disclosure is not limited to the pinhole camera model, and other camera models can also be utilized to generate the intrinsic parameters and the distortion parameters. Details about how to perform the geometric camera calibration based on the camera model are widely discussed and well known by a skilled person in the art, and not to be mentioned here.

The intrinsic parameters are related to coordinate system transformations from the camera coordinate system O120to a two-dimensional pixel coordinate system (for pixel sensors of the camera120, not shown in figures) of the image IMG. The intrinsic parameters are affected by internal parameters inside the camera120, for example, a focal length, the optical center C120and a skew coefficient of the camera120.

In this case, the intrinsic parameters generated in S315can be stored, and be utilized to adjust an image frame of the camera120while the camera120is capturing an image without involving the calibration chart180. For example, when the calibration process ends, the stored intrinsic parameters can be utilized to adjust the image frame of the camera120while the camera120is capturing an image or shooting a video or a film about the object OBJ (referring toFIG.1).

The distortion parameters are related to nonlinear lens distortions of the camera120. In some embodiments, the distortion parameters can also be calculated by the processing unit190in the geometric camera calibration.

In this case, the distortion parameters generated in S315can be stored to adjust an image frame of the camera120while the camera120is capturing an image without involving the calibration chart180. For example, when the calibration process ends, the stored distortion parameters can be utilized to adjust the image frame of the camera120while the camera120is capturing an image or shooting a video or a film about the object OBJ (referring toFIG.1).

It is noticed that, ideally, the intrinsic parameters calculated in the S315will be the same among the N images IMGa˜IMGn; the distortion parameters calculated in the S315will be the same among the N images IMGa˜IMGn; it is because that the intrinsic parameters and the distortion parameters are affected by internal factors of the camera120.

In step S320, the control method300butilizes a computer vision algorithm to generate N third rotation-translation matrices RT3a˜RT3n, between the camera coordinate system O120of the camera120and the calibration chart180, according to the feature pattern182of the calibration chart180appeared in the images IMGa˜IMGn. Each one of the N third rotation-translation matrices RT3a˜RT3nis generated in a similar way discussed in aforesaid embodiments of step S320(generating one third rotation-translation matrix RT3according to one image IMG) shown inFIG.3.

In step S331, the processing unit190calculates N candidate rotation-translation matrices RT4a˜RT4n, between the camera coordinate system O120and the first trackable device141, according to the first rotation-translation matrices RT1a˜RT1n, the second rotation-translation matrices RT2a˜RT2n, the N third rotation-translation matrices RT3a˜RT3nand the fifth rotation-translation matrix RT5.

Details about the steps S310-S331of the control method300bshown inFIG.5can be realized by performing N times of the step S310-S330of the control method300awhile moving the camera120to different positions.

As shown inFIG.5, step S332is executed by the processing unit190, to analyzing the N candidate rotation-translation matrices RT4a˜RT4nin statistics, so as to calculate the fourth rotation-translation matrix RT4according to analyzation of the N candidate rotation-translation matrices RT4a˜RT4n.

In some embodiments, the fourth rotation-translation matrix RT4can be generated according to an average of the N candidate rotation-translation matrices RT4a˜RT4n.

In some other embodiments, the fourth rotation-translation matrix RT4can be generated according to a median of the N candidate rotation-translation matrices RT4a˜RT4n.

In some other embodiments, a standard deviation can be generated according to the N candidate rotation-translation matrices RT4a˜RT4n. Afterward, the standard deviation can be utilized to determine whether each of the candidate rotation-translation matrices RT4a˜RT4nis trustworthy or not. The control method300bcan delete untrustworthy candidates, and calculate the fourth rotation-translation matrix RT4according to trustworthy candidates.

Step S340is executed to store the fourth rotation-translation matrix RT4, the intrinsic parameters and the distortion parameters.

The control method300bshown inFIG.5is able to the fourth rotation-translation matrix RT4according to N different images IMGa˜IMGn during the calibration process. It will be helpful to eliminate a potential error during the geometric camera calibration in S320.

Reference is further made toFIG.7AandFIG.7B.FIG.7AandFIG.7Bare schematic diagrams illustrating the tracking system100with a function to track the optical center C120of the camera120in a normal application after the calibration process. As shown inFIG.7AandFIG.7B, in the normal application (e.g., shooting a video about the object OBJ by the camera120), the camera120may move to different positions in the spatial area SA.

As shown inFIG.7A, the camera120is moved to the left side in the spatial area SA. In the case, the tracking station160of the tracking system100is able to track the first trackable device141to generate a current first rotation-translation matrix RT1x. Based on the current first rotation-translation matrix RT1xand the stored fourth rotation-translation matrix RT4, the tracking system100is able to precisely track the camera120.

As shown inFIG.7B, the camera120is moved to the right side in the spatial area SA. In the case, the tracking station160of the tracking system100is able to track the first trackable device141to generate a current first rotation-translation matrix RT1y. Based on the current first rotation-translation matrix RT1yand the stored fourth rotation-translation matrix RT4, the tracking system100is able to precisely track the camera120.

In this case, if the object OBJ captured by the camera120is merged with a virtual background (e.g., outer space) and some virtual objects (e.g., spaceships and aliens) in an immersive scenario, the view point of the camera120relative to the object OBJ can be precisely tracked, such that the object OBJ will appear to be real in the immersive scenario.

Reference is further made toFIG.8, which is a flow chart illustrating a control method300cperformed by the tracking system100inFIG.7AandFIG.7Bafter the calibration process.

As shown inFIG.7A,FIG.7BandFIG.8, step S351is executed to track the first trackable device141by the tracking station160to generate the current first rotation-translation matrix RT1x/RT1y. Step S352is executed to track the camera120according to the current first rotation-translation matrix RT1x/RT1yand the fourth rotation-translation matrix RT4stored in previous calibration process. Step S353is executed by the camera120to capture images without involving the calibration chart (referring to the calibration chart180shown inFIG.2). For example, the camera120is able to capture images, streaming images and/or videos about different objects (e.g., actors, buildings, animals or landscapes). Step S354is executed to adjust an image frame of the camera120according to the stored intrinsic parameters and the stored distortion parameters.

Another embodiment of the disclosure includes a non-transitory computer-readable storage medium, which stores at least one instruction program executed by a processing unit (referring to the processing unit190shown inFIG.2discussed in aforesaid embodiments) to perform control methods300a,300band300cas shown inFIG.3,FIG.5andFIG.8.

Although the present invention has been described in considerable detail with reference to certain embodiments thereof, other embodiments are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the embodiments contained herein.

It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present invention without departing from the scope or spirit of the invention. In view of the foregoing, it is intended that the present invention cover modifications and variations of this invention provided they fall within the scope of the following claims.