Patent ID: 12223676

DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the present preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.

SeeFIG.2, which shows a schematic diagram of a host according to an embodiment of the disclosure. In various embodiments, the host200can be any electronic device such as a smart device and/or a computer device. In some embodiments, the host200can be an HMD of a reality system (e.g., a VR system, an augmented reality (AR) system, a mixed reality (MR) system, or the like). In some embodiments, the host200can be a (standalone) HMD providing a visual content (e.g., a VR content) for the user/wearer to see, but the disclosure is not limited thereto. In other embodiments, the host200can be an automated guided vehicle or other autonomous robots, but the disclosure is not limited thereto.

InFIG.2, the host200includes a storage circuit202and a processor204. The storage circuit202is one or a combination of a stationary or mobile random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or any other similar device, and which records a program code and/or a plurality of modules that can be executed by the processor204.

The processor204is coupled with the storage circuit202, and the processor204may be, for example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like.

In some embodiments, the host200can be configured to perform tracking functions, such as inside-out tracking and/or outside-in tracking, to track the pose of the host200. In some embodiments, the host200can be disposed with one or more cameras (e.g., the front camera) for capturing the images of the environment where the host200locates. In one embodiment, the host200can track the pose of the host200based on the captured images by performing, for example, simultaneous localization and mapping (SLAM). In some embodiments, the pose of the host200can be characterized by one or more component of the host200on one or more degree of freedom (DOF), such as the so-called 6DOF of the host200, but the disclosure is not limited thereto.

In the embodiments of the disclosure, the processor204accesses the modules and/or the program code stored in the storage circuit202to implement the method for pose correction provided in the disclosure, which would be further discussed in the following.

SeeFIG.3, which shows a flow chart of the method for pose correction according to an embodiment of the disclosure. The method of this embodiment may be executed by the host200inFIG.2, and the details of each step inFIG.3will be described below with the components shown inFIG.2. In addition, for better explaining the concept of the disclosure,FIG.4would be used as an example, whereinFIG.4shows a schematic diagram of an application scenario according to an embodiment of the disclosure.

InFIG.4, the host200(e.g., an HMD) is assumed to move in a specific field499, and the host200can capture images when moving around in the specific field499.

In step S310, the processor204obtains a first image IM1, which may be one of the images captured by the host200.

In one embodiment, the processor204determines whether one or more reference object exists in the first image IM1. In the embodiments of the disclosure, the reference object(s) can be some objects particularly disposed in the environment of the specific field499as a reference for pose correction. In various embodiments, the appearances of the reference objects can be different or the same.

In the scenario ofFIG.4, a first reference object410can be one of the reference objects, which may be implemented as an object printed with specific patterns. In other embodiments, the reference objects can be any object (e.g., furniture, a (electronic) device, decorations, appliance, etc.) predetermined by the designer, but the disclosure is not limited thereto. In some embodiments, the positions and the orientations of the reference objects are fixed in the specific field499.

In one embodiment, for each of the reference objects, a corresponding look-up table can be constructed in advance (such as at an installing phase of the specific field499). For example, a look-up table corresponding to one of the reference objects (referred to as a specific reference object) can be exemplarily shown as the following Table 1.

TABLE 1datareference relative positionreferencepredeterminedcombinationdistanceangleareapose150cm0degree100pixelsPose12100cm30degrees50pixelsPose23150cm60degrees25pixelsPose3. . .. . .. . .. . .. . .

As can be seen in the above, Table 1 includes multiple data combinations, and each data combination includes a reference relative position, a reference area, and a reference predetermined pose. In the embodiments of the disclosure, the reference relative position of each data combination can be characterized by a distance between a reference device and the specific reference object and an angle of the reference device with respect to the specific reference object. In addition, the reference area of each data combination is the area occupied by the specific reference object in an image captured by the reference device when the relative position between the reference device and the specific reference object matches the corresponding reference relative position.

In one embodiment, the reference device can be another HMD used to measure the reference area and the predetermined pose of each data combination. For example, during constructing the data combination 1 of the look-up table of the specific reference object, the reference device can be set to be in the corresponding reference relative position (i.e., spaced from the specific reference object by 50 cm and having an angle of 0 degree with respect to the specific reference object) and capture an image of the specific reference object. In this case, the current pose of the reference device can be recorded as Pose1, and the area occupied by the specific reference object in the captured image can be recorded as the reference area (e.g., 100 pixels) of the data combination 1.

For example, during constructing the data combination 2 of the look-up table of the specific reference object, the reference device can be set to be in the corresponding reference relative position (i.e., spaced from the specific reference object by 100 cm and having an angle of 30 degrees with respect to the specific reference object) and capture an image of the specific reference object. In this case, the current pose of the reference device can be recorded as Pose2, and the area occupied by the specific reference object in the captured image can be recorded as the reference area (e.g., 50 pixels) of the data combination 2. For another example, during constructing the data combination 3 of the look-up table of the specific reference object, the reference device can be set to be in the corresponding reference relative position (i.e., spaced from the specific reference object by 150 cm and having an angle of 60 degrees with respect to the specific reference object) and capture an image of the specific reference object. In this case, the current pose of the reference device can be recorded as Pose3, and the area occupied by the specific reference object in the captured image can be recorded as the reference area (e.g., 25 pixels) of the data combination 3. Other data combinations in Table 1 can be constructed/measured/generated based on the similar principle.

For each of the reference objects, the corresponding look-up table can be constructed in advance based on the above teachings, and so as the first reference object410.

In one embodiment, after obtaining the first image IM1, the processor204may determine whether any of the reference objects exists in the first image IM1. In the scenario ofFIG.4, it is assumed that the first reference object410is determined to be existing in the first image IM1, and hence the processor204can proceed to step S320.

In step S320, in response to determining that the first reference object410exists in the first image IM1, the processor204determines a first relative position RL1between the host200and the first reference object410.

InFIG.4, the first relative position RL1between the host200and the first reference object410is characterized by a first distance D1between the host200and the first reference object410and a first angle A1of the host200with respect to the first reference object410.

In step S330, the processor204obtains a first reference pose based on the first relative position RL1. In one embodiment, the processor204obtains a first look-up table corresponding to the first reference object410, wherein the first look-up table includes a plurality of first data combinations, and each first data combination includes a first reference relative position, a first reference area, and a first predetermined pose.

For better understanding the concept of the disclosure, Table would be assumed to be the first look-up table corresponding to the first reference object410. In this case, the first data combinations may respectively correspond to the data combinations 1, 2, 3, etc.

Next, the processor204obtains a first specific data combination from the first data combinations based on the first relative position RL1, wherein the first reference relative position of the first specific data combination corresponds to the first relative position RL1.

For example, if the processor204determines that the first distance D1is 50 cm and the first angle A1is 0 degree, the processor204may determine that the data combination 1 in Table 1 is the first specific data combination. For another example, if the processor204determines that the first distance D1is 100 cm and the first angle A1is 30 degrees, the processor204may determine that the data combination 2 in Table 1 is the first specific data combination. For yet another example, if the processor204determines that the first distance D1is 150 cm and the first angle A1is 60 degrees, the processor204may determine that the data combination 3 in Table 1 is the first specific data combination, but the disclosure is not limited thereto.

Afterwards, the processor204determines the first predetermined pose of the first specific data combination as the first reference pose. For example, if the processor204determines that the data combination 1 in Table 1 is the first specific data combination, the processor204can correspondingly determine that Pose1 (i.e., the predetermined pose of the data combination 1) in Table 1 is the first reference pose. For another example, if the processor204determines that the data combination 2 in Table 1 is the first specific data combination, the processor204can correspondingly determine that Pose2 (i.e., the predetermined pose of the data combination 2) in Table 1 is the first reference pose. For yet another example, if the processor204determines that the data combination 3 in Table 1 is the first specific data combination, the processor204can correspondingly determine that Pose3 (i.e., the predetermined pose of the data combination 3) in Table 1 is the first reference pose, but the disclosure is not limited thereto.

In step S340, the processor204corrects the pose of the host200based on the first reference pose. In one embodiment, the processor204obtains at least one difference between the pose of the host200and the first reference pose and compensates the pose of the host200by the at least one difference.

In one embodiment, the at least one difference is characterized by at least one component difference on at least one DOF between the pose of the host200and the first reference pose.

For example, if the 6DOF of the pose of the host200is represented by (x-a, y-b, z-c, φ-d, θ-e, ψ-f) and the 6DOF of the first reference pose is represented by (x, y, z, φ, θ, ψ), the processor204may determine the difference between the pose of the host200and the first reference pose as (a, b, c, d, e, f). In this case, the processor204may correct the pose of the host200via compensating the pose of the host200by the (a, b, c, d, e, f), such that the compensated/corrected pose of the host200becomes (x, y, z, ψ, θ, ψ), but the disclosure is not limited thereto.

Accordingly, the embodiments of the disclosure can correct the pose of the host200when a reference object is detected in the captured image. In this case, when the host200moves around in the specific field499, the pose of the host200can be corrected from time to time, and hence the issue of pose drifting error accumulation mentioned in the above can be mitigated.

In some embodiments, the specific field499where the host200moves can be distributed with multiple identical/different reference objects, and each of the reference object can be designed with a corresponding look-up table. In this case, whenever one reference object is detected by the host200in the captured image, the host200can obtain the relative position between the host200and this reference object and accordingly search for the corresponding predetermined pose in the look-up table of this reference object. Once the predetermined pose (e.g., the first reference pose) corresponding to the relative position between the host200and this reference object is obtained, the host200can correct the pose of the host200based on this predetermined pose.

In one embodiment, the processor204can further determine whether the pose of the host200needs to be corrected. SeeFIG.5A, which shows a flow chart of the method for pose correction according toFIG.3.

InFIG.5A, after the processor204performs steps S310and S320, the processor204can perform step S510to determine, based on the first relative position RL1, whether the pose of the host200needs to be corrected. If yes, the processor204may proceed to steps S330and S340as taught in the above. If not, the processor204may proceed to step S520to not correct the pose of the host200.

SeeFIG.5B, which shows a flow chart of determining whether the pose of the host needs to be corrected according toFIG.5A. For better understanding, the scenario inFIG.4would be used as an example again.

In step S511, the processor204obtains the first look-up table (e.g., Table 1) corresponding to the first reference object410. In step S512, the processor204obtains the first specific data combination from the first data combinations based on the first relative position RL1. Details of steps S511and S512can be referred to the above embodiments, which would not be repeated herein.

For facilitating the following discussion, the data combination 1 would be assumed to be the first specific data combination obtained in step S512, but the disclosure is not limited thereto. In this case, the first reference area of the first specific data combination would be 100 pixels according to Table 1.

In step S513, the processor204determines a first area corresponding to the first reference object410in the first image IM1. In one embodiment, the first area corresponding to the first reference object410in the first image IM1is an area occupied by the first reference object410in the first image IM1. InFIG.4, assuming that the width and height of the first reference object410in the first image IM1is W and H, the processor204may determine that the first area as W×H pixels. For example, if W and H are both 10 pixels, the processor204may determine the first area as 100 pixels.

In other embodiments, the area occupied by the first reference object410in the first image IM1may be irregularly shaped, and the processor204may accordingly count the pixels in the area occupied by the first reference object410in the first image IM1to obtain the first area, but the disclosure is not limited thereto.

In step S514, the processor204determines whether the first area corresponds to the first reference area of the first specific data combination. If not, the processor204can proceed to step S515to determine that the pose of the host200needs to be corrected; if yes, the processor204can proceed to step S516to determine that the pose of the host200does not need to be corrected.

In one embodiment, the processor204can determine whether the first area is equal to the first reference area. If yes, the processor204can determine that the first area corresponds to the first reference area, or otherwise the processor204can determine that the first area does not correspond to the first reference area.

For example, if the first area is 100 pixels, the processor204may determine that the first area corresponds to the first reference area since the first area is equal to the considered first reference area (i.e., 100 pixels). For another example, if the first area is not 100 pixels, the processor204may determine that the first area does not correspond to the first reference area since the first area is not equal to the considered first reference area (i.e., 100 pixels), but the disclosure is not limited thereto.

In another embodiment, the processor204may determine that an area difference between the first area and the first reference area is larger than a threshold. If not, the processor204can determine that the first area corresponds to the first reference area, or otherwise the processor204can determine that the first area does not correspond to the first reference area.

For example, if the first area is 96 pixels and the threshold is 5 pixels, the processor204may determine that the first area corresponds to the first reference area since the area difference (i.e., 4 pixels) between the first area and the considered first reference area (i.e., 100 pixels) is not larger than the threshold. For another example, if the first area is 94 pixels and the threshold is 5 pixels, the processor204may determine that the first area does not correspond to the first reference area since the area difference (i.e., 6 pixels) between the first area and the considered first reference area (i.e., 100 pixels) is larger than the threshold, but the disclosure is not limited thereto.

After step S515, the processor204may perform steps S330and S340, and the details of steps S330and S340can be referred to the above teachings.

On the other hand, after step S516, the processor204can proceed to step S520to not correct the pose of the host200.

In one embodiment, the processor204may determine whether any of the reference objects exists in each of the images captured by the camera.

In other embodiments, the processor204may determine to selectively detect whether the reference objects exist in the images.

For example, after processing/analysing the first image IM1, the processor204may determine a time difference and accordingly obtain a second image, wherein the second image is captured later than the first image by the time difference. That is, if the first image IM1is captured at a timing point T1, the processor204may determine the image captured at a timing point T2 as the second image, wherein the timing point T2 is later than the timing point T1 by the time difference.

With the second image, the processor204may determine whether the second image includes the reference objects.

In one embodiment, in response to determining that at least one image is captured between the first image IM1and the second image, the processor204may not determine whether the reference objects exist in each image between the first image IM1and the second image. That is, the processor204may skip the detection for the reference objects in the images captured between the first image IM1and the second image. Accordingly, the computation resources of the host200can be reduced.

In various embodiments, the time difference between the first image IM1and the second image can be determined in different ways.

In one embodiment, the time difference between the first image IM1and the second image can be determined to be fixed. That is, the processor204can periodically perform the detection for the reference objects. For example, the processor204may determine whether the reference objects exist in each of the first image IM1, the second image, a third image, a fourth image, and so on, wherein the third image and the fourth image are respectively captured at timing points T3 and T4. The time difference between the timing point T3 and T2 and the time difference between the timing point T4 and T3 are the same as the time difference between the timing points T2 and T1.

In one embodiment, the processor204can determine the time difference between the first image IM1and the second image based on a distribution of the reference objects in the specific field499where the host200moves.

For example, if the reference objects are sparsely distributed in the specific field499, the processor204may determine the time difference as a larger value since the reference objects are less possible to be detected in images that are close in time. On the other hand, if the reference objects are densely distributed in the specific field499, the processor204may determine the time difference as a smaller value since the reference objects are possible to be detected in images that are close in time.

From another perspective, the time difference between the first image IM1and the second image may be positively related to the (average) distances between the reference objects in the specific field499.

In one embodiment, the processor204may determine the time difference between the first image IM1and the second image based on a moving speed of the host200in the specific field499where the host200moves.

For example, if the host200moves slowly in the specific field499, the processor204may determine the time difference as a larger value since the reference objects are less possible to be detected in images that are close in time. On the other hand, if the host200moves fast in the specific field499, the processor204may determine the time difference as a smaller value since the reference objects are possible to be detected in images that are close in time.

From another perspective, the time difference between the first image IM1and the second image may be negatively related to the moving speed of the host200in the specific field499.

In one embodiment, the processor204may determine the time difference between the first image IM1and the second image based on a predicted moving trace of the host200. Specifically, the moving trace of the host200can be predicted based on the historical moving traces, and when the predicted moving trace of the host200is close to some of the reference objects in the specific field499, the processor204may determine the time difference as a smaller value since the reference objects are possible to be detected in the following images.

On the other hand, when the predicted moving trace of the host200is far from any of the reference objects in the specific field499, the processor204may determine the time difference as a larger value since the reference objects are less possible to be detected in the following images, but the disclosure is not limited thereto.

In one embodiment, in response to determining that the first reference object410also exists in the second image, the processor204can determine the first relative position RL1between the host200and the first reference object410. Afterwards, the processor204subsequently perform steps S330and S340, and the details thereof can be referred to the above embodiments.

In other embodiments, in response to determining that other reference object exists in the second image, the processor204can accordingly correct the pose of the host200based on the above teachings.

In some embodiments where two or more reference objects exist in the first image IM1, the processor204can determine to correct the pose of the host200based on which of the reference object. Detailed discussions would be provided in the following.

SeeFIG.6, which shows a flow chart of the method for pose correction according to an embodiment of the disclosure. For better understanding,FIG.7would be used as an example, whereinFIG.7is a schematic diagram of an application scenario according to an embodiment of the disclosure.

In the embodiment, after the processor204performs steps S310to S330, the processor204can further determine whether other reference object(s) also exists in the first image IM1before performing step S340.

In step S610, in response to determining that a second reference object420also exists in the first image IM1, the processor204determines a second relative position RL2between the host200and the second reference object420.

InFIG.7, the second relative position RL2between the host200and the second reference object420is characterized by a second distance D2between the host200and the second reference object420and a second angle A2of the host200with respect to the second reference object420.

In step S620, the processor204obtains a second reference pose based on the second relative position RL2. In one embodiment, the processor204obtains a second look-up table corresponding to the second reference object420. As mentioned in the above, each reference object may have the corresponding look-up table constructed in advance, and the second look-up table may be understood as the corresponding look-up table of the second reference object420. In the embodiment, the second look-up table includes a plurality of second data combinations, and each second data combination comprises a second reference relative position, a second reference area, and a second predetermined pose, wherein the form of the second look-up table can be similar to Table 1. For better understanding, Table 1 would be used as an example of the second look-up table, but the disclosure is not limited thereto.

Next, the processor204obtains a second specific data combination from the second data combinations based on the second relative position RL2, wherein the second reference relative position of the second specific data combination corresponds to the second relative position.

For example, if the processor204determines that the second distance D2is 50 cm and the second angle A2is 0 degree, the processor204may determine that the data combination 1 in Table 1 is the second specific data combination. For another example, if the processor204determines that the second distance D2is 100 cm and the second angle A2is 30 degrees, the processor204may determine that the data combination 2 in Table 1 is the second specific data combination. For yet another example, if the processor204determines that the second distance D2is 150 cm and the second angle A2is 60 degrees, the processor204may determine that the data combination 3 in Table 1 is the second specific data combination, but the disclosure is not limited thereto.

Afterwards, the processor204determines the second predetermined pose of the second data combination as the second reference pose. For example, if the processor204determines that the data combination 1 in Table 1 is the second specific data combination, the processor204can correspondingly determine that Pose1 (i.e., the predetermined pose of the data combination 1) in Table 1 is the second reference pose. For another example, if the processor204determines that the data combination 2 in Table 1 is the second specific data combination, the processor204can correspondingly determine that Pose2 (i.e., the predetermined pose of the data combination 2) in Table 1 is the second reference pose. For yet another example, if the processor204determines that the data combination 3 in Table 1 is the second specific data combination, the processor204can correspondingly determine that Pose3 (i.e., the predetermined pose of the data combination 3) in Table 1 is the second reference pose, but the disclosure is not limited thereto.

In step S630, the processor204determining, based on the first relative position RL1and the second relative position RL2, to use the first reference pose or the second reference pose to correct the pose of the host200.

In one embodiment, the processor204may determine whether the first distance D1is smaller than the second distance D2. In response to determining that the first distance D1is smaller than the second distance D2, the processor204determines to use the first reference pose to correct the pose of the host200. On the other hand, in response to determining that the first distance D1is not smaller than the second distance D2, the processor204determines to use the second reference pose to correct the pose of the host200.

In another embodiment, the processor204may determine determining a second area corresponding to the second reference object420in the first image IM1, wherein the second area corresponding to the second reference object420in the first image IM1is the an area occupied by the second reference object420in the first image IM1. Next, the processor204may determine whether the first area is larger than the second area. In one embodiment, in response to determining that the first area is larger than the second area, the processor204determines to use the first reference pose to correct the pose of the host200. On the other hand, in response to determining that the first area is not larger than the second area, the processor204determines to use the second reference pose to correct the pose of the host200.

InFIG.6, after determining to use the first reference pose to correct the pose of the host200, the processor204performs step S640to correct the pose of the host200based on the first reference pose and neglecting the second reference pose. The details of correcting the pose of the host200based on the first reference pose can be referred to the above embodiments, which would not be repeated herein.

On the other hand, after determining to use the second reference pose to correct the pose of the host200, the processor204performs step S650to correct the pose of the host based on the second reference pose and neglecting the first reference pose. The details of correcting the pose of the host200based on the second reference pose is similar to the way of correcting the pose of the host200based on the first reference pose, and hence the details thereof would not be repeated herein.

SeeFIG.8, which shows an application scenario according to an embodiment of the disclosure. InFIG.8, assuming that the host200moves along the actual path. During the process of the host200moving from Point 0 to Point 1, the pose drifting error of the pose of the host200may be gradually accumulated, such that the estimated path of the host200gradually deviates from the actual path. However, when the host200reaches Point 1 and determines that the reference object811is detected in the captured image, the pose of the host200can be corrected based on the reference object811, such that the estimated path of the host200can be corrected to fit the actual path at Point 1.

During the process of the host200moving from Point 1 to Point 2, the pose drifting error of the pose of the host200may be gradually accumulated, such that the estimated path of the host200gradually deviates from the actual path. However, when the host200reaches Point 2 and determines that the reference object812is detected in the captured image, the pose of the host200can be corrected based on the reference object812, such that the estimated path of the host200can be corrected to fit the actual path at Point 2.

During the process of the host200moving from Point 2 to Point 3, the pose drifting error of the pose of the host200may be gradually accumulated, such that the estimated path of the host200gradually deviates from the actual path. However, when the host200reaches Point 3 and determines that the reference object813is detected in the captured image, the pose of the host200can be corrected based on the reference object813, such that the estimated path of the host200can be corrected to fit the actual path at Point 3.

During the process of the host200moving from Point 3 to Point 4, the pose drifting error of the pose of the host200may be gradually accumulated, such that the estimated path of the host200gradually deviates from the actual path. However, when the host200reaches Point 4 and determines that the reference object814is detected in the captured image, the pose of the host200can be corrected based on the reference object814, such that the estimated path of the host200can be corrected to fit the actual path at Point 4.

In summary, the embodiments of the disclosure can correct the pose of the host when a reference object is detected in the captured image. In this case, when the host moves around in the specific field, the pose of the host can be corrected from time to time, and hence the issue of pose drifting error accumulation mentioned in the above can be mitigated.

It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present invention without departing from the scope or spirit of the disclosure. In view of the foregoing, it is intended that the present disclosure cover modifications and variations of this invention provided they fall within the scope of the following claims and their equivalents.