Systems and methods for object tracking using fused data

In one embodiment, a method includes capturing, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment, identifying one or more features from the first image to estimate a pose of the hand of the user, estimating a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller, receiving IMU data of the controller, and estimating a second pose of the controller by updating the first pose of the controller using the IMU data of the controller. The method utilizes multiple data sources to track the controller under various conditions of the environment to provide an accurate controller tracking consistently.

TECHNICAL FIELD

This disclosure generally relates to an object tracking, and more specifically methods, apparatus, and system for an object tracking based on a fusion of feature estimation and sensor data.

BACKGROUND

Input instructions provided to AR/VR devices is typically based on controller tracking or hand tracking. A controller can be tracked using the known patterns formed by infrared (IR) light emitting diodes (LEDs) on the controller and input an instruction in a specific location in an environment via the button on the controller. An input instruction can also be made by a hand gesture by tracking features of the hand. For example, a user can turn a page of a virtual book by tracking a swipe gesture of the hand. However, controller tracking is more costly because of the additional hardware required, e.g., IR cameras and IR LED lights on the controller which could sometimes be interfered by occlusions or other light sources, and hand tracking is less accurate.

SUMMARY OF PARTICULAR EMBODIMENTS

To address the foregoing problems, disclosed are methods, apparatuses, and a system, to track a controller by estimating a grip of a hand and adjusting the grip of the hand based on inertial measurement unit (IMU) data from the controller. The present disclosure provides a method to track a controller without implementing LEDs in the controller (e.g., without tracking a pattern of LED lights), so that the method disclosed in the present application provides a cost-efficient, accurate way to track the controller. The method disclosed in the present application may estimate a grip of a user's hand based on feature-tracking identified from captured images of the user's hand and then estimate a pose of the controller using the estimated grip of the user's hand. Furthermore, the method of the present application may receive IMU data of the controller to adjust the estimated pose of the controller and provide a final pose of the controller at a faster frequency.

The embodiments disclosed herein are only examples, and the scope of this disclosure is not limited to them. Particular embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed herein. According to one embodiment of a method, the method comprises, by a computing system, capturing, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment. The method further comprises identifying one or more features from the first image to estimate a pose of the hand of the user. The method yet further comprises estimating a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller. The method further comprises receiving IMU data of the controller. The method further comprises estimating a second pose of the controller by updating the first pose of the controller using the IMU data of the controller.

Certain aspects of the present disclosure and their embodiments may provide solutions to these or other challenges. There are, proposed herein, various embodiments which address one or more of the issues disclosed herein. The methods disclosed in the present disclosure may provide a tracking method for a controller, which estimates and adjusts the pose of the controller based on the estimation of the grip and the IMU data of the controller. Furthermore, based on the pose of the controller relative to the environment and the user's hand, the method disclosed in the present application may also provide an IMU-predicted pose of the user's hand to reduce a search range of the user's hand in a next frame. Therefore, particular embodiments disclosed in the present application may track the controller cost-efficiently (e.g., no needs to install LEDs) and improve the process time to perform tracking tasks.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Current AR/VR devices are commonly paired with a portable/wearable device (e.g., a controller) to provide the user an easy, intuitive way to input instructions for the AR/VR devices. The controller is usually equipped with at least one inertial measurement unit (IMU) and infrared (IR) light emitting diodes (LEDs) for the AR/VR devices to estimate a pose of the controller and/or to track a location of the controller, such that the user may perform certain functions via the controller. For example, the user may use the controller to display a visual object in a corner of the room. However, equipping LEDs increases the cost of manufacturing the controller, and tracking the controller via determining a pattern of LED lights could be interfered under certain environment conditions. Also, purely relying on feature-tracking to track a controller could be inaccurate. Particular embodiments disclosed in the present disclosure provide a method to estimate a pose of the controller by fusing feature-tracking data of the user's hand and IMU data of the controller.

Furthermore, particular embodiments disclosed in the present disclosure may provide an IMU-predicted pose of the user's hand based on the fusion of the estimated grip of the hand and the IMU data of the controller to facilitate hand-tracking in a next frame. Utilizing the IMU data of the controller to adjust the grip of the hand can update the pose of the controller more frequently to keep an efficient, accurate tracking. Particular embodiments disclosed in the present disclosure may be applied to any kind of tracking system, such as visual inertial odometry (VIO)-based simultaneous localization and mapping (SLAM) tracking system, with efficiency and less cost.

FIGS. 1A-1Billustrate an example tracking system for tracking a controller, in accordance with certain embodiments. InFIG. 1A, the tracking system100comprises a central module (not shown) and a controller module110(e.g., a controller). The central module comprises a camera and at least one processor to track the controller module110in an environment. In particular embodiments, the central module may be implemented in a wearable device, such as a head-mounted device, to capture an image of an object to be tracked (e.g., a controller implemented with the controller module110). For example, the wearable device with the camera may perform an inside-out tracking (e.g., SLAM) for an object. In particular embodiments, the object to be tracked may also be tracked by one or more cameras implemented/fixed in the environment, e.g., an outside-in tracking.

The camera of the central module may capture a first frame120depicting at least part of a user's hand. More specifically, the first frame120depicts at least a part of the user's hand holding the controller module110. The central module may identify one or more features122,124,126of at least part of a user's hand from the first frame120. In particular embodiments, the first frame120may comprise one or more feature at least depicting the user's hand holding the controller module110. InFIG. 1B, the controller module110comprises a handle112for a user to hold. The central module identifies the features122,124,126of a user's hand which may be used to estimate a pose of the user's hand. For example, an area122where the purlicue of the hand overlaps with the controller110, the ulnar border of the hand124where represents a user's hand holding the controller110, and an area126including the finger tips and the controller110. The identified features122,124,126from the first frame120may be used to estimate a pose/location of the user's hand. Furthermore, the pose of the user's hand may be used to estimate a grip of the user's hand. For example, the pose of the user's hand may be a skeleton/a primary geometry of the user's hand representing a hand gesture of the user. The estimated grip of the user's hand may be utilized to estimate a pose of a controller module110based on the estimated grip of the user's hand which defines a relative pose between the hand of the user and the controller module110.

The controller module110comprises at least one IMU, such that the controller module110may provide IMU data to the central module to update/adjust the estimated pose of the controller module110. The controller module110may provide the IMU data at a frequency which is faster than a frequency of the central module taking a frame of the user and the controller module110. For example, the central module may capture a second frame130of the user holding the controller module110and identify the features122,124,126or any other potential features which can be used to estimate the pose of the user's hand from the second frame130. Before the central module estimates an updated pose of the user's hand based on the identified features in the second frame130, the central module may use the received IMU data of the controller module110to adjust the estimated pose of the controller module110which was estimated based on the grip of the hand estimated from the first frame120. In particular embodiments, the central module may provide/update a pose of the user's hand at a frequency of 30 Hz (e.g., based on captured frames) for estimating a pose of the controller module110, and the controller module110may provide the IMU data at a frequency of 500 Hz to the central module for updating the estimated pose of the controller module110, such that the pose of the controller module110can be tracked/adjusted at a faster frequency based on the IMU data of the controller module110to keep the accuracy and efficiency of tracking the controller module110. In particular embodiments, the central module may output the pose of the controller based on either tracking result (e.g., feature tracking or IMU tracking) as needed.

In particular embodiments, the captured frames may be a visible-light image which is identified to comprise at least one feature which can be used to estimate a pose of the user's hand. The visible-light image may be an RGB image, a CMYK image, a greyscale image, or any suitable image for estimating a pose of the user's hand. In particular embodiments, the identified features122,124,126from the captured frames120,130are configured to be accurately tracked by a camera of the central module to determine a motion, orientation, and/or spatial position of the controller module110(e.g., correspondence data of the controller module110) for reproduction in a virtual/augmented environment. In particular embodiments, the estimated pose of the controller module110may be adjusted by a spatial movement (X-Y-Z positioning movement) determined based on the identified features122,124,126between frames (e.g., the first frame120and the second frame130). For example, the central module may determine an updated spatial position of the user's hand in a frame k+1, e.g., a frame captured during operation, and compare it with a previous spatial position of the user's hand in a frame k, e.g., a frame captured previously or stored in a storage, to readjust the pose of the user's hand. Detailed operations and actions performed at the central module for tracking the controller module may be further described inFIGS. 2 to 5.

FIG. 2illustrates an example tracking system200comprising a central module and a controller module, in accordance with certain embodiments. The tracking system200comprises a central module implemented in a headset which is worn by a user, and a controller module implemented in a controller which is held by the user. In particular embodiments, the user may have two controllers paired with the headset for each hand. The headset comprises at least one camera, at least one IMU, and at least one processor which is configured to process instructions for tracking a controller. Furthermore, the controller comprises at least one IMU which is configured to provide IMU data of the controller to the central module of the headset, and at least one processer which is configured to process instructions/calibrations sent from the headset.

The camera of the headset captures one or more image of the user and the controller202in an environment and identifies one or more features of the user's hand from the image202for hand tracking204via machine learning or deep learning. Based on the identified features which can be used to estimate/determine a pose of the user's hand, the processor of the headset may estimate a pose of the user's hand and/or a location of the user's hand based on the identified features. In particular embodiments, the pose of the user's hand may be estimated based on repeated feature identified over a series of images. In particular embodiments, the processor of the headset may estimate a pose of the user's hand relative to the environment206based on the results of hand tracking204.

In particular embodiments, the IMU of the headset208may also provide IMU data of the headset to the processor of the headset, and the processor of the headset may estimate a pose of the headset relative to the environment212via inside-out tracking210based on the IMU data of the headset. In particular embodiments, the processor of the headset may estimate a pose of the headset relative to the environment212via inside-out tracking210based on the IMU data of the headset and the camera image202. For example, the IMU data of the headset may provide information of angler velocity, acceleration, and motion of the headset to calculate a pose of the headset in the environment. Furthermore, the processor of the headset may utilize the pose of the headset relative to the environment212to facilitate the hand tracking204. For example, the pose of the headset relative to environment212may be fed to facilitate the hand tracking204by comparing a pose/location of the headset relative to the environment212with the image of the user and the controller202in the environment to adjust/estimate the pose of the user's hand.

The processor of the headset may then estimate a grip of the user's hand214based on the estimated pose of the user's hand206and estimate a pose of the controller relative to the environment216based on the estimated grip of the user's hand214. For example, the processor of the headset may use the pose of the user's hand (including the identified features from the user's hand) to estimate the user's hand representing a gesture of holding the controller, such that, based on an inverse of the gesture/pose of the user's hand, the processor of the headset may generate a pose of the controller.

Furthermore, the IMU of the controller provides IMU data of the controller220to the headset for data fusion218to adjust the pose of the controller estimated based on the grip of the user's hand. The data fusion unit218may utilize the IMU data to calculate an IMU-predicted pose of the controller unit222. The IMU-predicted pose of the controller unit222may be utilized by the grip estimator unit214to adjust the pose of the controller relative to the environment and estimate an inverse grip of the user's hand214, where the inverse grip infers the pose of the user's hand214based on the pose of the adjusted pose of the controller. In particular embodiments, the final pose of the controller224may be provided based on the operations/needs of the headset. For example, the final pose of the controller224may be estimated in-between two captured frames (e.g., before the next estimation of the grip). On the other hand, the final pose of the controller224may also be estimated based on the IMU-adjusted grip, e.g., the estimated grip adjusted by the received IMU data of the controller. The processor of the headset may estimate the final pose of the controller224at a certain frequency based on a request or a demand to save power.

In addition, based on the data provided by the data fusion218, the processor of the headset may provide an IMU-predicted pose of the hand226based on the IMU-predicted pose of the controller222and use the IMU-predicted pose of the hand226to facilitate the hand tracking204. For example, the IMU-predicted pose of the controller222can be provided at a faster frequency (e.g., 500 Hz to 1 kHz) to fill in the gap between two frames. By applying the inverse grip estimation to the IMU-predicted pose of the controller222, the headset can generate an IMU-predicted pose of the hand226. The IMU-predicted pose of the hand226can be used to reduce a search range of the hand in the next frame to improve process time of the hand tracking204.

FIG. 3illustrates an example diagram of a tracking system300comprising a central module310and a controller module340, in accordance with certain embodiments. The central module310comprises a camera312, an IMU314, a hand and headset tracking unit316, and a controller tracking unit318to perform a tracking/adjustment for the controller module340in an environment. The central module310is paired with the controller module340to perform certain functions via the controller module340. The controller module340comprises at least one IMU342configured to provide IMU data344for the central module310to track the controller module340. In particular embodiments, the controller module340sends the IMU data344to the controller tracking unit318for computing predictions of a corresponding module, e.g., correspondence data of the controller module340. In particular embodiments, the central module340measures the pose of the controller module340at a frequency from 500 Hz to 1 kHz based on the IMU data344of the controller module340.

In order to generate/estimate a pose of the controller module340during operation, the camera312of the central module310may capture an image or a series of images320when the controller module340is within a field of view (FOV) range of the camera for tracking the controller module340. In particular embodiments, the image320depict at least a part of the user's hand holding the controller module340. The camera312of the central module310sends the image320to the hand/headset tracking unit316for an estimation of a pose of the user's hand based on features identified from the images320.

The hand/headset tracking unit316identifies one or more features of the user's hand from the image320via machine learning, deep learning, or any suitable computing methods. Based on the identified features which can be used to estimate/determine a pose of the user's hand324, the hand/headset tracking unit316of the central module310estimates a pose of the user's hand324and/or a location of the user's hand in the environment based on the identified features of the user's hand. In particular embodiments, the pose of the user's hand324may be estimated based on repeated feature identified over a series of images. The hand/headset tracking unit316of the central module310estimates the pose of the user's hand324at a frequency based on a processing capability or a requirement. In particular embodiments, the hand/headset tracking unit316of the central module310estimates the pose of the user's hand324at a frequency of 30 Hz.

In particular embodiments, the IMU314of the central module310also sends IMU data322to the hand/headset tracking unit316to facilitate the estimation of the pose of the headset. For example, the hand/headset tracking unit316may perform an inside-out tracking to estimate a pose of the central module310. Based on the image320(including the controller module340in the environment) and the IMU data322of the central module316, the hand/headset tracking unit316of the central module310may estimate the pose of the central module310, so that the estimated pose of the user's hand324(estimated based the images320) may be adjusted by the pose of the central module310(e.g., the location of the central module310relative to the user's hand in the environment).

The hand/headset tracking unit316of the central unit310sends the pose of the user's hand324to the controller tracking unit318for controller tracking. The controller tracking unit318comprises a grip estimation unit326configured to estimate a grip of the user's hand and a data fusion unit328configured to fuse/integrate data sent from the grip estimation unit326and data sent from the controller module340.

The grip estimation unit326of the controller tracking unit318receives the pose of the user's hand324from the hand/headset tracking unit316and estimates a grip of the user's hand based on the pose of the user's hand324. Furthermore, the grip estimation unit326estimates a pose of the controller module340based on the grip of the user's hand. For example, the pose of the user's hand324may reveal a gesture of the user holding the controller module340. Therefore, based on the pose of the user's hand324, the grip estimation unit326may estimate the grip of the user's hand and then estimate the pose of the controller module relative to the environment330based on the grip of the user's hand that defines a relative pose between the user's hand and the controller module340. Furthermore, the grip estimation unit326sends the pose of the controller relative to the environment330to the data fusion unit328.

The data fusion unit328of the controller tracking unit318receives the pose of the controller relative to the environment330from the grip estimation unit326of the controller tracking module318in the central module310, and further receives the IMU data344from the controller module340. The data fusion unit328may integrate the pose of the controller module relative to the environment330with the IMU data344of the controller module340to output an adjusted/final pose of the controller module for the central module310to perform a corresponding instruction accurately via the controller module340. In particular embodiments, the data fusion unit328may output the adjusted pose of controller module at a frequency based on the request or the processing speed of the central module310. In particular embodiments, the data fusion unit328may output the adjusted pose of the controller module at a frequency which is faster than the frequency of estimating the pose of the user's hand, such as 30 Hz, since the data fusion unit328can update the pose of the controller module330sent from the grip estimation unit326when it receives the IMU data344from the controller module340.

Furthermore, the data fusion unit328may also provide an IMU-predicted pose of the controller unit332based on the IMU data344of the controller module340to the grip estimation unit326, such that the grip estimation unit326may adjust the pose of the controller module340estimated based on the captured frames. The grip estimation unit326may provide an IMU-predicted pose of the user's hand346based on the IMU data344of the controller module340to the hand tracking unit316to facilitate the process of hand tracking. With the IMU-predicted pose of the user's hand346, the hand tracking unit316may identify features of the user's hand within a predicted range in a next captured frame, so that the hand tracking unit316may complete the hand tracking with less process time.

Furthermore, the central module310may also utilize these captured images320including identified features to conduct extensive services and functions, such as generating a state of the user/the controller module340, locating the user/the controller module340locally or globally, and/or rendering a virtual tag/object in the environment via the controller module340. In particular embodiments, the central module310may also use the IMU data322in assistance of generating the state of the user. In particular embodiments, the central module310may use the state information of the user relative to the controller module340in the environment based on the captured images320, to project a virtual object in the environment or set a virtual tag in a map via the controller module340.

In particular embodiments, the tracking system300may be implemented in any suitable computing device, such as, for example, a personal computer, a laptop computer, a cellular telephone, a smartphone, a tablet computer, an augmented/virtual reality device, a head-mounted device, a portable smart device, a wearable smart device, or any suitable device which is compatible with the tracking system300. In the present disclosure, a user which is being tracked and localized by the tracking device may be referred to a device mounted on a movable object, such as a vehicle, or a device attached to a person. In the present disclosure, a user may be an individual (human user), an entity (e.g., an enterprise, business, or third-party application), or a group (e.g., of individuals or entities) that interacts or communicates with the tracking system300. In particular embodiments, the central module310may be implemented in a head-mounted device, and the controller module340may be implemented in a remote controller separated from the head-mounted device. The head-mounted device comprises one or more processors configured to implement the camera312, the IMU314, the hand/headset tracking unit316, and the controller unit318of the central module310. In one embodiment, each of the processors is configured to implement the camera312, the IMU314, the hand/headset tracking unit316, and the controller unit318separately. The remote controller comprises one or more processors configured to implement the IMU342of the controller module340. In one embodiment, each of the processors is configured to implement the IMU342separately.

This disclosure contemplates any suitable network to connect each element in the tracking system300or to connect the tracking system300with other systems. As an example and not by way of limitation, one or more portions of network may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these. Network may include one or more networks.

FIG. 4illustrates an example diagram of a tracking system400with mapping service, in accordance with certain embodiments. The tracking system400comprises a controller module410, a central module420, and a cloud430. The controller module410comprises at least one IMU412, and a processor414. The controller module410receives one or more instructions442from the central module420to perform specific functions. The controller module410is configured to send IMU data440to the central module420for a pose estimation during operation, so that the central module420may perform the instructions442via the controller module410accurately in a map or in the environment.

The central module420comprises a camera422, at least one IMU424, a hand tracking unit426, and a controller tracking unit428. The central module420is configured to track the controller module410based on various methods, e.g., the method disclosed inFIG. 1AthroughFIG. 3. The camera422of the central module420may capture one or more frames of the controller module410being held by a user, and the IMU424of the central module420may provide IMU data of the central module420to the hand tracking unit426. The hand tracking unit426may identify features from the captured frames via machine learning to estimate a pose of the user's hand and adjust the pose of the user's hand based on the IMU data of the central module420. Furthermore, the hand tracking unit426sends the pose of the user's hand to the controller tracking unit428to estimate a pose of the controller module410. The controller tracking unit428receives the pose of the user's hand and the IMU data440of the controller module410and estimates the pose of the controller module410by fusing the received data.

In particular embodiments, the controller tracking unit428may determine correspondence data based on the features identified in different frames. The correspondence data may comprise observations and measurements of the feature, such as a location of the feature of the controller module410in the environment. Furthermore, the controller tracking unit428may also perform a stereo computation collected near the predetermined feature to provide additional information for the central module420to track the controller module410. In addition, the controller tracking unit428of the central module420may request a live map from the cloud430corresponding to the correspondence data. In particular embodiments, the live map may comprise map data444. The controller tracking unit428of the central module420may also request a remote relocalization service444for the controller module410to be located in the live map locally or globally. In particular embodiments, the pose of the controller module410relative to the environment may be built based on the frames captured by the camera422, e.g., a map built locally. In particular embodiments, the controller tracking unit428of the central module420may also send the correspondence data of the controller module410to the cloud430for an update of the map stored in the cloud430(e.g., with the environment built locally).

FIG. 5illustrates an example method500for tracking a controller, in accordance with certain embodiments. A controller module of a tracking system may be implemented in a portable device (e.g., a remote controller with input buttons, a smart puck with touchpad, etc.). A central module of the tracking system may be implemented in a wearable device (e.g., a head-mounted device, etc.), or be provided to or displayed on any computing system (e.g., an end user's device, such as a smartphone, virtual reality system, gaming system, etc.), and be paired with the controller module. The method500may begin at step510with capturing, using a camera, a first image depicting at least a part of a hand of the user holding a controller in an environment. In particular embodiments, the camera may be one or more cameras implemented in a wearable device worn by a user. In particular embodiments, the wearable device may be a controller. In particular embodiments, the wearable device may be equipped with one or more IMUs.

At step520, the method500may identify one or more features from the first image to estimate a pose of the hand of the user. In particular embodiments, the method500may further receive IMU data of the wearable device to estimate a pose of the wearable device and update the pose of the hand of the user based on the pose of the wearable device. Furthermore, the pose of the wearable device is estimated based on the IMU data of the wearable device and the first image of the user.

At step530, the method500may estimate a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller.

At step540, the method500may receive IMU data of the controller. In particular embodiments, the IMU data of the controller may be received at a faster frequency than a frequency that the first image is captured. For example, the first image may be captured at a first frequency and the IMU data of the controller may be received at a second frequency. The second frequency (e.g., 500 Hz) is higher than the first frequency (e.g., 30 Hz).

At step550, the method500may estimate a second pose of the controller by updating the first pose of the controller using the IMU data of the controller. In particular embodiment, the method500may estimate an IMU-predicted pose of the hand based on the updated first pose of the controller and the IMU data of the controller and estimate a second pose of the hand based on the IMU-predicted pose of the hand. In particular embodiments, the method500may estimate the second pose of the controller by estimating a pose of the controller relative to the environment based on the estimated grip, adjusting the pose of the controller relative to the environment based on the IMU data of the controller, estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the IMU of the controller, and estimating the second pose of the controller based on the adjusted pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.

In particular embodiments, the method500may further capture, using the camera, a second image of the user depicting at least a part of the hand of the user holding the controller in the environment, identify the one or more features from the second image of the user, and estimate a third pose of the hand based on the one or more features identified from the second image of the user. Furthermore, a frequency of estimating the second pose of the hand (e.g., 500 Hz) is higher than a frequency of estimating the third pose of the hand (e.g., 30 Hz).

In particular embodiments, the wearable device may comprise the camera configured to capture images of the user, a hand-tracking unit configured to estimate the pose of the hand of the user, and a controller-tracking unit configured to estimate the second pose of the controller.

Particular embodiments may repeat one or more steps of the method ofFIG. 5, where appropriate. Although this disclosure describes and illustrates particular steps of the method ofFIG. 5as occurring in a particular order, this disclosure contemplates any suitable steps of the method ofFIG. 5occurring in any suitable order. Moreover, although this disclosure describes and illustrates an example method for local localization including the particular steps of the method ofFIG. 5, this disclosure contemplates any suitable method for local localization including any suitable steps, which may include all, some, or none of the steps of the method ofFIG. 5, where appropriate. Furthermore, although this disclosure describes and illustrates particular components, devices, or systems carrying out particular steps of the method ofFIG. 5, this disclosure contemplates any suitable combination of any suitable components, devices, or systems carrying out any suitable steps of the method ofFIG. 5.

In particular embodiments, computer system600includes a processor602, memory604, storage606, an input/output (I/O) interface608, a communication interface610, and a bus612. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor602includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor602may retrieve (or fetch) the instructions from an internal register, an internal cache, memory604, or storage606; decode and execute them; and then write one or more results to an internal register, an internal cache, memory604, or storage606. In particular embodiments, processor602may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor602including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor602may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory604or storage606, and the instruction caches may speed up retrieval of those instructions by processor602. Data in the data caches may be copies of data in memory604or storage606for instructions executing at processor602to operate on; the results of previous instructions executed at processor602for access by subsequent instructions executing at processor602or for writing to memory604or storage606; or other suitable data. The data caches may speed up read or write operations by processor602. The TLBs may speed up virtual-address translation for processor602. In particular embodiments, processor602may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor602including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor602may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors602. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

In particular embodiments, memory604includes main memory for storing instructions for processor602to execute or data for processor602to operate on. As an example and not by way of limitation, computer system600may load instructions from storage606or another source (such as, for example, another computer system600) to memory604. Processor602may then load the instructions from memory604to an internal register or internal cache. To execute the instructions, processor602may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor602may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor602may then write one or more of those results to memory604. In particular embodiments, processor602executes only instructions in one or more internal registers or internal caches or in memory604(as opposed to storage606or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory604(as opposed to storage606or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor602to memory604. Bus612may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor602and memory604and facilitate accesses to memory604requested by processor602. In particular embodiments, memory604includes random access memory (RAM). This RAM may be volatile memory, where appropriate. Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory604may include one or more memories604, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

In particular embodiments, storage606includes mass storage for data or instructions. As an example and not by way of limitation, storage606may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc, a magneto-optical disc, magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage606may include removable or non-removable (or fixed) media, where appropriate. Storage606may be internal or external to computer system600, where appropriate. In particular embodiments, storage606is non-volatile, solid-state memory. In particular embodiments, storage606includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or a combination of two or more of these. This disclosure contemplates mass storage606taking any suitable physical form. Storage606may include one or more storage control units facilitating communication between processor602and storage606, where appropriate. Where appropriate, storage606may include one or more storages606. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

According to various embodiments, an advantage of features herein is that the present application can provide a tracking method which does not require a paired controller to equip with LEDs, and yet remains accurate and cost-efficient tracking. The tracking method estimates a pose of the user's hand based on features identified from captured images, and then estimates a grip of the user's hand based on the pose of the user's hand, such that the tracking method can estimate a pose of the controller based on the grip. Furthermore, the tracking method can adjust/calibrate the pose of the controller based on IMU data of the controller. In addition, the processing time of the tracking method can also be improved by the predictions provided by IMU data. Particular embodiments of the present disclosure also enable to track the controller without the LEDs or when the LEDs disposed on the controller fail. Therefore, particular embodiments disclosed in the present disclosure may provide an improved, cost-efficient tracking method for the controller.