Patent Description:
Input instructions provided to AR/VR devices is typically based on controller tracking or hand tracking. A controller can be tracked using the known patterns formed by infrared (IR) light emitting diodes (LEDs) on the controller and input an instruction in a specific location in an environment via the button on the controller. An input instruction can also be made by a hand gesture by tracking features of the hand. For example, a user can turn a page of a virtual book by tracking a swipe gesture of the hand. However, controller tracking is more costly because of the additional hardware required, e.g., IR cameras and IR LED lights on the controller which could sometimes be interfered by occlusions or other light sources, and hand tracking is less accurate. <CIT> is directed to systems and methods for providing a wireless hand-held inertial controller for use with a head-mounted, augmented or virtual reality display or other conventional display that operates with six degrees of freedom by fusing (i) data related to the position of the hand-held inertial controller derived from a depth camera located on the display with (ii) data relating to the orientation of the hand-held inertial controller derived from an inertial measurement unit located in the hand-held inertial controller. <CIT> discloses systems and methods for analyzing game control input data. A machine-readable medium having embodied thereon instructions for analyzing game control input data is also disclosed.

To address the foregoing problems, disclosed are methods, apparatuses, and a system, to track a controller by estimating a grip of a hand and adjusting the grip of the hand based on inertial measurement unit (IMU) data from the controller. The present disclosure provides a method to track a controller without implementing LEDs in the controller (e.g., without tracking a pattern of LED lights), so that the method disclosed in the present application provides a cost-efficient, accurate way to track the controller. The method disclosed in the present application estimates a grip of a user's hand based on feature-tracking identified from captured images of the user's hand and then estimate a pose of the controller using the estimated grip of the user's hand. Furthermore, the method of the present application receives IMU data of the controller to adjust the estimated pose of the controller and provide a final pose of the controller at a faster frequency.

The embodiments disclosed herein are only examples, and the scope of this disclosure is not limited to them. The invention is defined as set out in the appended claims.

According to an aspect of the invention there is provided a method comprising, by a computing system, capturing, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment; identifying one or more features of at least the part of the hand of the user depicted in the first image to estimate a pose of the hand of the user; estimating a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller; receiving IMU data of the controller; and estimating a second pose of the controller by updating the first pose of the controller using the IMU data of the controller.

The first image may be captured at a first frequency and the IMU data of the controller is received at a second frequency, wherein the second frequency is higher than the first frequency.

The method may further comprise: receiving IMU data of the wearable device to estimate a pose of the wearable device; and updating the pose of the hand of the user based on the pose of the wearable device.

The pose of the wearable device may be estimated based on the IMU data of the wearable device and the first image of the user.

The method may further comprise: estimating an IMU-predicted pose of the hand based on the updated first pose of the controller and the IMU data of the controller; and estimating a second pose of the hand based on the IMU-predicted pose of the hand.

The method may further comprise: capturing a second image of the user using the one or more cameras, the second image of the user depicting at least a part of the hand of the user holding the controller in the environment; identifying the one or more features from the second image of the user; and estimating a third pose of the hand based on the one or more features identified from the second image of the user.

A frequency of estimating the second pose of the hand may be higher than a frequency of estimating the third pose of the hand.

The wearable device may comprise: the one or more cameras configured to capture images of the user; a hand-tracking unit configured to estimate the pose of the hand of the user; and a controller-tracking unit configured to estimate the second pose of the controller.

Estimating the second pose of the controller may comprise: estimating a pose of the controller relative to the environment based on the estimated grip and the estimated pose of the hand of the user relative to the environment; adjusting the pose of the controller relative to the environment based on the IMU data of the controller; estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the IMU of the controller; and estimating the second pose of the controller based on the adjusted pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.

According to an aspect of the invention there is provided one or more computer-readable non-transitory storage media embodying software that is operable when executed to: capture, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment; identify one or more features of at least the part of the hand of the user depicted in the first image to estimate a pose of the hand of the user; estimate a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller; receive inertial measurement unit (IMU) data of the controller; and estimate a second pose of the controller by updating the first pose of the controller using the IMU data of the controller.

The software may be further operable when executed to: receive IMU data of the wearable device to estimate a pose of the wearable device; and update the pose of the hand of the user based on the pose of the wearable device.

The software may be further operable when executed to: estimate an IMU-predicted pose of the hand based on the updated first pose of the controller and the IMU data of the controller; and estimate a second pose of the hand based on the IMU-predicted pose of the hand.

The software may be further operable when executed to: capture a second image of the user using the one or more cameras, the second image of the user depicting at least a part of the hand of the user holding the controller in the environment; identify the one or more features from the second image of the user; and estimate a third pose of the hand based on the one or more features identified from the second image of the user.

According to an aspect of the invention there is provided a system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by the one or more of the processors to cause the system to: capture, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment; identify one or more features of at least the part of the hand of the user depicted in the first image to estimate a pose of the hand of the user; estimate a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller; receive inertial measurement unit (IMU) data of the controller; and estimate a second pose of the controller by updating the first pose of the controller using the IMU data of the controller.

The instructions may be further operable when executed to: receive IMU data of the wearable device to estimate a pose of the wearable device; and update the pose of the hand of the user based on the pose of the wearable device.

The present invention is defined in the attached claims directed to a method, a storage medium, and a system.

Certain aspects of the present disclosure and their embodiments may provide solutions to these or other challenges. There are, proposed herein, various embodiments which address one or more of the issues disclosed herein. The methods disclosed in the present disclosure provide a tracking method for a controller, which estimates and adjusts the pose of the controller based on the estimation of the grip and the IMU data of the controller. Furthermore, based on the pose of the controller relative to the environment and the user's hand, the method disclosed in the present application may also provide an IMU-predicted pose of the user's hand to reduce a search range of the user's hand in a next frame. Therefore, particular embodiments disclosed in the present application may track the controller cost-efficiently (e.g., no needs to install LEDs) and improve the process time to perform tracking tasks.

Particular embodiments of the present disclosure may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured content (e.g., real-world photographs). The artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may be associated with applications, products, accessories, services, or some combination thereof, that are, e.g., used to create content in an artificial reality and/or used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

The embodiments disclosed herein are only examples, and the scope of this disclosure is not limited to them. The present invention is defined attached claims directed to a method, a storage medium, and a system.

Current AR/VR devices are commonly paired with a portable/wearable device (e.g., a controller) to provide the user an easy, intuitive way to input instructions for the AR/VR devices. The controller is usually equipped with at least one inertial measurement unit (IMU) and infrared (IR) light emitting diodes (LEDs) for the AR/VR devices to estimate a pose of the controller and/or to track a location of the controller, such that the user may perform certain functions via the controller. For example, the user may use the controller to display a visual object in a corner of the room. However, equipping LEDs increases the cost of manufacturing the controller, and tracking the controller via determining a pattern of LED lights could be interfered under certain environment conditions. Also, purely relying on feature-tracking to track a controller could be inaccurate. Particular embodiments disclosed in the present disclosure provide a method to estimate a pose of the controller by fusing feature-tracking data of the user's hand and IMU data of the controller.

Furthermore, particular embodiments disclosed in the present disclosure may provide an IMU-predicted pose of the user's hand based on the fusion of the estimated grip of the hand and the IMU data of the controller to facilitate hand-tracking in a next frame. Utilizing the IMU data of the controller to adjust the grip of the hand can update the pose of the controller more frequently to keep an efficient, accurate tracking. Particular embodiments disclosed in the present disclosure may be applied to any kind of tracking system, such as visual inertial odometry (VIO)-based simultaneous localization and mapping (SLAM) tracking system, with efficiency and less cost.

<FIG> illustrate an example tracking system for tracking a controller, in accordance with certain embodiments. In <FIG>, the tracking system <NUM> comprises a central module (not shown) and a controller module <NUM> (e.g., a controller). The central module comprises a camera and at least one processor to track the controller module <NUM> in an environment. The central module is implemented in a wearable device, such as a head-mounted device, to capture an image of an object to be tracked (e.g., a controller implemented with the controller module <NUM>). For example, the wearable device with the camera may perform an inside-out tracking (e.g., SLAM) for an object. In an example not covered by the claims, the object to be tracked is tracked by one or more cameras implemented/fixed in the environment, e.g., an outside-in tracking.

The camera of the central module captures a first frame <NUM> depicting at least part of a user's hand. More specifically, the first frame <NUM> depicts at least a part of the user's hand holding the controller module <NUM>. The central module identifies one or more features <NUM>, <NUM>, <NUM> of at least part of a user's hand from the first frame <NUM>. The first frame <NUM> comprises one or more feature at least depicting the user's hand holding the controller module <NUM>. In <FIG>, the controller module <NUM> comprises a handle <NUM> for a user to hold. The central module identifies the features <NUM>, <NUM>, <NUM> of a user's hand which is used to estimate a pose of the user's hand. For example, an area <NUM> where the purlicue of the hand overlaps with the controller <NUM>, the ulnar border of the hand <NUM> where represents a user's hand holding the controller <NUM>, and an area <NUM> including the finger tips and the controller <NUM>. The identified features <NUM>,<NUM>,<NUM> from the first frame <NUM> are used to estimate a pose/location of the user's hand. Furthermore, the pose of the user's hand is used to estimate a grip of the user's hand. For example, the pose of the user's hand may be a skeleton/a primary geometry of the user's hand representing a hand gesture of the user. The estimated grip of the user's hand is utilized to estimate a pose of a controller module <NUM> based on the estimated grip of the user's hand which defines a relative pose between the hand of the user and the controller module <NUM>.

The controller module <NUM> comprises at least one IMU, such that the controller module <NUM> provides IMU data to the central module to update/adjust the estimated pose of the controller module <NUM>. The controller module <NUM> may provide the IMU data at a frequency which is faster than a frequency of the central module taking a frame of the user and the controller module <NUM>. For example, the central module may capture a second frame <NUM> of the user holding the controller module <NUM> and identify the features <NUM>, <NUM>, <NUM> or any other potential features which can be used to estimate the pose of the user's hand from the second frame <NUM>. Before the central module estimates an updated pose of the user's hand based on the identified features in the second frame <NUM>, the central module uses the received IMU data of the controller module <NUM> to adjust the estimated pose of the controller module <NUM> which was estimated based on the grip of the hand estimated from the first frame <NUM>. In particular embodiments, the central module may provide/update a pose of the user's hand at a frequency of <NUM> (e.g., based on captured frames) for estimating a pose of the controller module <NUM>, and the controller module <NUM> may provide the IMU data at a frequency of <NUM> to the central module for updating the estimated pose of the controller module <NUM>, such that the pose of the controller module <NUM> can be tracked/adjusted at a faster frequency based on the IMU data of the controller module <NUM> to keep the accuracy and efficiency of tracking the controller module <NUM>. In particular embodiments, the central module may output the pose of the controller based on either tracking result (e.g., feature tracking or IMU tracking) as needed.

In particular embodiments, the captured frames may be a visible-light image which is identified to comprise at least one feature which can be used to estimate a pose of the user's hand. The visible-light image may be an RGB image, a CMYK image, a greyscale image, or any suitable image for estimating a pose of the user's hand. In particular embodiments, the identified features <NUM>, <NUM>, <NUM> from the captured frames <NUM>, <NUM> are configured to be accurately tracked by a camera of the central module to determine a motion, orientation, and/or spatial position of the controller module <NUM> (e.g., correspondence data of the controller module <NUM>) for reproduction in a virtual/augmented environment. In particular embodiments, the estimated pose of the controller module <NUM> may be adjusted by a spatial movement (X-Y-Z positioning movement) determined based on the identified features <NUM>, <NUM>, <NUM> between frames (e.g., the first frame <NUM> and the second frame <NUM>). For example, the central module may determine an updated spatial position of the user's hand in a frame k+<NUM>, e.g., a frame captured during operation, and compare it with a previous spatial position of the user's hand in a frame k, e.g., a frame captured previously or stored in a storage, to readjust the pose of the user's hand. Detailed operations and actions performed at the central module for tracking the controller module may be further described in <FIG>.

<FIG> illustrates an example tracking system <NUM> comprising a central module and a controller module, in accordance with certain embodiments. The tracking system <NUM> comprises a central module implemented in a headset which is worn by a user, and a controller module implemented in a controller which is held by the user. In particular embodiments, the user may have two controllers paired with the headset for each hand. The headset comprises at least one camera, at least one IMU, and at least one processor which is configured to process instructions for tracking a controller. Furthermore, the controller comprises at least one IMU which is configured to provide IMU data of the controller to the central module of the headset, and at least one processer which is configured to process instructions/calibrations sent from the headset.

The camera of the headset captures one or more image of the user and the controller <NUM> in an environment and identifies one or more features of the user's hand from the image <NUM> for hand tracking <NUM> via machine learning or deep learning. Based on the identified features which can be used to estimate/determine a pose of the user's hand, the processor of the headset may estimate a pose of the user's hand and/or a location of the user's hand based on the identified features. In particular embodiments, the pose of the user's hand may be estimated based on repeated feature identified over a series of images. In particular embodiments, the processor of the headset may estimate a pose of the user's hand relative to the environment <NUM> based on the results of hand tracking <NUM>.

In particular embodiments, the IMU of the headset <NUM> may also provide IMU data of the headset to the processor of the headset, and the processor of the headset may estimate a pose of the headset relative to the environment <NUM> via inside-out tracking <NUM> based on the IMU data of the headset. In particular embodiments, the processor of the headset may estimate a pose of the headset relative to the environment <NUM> via inside-out tracking <NUM> based on the IMU data of the headset and the camera image <NUM>. For example, the IMU data of the headset may provide information of angler velocity, acceleration, and motion of the headset to calculate a pose of the headset in the environment. Furthermore, the processor of the headset may utilize the pose of the headset relative to the environment <NUM> to facilitate the hand tracking <NUM>. For example, the pose of the headset relative to environment <NUM> may be fed to facilitate the hand tracking <NUM> by comparing a pose/location of the headset relative to the environment <NUM> with the image of the user and the controller <NUM> in the environment to adjust/estimate the pose of the user's hand.

The processor of the headset may then estimate a grip of the user's hand <NUM> based on the estimated pose of the user's hand <NUM> and estimate a pose of the controller relative to the environment <NUM> based on the estimated grip of the user's hand <NUM>. For example, the processor of the headset may use the pose of the user's hand (including the identified features from the user's hand) to estimate the user's hand representing a gesture of holding the controller, such that, based on an inverse of the gesture/pose of the user's hand, the processor of the headset may generate a pose of the controller.

Furthermore, the IMU of the controller provides IMU data of the controller <NUM> to the headset for data fusion <NUM> to adjust the pose of the controller estimated based on the grip of the user's hand. The data fusion unit <NUM> may utilize the IMU data to calculate an IMU-predicted pose of the controller unit <NUM>. The IMU-predicted pose of the controller unit <NUM> may be utilized by the grip estimator unit <NUM> to adjust the pose of the controller relative to the environment and estimate an inverse grip of the user's hand <NUM>, where the inverse grip infers the pose of the user's hand <NUM> based on the pose of the adjusted pose of the controller. In particular embodiments, the final pose of the controller <NUM> may be provided based on the operations/needs of the headset. For example, the final pose of the controller <NUM> may be estimated in-between two captured frames (e.g., before the next estimation of the grip). On the other hand, the final pose of the controller <NUM> may also be estimated based on the IMU-adjusted grip, e.g., the estimated grip adjusted by the received IMU data of the controller. The processor of the headset may estimate the final pose of the controller <NUM> at a certain frequency based on a request or a demand to save power.

In addition, based on the data provided by the data fusion <NUM>, the processor of the headset may provide an IMU-predicted pose of the hand <NUM> based on the IMU-predicted pose of the controller <NUM> and use the IMU-predicted pose of the hand <NUM> to facilitate the hand tracking <NUM>. For example, the IMU-predicted pose of the controller <NUM> can be provided at a faster frequency (e.g., <NUM> to <NUM>) to fill in the gap between two frames. By applying the inverse grip estimation to the IMU-predicted pose of the controller <NUM>, the headset can generate an IMU-predicted pose of the hand <NUM>. The IMU-predicted pose of the hand <NUM> can be used to reduce a search range of the hand in the next frame to improve process time of the hand tracking <NUM>.

<FIG> illustrates an example diagram of a tracking system <NUM> comprising a central module <NUM> and a controller module <NUM>, in accordance with certain embodiments. The central module <NUM> comprises a camera <NUM>, an IMU <NUM>, a hand and headset tracking unit <NUM>, and a controller tracking unit <NUM> to perform a tracking/adjustment for the controller module <NUM> in an environment. The central module <NUM> is paired with the controller module <NUM> to perform certain functions via the controller module <NUM>. The controller module <NUM> comprises at least one IMU <NUM> configured to provide IMU data <NUM> for the central module <NUM> to track the controller module <NUM>. In particular embodiments, the controller module <NUM> sends the IMU data <NUM> to the controller tracking unit <NUM> for computing predictions of a corresponding module, e.g., correspondence data of the controller module <NUM>. In particular embodiments, the central module <NUM> measures the pose of the controller module <NUM> at a frequency from <NUM> to <NUM> based on the IMU data <NUM> of the controller module <NUM>.

In order to generate/estimate a pose of the controller module <NUM> during operation, the camera <NUM> of the central module <NUM> may capture an image or a series of images <NUM> when the controller module <NUM> is within a field of view (FOV) range of the camera for tracking the controller module <NUM>. In particular embodiments, the image <NUM> depict at least a part of the user's hand holding the controller module <NUM>. The camera <NUM> of the central module <NUM> sends the image <NUM> to the hand / headset tracking unit <NUM> for an estimation of a pose of the user's hand based on features identified from the images <NUM>.

The hand / headset tracking unit <NUM> identifies one or more features of the user's hand from the image <NUM> via machine learning, deep learning, or any suitable computing methods. Based on the identified features which can be used to estimate/determine a pose of the user's hand <NUM>, the hand / headset tracking unit <NUM> of the central module <NUM> estimates a pose of the user's hand <NUM> and/or a location of the user's hand in the environment based on the identified features of the user's hand. In particular embodiments, the pose of the user's hand <NUM> may be estimated based on repeated feature identified over a series of images. The hand / headset tracking unit <NUM> of the central module <NUM> estimates the pose of the user's hand <NUM> at a frequency based on a processing capability or a requirement. In particular embodiments, the hand / headset tracking unit <NUM> of the central module <NUM> estimates the pose of the user's hand <NUM> at a frequency of <NUM>.

In particular embodiments, the IMU <NUM> of the central module <NUM> also sends IMU data <NUM> to the hand / headset tracking unit <NUM> to facilitate the estimation of the pose of the headset. For example, the hand / headset tracking unit <NUM> may perform an inside-out tracking to estimate a pose of the central module <NUM>. Based on the image <NUM> (including the controller module <NUM> in the environment) and the IMU data <NUM> of the central module <NUM>, the hand / headset tracking unit <NUM> of the central module <NUM> may estimate the pose of the central module <NUM>, so that the estimated pose of the user's hand <NUM> (estimated based the images <NUM>) may be adjusted by the pose of the central module <NUM> (e.g., the location of the central module <NUM> relative to the user's hand in the environment).

The hand / headset tracking unit <NUM> of the central unit <NUM> sends the pose of the user's hand <NUM> to the controller tracking unit <NUM> for controller tracking. The controller tracking unit <NUM> comprises a grip estimation unit <NUM> configured to estimate a grip of the user's hand and a data fusion unit <NUM> configured to fuse/integrate data sent from the grip estimation unit <NUM> and data sent from the controller module <NUM>.

The grip estimation unit <NUM> of the controller tracking unit <NUM> receives the pose of the user's hand <NUM> from the hand / headset tracking unit <NUM> and estimates a grip of the user's hand based on the pose of the user's hand <NUM>. Furthermore, the grip estimation unit <NUM> estimates a pose of the controller module <NUM> based on the grip of the user's hand. For example, the pose of the user's hand <NUM> may reveal a gesture of the user holding the controller module <NUM>. Therefore, based on the pose of the user's hand <NUM>, the grip estimation unit <NUM> may estimate the grip of the user's hand and then estimate the pose of the controller module relative to the environment <NUM> based on the grip of the user's hand that defines a relative pose between the user's hand and the controller module <NUM>. Furthermore, the grip estimation unit <NUM> sends the pose of the controller relative to the environment <NUM> to the data fusion unit <NUM>.

The data fusion unit <NUM> of the controller tracking unit <NUM> receives the pose of the controller relative to the environment <NUM> from the grip estimation unit <NUM> of the controller tracking module <NUM> in the central module <NUM>, and further receives the IMU data <NUM> from the controller module <NUM>. The data fusion unit <NUM> may integrate the pose of the controller module relative to the environment <NUM> with the IMU data <NUM> of the controller module <NUM> to output an adjusted/final pose of the controller module for the central module <NUM> to perform a corresponding instruction accurately via the controller module <NUM>. In particular embodiments, the data fusion unit <NUM> may output the adjusted pose of controller module at a frequency based on the request or the processing speed of the central module <NUM>. In particular embodiments, the data fusion unit <NUM> may output the adjusted pose of the controller module at a frequency which is faster than the frequency of estimating the pose of the user's hand, such as <NUM>, since the data fusion unit <NUM> can update the pose of the controller module <NUM> sent from the grip estimation unit <NUM> when it receives the IMU data <NUM> from the controller module <NUM>.

Furthermore, the data fusion unit <NUM> may also provide an IMU-predicted pose of the controller unit <NUM> based on the IMU data <NUM> of the controller module <NUM> to the grip estimation unit <NUM>, such that the grip estimation unit <NUM> may adjust the pose of the controller module <NUM> estimated based on the captured frames. The grip estimation unit <NUM> may provide an IMU-predicted pose of the user's hand <NUM> based on the IMU data <NUM> of the controller module <NUM> to the hand tracking unit <NUM> to facilitate the process of hand tracking. With the IMU-predicted pose of the user's hand <NUM>, the hand tracking unit <NUM> may identify features of the user's hand within a predicted range in a next captured frame, so that the hand tracking unit <NUM> may complete the hand tracking with less process time.

Furthermore, the central module <NUM> may also utilize these captured images <NUM> including identified features to conduct extensive services and functions, such as generating a state of the user/the controller module <NUM>, locating the user/the controller module <NUM> locally or globally, and/or rendering a virtual tag/object in the environment via the controller module <NUM>. In particular embodiments, the central module <NUM> may also use the IMU data <NUM> in assistance of generating the state of the user. In particular embodiments, the central module <NUM> may use the state information of the user relative to the controller module <NUM> in the environment based on the captured images <NUM>, to project a virtual object in the environment or set a virtual tag in a map via the controller module <NUM>.

In particular embodiments, the tracking system <NUM> may be implemented in any suitable computing device, such as, for example, a personal computer, a laptop computer, a cellular telephone, a smartphone, a tablet computer, an augmented/virtual reality device, a head-mounted device, a portable smart device, a wearable smart device, or any suitable device which is compatible with the tracking system <NUM>. In the present disclosure, a user which is being tracked and localized by the tracking device may be referred to a device mounted on a movable object, such as a vehicle, or a device attached to a person. In the present disclosure, a user may be an individual (human user), an entity (e.g., an enterprise, business, or third-party application), or a group (e.g., of individuals or entities) that interacts or communicates with the tracking system <NUM>. In particular embodiments, the central module <NUM> may be implemented in a head-mounted device, and the controller module <NUM> may be implemented in a remote controller separated from the head-mounted device. The head-mounted device comprises one or more processors configured to implement the camera <NUM>, the IMU <NUM>, the hand / headset tracking unit <NUM>, and the controller unit <NUM> of the central module <NUM>. In one embodiment, each of the processors is configured to implement the camera <NUM>, the IMU <NUM>, the hand / headset tracking unit <NUM>, and the controller unit <NUM> separately. The remote controller comprises one or more processors configured to implement the IMU <NUM> of the controller module <NUM>. In one embodiment, each of the processors is configured to implement the IMU <NUM> separately.

This disclosure contemplates any suitable network to connect each element in the tracking system <NUM> or to connect the tracking system <NUM> with other systems. As an example and not by way of limitation, one or more portions of network may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these. Network may include one or more networks.

<FIG> illustrates an example diagram of a tracking system <NUM> with mapping service, in accordance with certain embodiments. The tracking system <NUM> comprises a controller module <NUM>, a central module <NUM>, and a cloud <NUM>. The controller module <NUM> comprises at least one IMU <NUM>, and a processor <NUM>. The controller module <NUM> receives one or more instructions <NUM> from the central module <NUM> to perform specific functions. The controller module <NUM> is configured to send IMU data <NUM> to the central module <NUM> for a pose estimation during operation, so that the central module <NUM> may perform the instructions <NUM> via the controller module <NUM> accurately in a map or in the environment.

The central module <NUM> comprises a camera <NUM>, at least one IMU <NUM>, a hand tracking unit <NUM>, and a controller tracking unit <NUM>. The central module <NUM> is configured to track the controller module <NUM> based on various methods, e.g., the method disclosed in <FIG>. The camera <NUM> of the central module <NUM> may capture one or more frames of the controller module <NUM> being held by a user, and the IMU <NUM> of the central module <NUM> may provide IMU data of the central module <NUM> to the hand tracking unit <NUM>. The hand tracking unit <NUM> may identify features from the captured frames via machine learning to estimate a pose of the user's hand and adjust the pose of the user's hand based on the IMU data of the central module <NUM>. Furthermore, the hand tracking unit <NUM> sends the pose of the user's hand to the controller tracking unit <NUM> to estimate a pose of the controller module <NUM>. The controller tracking unit <NUM> receives the pose of the user's hand and the IMU data <NUM> of the controller module <NUM> and estimates the pose of the controller module <NUM> by fusing the received data.

In particular embodiments, the controller tracking unit <NUM> may determine correspondence data based on the features identified in different frames. The correspondence data may comprise observations and measurements of the feature, such as a location of the feature of the controller module <NUM> in the environment. Furthermore, the controller tracking unit <NUM> may also perform a stereo computation collected near the predetermined feature to provide additional information for the central module <NUM> to track the controller module <NUM>. In addition, the controller tracking unit <NUM> of the central module <NUM> may request a live map from the cloud <NUM> corresponding to the correspondence data. In particular embodiments, the live map may comprise map data <NUM>. The controller tracking unit <NUM> of the central module <NUM> may also request a remote relocalization service <NUM> for the controller module <NUM> to be located in the live map locally or globally. In particular embodiments, the pose of the controller module <NUM> relative to the environment may be built based on the frames captured by the camera <NUM>, e.g., a map built locally. In particular embodiments, the controller tracking unit <NUM> of the central module <NUM> may also send the correspondence data of the controller module <NUM> to the cloud <NUM> for an update of the map stored in the cloud <NUM> (e.g., with the environment built locally).

<FIG> illustrates a method <NUM> for tracking a controller, in accordance with certain embodiments. A controller module of a tracking system may be implemented in a portable device (e.g., a remote controller with input buttons, a smart puck with touchpad, etc.). A central module of the tracking system is implemented in a wearable device (e.g., a head-mounted device, etc.), or be provided to or displayed on any computing system (e.g., an end user's device, such as a smartphone, virtual reality system, gaming system, etc.), and be paired with the controller module. The method <NUM> begins at step <NUM> with capturing, using a camera, a first image depicting at least a part of a hand of the user holding a controller in an environment. The camera is one or more cameras implemented in a wearable device worn by a user. The wearable device is a controller. The wearable device is equipped with one or more IMUs.

At step <NUM>, the method <NUM> identifies one or more features of at least the part of the hand of the user depicted in the first image to estimate a pose of the hand of the user. The method <NUM> receives IMU data of the wearable device to estimate a pose of the wearable device and update the pose of the hand of the user based on the pose of the wearable device. Furthermore, the pose of the wearable device is estimated based on the IMU data of the wearable device and the first image of the user.

At step <NUM>, the method <NUM> estimates a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller.

At step <NUM>, the method <NUM> receives IMU data of the controller. In particular embodiments, the IMU data of the controller may be received at a faster frequency than a frequency that the first image is captured. For example, the first image may be captured at a first frequency and the IMU data of the controller may be received at a second frequency. The second frequency (e.g., <NUM>) is higher than the first frequency (e.g., <NUM>).

At step <NUM>, the method <NUM> estimates a second pose of the controller by updating the first pose of the controller using the IMU data of the controller. In particular embodiment, the method <NUM> may estimate an IMU-predicted pose of the hand based on the updated first pose of the controller and the IMU data of the controller and estimate a second pose of the hand based on the IMU-predicted pose of the hand. In particular embodiments, the method <NUM> may estimate the second pose of the controller by estimating a pose of the controller relative to the environment based on the estimated grip, adjusting the pose of the controller relative to the environment based on the IMU data of the controller, estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the IMU of the controller, and estimating the second pose of the controller based on the adjusted pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.

In particular embodiments, the method <NUM> may further capture, using the camera, a second image of the user depicting at least a part of the hand of the user holding the controller in the environment, identify the one or more features from the second image of the user, and estimate a third pose of the hand based on the one or more features identified from the second image of the user. Furthermore, a frequency of estimating the second pose of the hand (e.g., <NUM>) is higher than a frequency of estimating the third pose of the hand (e.g., <NUM>).

In particular embodiments, the wearable device comprises the camera configured to capture images of the user, a hand-tracking unit configured to estimate the pose of the hand of the user, and a controller-tracking unit configured to estimate the second pose of the controller.

Particular embodiments may repeat one or more steps of the method of <FIG>, where appropriate. Although this disclosure describes and illustrates particular steps of the method of <FIG> as occurring in a particular order, this disclosure contemplates any suitable steps of the method of <FIG> occurring in any suitable order. Moreover, although this disclosure describes and illustrates an example method for local localization including the particular steps of the method of <FIG>, this disclosure contemplates any suitable method for local localization including any suitable steps, which may include all, some, or none of the steps of the method of <FIG>, where appropriate. Furthermore, although this disclosure describes and illustrates particular components, devices, or systems carrying out particular steps of the method of <FIG>, this disclosure contemplates any suitable combination of any suitable components, devices, or systems carrying out any suitable steps of the method of <FIG>.

<FIG> illustrates an example computer system <NUM>. In particular embodiments, one or more computer systems <NUM> perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems <NUM> provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems <NUM> performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems <NUM>. Herein, reference to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, reference to a computer system may encompass one or more computer systems, where appropriate.

In particular embodiments, processor <NUM> includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor <NUM> may retrieve (or fetch) the instructions from an internal register, an internal cache, memory <NUM>, or storage <NUM>; decode and execute them; and then write one or more results to an internal register, an internal cache, memory <NUM>, or storage <NUM>. In particular embodiments, processor <NUM> may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor <NUM> including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor <NUM> may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory <NUM> or storage <NUM>, and the instruction caches may speed up retrieval of those instructions by processor <NUM>. Data in the data caches may be copies of data in memory <NUM> or storage <NUM> for instructions executing at processor <NUM> to operate on; the results of previous instructions executed at processor <NUM> for access by subsequent instructions executing at processor <NUM> or for writing to memory <NUM> or storage <NUM>; or other suitable data. The data caches may speed up read or write operations by processor <NUM>. The TLBs may speed up virtual-address translation for processor <NUM>. In particular embodiments, processor <NUM> may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor <NUM> including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor <NUM> may include one or more arithmetic logic units (ALUs); be a multicore processor; or include one or more processors <NUM>. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

In particular embodiments, memory <NUM> includes main memory for storing instructions for processor <NUM> to execute or data for processor <NUM> to operate on. As an example and not by way of limitation, computer system <NUM> may load instructions from storage <NUM> or another source (such as, for example, another computer system <NUM>) to memory <NUM>. Processor <NUM> may then load the instructions from memory <NUM> to an internal register or internal cache. To execute the instructions, processor <NUM> may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor <NUM> may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor <NUM> may then write one or more of those results to memory <NUM>. In particular embodiments, processor <NUM> executes only instructions in one or more internal registers or internal caches or in memory <NUM> (as opposed to storage <NUM> or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory <NUM> (as opposed to storage <NUM> or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor <NUM> to memory <NUM>. Bus <NUM> may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor <NUM> and memory <NUM> and facilitate accesses to memory <NUM> requested by processor <NUM>. In particular embodiments, memory <NUM> includes random access memory (RAM). This RAM may be volatile memory, where appropriate. Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory <NUM> may include one or more memories <NUM>, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

In particular embodiments, communication interface <NUM> includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between computer system <NUM> and one or more other computer systems <NUM> or one or more networks. As an example and not by way of limitation, communication interface <NUM> may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface <NUM> for it. As an example and not by way of limitation, computer system <NUM> may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system <NUM> may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless network or a combination of two or more of these. Computer system <NUM> may include any suitable communication interface <NUM> for any of these networks, where appropriate. Communication interface <NUM> may include one or more communication interfaces <NUM>, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

According to various embodiments, an advantage of features herein is that the present application can provide a tracking method which does not require a paired controller to equip with LEDs, and yet remains accurate and cost-efficient tracking. The tracking method estimates a pose of the user's hand based on features identified from captured images, and then estimates a grip of the user's hand based on the pose of the user's hand, such that the tracking method can estimate a pose of the controller based on the grip. Furthermore, the tracking method can adjust/calibrate the pose of the controller based on IMU data of the controller. In addition, the processing time of the tracking method can also be improved by the predictions provided by IMU data. Particular embodiments of the present disclosure also enable to track the controller without the LEDs or when the LEDs disposed on the controller fail. Therefore, particular embodiments disclosed in the present disclosure may provide an improved, cost-efficient tracking method for the controller.

While processes in the figures may show a particular order of operations performed by certain embodiments of the invention, it should be understood that such order is exemplary (e.g., alternative embodiments may perform the operations in a different order, combine certain operations, overlap certain operations, etc.).

Claim 1:
A method (<NUM>) comprising, by a computing system:
capturing (<NUM>), using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment;
identifying (<NUM>) one or more features of at least the part of the hand of the user depicted in the first image to estimate a pose of the hand of the user;
estimating (<NUM>) a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller;
receiving (<NUM>) inertial measurement unit, IMU, data of the controller; and
estimating (<NUM>) a second pose of the controller by updating the first pose of the controller using the IMU data of the controller.