Patent Description:
Current AR/VR controllers are being tracked using the known patterns formed by infrared (IR) light emitting diodes (LEDs) on the controllers. Although each controller has an IMU and the IMU data could be used to determine the pose of the controller, the estimated pose will inevitably drift over time. Thus, periodically, the IMU-based pose estimations of the controller would need to be realigned with the observed patterns observed by the camera. In addition, tracking based on the IR LEDs have several shortcomings. For example, bright sunlight or other infrared light sources would cause tracking to fail. Furthermore, when the controller is close to the user's head, the IR LEDs may not be visible to allow for proper tracking.

Prior art document <CIT> discloses a method for tracking the poses of a wearable device, by recognizing feature points on the surface of the wearable device.

To address the foregoing problems, disclosed are methods, apparatuses, and a system, to track a controller by capturing a short exposure frame and a long exposure frame of an object alternately, such as performing an infrared (IR)-based tracking and a visual inertial odometry (VIO) tracking alternately by a camera. The present disclosure provides a method to realign a location of the controller by taking an IR image of the controller with a shorter exposure time and a visible-light image with a longer exposure time alternately. The method disclosed in the present application considers the condition of the environment to track the controller based on the IR-based observations or the visible-light observations. Furthermore, the method disclosed in the present application may re-initiate the tracking of the controller periodically or when the controller is visible in the field of view of the camera, so that an accuracy of the estimated pose of the controller can be improved over time.

The embodiments disclosed herein are only examples, and the scope of this disclosure is not limited to them. Particular embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed herein. The method comprises, by a computing system, receiving motion data captured by one or more motion sensors of a wearable device. The method further comprises generating a pose of the wearable device based on the motion data. The method yet further comprises capturing a first frame of the wearable device by a camera using a first exposure time. The method additionally comprises identifying, in the first frame, a pattern of lights disposed on the wearable device. The method further comprises capturing a second frame of the wearable device by the camera using a second exposure time. The method further comprises identifying, in the second frame, predetermined features of the wearable device. In particular embodiments, the predetermined features may be features identified in a previous frame. The method yet further comprises adjusting the pose of the wearable device in an environment based on at least one of (<NUM>) the identified pattern of lights in the first frame and (<NUM>) the identified predetermined features in the second frame.

In accordance with one aspect of the present invention, there is provided a method comprising, by a computing system: receiving motion data captured by one or more motion sensors of a wearable device; generating a pose of the wearable device based on the motion data; capturing a first frame of the wearable device by a camera using a first exposure time; identifying, in the first frame, a pattern of lights disposed on the wearable device; capturing a second frame of the wearable device by the camera using a second exposure time; identifying, in the second frame, predetermined features of the wearable device; and adjusting the pose of the wearable device in an environment based on the identified pattern of lights in the first frame and the identified predetermined features in the second frame.

The camera captures the first frame of the wearable device using the first exposure time when the environment has a first light condition; and the second frame of the wearable device using the second exposure time when the environment has a second light condition.

In some embodiments, the second light condition may comprise one or more of: an environment having bright light; an environment having a light source to interfere the pattern of lights of the wearable device; and the camera not being able to capture the pattern of lights.

In some embodiments, the wearable device may be equipped with one or more inertial measurement units (IMUs) and one or more infrared (IR) light emitting diodes (LEDs); the first frame is an IR image; and the second frame is a visible-light image.

The second exposure time is longer than the first exposure time.

In some embodiments, the pose of the wearable device may be generated at a faster frequency than a frequency that the first frame and the second frame are captured.

In some embodiments, the method may further comprise: capturing a third frame of the wearable device by the camera using the second exposure time; identifying, in the third frame, one or more features corresponding to the predetermined features of the wearable device; determining correspondence data between the predetermined features and the one or more features; and tracking the wearable device in the environment based on the correspondence data.

The system comprises: the camera configured to capture the first frame and the second frame of the wearable device. Furthermore, the system is configured to identify the pattern of lights and the predetermined features of the wearable device; and to adjust the pose of the wearable device.

In some embodiments, the camera may be located within a head-mounted device; and wherein the wearable device is a controller separated from the head-mounted device.

In some embodiments, the head-mounted device may comprise one or more processors, wherein the one or more processors are configured to implement the camera, the identifying unit, and the filter unit.

In accordance with a further aspect of the present invention, there is provided one or more computer-readable non-transitory storage media embodying software that is operable when executed to: receive motion data captured by one or more motion sensors of a wearable device; generate a pose of the wearable device based on the motion data; capture a first frame of the wearable device by a camera using a first exposure time; identify, in the first frame, a pattern of lights disposed on the wearable device; capture a second frame of the wearable device by the camera using a second exposure time; identify, in the second frame, predetermined features of the wearable device; and adjust the pose of the wearable device in an environment based on the identified pattern of lights in the first frame and the identified predetermined features in the second frame.

The software is further operable when executed to: capture the first frame of the wearable device using the first exposure time when the environment has a first light condition; and capture the second frame of the wearable device using the second exposure time when the environment has a second light condition.

In some embodiments, the pose of wearable device may be generated at a faster frequency than a frequency that the first frame and the second frame are captured.

In some embodiments, the software may be further operable when executed to: capture a third frame of the wearable device by the camera using the second exposure time; identify, in the third frame, one or more features corresponding to the predetermined features of the wearable device; determine correspondence data between the predetermined features and the one or more features; and track the wearable device in the environment based on the adjusted pose and the correspondence data.

In some embodiments, the camera may be located within a head-mounted device; and wherein the wearable device is a remote controller separated from the head-mounted device.

In accordance with a further aspect of the present invention, there is provided a system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by the one or more of the processors to cause the system to: receive motion data captured by one or more motion sensors of a wearable device; generate a pose of the wearable device based on the motion data; capture a first frame of the wearable device by a camera using a first exposure time; identify, in the first frame, a pattern of lights disposed on the wearable device; capture a second frame of the wearable device by the camera using a second exposure time; identify, in the second frame, predetermined features of the wearable device; and adjust the pose of the wearable device in an environment based on the identified pattern of lights in the first frame and the identified predetermined features in the second frame.

The instructions further cause the system to: capture the first frame of the wearable device using the first exposure time when the environment has a first light condition; and capture the second frame of the wearable device using the second exposure time when the environment has a second light condition.

The invention is defined by the attached claims directed to a method, a storage medium, and a system.

Certain aspects of the present disclosure and their embodiments may provide solutions to these or other challenges. There are, proposed herein, various embodiments which address one or more of the issues disclosed herein. The methods disclosed in the present disclosure may provide a tracking method for a controller, which adjusts the pose of the controller estimated by IMU data collected from the IMU(s) disposed on the controller based on an IR image and/or a visible-light image captured by a camera of the head-mounted device. The methods disclosed in the present disclosure may improve the accuracy of the pose of the controller, even if the user is under an environment with various light conditions or light interferences. Furthermore, particular embodiments disclosed in the present application may generate the pose of the controller based on the IMU data and the visible-light images, so that the IR-based tracking may be limited under a certain light condition to save power and potentially lower cost for manufacturing the controller. Therefore, the alternative tracking system disclosed in the present disclosure may improve the tracking task efficiently in various environment conditions.

Particular embodiments of the present disclosure may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured content (e.g., real-world photographs). The artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may be associated with applications, products, accessories, services, or some combination thereof, that are, e.g., used to create content in an artificial reality and/or used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

The embodiments disclosed herein are only examples, and the scope of this disclosure is not limited to them. Particular embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed above. Embodiments according to the invention are in particular disclosed in the attached claims directed to a method, a storage medium, and a system.

The patent or application file contains drawings executed in color.

For extensive services and functions provided by current AR/VR devices, a controller is commonly paired with the AR/VR devices to provide the user an easy, intuitive way to input instructions for the AR/VR devices. The controller is usually equipped with at least one inertial measurement units (IMUs) and infrared (IR) light emitting diodes (LEDs) for the AR/VR devices to estimate a pose of the controller and/or to track a location of the controller, such that the user may perform certain functions via the controller. For example, the user may use the controller to display a visual object in a corner of the room or generate a visual tag in an environment. The estimated pose of the controller will inevitably drift over time and require a realignment by an IR-based tracking. However, the IR-based tracking may be interfered by other LED light sources and/or under an environment having bright light. Furthermore, the IR-based tracking may fail due to the IR LEDs of the controller not being visible to allow for proper tracking. Particular embodiments disclosed in the present disclosure provide a method to alternately take an IR image and a visible-light image for adjusting the pose of the controller based on different light levels, environmental conditions, and/or a location of the controller.

Particular embodiments disclosed in the present disclosure provide a method to realign the pose of the controller utilizing an IR tracking or a feature tracking depending on whichever happens first. During an initialization of a controller, particular embodiments of the present application may predetermine certain features, e.g., reliable features to track the controller, by setting/painting on these features in a central module, so that the central module can identify these features in a visible-light image to adjust a pose of the controller when the pose of the controller drifts along operation.

<FIG> illustrates an example VIO-based SLAM tracking system architecture, in accordance with certain embodiments. The tracking system <NUM> comprises a central module <NUM> and at least one controller module <NUM>. The central module <NUM> comprises a camera <NUM> configured to capture a frame of the controller module <NUM> in an environment, an identifying unit <NUM> configured to identify patches and features from the frame captured by the camera <NUM>, and at least one processor <NUM> configured to estimate geometry of the central module <NUM> and the controller module <NUM>. For example, the geometry comprises 3D points in a local map, a pose/motion of the controller module <NUM> and/or the central module <NUM>, a calibration of the central module <NUM>, and/or a calibration of the controller module <NUM>. The controller module <NUM> comprises at least one IMU <NUM> configured to collect raw IMU data <NUM> of the controller module <NUM> upon receiving an instruction <NUM> from the central module <NUM>, and to send the raw IMU data <NUM> to the processor <NUM> to generate a pose of the controller module <NUM>, such that the central module <NUM> may learn and track a pose of the controller module <NUM> in the environment. The controller module <NUM> can also provide raw IMU data <NUM> to the identifying unit <NUM> for computing a prediction, such as correspondence data, for a corresponding module. Furthermore, the controller module <NUM> may comprise trackable markers selectively distributed on the controller module <NUM> to be tracked by the central module <NUM>. For example, the trackable markers may be a plurality of light (e.g., light emitting diodes) or other trackable markers that can be tracked by the camera <NUM>.

In particular embodiments, the identifying unit <NUM> of the central module <NUM> receives an instruction <NUM> to initiate the controller module <NUM>. The identifying unit <NUM> instructs the camera <NUM> to capture a first frame of the controller module <NUM> for the initialization upon the receipt of the instruction <NUM>. The first frame <NUM> may comprise one or more predetermined features <NUM> which are set or painted on in the central module <NUM>. For example, the predetermined features <NUM> may be features identified in previous frames to track the controller module <NUM>, and these identified features which are repeatedly recognized in the previous frames are considered reliable features for tracking the controller module <NUM>. The camera <NUM> of the central module <NUM> may then start to capture a second frame <NUM> after the initialization of the controller module <NUM>. For example, the processor <NUM> of the central module <NUM> may start to track the controller module <NUM> by capturing the second frame <NUM>. In one embodiment, the second frame <NUM> may be a visible-light image which comprises the predetermined feature <NUM> of the controller module <NUM>, so that the central module <NUM> may adjust the pose of the controller module <NUM> based on the predetermined feature <NUM> captured in the second frame <NUM>. In another embodiment, the second frame may be an IR image which captures the plurality of lights disposed on the controller module <NUM>, such that the central module <NUM> may realign the pose of the controller module <NUM> based on a pattern <NUM> of lights formed by the plurality of lights on the controller module <NUM>. Also, the IR image can be used to track the controller module <NUM> based on the pattern <NUM> of lights, e.g., constellation of LEDs, disposed on the controller module <NUM>, and furthermore, to update the processor <NUM> of the central module <NUM>. In particular embodiments, the central module <NUM> may be set to take an IR image and a visible-light image alternately for realignment of the controller module <NUM>. In particular embodiments, the central module <NUM> may determine to take either an IR image or a visible-light image for realignment of the controller module <NUM> based on a light condition of the environment. Detailed operations and actions performed at the central module <NUM> may be further described in <FIG>.

In certain embodiments, the identifying unit <NUM> may further capture a third frame following the second frame <NUM> and identify, in the third frame, one or more patches corresponding to the predetermined feature <NUM>. In this particular embodiment, the second frame <NUM> and the third frame, and potentially one or more next frames, are the visible-light frames, e.g., the frames taken with a long-exposure time, such that the central module <NUM> can track the controller module <NUM> based on the repeatedly-identified features over frames. The identifying unit <NUM> may then determine correspondence data <NUM> of a predetermined feature <NUM> between patches corresponding to each other identified in different frames, e.g., the second frame <NUM> and the third frame, and send the correspondence data <NUM> to the processor <NUM> for further analysis and service, such as adjusting the pose of the controller module <NUM> and generating state information of the controller module <NUM>. In particular embodiments, the state information may comprise a pose, velocity, acceleration, spatial position and motion of the controller module <NUM>, and potentially a previous route, of controller module <NUM> relative to an environment built by the series of frames captured by the cameras <NUM> of the central module <NUM>.

<FIG> illustrates an example tracking system for a controller based on an IR image and/or a visible-light image, in accordance with certain embodiments. The tracking system <NUM> comprises a central module (not shown) and a controller module <NUM>. The central module comprises a camera and at least one processor to track the controller module <NUM> in an environment. In particular embodiments, the camera of the central module may capture a first frame <NUM> to determine or set up predetermined features <NUM> of the controller module <NUM> for tracking during initialization stage. For example, during the initialization/startup phase of the controller module <NUM>, a user would place the controller module <NUM> in a range of field of view (FOV) of the camera of the central module to initiate the controller module <NUM>. The camera of the central module may capture the first frame <NUM> of the controller module <NUM> in this startup phase to determine one or more predetermined features <NUM> to track the controller module <NUM>, such as an area where the purlicue of the hand overlaps with the controller module <NUM> and the ulnar border of the hand where represents a user's hand holding the controller module <NUM>. In particular embodiments, the predetermined features <NUM> can also be painted on (e.g., via small QR codes). In particular embodiments, the predetermined feature <NUM> may be a corner of a table or any other trackable features identified in a visible-light frame. In particular embodiments, the predetermined feature <NUM> may be IR patterns "blobs" in an IR image, e.g., the constellations of LEDs captured in the IR image.

In particular embodiments, the controller module <NUM> comprises at least one IMU and a plurality of IR LEDs, such that the controller module <NUM> can be realigned during operation based on either a second frame <NUM> capturing a pattern <NUM> of the IR LEDs or a second frame <NUM> capturing the predetermined features <NUM>. For example, the central module may generate a pose of the controller module <NUM> based on raw IMU data sending from the controller module <NUM>. The generated pose of the controller module <NUM> may be shifted over time and required a realignment. The central module may determine to capture a second frame <NUM> which captures the controller module <NUM> for adjusting the generated pose of the controller <NUM> based on a light condition in the environment. In one embodiment, the second frame <NUM> may be an IR image comprising a pattern <NUM> of the IR LEDs. When the IR pattern is a known a priori, the second frame, which is an IR image, can be used to realign or track the controller module <NUM> without multiple frames. In another embodiment, the second frame <NUM> may be a visible-light image which is identified to comprise at least one predetermined feature <NUM>. The visible-light image may be an RGB image, a CMYK image, or a greyscale image.

In particular embodiments, the central module may capture an IR image and a visible-light image alternately by a default setting, such that the central module may readjust the generated pose of the controller module <NUM> based on either the IR image or the visible-light image whichever is captured first for readjustment. In particular embodiments, the central module may capture the IR image when the environment comprises a first light condition. The first light condition may comprise one or more of an indoor environment, an environment not having bright light in the background, an environment not having a light source to interfere the pattern <NUM> of IR LEDs of the controller module <NUM>. For example, the environment may not comprise other LEDs to interfere the pattern <NUM> formed by the IR LEDs of the central module to determine a location of the controller module <NUM>.

In particular embodiments, the central module may capture the visible image when the environment comprises a second light condition. The second light condition may comprise one or more of an environment having bright light, an environment having a light source to interfere the pattern <NUM> of IR LEDs of the controller module <NUM>, and the camera of the central module not being able to capture the pattern of lights. For example, when a user is holding a controller implemented with the controller module <NUM> too close to a head-mounted device implemented with the central module, the camera of the central module cannot capture a complete pattern <NUM> formed by the IR LEDs of the controller module <NUM> to determine a location of the controller module <NUM> in the environment. Detailed operations and actions performed at the central module may be further described in <FIG>.

<FIG> illustrates an example controller <NUM> implemented with a controller module, in accordance with certain embodiments. The controller <NUM> comprises a surrounding ring portion <NUM> and a handle portion <NUM>. The controllers <NUM> is implemented with the controller module described in the present disclosure and includes a plurality of tracking features positioned in a corresponding tracking pattern. In particular embodiments, the tracking features can include, for example, fiducial markers or light emitting diodes (LED). In particular embodiments described herein the tracking features are LED lights, although other lights, reflectors, signal generators or other passive or active markers can be used in other embodiments. For example, the controller <NUM> may comprise a contrast feature on the ring portion <NUM> or the handle portion <NUM>, e.g., a strip with contrast color around the surface of the ring portion <NUM>, and/or a plurality of IR LEDs <NUM> embedded in the ring portion <NUM>. The tracking features in the tracking patterns are configured to be accurately tracked by a tracking camera of a central module to determine a motion, orientation, and/or spatial position of the controller <NUM> for reproduction in a virtual/augmented environment. In particular embodiments, the controller <NUM> includes a constellation or pattern of lights <NUM> disposed on the ring portion <NUM>.

In particular embodiment, the controller <NUM> comprises at least one predetermined feature <NUM> for the central module to readjust a pose of the controller <NUM>. The pose of the controller <NUM> may be adjusted by a spatial movement (X-Y-Z positioning movement) determined based on the predetermined features <NUM> between frames. For example, the central module may determine an updated spatial position of the controller <NUM> in frame k+<NUM>, e.g., a frame captured during operation, and compare it with a previous spatial position of the controller <NUM> in frame k, e.g., a frame captured in the initialization of the controller <NUM>, to readjust the pose of the controller <NUM>.

<FIG> illustrates an example diagram of a tracking system <NUM> comprising a central module <NUM> and a controller module <NUM>, in accordance with certain embodiments. The central module <NUM> comprises a camera <NUM>, an identifying unit <NUM>, a tracking unit <NUM>, and a filter unit <NUM> to perform a tracking/adjustment for the controller <NUM> in an environment. The controller module <NUM> comprises a plurality of LEDs <NUM> and at least one IMU <NUM>. In particular embodiments, the identifying unit <NUM> of the central module <NUM> may send instructions <NUM> to initiate the controller module <NUM>. In particular embodiments, the initialization for the controller module <NUM> may comprise capturing a first frame of the controller module <NUM> and predetermining one or more features in the first frame for tracking/identifying the controller module <NUM>. The instructions <NUM> may indicate the controller module <NUM> to provide raw IMU data <NUM> for the central module <NUM> to track the controller module <NUM>. The controller module <NUM> sends the raw IMU data <NUM> collected by the IMU <NUM> to the filter unit <NUM> of the central module <NUM> upon a receipt of the instructions <NUM>, to order to generate/estimate a pose of the controller module <NUM> during operation. Furthermore, the controller module <NUM> sends the raw IMU data <NUM> to the identifying unit <NUM> for computing predictions of a corresponding module, e.g., correspondence data of the controller module <NUM>. In particular embodiments, the central module <NUM> measures the pose of the controller module <NUM> at a frequency from <NUM> to <NUM>.

After initialization of the controller module <NUM>, the camera <NUM> of the central module <NUM> may capture a second frame when the controller module <NUM> is within a FOV range of the camera for a realignment of the generated pose of the controller module <NUM>. In particular embodiments, the camera <NUM> may capture the second frame of the controller module <NUM> for realignment as an IR image or a visible-light image alternately by a default setting. For example, the camera <NUM> may capture an IR image and a visible-light image alternately at a slower frequency than the frequency of generating the pose of the controller module <NUM>, e.g., <NUM>, and utilize whichever image captured first or capable for realignment, such as an image capturing a trackable pattern of the LEDs <NUM> of the controller module <NUM> or an image capturing predetermined features for tracking the controller module <NUM>.

In particular embodiments, the identifying unit <NUM> may determine a light condition in the environment to instruct the camera <NUM> to take a specific type of frame. For example, the camera <NUM> may provide the identifying unit <NUM> a frame <NUM> based on a determination of the light condition <NUM>. In one embodiment, the camera <NUM> may capture an IR image comprising a pattern of LEDs <NUM> disposed on the controller module <NUM>, when the environment does not have bright light in the background. In another embodiment, the camera <NUM> may capture a visible-light image of the controller module <NUM>, when the environment has a similar light source to interfere the pattern of LEDs <NUM> of the controller module <NUM>. In particular embodiments, the camera <NUM> captures an IR image using a first exposure time and captures a visible-light image using a second exposure time. The second exposure time may be longer than the first exposure time considering the movement of the user and/or the light condition of the environment.

In particular embodiments where no LEDs <NUM> of the controller module <NUM> are used, the central module <NUM> may track the controller module <NUM> based on visible-light images. A neural network may be used to find the controller module <NUM> in the visible-light images. The identifying unit <NUM> of the central module <NUM> may the identify features which are constantly observed over several frames, e.g., the predetermined features and/or reliable features for tracking the controller module <NUM>, in the frames captured by the camera <NUM>. The central module <NUM> may utilize these features to compute/adjust the pose of the controller module <NUM>. In particular embodiments, the features may comprise patches of images corresponding to the controller module <NUM>, such as the edges of the controller module <NUM>.

In particular embodiments, the identifying unit <NUM> may further send the identified frames <NUM> to the filter unit <NUM> for adjusting the generated pose of the controller module <NUM>. When the filter unit <NUM> receives an identified frame <NUM>, which can either be an IR image capturing the pattern of lights or a visible-light image comprising patches for tracking the controller module <NUM>, the filter unit <NUM> may determine a location of the controller module <NUM> in the environment based on the pattern of lights of the controller module <NUM> or the predetermined feature identified in the patches from the visible-light image. In particular embodiments, a patch may be a small image signature of a feature (e.g., corner or edge of the controller) that is distinct and easily identifiable in an image/frame, regardless of the angle at which the image was taken by the camera <NUM>.

Furthermore, the filter unit <NUM> may also utilize these identified frames <NUM> to conduct extensive services and functions, such as generating a state of a user/device, locating the user/device locally or globally, and/or rendering a virtual tag/object in the environment. In particular embodiments, the filter unit <NUM> of the central module <NUM> may also use the raw IMU data <NUM> in assistance of generating the state of a user. In particular embodiments, the filter unit <NUM> may use the state information of the user relative to the controller module <NUM> in the environment based on the identified frames <NUM>, to project a virtual object in the environment or set a virtual tag in a map via the controller module <NUM>.

In particular embodiment, the identifying unit <NUM> may also send the identified frames <NUM> to the tracking unit <NUM> for tracking the controller module <NUM>. The tracking unit <NUM> may determine correspondence data <NUM> based on the predetermined features in different identified frames <NUM>, and track the controller module <NUM> based on the determined correspondence data <NUM>.

In particular embodiments, the central module <NUM> captures at least the following frames to track/realign the controller module <NUM>: (<NUM>) an IR image; (<NUM>) a visible-light image; (<NUM>) an IR image; and (<NUM>) a visible-light image. In a particular embodiment, the identifying unit <NUM> of the central module <NUM> may identify IR patterns in captured IR images. When the IR patterns in the IR images are matched against an a priori pattern, such as the constellation of LED positions on the controller module <NUM> identified in the first frame, a single IR image can be sufficient to be used by the filter unit <NUM> for state estimation and/or other computations. In another embodiment of a feature-based tracking, the identifying unit <NUM> of the central module <NUM> may identify a feature to track in a first visible-light image, and the identifying unit <NUM> may then try to identify the same feature in a second visible-light frame, which feature is corresponding to the feature identified in the first visible-light image. When the identifying unit <NUM> repeatedly observes the same feature over at least two visible-light frames, these observations, e.g., identified features, in these frames can be used by the filter unit <NUM> for state estimation and/or other computations. Furthermore, in particular embodiments, the central module <NUM> can also use a single visible-light frame to update the state estimation based on a three-dimensional model of the controller module <NUM>, such as a computer-aided design (CAD) model of the controller module <NUM>.

In particular embodiments, the tracking system <NUM> may be implemented in any suitable computing device, such as, for example, a personal computer, a laptop computer, a cellular telephone, a smartphone, a tablet computer, an augmented/virtual reality device, a head-mounted device, a portable smart device, a wearable smart device, or any suitable device which is compatible with the tracking system <NUM>. In the present disclosure, a user which is being tracked and localized by the tracking device may be referred to a device mounted on a movable object, such as a vehicle, or a device attached to a person. In the present disclosure, a user may be an individual (human user), an entity (e.g., an enterprise, business, or third-party application), or a group (e.g., of individuals or entities) that interacts or communicates with the tracking system <NUM>. In particular embodiments, the central module <NUM> may be implemented in a head-mounted device, and the controller module <NUM> may be implemented in a remote controller separated from the head-mounted device. The head-mounted device comprises one or more processors configured to implement the camera <NUM>, the identifying unit <NUM>, the tracking unit <NUM>, and the filter unit <NUM> of the central module <NUM>. In one embodiment, each of the processors is configured to implement the camera <NUM>, the identifying unit <NUM>, the tracking unit <NUM>, and the filter unit <NUM> separately. The remote controller comprises one or more processors configured to implement the LEDs <NUM> and the IMU <NUM> of the controller module <NUM>. In one embodiment, each of the processors is configured to implement the LEDs <NUM> and the IMU <NUM> separately.

This disclosure contemplates any suitable network to connect each element in the tracking system <NUM> or to connect the tracking system <NUM> with other systems. As an example and not by way of limitation, one or more portions of network may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these. Network may include one or more networks.

<FIG> illustrates an example diagram of a tracking system <NUM> with mapping service, in accordance with certain embodiments. The tracking system <NUM> comprises a controller module <NUM>, a central module <NUM>, and a cloud <NUM>. The controller module <NUM> comprises an IMU unit <NUM>, a light unit <NUM>, and a processor <NUM>. The controller module <NUM> receives one or more instructions <NUM> from the central module <NUM> to perform specific functions. For example, the instruction <NUM> comprises, but is not limited to, an instruction to initiate the controller module <NUM>, an instruction to switch off the light unit <NUM>, and an instruction to tag a virtual object in the environment. The controller module <NUM> is configured to send raw IMU data <NUM> to the central module <NUM> for a pose estimation during operation, so that the processor <NUM> of the controller module <NUM> may perform the instructions <NUM> accurately in a map or in the environment.

The central module <NUM> comprises a camera <NUM>, an identifying unit <NUM>, a tracking unit <NUM>, and a filter unit <NUM>. The central module <NUM> may be configured to track the controller module <NUM> based on various methods, e.g., an estimated pose of the controller module <NUM> determined by the raw IMU data <NUM>. Furthermore, the central module <NUM> may be configured to adjust the estimated pose of the controller module <NUM> during operation based on a frame of the controller module <NUM> captured by the camera <NUM>. In particular embodiments, the identifying unit <NUM> of the central module <NUM> may determine a program to capture a frame of the controller module <NUM> based on a light condition of the environment. The program comprises, but is not limited to, capturing an IR image and a visible-light image alternately and capturing a visible-light image only. The IR image is captured by a first exposure time, and the visible-light image is captured by a second exposure time. In particular embodiments, the second exposure time may be longer than the first exposure time. The identifying unit <NUM> may then instruct the camera <NUM> to take a frame/image of the controller module <NUM> based on the determination, and the camera <NUM> would provide the identifying unit <NUM> a specific frame according to the determination. In particular embodiments, the identifying unit <NUM> may also instruct the controller module <NUM> to switch off the light unit <NUM> specific to a certain light condition, e.g., another LED source nearby, to save power.

The identifying unit <NUM> identifies the frame upon the receipt from the camera <NUM>. In particular, the identifying unit <NUM> may receive a frame whichever is being captured first when the controller module <NUM> requires a readjustment of its pose. For example, the camera <NUM> captures an IR image and a visible-light image alternately at a slow rate, e.g., a frequency of <NUM>, and then sends a frame to the identifying unit <NUM> when the controller module <NUM> is within the FOV of the camera <NUM>. Therefore, the frame being captured could be either the IR image or the visible-light image. In particular embodiments, the identifying unit <NUM> may identify a pattern formed by the light unit <NUM> of the controller module <NUM> in the captured frame. The pattern formed by the light unit <NUM> may indicate that a position of the controller module <NUM> relative to the user/the central module <NUM> and/or the environment. For example, in response to a movement/rotation of the controller module <NUM>, the pattern of the light unit <NUM> changes. In particular embodiments, the identifying unit <NUM> may identify predetermined features for tracking the controller module <NUM> in the captured frame. For example, the predetermined features of the controller module <NUM> may comprise a user's hand gesture when holding the controller module <NUM>, so that the predetermined features may indicate a position of the controller module <NUM> relative to the user/the central module <NUM>. The identifying unit <NUM> may then send the identified frames to the filter unit <NUM> for an adjustment of the pose of the controller module <NUM>. In particular embodiments, the identifying unit <NUM> may also send the identified frames to the tracking unit <NUM> for tracking the controller unit <NUM>.

The filter unit <NUM> generates a pose of the controller module <NUM> based on the received raw IMU data <NUM>. In particular embodiments, the filter unit <NUM> generates the pose of the controller module <NUM> at a faster rate than a rate of capturing a frame of the controller module. For example, the filter unit <NUM> may estimate and update the pose of the controller module <NUM> at a rate of <NUM>. The filter unit <NUM> then realign/readjust the pose of the controller module <NUM> based on the identified frames. In particular embodiments, the filter unit <NUM> may adjust the pose of the controller module <NUM> based on the pattern of the light unit <NUM> of the controller module <NUM> in the identified frame. In particular embodiments, the filter unit <NUM> may adjust the pose of the controller module <NUM> based on the predetermined features identified in the frame.

In particular embodiments, the tracking unit <NUM> may determine correspondence data based on the predetermined features identified in different frames. The correspondence data may comprise observations and measurements of the predetermined feature, such as a location of the predetermined feature of the controller module <NUM> in the environment. Furthermore, the tracking unit <NUM> may also perform a stereo computation collected near the predetermined feature to provide additional information for the central module <NUM> to track the controller module <NUM>. In addition, the tracking unit <NUM> of the central module <NUM> may request a live map from the cloud <NUM> corresponding to the correspondence data. In particular embodiments, the live map may comprise map data <NUM>. The tracking unit <NUM> of the central module <NUM> may also request a remote relocalization service <NUM> for the controller module <NUM> to be located in the live map locally or globally.

Furthermore, the filter unit <NUM> may estimate a state of the controller module <NUM> based on the correspondence data and the raw IMU data <NUM>. In particular embodiments, the state of the controller module <NUM> may comprise a pose of the controller module <NUM> relative to an environment which is built based on the frames captured by the camera <NUM>, e.g., a map built locally. In addition, the filter unit <NUM> may also send the state information of the controller module <NUM> to the cloud <NUM> for a global localization or an update of the map stored in the cloud <NUM> (e.g., with the environment built locally).

<FIG> illustrates an example method <NUM> for capturing an IR image based on a first light condition in an environment, in accordance with certain embodiments. A controller module of a tracking system may be implemented in the wearable device (e.g., a remote controller with input buttons, a smart puck with touchpad, etc.). A central module of the tracking system may be provided to or displayed on any computing system (e.g., an end user's device, such as a smartphone, virtual reality system, gaming system, etc.), and be paired with the controller module implemented in the wearable device. The method <NUM> may begin at step <NUM> receiving, from the wearable device, motion data captured by one or more motion sensors of the wearable device. In particular embodiments, the wearable device may be a controller. In particular embodiments, the wearable device may be equipped with one or more IMUs and one or more IR LEDs.

At step <NUM>, the method <NUM> may generate, at the central module, a pose of the wearable device based on the motion data sent from the wearable device.

At step <NUM>, the method <NUM> may identify, at the central module, a first light condition of the wearable device. In particular embodiments, the first light condition may comprise one or more of an indoor environment, an environment having dim light, an environment without a light source similar to the IR LEDs of the wearable device, and a camera of the central module being able to capture a pattern of IR LEDs of the wearable device for tracking.

At step <NUM>, the method <NUM> may capture a first frame of the wearable device by a camera using a first exposure time. In particular embodiments, the first frame may be an IR image. In particular embodiments, the pose of the wearable device may be generated at a faster frequency than a frequency that the first frame is captured.

At step <NUM>, the method <NUM> may identify, in the first frame, a pattern of lights disposed on the wearable device. In particular embodiments, the pattern of lights may be composed of the IR LEDs of the wearable device.

<FIG> illustrates an example method <NUM> for adjusting the pose of a wearable device by capturing the IR image and the visible-light image alternately based on the first light condition in the environment, in accordance with certain embodiments. The method <NUM> may begin, at step <NUM> follows the step <NUM> in the method <NUM>, capturing a second frame of the wearable device by the camera using a second exposure time. In particular embodiments, the second exposure time may be longer than the first exposure time. In particular embodiments, the second frame may be a visible-light image. For example, the visible-light image may be an RGB image. In particular embodiments, the pose of the wearable device may be generated at a faster frequency than a frequency that the second frame is captured.

At step <NUM>, the method <NUM> may identify, in the second frame, predetermined features of the wearable device. In particular embodiment, the predetermined features may be predetermined during the initialization/startup phase for the controller module. In particular embodiments, the predetermined features may be painted on (e.g., via small QR codes) in the controller module. In particular embodiments, the predetermined features may be reliable features for tracking the wearable device determined from previous operations. For example, the reliable feature may be a feature identified repeatedly in the previous frames for tracking the wearable device.

At step <NUM>, the method <NUM> may adjust the pose of the wearable device in the environment based on at least one of (<NUM>) the identified pattern of lights in the first frame or (<NUM>) the identified predetermined features in the second frame. In particular embodiments, the method may adjust the pose of the wearable device based on the identified pattern of lights or the identified predetermined feature whichever is captured/identified first. In particular embodiments, the method may train or update neural networks based on the process of adjusting the pose of the wearable device. The trained neural networks may further be used in tracking and/or image refinement.

In particular embodiments, the method <NUM> may further capture a third frame of the wearable device by the camera using the second exposure time, identify, in the third frame, one or more features corresponding to the predetermined features of the wearable device, determine correspondence data between the predetermined features and the one or more features, and track the wearable device in the environment based on the correspondence data.

In particular embodiments, the computing system may comprise the camera configured to capture the first frame and the second frame of the wearable device, an identifying unit configured to identify the pattern of lights and the predetermined features of the wearable device, and a filter unit configured to adjust the pose of the wearable device. In particular embodiments, the central module may be located within a head-mounted device, and the controller module may be implemented in a controller separated from the head-mounted device. In particular embodiments, the head-mounted device may comprise one or more processors, and the one or more processors are configured to implement the camera, the identifying unit, and the filter unit.

In particular embodiments, the method <NUM> may be further configured to capture the first frame of the wearable device using the first exposure time when the environment has the first light condition. In particular embodiments, the method <NUM> may be further configured to capture the second frame of the wearable device using the second exposure time when the environment has a second light condition. The second light condition may comprise one or more of an environment having bright light, an environment having a light source to interfere the pattern of lights of the wearable device, and the camera not being able to capture the pattern of lights.

Particular embodiments may repeat one or more steps of the method of <FIG>, where appropriate. Although this disclosure describes and illustrates particular steps of the method of <FIG> as occurring in a particular order, this disclosure contemplates any suitable steps of the method of <FIG> occurring in any suitable order. Moreover, although this disclosure describes and illustrates an example method for local localization including the particular steps of the method of <FIG>, this disclosure contemplates any suitable method for local localization including any suitable steps, which may include all, some, or none of the steps of the method of <FIG>, where appropriate. Furthermore, although this disclosure describes and illustrates particular components, devices, or systems carrying out particular steps of the method of <FIG>, this disclosure contemplates any suitable combination of any suitable components, devices, or systems carrying out any suitable steps of the method of <FIG>.

<FIG> illustrates an example method <NUM> for adjusting a pose of the wearable device by capturing a visible-light image based on a second light condition in an environment, in accordance with certain embodiments. A controller module of a tracking system may be implemented in the wearable device (e.g., a remote controller with input buttons, a smart puck with touchpad, etc.). A central module of the tracking system may be provided to or displayed on any computing system (e.g., an end user's device, such as a smartphone, virtual reality system, gaming system, etc.), and be paired with the controller module implemented in the wearable device. The method <NUM> may begin at step <NUM> receiving, from the wearable device, motion data captured by one or more motion sensors of the wearable device. In particular embodiments, the wearable device may be a controller. In particular embodiments, the wearable device may be equipped with one or more IMUs and one or more IR LEDs.

At step <NUM>, the method <NUM> may identify, at the central module, a second light condition of the wearable device. In particular embodiments, the second light condition may comprise one or more of an environment having bright light, an environment having a light source similar to the IR LEDs of the wearable device, and the camera not being able to capture the pattern of lights.

At step <NUM>, the method <NUM> may capture a second frame of the wearable device by the camera using a second exposure time. In particular embodiments, the second frame may be a visible-light image. For example, the visible-light image may be an RGB image. In particular embodiments, the pose of the wearable device may be generated at a faster frequency than a frequency that the second frame is captured.

At step <NUM>, the method <NUM> may adjust the pose of the wearable device in the environment based on the identified predetermined features in the second frame.

In particular embodiments, the computing system may comprise the camera configured to capture the first frame and the second frame of the wearable device, an identifying unit configured to identify the pattern of lights and the predetermined features of the wearable device, and a filter unit configured to adjust the pose of the wearable device. In particular embodiments, the central module may be located within a head-mounted device, and the controller module may be implemented in a controller separated from the head-mounted device. In particular embodiments, the head-mounted device may comprise one or more processors, and the one or more processors are configured to implement the camera, the identifying unit, and a filter unit.

In particular embodiments, the method <NUM> may be further configured to capture the second frame of the wearable device using the second exposure time when the environment has a second light condition. The second light condition may comprise one or more of an environment having bright light, an environment having a light source to interfere the pattern of lights of the wearable device, and the camera not being able to capture the pattern of lights.

Particular embodiments may repeat one or more steps of the method of <FIG>, where appropriate. Although this disclosure describes and illustrates particular steps of the method of <FIG> as occurring in a particular order, this disclosure contemplates any suitable steps of the method of <FIG> occurring in any suitable order.

<FIG> illustrates an example computer system <NUM>. In particular embodiments, one or more computer systems <NUM> perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems <NUM> provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems <NUM> performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems <NUM>. Herein, reference to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, reference to a computer system may encompass one or more computer systems, where appropriate.

In particular embodiments, processor <NUM> includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor <NUM> may retrieve (or fetch) the instructions from an internal register, an internal cache, memory <NUM>, or storage <NUM>; decode and execute them; and then write one or more results to an internal register, an internal cache, memory <NUM>, or storage <NUM>. In particular embodiments, processor <NUM> may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor <NUM> including any suitable number of any suitable internal caches, where appropriate. As an example and not by way of limitation, processor <NUM> may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in the instruction caches may be copies of instructions in memory <NUM> or storage <NUM>, and the instruction caches may speed up retrieval of those instructions by processor <NUM>. Data in the data caches may be copies of data in memory <NUM> or storage <NUM> for instructions executing at processor <NUM> to operate on; the results of previous instructions executed at processor <NUM> for access by subsequent instructions executing at processor <NUM> or for writing to memory <NUM> or storage <NUM>; or other suitable data. The data caches may speed up read or write operations by processor <NUM>. The TLBs may speed up virtual-address translation for processor <NUM>. In particular embodiments, processor <NUM> may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor <NUM> including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor <NUM> may include one or more arithmetic logic units (ALUs); be a multicore processor; or include one or more processors <NUM>. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

In particular embodiments, memory <NUM> includes main memory for storing instructions for processor <NUM> to execute or data for processor <NUM> to operate on. As an example and not by way of limitation, computer system <NUM> may load instructions from storage <NUM> or another source (such as, for example, another computer system <NUM>) to memory <NUM>. Processor <NUM> may then load the instructions from memory <NUM> to an internal register or internal cache. To execute the instructions, processor <NUM> may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor <NUM> may write one or more results (which may be intermediate or final results) to the internal register or internal cache. Processor <NUM> may then write one or more of those results to memory <NUM>. In particular embodiments, processor <NUM> executes only instructions in one or more internal registers or internal caches or in memory <NUM> (as opposed to storage <NUM> or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory <NUM> (as opposed to storage <NUM> or elsewhere). One or more memory buses (which may each include an address bus and a data bus) may couple processor <NUM> to memory <NUM>. Bus <NUM> may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor <NUM> and memory <NUM> and facilitate accesses to memory <NUM> requested by processor <NUM>. In particular embodiments, memory <NUM> includes random access memory (RAM). This RAM may be volatile memory, where appropriate. Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory <NUM> may include one or more memories <NUM>, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

In particular embodiments, communication interface <NUM> includes hardware, software, or both providing one or more interfaces for communication (such as, for example, packet-based communication) between computer system <NUM> and one or more other computer systems <NUM> or one or more networks. As an example and not by way of limitation, communication interface <NUM> may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface <NUM> for it. As an example and not by way of limitation, computer system <NUM> may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system <NUM> may communicate with a wireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (such as, for example, a Global System for Mobile Communications (GSM) network), or other suitable wireless network or a combination of two or more of these. Computer system <NUM> may include any suitable communication interface <NUM> for any of these networks, where appropriate. Communication interface <NUM> may include one or more communication interfaces <NUM>, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

According to various embodiments, an advantage of features herein is that a pose of a controller associated with a central module in a tracking system can be efficiently realigned during operation. The central module can realign the controller based on either an IR constellation tracking or a VIO-based tracking, such that the central module may track the controller in real-time and accurately without any restrictions from the environment. Particular embodiments of the present disclosure also enable to track the controller when LEDs disposed on the controller fail. Furthermore, when the central module determines that the IR constellation tracking is compromised, the central module can switch off the LEDs on the controller for power saving. Therefore, particular embodiments disclosed in the present disclosure may provide an improved, power-efficient tracking method for the controller.

While processes in the figures may show a particular order of operations performed by certain embodiments of the invention, it should be understood that such order is exemplary (e.g., alternative embodiments may perform the operations in a different order, combine certain operations, overlap certain operations, etc.).

Claim 1:
A method (<NUM>, <NUM>) comprising, by a computing system (<NUM>):
receiving (<NUM>) motion data captured by one or more motion sensors of a wearable device;
generating (<NUM>) a pose of the wearable device based on the motion data;
determining (<NUM>) that an environment of the device has a first light condition;
responsive to determining the first light condition, capturing (<NUM>) a first frame of the wearable device by a camera (<NUM>, <NUM>, <NUM>) using a first exposure time;
identifying (<NUM>), in the first frame, a pattern of lights disposed on the wearable device;
determining that an environment of the device has a second light condition;
responsive to determining the second light condition, capturing (<NUM>) a second frame of the wearable device by the camera (<NUM>, <NUM>, <NUM>) using a second exposure time that is longer than the first exposure time;
identifying (<NUM>), in the second frame, predetermined features of the wearable device; and
adjusting (<NUM>) the pose of the wearable device in an environment based on (<NUM>) the identified pattern of lights in the first frame and (<NUM>) the identified predetermined features in the second frame.