Patent Description:
Existing SLAM techniques often use some type of camera configuration that contains one or more cameras, each camera containing one or more optical sensors. A variety of optical sensor types may be used, including one-dimensional (1D), 2D, 3D, etc. For example, in a case where two fixed-point 2D cameras are employed, each camera may detect a target object. The SLAM application's algorithm may determine the distance between the target object and each camera and triangulate this information to determine the location (e.g.,. position) of the target object. Often, especially, when the target may be moving and/or rotating through space, an inertial measurement unit (IMU) may also be employed, along with a camera apparatus, to detect the target's linear acceleration and rotation rate. The combination of sensor input from both an IMU and a camera apparatus helps to enable a higher degree of location-tracking accuracy. Other SLAM techniques also exist, including radar SLAM, WiFi-SLAM, etc..

However, there are at least two problems with existing SLAM-based applications. First, when employing an IMU to track the acceleration and rotation of a moving target through space, the IMU location-tracking accuracy degrades continuously over time due at least to offset error (also known as "drift"), thus decreasing the accuracy of SLAM techniques. Second, while combining sensory data received from other equipment (e.g., cameras) can help improve the SLAM algorithm's accuracy, this equipment currently can add significant cost and require a complex setup procedure. Therefore, there is a need to improve existing SLAM techniques by providing a lower-cost and yet highly accurate location-tracking solution. Previously proposed arrangements are disclosed in <CIT>, <CIT>, and <NPL>.

Generally, techniques for determining user device location are described. The invention is defined by the appended set of claims, which provided for a video game system according to claim <NUM>, a user device according to claim <NUM>, and a non-transitory computer-readable storage medium according to claim <NUM>. The dependent claims define preferred realizations of the inventions.

An understanding of the nature and the advantages of the embodiments disclosed and suggested herein may be realized by reference to the remaining portions of the specification and the attached drawings.

Generally, systems and methods for utilizing a thermal sensor array to determine a user device location are described. Typically, a user may operate a user device (e.g., a video game controller, headset, paddle, etc.) within a 3D space, the 3D space being mapped to a portion of a physical space (e.g., a video game room). The user may interact with the user device by moving it in different directions and at different speeds within the 3D space. The user device (e.g., a video game controller, a virtual reality (VR) headset, etc.) will also typically interact with a computing hub (e.g., a video game console), which may in turn interact with other devices and cause them to perform a function (e.g., change a video image being displayed on a television (TV)). One of the interactions the user device may have with the computing hub is to determine its location within the 3D space and send the location to the computing hub. The user device may determine its location by utilizing a thermal sensor array of the user device. The thermal sensor array may be configured to receive thermal signals from one or more thermal beacons (e.g., IR LEDs), which are positioned in the physical space (e.g., affixed to the walls of the game room). Based on thermal data received from the thermal sensor array, the user device may determine its location in the 3D space, and transmit that location to the computing hub. The user device may also use the thermal data, together with other sensor data to determine its location.

In an example, the user device may also include and receive data from an IMU. The user device may also have stored a previous location (e.g., last determined location). In addition to the thermal data received from the thermal sensor array, the user device may input the IMU data and previous location data into a fusion sensor algorithm. The fusion sensor algorithm may use the different data inputs to determine the user device location. The user device may then store that location data to be used in the future, for example, as a previous location data.

The above examples are provided for illustrative purposes. The embodiments of the present disclosure are not limited as such. The embodiments similarly apply to a larger number of SLAM applications using thermal sensor arrays to determine location. These and other embodiments are further described herein next.

<FIG> illustrates an example of a system <NUM>, which includes a user device that further includes a thermal sensor array for use in determining a location of the user device, according to an embodiment of the present disclosure. In an example, the location determination implements a SLAM application In <FIG>, a gaming room <NUM> is depicted, wherein a user <NUM> wears a gaming headset <NUM> (e.g.,. the user device) to interact with a video game console <NUM>. The video game console in turn may cause, for example, a TV <NUM> to display movement of an object on the screen which corresponds to the gaming headset's <NUM> movement. It should be understood that although <FIG> and subsequent illustrations may depict a SLAM application within a gaming environment, the use of this type of scenario should not be construed to pose a limitation on the scope of the disclosure. For example, in some embodiments, a physical space such as a meeting room or theater space may be used and other location determination applications are possible. In general, and continuing with the gaming room example, the gaming room <NUM> may allow for a plurality of thermal beacons <NUM>-<NUM> (e.g., IR LEDs) to be positioned in the room <NUM>.

The user device <NUM> should include one or more thermal sensors (e.g., thermal sensor array or thermopile array) that are configured to receive thermal signals from one or more thermal beacons <NUM>-<NUM> positioned in the room <NUM>. It should be understood that, although a thermopile array is type of thermal sensor that is depicted in <FIG> and subsequent figures, other types of thermal sensors may be used as a suitable thermal sensor. Furthermore, although gaming headset <NUM> is a type of user device that is depicted below, other types of mobile computing devices that include thermal sensors may be used as a suitable user device. This may include, but is not limited to, a video game controller, a gaming paddle, a mobile phone, a laptop, etc..

In some embodiments, each beacon of the plurality of thermal beacons <NUM>-<NUM> positioned in the room <NUM> may emit a thermal signal (e.g., infrared light). Optionally, each thermal signal emitted by a thermal beacon may include an identifier that is unique to the particular thermal beacon. For example, the identifier may be a modulated frequency of the infrared signal, which the thermopile array of user device <NUM> may be able to detect as a unique signal that corresponds to a particular thermal beacon in the room. As discussed in more detail below, in some embodiments, user device <NUM> may be able to improve its location-tracking accuracy by triangulating its position relative to at least two thermal beacons in the room <NUM>.

In some embodiments, the plurality of thermal beacons <NUM>-<NUM> may be positioned in the room <NUM> such that, for a particular position of the user <NUM> within the 3D space of the room <NUM>, the thermopile array of the user device <NUM> may be able to detect thermal signals from at least two thermal beacons of the plurality within the thermopile array's field of view. For example, as depicted in <FIG>, thermal beacons <NUM>-<NUM> may each be affixed to a wall of the gaming room <NUM>, and be substantially equally spaced from each other. During a calibration process (discussed in detail below), the user <NUM> may be instructed to move the user device <NUM> in an arc-like (e.g., <NUM> degree) motion. This calibration process may allow the user device <NUM> to construct and store on the user device <NUM> a 3D model of the 3D space, the 3D space corresponding to at least a portion of the room <NUM>, and in which the user <NUM> may operate the user device <NUM> within. The calibration process may also verify that the thermopile array's field of view can detect at least two signals from a particular point within the 3D space in the room <NUM>. In other embodiments, and depending on the type of SLAM application (e.g., the range of use intended for the user device within the room <NUM>), two or more thermal beacons may be positioned in the physical space <NUM> in order for the user device <NUM> to determine its location.

<FIG> illustrates an example close-up view of a user device <NUM> which receives thermal signals from one or more thermal beacons. The user device <NUM> is a gaming headset and may correspond to the user device <NUM> of <FIG>. In an example, the headset <NUM> is a virtual reality (VR) or augmented reality (AR) headset. Generally, the headset <NUM> includes a housing <NUM> that may integrate components such as a display, processing units, memories, audio systems, I/O ports, graphics processing units, network communications devices, and other electronic, electrical, and mechanical components. The housing <NUM> further integrates (e.g., houses, attaches, or holds) additional components for location tracking, such that the additional components are rigidly connected with the housing <NUM>. These components include, for instance, an IMU <NUM> and a thermopile array <NUM>.

In an example, the IMU <NUM> includes accelerometer(s), gyroscope(s), and magnetometer(s). An accelerometer(s) measures movement along the X, Y, and Z axes. Generally, a gyroscope(s) measures <NUM> degree rotation. A magnetometer(s) determines orientation towards a magnetic field. As such, inertial data (e.g., including acceleration data, orientation data, and/or rotation data) indicative of a rotational motion of the headset <NUM> can be generated from readings of these sensors. Translational motion can also be generated based on speed and time of the user's <NUM> head motion via the headset <NUM>. For instance, a motion vector is defined. Speed is measured from the acceleration and distance is measured as a function of speed and time. Direction is derived from the rotation and orientation. The motion vector defines the distance and direction of the motion, thereby allowing a tracking of the translational motion of the headset <NUM> along the X, Y, and Z axes. Thus, by defining inertial data, distance, and direction in a motion vector, the motion of the user's headset <NUM> can be tracked over time. Accordingly, based on the inertial data, and by performing integration(s) with respect to time, the location of headset <NUM> may be determined for a specific point in time in 3D space. A processing unit of the IMU <NUM> (e.g., a signal processor) may generate location data from data sensed by such IMU sensors.

As described above, one of the limitations with utilizing an IMU to determine location is that an IMU typically suffers from accumulated error. Because a SLAM application may be continually integrating acceleration with respect to time to calculate velocity and position, any measurement errors, however small, are accumulated over time, leading to drift. Drift is measurement of the difference between where a user device may initially determine it is located compared to its actual location. As such, the headset <NUM> also may contain a thermopile array <NUM>, which the headset <NUM> can use to continually correct for drift errors and determine a more accurate location.

In an example, the thermopile array <NUM> is thermal infrared sensor array that is configured to measure temperature of one or more thermal beacons <NUM>, <NUM> from a distance by detecting the infrared energy from the one or more thermal beacons <NUM>, <NUM> (which may correspond to one or more of the thermal beacons <NUM>-<NUM> of <FIG>). The thermopile array may include thermocouples that are connected on a silicon chip, which are configured to convert thermal energy received from a thermal signal of a thermal beacon <NUM>, <NUM> into electrical energy in the form of a voltage output. Typically, the input thermal energy is proportional to the voltage output. Thus, because a user device <NUM> may detect more light energy (e.g., signal) the closer it is to a thermal beacon (e.g., creating a higher temperature differential), a thermopile array can be used to measure the distance between a user device <NUM> and a particular thermal beacon. In some embodiments, a processing unit connected to the thermopile array <NUM> (e.g., a signal processor which be integrated with the thermopile array <NUM>) may transmit thermal data, corresponding to distance from the user device <NUM> to the thermal beacon <NUM>, <NUM>, based on the voltage output by an element of the thermopile array <NUM>. In some embodiments, the thermopile may be composed of a single element (e.g., pixel), dual element, etc. In other embodiments, the thermopile may be a linear (e.g., <NUM>, <NUM>) or area (e.g., 32x32) array of pixels. Although the embodiments discussed herein discuss a thermopile array, this disclosure should not be construed to be so limiting. In some embodiments, any given pixel(s) of the thermopile array <NUM> may be able to detect thermal signals from one or more thermal beacons at a given time, provided that the thermal beacon is within the pixel's field of view. As discussed above, and further below in regards to <FIG>, the thermal data generated from the thermopile array <NUM> may be used to triangulate the distance between the user device <NUM> and at least two thermal beacons <NUM>, <NUM> to determine a 3D location of the user device <NUM> in a physical space <NUM>. In some embodiments, the thermal data from the thermopile array may also be combined with the IMU data to output a 3D location of the user device <NUM> in a physical space <NUM>.

<FIG> is a block diagram <NUM> of an example architecture for a user device <NUM> (which may correspond to user device <NUM> of <FIG> and/or user device <NUM> of <FIG>) and utilizing a thermopile array to implement SLAM, according to an embodiment of the present disclosure. The user device <NUM> may include at least one memory <NUM>, one or more processing units (or processor(s)) <NUM>, a thermopile array <NUM>, an IMU <NUM>, and a communications device <NUM>, among other components. The processor(s) <NUM> may be implemented as appropriate in hardware. The thermopile array <NUM> of the user device <NUM> may be configured to detect one or more thermal signals (e.g., infrared light) from one or more thermal beacons <NUM>, <NUM>. The communications device <NUM> may further be configured to communicate with a computing hub <NUM> (e.g., a video game console, virtual meeting server, etc.) using any suitable communication path. This may include, for instance, a wire or cable, fiber optics, a telephone line, a cellular link, a radio frequency (RF) link, a WAN or LAN network, the Internet, or any other suitable medium.

The memory <NUM> may store program instructions that are loadable and executable on the processor(s) <NUM>, as well as data generated during the execution of these programs. Depending on the configuration and type of user device <NUM>, the memory <NUM> may be volatile (such as random access memory (RAM)) and/or non-volatile (such as read-only memory (ROM), flash memory, etc.). In some implementations, the memory <NUM> may include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM) or ROM. The user device <NUM> may also include additional storage (not shown), such as either removable storage or non-removable storage including, but not limited to, magnetic storage, optical disks, and/or tape storage. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing devices.

Turning to the contents of the memory <NUM> in more detail, the memory <NUM> may include an operating system <NUM> and one or more application modules or services for implementing the features disclosed herein, including a calibration module <NUM>, a sensor fusion algorithm module <NUM>, and a thermopile array algorithm module <NUM>. It should be understood that any of the tasks performed by one module may be performed by one or more of the other modules, and, as such, the module definitions provided herein are included for illustrative purposes.

The operating system <NUM> may provide executable program instructions for the general administration and operation of user device <NUM> and typically will include a computer-readable storage medium (e.g., a hard disk, random access memory, read only memory, etc.) storing instructions that, when executed by a processor of the user device <NUM>, allow the user device <NUM> to perform its intended functions. Suitable implementations for the operating system are known or commercially available and are readily implemented by persons having ordinary skill in the art, particularly in light of the disclosure herein.

The calibration module <NUM> may be responsible for determining and maintaining in memory <NUM>, a 3D model of a 3D space, wherein the 3D space may correspond to at least a portion of a physical space (e.g., gaming room <NUM>). In some embodiments, the calibration module <NUM> may also be responsible for verifying that the thermopile array <NUM> of the user device <NUM>, for a given position within the 3D space, can detect a thermal signal from at least two beacons of the plurality of beacons <NUM>-<NUM> (e.g., the beacons are within the field of view of one or more elements of the thermopile array). This helps to enable triangulation of location via the thermopile array algorithm module <NUM>, discussed below. In some embodiments, the calibration module <NUM> may perform operations as described in <FIG>. In one embodiment, and using <FIG>'s gaming room <NUM> as an example, the 3D model (which may be represented as a 3D wireframe model) corresponds to the 3D space in which the user <NUM> expects to operate the user device <NUM> within. For example, the user may only intend to operate the user device <NUM> within a portion of the gaming room near the center (and radiating out from the center some distance that is reachable by a human). The calibration module <NUM> may instruct the user to begin the calibration by moving the device to the center of the room <NUM> (e.g., where they primarily intend to operate the device) and then, while standing and turning in place, move the device <NUM> in a <NUM> degree visual sweep of the room, divided into smaller arc segments (e.g., <NUM> degree turns). At each segment, as described above, the user device <NUM> may verify at that at least two thermal beacons are within the field of view of the thermopile. Additionally, the user device <NUM> may determine and store in memory <NUM> the distance between the user device <NUM> and each thermal beacon <NUM>, <NUM> visible within the arc segment. Once the user device <NUM> completes the sweep of the room, the calibration module <NUM> may use the location data determined between the user device <NUM> and each of the beacons <NUM>-<NUM> (relative to the center <NUM> of the room) to construct a 3D model of the 3D space. In some embodiments, the origin of the X, Y, and Z axes may be at the position (e.g., center <NUM>) where the user <NUM> initially positioned the user device <NUM> during calibration. In some embodiments, a full <NUM> degree visual sweep may not be needed to perform calibration. For example, if the user <NUM> will typically be operating the user device <NUM> facing forward towards the TV <NUM>, the calibration may only recommend a partial (e.g., <NUM> degree) sweep of the room. In that case, a smaller number of thermal beacons may be needed to pre-position in the room in order to construct the 3D model.

The sensor fusion algorithm module <NUM> may be responsible for determining a location of the user device <NUM>. In some embodiments, the sensor fusion module <NUM> may be performed after calibration <NUM> has been performed. The sensor fusion module <NUM> may combine sensory data input from one or more sources to improve accuracy in determining the location of the user device <NUM>. In an embodiment, the sensor fusion module <NUM> may receive thermal data as sensory input from the thermopile array <NUM> and IMU data as sensory input from the IMU <NUM>.

In some embodiments, before combining the sensory input data from the different sensors, the sensor fusion module <NUM> may first execute a thermopile array algorithm module <NUM>. The thermopile array algorithm module <NUM> may be responsible for triangulating, based on two or more distance values (e.g., corresponding to the distances between the user device <NUM> and at least two thermal beacons <NUM>, <NUM>), the location of the user device <NUM>. The location (e.g.,. position) may be in the form of X, Y, and Z coordinates within the 3D model determined during calibration. The thermopile array algorithm module <NUM>, in addition to outputting a location, also outputs a confidence value that corresponds with the location. The confidence value may increase or decrease depending on the number of thermal beacons detected within the thermopile array's field of view (e.g., the number of distance values obtained). For example, if only one thermal beacon is detected, although the thermopile array algorithm module <NUM> may still output a location value, the corresponding confidence value may be low. Conversely, if two or more thermal beacons are detected within the field of view, the module <NUM> may output a high confidence value. In some embodiments, the user device <NUM> may execute the thermopile array algorithm module <NUM> on a certain frequency. For example, the module <NUM> may run at <NUM> (approximately every <NUM> milliseconds). Typically, a higher frequency of generating an updated location based on the thermal data will improve location-tracking accuracy. For example, when fusing both thermal data and IMU data (discussed further below), more frequent location information based on thermal data will help to correct against drift error within the IMU data.

In another embodiment, and returning to the sensor fusion module <NUM> discussed above, the module <NUM> may receive location information (X, Y, and Z coordinates) from the thermopile array algorithm module <NUM> (e.g., after the algorithm <NUM> used the thermal data to determine a 3D location value and a corresponding confidence value) as well as location information (X, Y, and Z coordinates) from the IMU data from the IMU <NUM>. The sensor fusion module <NUM> may combine (or "fuse") the different location data together using one or more of a number of algorithms and output a single location in 3D space with higher accuracy.

In one embodiment, the sensor fusion module <NUM> may employ an artificial intelligence model that is trained to utilize sensor data from disparate sources to determine a location of the user device <NUM>. As used herein, the term "artificial intelligence" refers to any suitable computer-implemented artificial intelligence technique including machine learning (supervised or unsupervised), natural language processing, machine perception, computer vision, affective computing, statistical learning and classification (including use of hidden Markov models and Bayesian network models), reinforcement learning including neural networks, search algorithms and optimization algorithms (including evolutionary computing) and automated reasoning. As an example, a neural network may be trained to receive the IMU data, thermal data, and a previous location(s) data as input. This information may be used to output a corrected location of the user device <NUM> and/or to predict a next location where the user device <NUM> will be within a certain time interval.

In another embodiment, the sensor fusion module <NUM> may employ a fixed-point algorithm, such as employing a Kalman filter. The Kalman filter may use the sensory input data from the thermopile array, the IMU, and previous location data, to build a prediction model. The prediction model may take into account state variables (e.g., previous location(s), frequency of polling for thermal data, previous measurements of drift error, etc.).

<FIG> illustrates an example flow <NUM> for performing calibration of a user device <NUM> of the system <NUM> of <FIG>, according to embodiments of the present disclosure. Although the operations are illustrated in a particular order, some of the operations can be reordered or omitted. Also, some or all of the flow <NUM> (or any other flows described herein, or variations, and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware or combinations thereof. The code may be stored on a computer-readable storage medium, for example, in the form of a computer program including a plurality of instructions executable by one or more processors. The computer-readable storage medium may be non-transitory.

In an example, the flow includes an operation <NUM>, which involves positioning thermal beacons <NUM>-<NUM> within a physical space <NUM>, which may be a gaming environment. Using the example of a video game system <NUM>, a video game console <NUM> may prompt the user <NUM> (e.g., visually on a TV screen, audibly via a headset, or some other mechanism) to begin the calibration process by positioning thermal beacons <NUM>-<NUM> in the physical space <NUM> (e.g., game room). Depending on the type of gaming application, the video game console <NUM> may instruct the user on how many thermal beacons should be used and what degree of coverage (e.g., <NUM> degrees, <NUM> degrees, etc.) is recommended. For example, if the application requires the user <NUM> to be able to move the video game controller <NUM> (e.g., headset) in a <NUM> degree visual sweep of the perimeter of the room <NUM> while playing the game, then the user <NUM> may be recommended to place thermal beacons around the perimeter of the room, as depicted in <FIG>. For example, the thermal beacons <NUM>-<NUM> may be positioned both in the center of the four walls <NUM>, <NUM>, <NUM>, <NUM> and near the wall corners <NUM>, <NUM>, <NUM>, <NUM>. In this way, for a given position of the headset <NUM> within the gaming environment, the thermopile array <NUM> of the headset <NUM> may be able to detect thermal signals from at least two thermal beacons <NUM>, <NUM> within its field of view.

In an example, the flow includes an operation <NUM>, which involves receiving an instruction that requests the user to move the user device <NUM> to a first (or next) location. It should be understood that this instruction may come from directly from the user device <NUM> itself (e.g., an audio command from a headset), or from a computing hub <NUM> (e.g., video game console) that is communicatively connected to the user device <NUM>. In the case where the instruction may come from the computing hub <NUM>, the user device <NUM> may iterate through a next step (e.g., operation <NUM>, discussed below) in the calibration process and then send a confirmation message (or information/error message) to the computing hub <NUM>, which in turn may relay the next instruction message to the user <NUM>.

In the case where the user <NUM> is being instructed to move the user device <NUM> to a first location, and continuing with the gaming example above, the headset <NUM> (e.g., via audio instruction, flashing light, etc.) may instruct the user <NUM> to move to the center of the room <NUM>, which the user <NUM> intends to be the center of the gaming experience. The headset <NUM> may use this position <NUM> as the origin of the 3D model that the headset <NUM> builds and stores on the headset <NUM> as a result of the calibration process <NUM>. It should be understood that the optimal location of the origin may depend on the type of application, and, in some embodiments, may not be in the center of a room. In some embodiments, the user <NUM> may then signal to the video game console <NUM> or the headset <NUM> that the user has completed the previous steps and is ready to proceed.

In an example, the flow includes an operation <NUM>, where the user device <NUM> may determine a first (or next) location of the user device <NUM> based on thermal data corresponding to thermal signals emitted by at least two thermal beacons of a plurality of thermal beacons. In the case where the user device <NUM> is determining a first location, the user <NUM> may already be in place, based on the previous operation <NUM>. Continuing with the gaming example above, the headset's calibration module <NUM> may utilize thermal data received from the thermopile array <NUM> to determine at least two distance values, respectively, between the thermopile array <NUM> and at least two beacons <NUM>, <NUM> within the thermopile array's <NUM> field of view. Based on this distance information, the calibration module <NUM> may triangulate the headset's position and store that position and the distance values in memory <NUM>. In the event that the calibration module <NUM> is unable to detect at least two beacons within the field of view, it may prompt the user <NUM> with a warning (e.g., to check the positioning of the thermal beacons).

In some embodiments, at operation <NUM>, once the calibration module <NUM> has determined a first (or next) location based on information from at least two thermal beacons within its field of view, the calibration module <NUM> may determine if the calibration is complete. In some embodiments, the calibration module <NUM> may perform this determination <NUM> by determining if it has enough information to construct a 3D model of the 3D space. In other embodiments, the calibration module <NUM> may pre-determine that it must capture all four sides of a room <NUM>, and continually prompt the user to turn to a next position until it has capture data for all four sides. The calibration module <NUM> may use any suitable mechanism to determine if the calibration is completed. If the calibration is completed, then the flow may proceed to operation <NUM>, described further below.

If the calibration procedure has not been completed, then the flow may loop back to operation <NUM>, instructing the user to move to a next location (e.g., position). Continuing with the gaming example above, the headset <NUM> may output an audible instruction, tactile feedback, or any other suitable method to indicate to the user <NUM> to move to the next location. In one embodiment, the headset <NUM> may instruct the user to "turn in place approximately <NUM> degrees and then stop and wait for further instructions. " The calibration module <NUM> may then continue with the calibration and perform operation <NUM>, as discussed above. This loop may continue until the calibration is completed.

In an example, at operation <NUM>, the calibration module <NUM> may determine that it has enough data to construct a 3D model of the physical space <NUM>, with the first location serving as the origin of the 3D model. Each of the determined distances (e.g., to the thermal beacons <NUM>-<NUM>) relative to the first location may be used to determine the dimensions of 3D model. In some embodiments, the calibration module <NUM> may use any suitable mechanism to construct the 3D model. Once the 3D model has been constructed, the calibration module <NUM> may store the model on the user device <NUM> (e.g., in memory <NUM>, or other storage).

<FIG> illustrates an example flow <NUM> for implementing SLAM on a user device that includes a thermopile array, according to embodiments of the present disclosure. In some embodiments, flow <NUM> may be performed after the calibration flow <NUM> of <FIG> is performed and a 3D model of the 3D space has been generated. Although the flow operations below discuss utilizing a paddle as a user device (e.g., instead of a headset <NUM> within the video game system of <FIG>), any suitable user device and/or system may be used.

In an example, at operation <NUM>, the user device may receive thermal data corresponding to a thermal signal emitted from a thermal beacon of a plurality of thermal beacons, the thermal signal being detected by the thermal sensor array of the user device. The user <NUM> may operate a paddle to play virtual ping-pong, whereby the game involves the user moving the paddle into different positions as the user plays the game. The user <NUM> may also rotate the paddle and swing the paddle with different rates of acceleration. As described above, depending in part on the type of application, the frequency at which the paddle may poll the thermopile array <NUM> to receive thermal data may be increased to achieve higher location-tracking accuracy with respect to time. In some embodiments, this may involve the thermopile array algorithm module <NUM> receiving a voltage reading (corresponding to a particular thermal beacon's IR light signal) from the thermopile array at a frequency of at least <NUM> (e.g., receiving an update approximately every <NUM> milliseconds).

In an example, at operation <NUM>, the user device (e.g., paddle) may determine its location in 3D space, based on the data received from the thermopile array. Specifically, in some embodiments, for each voltage reading, the thermopile array algorithm module <NUM> may compute an associated distance to a thermal beacon. As described above, the module <NUM> may then triangulate its location using thermal signals from at least two thermal beacons. In some embodiments, the module <NUM> may use the 3D model of the 3D space within the gaming room <NUM> (previously stored during the calibration process) to determine the location of the paddle in the 3D space. In some embodiments, the paddle may utilize a confidence value (e.g., generated by the thermopile array algorithm module <NUM>) to determine its location. If, for example, at a particular time, only one thermal beacon was detected, the confidence value may be lower. As such, the paddle may determine to disregard that location value, combine it with other location data to improve accuracy, or perform any other suitable behavior. It should be understood that, in some embodiments, the paddle may utilize only thermal data (apart from other sensor data) to determine the paddle's location. However, in other embodiments, the paddle may utilize, in addition to thermal data, other sensory input (e.g., IMU data) and/or variables to determine the paddle's location, which may be further input into a sensory fusion algorithm <NUM> for combining (as discussed in <FIG> below).

It should be understood that, contrasted with other SLAM applications that may require computing external to the user device <NUM> to determine the user device's location (e.g., external optical sensors, processing by a video game console, etc.), one technical advantage of the present disclosure is that it enables the user device <NUM> to determine its location using its own internal components (based in part on thermal signals received from the thermal beacons). This may improve system reliability, for example, in terms of reducing network connectivity issues such as latency, bandwidth limitations, etc..

In an example, at operation <NUM>, and continuing with the example above, the paddle may transmit the location of the paddle to the video game console <NUM>. In some embodiments, the location data may be transmitted by the communications device <NUM> using any suitable communications path (e.g., WiFi), as discussed above. The video game console <NUM> may process the location data using any suitable mechanism. In one example, the video game console <NUM> may use the location data to move an object being displayed on the TV <NUM> (that corresponds to the paddle) to a new location, wherein the new location corresponds to the change of the paddle's location in the 3D space of the gaming room <NUM>.

In an example, at operation <NUM>, the user device <NUM> may store its own location on the user device <NUM>. In some embodiments, previous location data may be in the form of X, Y, and Z coordinates that correspond to the 3D model previously generated during calibration (see <FIG>). The previous location data may be stored in memory <NUM> of the user device <NUM>. In some embodiments, the memory <NUM> may store a history of previous locations, whereby a sensor fusion algorithm <NUM> of the user device <NUM> (e.g., employing a machine learning algorithm or fixed-point algorithm) may use one or more of the historical location data points as input to determine the user device's <NUM> current location. In yet other embodiments, historical location data may be used by an algorithm to predict a future location.

<FIG> illustrates an example flow <NUM> for implementing SLAM on a user device that includes a thermopile array and an IMU, according to embodiments of the present disclosure. Similar to the flow <NUM> of <FIG>, flow <NUM> may be performed after the calibration flow <NUM> of <FIG> is performed and a 3D model of the 3D space has been generated. Also, similar to flow <NUM>, although the flow operations below discuss utilizing a paddle as a user device <NUM> within the video game system of <FIG>, any suitable user device and/or system may be used. In some embodiments, flow <NUM> includes example operations that can be implemented as sub-operations of the example flow <NUM>.

In an example, at operation <NUM>, similar to operation <NUM> of <FIG>, the user device may receive thermal data corresponding to a thermal signal emitted from a thermal beacon of a plurality of thermal beacons, the thermal signal being detected by the thermal sensor array of the user device. In some embodiments, this data may be further processed by the thermopile array algorithm module <NUM>.

In an example, at operation <NUM>, the user device may retrieve previous location data that was stored on the user device. For example, this may be data that was stored by a previous operation <NUM>. In some embodiments, previous location data may be in the form of X, Y, and Z coordinates that correspond to the 3D model previously generated during calibration (see <FIG>). In other embodiments, the previous location data may also include other data, including, but not limited to, inertial data (e.g., rotational rate, linear acceleration of the user device in 3D space at a point in time). This data may be received by one or more sensor units devices of the user device <NUM> (e.g., IMU <NUM>), as discussed further below.

In an example, at operation <NUM>, the user device may receive IMU data from an IMU <NUM> of the user device. In some embodiments, the IMU data may include acceleration data, orientation data, and/or rotation data of the user device in 3D space, as discussed in reference to IMU <NUM> of <FIG>. Accordingly, the IMU data may also be used to determine the location of the user device in 3D space at a point in time.

In an example, at operation <NUM>, the user device may determine its location in 3D space by inputting thermal data (e.g., received at operation <NUM>), previous location data (e.g., received at operation <NUM>), and/or IMU data (e.g., received at operation <NUM>) into a sensor fusion algorithm (which may correspond to the sensor fusion algorithm module <NUM> of <FIG>). In some embodiments, other sensory input from other sensors may be also used as input to the sensor fusion algorithm, including, but not limited to, a global positioning system (GPS) tracker. In some embodiments, each of the data received by the sensor fusion algorithm from operations <NUM> and <NUM> may correspond to data measuring a location of the user device at substantially the same point in time. However, in other embodiments, the data received by the sensor fusion algorithm from operations <NUM> and <NUM> may correspond to data measuring a location of the user device at different time intervals. In an example, and similar to as discussed above regarding operation <NUM>, the thermopile array algorithm <NUM> may determine the user device location based on thermal data from the thermopile array <NUM> at a frequency of <NUM> (e.g., updating every <NUM> milliseconds). In contrast and for example, a <NUM> IMU may output an acceleration rate and rotational rate for a sample period representing the total motion of the IMU over <NUM> milliseconds. The sensor fusion algorithm <NUM> may use the location data from derived from the thermopile array as a drift correction factor to correct for drift errors within the IMU location data. The sensor fusion algorithm may also utilize previous location data from operation <NUM> to increase the algorithm's accuracy, as discussed above.

In an example, at operation <NUM>, and similar to operation <NUM> from <FIG>, the user device transmits its location that it determined in operation <NUM> to the video game console.

In an example, at operation <NUM>, and similar to operation <NUM> from <FIG>, the user device may store its own location on the user device.

<FIG> illustrates an example of a hardware system suitable for implementing a computer system <NUM> in accordance with various embodiments. The computer system <NUM> represents, for example, components of a video game system, a mobile user device, a proximity device, a wearable gesture device, and/or a central computer. The computer system <NUM> includes a central processing unit (CPU) <NUM> for running software applications and optionally an operating system. The CPU <NUM> may be made up of one or more homogeneous or heterogeneous processing cores. Memory <NUM> stores applications and data for use by the CPU <NUM>. Storage <NUM> provides non-volatile storage and other computer readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. User input devices <NUM> communicate user inputs from one or more users to the computer system <NUM>, examples of which may include keyboards, mice, joysticks, touch pads, touch screens, still or video cameras, and/or microphones. Network interface <NUM> allows the computer system <NUM> to communicate with other computer systems via an electronic communications network, and may include wired or wireless communication over local area networks and wide area networks such as the Internet. An audio processor <NUM> is adapted to generate analog or digital audio output from instructions and/or data provided by the CPU <NUM>, memory <NUM>, and/or storage <NUM>. The components of computer system <NUM>, including the CPU <NUM>, memory <NUM>, data storage <NUM>, user input devices <NUM>, network interface <NUM>, and audio processor <NUM> are connected via one or more data buses <NUM>.

A graphics subsystem <NUM> is further connected with the data bus <NUM> and the components of the computer system <NUM>. The graphics subsystem <NUM> includes a graphics processing unit (GPU) <NUM> and graphics memory <NUM>. The graphics memory <NUM> includes a display memory (e.g., a frame buffer) used for storing pixel data for each pixel of an output image. The graphics memory <NUM> can be integrated in the same device as the GPU <NUM>, connected as a separate device with the GPU <NUM>, and/or implemented within the memory <NUM>. Pixel data can be provided to the graphics memory <NUM> directly from the CPU <NUM>. Alternatively, the CPU <NUM> provides the GPU <NUM> with data and/or instructions defining the desired output images, from which the GPU <NUM> generates the pixel data of one or more output images. The data and/or instructions defining the desired output images can be stored in the memory <NUM> and/or graphics memory <NUM>. In an embodiment, the GPU <NUM> includes 3D rendering capabilities for generating pixel data for output images from instructions and data defining the geometry, lighting, shading, texturing, motion, and/or camera parameters for a scene. The GPU <NUM> can further include one or more programmable execution units capable of executing shader programs.

The graphics subsystem <NUM> periodically outputs pixel data for an image from the graphics memory <NUM> to be displayed on the display device <NUM>. The display device <NUM> can be any device capable of displaying visual information in response to a signal from the computer system <NUM>, including CRT, LCD, plasma, and OLED displays. The computer system <NUM> can provide the display device <NUM> with an analog or digital signal.

In accordance with various embodiments, the CPU <NUM> is one or more generalpurpose microprocessors having one or more processing cores. Further embodiments can be implemented using one or more CPUs <NUM> with microprocessor architectures specifically adapted for highly parallel and computationally intensive applications, such as media and interactive entertainment applications.

The components of a system may be connected via a network, which may be any combination of the following: the Internet, an IP network, an intranet, a wide-area network ("WAN"), a local-area network ("LAN"), a virtual private network ("VPN"), the Public Switched Telephone Network ("PSTN"), or any other type of network supporting data communication between devices described herein, in different embodiments. A network may include both wired and wireless connections, including optical links. Many other examples are possible and apparent to those skilled in the art in light of this disclosure. In the discussion herein, a network may or may not be noted specifically.

Specific details are given in the description to provide a thorough understanding of the embodiments. However, it will be understood by one of ordinary skill in the art that the embodiments may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail in order to avoid obscuring the embodiments.

Also, it is noted that the embodiments may be described as a process which is depicted as a flow diagram or block diagram. Although each may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process may have additional steps not included in the figure.

Moreover, as disclosed herein, the term "memory" or "memory unit" may represent one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices, or other computer-readable mediums for storing information. The term "computer-readable medium" includes, but is not limited to, portable or fixed storage devices, optical storage devices, wireless channels, a sim card, other smart cards, and various other mediums capable of storing, containing, or carrying instructions or data.

Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a computer-readable medium such as a storage medium. Processors may perform the necessary tasks.

Claim 1:
A video game system (<NUM>) comprising:
a video game console (<NUM>);
a plurality of thermal beacons (<NUM>, <NUM>, <NUM>); and
a user device (<NUM>) communicatively coupled with the video game console, the user device comprising:
a thermopile array (<NUM>);
a processor; and
a memory storing instructions that, upon execution by the processor, cause the processor to:
receive thermal data from the thermopile array, the thermal data corresponding to a thermal signal emitted from each of one or more thermal beacons of the plurality of thermal beacons and detected by the thermopile array;
determine a confidence value corresponding to the thermal data, the confidence value based on a number of thermal beacons of the plurality of thermal beacons in a field of view of the thermopile array;
determine, based on the thermal data and the confidence value, a location of the user device in a three-dimensional, 3D, space; and
transmit the location of the user device to the video game console.