Patent Description:
Electromagnetic (EM) position tracking systems may use near-field EM fields, transmitters, and receivers to determine position information for particular devices. The EM position tracking systems, in general, may use transmitters that generate an EM signal (e.g., field) that is detected at remote receivers. In one example, a transmitter generates an EM field using a transmitter coil to induce a current on a receiver coil associated with a remote receiver. The receiver generates values corresponding to the EM field magnitude which are then processed to compute a position and/or orientation (i.e., pose) of the receiver relative to the transmitter.

<CIT> and <CIT> illustrate the state of the art in multisensor tracking for mixed reality, employing sensor fusion to enhance tracking accuracy by integrating data from different sensor modalities.

A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

In one general aspect, a handheld electronic device for controlling three-dimensional content displayed in a user interface of a computing device. The handheld electronic device includes an electromagnetic sensing system for detecting, for the handheld electronic device, a pose of the handheld electronic device in three-dimensional space. The handheld electronic device also includes an inertial measurement unit sensor for detecting, for the handheld electronic device, an orientation in three-dimensional space of the handheld electronic device in three-dimensional space. The handheld electronic device also includes at least one processor coupled to memory and the at least one processor is configured to generate commands to manipulate the three-dimensional content in the computing device. The commands may be generated based on a determined proximity of the handheld electronic device relative to a receiver module associated with the computing device. The determined proximity may trigger selection of data for use in generation of the commands. The data may include the pose of the electromagnetic sensing system when the determined proximity indicates that the handheld electronic device is within range of the receiver module, and the orientation of the inertial measurement unit sensor when the determined proximity indicates that the handheld electronic device is out of range of the receiver module. The handheld electronic device also includes at least one communication module to trigger transmission of the commands to manipulate the three-dimensional content displayed in the computing device based on detected changes in pose of the handheld electronic device. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

Implementations may include one or more of the following features. The handheld electronic device where the handheld electronic device is an air mouse device configured to manipulate three-dimensional computer-aided design objects displayed in the user interface of the computing device. The manipulation of the three-dimensional computer-aided design objects may be based on tracking the pose of the handheld electronic device in three-dimensional space. The handheld electronic device where the handheld electronic device is configured to prioritize use of the electromagnetic sensing system using six degrees of freedom while within range of the receiver module and switch to perform sensing in three degrees of freedom using the inertial measurement unit upon detecting that the handheld electronic device is out of range of the receiver module associated with the computing device. The handheld electronic device where the computing device is removably attached to a dongle where the dongle includes a second processor, a first communication interface with the computing device, and a second communication interface with the handheld electronic device. The dongle may be operable to collect, from the at least one processor of the handheld electronic device and using the first communication interface, data associated with the pose of the handheld electronic device, convert, using the second processor, the data from the electromagnetic sensing system or from the inertial measurement unit sensor, to the commands, and transmit the commands to the computing device using the second communication interface. In some implementations, the dongle further includes an electromagnetic receiver module to interface with a transmitter module associated with the handheld electronic device. In some implementations, the handheld electronic device is configured to communicate pose information to the dongle via a wireless protocol and the dongle is configured to communicate to the computing device via a wired protocol. In some implementations, the wireless protocol is Radio Frequency (RF) and the wired protocol is Universal Serial Bus (USB).

In some implementations, the handheld electronic device also includes a microphone configured to receive voice-based queries for communication from the handheld electronic device to the computing device and a speaker configured to generate audio playback from the handheld electronic device. The audio playback may include information responsive to the voice-based queries.

In some implementations, the handheld electronic device is detected to be out of range of the computing device if a metric correlated to signal noise associated with the electromagnetic sensing system is above a noise threshold. In some implementations, the handheld electronic device further includes a removable trackball. The trackball may include the at least one communication module. The communication module may be configured to transmit commands to control a three-dimensional object in the content according to movements of the trackball in three-dimensional space. In some implementations, a change in a detected pose associated with the trackball causes a corresponding change in a pose of the three-dimensional object. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

In another general aspect, a pose tracking system for an air mouse device is described. The pose tracking system includes an electromagnetic receiver device associated with the air mouse device and configured to determine a <NUM>-DoF pose between a remote electromagnetic transmitter and the electromagnetic receiver device. The remote electromagnetic transmitter may be associated with a computing device. The pose tracking system may also include an inertial measurement sensor configured to determine a <NUM>-DoF pose of the air mouse device. The pose tracking system may also include at least one processor coupled to memory and configured to generate commands using the <NUM>-DoF pose to manipulate three-dimensional content displayed on the computing device, in response to detecting air mouse poses while the air mouse device is communicably coupled to the electromagnetic transmitter. The at least one process may also be configured to generate commands using the <NUM>-DoF pose to manipulate three-dimensional content displayed on the computing device, in response to detecting air mouse poses while the air mouse device is beyond a predetermined range from the electromagnetic receiver device. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

Implementations may include one or more of the following features. The pose tracking system where the predetermined range is based at least in part on at least one metric that corresponds to a received signal strength detected at the electromagnetic receiver device. The pose tracking system further including at least one communication module to transmit the generated commands to the computing device to control the three-dimensional content displayed on the computing device. The pose tracking system where the air mouse device further includes a removable trackball, the trackball including an electromagnetic sensing system to detect a plurality of poses associated with the air mouse device in which the at least one processor is configured to generate the commands and to transmit commands to control the three-dimensional content according to movements of the trackball in three-dimensional space. The pose tracking system where the processor is configured to generate commands using the <NUM>-DoF pose to trigger manipulation of three-dimensional content displayed on the computing device until the air mouse device is within range of the electromagnetic receiver device and automatically switch to generating commands using the <NUM>-DoF pose to trigger manipulation of three-dimensional content displayed on the computing device when the air mouse device is within range of the electromagnetic receiver device. The air mouse device where the air mouse device is configured to manipulate three-dimensional computer-aided design objects displayed in the user interface of the computing device, the manipulation of the three-dimensional computer-aided design objects being based on tracking the position and orientation of the air mouse device in three-dimensional space. The air mouse device where the air mouse device is configured to prioritize use of the electromagnetic sensing system using six degrees of freedom while within range of the computing device and switch to perform sensing in three degrees of freedom using the inertial measurement unit upon detecting that the air mouse device is out of range of the computing device. The air mouse device where the air mouse device is detected to be out of range of the computing device if a metric correlated to signal noise associated with the electromagnetic sensing system is above a predefined noise threshold. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

In another general aspect, an air mouse device is described for controlling content displayed in a user interface of a computing device. The air mouse device may include an electromagnetic sensing system for detecting, for the air mouse device, a three-dimensional position and a three-dimensional orientation responsive to detected movements of the air mouse device in three-dimensional space and an inertial measurement unit sensor for detecting, for the air mouse device, a three-dimensional orientation in three-dimensional space. The air mouse device may also include at least one processor coupled to memory, the at least one processor configured to generate commands to manipulate the content in the computing device. The commands may be generated based on a determined proximity of the air mouse device relative to the computing device. the determined proximity may trigger selection of data for use in generation of the commands, where the data includes the three-dimensional position and the three-dimensional orientation associated with the electromagnetic sensing system when the determined proximity indicates that the air mouse device is within range of the computing device, and the three-dimensional orientation of the inertial measurement unit sensor when the determined proximity indicates that the air mouse device is out of range of the computing device. The air mouse device may also include at least one communication module to trigger transmission of the commands to manipulate the content displayed in the computing device. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

Implementations may include one or more of the following features. The air mouse device where the air mouse device is configured to manipulate three-dimensional computer-aided design objects displayed in the user interface of the computing device, the manipulation of the three-dimensional computer-aided design objects being based on tracking the position and orientation of the air mouse device in three-dimensional space. The air mouse device where the air mouse device is configured to: prioritize use of the electromagnetic sensing system using six degrees of freedom while within range of the computing device and switch to perform sensing in three degrees of freedom using the inertial measurement unit upon detecting that the air mouse device is out of range of the computing device. The air mouse device where the air mouse device is detected to be out of range of the computing device if a metric correlated to signal noise associated with the electromagnetic sensing system is above a predefined noise threshold. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

This document describes example input devices and tracking of such input devices for use with computing platforms. In particular, systems and techniques described herein may be used for tracking input devices for use in manipulating three-dimensional (3D) models and/or objects provided for display in a computing device. In some implementations, the input devices described herein may be tracked in a mode using six degrees-of-freedom (<NUM>-DoF). In some implementations, the input devices described herein may be tracked in a mode using three degrees-of-freedom (<NUM>-DoF). The systems and techniques described herein can determine which mode to use based on proximity of the input device to one or more other devices.

In some implementations, the systems and techniques described in this document may be used to determine and/or toggle the mode of operation and/or tracking for the input devices. According to example implementations described throughout this disclosure, the input devices may be capable of determining full <NUM>-DoF pose data for use in tracking the input devices to allow accurate manipulation of 3D virtual content in an intuitive manner.

For example, the systems and techniques described herein may include input devices that make use of electromagnetic (EM) fields to track 3D position and/or 3D orientation of a still or moving input device. The input devices may make use of an EM tracking system that is divided into a transmitter portion that produces EM fields and a receiver portion that senses EM fields. In some implementations, the transmitter portion resides on a computing device being accessed with the input device while the receiver portion resides on the input device. In some implementations, the transmitter portion resides on the input device while the receiver portion resides on the computing device being accessed with the input device. In some implementations, the transmitter or receiver portion is instead on a third device in the EM tracking system that operates as a base station and may or may not be mechanically (or communicably) coupled to either the input device or the computing device.

The implementations described throughout this disclosure may utilize an input device, such as an air mouse device that incorporates a non-line-of-sight (NLOS) sensing technology, <NUM>-DoF electromagnetic tracking (EM) technology, and <NUM>-DoF tracking technology to track both position and orientation (e.g., pose) of at least one element embedded in the air mouse device. The EM-based air mouse device described herein may provide an advantage over non-EM systems in that the EM-based air mouse device is NLOS and may maintain device tracking despite occlusion of onboard or external sensing elements in the tracking system. Thus, the air mouse device described herein may continue tracking while other technologies (e.g., light-based sensors, ultrasonic sensors, and/or other mediums) may fail to maintain tracking when their respective sensing elements are occluded. Employing the EM tracking technology in the air mouse devices described herein allows for a <NUM>-DoF device that is robust against occlusions caused by palms, fingers, or other body parts blocking sensors of the air mouse device.

The systems and techniques described herein solve a technical problem of accurately tracking an input device (e.g., an air mouse device) to accurately convey, to a user of the input device, manipulations of 3D content in 3D space within a user interface of a computing device. One example technical solution may include an air mouse computer input device that incorporates NLOS, <NUM>-DoF EM tracking technology to track the pose associated with a tracking element embedded in the input device. In some implementations, a secondary sensor may be installed in the input device to provide failsafe backup tracking for the input device.

The technical solutions described herein may provide a technical effect of ensuring a one to one motion correlation in six degrees of freedom between movements carried out by the input device and any resulting movement of an object in virtual 3D space on the computing device. The systems and techniques described herein may provide improved input device movement stability, improved tracking accuracy for the input device, and efficient tracking in <NUM>-DoF space.

Implementations of the devices described herein can provide advantages over conventional devices. For example, the devices described herein can use NLOS <NUM>-DoF EM tracking while in range of a particular EM base and can automatically switch to tracking with a non-EM based sensor when out of range of the EM base. Such a switching mechanism can ensure that movements (and pose changes) of an input device may be tracked and transmitted regardless of range of the input device to a particular computing device. Users can move content in the computing device without loss of range errors and without loss of input device representation on the computing device. The handheld electronic device may be detected to be out of range of the computing device if a metric correlated with signal noise is above a noise threshold associated with the electromagnetic sensing system. For example, in one embodiment the EM tracking system is configured to provide acceptable tracking performance within a one meter range between the EM transmitter device and EM receiver device. Whenever the system detects that the range has exceeded one meter, for example, the system determines that the receiver is out of range of the transmitter. In another embodiment, the receiver is determined to be out of range of the transmitter when the signal-to-noise ratio of the received EM field falls below a threshold.

In some embodiments of the air mouse/input devices described herein, an EM tracking element may be used with (or instead of) an inertial measurement unit (IMU) sensor to provide absolute or relative pose tracking. In some implementations, the IMU sensor may be used in combination with the EM tracking element to provide for additional stability and tracking accuracy. Other input devices, dongles, buttons, and trackpads can be optionally added for additional input events. In addition, a number of software algorithms can be executed by the systems described herein to recognize pose trajectories for use with specific input gestures that may trigger commands on a computing device. For example, the systems described herein may convert detected pose trajectories into recognized gestures. The gestures may be interpreted as input commands by a system-associated computing device.

In the implementations described herein, the input devices include a number of example air mouse devices. In general, each air mouse device may house one or more subdevices, one or more modules, and/or any number of mechanical and electrical components to provide a powered and functional portable input device. Other external enclosures may be rigidly, flexibly, and/or communicably attached to the air mouse devices.

<FIG> is a block diagram of an example pose tracking system <NUM>, in accordance with implementations described herein. The system <NUM> may be utilized to perform positional tracking for one or more air mouse devices in use with one or more computing devices. The pose tracking system <NUM> may provide NLOS, <NUM>-DoF tracking for any number of air mouse devices within range of a near-field electromagnetic field generated by system <NUM>. The pose tracking system <NUM> may provide <NUM>-DoF tracking for any number of air mouse devices out of range of the near-field electromagnetic field, but within communicable range of the computing device, for example.

In the example system <NUM>, an air mouse device <NUM> can be moved by a user accessing content on a computing device <NUM> or mobile device <NUM>, for example. Accessing content with the air mouse device <NUM> may include generating, modifying, moving and/or selecting the content shown within computing device <NUM>.

A base device system <NUM> may detect the movement and/or pose change of the air mouse device <NUM> and can perform a number of calculations, determinations, and/or processes to determine the pose and any change in pose as the air mouse device <NUM> moves through 3D space. The pose may be used with any number of algorithms described herein to track the air mouse device <NUM> to properly move and render the content for display on computing device <NUM> and/or mobile device <NUM> over network <NUM>, for example. In some implementations, the pose and other content may be used and/or transmitted directly from the air mouse device <NUM> without the use of the network <NUM>. Similarly, the pose or other data may be transmitted from the base device <NUM> to device <NUM> and/or device <NUM> without the use of network <NUM>. In some implementations, the devices of system <NUM> may communicate using point-to-point communication mechanisms (e.g., BLE, USB, etc.).

Once the pose of the air mouse device is computed relative to a base device system <NUM>, the pose can be utilized in relative space with system <NUM>. For example, if a user accesses the air mouse device <NUM> to manipulate 3D content <NUM> in a CAD program, a relative pose may be sufficient to allow tracking of device <NUM> as the user moves the content because the base device system <NUM> is likely stationary and the system would not benefit from additional calculations of translating the pose into world space. In some implementations, the pose can be translated to world space using the known pose of the base device system <NUM> in world space.

As shown in <FIG>, the air mouse device <NUM> includes transmit coils and/or receive coils (e.g., transmitter/receiver coils <NUM>), at least one processor (e.g., central processing unit (CPU) <NUM>), at least one IMU sensor <NUM>, one or more analog to digital convertors (ADCs) or one or more digital to analog converters (DACs) <NUM>, and one or more amplifiers <NUM>. In some implementations, the DACs <NUM> may be combined with one or more amplifiers <NUM>, which may function to amplify as the signals are converted to analog signals. Other hardware and software may be present in system <NUM>, for example, to communicate over additional wired or wireless protocols.

The base device system <NUM> includes transmitter/receiver coils <NUM>, at least one CPU <NUM>, memory <NUM>, an ADC/DAC <NUM>, and one or more amplifiers <NUM>. Other hardware and software may be present in system <NUM>, for example, to communicate over additional wired or wireless protocols.

The transmitter/receiver coils <NUM> and transmitter/receiver coils <NUM> may represent either transmitter coils or receiver coils that may respectively emit and/or sense EM fields (not shown). In general, an EM field may be generated by the base device system <NUM> using a transmitter coil. The air mouse device <NUM> may, using receiver coils, read EM data generated by the system <NUM>. Other configurations are possible. For example, the EM field and data may be generated by the air mouse device <NUM> while the base device system <NUM> reads the EM data (as shown in <FIG>).

In general, the pose tracking system <NUM> uses near-field electromagnetic fields to determine pose data associated with input devices, such as air mouse device <NUM>. The pose data may be used to track the air mouse device <NUM> as a user moves the device (or when the device is stationary). In operation, the pose tracking system <NUM> may utilize at least one transmitter that generates an EM field on a tri-axis coil (e.g., transmitter coil(s) <NUM>) to induce a current on a second tri-axis coil (e.g., receiver coil(s) <NUM>) at a receiver (e.g., base device system <NUM>). The receiver generates a number of readings which are then processed by system <NUM>, for example, to compute a position and orientation of the transmitter (e.g., on air mouse device <NUM>) relative to the receiver (e.g., on base device system <NUM>).

In some implementations, an EM field may be generated by the system <NUM> using a transmitter coil <NUM> and the amplifier <NUM>. For example, the transmitter coil <NUM> may be a tri-axis coil (e.g., three coils) that may generate three EM fields (one for each different axis). The EM field may be provided at a strength (i.e., transmit power) based on electrical power provided by the amplifier(s) <NUM> to the transmitter coil <NUM>. The amplifier <NUM> may be a programmable amplifier generally configured to generate electrical power at a magnitude based on received control signaling. In general, the amplifiers <NUM> and <NUM> may function to amplify received signals.

Continuing with the above example, the air mouse device <NUM> may include components to read EM data. For example, device <NUM> includes receiver coils <NUM> and ADC <NUM>. In some implementations, the receiver coil <NUM> is a tri-axis coil configured to generate an analog electrical signal having a magnitude and/or phase indicative of a detected EM field. The ADC <NUM> is generally configured to receive the generated analog signal and convert the analog signal to a digital value indicative of the analog signal of the detected EM field. In this example, the digital values generated by the ADC <NUM> represent EM data that can be used for pose identification, as described further below.

The air mouse device <NUM> may also include a processor <NUM>, which may communicate, with processor <NUM> in the system <NUM>. The communications may include relative pose data of the system <NUM> in relation to air mouse device <NUM> (or alternatively pose data of the device <NUM> in relation to system <NUM>). Such processors <NUM> and <NUM> are configured to execute instructions (e.g., computer programs) in order to carry out specific tasks. In some implementations, at least one of the processors <NUM> and <NUM> executes instructions to identify a relative pose between the system <NUM> and the device <NUM> based on the EM data provided by the ADC <NUM>. Memory <NUM> and memory <NUM> may be utilized throughout communications and interactions amongst system <NUM> and device <NUM>.

The system <NUM> may be used to track 3D position and 3D orientation (i.e., <NUM>-DoF tracking). The input devices may make use of an EM tracking system that is divided into a transmitter portion that produces EM fields and a receiver portion that senses EM fields. In one non-limiting example, the transmitter portion is housed on system <NUM> and includes transmitter coils <NUM>, ADC <NUM>, and amplifier <NUM>. The transmitter portion may generate three fields on a tri-axis coil (not shown) while the receiver portion employs a tri-axis coil (not shown) as an antenna to sense the fields generated by the transmitter portion. The receiver portion may include receiver coils <NUM>, ADC/DAC <NUM>, and amplifiers <NUM>. Each of the coils on the receiver portion may sense all three fields generated by the coils associated with the transmitter portion. This interaction results in at least nine EM measurements. Using these measurements, the pose of the receiver coils <NUM> relative to the transmitter coils <NUM> can be computed from the nine measurements by employing a dipole model (not shown) to equate sensed EM magnitudes (in the EM data) to a pose. As used herein, a pose may refer to a position, an orientation, or both.

In another non-limiting example, the transmitter portion is housed on device <NUM> and includes transmitter coils <NUM> and DAC <NUM>, and amplifier <NUM>. The transmitter portion may generate three fields on a tri-axis coil while the receiver portion employs a tri-axis coil (e.g., coils <NUM>) as an antenna to sense the fields generated by the transmitter portion. The receiver portion may be housed on system <NUM> and may include receiver coils <NUM>, ADC/DAC <NUM>, and amplifiers <NUM>. Each of the coils on the receiver portion may sense all three fields generated by the coils associated with the transmitter portion. This interaction results in at least nine EM measurements. Using these measurements, the pose of the receiver coils <NUM> relative to the transmitter coils <NUM> can be computed from the nine measurements by employing a dipole model (not shown) to equate sensed EM magnitudes (in the EM data) to a pose.

As shown in <FIG>, the air mouse device <NUM> may also include an IMU sensor <NUM>. The IMU sensor <NUM> may function to detect, for the air mouse device, a 3D orientation in 3D space based on the IMU measurements. The IMU sensor <NUM> may include one or more accelerometers, gyroscopes, magnetometers, and other such sensors. In general, the IMU sensor <NUM> may detect motion, movement, velocity, and/or acceleration of the air mouse device <NUM>, for example.

In some implementations, a pose of the air mouse device <NUM>, for example, may be detected based on data provided by the IMU sensor <NUM>. Based on the detected pose, the system <NUM> may update content on a computing device <NUM> (or base device <NUM>) to reflect a changed pose of the air mouse device <NUM>.

As used herein, the term transmitter/receiver may represent a single transmitter, a single receiver, a transmitter and a receiver (e.g., a transceiver), multiple transmitters, or multiple receivers. In some implementations, the transmitter (TX) and receiver (RX) are distributed over different devices, including a stationary base device (e.g., associated or within computing device <NUM>) that tracks the position and/or orientation of a movable input device. In some implementations, the base device system <NUM> may include transmitter coils while the air mouse device <NUM> may include receiver coils. In some implementations, the base device system <NUM> may include receiver coils while the air mouse device <NUM> may include transmitter coils. In some implementations, either or both devices <NUM>, <NUM> may include transceivers that may transmit and receive depending upon which EM field excites coils of the respective transceiver.

In some implementations, the base device system <NUM> is connected to computing device <NUM> and/or mobile device <NUM>. For example, the base device system <NUM> may be rigidly connected to computing device <NUM> and/or mobile device <NUM>. In some examples, the base device system <NUM> may be communicably coupled to (e.g., wired or wirelessly) computing device <NUM> and/or mobile device <NUM>. In some examples, the base device system <NUM> may be a dongle coupled to computing device <NUM> and/or mobile device <NUM> and may communicate with air mouse device <NUM> via the dongle. In some implementations, the base device system <NUM> is housed within computing device <NUM> or mobile device <NUM>.

In the example system <NUM>, the devices <NUM>, <NUM>, and <NUM> may be (or may be part of) a laptop computer, a desktop computer, a mobile computing device, a tablet computing device, or a gaming console. Devices <NUM>, <NUM>, and <NUM> can include hardware and/or software for executing applications and application content. In addition, devices <NUM>, <NUM>, and <NUM> can include (or have access to) hardware and/or software that can recognize, monitor, and track a pose of the air mouse device <NUM>, when these devices are placed in front of or held within a range of positions relative to the air mouse device <NUM>. In some implementations, devices <NUM>, <NUM>, and <NUM> can provide additional content to air mouse device <NUM> over network <NUM>. In some implementations, devices <NUM>, <NUM>, <NUM>, and <NUM> can be connected to/interfaced with one or more of each other either paired or connected through network <NUM>. The connection can be wired or wireless. The network <NUM> can be a public communications network or a private communications network.

The system <NUM> may include or have access to electronic storage (e.g., memory <NUM> and memory <NUM>). The electronic storage can include non-transitory storage media that electronically stores information. The electronic storage may be configured to store captured pose data, raw sensor data, application data, and/or other computing data.

In some implementations, the base device system <NUM> may be integrated with the computing device <NUM> which may include (or be coupled to) a dongle <NUM>, as described in detail with respect to <FIG> below. In some implementations, the air mouse device <NUM> may include a trackball <NUM>, as described in detail with respect to <FIG> below.

In some implementations, the air mouse device <NUM> may include a communications module <NUM>. The communications module <NUM> may be used to transmit data to CPU <NUM> or other devices in system <NUM>. In some implementations, the communications module <NUM> may transmit commands to manipulate content <NUM> in device <NUM> based on movements and/or pose changes detected from air mouse device <NUM>. Although communication modules are not explicitly depicted for computing devices <NUM>, <NUM>, and for base device system <NUM>, each of these devices may have the ability to send and receive messages, data, commands, and the like amongst any of the devices of system <NUM>.

In some implementations, the air mouse device <NUM> includes a gesture module <NUM>. In some implementations, the base device <NUM> instead includes the gesture module <NUM>. In yet other implementations, the computing device <NUM> includes the gesture module <NUM>. The gesture module <NUM> represents algorithms and/or software that can convert pose trajectories into recognized gestures. The pose trajectories may be generated by a user accessing air mouse device <NUM>. The gestures may be interpreted as input commands by an associated computing device <NUM> housing system <NUM>, for example. In some implementations, a pose of the air mouse device may control a two dimensional (2D) mouse cursor on an associated computing device <NUM>. The mouse cursor may be moved by either translating the air mouse device <NUM> or changing the pointing angle of the air mouse device <NUM>.

In some implementations, the air mouse device <NUM> includes a microphone (not shown) and/or a speaker (not shown), as described in detail with respect to <FIG>.

<FIG> is a block diagram of an example pose tracking system <NUM> for an air mouse device <NUM>, in accordance with implementations described herein. The tracking system <NUM> may be an EM tracking system that utilizes transmitter coils and receiver coils to perform tracking of the air mouse device <NUM>. In some implementations, the pose tracking system <NUM> may include non-electromagnetic sensors and devices to assist in tracking of the air mouse device <NUM>, as a user moves the device <NUM> to interact with content in a computing device (associated with or integrated with a base device <NUM>).

As shown in this example, the system <NUM> includes an input device (e.g., air mouse device <NUM>) and a base device <NUM>. The air mouse device <NUM> may be a handheld electronic device for controlling 3D content displayed in a user interface of a computing device. For example, the device <NUM> may be an air mouse device <NUM> for use with content <NUM> on computing device <NUM> in which base device system <NUM> is installed on (or accessible to) computing device <NUM>.

The tracking system <NUM> is generally configured to identify relative poses between the air mouse device <NUM> and the base device <NUM> by generating an EM field <NUM>, measuring a magnitude and/or phase of the generated EM field <NUM> (referred to herein generally as "EM data"), and computing a relative pose based on the corresponding EM data. The tracking system <NUM> can therefore be incorporated into a variety of devices and systems that employ pose identification algorithms. In some implementations, a portion of the tracking system <NUM> is incorporated into an air mouse device while other portions of the tracking system <NUM> are incorporated into a computing device. Thus, in some configurations, the base device <NUM> is a computing device (e.g., base device system <NUM>, computing device <NUM>, or mobile device <NUM>) while the input device is a handheld air mouse device <NUM>/<NUM>. In some configurations, the base device <NUM> is the input device while the air mouse device <NUM> is a stationary computing device (e.g., base device system <NUM>, computing device <NUM>, or mobile device <NUM>). Other configurations are possible, including an EM field that is generated by device <NUM> to ensure that EM data may be read at the base device <NUM>.

In operation of system <NUM>, an EM field <NUM> is generated by the base device <NUM>. The base device <NUM> includes an EM transmitter module <NUM> to generate the EM fields. The EM transmitter module <NUM> includes a transmitter coil <NUM>, an amplifier <NUM>, and a DAC <NUM>. The transmitter coil <NUM> may represent, for example, a tri-axis coil configured to generate the EM field <NUM> at a particular strength (e.g., transmit power). The transmit power may be based at least in part on the electrical power provided by the amplifier <NUM>. The amplifier <NUM> may be configured to generate the electrical power at a magnitude based on received control signaling of device <NUM>.

In response to detecting the generated EM field <NUM>, the air mouse device <NUM> reads EM data from the EM field <NUM> using an onboard EM receiver module <NUM>. In general, the EM transmitter module <NUM> may function as a portion of an electromagnetic sensing system for detecting poses associated with the air mouse device <NUM>, for example. The air mouse device <NUM> may include a remaining portion of the electromagnetic sensing system in an EM receiver module, as described with respect to EM receiver module <NUM>. In some implementations, the EM receiver module includes one or more processors (not shown).

The base device <NUM> includes a CPU (i.e., processor <NUM>) that may bidirectionally communicate with air mouse device <NUM> via communication link <NUM>. For example, EM data, identified poses, identified pose trajectories, and/or other information may be exchanged between air mouse device <NUM> and base device <NUM>. For example, in some implementations, the processor <NUM> identifies poses based on the EM data and communicates the identified poses to the processor <NUM>. In some implementations, the processor <NUM> communicates the EM data to the processor <NUM>, which identifies poses based on the EM data.

The communication link <NUM> can be a wired communication link (e.g., USB, serial, Ethemet, etc.), a wireless communication link (e.g., Bluetooth, WiFi, ZigBee, RF, etc.), and the like, or a combination thereof. In other embodiments, the EM data can be sent to another device and/or processor (not shown) and the other device and/or processor may compute a pose from the EM data. In some implementations, the EM data may be stored locally within device <NUM> or <NUM>, locally within the system <NUM>, and/or remote from system <NUM>.

As shown in <FIG>, the air mouse device <NUM> is an input device that includes an EM receiver module <NUM> to generate EM data from detected EM fields. The EM receiver module <NUM> includes a receiver coil <NUM>, an amplifier <NUM>, and an ADC <NUM>. In some implementations, the receiver coil <NUM> may be a tri-axis coil configured to detect an analog electrical signal having a magnitude and/or phase indicative of a particular detected EM field. The ADC <NUM> may be configured to receive the generated analog signal and convert the analog signal to a digital value indicative of the analog signal represented in the EM field <NUM>. The digital values generated by the ADC <NUM> are EM data that can be used for pose identification for air mouse device <NUM>.

The air mouse device <NUM> also includes an IMU sensor <NUM>. The IMU sensor <NUM> may provide additional pose information about device <NUM>. One or more of the processors <NUM> and <NUM> (or another device processor) can use the additional pose information to supplement or augment the poses identified based on the EM data. For example, in some embodiments the processor <NUM> can use the additional pose information to identify potential errors in the poses determined based on the EM data, and to address the identified errors.

In some implementations, the processor <NUM> may use the additional pose information determined using IMU <NUM> to track air mouse device <NUM>, for example, when device <NUM> is out of range of the EM field produced by transmitter module <NUM>. Thus, the IMU <NUM> may be utilized as a backup pose determiner in order to continue tracking the device <NUM>. For example, if device <NUM> is the air mouse device <NUM> and the user operating device <NUM> moves out of range of base device <NUM> (e.g., installed within computing device <NUM>), the system <NUM> can still determine pose information well enough to track a moving device <NUM> such that content (e.g., 3D objects) within the computing device <NUM> can be moved based on movements of tracked air mouse device <NUM>. In this example, the pose information may be <NUM>-DoF pose data based on detected 3D orientations of the device <NUM>/<NUM>.

If the IMU <NUM> is utilized as a backup pose determiner, the systems described herein may include one or more processors (e.g., processor <NUM> on device <NUM>) that are configured to generate commands to manipulate 3D content in the computing device <NUM>, for example. In some implementations, the IMU may generate commands using the <NUM>-DoF pose to manipulate 3D content displayed on the computing device, in response to detecting air mouse poses while the air mouse device is communicably coupled to the electromagnetic transmitter. To do so, the systems described herein may begin to use pose information gathered from IMU <NUM> (instead of from the EM sensing system including both transmitter module <NUM> and receiver module <NUM>). The pose information may be used to track movement of air mouse device <NUM>. The movements may be converted into commands that may be carried out on the content <NUM> shown in the user interface of computing device <NUM>, for example.

In some implementations, the commands may be generated based on a determined proximity of the air mouse device (e.g., air mouse device <NUM>) relative to a base device (e.g., base device <NUM>). The base device <NUM> may be associated with (or in communication with) the computing device <NUM>, for example. The determined proximity may be an indication that the air mouse device <NUM> is in or out of range of the base device <NUM> (and/or computing device <NUM>, if the transform between the base device <NUM> and the computing device <NUM> can be estimated). The determined proximity may trigger selection of which particular data to use when generating the commands. For example, the system <NUM> may utilize data (e.g., pose data) from the electromagnetic sensing system when the determined proximity indicates that the device <NUM> is within range of the base device <NUM>. The system <NUM> may instead utilize pose data from the IMU <NUM> (e.g., three-dimensional orientation) when the determined proximity indicates that the device <NUM> is out of range of the base device <NUM> and/or computing device <NUM>.

Upon generation of one or more commands, a communication module <NUM> may access CPU <NUM> to trigger transmission of the one or more commands to manipulate the three-dimensional content displayed in the computing device.

<FIG> is a block diagram of another example pose tracking system for an air mouse device <NUM>, in accordance with implementations described herein. Similar to the pose tracking system <NUM>, system <NUM> may be an EM tracking system that utilizes transmitter coils and receiver coils to perform tracking of the air mouse device <NUM>. In some implementations, the pose tracking system <NUM> may include non-electromagnetic sensors and devices to assist in tracking of the air mouse device <NUM>, as a user moves the air mouse device <NUM> to interact with content in a computing device (e.g., computing device <NUM>) associated a base device <NUM>.

The pose tracking system <NUM> is similar to the pose tracking system <NUM> of <FIG>, but the pose tracking system <NUM> places an EM receiver module <NUM> at the base device <NUM> and an EM transmitter module <NUM> at the air mouse device <NUM>. For example, as shown in <FIG>, the base device <NUM> includes the EM receiver module <NUM> with a receiver coil <NUM>, an amplifier <NUM>, and an ADC <NUM>. The base device also includes a processor <NUM>.

The input device <NUM> includes the EM transmitter module <NUM> to generate EM fields which are sensed at the base device <NUM>. The EM transmitter module <NUM> includes a transmitter coil <NUM>, an amplifier <NUM>, and a DAC <NUM>. In some implementations, the transmitter coil <NUM> is a tri-axis coil configured to function with the DAC <NUM> and amplifier <NUM> to generate an analog electromagnetic field.

In some implementations, the processor <NUM> (or processor <NUM> or another device processor) may use the additional pose information determined using IMU <NUM> to track air mouse device <NUM>, for example, when device <NUM> is out of range of the EM fields described herein. Thus, if the device <NUM> is an air mouse device <NUM> and the user operating the device <NUM> moves out of range of base device <NUM> (e.g., installed within computing device <NUM>), the system <NUM> can still determine pose information well enough to track a moving air mouse device <NUM> such that content (e.g., 3D objects) within the computing device <NUM> can be moved based on movements of tracked air mouse device <NUM>.

In operation of system <NUM>, the EM transmit module <NUM> is configured to generate EM field <NUM>. The EM receiver module <NUM> is configured to generate EM data from the sensed EM field <NUM>. In some implementations, IMU data from the air mouse may be transmitted via a communication link <NUM> from processor <NUM> to processor <NUM>. In some implementations, the EM data can be used to identify relative poses between the base device <NUM> and the device <NUM>. Similarly, the IMU data may be used to identify poses of the air mouse device <NUM>.

<FIG> is a block diagram depicting an example range of tracking for use with one or more air mouse devices described herein. In the depicted example, an air mouse device <NUM> is shown a distance from computing device <NUM>. The air mouse device <NUM> may be used to modify 3D content in the display of computing device <NUM>. The air mouse device <NUM> may represent air mouse device <NUM> (<FIG>) while computing device <NUM> may represent computing device <NUM> housing system <NUM>.

In operation, the air mouse <NUM> may be moved away from a computing device <NUM> that houses receiver circuitry and/or transmitter circuitry for determining pose information which may be used to track device <NUM>. For example, a user may operate device <NUM> to move the device (from <NUM> to 402A) to manipulate 3D content in device <NUM>. Here, the user has moved device <NUM> away from the computing device <NUM>, as indicated by arrows <NUM> and <NUM>. The device is illustrated as moving in a straight line, but any route of movement may be possible. The line from position <NUM> through position <NUM> is intended to illustrate a particular distance over which device <NUM> functions may change based on a distance away from device <NUM>. For example, since air mouse device <NUM> includes EM-based circuitry, moving device <NUM> from a particular antenna (i.e., coils) within device <NUM> can cause signal decay and signal failure if the device <NUM> travels beyond a system-defined operating distance from the computing device <NUM>. In particular, noise may increase as the distance is increased between a receiver module (e.g., receiver module <NUM>) and a transmitter module (e.g., transmitter module <NUM>). Elevated noise may cause signal failure or an inability to utilize data from either device <NUM>, <NUM>.

The systems described herein can detect when a particular receiver module is out of range of a particular transmitter module. That is, when the noise (or a metric that is correlated to noise, such as distance) between the receiver and the transmitter is greater than a threshold, the systems described herein may fall back to <NUM>-DoF pose estimation and subsequent device tracking. In some implementations, the threshold may be a predefined threshold based on the hardware used in the system, and in other implementations, the threshold is dynamically computed. In some examples, the threshold might be set at a distance between the transmitter and the receiver that allows for pose determination, but may be a large enough distance to induce noise effects that cause signal degradation and/or data transmit failure. In either event, if a particular receiver is detected as being out of range of a transmitter, the system may rely on a secondary sensor (e.g., IMU sensor <NUM>) to provide <NUM>-DoF tracking until the transmitter and receiver are detected to be within range again. In one example, once the air mouse device <NUM> is detected within range of the computing device <NUM> (housing base device system <NUM>), the EM tracking system <NUM> may return to performing <NUM>-DoF pose tracking.

As shown in <FIG>, if the user moves air mouse device <NUM> from location <NUM> to a distance represented by location <NUM> or location <NUM> (within the <NUM>-DoF range), the EM-based pose determination and device tracking may be performed by system <NUM>. The maximum distance may be represented by device electronics and power protocols used. In an example operation of device <NUM>, about one meter may be represented from location <NUM> to location <NUM>. However, such distances may be programmable smaller or larger than about one meter.

If the device <NUM> is moved beyond location <NUM>, for example to location <NUM> or <NUM> (e.g., within the <NUM>-DoF range), the system <NUM> can revert to using IMU sensor <NUM> to perform <NUM>-DoF pose determination and device tracking. If the air mouse device <NUM> is returned to location <NUM> or otherwise placed within range of device <NUM> (housing an EM transmitter/receiver device), for example, the system <NUM> can use the <NUM>-DoF (EM-based) pose determination and device tracking.

In some implementations, an example air mouse tracking system may include a handheld air mouse device and a host computing device. The handheld air mouse device may include an IMU, a first processor, and either an EM receiver device or an EM transmitter device. The host computing device may include an EM receiver if the handheld air mouse device includes an EM transmitter device. The host computing device may include an EM transmitter device if the handheld air mouse device includes an EM receiver device. The host computing device may also include a display and a second processor in communication with the first processor. The first processor may be configured to collect pose data derived from EM data and/or IMU data. The pose data may be used to compute (by the first processor or the second processor) an estimated pose between the EM receiver and EM transmitter modules. The pose may be used to generate commands to manipulate three-dimensional content in the computing device.

In some implementations, the EM receiver device or the EM transmitter device may be in a base device that contains a third processor and is external to the host computing device. Estimated poses may be computed by a single processor or any combination of the first processor, the second processor, and/or the third processor. In some implementations, the estimated pose is a <NUM>-DoF pose computed from the EM data when a metric correlated with the noise in the EM data is below a threshold. The estimated pose is a <NUM>-DoF pose computed from the IMU data when the metric is above the threshold.

In another example, a handheld electronic device is described for controlling three-dimensional content displayed in a user interface of a computing device. The handheld electronic device may include an EM receiver module, an IMU, a first communications link, and at least one processor. The at least one processor may be configured to collect EM data from the EM receiver module, collect IMU data from the IMU, and transmit pose data over the first communications link, where the pose data is derived from EM data and/or IMU data and where the pose data is used to estimate a pose of the handheld electronic device. In addition, the pose may be used to generate commands to manipulate the three-dimensional content in the computing device.

In some implementations, the transmitter is housed within the computing device. In some implementations, the transmitter is embedded in a dongle plugged into the computing device. In some implementations, the dongle includes a second communications module that maintains a communications link with the first communications module, receives pose data from the handheld electronic device over the communications link, and forwards data derived from the pose data to the computing device over USB. In some implementations, the transmitter module is instead housed in the air mouse device while the receiver is housed within the computing device.

<FIG> are block diagrams depicting example gesture recognition for use with one or more air mouse devices described herein. The air mouse device <NUM> may be described with respect to similar device <NUM>, device <NUM>, or device <NUM>. Gesture recognition may be carried out by the gesture module within the air mouse device (e.g., gesture module <NUM> in air mouse device <NUM>), the base device <NUM>, or the computing device <NUM>. The gesture module includes algorithms and/or software that can convert pose trajectories determined for the air mouse device into recognized gestures. The pose trajectories may be generated by a user accessing air mouse device <NUM>. The gestures may be interpreted as input commands by an associated computing device <NUM> housing system <NUM>, for example. In some implementations, a pose of the air mouse device may control a two dimensional (2D) mouse pointer on an associated computing device <NUM>. In general, the mouse pointer may be moved by either translating the air mouse device <NUM> or changing the pointing angle of the air mouse device <NUM>.

As shown in <FIG>, a user may be using an air mouse device <NUM> to access software and content in a user interface <NUM> in a computing device <NUM>. The air mouse device <NUM> may be moved at an angle to cause a proportional change in the position of a 2D mouse cursor.

For example, a user may perform an arc movement with device <NUM> from position [<NUM>] <NUM> to position [<NUM>] <NUM>, as shown by device 502A at arrow <NUM>. In response, the system <NUM> may use pose information obtained from EM-based tracking system (split between system <NUM> and device <NUM>) to accurately move a mouse pointer <NUM> a proportional distance from a left position [<NUM>] <NUM> to a right position [<NUM>] <NUM> in the user interface <NUM>, resulting in the mouse pointer being shown by mouse pointer <NUM>. That is, if the air mouse device <NUM> is moved with the angle change shown by arrow <NUM>, the mouse pointer <NUM> is also moved from left to right for a distance proportional to the angle change.

As shown in <FIG>, a user may be using the air mouse device <NUM> to access software and content in the user interface <NUM> in the computing device <NUM>. The air mouse device <NUM> may be moved from left to right to cause a proportional change in the position of a 2D mouse cursor.

For example, a user may perform a left to right movement with device <NUM> (to 502B) from position [<NUM>] <NUM> to position [<NUM>] <NUM>, as shown by device 502B at arrow <NUM>. In response, the system <NUM> may use pose information obtained from EM-based tracking system (e.g., split between system <NUM> and device <NUM>) to accurately move a mouse pointer <NUM> a proportional distance from a left position [<NUM>] <NUM> to a right position [<NUM>] <NUM> in the user interface <NUM>, resulting in the mouse pointer being shown by mouse pointer <NUM>. That is, if the air mouse device <NUM> is moved as shown by arrow <NUM>, the mouse pointer <NUM> is also moved from left to right for a distance proportional to the translation change. Thus, the mouse pointer position change is proportional to the position of the air mouse device in 3D space.

As shown in <FIG>, a user may be using the air mouse device <NUM> to access software and 3D content <NUM> in the user interface <NUM> in the computing device <NUM>. The air mouse device <NUM> may be moved from left to right and forward in space to cause a proportional change in the position of a 2D or 3D mouse cursor configured to move in the virtual space displayed on a computing device.

For example, a user may perform a left to right movement with device <NUM> (to 502C) from position [<NUM>] <NUM> to position [<NUM>] <NUM>, as shown by device 502C at arrow <NUM>. In response, the system <NUM> may use pose information obtained from EM-based tracking system (e.g., split between system <NUM> and device <NUM>) to accurately move a mouse pointer <NUM> a proportional distance from a left position [<NUM>] <NUM> to a right position [<NUM>] <NUM> in the user interface <NUM>, resulting in the mouse pointer being shown by mouse pointer <NUM>. Thus, the mouse pointer position change is proportional to the position of the air mouse device in 3D space.

Similarly, the user may perform a front to back (i.e., forward) movement in 3D space with device <NUM> (to 502D) from position [<NUM>] <NUM> to position [<NUM>] <NUM>, as shown by device 502D at arrow <NUM>. In response, the system <NUM> may use pose information obtained from EM-based tracking system (e.g., split between system <NUM> and device <NUM>) to accurately move the mouse pointer <NUM> a proportional distance from the first position [<NUM>] <NUM> to a second position [<NUM>] <NUM> in the user interface <NUM>, resulting in the mouse pointer being shown by mouse pointer <NUM>. Thus, the mouse pointer position change is proportional to the position of the air mouse device in virtual 3D space by moving the air mouse device in 3D space.

In some implementations, the system <NUM> can capture and interpret complex gestures. For example, such gestures may be captured and interpreted as commands to the associated computing device <NUM> housing system <NUM>, for example. One example gesture may include a user shaking the air mouse device <NUM> left to right back and forth. The system <NUM> may interpret the movements as a command to close the current operating system/application window. In another example, a gesture may include pressing a button and pointing the air mouse device <NUM> downward to trigger the system <NUM> to cause a window focused in interface <NUM> to scroll downward. In yet another example, gesture may include using the device <NUM> to enter, as input, a letter (e.g., "x") in the air (e.g., 3D space). Such a gesture may result in the system <NUM> interpreting the gesture as a command to close the window. Each of these example gestures may be interpreted and transmitted as commands (e.g., via communication module <NUM>) to a software application and/or operating system on the computing device <NUM>. The gestures may be interpreted by gesture module <NUM> utilizing any EM-based sensors and/or components and non-EM based sensors and/or components described in <FIG> and/or systems throughout this disclosure.

In some implementations, the systems described herein may capture and interpret gestures as commands to a computing system in a 3D context. In particular, system <NUM> may capture and interpret gestures as commands in a 3D context. For example, the 3D object <NUM> may be displayed on the computing device <NUM>. The system <NUM> may determine that the user wishes to interact with content <NUM> based on contextual cues. If system <NUM> determines that the user has some context with 3D object <NUM>, the system <NUM> may lock the motion of the air mouse device <NUM> to the 3D object <NUM> such that performing movements with device <NUM> causes one to one movements of the 3D object <NUM>. For example, pulling the air mouse device <NUM> backward (not shown) may cause the 3D object <NUM> to appear larger and closer in the user interface <NUM>, effectively zooming in on the 3D object <NUM>. Similarly, a gesture that includes rotating the air mouse device <NUM> in the air may cause a corresponding rotation of the 3D object <NUM> in the user interface <NUM>. In another example, the use of two air mouse devices (e.g., one device <NUM> in each of a left hand and a right hand) may allow a user to intuitively grab (e.g., naturally handle) the 3D object <NUM> and pull or stretch the object. The system <NUM> may effectively simulate grabbing a first end and a second end, respectively, using the left air mouse device <NUM> to simulate handling the first end, and using the right air mouse device <NUM> to simulate handling the second end.

<FIG> is a block diagram depicting an example air mouse device 600A in accordance with implementations described herein. The air mouse device 600A may control a 2D pointer in a computing device by rolling a trackball <NUM> over a trackpad <NUM>. For example, the air mouse device 600A may be accessed by a user to move the content <NUM> on computing device <NUM> (<FIG>) in two dimensions as the user rolls the trackball <NUM> around the trackpad <NUM>. If the user rolls trackball <NUM> rightward, as shown by arrow <NUM>, the content <NUM> may be moved left to right. In this example, the trackball <NUM> may provide pose information and commands may be generated in response to the pose information.

<FIG> is a block diagram depicting an example 3D air mouse device 600B in accordance with implementations described herein. The air mouse device 600B is removable from trackpad <NUM>. The air mouse device 600B may control 3D virtual objects in a computing device by lifting a trackball <NUM> from trackpad <NUM> and moving the trackball <NUM> in 3D space (e.g., the x-y-z axis in <FIG>). For example, a user may lift trackball <NUM> and tum the ball rightward (or leftward) from a perpendicular y, as shown by arrow <NUM>. Such a movement may tilt content <NUM> (<FIG>) at any angle that the user may choose when twisting and/or turning the trackball <NUM>. Similarly, the user may manipulate content <NUM> to twist the content backward (or forward) from a horizontal z, as indicated by arrow <NUM>. In addition, the user may rotate the trackball in 3D space around the y-axis, for example, as indicated by arrow <NUM>. Using the tracking EM-based systems described herein, the rotation of the trackball <NUM> triggers rotation of the content <NUM> (e.g., a three-dimensional object) displayed in a computing device. Similarly, translating the trackball <NUM> in space causes a proportional translation of the content <NUM> in virtual space shown in a UI of the computing device.

The user may move trackball <NUM> left, right, up, down, forward, and backward including tilting, rotating, revolving movements to affect the movement of content <NUM>, for example, in 3D space. The movement may be a one to one movement where a user moves the trackball in space and the content <NUM> is moved the same distance in the computing device (or moved a predetermined proportional distance based on the physical movement of the trackball <NUM>).

In general, the trackball <NUM> may provide pose information and commands may be generated in response to the pose information. Such commands may manipulate, control, or otherwise modify the content <NUM> in the computing device. To function in system <NUM>, the trackball may include at least one transmitter module (e.g., transmitter module <NUM>) to transmit commands to control a three-dimensional object (e.g., content <NUM>) according to movements of the trackball <NUM> in 3D space. In some implementations, the trackball may instead include at least one receiver module while a base computing device includes the transmitter module.

In some implementations, the input devices described herein include one or more buttons that provide for an additional input events. Additionally the input devices can incorporate any or all of a slider, one dimensional (1D) touchpad, a roller, or similar component that allows for additional 1D input. In some implementations, the input devices described herein can include a 2D touchpad, a joystick, and/or similar component that allows for 2D analog input into the respective input device.

<FIG> is a block diagram of an example pose tracking system <NUM>, in accordance with implementations described herein. The pose tracking system <NUM> may include an input device <NUM> and a computing device <NUM> removably attachable to a dongle <NUM>. The dongle <NUM> may include components similar to components in the base device <NUM> (<FIG>) or <NUM> (<FIG>). The input device <NUM> may be a handheld electronic device for controlling 3D content displayed in a user interface (not shown) of computing device <NUM>. For example, the device <NUM> may be an input device <NUM> for use with content <NUM> on computing device <NUM> in which base device system <NUM> is installed on, connected to, or otherwise accessible to computing device <NUM>.

In the depicted example, the dongle <NUM> may include a USB connector <NUM> that may be mated with the computing device <NUM> by a connector <NUM>. In general, the input device <NUM> may be a handheld device such as a controller, an air mouse, a mobile device, or a tablet device, etc. The input device <NUM> may be configured to communicate pose information to the dongle <NUM> via a first wireless protocol (e.g., via radio frequency, Wi-Fi, or other wireless signal in the electromagnetic spectrum). The dongle <NUM> may communicate to the computing device <NUM> using a wired protocol such as Universal Serial Bus (USB), ZigBee, serial, firewire, thunderbolt, lightning, analog audio, digital audio, and the like. In some implementations, the dongle <NUM> includes an EM transmitter module <NUM> to interface (and/or communicate) with an EM receiver module <NUM> associated with the input device <NUM>. Although a USB interface for the dongle <NUM> is depicted and may provide both a reliable mechanical interface as well as digital communication protocol in a small form factor, any other interface (such as ZigBee, serial, firewire, thunderbolt, lightning, analog audio, digital audio, etc.) could be used in any of the devices described herein.

The tracking system <NUM> may be an EM tracking system that utilizes transmitter coils and receiver coils to perform tracking of the input device <NUM>. In some implementations, the pose tracking system <NUM> may include non-electromagnetic sensors and devices to assist in tracking of the input device <NUM>, as a user moves the device <NUM> to interact with content in a computing device (associated with or integrated with the computing device <NUM> connected to dongle <NUM>).

In some implementations, the EM transmitter module <NUM> is instead in the input device <NUM> and the EM receiver module <NUM> is swapped to the dongle <NUM>. In this implementation of the system, the EM data may be collected at the dongle and may be converted to poses by the CPU <NUM> in the dongle <NUM> or by the CPU <NUM> in the computing device <NUM>.

The tracking system <NUM> is generally configured to identify relative poses between the input device <NUM> and the base device (e.g., dongle <NUM>) by generating an EM field <NUM>, measuring a magnitude and/or phase of the generated EM field <NUM> (referred to herein generally as "EM data"), and computing a relative pose based on the corresponding EM data. Other configurations are possible, including an EM field that is generated by device <NUM> to ensure that EM data may be read at the base computing device <NUM> (via dongle <NUM>, for example).

In operation of system <NUM>, an EM field <NUM> is generated by the dongle <NUM>. The dongle <NUM> includes the EM transmitter module <NUM> to generate the EM fields. The EM transmitter module <NUM> includes a transmitter coil <NUM>, an amplifier <NUM>, and a DAC <NUM>. The transmitter coil <NUM> may represent, for example, a tri-axis coil configured to generate the EM field <NUM> at a particular strength (e.g., transmit power). The transmit power may be based at least in part on the electrical power provided by the amplifier <NUM>. The amplifier <NUM> is configured to generate the electrical power at a magnitude based on received control signaling of device <NUM>.

In response to detecting the generated EM field <NUM>, the input device <NUM> reads EM data from the EM field <NUM> using the onboard EM receiver module <NUM>. In general, the EM receiver module <NUM> may function as a portion of an electromagnetic sensing system for detecting 3D positions and/or 3D orientations (i.e., poses) associated with the input device <NUM>, for example. In this example, the dongle <NUM> may include a remaining portion of the electromagnetic sensing system in an EM transmitter module <NUM>. In some implementations, the EM receiver module <NUM> may function as a portion of an electromagnetic sensing system for detecting 3D positions and/or 3D orientations (i.e., poses) associated with detected movements of the input device <NUM>.

The dongle <NUM> also includes a CPU (i.e., processor <NUM>) that may bidirectionally communicate with input device <NUM> via CPU (i.e., processor <NUM>) and/or CPU (i.e., processor <NUM>, as shown by communication link <NUM> and/or <NUM>, respectively. For example, EM data, identified poses, and/or other information may be exchanged between device <NUM> and computing device <NUM> via dongle <NUM> using any of processor <NUM>, <NUM>, and/or <NUM>. For example, in some implementations, the processor <NUM> identifies poses based on the EM data identified via module <NUM> and communicates the identified poses to the processor <NUM> directly or via processor <NUM> on the dongle <NUM>. The dongle can then pass (via USB, for example) the poses to the computing device <NUM>. In some implementations, the processor <NUM> communicates the EM data to the processor <NUM> or processor <NUM>, which identifies poses based on the EM data.

The communication links <NUM> and <NUM> can be a wired communication link, a wireless communication link (e.g., Bluetooth, ZigBee, RF, etc.), and the like, or a combination thereof. In other embodiments, the EM data can be sent to another device and/or processor (not shown) and the other device and/or processor may compute a pose from the EM data. In some implementations, the EM data may be stored locally within devices <NUM>, <NUM>, or <NUM>, locally within the system <NUM>, and/or remote from system <NUM>.

The input device <NUM> also includes the EM receiver module <NUM> to generate EM data from detected EM fields. The EM receiver module <NUM> includes a receiver coil <NUM>, an amplifier <NUM>, and an ADC <NUM>. In some implementations, the receiver coil <NUM> is a tri-axis coil configured to detect an analog electrical signal having a magnitude and/or phase indicative of a particular detected EM field. The ADC <NUM> is generally configured to receive the generated analog signal and convert the analog signal to a digital value indicative of the analog signal represented in the EM field <NUM>. The digital values generated by the ADC <NUM> are EM data that can be used for pose identification for input device <NUM>.

The input device <NUM> also includes an IMU sensor <NUM>. The IMU sensor <NUM> may provide additional pose information about device <NUM>. One or more of the processors <NUM>, <NUM>, or <NUM> (or another device processor) can use the additional pose information to supplement or augment the poses identified based on the EM data. For example, in some embodiments the processor <NUM> can use the additional pose information to identify potential errors in the poses determined based on the EM data, and to address the identified errors.

In some implementations, the processor <NUM> may use the additional pose information determined using IMU <NUM> to track input device <NUM>, for example, when device <NUM> is out of range of the EM field produced by transmitter module <NUM>. Thus, the IMU <NUM> may be utilized as a backup pose determiner in order to continue determining pose information and tracking data for device <NUM>. For example, if device <NUM> is the input device <NUM> and the user operating device <NUM> moves out of range of computing device <NUM> (or dongle <NUM>), the system <NUM> can still determine pose information well enough to track a moving device <NUM> such that content (e.g., cursor or 3D objects) within the computing device <NUM> can be properly moved based on movements of tracked input device <NUM>. In this example, the pose information may be <NUM>-DoF pose data based on detected 3D orientations of the device <NUM>/<NUM>.

In some implementations, the dongle <NUM> includes the processor <NUM> and a wireless interface communicably coupled to the computing device <NUM> and communicably coupled to the input device <NUM>. The dongle <NUM> may be operable to collect, from the processor <NUM>, pose data associated with the input device <NUM>. In some implementations, the dongle <NUM> may be operable to determine pose information from input device <NUM> when the device is not moving. The dongle <NUM> may then convert, using the processor <NUM> or <NUM>, the position and orientation data (i.e., pose data). In some implementations, the dongle <NUM> may instead convert, using the processor <NUM> or <NUM>, the position and/or orientation data retrieved from the IMU <NUM> of device <NUM>. In either case, the position and/or orientation data may be used to generate commands for execution by the computing device <NUM>. The commands may be transmitted from the dongle <NUM> to the computing device <NUM> over the USB interface generated by connectors <NUM>/<NUM>, for example.

The input device <NUM> may also include a microphone <NUM> and a speaker <NUM>. The input device <NUM> may include the microphone <NUM> for capturing audio spoken by user(s) accessing device <NUM>. For example, the microphone may be configured to receive voice-based queries for generating communication from the input device <NUM> to the computing device <NUM>. In some implementations, the captured audio is stored on the mouse for future playback or upload to a computer. In other embodiments, the audio is streamed to a computing device via a wireless connection for either storage or processing.

In some implementations, the speaker <NUM> is configured to generate audio playback from the input device <NUM>. The audio playback may include information responsive to the voice-based queries received at the microphone <NUM>. For example, if a user spoke a question or command into the microphone <NUM>, the system <NUM> may stream the question to the computing device <NUM> (directly or via dongle <NUM>), the computing device <NUM> may respond with an answer that is played back to the user through the speaker <NUM>.

Although the microphone <NUM> and speaker <NUM> are not shown in systems <NUM>, <NUM>, <NUM>, 600A, and 600B, both devices <NUM> and <NUM> may be included in a similar fashion, as described above.

In some implementations, the dongle <NUM> can be attached to the input device <NUM> for storage when not plugged into the computing device <NUM>. In some implementations, the input device <NUM> may contain a female connector on a rear surface (e.g., for charging) and the dongle <NUM> may contain a male connector that can be mated with the input device female connector for storage. In another implementation, the input device <NUM> contains a compartment in which the dongle <NUM> can be stored.

<FIG> is a flow chart diagramming an implementation of a process <NUM> to track an input device in order to determine which data is used to generate commands for an associated computing device, in accordance with implementations described herein. The method <NUM> is described with respect to an example implementation at the EM tracking system <NUM> of <FIG>, but it will be appreciated that the method can be implemented at EM tracking systems having other configurations.

At block <NUM>, the process <NUM> may include detecting (at air mouse device <NUM>) first data associated with the air mouse device <NUM>. The first data may include movement data, position data, orientation data, or any combination thereof. The first data may be detected using at least one onboard EM-based sensor in system <NUM>. For example, the device <NUM> may use EM receiver module <NUM> to detect position data and/or orientation data associated with the air mouse device <NUM>. The EM receiver module may be associated with and/or capable of communication with the base device <NUM>, which may be installed upon (or otherwise accessible to the computing device <NUM> displaying content in which the air mouse device <NUM> may control. In some implementations, the EM receiver module <NUM> may detect the first data to determine pose information for the device <NUM>. The first data and the second data may pertain to position data, orientation, data, or both (i.e., pose data).

At block <NUM>, the process <NUM> may include detecting additional data (i.e., second data) associated with the air mouse device <NUM>. The second data may be detected using at least one onboard non-EM based sensor. For example, the non-EM based sensor may include the IMU <NUM> and the second data may include accelerometer and gyroscope information about device <NUM>. The second data may be detected with respect to device <NUM> as a user moves device <NUM> in 3D space. In some implementations, the second data may be detected when device <NUM> is not moving.

At some point, a user may move device <NUM> and may cause device <NUM> to become in or out of range of an EM field associated with a base device (e.g., EM transmitter module <NUM> on device <NUM> installed in computing device <NUM>). The range may be a predefined proximity range associated with the computing device. For example, the range may be defined based on a particular strength (e.g., transmit power) of the EM transmitter module <NUM> used to generate the EM field transmitting data between device <NUM> and device <NUM>. An example range may include about one to about three meters. Other ranges are possible.

At block <NUM>, a decision may be made for whether or not the device <NUM> is within range of the computing device <NUM> (connected to transmitter module <NUM>). In response to determining that the air mouse device <NUM> is within a predefined proximity range of the computing device <NUM> and/or transmitter module <NUM>, the device <NUM> may generate pose-related data (block <NUM>) that a host computing device may translate into commands to interact with content displayed on the computing device using the first data. For example, commands may be generated by device <NUM> based at least in part on EM-based data because the device <NUM> is within range to use and transmit such data.

In response to determining that the air mouse device <NUM> is out of range of the predefined proximity range of the computing device, the device <NUM> may generate commands (block <NUM>) to interact with content displayed on the computing device using the second data. For example, commands may be generated using IMU data based on detecting that the EM-based elements are unavailable for use.

Regardless of which data (first data or second data) is used, the air mouse device <NUM> may trigger transmission of pose-related data to the host computing device which may be used to manipulate the content displayed in the computing device, at block <NUM>. The system <NUM> may continue to execute and determine first data and second data. Each time commands are to be generated, the system again begins to determine whether or not input device is within range of the computing device, as shown by arrow <NUM>. For example, device <NUM> can continue to detect proximity to device <NUM> to determine whether or not EM data can be collected to identify poses. Thus, the process flow returns to block <NUM>.

<FIG> shows an example computer device <NUM> and an example mobile computer device <NUM>, which may be used with the techniques described here. Features described with respect to the computer device <NUM> and/or mobile computer device <NUM> may be included in the portable computing device <NUM> described above. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices.

In addition, GPS (Global Positioning System) receiver module <NUM> may provide additional navigation- and locationrelated wireless data to device <NUM>, which may be used as appropriate by applications running on device <NUM>.

Implementations of the various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device or in a propagated signal, for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. A computer program, such as the computer program(s) described above, can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

Method steps may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output.

Information carriers suitable for embodying computer program instructions and data include all forms of nonvolatile memory, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in special purpose logic circuitry.

To provide for interaction with a user, implementations may be implemented on a computer having a display device, e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.

Implementations may be implemented in a computing system that includes a backend component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a frontend component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation, or any combination of such backend, middleware, or frontend components. Components may be interconnected by any form or medium of digital data communication, e.g., a communication network.

The computing device according to example embodiments described herein may be implemented using any appropriate combination of hardware and/or software configured for interfacing with a user including a user device, a user interface (UI) device, a user terminal, a client device, or a customer device. The computing device may be implemented as a portable computing device, such as, for example, a laptop computer. The computing device may be implemented as some other type of portable computing device adapted for interfacing with a user, such as, for example, a PDA, a notebook computer, or a tablet computer. The computing device may be implemented as some other type of computing device adapted for interfacing with a user, such as, for example, a PC. The computing device may be implemented as a portable communication device (e.g., a mobile phone, a smart phone, a wireless cellular phone, etc.) adapted for interfacing with a user and for wireless communication over a network including a mobile communications network.

The computer system (e.g., computing device) may be configured to wirelessly communicate with a network server over a network via a communication link established with the network server using any known wireless communications technologies and protocols including radio frequency (RF), microwave frequency (MWF), and/or infrared frequency (IRF) wireless communications technologies and protocols adapted for communication over the network.

In accordance with aspects of the disclosure, implementations of various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may be implemented as a computer program product (e.g., a computer program tangibly embodied in an information carrier, a machine-readable storage device, a computer-readable medium, a tangible computer-readable medium), for processing by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). In some implementations, a tangible computer-readable storage medium may be configured to store instructions that when executed cause a processor to perform a process. A computer program, such as the computer program(s) described above, may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be processed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments. Example embodiments, however, may be embodied in many alternate forms and should not be construed as limited to only the embodiments set forth herein.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments. It will be further understood that the terms "comprises," "comprising," "includes," and/or "including," when used in this specification, specify the presence of the stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof.

It will be understood that when an element is referred to as being "coupled," "connected," or "responsive" to, or "on," another element, it can be directly coupled, connected, or responsive to, or on, the other element, or intervening elements may also be present. In contrast, when an element is referred to as being "directly coupled," "directly connected," or "directly responsive" to, or "directly on," another element, there are no intervening elements present. As used herein the term "and/or" includes any and all combinations of one or more of the associated listed items.

Spatially relative terms, such as "beneath," "below," "lower," "above," "upper," and the like, may be used herein for ease of description to describe one element or feature in relationship to another element(s) or feature(s) as illustrated in the figures. Thus, the term "below" can encompass both an orientation of above and below. The device may be otherwise oriented (rotated <NUM> degrees or at other orientations) and the spatially relative descriptors used herein may be interpreted accordingly.

Example embodiments of the present inventive concepts are described herein with reference to cross-sectional illustrations that are schematic illustrations of idealized embodiments (and intermediate structures) of example embodiments. Thus, example embodiments of the present inventive concepts should not be construed as limited to the particular shapes of regions illustrated herein but are to include deviations in shapes that result, for example, from manufacturing. Accordingly, the regions illustrated in the figures are schematic in nature and their shapes are not intended to illustrate the actual shape of a region of a device and are not intended to limit the scope of example embodiments.

It will be understood that although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. Thus, a "first" element could be termed a "second" element without departing from the teachings of the present embodiments.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this present inventive concept belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and/or the present specification and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Claim 1:
A handheld electronic device configured for controlling three-dimensional content displayed in a user interface of a computing device, the handheld electronic device including:
an electromagnetic sensing system for detecting, for the handheld electronic device, a pose of the handheld electronic device in three-dimensional space, wherein the pose is defined by a three- dimensional position and a three-dimensional orientation of the handheld electronic device;
an inertial measurement unit sensor for detecting, for the handheld electronic device, an orientation in three-dimensional space of the handheld electronic device in three-dimensional space;
at least one processor coupled to memory, the at least one processor configured to generate commands to manipulate the three-dimensional content in the computing device,
characterized in that
the commands are generated based on a determined proximity of the handheld electronic device relative to a receiver module associated with the computing device, the determined proximity triggering selection of data for use in generation of the commands, the data including:
the pose of the electromagnetic sensing system when the determined proximity indicates that the handheld electronic device is within range of the receiver module, and
the orientation of the inertial measurement unit sensor when the determined proximity indicates that the handheld electronic device is out of range of the receiver module; and
at least one communication module to trigger transmission of the commands to manipulate the three-dimensional content displayed in the computing device based on detected changes in pose of the handheld electronic device.