Correcting angle error in a tracking system

To correct an angle error, acceleration data is received corresponding to a tracked object in a reference frame of the tracked object. Positional data of the tracked object is received from a positional sensor, and positional sensor acceleration data is computed from the received positional data. The acceleration data is transformed into a positional sensor reference frame using a rotation estimate. An amount of error between the transformed acceleration data and the positional sensor acceleration data is determined. The rotation estimate is updated responsive to the determined amount of error.

FIELD OF THE INVENTION

The present invention relates generally to a tracking system, and more particularly to compensating for errors in the angles of the tracking system.

DESCRIPTION OF THE RELATED ART

A growing trend in the computer gaming industry is to develop games that increase the interaction between a user and a gaming system. One way of accomplishing a richer interactive experience is to use game controllers whose movement is tracked by the gaming system in order to track the player's movements and use these movements as inputs for the game. Generally speaking, gesture input refers to having an electronic device such as a computing system, video game console, smart appliance, etc., react to some gesture captured by a video camera or other positional sensor that tracks an object.

Typically, in order to produce reliable measurements of the location and motion of the user, the gaming system needs to be calibrated. Such calibration is commonly necessary each time the gaming system is used. The calibration process determines initial values for angles (e.g., yaw, pitch, and roll) of the controller in relation to the positional sensor. Since a user typically moves the controller frequently, and these movements are variable with many different accelerations, these initial angles may need to be updated frequently. These updated angles may be estimated by integrating angular rate measurements from inertial sensors on the controller. However, a slight error is introduced each time this integration is performed. Thus, over time, the values for the angles will drift, leading to a certain amount of angle error.

In certain circumstances, the pitch and roll angles may be determined absolutely by measuring the direction of gravity when the controller is not accelerating. Magnetometers may be used to determine absolute yaw angle, but magnetometers are affected by metal and magnetic interference.

DETAILED DESCRIPTION

Described herein is a method and apparatus for correcting angle errors in a tracking system for use in a gaming system. In one embodiment of the invention, to correct an angle error, acceleration data is received corresponding to a tracked object in a reference frame of the tracked object. Positional data of the tracked object is received from a positional sensor, and positional sensor acceleration data is computed from the received positional data. The acceleration data is transformed into a positional sensor reference frame using a rotation estimate. An amount of error between the transformed acceleration data and the positional sensor acceleration data is determined. The rotation estimate is updated responsive to the determined amount of error.

The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. In one embodiment, the apparatus for performing the operations herein includes a game console (e.g., a Sony Playstation®, a Nintendo Wii®, a Microsoft Xbox®, etc.). A computer program may be stored in a computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks (e.g., compact disc read only memory (CD-ROMs), digital video discs (DVDs), Blu-Ray Discs™, etc.), and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions.

A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, a machine-readable medium includes a machine readable storage medium (e.g., read only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory devices, etc.), a machine readable transmission medium (electrical, optical, acoustical or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.)), etc.

FIG. 1illustrates a perspective view of a tracking system100, in accordance with one embodiment of the present invention. The tracking system100includes a positional sensor105, an object110that is tracked by the positional sensor105, and a computing device115that processes data received by the positional sensor105and by the object110. In one embodiment, the tracking system100is a component of a gaming system. Alternatively, the tracking system100may be a component of a motion capture system.

The positional sensor105is a sensor that measures positions of the object110in two-dimensional or three-dimensional space relative to the positional sensor105. The positional sensor105may be a video camera, a Z-camera, a stereo camera, a web camera, an ultrasonic sensor array, a photonic detector, or other device capable of measuring the position of the object110. For example, if the positional sensor105is a camera, the positional sensor105generates a plurality of images including the object110as it is moved.

Positional measurements taken by the positional sensor105are in a reference frame150of the positional sensor105that is defined by a positional sensor measurement plane and a vector normal to the positional sensor measurement plane. A reference frame is defined herein as a coordinate system within which to measure an object's position, orientation and other properties. The terms reference frame and coordinate system are used interchangeably throughout this application.

As shown, the positional sensor105is positioned on top of a television set120, with a negative pitch145relative to a floor150. The pitch145is an angle between a horizontal axis of the positional sensor's reference frame150that is in the measurement plane of the positional sensor105and a plane perpendicular to gravity135. As long as the pitch145is a non-zero value, the positional sensor105has a reference frame150that is different from a world reference frame140(defined as a reference frame that has an axis (e.g., y2) aligned with gravity135).

In one embodiment of the invention, the positional sensor105is a standard video camera. In such an embodiment, the positional sensor105may capture depth information (distance130between the positional sensor105and the object110) based on predefined information that identifies a size of the object110and/or based on predefined information that identifies a field of view (FOV)125of the positional sensor105. The field of view125is the angular extent of a given scene imaged by the positional sensor105. The field of view defines the distortion (e.g., amount of zoom) of an image caused by a camera lens. As the object110is moved further from the positional sensor105(that is, as the distance130is increased), an image of the object110as captured by the positional sensor105becomes smaller. Therefore, the distance130of the object110to the positional sensor105can be determined based on a ratio of the image size of the tracked object110(e.g., as measured in pixels) to a known actual size of the tracked object110provided that a field of view125of the positional sensor105is known.

In another embodiment of the invention, the positional sensor105is a Z-camera (a single lens video camera capable of capturing video with depth information) or a stereo camera (video camera with 2 or more lenses that can capture three-dimensional images). In such an embodiment, the positional sensor105can capture depth information without being pre-configured with information identifying a size of the object110.

In yet another embodiment of the invention, the positional sensor105is a sensor array such as an ultrasonic sensor array or a photonic detector. Such a positional sensor105detects the distance between the positional sensor105and the object110using time of flight or phase coherence (e.g., of light or sound), and detects vertical and horizontal positions of the object110using triangulation.

The object110is an electronic device that includes one or more inertial sensors. The inertial sensors may measure accelerations along a single axis or multiple axes, and may measure linear as well as angular accelerations. In one embodiment, the object110is a hand held electronic device or a portion of a handheld electronic device such as a game controller, as shown inFIGS. 2A and 2B. In another embodiment, the object110is a motion capture (mocap) ball, as shown inFIG. 2C. The object110may have an arbitrary shape, such as a square, sphere, triangle, or more complicated shape. In one embodiment, the object110has a spherical shape.

FIG. 2Aillustrates a game controller200having a ball section205, in accordance with one embodiment of the present invention.FIG. 2Billustrates another game controller210having a ball section215, in accordance with another embodiment of the present invention. In certain embodiments, the ball sections205and215correspond to object110ofFIG. 1.

The ball sections205,215can be of different colors, and in one embodiment, the ball sections205,215can light up. Although a spherical ball section is illustrated, the ball sections205,215can have other shapes for visual tracking purposes, such as a partial sphere, an imperfect sphere, an elongated ball (like one used in American football or in rugby), a cube-like shape, etc. In one embodiment, the ball section205,215is 4 cm. in diameter. However, other larger or smaller sizes are also possible. Larger sizes help with visual recognition. For example, a ball with a 5 cm. diameter can provide about 55 percent more pixels for image recognition than a 4 cm. ball.

FIG. 2Cillustrates multiple mocap balls220disposed on a user225, in accordance with one embodiment of the present invention. Mocap balls220are markers that are worn by a user225near each joint to enable a positional sensor to capture and identify the user's motion based on the positions or angles between the mocap balls220. In one embodiment, the mocap balls220are attached to a motion capture suit.

Returning toFIG. 1, object110and positional sensor105are connected with computing device115through wired and/or wireless connections. Examples of wired connections include connections made via an IEEE 1394 (firewire) cable, an ethernet cable, and a universal serial bus (USB) cable, etc. Examples of wireless connections include wireless fidelity (WiFi™) connections, Bluetooth® connections, Zigbee® connections, and so on. In the illustrated embodiment, object110is wirelessly connected with computing device115and positional sensor105is connected with computing device115via a wired connection.

Computing device115may be a video game console, a personal computer, a game kiosk, or other computing apparatus. Computing device115may execute games or other applications that can respond to user input from object110. The object110is tracked, and motion of the object110provides the user input.

Before the tracking system100can accurately track the object110, the tracking system100needs to be calibrated. For example, calibrating the tracking system100may include computing a pitch145of the positional sensor105and computing a relative yaw between the positional sensor105and the object110. The relative yaw between the object110and the positional sensor105represents the differences in heading between the object110and the positional sensor105. In one embodiment, zero yaw is defined as being achieved between the positional sensor105and the object110when the object is pointed perpendicular to an imaging plane of the positional sensor105. Alternatively, zero yaw may be defined as being achieved when the object110is pointed directly toward the positional sensor105. If the positional sensor105is a camera with an unknown field of view125, calibrating the tracking system100also includes computing the field of view125of the positional sensor105. If the object110has an unknown size, calibrating the tracking system100may also include determining the size of the object110.

In order for the inertial sensor disposed in the object110to gather sufficient inertial data for calibration, the object110should be moved by a user. The object110should be moved within the frame of view125of the positional sensor105to ensure that each inertial data measurement has a corresponding position measured by the positional sensor105. An effectiveness of the calibration can be increased if the object110is moved through a path within the field of view125of the positional sensor105such that it exceeds a minimum threshold and occurs in at least two dimensions (e.g., a plane). The path may include movement towards and/or away from the positional sensor105. As will be described later herein, inertial data corresponding to the object110is also required when correcting for angle errors in the tracking system.

Received inertial data has an uncertainty that is defined by an amount of signal noise that accompanies the inertial data. As the magnitude of accelerations measured by the object110decreases, a signal to noise ratio (ratio of a signal power to the noise power corrupting the signal) increases. A decrease in the signal to noise ratio causes the inertial data to become less accurate. In one embodiment, the minimum acceleration threshold is set to prevent the inertial data from falling below a minimum signal to noise ratio.

An example of an object being moved through a path in accordance with one embodiment of the present invention is illustrated inFIG. 3. As shown, the object is a ball305attached to the end of a game controller310and the path315is a circular path about a user's head. Circular paths are advantageous in that movement in a circle provides constant acceleration. Therefore, a circular path provides increased inertial data. The entire path315occurs within a field of view320of a positional sensor, and includes motion toward and away from the positional sensor.

FIG. 4illustrates a block diagram of a tracking system400, in accordance with one embodiment of the present invention. The tracking system400includes a computing device415physically connected with a positional sensor405and wirelessly connected with an object410that is tracked by the positional sensor405. Of course, it should be understood that in some embodiments of the invention the positional sensor405is wirelessly connected with the computing device415. In one embodiment, the tracking system400corresponds to tracking system100ofFIG. 1.

In one embodiment of the invention the object410includes one or more inertial sensors420in a fixed position within the object410, however, in alternative embodiments of the invention the inertial sensor(s)420are outside of the object410(e.g., within a controller coupled with the object110). In one embodiment, the inertial sensors420include one or more gyroscopes and one or more accelerometers. Gyroscopes use principals of angular momentum to detect changes in orientation (e.g., changes in pitch, roll and twist). Accelerometers measure accelerations along one or more axes. The gyroscope and accelerometer may be separate sensors, or may be combined into a single sensor. In one embodiment, the gyroscope and accelerometer are micro-electromechanical systems (MEMS) devices. As the object410is moved, the inertial sensors420gather inertial data corresponding to the object410(e.g., acceleration data) and transmit the data to the computing device415. Inertial data gathered by the inertial sensors420is in a reference frame of the object410.

As the object410is moved, the positional sensor405captures positional measurements of the object410that may include image size and image location information. For example, in the case the positional sensor is a camera, the positional sensor405generates a plurality of images of the object410while the object is moved (of course, it should be understood that in one embodiment of the invention, the positional sensor405generates images of the object410regardless of whether the object410is being moved). The positional sensor405then transmits the positional measurements to the computing device415. In one embodiment, the positional sensor405streams the positional measurements to the computing device415in real time as the measurements are captured.

In one embodiment of the invention, as the object410is tracked by the positional sensor405, changing positions of the object410are used as an input to the computing device415to control a game, computer application, etc. For example, changing positions of the object410can be used to control a character in a first person or third person perspective game, to move a mouse cursor on a screen, and so on. In another embodiment, the inertial data received from the object410is used as an input to the computing device415. Alternatively, the inertial data may be used in combination with the positional measurement data obtained by the positional sensor405to provide a precise and accurate input for the computing device415.

The computing device415may be a video game console, personal computer, game kiosk, etc. In one embodiment of the invention, the computing device415includes an inertial data logic component425, a positional measurement logic component430and an angle error estimation logic component435, each of which performs different operations.

Positional measurement logic component430analyzes positional measurement data (e.g., images) received from the positional sensor405to find the position of the object410(e.g., in the positional sensor reference frame150). For example, if the positional sensor405is a digital camera, the positional measurement logic component430analyzes images received to find the object410in the images. In one embodiment of the invention, an image location of the object410is determined by analyzing a pixel group that represents the object410in the image to find the object's centroid. In one embodiment, a Gaussian distribution of each pixel can be calculated and used to provide sub-pixel accuracy of the centroid location.

In one embodiment of the invention, the positional measurement logic component430converts the positional location information to a three dimensional coordinate system of the positional sensor405. The positional measurement locations may be converted to the three dimensional coordinate system using the following equations:

Once the positional measurement data is converted into the three dimensional coordinate system, the positional measurement logic component430takes a second derivative of the location information with respect to time to compute an acceleration of the object410in the three dimensional coordinate system of the positional sensor405based on changing positions of the object410over time. The positional measurement logic component430then provides the computed positional sensor acceleration data to the angle error estimation logic component435.

The inertial data logic component425processes the inertial data. In one embodiment of the invention, for each angle to be corrected, the inertial data logic component425removes an acceleration caused by gravity from the inertial data, and transforms the remaining acceleration from the frame of reference of the object410to the frame of reference of the positional sensor405using a rotation estimate for that angle. The resulting data (in its transformed state) is passed to the angle error estimation logic component435. According to one embodiment of the invention, the rotation estimate is based on the previously determined value of the angle. For example, the first time the error correction process is performed, the rotation estimate is based on the initial value for the angle.

In one embodiment of the invention, the transformation of the inertial data from the object's frame of reference to the positional sensor's frame of reference is performed only if the inertial data received meets a certain threshold (e.g., the data has a magnitude greater than a predetermined noise floor). If the inertial data does meet that certain threshold, the transformation will not take place and the error correction process will not continue.

In one embodiment of the invention, the received inertial data is weighted based on the magnitude of that inertial data. For example, the accuracy of any error correction generally increases as the magnitude of received acceleration data increases. Thus, larger magnitudes of received acceleration data received from the object410(via the inertial sensor(s)420) are weighted more than lower magnitudes of received acceleration data.

The angle error estimation logic component435compares the computed acceleration from the positional sensor logic component430with the inertial data from the inertial data logic component425to estimate the amount of error in an angle (e.g., yaw angle of the tracked object410relative to the positional sensor405, pitch angle, or roll angle) in the tracking system400. It should be understood that each angle has an initial value (e.g., computed during a calibration of the tracking system, etc.). For example, in the case of correcting for yaw angle drift, the pitch of the positional sensor405is known and an initial relative yaw between the positional sensor405and the tracked object410has been determined. Since a user typically moves the tracked object410frequently, and these movements are variable with many different accelerations, these initial angles need to be frequently updated. In one embodiment of the invention, an updated angle may be estimated by integrating angular rate measurements from the inertial sensor(s)420. For example, in the case of the yaw angle, the yaw angle of the tracked object410may be continually computed by integrating data from the inertial sensor(s)420. However, each time this integration is performed, there is a slight error. Thus, over time, the yaw angle value will drift. Thus, the yaw angle starts as known (the initial yaw angle) but will drift over time. As another example, in the cause of tilt (pitch and roll angles), the tilt of the object410may be continually determined from gravity if the object410is held still for a certain amount of time (e.g., typically less than a second). However, if the object410is not moved (e.g., a user does not move a game controller coupled with the object410for a certain amount of time) tilt may not be determined from gravity.

The transformed inertial data can be represented by a first set of vectors, and the positional sensor acceleration data (computed from the positional data gathered from the positional sensor405) can be represented as a second set of vectors. The positional sensor acceleration data and the transformed inertial data represent different measurements of the same object motion. Therefore, the positional sensor acceleration data and the transformed inertial data include a matched set of vectors, in which each vector in the first set of vectors corresponds to a vector in the second set of vectors. Corresponding vectors have the same origin and the same magnitude. Accordingly, the transformed inertial data can be compared with the positional sensor acceleration data to determine an amount of error in the angle (if the two data sets match, then the rotation estimate was accurate and the angle does not have an error to correct).

In one embodiment of the invention, to determine the amount of error for an angle due to drifting (e.g., yaw, pitch, or roll), the positional sensor acceleration data is transformed by rotating vectors of the positional sensor acceleration data by multiple potential angles and the transformation is compared with the transformed inertial data vectors. Many different combinations of potential angles may be tried. The combination of a potential angle that yields the smallest difference between the rotated vectors of the positional sensor acceleration and the transformed inertial data vectors is the estimated correct angle. The estimated amount of error is the difference between the angle estimated from integrating angular rate measurements from the inertial sensor(s)420(the inertial data) and the estimated correct angle. A gradient descent technique, partial least squares technique, least squares fit technique, or other error minimization technique may be used to minimize the number of combinations of potential angles that are tested to find the estimated correct angle.

For example, each vectorVithat comprises object motion is represented in the first set of vectors asVi1(the positional sensor acceleration data vectors) and in the second set of vectors asVi2(the transformed inertial data vectors). For all vectorsVi1in the first set of vectors, the vectorVi1is transformed by a three dimensional rotation that is based on the potential angles, ending up with a transformed vectorVi1′. A differenceDθis then found betweenVi1′ andVi2for each vectorVi. The sum of the differences between the rotated vectorsVi1′ of the first set of vectors and the matching vectorsVi2of the second set of vectors is then calculated using the following equation:

The angle θ that yields the smallest Dθis the correct estimated three dimensional angle. Therefore, the potential angle that makes up the three dimensional angle θ represents the estimated correct angle. The estimated amount of error may be determined by the difference between the angle estimated from integrating angular rate measurements from the inertial sensor(s)410(the inertial data) and the estimated correct angle. Although it has been described that the positional sensor acceleration data vectors are transformed and compared with the transformed inertial data vectors, it should be understood that it does not matter which set of vectors is rotated since the angle between the two sets of the vectors is the same regardless of which set is rotated.

In another embodiment of the invention, to determine the amount of error for an angle due to drifting (e.g., yaw, pitch, or roll), a three dimensional rotation is computed that aligns the coordinate system of the transformed inertial data and the positional sensor acceleration data. A three dimensional rotation can be defined by an axis and an angle. The rotation axis ê for the three dimensional rotation can be solved for by taking the cross product of vectors of the first set of vectors Vi1with vectors of the second set of vectors Vi2, as follows:
ê=Vi1×Vi2(equation 5)

The dot product of Vi1and Vi2can also be taken to find a scaler s that represents the projection of Vi1onto Vi2, as follows:
s=Vi1·Vi2(equation 6)

The angle of the three dimensional rotation θ can then be solved for using the rotation axis and the scalar as follows:

The three dimensional rotation may then be decomposed into either of the reference frames to determine the angles of rotation about each of the axes of that reference frame. For example, the three dimensional rotation can be projected onto the x-axis to determine the amount of rotation that is occurring about the x-axis, and can be projected onto the y-axis to determine the amount of rotation that is occurring about the y-axis. If the three dimensional angle is projected into the object reference frame, the rotation about the axis that is aligned with gravity is the yaw, and the rotation about the axis that is perpendicular to gravity and in the positional sensor plane of the positional sensor is the pitch. The three dimensional rotation can be decomposed into rotations about the axes of the reference frames using, for example, three orthogonal matrices, a rotation matrix or a quaternion.

Quaternions form a four dimensional normed division algebra over the real numbers. Quaternions provide a useful representation of coordinate transformations because they can be computed faster than matrix transformations, and never lose their orthogonality. A quaternion representation of a rotation is written as a four dimensional vector:
q=[q1q2q3q4]T(equation 8)

In terms of the world reference frame, the quaternion's elements are expressed as follows:

⁢q1=e^x⁢⁢sin⁡(θ2)❘vi⁢⁢1×vi⁢⁢2❘(equation⁢⁢9)q2=e^y⁢⁢sin⁡(θ2)vi⁢⁢1×vi⁢⁢2(equation⁢⁢10)q3=e^z⁢⁢sin⁡(θ2)vi⁢⁢1×vi⁢⁢2(equation⁢⁢11)q4=cos⁡(θ2)(equation⁢⁢12)
Where êx, êy, and êzrepresent the unit vectors represent the unit vectors along the x, y and z axes of the object reference frame, respectively.

The quaternion fully represents both the yaw and pitch rotations that are necessary to align the reference frames. However, to provide a more intuitive result, the quaternion may be converted to Euler angles of pitch and yaw according to the following formulas:

The decomposed angle corresponds to the estimated correct angle. The estimated amount of error can be determined by calculating the difference between the angle estimated from integrating angular rate measurements from the inertial sensor(s)410) (the inertial data) and the estimated correct angle.

In another embodiment of the invention, the sets of vectors may be projected onto a plane and a computation performed to compute a two-dimensional angle. For example, the positional sensor acceleration data vectors (Vi1) and the transformed inertial data vectors (Vi2) are projected onto a plane and the rotation is computed around the plane's normal which aligns the two sets of vectors in the plane. For example, for yaw angle correction, the transformed inertial data vectors (the acceleration data vectors) and the positional sensor acceleration data vectors are projected onto the XZ plane, and a rotation is found around the Y axis. Typically, this computation may be performed when at least one angle is known. For example, typically the pitch of the positional sensor405is known from an initial calibration of the tracking system (that is, the positional sensor405is typically fixed). The following equations may be used to project the positional sensor acceleration data vectors (Vi1) and the transformed inertial data vectors (Vi2) onto the plane, given the plane's unit-length normal vector ĝ:
projected vector (vi1′)=vi1−ĝ(vi1·ĝ)   (equation 15)
projected vector (vi2′)=vi2−ĝ(vi2·ĝ)   (equation 16)

The projected vectors are then rotated about the plane's normal. For example, the equations 9, 10, and 11 may be used to rotate the projected vectors around the x, y, and z axes respectively. The projected vectors are then compared to determine an estimated correct angle. The amount of error in the angle is computed based on the difference between the estimated correct angle and the angle computed from integration of the inertial data.

After the estimated error is determined, a correction is made to the rotation estimate. According to one embodiment of the invention, the correction amount is a portion of the estimated error (e.g., a percentage of the estimated error). Thus, the rotation estimate used to transform the acceleration received from the sensor(s)420of the object410from the frame of reference of the object410to the frame of reference of the positional sensor405is updated according to the correction amount.

According to one embodiment of the invention, the correction amount is a function of an amount of time that has passed since a previous correction. Thus, in one embodiment of the invention, the more time elapsed since the last correction the greater the amount of correction. For instance, if a user has not moved the object410(e.g., if the user has placed the controller coupled with the object410down) for a certain amount of time (e.g., 5-10 minutes), a considerable amount of drift may have occurred. In this case, the amount of correction may be a large percentage of the estimated error, or even the full percentage of the estimated error. According to one embodiment of the invention, if the user has not moved the object410for a certain amount of time, the user will be locked from using the object410in the tracking system until the user moves the object410in a motion path that gives a minimum threshold of acceleration and positional data. In one embodiment of the invention, the object410emits a certain color of light and/or sound when a user needs to move the object410to produce a minimum threshold of acceleration and positional data.

In another embodiment of the invention, the amount of correction is a function on the magnitude of the vector sets. For example, as described previously, in one embodiment of the invention, the vectors are weighted (the larger the magnitude of a vector the larger the weight). The larger the weight of the vectors, the larger the amount of correction. Of course, it should be understood that any combination of the above embodiments may be used (e.g., a percentage of the estimated error, a function of amount of time that has passed, and/or a function on the weights associated with the magnitude of the vectors).

FIG. 5shows a schematic diagram of a multiplayer environment500, in which visual information is used to determine the locations of different controllers held by players, according to one embodiment. In the multiplayer environment500, positional sensor508(illustrated and described below as a camera) obtains image data of a playing field518, and the image data is analyzed to obtain the location of ball-attached controllers C1, C2, C4and C5. Distances dz1, dz2, dz4, and dz5are estimated by analyzing the shape and size of the respective balls in the captured image. A computing device502uses the obtained coordinates and distances to produce representations of the players in screen504, avatars512aand512brespectively. A typical distance for good image recognition is about 10 ft (3 mtr). One advantage of using visual recognition is that improvements in image capture and image recognition can be included in the system without having to change the controller.

FIG. 6is a flow diagram illustrating an exemplary method600of compensating for angle error in a tracking system according to one embodiment of the invention. The method may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (such as instructions run on a processing device), or a combination thereof. In one embodiment, method600is performed by computing device415ofFIG. 4.

Referring toFIG. 6, at block605the computing device415receives positional data from the positional sensor405as the tracked object410is moved. In one embodiment of the invention, the computing device receives images of the object in a two-dimensional reference frame of the imaging device. Flow moves from block605to block610

At block610, the computing device415receives inertial data (e.g., via the inertial sensor(s)420) from the tracked object410as the tracked object is moved. The inertial data is in a reference frame of the object410. Flow moves from block610to block615. At block615, the computing device415converts the positional data to a coordinate system of the positional sensor405. Flow moves from block615to block620, where the computing device415computes positional sensor acceleration data of the object410from the positional data. For example, the computing device415may take a second derivative of the positional data with respect to time to compute the positional sensor acceleration data of the object410. In one embodiment of the invention, the positional sensor acceleration data is computed in real time as the positional sensor data is received. Alternatively, the positional sensor acceleration data may be computed periodically (e.g., every 10 milliseconds, every half second, etc.), or when the motion is completed. Flow moves from block620to block625.

At block625, the computing device415transforms the received inertial data into a reference frame of the positional sensor using a rotation estimate. According to one embodiment of the invention, the rotation estimate is based on the previously determined value of the angle. For example, the first time the error correction process is performed, the rotation estimate is based on the initial value for the angle (e.g., determined during a configuration of the tracking system). Flow moves from block625to block630.

At block630, the computing device415determines an amount of error between the transformed inertial data and the positional sensor acceleration data.FIG. 7is a flow diagram illustrating one method of determining an amount of error between the transformed inertial data and the positional sensor acceleration data according to one embodiment of the invention. Thus, the operations ofFIG. 7are performed within the block630according to one embodiment of the invention. At block705, the computing device415transforms the positional sensor acceleration vectors by multiple potential angles. Flow moves from block705to block710, where the computing device415determines the difference between the transformed positional sensor acceleration vectors and the transformed inertial data vectors. Flow moves from block710to block715, where the computing device415sets the smallest difference between the vectors to be the estimated correct angle. Flow moves from block715to block720where the computing device415determines the amount of error in the angle based on the difference between the estimated correct angle and the angle derived from integrating angular rate measurements from the inertial sensor(s)420(the inertial data).

FIG. 8is a flow diagram illustrating another method of determining an amount of error between the transformed inertial data and the positional sensor acceleration data according to one embodiment of the invention. Thus, the operations ofFIG. 8are performed within the block630according to one embodiment of the invention. At block805, the computing device415determines a three-dimensional rotation that aligns the coordinate system of the transformed inertial data and the positional sensor acceleration data. Flow moves from block805to block810, where the computing device415decomposes the three-dimensional angle into the angle which is being corrected. Flow moves from block810to block815, where the computing device415sets the decomposed angle to be the estimated correct angle. Flow moves from block815to block820where the computing device415determines the amount of error in the angle based on the difference between the estimated correct angle and the angle derived from integrating angular rate measurements from the inertial sensor(s)420(the inertial data).

FIG. 9is a flow diagram illustrating yet another method of determining an amount of error between the transformed inertial data and the positional sensor acceleration data according to one embodiment of the invention. Thus, the operations ofFIG. 9are performed within the block630according to one embodiment of the invention. At block905, the computing device415projects the positional sensor acceleration data vectors and the transformed inertial data vectors onto the same two-dimensional plane (e.g., using the equations 15 and 16 described above). Flow moves from block905to block910, where the computing device415rotates the projections around the plane's normal (e.g., using one of the equations 9, 10, or 11 described above). Flow moves from block910to block915, where the computing device415determines the amount of error in the angle based on the difference between the estimated correct angle and the angle computed from integration of the inertial data.

Referring back toFIG. 6, flow moves from block630to block635. At block635, the computing device415updates the rotation estimate responsive to the determined error. Thus, the rotation estimate used to transform the received inertial data into a reference frame of the positional sensor as described in block625is updated responsive to the determined amount of error. According to one embodiment of the invention, the amount the rotation estimate is updated is a portion of the estimated error (e.g., a percentage of the determined estimated error).

According to one embodiment of the invention, the correction amount is a function of an amount of time that has passed since a previous rotation estimate update. Thus, in one embodiment of the invention, the more time elapsed since the last update the greater the amount of correction. For instance, if a user has not moved the object410(e.g., if the user has placed the controller coupled with the object410down) for a certain amount of time (e.g., 5-10 minutes), a considerable amount of drift may have occurred. In this case, the amount of correction may be a large percentage of the estimated error, or even the full percentage of the estimated error. According to one embodiment of the invention, if the user has not moved the object410for a certain amount of time, the user will be locked from using the object410in the tracking system until the user moves the object410in a motion path that gives a minimum threshold of acceleration and positional data. In one embodiment of the invention, the object410emits a certain color of light and/or sound when a user needs to move the object410to produce a minimum threshold of acceleration and positional data.

In another embodiment of the invention, the amount of correction is a function on the magnitude of the vector sets. For example, as described previously, in one embodiment of the invention, the vectors are weighted (the larger the magnitude of a vector the larger the weight). The greater the magnitude of a vector, the less the vector is influenced by noise. The larger the weight of the vectors, the larger the amount of correction. Thus, the accuracy of the estimated error increases by weighting the vectors. Of course, it should be understood that any combination of the above embodiments may be used (e.g., a percentage of the estimated error, a function of amount of time that has passed, and/or a function on the weights associated with the magnitude of the vectors).

FIG. 10illustrates hardware and user interfaces that may be used to determine controller location, in accordance with one embodiment of the present invention.FIG. 10schematically illustrates the overall system architecture of the Sony® Playstation 3® entertainment device, a console that may be compatible for implementing a three-dimensional controller locating system in accordance with one embodiment of the present invention. A system unit1400is provided, with various peripheral devices connectable to the system unit1400. The system unit1400comprises: a Cell processor1428; a Rambus® dynamic random access memory (XDRAM) unit1426; a Reality Synthesizer graphics unit1430with a dedicated video random access memory (VRAM) unit1432; and an I/O bridge1434. The system unit1400also comprises a Blu Ray® Disk BD-ROM® optical disk reader1440for reading from a disk1440aand a removable slot-in hard disk drive (HDD)1436, accessible through the I/O bridge1434. Optionally the system unit1400also comprises a memory card reader1438for reading compact flash memory cards, Memory Stick® memory cards and the like, which is similarly accessible through the I/O bridge1434.

The I/O bridge1434also connects to multiple Universal Serial Bus (USB) 2.0 ports1424; a gigabit Ethernet port1422; an IEEE 802.11b/g wireless network (Wi-Fi) port1420; and a Bluetooth® wireless link port1418capable of supporting of up to seven Bluetooth connections.

In operation, the I/O bridge1434handles all wireless, USB and Ethernet data, including data from one or more game controllers1402-1403. For example when a user is playing a game, the I/O bridge1434receives data from the game controller1402-1403via a Bluetooth link and directs it to the Cell processor1428, which updates the current state of the game accordingly.

The wireless, USB and Ethernet ports also provide connectivity for other peripheral devices in addition to game controllers1402-1403, such as: a remote control1404; a keyboard1406; a mouse1408; a portable entertainment device1410such as a Sony Playstation Portable® entertainment device; a video camera such as an EyeToy® video camera1412; a microphone headset1414; and a microphone1415. Such peripheral devices may therefore in principle be connected to the system unit1400wirelessly; for example the portable entertainment device1410may communicate via a Wi-Fi ad-hoc connection, whilst the microphone headset1414may communicate via a Bluetooth link.

The provision of these interfaces means that the Playstation 3 device is also potentially compatible with other peripheral devices such as digital video recorders (DVRs), set-top boxes, digital cameras, portable media players, Voice over IP telephones, mobile telephones, printers and scanners.

In addition, a legacy memory card reader1416may be connected to the system unit via a USB port1424, enabling the reading of memory cards1448of the kind used by the Playstation® or Playstation 2® devices.

The game controllers1402-1403are operable to communicate wirelessly with the system unit1400via the Bluetooth link, or to be connected to a USB port, thereby also providing power by which to charge the battery of the game controllers1402-1403. Game controllers1402-1403can also include memory, a processor, a memory card reader, permanent memory such as flash memory, light emitters such as LEDs or infrared lights, microphone and speaker for ultrasound communications, an acoustic chamber, a digital camera, an internal clock, a recognizable shape such as a spherical section facing the game console, and wireless communications using protocols such as Bluetooth®, WiFi™, etc.

Game controller1402is a controller designed to be used with two hands, and game controller1403is a single-hand controller with a ball attachment, as previously described inFIGS. 1A-4A. In addition to one or more analog joysticks and conventional control buttons, the game controller is susceptible to three-dimensional location determination. Consequently gestures and movements by the user of the game controller may be translated as inputs to a game in addition to or instead of conventional button or joystick commands. Optionally, other wirelessly enabled peripheral devices such as the Playstation™ Portable device may be used as a controller. In the case of the Playstation™ Portable device, additional game or control information (for example, control instructions or number of lives) may be provided on the screen of the device. Other alternative or supplementary control devices may also be used, such as a dance mat (not shown), a light gun (not shown), a steering wheel and pedals (not shown) or bespoke controllers, such as a single or several large buttons for a rapid-response quiz game (also not shown).

The remote control1404is also operable to communicate wirelessly with the system unit1400via a Bluetooth link. The remote control1404comprises controls suitable for the operation of the Blu Ray™ Disk BD-ROM reader1440and for the navigation of disk content.

The Blu Ray™ Disk BD-ROM reader1440is operable to read CD-ROMs compatible with the Playstation and PlayStation 2 devices, in addition to conventional pre-recorded and recordable CDs, and so-called Super Audio CDs. The reader1440is also operable to read DVD-ROMs compatible with the Playstation 2 and PlayStation 3 devices, in addition to conventional pre-recorded and recordable DVDs. The reader1440is further operable to read BD-ROMs compatible with the Playstation 3 device, as well as conventional pre-recorded and recordable Blu-Ray Disks.

The system unit1400is operable to supply audio and video, either generated or decoded by the Playstation 3 device via the Reality Synthesizer graphics unit1430, through audio and video connectors to a display and sound output device1442such as a monitor or television set having a display1444and one or more loudspeakers1446. The audio connectors1450may include conventional analogue and digital outputs whilst the video connectors1452may variously include component video, S-video, composite video and one or more High Definition Multimedia Interface (HDMI) outputs. Consequently, video output may be in formats such as PAL or NTSC, or in 720p, 1080i or 1080p high definition.

Audio processing (generation, decoding and so on) is performed by the Cell processor1428. The Playstation 3 device's operating system supports Dolby® 5.1 surround sound, Dolby® Theatre Surround (DTS), and the decoding of 7.1 surround sound from Blu-Ray® disks.

In the present embodiment, the video camera1412comprises a single charge coupled device (CCD), an LED indicator, and hardware-based real-time data compression and encoding apparatus so that compressed video data may be transmitted in an appropriate format such as an intra-image based MPEG (motion picture expert group) standard for decoding by the system unit1400. The camera LED indicator is arranged to illuminate in response to appropriate control data from the system unit1400, for example to signify adverse lighting conditions. Embodiments of the video camera1412may variously connect to the system unit1400via a USB, Bluetooth or Wi-Fi communication port. Embodiments of the video camera may include one or more associated microphones and also be capable of transmitting audio data. In embodiments of the video camera, the CCD may have a resolution suitable for high-definition video capture. In use, images captured by the video camera may for example be incorporated within a game or interpreted as game control inputs. In another embodiment the camera is an infrared camera suitable for detecting infrared light.

In general, in order for successful data communication to occur with a peripheral device such as a video camera or remote control via one of the communication ports of the system unit1400, an appropriate piece of software such as a device driver should be provided. Device driver technology is well-known and will not be described in detail here, except to say that the skilled man will be aware that a device driver or similar software interface may be required in the present embodiment described.

FIG. 11illustrates additional hardware that may be used to process instructions, in accordance with one embodiment of the present invention. Cell processor1428ofFIG. 10has an architecture comprising four basic components: external input and output structures comprising a memory controller1560and a dual bus interface controller1570A, B; a main processor referred to as the Power Processing Element1550; eight co-processors referred to as Synergistic Processing Elements (SPEs)1510A-H; and a circular data bus connecting the above components referred to as the Element Interconnect Bus1580. The total floating point performance of the Cell processor is 218 GFLOPS, compared with the 6.2 GFLOPs of the Playstation 2 device's Emotion Engine.

The Power Processing Element (PPE)1550is based upon a two-way simultaneous multithreading Power1470compliant PowerPC core (PPU)1555running with an internal clock of 3.2 GHz. It comprises a 512 kB level 2 (L2) cache and a 32 kB level 1 (L1) cache. The PPE1550is capable of eight single position operations per clock cycle, translating to 25.6 GFLOPs at 3.2 GHz. The primary role of the PPE1550is to act as a controller for the Synergistic Processing Elements1510A-H, which handle most of the computational workload. In operation the PPE1550maintains a job queue, scheduling jobs for the Synergistic Processing Elements1510A-H and monitoring their progress. Consequently each Synergistic Processing Element1510A-H runs a kernel whose role is to fetch a job, execute it and synchronized with the PPE1550.

Each Synergistic Processing Element (SPE)1510A-H comprises a respective Synergistic Processing Unit (SPU)1520A-H, and a respective Memory Flow Controller (MFC)1540A-H comprising in turn a respective Dynamic Memory Access Controller (DMAC)1542A-H, a respective Memory Management Unit (MMU)1544A-H and a bus interface (not shown). Each SPU1520A-H is a RISC processor clocked at 3.2 GHz and comprising 256 kB local RAM1530A-H, expandable in principle to 4 GB. Each SPE gives a theoretical 25.6 GFLOPS of single precision performance. An SPU can operate on 4 single precision floating point members, 4 32-bit numbers, 8 16-bit integers, or 16 8-bit integers in a single clock cycle. In the same clock cycle it can also perform a memory operation. The SPU1520A-H does not directly access the system memory XDRAM1426; the 64-bit addresses formed by the SPU1520A-H are passed to the MFC1540A-H which instructs its DMA controller1542A-H to access memory via the Element Interconnect Bus1580and the memory controller1560.

The Element Interconnect Bus (EIB)1580is a logically circular communication bus internal to the Cell processor1428which connects the above processor elements, namely the PPE1550, the memory controller1560, the dual bus interface1570A,B and the 8 SPEs1510A-H, totaling 12 participants. Participants can simultaneously read and write to the bus at a rate of 8 bytes per clock cycle. As noted previously, each SPE1510A-H comprises a DMAC1542A-H for scheduling longer read or write sequences. The EIB comprises four channels, two each in clockwise and anti-clockwise directions. Consequently for twelve participants, the longest step-wise data-flow between any two participants is six steps in the appropriate direction. The theoretical peak instantaneous EIB bandwidth for 12 slots is therefore 96 B per clock, in the event of full utilization through arbitration between participants. This equates to a theoretical peak bandwidth of 307.2 GB/s (gigabytes per second) at a clock rate of 3.2 GHz.

The memory controller1560comprises an XDRAM interface1562, developed by Rambus Incorporated. The memory controller interfaces with the Rambus XDRAM1426with a theoretical peak bandwidth of 25.6 GB/s.

The dual bus interface1570A,B comprises a Rambus FlexIO® system interface1572A,B. The interface is organized into 12 channels each being 8 bits wide, with five paths being inbound and seven outbound. This provides a theoretical peak bandwidth of 62.4 GB/s (36.4 GB/s outbound, 26 GB/s inbound) between the Cell processor and the I/O Bridge700via controller170A and the Reality Simulator graphics unit200via controller170B.

Data sent by the Cell processor1428to the Reality Simulator graphics unit1430will typically comprise display lists, being a sequence of commands to draw vertices, apply textures to polygons, specify lighting conditions, and so on.

While embodiments of the invention have been described with reference to correcting a single angle error (e.g., yaw, pitch, or roll), it should be understood that in other embodiments of the invention multiple angle errors may be corrected simultaneously.