ESTIMATING THE GRAVITY VECTOR IN A WORLD COORDINATE SYSTEM USING AN ACCELEROMETER IN A MOBILE DEVICE

An accelerometer located within a mobile device is used to estimate a gravity vector on a target plane in a world coordinate system. The accelerometer makes multiple measurements, each measurement being taken when the mobile device is held stationary on the target plane and a surface of the mobile device faces and is in contact with a planar portion of the target plane. An average of the measurements is calculated. A rotational transformation between an accelerometer coordinate system and a mobile device's coordinate system is retrieved from a memory in the mobile device, where the mobile device's coordinate system is aligned with the surface of the mobile device. The rotational transformation is applied to the averaged measurements to obtain an estimated gravity vector in a world coordinate system defined by the target plane.

DETAILED DESCRIPTION

The word “exemplary” or “example” is used herein to mean “serving as an example, instance, or illustration.” Any aspect or embodiment described herein as “exemplary” or as an “example” in not necessarily to be construed as preferred or advantageous over other aspects or embodiments.

Embodiments of the invention provide a method for estimating the gravity vector with respect to a target plane using an accelerometer in a mobile device, such as a mobile phone. For AR applications, the target plane is a feature plane on which the AR target is going to be displayed. The target plane can have any orientation; for example, the target plane can be aligned with the horizontal axis, aligned with the vertical axis, or tilted with respect to the horizontal or vertical axis. This target plane defines a world coordinate system for AR applications. The mobile device can serve as a convenient tool for estimating or measuring the gravity vector in the world coordinate system. It is appreciated that the estimation technique described herein is not limited to AR applications; it is applicable to a wide range of applications where the target plane may be a plane relative to which the gravity vector is unknown and needs to be measured.

As used herein, the term “world coordinate system,” “tracking coordinate system” or “target coordinate system” refers to a coordinate system having a 2-D coordinate plane defined by the target plane. That is, the x-y (or x-z or y-z) coordinate plane of the world coordinate system is parallel to the target plane. The term “accelerometer coordinate system” refers to the coordinate system of the accelerometer within the mobile device. The term “device coordinate system” or “surface coordinate system” refers to a coordinate system having a 2-D coordinate plane defined by a surface of the mobile device. In one embodiment, a mobile device may have only one device coordinate system defined by one surface (either front or back surface) of the mobile device. This surface is referred to as a “coordinate surface” of the mobile device. For a mobile device having its front surface parallel to its back surface, both front and back surfaces of the mobile device can be the coordinate surface. If the front surface of a mobile device is not parallel to the back surface, only one surface (either front or back) of the mobile device that defines the device coordinate system is the coordinate surface. In another embodiment, a mobile device may have two device coordinate systems; one defined by the front surface and the other one defined by the back surface. The mobile device may select either the front surface or the back surface as the coordinate surface. In one embodiment, the mobile device can be configured to retrieve one of two rotational transformations from the memory for alignment with the front or back surface, where the two rotational transformations include a first transformation between the coordinate system of the accelerometer and the front surface of the mobile device, and a second transformation between the coordinate system of the accelerometer and the back surface of the mobile device.

In one embodiment, a user places the coordinate surface of the mobile device firmly on the target plane such that the coordinate surface is parallel to the target plane. After the mobile device is firmly placed and motionless, the accelerometer within the mobile device makes one or more measurements. Each measurement is a measured gravity vector in the accelerometer coordinate system. If the rotational transformation between the device coordinate system and the accelerometer coordinate system is known, a conversion engine in the mobile device can convert the gravity measurements from the accelerometer coordinate system to the device coordinate system. As the coordinate surface of the mobile device is parallel to the target plane, the gravity measurements in the device coordinate system are the same as those in the world coordinate system. The gravity measurements can be averaged over a time window to obtain an accurate estimate of the gravity vector relative to the target plane in the world coordinate system.

The gravity vector measured in the world coordinate system can be used for AR, SLAM and various other applications. In one embodiment, the accelerometer is factory-calibrated such that the coordinate system of the accelerometer aligns with the surface of the mobile device. Other calibration techniques, such as user-performed calibration, can also be used. The alignment result is a rotational transformation, which can be stored in the memory of the mobile device. Thus, a mobile device can be aligned (calibrated) once and the alignment result can be used in the subsequent measurements.

FIG. 1is a block diagram illustrating a system in which embodiments of the invention may be practiced. The system may be a mobile device100, which may include a processor110, a memory120, an interface160and one or more sensors such as an accelerometer130and a gyroscope140. In one embodiment, the mobile device100may include both the accelerometer130and the gyroscope140; in an alternative embodiment, the mobile device100may include only the accelerometer130. It should be appreciated that the mobile device100may also include a display device, a user interface (e.g., keyboard, touch-screen, etc.), a power device (e.g., a battery), as well as other components typically associated with a mobile communication device. For example, the interface160may be a wireless transceiver to transmit and receive wireless signals through a wireless link to/from a wireless network, or may be wired interface for direct connection to networks (e.g., the Internet). Thus, the mobile device100may be a mobile phone (e.g., cell phone, smart phone, etc.), personal digital assistant, mobile computer, tablet, personal computer, laptop computer, e-reader, or any type of mobile device that has motion sensing and/or rotation sensing capabilities.

In one embodiment, the processor110may include a conversion engine115, which may be implemented in hardware, firmware, software, or a combination of any of the above. In one embodiment, the processor110may be a general-purpose processor or a special-purpose processor configured to execute instructions for performing the operations of conversion engine115that retrieves a stored rotational transformation from the memory120, where the rotational transformation transforms an acceleration measurement in the accelerometer coordinate system into a corresponding vector in the device coordinate system. The conversion engine115may apply the rotational transformation to accelerometer measurements to compute the gravity vector in the world coordinate system in a process to be described below.

The memory120may be coupled to the processor110to store instructions for execution by the processor110. The memory120may store a device profile121, which includes the rotational transformation between the accelerometer coordinate system and the device coordinate system. In an embodiment where the mobile device100may choose one of its surfaces as the coordinate surface, the memory120may store more than one rotational transformation; e.g., one rotational transformation for the front surface and another rotational transformation for the back surface. According to embodiments of the invention, the device profile121of the mobile device may store the rotational transformation with other sensor calibration parameters such as the measuring scale, crosstalk of sensors and the alignment between the sensors and the camera, sensor biases, if any, on the mobile device.

It should be appreciated that embodiments of the invention as will be hereinafter described may be implemented in conjunction with the execution of instructions by the processor110of the mobile device100and/or other circuitry of the mobile device100and/or other devices. Particularly, circuitry of the mobile device100, including but not limited to the processor110, may operate under the control of a program, routine, or the execution of instructions to execute methods or processes in accordance with embodiments of the invention. For example, such a program may be implemented in firmware or software (e.g. stored in the memory120and/or other locations) and may be implemented by processors, such as the processor110, and/or other circuitry of the mobile device100. Further, it should be appreciated that the terms processor, microprocessor, circuitry, controller, etc., refer to any type of logic or circuitry capable of executing logic, commands, instructions, software, firmware, functionality and the like.

FIG. 2Aillustrates a side profile of the mobile device100according to one embodiment. The mobile device100, as viewed inFIG. 2A, has a top surface and a bottom surface, where the bottom surface may be either the front or back side of the mobile device100.FIG. 2Ashows that a device coordinate system182having an x-y coordinate plane parallel to the bottom surface. As such, the bottom surface is a coordinate surface170for the mobile device100, which is the surface to which the device coordinate system182aligns. This coordinate surface170is to be placed on a target plane180that defines the world coordinate system.

FIG. 2Billustrates a side profile of the mobile device100according to another embodiment. The mobile device100, as viewed inFIG. 2B, has a concaved bottom surface, where the concaved bottom surface may be either the front or back side of the mobile device100. The convex hull of the concaved bottom surface defines a planar surface175.FIG. 2Bshows that the x-y coordinate plane of the device coordinate system182is parallel to the planar surface175. As such, the planar surface175is the coordinate surface for the mobile device100, which is the surface to which the device coordinate system182aligns. This coordinate surface is to be placed on the target plane180that defines the world coordinate system.

In an embodiment where the mobile device100can select either its front surface or the back surface as the coordinate surface, the selection is based on which surface of the mobile device100is placed and in contact with the target plane180. The device coordinate system182is defined based on the selection. Once the selection is made, the mobile device100may retrieve the corresponding rotational transformation for the device coordinate system182.

The accelerometer130measures the gravity vector when it is stationary. The accelerometer coordinate system183is the coordinate system in which all of the accelerometer measurements lie. The accelerometer coordinate system183is not necessarily aligned with the device coordinate system182. The accelerometer coordinate system183and the device coordinate system182are misaligned when the x-y plane of the surface coordinate system182is not aligned with the ax-ay plane of the accelerometer coordinate system183. The misalignment described herein is the rotational misalignment. This rotational misalignment can be calibrated by the factory manufacturing the mobile device100, or by a user who performs a user calibration process. The calibration result is the rotational transformation stored in the memory120ofFIG. 1. In one embodiment, the rotational transformation is in the form of a rotational matrix. The rotational transformation transforms an accelerometer measurement from the accelerometer coordinate system183into the device coordinate system182. If the accelerometer coordinate system183is aligned with the device coordinate system182, the rotational matrix is an identify matrix and no transformation is necessary.

The gravity vector g, which equals 9.81 m/s2measured at sea level, points straight down to the earth center. The gravity vector g may be represented by [0, 0, 9.81] in a coordinate system having a z axis pointing straight down to the earth center. The gravity vector measured by the accelerometer130is g′, which has the same vector length (9.81 m/s2measured at sea level) as the gravity vector g, but may be a rotated version of g due to the orientation of the accelerometer130and calibration errors. For example, the measured gravity vector g′ may be [5, 2.69, 8] in the accelerometer coordinate system183. If the accelerometer coordinate system183and the device coordinate system182are aligned, the gravity vector in the device coordinate system182will also be g′. However, when the accelerometer coordinate system183and the device coordinate system182are misaligned, the gravity vector g″ in the device coordinate system182is further rotated from g′ by an angle in the 3-D Euclidian frame. The rotational transformation stored in the mobile device100is a transformation from g′ to g″. A bias exists in the gravity measurement if the length of the measured gravity vector is greater than 9.81. Techniques for removing biases from gravity measurements include the extended Kalman filtering, which produces accurate results when the amount of bias is small (e.g., within +/−3 degrees as sines and cosines can be approximated by straight lines in that region). Alternative filtering techniques may also be used.

FIG. 3A-3Dillustrate examples of target planes210-240with different orientations, such as tilted with respect to the horizontal axis (FIG. 3AandFIG. 3B), vertical (FIG. 3C) and horizontal (FIG. 3D) orientations. Each target plane210-240has a planar surface facing the mobile device100, or at least a portion of the surface facing the mobile device100is planar. To estimate the gravity vector relative to the target plane210-240, a user may hold the mobile device100firmly on the planar surface of the target plane without motion. The accelerometer130can then take measurements while the mobile device100is stationary and motionless. The surface of the mobile device100facing the target plane210-240is its coordinate surface (e.g., the surface170or175ofFIG. 2AandFIG. 2B); that is, the mobile device's surface that is aligned with the device coordinate system182. This coordinate surface (170or175) is held parallel to the planar surface of the target plane210-240when the accelerometer130takes measurements.

FIG. 4illustrates an embodiment of a method400for estimating the gravity vector. In one embodiment, the method400is performed by a mobile device, such as the processor110ofFIG. 1A, using the measurements of the accelerometer130. In one embodiment, the method400may be performed by hardware, software, firmware, or a combination of any of the above.

In one embodiment, a processor of a mobile device receives a plurality of measurements from the accelerometer located within the mobile device (block401). Each of the measurements is taken when the mobile device is held stationary on the target plane and a surface of the mobile device faces and is in contact with a planar portion of the target plane. The processor calculates an average of the measurements (block402), and retrieves a rotational transformation between an accelerometer coordinate system and a device coordinate system from a memory in the mobile device (block403), wherein the device coordinate system is aligned with the surface of the mobile device. Accelerometer bias, if any, is removed from the average of the measurements. The rotational transformation is applied to the averaged measurements (with the bias removed) to obtain an estimated gravity vector in a world coordinate system defined by the target plane (block404).

In one embodiment, the mobile device100may run an application that prompts the user to place the device on a target plane in order to start estimation of the gravity vector relative to the target plane. The application may include instructions to direct the accelerometer130to take multiple measurements upon receiving a trigger from a user, and/or when the accelerometer130senses that the device is stationary. After applying the rotational transformation to the measurements according to the method400, the application may use the estimated gravity vector in the world coordinate system to compute additional parameters for AR, SLAM applications or other purposes, such as accelerator (and gyroscope) assisted AR, map building in SLAM, dynamic objects handling in SLAM, aligning SLAM maps to a horizontal coordinate system, and the like.

For example, when a user wants to localize the mobile device100relative to the world (i.e., a target plane), one additional parameter that the mobile device100may compute is the position vector of the mobile device100in the world coordinate system. The position vector indicates how far the mobile device100is from the origin of the world coordinate system and which direction the mobile device100is moving. The position vector can be computed using the accelerometer130while the mobile device100is moving. The acceleration of the mobile device100in the world coordinate system can be extracted from the accelerometer measurements by subtracting the gravity vector relative to the target plane from the gravity vector measured by the accelerometer130. The velocity of the mobile device100can be obtained by integrating the acceleration over time, and the position vector of the mobile device100can be obtained by integrating the velocity over time. The orientation of the mobile device100can be measured by the gyroscope140within the mobile device100, which tracks angular rotation of the mobile device100.

It should be appreciated that when the mobile device100described above is a wireless mobile device, it may communicate via one or more wireless communication links through a wireless network that are based on or otherwise support any suitable wireless communication technology. For example, in some aspects computing device or server may associate with a network including a wireless network. In some aspects the network may comprise a body area network or a personal area network (e.g., an ultra-wideband network). In some aspects the network may comprise a local area network or a wide area network. A wireless device may support or otherwise use one or more of a variety of wireless communication technologies, protocols, or standards such as, for example, CDMA, TDMA, OFDM, OFDMA, WiMAX, and Wi-Fi. Similarly, a wireless device may support or otherwise use one or more of a variety of corresponding modulation or multiplexing schemes. A wireless device may thus include appropriate components (e.g., air interfaces) to establish and communicate via one or more wireless communication links using the above or other wireless communication technologies. For example, a device may comprise a wireless transceiver with associated transmitter and receiver components (e.g., a transmitter and a receiver) that may include various components (e.g., signal generators and signal processors) that facilitate communication over a wireless medium. As is well known, a mobile wireless device may therefore wirelessly communicate with other mobile devices, cell phones, other wired and wireless computers, Internet web-sites, etc.

The techniques described herein can be used for various wireless communication systems such as Code Division Multiple Access (CDMA), Time division multiple access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency-Division Multiple Access (OFDMA), Single Carrier FDMA (SC-FDMA) and other systems. The terms “system” and “network” are often used interchangeably. A CDMA system can implement a radio technology such as Universal Terrestrial Radio Access (UTRA), CDMA2000, etc. UTRA includes Wideband-CDMA (W-CDMA) and other variants of CDMA. CDMA2000 covers Interim Standard (IS)-2000, IS-95 and IS-856 standards. A TDMA system can implement a radio technology such as Global System for Mobile Communications (GSM). An OFDMA system can implement a radio technology such as Evolved Universal Terrestrial Radio Access; (Evolved UTRA or E-UTRA), Ultra Mobile Broadband (UMB), Institute of Electrical and Electronics Engineers (IEEE) 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, Flash-OFDM®, etc. Universal Terrestrial Radio Access (UTRA) and E-UTRA are part of Universal Mobile Telecommunication System (UMTS). 3GPP Long Term Evolution (LTE) is an upcoming release of UMTS that uses E-UTRA, which employs OFDMA on the downlink and SC-FDMA on the uplink. UTRA, E-UTRA, UMTS, LTE and GSM are described in documents from an organization named “3rd Generation Partnership Project” (3GPP). CDMA2000 and UMB are described in documents from an organization named “3rd Generation Partnership Project 2” (3GPP2).

The techniques described herein may be incorporated into (e.g., implemented within or performed by) a variety of mobile apparatuses (e.g., devices). For example, one or more aspects taught herein may be incorporated into a phone (e.g., a cellular phone), a personal data assistant (“PDA”), a tablet, a mobile computer, a laptop computer, a tablet, an entertainment device (e.g., a music or video device), a headset (e.g., headphones, an earpiece, etc.), a medical device (e.g., a biometric sensor, a heart rate monitor, a pedometer, an EKG device, etc.), a user I/O device, a point-of-sale device, an entertainment device, or any other suitable device. These devices may have different power and data requirements