Patent ID: 12250458

DETAILED DESCRIPTION OF THE EMBODIMENTS

The example techniques described in this disclosure relate to image stabilization and visual-inertial odometry (VIO). VIO is a technique to estimate state (e.g., position and orientation (pose) and velocity) of a device by using images captured by a camera of the device and information indicative of the angular velocity and/or movement of the device (e.g., based on output from inertial measurement unit (IMU) and/or accelerometer). The movement of the device may cause high-speed motion jitter and random noise that results in the image used for VIO to be blurry. Because the image used for VIO is blurry, there may be error in the localization of device. The error in the localization of the device may refer to error or even failure in the determination of the pose and velocity of the device and error or even failure in the determination of the location and position of other objects in the image.

In VIO techniques, based on the output of the IMU for two consecutive frames of image data, it may be possible for the device to determine the pose of the device. Also, the device may be able to estimate the pixel coordinates of feature points in a next frame. In this way, the device performs feature tracking, which may be useful for fast motion. Based on the feature tracking, the device may determine its current pose and location of objects in the frame for mapping the area surrounding the device. As one example, the gravity vector from the IMU (e.g., accelerometer of the IMU) can be used to convert the estimated position of the device into a world frame for navigation.

However, there may be errors relying on the output from the IMU for VIO techniques if there are blurry images. For instance, a loose connection in the IMU may result in vibration. Also, tracking of feature points may be not be reliable in blurry images.

Image stabilization is an example way in which to reduce the blurriness of an image, and is generally a separate operation from the VIO. For instance, image stabilization (IS) reduces the blurring effect caused by the motion of the device or image capturing during exposure. There is optical image stabilization (OIS) and electronic image stabilization (EIS). Optical image stabilization moves the camera in opposite direction from movement of camera device, and is a hardware-based solution. Electronic image stabilization provides software-based solution to compensate for movement of the camera. For example, the EIS unit may determine intentional and unintentional movements, and then adjust each frame with image warping. EIS may be implemented with relative ease as compared to OIS because EIS is software driven and may not be impacted from hardware failure like OIS techniques can. However, there may be loss of image quality at boundaries using EIS techniques.

Although VIO and IS techniques are separate techniques, this disclosure describes example ways in which to integrate the VIO and IS techniques. For example, rather than utilizing two separate units for VIO and IS, the example techniques iteratively update intermediate results used for both VIO and IS, resulting in efficient resource utilization. Furthermore, because the VIO and IS techniques are integrated together, the result from IS can be used to improve VIO, and vice-versa, forming a closed-loop design that results in better images and more accurate pose estimation for determination of localization (e.g., determination of where the device is relative to other objects in the image).

As described in more detail below, an integrated IS/VIO unit may be configured to utilize a tracking filter that receives as input the angular velocity and movement of the device determined from the IMU across a current frame and a subsequent frame and is configured to generate output indicative of angular velocity and movement of the device with unintentional movement of the device removed. The tracking filter may also utilize the predicted state (e.g., pose information) previously determined by a VIO unit of the IS/VIO unit to generate the output indicative of angular velocity and movement of the device with unintentional movement of the device removed.

An IS unit of the IS/VIO unit may utilize the output from the tracking filter to perform image stabilization on the current frame. The IS unit may output the image stabilized current frame to the VIO unit, and the VIO unit utilizes the image stabilized current frame to generate the predicted state. In this way, the IS unit and the VIO unit of the integrated IS/VIO unit form a closed-loop feedback design to iteratively update image stabilization and determination of pose.

As described in more detail, there may be a one-frame latency for image stabilization and determination of pose and velocity. For example, for image stabilization of a current frame, the IS unit may utilize a subsequent frame. Hence, in this disclosure, the current frame (e.g., frame n) is the frame for which image stabilization is being performed, but a subsequent frame (e.g., frame n+1) has been captured for image stabilization of the current frame. However, for the determination of pose and velocity, that determination may be for the subsequent frame (e.g., frame n+1). Accordingly, the image stabilization may be for the current frame and the pose and velocity information may be for the subsequent frame.

For instance, after the current frame, the subsequent frame becomes the current frame. Since the subsequent frame was utilized for image stabilization of the current frame, when the subsequent frame becomes the current frame, various intermediate values that were calculated are already stored within a priority queue of the integrated IS/VIO unit (e.g., predicted state and pose information for the subsequent state). Accordingly, rather than re-calculating various values, the already stored values in the priority queue can be utilized for image stabilization and pose information when the subsequent frame becomes the current frame, in addition to pose information for the next frame (e.g., frame after the current frame, where the previous subsequent frame is now the current frame).

FIG.1is a block diagram of a device configured to perform one or more of the example techniques described in this disclosure. Examples of camera device10include stand-alone digital cameras or digital video camcorders, camera-equipped wireless communication device handsets, such as mobile telephones having one or more cameras, cellular or satellite radio telephones, camera-equipped personal digital assistants (PDAs), panels or tablets, gaming devices, computer devices that include cameras, such as so-called “web-cams,” or any device with digital imaging or video capabilities.

As illustrated in the example ofFIG.1, camera device10includes camera12(e.g., having an image sensor and lens), camera processor14and local memory20of camera processor14, a central processing unit (CPU)16, a graphical processing unit (GPU)18, user interface22, memory controller24that provides access to system memory30, and display interface26that outputs signals that cause graphical data to be displayed on display28. Although the example ofFIG.1illustrates camera device10including one camera12, in some examples, camera device10may include a plurality of cameras, such as for omnidirectional image or video capture.

Also, although the various components are illustrated as separate components, in some examples the components may be combined to form a system on chip (SoC). As an example, camera processor14, CPU16, GPU18, and display interface26may be formed on a common integrated circuit (IC) chip. In some examples, one or more of camera processor14, CPU16, GPU18, and display interface26may be in separate IC chips. Various other permutations and combinations are possible, and the techniques should not be considered limited to the example illustrated inFIG.1.

The various components illustrated inFIG.1(whether formed on one device or different devices) may be formed as at least one of fixed-function or programmable circuitry such as in one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), or other equivalent integrated or discrete logic circuitry. Examples of local memory20and system memory30include one or more volatile or non-volatile memories or storage devices, such as random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, a magnetic data media or an optical storage media.

The various units illustrated inFIG.1communicate with each other using bus32. Bus32may be any of a variety of bus structures, such as a third generation bus (e.g., a HyperTransport bus or an InfiniBand bus), a second generation bus (e.g., an Advanced Graphics Port bus, a Peripheral Component Interconnect (PCI) Express bus, or an Advanced eXtensible Interface (AXI) bus) or another type of bus or device interconnect. The specific configuration of buses and communication interfaces between the different components shown inFIG.1is merely exemplary, and other configurations of camera devices and/or other image processing systems with the same or different components may be used to implement the techniques of this disclosure.

Camera processor14is configured to receive image frames from camera12, and process the image frames to generate output frames for display. CPU16, GPU18, camera processor14, or some other circuitry may be configured to process the output frame that includes image content generated by camera processor14into images for display on display28. In some examples, GPU18may be further configured to render graphics content on display28.

In some examples, camera processor14may be configured as an image processing pipeline (sometimes called an image signal processors (ISP)). For instance, camera processor14may include a camera interface that interfaces between camera12and camera processor14. Camera processor14may include additional circuitry to process the image content. Although one camera processor14is shown with one camera12, in some examples, device10may include a plurality of cameras. Camera processor14may be a common camera processor for each of the cameras. In some examples, there may be a plurality of camera processors for one or more of the plurality of cameras.

Camera processor14outputs the resulting frames with image content (e.g., pixel values for each of the image pixels) to system memory30via memory controller24. As described in more detail, CPU16may utilize the frames to correct for unintentional movement of camera device10during capturing of the frames. For instance, CPU16includes image stabilization (IS) and visual-inertial odometry (VIO) unit36(IS/VIO unit36). IS/VIO unit36may utilize the frames outputted by camera processor14and the output from inertial measurement unit (IMU)34to perform image stabilization and determine pose and localization information (e.g., state) of camera device10.

This disclosure describes the examples techniques as being performed by CPU16(e.g., via IS/VIO unit36). However, the example techniques should not be considered limited to CPU16performing the example techniques. Moreover, IS/VIO unit36may be considered as software executing on the hardware of CPU16. Accordingly, system memory30may store computer-readable instructions (e.g., instructions for IS/VIO unit36) that cause one or more processors (e.g., CPU16) to perform the example techniques described in this disclosure. The example techniques described in this disclosure are not limited to being performed by software executing on hardware. In some examples, IS/VIO unit36may be fixed-function circuitry configured to perform the example techniques described in this disclosure.

CPU16may comprise a general-purpose or a special-purpose processor that controls operation of camera device10. A user may provide input to camera device10to cause CPU16to execute one or more software applications. The software applications that execute on CPU16may include, for example, a media player application, a video game application, a graphical user interface application or another program (e.g., including IS/VIO unit36). The user may provide input to camera device via one or more input devices (not shown) such as a keyboard, a mouse, a microphone, a touch pad or another input device that is coupled to camera device10via user interface22.

One example of the software application is a camera application. CPU16executes the camera application, and in response, the camera application causes CPU16to generate content that display28outputs. GPU18may be configured to process the content generated by CPU16for rendering on display28. For instance, display28may output information such as light intensity, whether flash is enabled, and other such information. The user of camera device10may interface with display28to configure the manner in which the images are generated (e.g., with or without flash, focus settings, exposure settings, and other parameters). The camera application also causes CPU16to instruct camera processor14to capture and process the frames of image content captured by camera12in the user-defined manner.

Memory controller24facilitates the transfer of data going into and out of system memory30. For example, memory controller24may receive memory read and write commands, and service such commands with respect to memory30in order to provide memory services for the components in camera device10. Memory controller24is communicatively coupled to system memory30. Although memory controller24is illustrated in the example of camera device10ofFIG.1as being a processing circuit that is separate from both CPU16and system memory30, in other examples, some or all of the functionality of memory controller24may be implemented on one or both of CPU16and system memory30.

System memory30may store program modules and/or instructions and/or data that are accessible by camera processor14, CPU16, and GPU18. For example, system memory30may store user applications (e.g., instructions for the camera application), resulting frames from camera processor14, etc. As another example, as described above, system memory30may store the instructions for IS/VIO unit36so that IS/VIO unit36executes on CPU16. System memory30may additionally store information for use by and/or generated by other components of camera device10. For example, system memory30may act as a device memory for camera processor14.

Camera processor14, CPU16, and GPU18may store image data, and the like in respective buffers that are allocated within system memory30. Display interface26may retrieve the data from system memory30and configure display28to display the image represented by the generated image data. For example, display28may output the output frame generated by camera processor14. In some examples, display interface26may include a digital-to-analog converter (DAC) that is configured to convert the digital values retrieved from system memory30into an analog signal consumable by display28to drive elements of the displays. In other examples, display interface26may pass the digital values directly to display28for processing.

Display28may include a monitor, a television, a projection device, a liquid crystal display (LCD), a plasma display panel, a light emitting diode (LED) array, or another type of display unit. Display28may be integrated within camera device10. For instance, display28may be a screen of a mobile telephone handset or a tablet computer. Alternatively, display28may be a stand-alone device coupled to camera device10via a wired or wireless communications link. For instance, display28may be a computer monitor or flat panel display connected to a personal computer via a cable or wireless link.

As illustrated, camera device10includes inertial measurement unit (IMU)34. In one or more examples, IMU34is a six-axis IMU. The six-axis IMU may couple a 3-axis accelerometer with a 3-axis gyroscope. Accelerometers measure linear acceleration, while gyroscopes measure rotational motion. For example, the accelerometer of IMU34may indicate the movement of camera device10in 3-axis (x, y, z coordinates) and the gyroscope of IMU34may indicate the angular velocity of camera device10in 3-axis (x, y, z coordinates).

As described above, camera device10may output, for display (e.g., via display28), frames captured by camera12. However, in some examples, due to movement of camera device10(e.g., shaking from the user holding camera device10or some other unintentional movement), the image content in the frames displayed on display28may be blurry, which negatively impacts viewer experience.

Moreover, in some examples, camera device10may be configured to generate information indicative of its position and orientation (pose) and velocity based on the frames captured by camera12. An example technique for determining the pose and velocity of camera device10, as well as a prediction of where camera device10will be is referred to as visual-inertial odometry (VIO) techniques.

For example, for virtual reality (VR)/augmented reality (AR), the pose of camera device10, as well as locations of objects captured in the frames, may be utilized to generate the VR/AR experience (e.g., localization of camera device10relative to other objects). If the pose information of camera device10is based on blurry images, the resulting localization by the VIO techniques may be erroneous, causing negative viewer experience. There may be other reasons for utilizing VIO, and VR/AR is one example.

This disclosure describes example techniques to integrate image stabilization (IS) and VIO into IS/VIO unit36to generate image stabilized frames with reduced blurriness and provide more accurate pose and velocity information. For instance, IS/VIO unit36includes an IS unit and a VIO unit, as described in more detail with respect toFIG.2. However, the IS unit and the VIO unit of IS/VIO unit36share input and outputs, so that any intermediate values that are generated can be iteratively updated in a manner which resources used for image stabilization and visual-inertial odometry can be shared. As one example, the output from IS unit of IS/VIO unit36is used as input to the VIO unit of IS/VIO unit36, and the output of the VIO unit of IS/VIO unit36is feedback input for the IS unit of IS/VIO unit36, forming a closed-loop design. Also, intermediate values generated, as part of image stabilization or visual-inertial odometry, may be stored in a priority queue in a manner in which the same data can be reused rather than being recalculated.

As one example, IS/VIO unit36may receive a first set of information, the first set of information being indicative of angular velocity and movement of camera device based on change in position of camera device10across a current frame and a subsequent frame. In this disclosure, there may be a one frame (or possibly one or more frame) latency in image stabilization. For example, to perform image stabilization for a current frame (e.g., frame n), IS/VIO unit36utilizes a subsequent frame (e.g., frame n+1), where camera device10captures the subsequent frame after capturing the current frame. Hence, the example techniques are described with respect to stabilization with respect to a current frame, but the current frame may not be the most recently captured frame. Rather, to perform stabilization a subsequent frame (e.g., a frame captured after the current frame, and in some examples, the immediately next frame) is utilized by IS/VIO unit36for stabilization.

However, for determining the pose and velocity information, IS/VIO unit36may determine such pose and velocity information for camera device10for the subsequent frame (e.g., frame n+1). Accordingly, the image stabilization may be for frame n, but the pose and velocity information may be determined from frame n+1.

For example, IMU34may output angular velocity (w) and movement information (e.g., acceleration represented by “a”). The angular velocity may be the angular velocity of camera device10between when camera device10captured a current frame and a subsequent frame. The movement information may be the movement of camera device10between when camera device10captured the current frame and the subsequent frame. In some examples, one or more filters (e.g., random noise filters) may filter the angular velocity and movement information.

IS/VIO unit36may filter the first set of information to compensate for unintentional movement of camera device10to generate a second set of information, the second set of information being indicative of angular velocity and movement of camera device10without unintentional movement. For example, during capturing of frames for a video, the user of camera device10may intentionally move camera device (e.g., to track movement of an object, to pan across a scene, etc.). During this intentional movement of camera device10, there may be an unintentional movement component, where camera device10moves due to slight involuntary shaking by the user. In some examples, the involuntary shaking by the user may also occur between capturing the current frame and the subsequent frame even when the user is not intentionally moving camera device10.

The unintentional movement of the user may be relatively slight. For instance, arm fatigue may make it difficult for the user to hold the camera steady. Slight shifts in the position of user, such as rebalancing, shifting for comfort, etc. can also result in unintentional movement of camera device10. Such unintentional movement can result in blurry images and poor pose and localization determination as described above.

There may be various ways in which to filter the first set of information to compensate for the unintentional movement of camera device10to generate the second set of information. As one example, IS/VIO unit36may utilize extended Kalman filters (EKFs) to model motions and utilize probability information to determine whether movement was intentional or unintentional. In addition, to perform the filtering, the IS/VIO unit36may utilize previously determined position and orientation of camera device10(e.g., from the VIO unit of IS/VIO unit36) and previously generated information (e.g., generated for frames prior to the current frame) of angular velocity and movement of camera device10without unintentional movement as inputs to the EKFs. The result from the filtering may be the second set of information, where the second set of information is indicative of angular velocity and movement of camera device10without unintentional movement.

IS/VIO unit36may perform image stabilization on the current frame based on both the first set of information and the second set of information to generate an image stabilized current frame. As described above, the first set of information is indicative of angular velocity and movement of camera device10based on change in position of camera device10across the current frame and the subsequent frame. The second set of information is indicative of angular velocity and movement of camera device10without unintentional movement of camera device10. In one or more examples, for image stabilization, IS/VIO unit36utilize both the first set of information and the second set of information for adjusting the location of a region in the current frame, such that the adjusted location of the current frame compensates for the unintentional movement of camera device10.

For example, the location of a region in the current frame may be incorrect and due to the unintentional movement of camera device10. IS/VIO unit36may utilize the first set of information and the second set of information to adjust the location of the region such that in the image stabilized current frame the location of the region is correct and the blurriness cause by unintentional movement of camera device10is minimized.

IS/VIO unit36may output, for display on display28, the image stabilized current frame. Also, the VIO unit of IS/VIO unit36may determine position and orientation (pose) of camera device10based on the image stabilized current frame. IS/VIO unit36may then output information indicative of the determined position and orientation, which may be utilized by GPU18or other components for AR/VR. Because the image stabilized current frame has reduced blurriness, there may be greater accuracy in the determined position and orientation of camera device10.

FIG.2is a block diagram illustrating an example of an image stabilization and visual-inertial odometry (VIO) unit36ofFIG.1in further detail. InFIG.2, the dashed boxes illustrate the output from a unit. For example, as illustrated inFIG.2, IMU34outputs the angular velocity and movement of camera device10across a current frame and a subsequent frame. The angular velocity may be represented in three-dimensions as wx, wy, and wz, and the movement may be represented in three-dimensions as ax, ay, and az.

In one or more examples, noise filter40A may noise filter the angular velocity, and noise filter40B may filter the movement information. Noise filter40A and noise filter40B may be digital filters formed using a digital signal processor (DSP). For example, camera device10may include a DSP and noise filter40A and noise filter40B may be formed on the DSP. As another example, noise filter40A and noise filter40B may be formed within CPU16. The output from noise filter40A is w′x, w′y, and w′z, and the output from noise filter40B is a′x, a′y, and a′z.

In some examples, IMU34may suffer from random noise in the measurement, and noise filter40A and noise filter40B may be configured to filter out such random noise. The random noise that is added by IMU34tends to be low frequency below cutoff frequency or high frequency above cutoff frequency, and therefore, noise filter and noise filter40B may be averaging filters to filter out the random noise beyond bandwidth. As one example, the averaging filter for noise filter40A and noise filter can be represented as follows: yi=(1−α)yi-1+α(xi−xi-1). In this example, xiis the input and yiis the output signal at time step i, xi-1is the input at time step i−1, and α is a time constant that may be set as follows: α=τ/(τ+dt). In one or more examples, τ can be user selected value used in designing noise filter40A and noise filter40B according to slope rate of output/input.

In one or more examples, the angular velocity (e.g., w′x, w′y, and w′z) may be considered as the rotation of camera device10and the movement (e.g., a′x, a′y, and a′z) may be considered as the translation vector of camera device10across the current frame and the subsequent frame. In one or more examples, transform unit42may utilize the rotation and translation to form a unified matrix that includes the angular velocity and movement information in a formulation that eases filtering to compensate for unintentional movement of camera device10. Transform unit42may be hardware unit or software unit executing on CPU16.

As illustrated, transform unit42generates transformation matrix T which includes the rotation (R) and translation (t). Accordingly, the output from transform unit42is shown as transform T=[R, t]. In one or more examples, transform T may be in the Lie group formulation. In mathematics, a Lie group is a group, whose elements are organized continuously in a smooth way in a matrix. The matrix may be differentiable in an exponential manifold space, and has closed form. Euler angles (roll, pitch, yaw) that are indicative of angular velocity may suffer from matrix rank degeneration when “gimbal lock” problem exists. Accordingly, it may be possible to transform such angular velocity and movement into a group of n×n invertible matrices with a Lie group, where the element of the matrix is exp(tX), by using the matrix exponential, t is in vector space Rn, X is the (n×n) matrices over complex field Cm.

The following describes one example way in which transform unit42may generate transformation matrix T. During manufacturing and prior to use of camera device10, the manufacturer (or possibly some other entity but for ease is described with the manufacturer) may perform cross-calibration between IMU34and camera12. The manufacturer may face camera12towards a chessboard pattern having unit “April tag patterns” with known scale. The manufacturer may then move camera device10in the following order: roll, pitch, yaw, back and forth, left and right, and top and down. Each of these steps may be repeated several times to collect both the output from camera12and IMU34. From the collected information, it may be possible to determine the camera orientation (Rwc) and the IMU orientation (Rwi) in the world frame. The relative rotation between camera device10and IMU34may then be acquired as Rwi/Rwc. Transform unit42may utilize the Rwc for two consecutive frames (e.g., current frame and subsequent frame) to generate the transformation matrix T.

As described above, the transformation matrix T between two consecutive frames (e.g., current frame and subsequent frame) include the relative rotation matrix (e.g., w′x, w′y, and w′z), represented as R, and the translation vector (e.g., a′x, a′y, and a′z), represented as t. The transformation matrix T may be represented as:

T1⁢0=[R1⁢0t1⁢001]

In the above equation, T10 represents the transformation matrix T across a current frame and a subsequent frame. R10 represents the rotation across the current frame and the subsequent frame, and t10 represent the translation across the current frame and the subsequent frame.

For instance, R10 is a function of the gyroscope angular velocity measurements of IMU34and the bias of the gyroscope. The following relation between a current frame (frame 1) and subsequent frame (frame 0) may be:
R10=Rw1−1Rw0.

Rw0 may be acquired by the discrete integration of angular velocities in world frame until the time step moving to the subsequent frame (frame 0). Rw0 may be dependent on the gyroscope angular velocity bias as well. Each integration step is:
Rwn+1=e{circumflex over ( )}(Rwn[w+bg]dt)Rwn.

Rwn represents the integrated rotation matrix from the world frame to the coordinate frame at time n.

For the translation vector t10, t10 can be related to the world coordinate frame as follows:
t10=two−tw1.

The discrete integration at each time step is:
twn+1=twn+∫nn+1vn dt
vn=vn+1∫nn+1(Rwn[a+ba]−g)dt
The equation for t10 can be simplified to
t10=Δtv1+½Δt2g+t*10

In the above equation, t*10is the integration part of the twn+1 and vn, but with the start velocity and gravity vector set to zero. Also, the rotation matrix may be independent of the translation vector. Accordingly, the transformation matrix T that is output from transform unit42may be as follows. In the following description, the transformation matrix T may be considered equivalent to the T10 in the following equation.

T1⁢0=[I3*3Δ⁢tv1+12⁢Δ⁢t2⁢g01*31][R1⁢0t1⁢0*01*31]

In this manner, transform unit42may generate the transformation matrix T that is used for filtering out the angular velocity and movement information caused by unintentional movement of camera device10. For example, as illustrated inFIG.2, IMU data (e.g., angular velocity and movement information from IMU34) is processed by an averaging filter to remove random noise (e.g., filtered by noise filter40A and noise filter40B). Transform unit42may then integrate the filtered angular velocity (e.g., w′x, w′y, and w′z) and acceleration (e.g., a′x, a′y, and a′z) into pose transformation with the Lie Group formulation (e.g., to generate transformation matrix T). For instance, transformation matrix T is a unified matrix that includes the integrated filtered angular velocity and acceleration information. Transformation matrix T may be considered as a first set of information, the first set of information being indicative of angular velocity and movement of camera device10based on change in position of camera device across a current frame and a subsequent frame.

As illustrated inFIG.2, IS/VIO unit36may receive the first set of information (e.g., transformation matrix T). IS/VIO unit36may also receive one or more frames (e.g., current frame and subsequent frame), where IS/VIO unit36may perform image stabilization on the current frame to remove the blurriness in the current frame, and well as more accurately determine pose and localization based on the current frame. IS/VIO unit36may receive the one or more frames continuously (e.g., in preview mode) at a particular frame rate, or may receive the one or more frames when instructed (e.g., like in a snapshot).

IS/VIO unit36may include image stabilization unit44, tracking filter unit46, priority queue48, and visual-inertial odometry (VIO) unit50. In some examples, priority queue48may be part of system memory30. In some examples, priority queue48may be part of local memory of CPU16(e.g., cache memory, registers, etc. of CPU16).

Tracking filter unit46may receive the first set of information (e.g., transformation matrix T) and filter the first set of information to compensate for unintentional movement of camera device10to generate a second set of information, the second set of information begin indicative of angular velocity and movement of camera device10without unintentional movement. As illustrated, tracking filter unit46may output transform TIM, where transform TIMis a filtered state without unintentional movement of camera device10. As example manner in which tracking filter unit46may generate TIMis described in more detail with respect toFIG.3.

In general, as illustrated inFIG.2, tracking filter unit46receives as input a previously predicted state (e.g., position and orientation previously determined for a frame prior to the current frame) that was generated by VIO unit50. Tracking filter unit46also receives the transformation matrix T, which includes the current angular velocity and movement information from IMU34. Tracking filter unit46mixes previously predicted state and the transformation matrix T, and applies filtering to the result of the mixing to decompose the transformation matrix T into two parts: a decomposed first part of the transformation and a decomposed second part of the transformation. The decomposed second part of the transformation may be the unintentional movement information. The decomposed first part of the transformation may be the angular velocity and movement information with the unintentional movement removed (e.g., with the decomposed second part of the transformation removed). The decomposed first part of the transformation is referred to as transformation matrix TIM.

Priority queue48may store both the transformation matrix T (e.g., the first set of information) and transformation matrix TIM(e.g., the second set of information). Image stabilization unit44may receive both the first of information (e.g., transformation matrix T) and the second set of information (e.g., transformation matrix TIM) and perform image stabilization to generate an image stabilized current frame (illustrated as corrected frames inFIG.2). Example techniques to perform image stabilization to generate the image stabilized current frame is described below with respect toFIGS.4A and4B.

VIO unit50may be configured to receive corrected frames (e.g., image stabilized current frame) as well as current, filtered IMU measurements (e.g., w′x, w′y, w′z, a′x, a′y, and a′z). VIO unit50may utilize the received information to determine the pose (e.g., position and orientation) and velocity of camera device10. The predicted state (e.g., pose and velocity) that VIO unit50outputs is an example of the pose an velocity information that VIO unit50generates. As illustrated, the predicted state information is fed into tracking filter unit46for generation of transformation matrix TIM, which is then used for image stabilization by image stabilization unit44. In this way, image stabilization unit44and VIO unit50form a closed loop design, where values generated by one are fed back to the other.

To determine pose (e.g., position and orientation) information for camera device10, VIO unit50may perform pre-integration on the angular velocity and movement information and feature detection in the image stabilized current frame received from image stabilization unit44. In some examples, the pre-integration on the angular velocity and movement information may be pre-integration on the angular velocity and movement information without the unintentional movement. That is, VIO unit50may perform pre-integration on transformation matrix TIMAs described above, transformation matrix TIMrepresents the information indicative of angular velocity and movement of camera device10without unintentional movement. Accordingly, VIO unit50may reuse the transformation matrix TIMthat is used by image stabilization unit44but for pre-integration.

VIO unit50may estimate pose and velocity through discrete sum of angular velocity and acceleration multiplied by time interval. Double integral of acceleration over time interval is the position estimation. In this integration process, biases of the acceleration and gyroscope measurements are also updated and subtracted from the raw measurements.

VIO unit50may also detect features in image stabilized current frame and the subsequent frame. These visual features are salient points compared to neighboring pixels in visual appearance, and a descriptor for each feature point will also be extracted by statistical results of patch around feature point. These distinctive feature points across consecutive image frames (e.g., the current frame and the subsequent frame) are utilized to establish the correspondence between frames for tracking. In some examples, VIO unit50may utilize a sliding-window of frames for optimization purposes. For example, 5-10 frames, with rich feature points, may be maintained for use by VIO unit50to determine pose information. With the output from the sliding-window optimization, VIO unit50may perform localization and generate the pose information for camera device10, and also a global map of3D points surrounding camera device10captured in the subsequent frame using the image stabilized frames.

The relative pose of two frames is estimated firstly by pre-integration of IMU measurements, then further optimized by feature tracking given the2D feature point correspondences between two frames. The optimizing is to minimize the summed reprojection errors between predicted pixel and observed pixel locations. The3D map points are firstly triangulated by2D correspondence of feature points in camera frame, then registered into global frame by camera to map transform.

FIG.3is a block diagram illustrating an example of a tracking filter unit ofFIG.2in further detail. As illustrated, tracking filter unit46includes mixing unit60, matrix-vector converter61, one or more extended Kalman filters (EKFs)62A-62C, model probability update unit64, fusion unit66, and vector-matrix converter67. Mixing unit60may receive, as input, the first set of information (e.g., transformation matrix T) and information indicative of previously determined position and orientation of camera device10(e.g., predicted state). In some examples, matrix-vector converter61may convert the Lie group transformation matrix T (e.g., the first set of information) to a vector utilizing the logarithm operation (e.g., log X). Conversion to matrix may be performed for calculation purposes and should not be considered as limiting.

The previously determined position and orientation of camera device10may have been previously determined by VIO unit50. As described above, the first set of information (e.g., transformation matrix T) is indicative of angular velocity and movement of camera device10based on a change in position of the device across a current frame captured by camera device10and a subsequent frame captured by camera device10.

Mixing unit60may mix the received information indicative of previously determined position and orientation of camera device10and the first set of information (e.g., transformation matrix T) to generate a mixed estimate of angular velocity and movement of camera device10. The mixed estimate of angular velocity and movement may be equal to ((P−1*predicted state)+Q−1*(transformation matrix T))/(P−1+Q−1), where P and Q are the respective covariance matrix of the predicted state and transformation matrix T, respectively.

EKFs62A-62C may each receive the mixed estimate from mixing unit60. EKFs62A-62C may be each configured to model different dynamics and kinematics of camera device10. For example, EKF62A may be configured to model constant velocity. EKF62B may be configured to model constant angular velocity. EKF62C may be configured to model constant acceleration. EKFs62A-62C may together model any intentional motion of camera device10to generate modeled motions indicative of intentional movement of camera device10. That is, any motion in reality can be represented by a combination of the motion models from EKFs62A-62C. Any other un-modeled motion patterns may be considered as unintentional movement (e.g., unintentional vibration).

For example, EKFs62A-62C may determine which portions of the angular velocity and movement from the mixing estimate from mixing unit60aligns with respective models of EKFs62A-62C. The modeled motion may be the motion that aligns with the respective models of EKFs62A-62C. Any portions of angular velocity and movement of the mixing estimate from mixing unit60that does not align with the respective models of EKFs62A-62C may represent unintentional movement of camera device10. Accordingly, tracking filter unit46may apply one or more extended Kalman filters (EKFs)62A-62C to the mixed estimate to generate one or more modeled motions. The modeled motions may be indicative of intentional movement of camera device10.

Model probability update unit64may determine model probabilities of the one or more modeled motions. Each model probability Pimatrix from paralleling EKFs62A-62C may be calculated according to each state covariance matrix Q′ over a sum of covariance matrices of all models of EKFs62A-62C: Pi=Qi−1/Σi=1i=nQi−1. The variable P is reused here as Pi, but Piis different than variable P used above for mixing unit60.

Fusion unit66may fuse the model probabilities and the one or more modeled motions to generate TIM, where TIMis information indicative of angular velocity and movement of camera device10without unintentional movement. In some examples, the output from fusion unit66may be a vector that vector-matrix converter67converts to a matrix (e.g., using the exponential operation eX). If transformation matrix T is a first set of information indicative of angular velocity and movement of camera device based on change in position of camera device10across a current frame and a subsequent frame, then transformation matrix TIMis a second set of information indicative of angular velocity and movement of camera device10without unintentional movement.

Fusion unit66may perform the fusing operation by a weighted sum of the outputs from each of EKFs62A-62C and respective model probabilities determined by model probability update unit64. For example, fusion unit66may determine Xt=Σi=1i=nPiXi. In this example, Xiis the output from each of EKFs62A-62C and Piis the respective probability from model probability update unit64. Xtrepresents TIM, but in vector form that is converted into the matrix TIMwith vector-matrix converter67.

Image stabilization unit44may utilize the transformation matrix T and the transformation matrix TIMto perform image stabilization. As one example, image stabilization unit44may determine the following:
[u0′,v0′,1]T=A(T)−1TIMA−1[u0,v0,1]T.

In the above equation, [u0, v0, 1] is coordinate of a region in the current frame prior to image stabilization. For example, u0 represents width and v0 represents height. [u0′, v0′, 1] is coordinate of the region in the image stabilized current frame. As above, u0′ represents width and v0′ represents height. “A” is the intrinsic matrix. As described above, transformation matrix T is a first set of information indicative of angular velocity and movement of camera device10based on change in position of camera device10across a current frame and a subsequent frame, and transformation matrix TIMis a second set of information indicative of angular velocity and movement of camera device10without unintentional movement. Transform unit42may generate transformation matrix T, and tracking filter unit46may generate transformation matrix TIMThe superscript T is used to represent the “transpose” and should not be confused with the transformation matrix T.

The following describes with respect toFIGS.4A and4Bthe manner in which to compute that [u0′, v0′, 1]=A*(T)−1*TIM*A−1*[u0, v0, 1]Tprovides image stabilization by compensating for unintentional movement. Image stabilization unit44may not need to perform the following equations each time. Rather, image stabilization unit44may be pre-configured to perform operations in accordance with [u0′, v0′, 1]T=A*(T)−1*TIM*A−1*[u0, v0, 1]T, and the following is to ease with understanding.

FIGS.4A and4Bare conceptual diagrams illustrating an example way in which image stabilization is performed in accordance with one or more example techniques described in this disclosure.FIG.4Aillustrates current frame70having current region72, where region72is a region in current frame70prior to image stabilization.FIG.4Billustrates subsequent frame76having region78. Subsequent frame76may be the frame captured by camera12after camera12captures current frame70. In some examples, subsequent frame76is consecutively after current frame70in capture. As described in more detail, image stabilization unit44may utilize image stabilization to rotate current region72to image stabilized region74.

In the example ofFIGS.4A and4B, the coordinates for region78may be as follows:Zc1[u1, v1, 1]T=A*T*A−1*Z0[u0, v0, 1]T, where [u1, v1, 1] represent coordinates for region78and [u0, v0, 1] represent coordinates of current region72, A is the intrinsic matrix, and T is the transformation matrix T.

As described above, transformation matrix T may include a decomposed first part of the transformation and a decomposed second part of the transformation. The decomposed second part of the transformation may be the unintentional movement information. The decomposed first part of the transformation may be the angular velocity and movement information with the unintentional movement removed (e.g., with the decomposed second part of the transformation removed). The decomposed first part of the transformation is referred to as transformation matrix TIM. The decomposed second part of the transformation is referred to as transformation matrix T.

The above equation for Zc1 can be rewritten with TIMand TNMas follows:
Zc1[u1,v1,1]T=A*TNM*TIM*A−1*Z0[u0,v0,1]T.

By moving TNMfrom the right side to the left side of the equation, the above equation can be rewritten as:
A*(TNM)−1*A−1*Zc1[u1,v1,1]T=A*TIM*A−1*Z0[u0,v0,1]T.

The left side of the above equation may be considered as the coordinates of region80in subsequent frame76. For instance, the above equation can be rewritten as:Zc1[u1′, v1′, 1]T=A*TIM*A−1*Z0[u0, v0, 1]T, where [u1′, v1′, 1] is the coordinates for region80in subsequent frame76.

Because the equation for Zc1[u1, v1, 1]Tis A*TNM*TIM*A−1*Z0[u0, v0, 1]Tthe equation for Zc1[u1′, v1′, 1]Tcan also be written as follows A*TNM*TIM*A−1*Z0[u0′, v0′, 1]T. In this equation, [u0′, v0′, 1] are the coordinates for region74in current frame70.

By combining the two equations for Zc1[u1′, v1′, 1]T, the result is:
A*TNM*TIM*A−1*Z0[u0′,v0′,1]T=A*TIM*A−1*Z0[u0,v0,1]T.

The above equation can be simplified to:
[u0′,v0′,1]T=A*(T)−1*TIM*A−1*[u0,v0,1]T.

Image stabilization unit44may perform the operations of the following equation: [u0′, v0′, 1]T=A*(T)−1*TIM*A−1*[u0, v0, 1]Ton current frame70to generate the image stabilized current frame. In some examples, rather than u0, v0 representing height and width of a region, u0 and v0 may represent coordinate of a pixel in current frame70. Image stabilization unit44may perform the operation of [u0′, v0′, 1]T=A*(T)−1*TIM*A−1*[u0, v0, 1]Ton a pixel-by-pixel basis to generate the image stabilized current frame. The equation A*(T)−1*TIM*A−1*[u0, v0, 1]Tincludes both transformation matrix T and transformation matrix TIM. Accordingly, image stabilization unit44may perform image stabilization on current frame70based on both the first set of information (e.g., transformation matrix T) and the second set of information (e.g., transformation matrix TIM) to generate an image stabilized current frame.

FIG.5is a flowchart illustrating an example method of operation in accordance with one or more examples described in this disclosure. For ease of description, the example is described with respect to IS/VIO unit36inFIG.2and tracking filter46of IS/VIO unit36inFIG.3.

Tracking filter46of IS/VIO unit36may receive a first set of information, the first set of information being indicative of angular velocity and movement of the device based on a change in position of the device across a current frame captured by camera device10and a subsequent frame captured by camera device10(90). For example, tracking filter46may receive transformation matrix T from transform unit42. As described above, transformation matrix T may be a unified transform matrix with information indicative of angular velocity and movement of the device integrated into the unified transform matrix.

In some examples, transform unit42may receive angular velocity and movement of camera device10in Euclidean space. The angular velocity and movement of camera device10information may be from IMU34and filtered by noise filters40A and40B. Transform unit42may transform, with a Lie Group transform, the angular velocity and movement of camera device10in Euclidean space to exponential manifold space to generate the first set of information. For instance, as described above, transform unit42may determine:

T1⁢0=[I3*3Δ⁢tv1+12⁢Δ⁢t2⁢g01*31][R1⁢0t1⁢0*01*31]

Tracking filter unit46may filter the first set of information to compensate for unintentional movement of the device to generate a second set of information, the second set of information being indicative of angular velocity and movement of the device without unintentional movement (92). For example, mixing unit60of tracking filter unit46may receive the first set of information (e.g., possibly after conversion to vector form with matrix-vector converter61) and receive information indicative of previously determined position and orientation of device10(e.g., predicted state inFIG.3).

Mixing unit60may mix the received information indicative of previously determined position and orientation of the device and the first set of information to generate a mixed estimate of angular velocity and movement of the device. EKFs62A-62C may apply one or more extended Kalman filters (EKFs) to the mixed estimate to generate one or more modeled motions. Model probability update unit64may determine model probabilities of the one or more modeled motions. Fusion unit66may fuse the model probabilities and the one or modeled motions to generate the second set of information (e.g., possibly with vector-matrix converter67converting to a matrix form) to generate T.

Image stabilization unit44may perform image stabilization on the current frame based on both the first set of information and the second set of information to generate an image stabilized current frame (94). For example, image stabilization unit44may perform A*(T)−1*TIM*A−1*[u0, v0, 1]T, wherein A is an intrinsic matrix, T is a matrix based on the first set of information, TIMis a matrix based on the second set of information, [u0, v0, 1] represents coordinate location of a pixel in the current frame prior to image stabilization.

VIO unit50may determine position and orientation and velocity information of camera device10based on image stabilized current frame and the subsequent frame (96). For example, image stabilization unit44may output the corrected frame to VIO unit50, and VIO unit50may determine velocity, position and orientation of the device based on the image stabilized current frame (e.g., perform pre-integration and evaluate feature points) and the subsequent frame. For instance, the pose information and velocity may be based on movement of feature points between two frames. In some examples, VIO unit50may determine the movement of feature points between the image stabilized current frame and the subsequent frame.

IS/VIO unit36may output image stabilized current frame and information indicative of determined position and orientation (98). For example, image stabilization unit44may output, for display, the image stabilized current frame. VIO unit50may output information indicative of determined position and orientation, and in some examples, also include velocity information for augmented reality or virtual reality.

FIG.6is a flowchart illustrating an example method of operation in accordance with one or more examples described in this disclosure. As described above, in some examples, there is a one-frame latency in image stabilization for a current frame. For instance, a subsequent frame is used for image stabilizing the current frame, and then VIO is performed on the subsequent frame using the image stabilized current frame. For example, the image stabilizing is on frame n, but the VIO for determining pose and velocity information is for frame n+1 based on the image stabilized frame n.

However, in some examples, there may be a benefit in determining pose and velocity information in real-time, rather than waiting for image stabilization. In such examples, the pose and velocity information may be updated using the image stabilized frame.

For example, VIO unit50may perform feature tracking using feature points on the subsequent frame (e.g., frame n+1) using un-stabilized current frame (e.g., un-stabilized frame n) for a real-time system (100). That is, VIO unit50may perform techniques similar to those described above but with respect to un-stabilized current frame.

In parallel, image stabilization unit44may perform image stabilization on the current frame (e.g., frame n) (102). Image stabilization unit44may utilize the example techniques described in this disclosure to perform the image stabilization. VIO unit50may update feature tracking on subsequent frame based on image stabilized current frame (104). As one example, VIO unit50may limit feature tracking to feature points proximate to feature points identified during the feature tracking with the un-stabilized current frame using the image stabilized current frame, and determine pose and velocity information based on the tracked feature points.

The following describes example techniques that may be used together or separately.Clause 1: A device for image processing includes a memory; and one or more processors coupled to the memory, the one or more processors configured to: receive a first set of information, the first set of information being indicative of angular velocity and movement of the device based on a change in position of the device across a current frame captured by the device and a subsequent frame captured by the device; filter the first set of information to compensate for unintentional movement of the device to generate a second set of information, the second set of information being indicative of angular velocity and movement of the device without unintentional movement; perform image stabilization on the current frame based on both the first set of information and the second set of information to generate an image stabilized current frame; and output, for display, the image stabilized current frame.Clause 2: The device of clause 1, wherein the one or more processors are further configured to: determine velocity, position and orientation of the device based on the image stabilized current frame and the subsequent frame; and output information indicative of the determined velocity, position and orientation for augmented reality or virtual reality.Clause 3: The device of any of clauses 1 and 2, wherein receiving the first set of information comprises receiving the information in a unified transform matrix with information indicative of angular velocity and movement of the device integrated into the unified transform matrix.Clause 4: The device of any of clauses 1 through 3, wherein receiving the first set of information comprises: receiving angular velocity and movement of the device in Euclidean space; and transforming, with a Lie Group transform, the angular velocity and movement of the device in Euclidean space to exponential manifold space to generate the first set of information.Clause 5: The device of any of clauses 1 through 4, wherein filtering the first set of information comprises: receiving information indicative of previously determined position and orientation of the device; mixing the received information indicative of previously determined position and orientation of the device and the first set of information to generate a mixed estimate of angular velocity and movement of the device; applying one or more extended Kalman filters (EKFs) to the mixed estimate to generate one or more modeled motions; determining model probabilities of the one or more modeled motions; and fusing the model probabilities and the one or modeled motions to generate the second set of information.Clause 6: The device of any of clauses 1 through 5, wherein performing image stabilization comprises determining: A*(T)−1*TIM*A−1*[u0, v0, 1]T, wherein A is an intrinsic matrix, T is a matrix based on the first set of information, TIMis a matrix based on the second set of information, [u0, v0, 1] represents coordinate location of a pixel in the current frame prior to image stabilization.Clause 7: The device of any of clauses 1 through 6, wherein the one or more processors are configured to receive angular velocity and movement information from an inertial movement unit (IMU) and filter the angular velocity and movement information to generate the first set of information.Clause 8: The device of any of clauses 1 through 7, wherein the device comprises a mobile telephone having one or more cameras.Clause 9: A method for image processing includes receiving a first set of information, the first set of information being indicative of angular velocity and movement of the device based on a change in position of the device across a current frame captured by the device and a subsequent frame captured by the device; filtering the first set of information to compensate for unintentional movement of the device to generate a second set of information, the second set of information being indicative of angular velocity and movement of the device without unintentional movement; performing image stabilization on the current frame based on both the first set of information and the second set of information to generate an image stabilized current frame; and outputting, for display, the image stabilized current frame.Clause 10: The method of clause 9, further includes determining velocity, position and orientation of the device based on the image stabilized current frame and the subsequent frame; and outputting information indicative of the determined velocity, position and orientation.Clause 11: The method of any of clauses 9 and 10, wherein receiving the first set of information comprises receiving the information in a unified transform matrix with information indicative of angular velocity and movement of the device integrated into the unified transform matrix.Clause 12: The method of any of clauses 9 through 11, wherein receiving the first set of information comprises: receiving angular velocity and movement of the device in Euclidean space; and transforming, with a Lie Group transform, the angular velocity and movement of the device in Euclidean space to exponential manifold space to generate the first set of information.Clause 13: The method of any of clauses 9 through 12, wherein filtering the first set of information comprises: receiving information indicative of previously determined position and orientation of the device; mixing the received information indicative of previously determined position and orientation of the device and the first set of information to generate a mixed estimate of angular velocity and movement of the device; applying one or more extended Kalman filters (EKFs) to the mixed estimate to generate one or more modeled motions; determining model probabilities of the one or more modeled motions; and fusing the model probabilities and the one or modeled motions to generate the second set of information.Clause 14: The method of any of clauses 9 through 13, wherein performing image stabilization comprises determining: A*(T)−1*TIM*A−1*[u0, v0, 1]T, wherein A is an intrinsic matrix, T is a matrix based on the first set of information, TIM is a matrix based on the second set of information, [u0, v0, 1] represents coordinate location of a pixel in the current frame prior to image stabilization.Clause 15: The method of any of clauses 9 through 14, further includes receiving angular velocity and movement information from an inertial movement unit (IMU) and filter the angular velocity and movement information to generate the first set of information.Clause 16: A computer-readable storage medium storing instructions thereon that when executed cause one or more processors to: receive a first set of information, the first set of information being indicative of angular velocity and movement of the device based on a change in position of the device across a current frame captured by the device and a subsequent frame captured by the device; filter the first set of information to compensate for unintentional movement of the device to generate a second set of information, the second set of information being indicative of angular velocity and movement of the device without unintentional movement; perform image stabilization on the current frame based on both the first set of information and the second set of information to generate an image stabilized current frame; and output, for display, the image stabilized current frame.Clause 17: The computer-readable storage medium of clause 16, further comprising instructions that cause the one or more processors to: determine velocity, position and orientation of the device based on the image stabilized current frame and the subsequent frame; and output information indicative of the determined velocity, position and orientation for augmented reality or virtual reality.Clause 18: The computer-readable storage medium of any of clauses 16 and 17, wherein the instructions that cause the one or more processors to receive the first set of information comprise instructions that cause the one or more processors to receive the information in a unified transform matrix with information indicative of angular velocity and movement of the device integrated into the unified transform matrix.Clause 19: The computer-readable storage medium of any of clauses 16 through 18, wherein the instructions that cause the one or more processors to receive the first set of information comprise instructions that cause the one or more processors to: receive angular velocity and movement of the device in Euclidean space; and transform, with a Lie Group transform, the angular velocity and movement of the device in Euclidean space to exponential manifold space to generate the first set of information.Clause 20: The computer-readable storage medium of any of clauses 16 through 19, wherein instructions that cause the one or more processors to filter the first set of information comprise instructions that cause the one or more processors to: receive information indicative of previously determined position and orientation of the device; mix the received information indicative of previously determined position and orientation of the device and the first set of information to generate a mixed estimate of angular velocity and movement of the device; apply one or more extended Kalman filters (EKFs) to the mixed estimate to generate one or more modeled motions; determine model probabilities of the one or more modeled motions; and fuse the model probabilities and the one or modeled motions to generate the second set of information.Clause 21: The computer-readable storage medium of any of clauses 16 through wherein the instructions that cause the one or more processors to perform image stabilization comprise instructions that cause the one or more processors to determine: A*(T)−1*TIM*A−1*[u0, v0, 1]T, wherein A is an intrinsic matrix, T is a matrix based on the first set of information, TIM is a matrix based on the second set of information, [u0, v0, 1] represents coordinate location of a pixel in the current frame prior to image stabilization.Clause 22: The computer-readable storage medium of any of clauses 16 through 21, further comprising instructions that cause the one or more processors to: receive angular velocity and movement information from an inertial movement unit (IMU) and filter the angular velocity and movement information to generate the first set of information.Clause 23: A device for image processing includes means for receiving a first set of information, the first set of information being indicative of angular velocity and movement of the device based on a change in position of the device across a current frame captured by the device and a subsequent frame captured by the device; means for filtering the first set of information to compensate for unintentional movement of the device to generate a second set of information, the second set of information being indicative of angular velocity and movement of the device without unintentional movement; means for performing image stabilization on the current frame based on both the first set of information and the second set of information to generate an image stabilized current frame; and means for outputting, for display, the image stabilized current frame.Clause 24: The device of clause 23, further includes means for determining velocity, position and orientation of the device based on the image stabilized current frame and the subsequent frame; and means for outputting information indicative of the determined velocity, position and orientation.Clause 25: The device of any of clauses 23 and 24, wherein the means for receiving the first set of information comprises means for receiving the information in a unified transform matrix with information indicative of angular velocity and movement of the device integrated into the unified transform matrix.Clause 26: The device of any of clauses 23 through 25, wherein the means for receiving the first set of information comprises: means for receiving angular velocity and movement of the device in Euclidean space; and means for transforming, with a Lie Group transform, the angular velocity and movement of the device in Euclidean space to exponential manifold space to generate the first set of information.Clause 27: The device of any of clauses 23 through 26, wherein filtering the first set of information comprises: means for receiving information indicative of previously determined position and orientation of the device; means for mixing the received information indicative of previously determined position and orientation of the device and the first set of information to generate a mixed estimate of angular velocity and movement of the device; means for applying one or more extended Kalman filters (EKFs) to the mixed estimate to generate one or more modeled motions; means for determining model probabilities of the one or more modeled motions; and means for fusing the model probabilities and the one or modeled motions to generate the second set of information.Clause 28: The device of any of clauses 23 through 27, wherein the means for performing image stabilization comprises means for determining: A*(T)−1*TIM*A−1*[u0, v0, 1]T, wherein A is an intrinsic matrix, T is a matrix based on the first set of information, TIM is a matrix based on the second set of information, [u0, v0, 1] represents coordinate location of a pixel in the current frame prior to image stabilization.Clause 29: The device of any of clauses 23 through 28, further includes means for receiving angular velocity and movement information from an inertial movement unit (IMU) and filter the angular velocity and movement information to generate the first set of information.Clause 30: The device of any of clauses 23 through 29, wherein the device comprises a mobile telephone having one or more cameras.

In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over, as one or more instructions or code, a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media. In this manner, computer-readable media generally may correspond to tangible computer-readable storage media which is non-transitory. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. It should be understood that computer-readable storage media and data storage media do not include carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and/or firmware.

Various examples have been described. These and other examples are within the scope of the following claims.