Motion information assisted 3A techniques

A method for adjusting parameters in a video capture device using motion information is disclosed. Motion information that indicates motion of a video capture device is determined. The motion information is compared to an upper bound and a lower bound. An aggressiveness level that indicates a change in a white balance gain for the video capture device is determined based on the comparison. A new white balance gain for the video capture device is determined based on the aggressiveness level. An exposure convergence holding time is adjusted based on the motion information. An exposure step size is increased based on the motion information. A brightness level of the video capture device is adjusted based on the convergence step size.

TECHNICAL FIELD

The present disclosure relates to digital video. Specifically, the present disclosure relates to motion information assisted 3A techniques.

BACKGROUND

Video capture devices, such as digital video cameras, may be used in different applications and environments. A video capture device may be capable of capturing video from a variety of distances relative to a target scene. Video capture devices may store captured video on a variety of media, e.g., video tape, hard drive, Digital Versatile Disk (DVD), etc.

Digital video capture devices may use a video sensor to capture video. The video capture device may include a number of video sensor configuration parameters that may be adjusted to better capture video in different environmental conditions. For example, the video capture device may include a number of sensitivity settings, e.g., white balance, exposure control, and focusing. Each of these settings may affect the smoothness and quality of captured video.

A video capture device may allow a user to manually select video sensor configuration parameters. By manually selecting the configuration parameters, the user may select settings appropriate for current environmental conditions to better capture video in that environment. Alternatively, or additionally, video capture devices may include some automatic settings that select the sensor configuration parameters based on current environmental conditions. The video capture devices may, for example, include light sensors that detect the brightness of the surrounding environment and select the configuration setting based on the amount of light detected. Many video capture devices also include automatic focusing of the camera. Video capture devices typically have many different parameters and settings that affect the quality of the video being taken. Therefore, there is a need for improved techniques for adjusting parameters in a video capture device.

DETAILED DESCRIPTION

A method for adjusting parameters in a video capture device using motion information is described. Motion information is determined by a video capture device that indicates motion of the video capture device. The motion information is compared to an upper bound and to a lower bound. An aggressiveness level is determined based on the comparison. The aggressiveness level indicates a change in a white balance gain for the video capture device. A new white balance gain for the video capture device is determined based on the aggressiveness level.

The motion information may be determined using data received from an accelerometer in the video capture device. The motion information may include an estimated average velocity of the video capture device. The motion information may be a motion vector. Determining a motion vector may include summing pixel values in each row of a first frame to form a first column vector. Pixel values in each row of a second frame may be summed to form a second column vector. Pixel values in each column of the first frame may be summed to form a first row vector. Pixel values in each column of the second frame may be summed to form a second row vector.

A first shift needed to reach a peak value of autocorrelation of the first row vector and the second row vector may be determined. The first shift may be a horizontal component of the motion vector. A second shift needed to reach a peak value of autocorrelation of the first column vector and the second column vector may be determined. The second shift may be a vertical component of the motion vector.

Determining the aggressiveness level may include setting the aggressiveness level to zero if a length of the motion vector is less than or equal to the lower bound. The aggressiveness level may be set to a value that is proportional to the length of the motion vector if the length of the motion vector is greater than the lower bound and less than the upper bound. The aggressiveness level may be set to a predetermined maximum if the length of the motion vector is greater than or equal to the upper bound.

A change from an old white balance gain to a new white balance gain may be bigger for a large aggressiveness level than for a small aggressiveness level. Determining the new white balance gain may include using an equation: new white balance gain=old gain*(1−w)+current gain*w. Old gain is the old white balance gain for a previous frame, current gain is a white balance gain for a current frame, and w is the aggressiveness level. An exposure convergence holding time may be adjusted based on the motion information. An exposure step size may be increased based on the motion information. A brightness level of the video capture device may be adjusted based on the convergence step size.

Increasing an exposure step size may be dependent on a length of a motion vector. Determining if the video capture device is panning may be based on the motion information. Performing auto focus in the video capture device may be based on the panning determination. Performing auto focus may include not performing auto focus if panning is detected and performing auto focus if panning is not detected.

An apparatus for adjusting parameters in a video capture device based on motion information is described. The apparatus includes a processor, memory in electronic communication with the processor, and instructions stored in the memory. The instructions are executable by the processor to determine motion information that indicates motion of a video capture device. The instructions are also executable by the processor to compare the motion information to an upper bound and to a lower bound. The instructions are further executable to determine an aggressiveness level based on the comparison. The aggressiveness level indicates a change in a white balance gain for the video capture device. The instructions are also executable to determine a new white balance gain for the video capture device based on the aggressiveness level.

A computer-program product for adjusting parameters in a video capture device is also described. The computer-program product includes a computer-readable medium having instructions thereon. The instructions include code for determining motion information that indicates motion of a video capture device. The instructions also include code for comparing the motion information to an upper bound and a lower bound. The instructions further include code for determining an aggressiveness level based on the comparison. The aggressiveness level indicates a change in a white balance gain for the video capture device. The instructions also include code for determining a new white balance gain for the video capture device based on the aggressiveness level.

An apparatus for adjusting parameters in a video capture device is described. The apparatus includes means for determining motion information that indicates motion of a video capture device. The apparatus also includes means for comparing the motion information to an upper bound and a lower bound. The apparatus further includes means for determining an aggressiveness level based on the comparison. The aggressiveness level indicates a change in a white balance gain for the video capture device. The apparatus also includes means for determining a new white balance gain for the video capture device based on the aggressiveness level.

A method for adjusting parameters in a video capture device based on motion information is also described. Motion information is determined by a video capture device that indicates motion of the video capture device. An exposure convergence holding time is adjusted based on the motion information. An exposure step size is increased based on the motion information. A brightness level of the video capture device is increased based on the convergence step size.

Increasing an exposure step size may be dependent on a length of the motion vector. Adjusting the exposure convergence holding time may include reducing the exposure convergence holding time by forty percent to sixty percent. A drop in the brightness level to a value outside a predetermined range of a luma target may be detected.

An apparatus for adjusting parameters in a video capture device based on motion information is described. The apparatus includes a processor, memory in electronic communication with the processor, and instructions stored in the memory. The instructions are executable by the processor to determine motion information that indicates motion of a video capture device. The instructions are also executable by the processor to adjust an exposure convergence holding time based on the motion information. The instructions are further executable by the processor to increase an exposure step size based on the motion information. The instructions are also executable by the processor to adjust a brightness level of the video capture device based on the convergence step size.

A computer-program product for adjusting parameters in a video capture device is described. The computer-program product includes a computer-readable medium having instructions thereon. The instructions include code for determining motion information that indicates motion of a video capture device. The instructions also include code for adjusting an exposure convergence holding time based on the motion information. The instructions further include code for increasing an exposure step size based on the motion information. The instructions also include code for adjusting a brightness level of the video capture device based on the convergence step size.

An apparatus for adjusting parameters in a video capture device is also described. The apparatus includes means for determining motion information that indicates motion of a video capture device. The apparatus also includes means for adjusting an exposure convergence holding time based on the motion information. The apparatus further includes means for increasing an exposure step size based on the motion information. The apparatus also includes means for adjusting a brightness level of the video capture device based on the convergence step size.

Motion vectors may be used to determine motion in a video capture device: still, panning, or others. Additionally, light condition changes may result in the adjustment of 3A (Auto-Exposure, Auto-White Balance, Auto-Focus). In one configuration, 3A adjustment is based on sensor-received statistics without using any motion vector related information. However, the panning motion of a video capture device may also result in the adjustment of 3A, which produces a smooth point-and-converge process. Therefore, the present systems and methods may adjust parameters in a video capture device using motion information. With motion information, the 3A adjustment may become a smart motion-aware control. In other words, motion information may be used in 3A adjustment to achieve the desired smooth point-and-converge performance during panning while keeping the original 3A adjustment design for still scene light condition changes.

FIG. 1Ais a block diagram illustrating a video capture device102that adjusts parameters using motion information. A video sensor104may include two video buffers106a-b, both of which may store video information, e.g., pixel values, pertaining to a captured video frame. The video sensor104may, for example, store the video information in the video buffers106during video preview. More specifically, the video sensor104may store video information in the video buffers106while the user is pointing the video capture device102at the scene of interest, but before the user actuates a button to capture the video, i.e., a record button. In one configuration, the video sensor104may capture and store the videos in the video buffers106within seconds, if not milliseconds or even shorter time periods, of one another. In this case, the video sensor104may store successive sets of video information pertaining to the same scene of interest in the video buffers106. This video information may then be stored as frames122of a captured video. This successive video buffering technique may occur unnoticed by a user attempting to capture the video. That is, from the user's perspective, only a single video may be captured via actuation of a button on the video capture device102. In another configuration, the video capture device102may include more or less than two video buffers106.

The video buffers106may include any volatile or non-volatile memory or storage device, such as FLASH memory, or such as a magnetic data storage device or optical data storage device. In an alternative configuration, the video buffers106may reside outside of the video sensor104, such as within other modules of the video capture device102, including a processor108.

The video sensor104may also include a video sensor configuration module110that configures the video sensor104in accordance with configuration parameters112received from the processor108. The configuration parameters112received from video processor108may include a white balance gain, an exposure convergence holding time, an auto exposure step size, or a panning determination to be used during auto focus. The video sensor104may expose video sensor elements to the scene of interest in accordance with the configuration parameters112to capture the video. In particular, based on the configuration parameters112, the video sensor configuration module110may perform auto white balancing, auto exposure control, and/or auto focusing. Auto white balancing may be the adjustment of color intensities in captured frames122. Auto exposure control may be the adjustment of an exposure convergence holding time and/or an exposure step size that determines the brightness level of captured frames. Auto focusing may be the adjusting of a lens apparatus to clearly focus on a targeted object or group of objects.

The processor108on the video capture device102may include a sensor control module114. The sensor control module114may include a motion detection module116that determines the amount of motion within the scene of interest. More specifically, the motion detection module116may generate one or more indicators that identify changes between the frames of a continuous video stream that are indicative of motion. This may include calculating global motion vectors or local motion vectors. As used herein, the term “global motion vector” refers to a vector calculated using two or more frames in a video sequence. As used herein, the term “local motion vector” refers to a vector calculated using two or more sub-frames in a video sequence. In other words, a global motion vector may indicate movement of everything or almost everything from one frame to another, i.e., a global motion vector may indicate movement of the video capture device102itself. In contrast, a local motion vector may indicate movement of a portion of a frame from one frame to another, i.e., a local motion vector indicates that something being filmed has moved relative to its environment, but the video capture device102has not necessarily moved. As used herein, the term “motion information” refers to global motion vectors, local motion vectors, or any other data that indicates movement of a video capture device102or movement of something being filmed by a video capture device102, e.g., data from an accelerometer in the video capture device102.

A 3A adjustment module118may analyze the motion information generated by the motion detection module116, and adjust one or more configuration parameters112of the video sensor104based on the amount of motion detected between two frames122. Although discussed herein as adjusting an auto white balance gain, an auto exposure control delay, and/or an auto exposure control step size, the 3A adjustment module118may adjust other configuration parameters of the video sensor104. In general, the 3A adjustment module118may increase the auto white balance gain of the video sensor104for scenes that include a relatively large amount of global motion. This may result in faster auto white balance convergence during motion, thus reducing color tone jitter in the captured video. The 3A adjustment module118may also decrease the auto exposure control delay and increase the auto exposure control step size during scenes with relatively large amounts of global motion. This may result in a faster convergence to a target luma value, e.g., the brightness level of captured video may be corrected faster during panning The 3A adjustment module118may also turn off auto focus during scenes with relatively large amounts of global motion. This may eliminate unwanted focusing during panning.

The processor108may also include 3A data120that is used by the 3A adjustment module118to determine the configuration parameters112. The 3A data120may, for example, include a plurality of different configuration parameters112for the different operating modes of the camera that may be chosen by a user. Furthermore, the 3A data120may include data about the current configuration parameters112that may be used by the 3A adjustment module118to produce future configuration parameters112. In one configuration, the 3A adjustment module118may select initial configuration parameters112for the video sensor104based on brightness, and adjust the configuration parameters112of the video sensor104based on analysis of subsequent motion information.

FIG. 1Bis a block diagram illustrating another video capture device102bthat adjusts parameters using motion information109. The video capture device102bofFIG. 1Bmay be one configuration of the video capture device102ofFIG. 1A. The video capture device102bmay include a video sensor104b. The video sensor104bmay include an accelerometer105. The accelerometer105may provide acceleration a107to a motion detection module116b. The acceleration107may be in x,y,z coordinates. The motion detection module116bmay be part of a sensor control module114bon a processor108b. The sensor control module114bofFIG. 1Bmay have similar functions as the sensor control module114ofFIG. 1A. The processor108bofFIG. 1Bmay have similar functions as the processor108ofFIG. 1A.

The motion detection module116bmay use the acceleration107to determine the velocity v=at and distance moved

s=at22.
For example, if the accelerometer105provides an estimated acceleration a107and the frame rate is 30 frames/second, the motion detection module116bmay determine the velocity and distance using t= 1/30 seconds. The motion detection module116bmay then determine the change in velocity and the change in distance. If the change in distance or the change in velocity are almost constant (no abrupt large change), then a panning process is initially found. The convergence speed may be switched from a normal mode to a motion information (MI) assisted convergence mode. In MI assisted convergence mode, the convergence speed may depend on the estimated currentVj, whereVjis the estimated average velocity of the video capture device102bover the latest 1 second.

To determineVj, the change in velocity for each frame may be determined. Then,

V_⁢j=1N⁢∑i=1N⁢⁢Δ⁢⁢Vi
may be computed where N is the current frame rate. The change in velocity for each frame may be determined using a buffer113ain the motion detection module116b. Alternatively, the change in velocity for each frame may be determined using a buffer113bin the accelerometer. With each new frame, statistics for the new frame may be input into the buffer113. The convergence speed may then be calculated as a function ofVj. IfVjhas an abrupt change, an interruption of the panning process is detected. The convergence mode may then be switched back to normal mode.

The motion detection module116bmay receive the frame rate/synchronization information111. The frame rate/synchronization information111may be received from the video sensor104b, the processor108b, or elsewhere in the video capture device102b. The frame rate/synchronization information111may include the synchronization information between the frame index and the velocity index. The motion detection module116bmay then output motion information109to a 3A adjustment module118. The motion information109may includeVj. The other components of the video capture device102bofFIG. 1Bmay perform in a similar manner to the components of the video capture device102ofFIG. 1A

FIG. 2is a block diagram illustrating a video processor208that produces configuration parameters212from video information224. The video information224may be received from one or more video buffers106and may include frames122or portions of frames122within a captured video stream. A motion detection module216may determine a global motion vector232or a local motion vector240from the video information224.

Motion vectors may be defined and computed in several ways. A global motion module226may compute global motion vectors232from global rows228and global columns230in one or more frames. In contrast, a local motion module234may compute a local motion vector240for an N×M block in a frame, e.g., using local rows236and local columns238. Specifically, a local motion vector240may be obtained by predicting motion in the temporal domain based on neighboring frames. These computations may be very complex depending on the size of the blocks and the search range in the neighboring frames. Motion information may also be detected by an accelerometer242built into a video capture device102.

The motion detection module216may determine motion information and send it to a 3A adjustment module218that may produce configuration parameters212using 3A data220. The 3A adjustment module218may include an auto white balancing (AWB) module244, an auto exposure control (AEC) module246, and an auto focus (AF) module248.

The auto white balance module244may determine a new white balance gain256based on motion information. In real world usage condition, the light source for a filmed video may not change very often. However, depending on the scene, an auto white balance decision may result in jittery captured video. Without a temporal filter (to slow down the white balance (WB) gain change), a user may notice very annoying color tone jitter in preview operation, as well as in the recorded videos and the continuous snapshots. Therefore, a fixed-parameter temporal filter may be used to adjust the white balance convergence process in the auto white balance module244. Specifically, a relatively long (around or more than 10 seconds) white balance convergence process may be applied in order to achieve a smooth AWB transition when the light temperature changes. Furthermore, 3 fixed aggressiveness levels (w) may be used for the camera mode in a video capture device102: w(low)=0.05, w(med)=0.15, w(high)=0.45. Additionally, one fixed aggressiveness level (w) may be used for general camcorder mode in a video capture device102: w=0.025. With this fixed-aggressiveness level, the auto white balance gain256may converge smoothly (over around 10 sec.) when the light temperature changes. The white balance for a video frame may be adjusted according to Equation (1):
new white balance gain=old gain*(1−w)+current gain*w(1)

where the new white balance gain256is the white balance gain actually applied to the video front end, (i.e., sensor configuration module110), old gain258is the white balance gain of the previous frame (e.g., based on lighting conditions), and current gain260is the white balance gain for the current frame (e.g., based on lighting conditions).

During panning, however, it may be desirable for the auto white balance convergence to be a smooth point-and-converge process. Therefore, motion information may be used to detect the panning movement of the video capture device102and assist the auto white balance module244to make the appropriate adjustment. If the panning motion is detected, the auto white balance convergence period may be reduced by adjusting the parameters of the temporal filter. In other words, using general camcorder mode with fixed aggressiveness may not converge fast enough during a panning process, i.e., using a fixed aggressiveness level may allow auto white balance decision instability when there is no panning but auto white balance is in an indecisive situation. In order to solve this problem, the auto white balance module244may use a panning-aware auto white balance convergence technique that allows the auto white balance to converge in a smooth and point-to-converge fashion during panning process, but allows no auto white balance instability. This may be performed using a panning mode aggressiveness level254that is adjusted based on motion information.

A panning mode aggressiveness level (w)254may be determined by Equation (2):

where ∥MV∥ is the norm of the motion vector (i.e., motion vector length262), Threshold_low250is a lower limit of the MV length262, Threshold_high252is an upper limit for the MV length262, and α is a scalar that keeps the value of the panning mode aggressiveness level (w)254within the range of (0, 0.05). Threshold_low250and Threshold_high252may be video capture device specific. Therefore, if the motion vector length262is less than or equal to Threshold_low250, w may equal zero. In other words, the auto white balance gain256may not be updated by the current auto white balance gain260, unless there is a sudden brightness change. Additionally, if the length of the motion vector is greater than or equal to Threshold_high252, w may equal 0.05. Therefore, 0.05 may be the highest panning mode aggressiveness level254for camcorder mode. If, however, the motion vector length262is between Threshold_low250and Threshold_high252, w may increase proportionally to the motion vector length262(as scaled by α). Therefore, one of the configuration parameters212sent to a sensor configuration module110may be the new white balance gain256that is adjusted using the panning mode aggressiveness level254.

Threshold_low250and Threshold_high252may depend on the way the motion vector is computed. Typically, hand jitter creates a certain amount of motion. However, the motion from hand jitter does not affect 3A data220convergence. Therefore, Threshold_low250may be determined by the hand motion triggered motion vector size. Threshold_high252may be determined by consistent panning motion in roughly the same direction. If the current frame and previous frame has a shift of ˜10%, the amount of motion may be determined to be Threshold_high252. Both Threshold_low250and Threshold_high252may be adjustable and tunable.

The auto exposure control module246may speed up an auto exposure control convergence process using motion information. Auto exposure control convergence may occur when the light intensity of a target scene changes. Auto exposure control convergence may be designed to start with an exposure convergence holding time270to prevent minor disturbances, followed by a smooth converging process back to a luma target266, i.e., an exposure step size268may be adjusted until the brightness level264re-enters an acceptable range within a luma target266.

During panning, it may be desirable for the auto exposure control convergence to be a point-and-converge process. As before, motion information may be computed to detect the panning movement of the video capture device102, e.g., a global motion vector232. If panning movement is detected, the auto exposure control (AEC) module246may reduce an exposure convergence holding time270and converge back to the luma target266in a fast but smooth process. In other words, when a video capture device102is not panning, the exposure convergence holding time270prevents disturbances. However, if panning is detected (from motion information), a shorter exposure convergence holding time270may be desirable. If panning is detected, the exposure convergence holding time270may be reduced by 40-60% of normal.

Furthermore, if panning motion is detected, the auto exposure control convergence process may also be sped up by increasing the exposure step size268based on the panning speed. In order to keep the convergence process as smooth as possible, the exposure step size268may only be increased or reduced by a factor within [0, 1], which is panning speed dependent. Equation (3) illustrates how an exposure step size268may be determined:
Step_Size(pan)=Step_Size(org)+F[V](3)

where Step_Size(pan) is the exposure step size268actually sent to a sensor configuration module110, Step_Size(org) is the previous exposure step size272, and F[V] is an incrementing function274that is dependent on the panning speed and is in the range [0,1]. The exposure step size can only be performance at an integer. The incrementing function (F[V])274may be accumulated until it reaches an integer, then the corresponding exposure step size268may be increased or reduced by 1.

Therefore, two of the configuration parameters212sent to a sensor configuration module110may be the exposure convergence holding time270and the exposure step size268that is adjusted using an incrementing function274.

The auto focus (AF) module248may decide whether to re-focus using motion information. It may be desirable to avoid re-focusing a video capture device102while it is panning Since the view of the video capture device102window is changing continuously with the panning movement, re-focusing during this process may cause indecisive focusing behavior, i.e., captured video may alternate rapidly between out-of-focus and in-focus. To avoid this, the auto focus module248may use motion vectors to determine when to re-focus. Specifically, if a panning motion of the video capture device102is detected, the video capture device102may not re-focus. Rather, re-focusing may proceed when the panning motion is stopped, as indicated by the length of the motion vectors. Motion vectors may also assist the auto focus module248to track the object of interest and help the auto focus module248to make accurate decisions by detecting scene changes.

One possible panning detection algorithm may use a focus value (FV) for panning detection that may be unreliable and inaccurate. This algorithm assumes that if there is no panning and object motion, the current FV and the average FV of the past three frames are identical or at least very close. If this difference is greater than 10% of the FV average of the past three samples, panning motion is detected. This is illustrated in Equation (4) where panning is detected if:
|FVcurrent−FVpast3samples|>10%×FVpast3samples(4)

However, this assumption may not be accurate since object movement may also satisfy this condition. In order to solve the problem of this auto focus panning detection algorithm, real motion information from the motion detection module216may be used to determine a panning detection276. Global motion vectors232(computed from global rows228and global columns230) may assist the auto focus module248with determining a panning detection276. Local motion vectors240(block based motion vectors inside of the frames) may assist the auto focus module248with object tracking. Therefore, one of the configuration parameters212sent to a sensor configuration module110may be the panning detection276that is adjusted using the motion information from the motion detection module216.

FIG. 3is a block diagram illustrating global motion vectors332. Captured video may include many frames. A first frame384may include a first object378a, a second object380a, and a third object382ain a particular configuration. A subsequently received second frame386may include the same objects that may be positioned the same relative to each other, but in a different position relative to the frame boundaries. In other words, the first object378b, second object380b, and third object382bmay also be in the second frame386, but shifted in a particular direction. This may indicate global motion, i.e., movement of the video capture device102. The global motion vector332may be the amount of shift in the entire scene. Therefore, a first global motion vector332a, a second global motion vector332b, and a third motion vector332cmay be identified based on shifts in different columns and/or rows of pixels in the frames384,386, but should be very similar. As shown in the composite frame387, a global motion vector332may have a horizontal component388a-cand a vertical component390a-c.

To compute global motion vectors332, a motion detection module216may generate 1-dimensional projection values for one or more frames and compare the projections of the frames to detect motion within the scene of interest. In particular, the motion detection module216may compute horizontal projections (global rows228), vertical projections (global columns230) or both for each frame. Horizontal projections are summations of the pixel values of a row of pixels of a frame, e.g., up to 1024 sub-sampled rows. Vertical projections are summations of the pixel values of a column of pixels of a frame, e.g., up to 1024 sub-sampled columns. For example, the motion detection module216may determine horizontal projections for a frame according to Equation (5):

where PH(j) denotes the summation of the pixels in the jth row (i.e., i is a column index and j is a row index), and Pix(i, j) denotes the pixel value of the pixel in the ith column and jth row. The motion detection module216may also determine vertical projections for a frame according to Equation (6):

where PV(i) denotes the summation of the pixels in the ith column. PH, therefore is the summation of the x-axis pixel values (as i varies and j remains static) of a particular frame. Likewise, PVis the summation of the y axis pixel values (as i remains static and j varies) of a particular frame.

The global motion vector332may be estimated from the global rows228and global columns230, i.e., a global column230of frame N+1 may represent a shifted global column230of frame N. The amount of shift may be the horizontal component388of the global motion vector332. Likewise, a global row228of frame N+1 may be a shifted global row228of frame N. The amount of shift may be the vertical component390of the global motion vector332. Once the vertical projections and horizontal projections for two frames have been determined, autocorrelation may be used to find the peak of the projections. The amount of shift needed to reach the peak is the global motion vector332.

Additionally, in one configuration, a global motion vector332may be determined from 2 vectors. Given vector N and vector N+1 of the same length L, the cross-correlation F may be computed according to Equation (7):
F=ΣN(n)*N+1(n+k),n=0, . . . ,(L−1)  (7)

where n and k are indices. The index k may be varied in a search range from −S to +S, (search range is 10% of the vector length). A vector with an index less than zero or greater than or equal to L may be appended with zeros.

The k that generates the largest F may be the estimated motion vector. Since the global rows228and global columns230are large numbers, the product and sum of the two vectors may create even greater numbers and cause overflow with fixed point computation. Therefore, the cross-correlation computation may be implemented as Sum of Absolute Difference (SAD) according to Equation (8):
F=Σabs(N(n)*N+1(n+k)),n=0, . . . , (L−1)  (8)

with SAD, the k that produces smallest F value may be used as the motion vector.

Furthermore, the history of the past global motion vectors332may be placed in a queue for a 3A adjustment module218to utilize. The queue may be first-in first-out (FIFO) for motion vectors (dx, dy) for 1 second (30 frames). The 3A adjustment module218may use a current motion vector (the last entry in the history queue) and the past history to adjust its convergence speed and re-focusing decision. If the motion is too large and exceeds the search range, the index k that produces the largest F value (with cross-correlation) or the smallest F value (with SAD) in the search range may be used.

FIG. 4is another block diagram illustrating global motion vectors232. A first frame484illustrates a frame of video captured while a video capture device102is panning A second frame486illustrates a frame of video captured following the first frame484, i.e., the mountains have shifted horizontally in the second frame486with respect to the first frame484. The global rows228and global columns230may be determined for the area enclosed by the dotted boxes492. For example the global rows228may be determined for the second dotted box492bin the first frame484and the fourth dotted box492din the second frame486while the global columns230may be determined for the first dotted box492ain the first frame484and the third dotted box492cin the second frame486. Alternatively, the global rows228may be determined for the first dotted box492aand the third dotted box492cand the global columns230may be determined for the second dotted box492band the fourth dotted box492d. Alternatively, the global rows228and the global columns230may be determined for the area where the first dotted box492aoverlaps with the second dotted box492bin the first frame484and the area where the third dotted box492coverlaps with the fourth dotted box492din the second frame486.

The second dotted box492band the fourth dotted box492dmay represent the windows/regions for computing the SAD of the row sum of the luminance Y (or red (R), green (G), blue (B)) in the neighboring two frames N and N+1 in order to detect panning motion in the vertical direction. The first dotted box492aand the third dotted box492cmay represent the windows/regions for computing the SAD of the column sum of Y (or R, G, B) in the neighboring two frames N and N+1 in order to detect panning motion in the horizontal direction.

The row difference window494and the column difference window495illustrate how panning may be detected using the global rows228and global columns230. The row difference window494illustrates a first difference curve496and a second difference curve497. The first difference curve496may be the SAD between the first frame484and the second dotted box492b. The second difference curve497may be the SAD between the first frame484and the fourth dotted box492. The location of the first difference curve496in relation to the second difference curve497in the row difference window494indicates that no vertical panning is detected between the first frame484and the second frame486.

Similarly, the column difference window495illustrates a third difference curve498and a fourth difference curve499. The third difference curve may be the SAD between the first frame484and the first dotted box492a. The fourth difference curve may be the SAD between the first frame484and the third dotted box492c. The location of the third difference curve498in relation to the fourth difference curve499in the column difference window495indicates that horizontal panning has been detected.

FIG. 4illustrates one example of how to detect panning motion using global motion vectors232. Additional ways of panning detection by motion information may also be used.

FIG. 5is a block diagram illustrating a local motion vector540. A local motion vector540may indicate movement in a sub-block of a frame. A first frame584may include a first object578a, a second object580a, and a third object582ain a particular configuration. A subsequently received second frame586may include the same objects, however in contrast toFIG. 3, the first object578bmay be in a different position but the second object580band third object582bmay be in the same or similar position. In other words, the first object578bhas moved while the second object580band third object582bhave not moved. This may indicate local motion, i.e., movement of a particular subject of filming, but not an entire scene. Therefore, the local motion vector540may indicate the movement of the first object578from the first frame584to the second frame586. As shown in the composite frame587, the local motion vector540may have a horizontal component588and a vertical component590.

FIG. 6is a flow diagram illustrating a method600for determining a new white balance gain256using motion information. The method600may be performed by a motion detection module216and an auto white balance module244in a video processor208. The motion detection module216may determine602motion information that indicates motion of a video capture device102, e.g., a global motion vector232. The motion information may be a motion vector. Alternatively, other types of motion information may be used, e.g., data from an accelerometer242. The auto white balance module244may determine604an upper bound for the motion information and a lower bound for the motion information, e.g., Threshold_high252and Threshold_low250, respectively. The auto white balance module244may also compare606the motion information to the upper bound and the lower bound. The auto white balance module244may also determine608an aggressiveness level based on the comparison, i.e., determining a panning mode aggressiveness level254determined using Equation (2). The panning mode aggressiveness level254may indicate the change to the white balance gain in the video capture device102, e.g., a large w may indicate that the new white balance gain256will be very different than the old white balance gain258and a small w may indicate that the new white balance gain256will not be very different than the old white balance gain258. The auto white balance module244may also determine610a new white balance gain256for the video capture device102based on the aggressiveness level, i.e., applying the panning mode aggressiveness level254to Equation 1 to determine a new white balance gain256. This new white balance gain256may be sent to a sensor configuration module110as a configuration parameter112.

FIG. 7is a flow diagram illustrating a method700for auto exposure control. Auto exposure control may be the process of adjusting configuration parameters212so that a brightness level264in captured video converges to a luma target266, or converges to within an acceptable range of the luma target266. The method700may be performed by an auto exposure control module246. The auto exposure control module246may receive714a luma target266and a previous exposure step size272as input. The auto exposure control module246may also low pass filter716the luma target266, i.e., apply weighting to the luma target266. The auto exposure control convergence process may then include three states: holding time control, convergence time control, and exposure step size control. The auto exposure control module246may adjust718the exposure convergence holding time270. In one configuration, the exposure convergence holding time270is about one second to prevent minor disturbances. The auto exposure control module246may also adjust720the convergence time control. This may include adjusting a brightness level264back to the luma target266smoothly, i.e., with little or no oscillation or overshoot. The auto exposure control module246may also adjust722the exposure step size268based on the current stage during the convergence process and changes in the overall brightness level264. The auto exposure control module246may also output724a current exposure step size268.

In one configuration, the exposure convergence holding time270may be shortened using panning aware auto exposure control, e.g., the holding time270may be shortened to 40-60% of its value if panning is detected from motion information. Additionally, the exposure step size268may be increased if panning is detected to speed up the convergence of the brightness level264to the luma target266, e.g., using Equation (3).

FIG. 8is a flow diagram illustrating a method800for panning aware auto exposure control in a video capture device102. The method800may be performed by a motion detection module216and an auto exposure control module246in a video processor208. The motion detection module216may determine826motion information that indicates motion of a video capture device102, e.g., a global motion vector232. The motion information may be a motion vector. Alternatively, other types of motion information may be used, e.g., data from an accelerometer242. The auto exposure control module246may adjust828an auto exposure convergence holding time270based on the motion information, e.g., reduce the holding time270to 40-60% of its value if panning is detected from motion information. The auto exposure control module246may also increase830an exposure step size268based on the motion information. This may include determining exposure step size268according to Equation (3) using a previous exposure step size272and an incrementing function (F[V])274. The auto exposure control module246may also adjust832a brightness level of the video capture device102based on the exposure step size268.

FIG. 9is a flow diagram illustrating a method900for holding time control during auto exposure control. In other words, the method900illustrated inFIG. 9may be performed instead of step718in the method700illustrated inFIG. 7. The method900may be modified if panning is detected based on motion information, i.e., the holding time270may be reduced by 40-60%. The method900may be performed by an auto exposure control module246. The holding time control may be achieved by skipping certain frames to prevent instantaneous brightness convergence. Holding time control may be performed when the following conditions are satisfied: (1) the brightness level264is out of the luma target266tolerance range, e.g., within 8% of the luma target266; (2) the luma target266has previously been achieved before the brightness level264changes; and (3) the lowest or highest brightness level264is reached during the brightness drop/increase process. In the illustrated method, an example of Threshold hold low is 6 and Threshold hold high is 215.

FIG. 10is a flow diagram illustrating a method1000for convergence time control during auto exposure control. In other words, the method1000illustrated inFIG. 10may be performed instead of step720in the method700illustrated inFIG. 7. The method1000may be modified if panning is detected based on motion information, i.e., the exposure step size268may be increased using Equation (3). The method1000may include a low frame rate case and a high frame rate case. The convergence speed may be dependent on the exposure step size268. To achieve a smooth convergence process, the exposure step sizes268may be adjusted according to its stage during the convergence process. Frame skipping may be performed to prevent oscillation or overshoot from occurring during convergence. When the brightness level264changes more than a predetermined threshold and the difficulty of bringing the lowest/highest brightness up/down is greater than another threshold, the frame skipping is performed every other frame. Frame skipping is only performed on certain stages during the convergence process to have the least effect on the convergence smoothness.

FIG. 11is a flow diagram illustrating a method1100for convergence time control during auto exposure control. In other words, the method1100illustrated inFIG. 11may be performed instead of step722in the method700illustrated inFIG. 7. The method1100may be modified if panning is detected based on motion information, i.e., the exposure step size268may be increased using Equation (3). The exposure step size268may be dependent on the current stage of the convergence process and the overall brightness level264change. The exposure step size238may be 0, +1, −1, +2, −2, +8, or −8 depending on the conditions illustrated in the left side of the method1100.

FIG. 12is a graph1200illustrating panning aware auto exposure control as a function of time. The graph1200includes a panning aware curve (solid line) and a panning unaware curve (dashed line) as a function of time. The panning aware curve may illustrate auto exposure control using motion information. The panning aware curve may include a panning aware brightness drop1202, a panning aware holding time1206, and a panning aware convergence1210. In contrast to the panning aware curve, the panning unaware curve may illustrate auto exposure control without using motion information. The panning unaware curve may include a panning unaware brightness level drop1204, a panning unaware holding time1208, and a panning unaware convergence1212.

Auto exposure control, either panning aware or panning unaware, may occur when a brightness level264drops below an acceptable range of a luma target1266, e.g., 8% of the luma target1266. After a qualifying brightness level264drop, an auto exposure control module246may perform auto exposure control based on motion information. Specifically, a holding time may be determined based on motion information. If panning aware auto exposure is used, the panning aware holding time1206may be 40-60% less than the panning unaware holding time1208. The panning unaware holding time1208may be a constant, (e.g., about 1 second), or may be determined by the auto exposure control module246. After the holding time, the auto exposure control module246may then adjust an exposure step size268until the brightness level264converges back within an acceptable range of the luma target1266. Panning aware convergence1210may be faster because the exposure step size268may be increased faster than the panning unaware convergence1212based on an incrementing function274.

FIG. 13is a graph1300illustrating increases in an exposure step size1368during panning aware auto exposure control. Specifically, the graph1300illustrates seven segments1314a-g, each of which may represent a frame122of captured video. The numbers on the top side of the graph1300may represent the exposure step size1368sent as a configuration parameter212. For example, the exposure step size1368is 1 for the first segment1314a, 1 for the second segment1314b, 1 for the third segment1314c, etc. The numbers on the bottom side of the graph1300may represent the value of an incrementing function (F[V])1374that may be used to adjust the exposure step size1368, e.g., using Equation (3). The incrementing function (F[V])1374may be based on the panning speed as indicated by motion information and may be in the range [0,1]. For example, the incrementing function1374is 0.3 for the first segment1314a, 0.6 for the second segment1314b, and 0.9 for the third segment1314c. However, when 0.2 is added to the incrementing function (F[V])1374before the fourth segment1314d, the incrementing function (F[V])1374for the fourth segment1314dmay wrap around to 0.1 instead of 1.1 and the exposure step size1368for the fourth segment1314dmay be increased from 1 to 2 according to Equation (3). In other words, the incrementing function1374may be accumulated until it reaches 1 or more, at which point the corresponding exposure step size1368may be increased or reduced by 1. Similarly, as the incrementing function1374reaches 1 again at the end of the sixth segment1314f, the incrementing function (F[V])1374for the seventh segment1314gmay wrap around to 0.0 instead of 1.0 and the exposure step size1368for the seventh segment1314gmay be increased from 1 to 2.

FIG. 14is a flow diagram illustrating a method1400for auto focusing a video capture device102based on motion information. The method1400may be performed by a motion detection module216and an auto focus module248in a video processor208. When panning, it may be desirable to avoid re-focusing since the view of the video capture device102window may be changing. Therefore, motion information may be used to assist the auto focus module248to determine whether to re-focus. The motion detection module216may determine1418motion information that indicates motion of a video capture device102, e.g., a global motion vector232. The motion information may be a motion vector. Alternatively, other types of motion information may be used, e.g., data from an accelerometer242. The auto focus module248may determine1420if the video capture device102is panning based on the motion information. One possible way to determine panning may be to use Equation (4). However, since this may not be very accurate, the auto focus module248may instead use the motion vector length262to determine panning, i.e., a panning detection276is true if the motion vector length262is larger than a predetermined threshold and false if the motion vector length262is less than a predetermined threshold. The auto focus module248may also perform1422auto focusing in the video capture device102based on the panning determination. This may include sending the panning detection276as a configuration parameter212to a sensor configuration module110. For example, the video capture device102may re-focus only if the panning detection276is false.

Additionally, local motion vectors240may be used to assist the auto focus module248for object tracking In other words, the size, shape, and position of a focus window may be adjusted such that the object of interest is brought into focus. This may occur for object tracking with moving backgrounds or close-to-still backgrounds.

FIG. 15is a block diagram illustrating various components that may be utilized in a computing device/electronic device1502. The computing device/electronic device1502may implement a video capture device102. Computing devices/electronic devices1502may include the broad range of digital computers including microcontrollers, hand-held computers, personal computers, servers, mainframes, supercomputers, minicomputers, workstations, and any variation or related device thereof.

The computing device/electronic device1502is shown with a processor1501and memory1503. The processor1501may control the operation of the computing device/electronic device1502and may be embodied as a microprocessor, a microcontroller, a digital signal processor (DSP) or other device known in the art. The processor1501typically performs logical and arithmetic operations based on program instructions1504stored within the memory1503. The instructions1504in the memory1503may be executable to implement the methods described herein.

The computing device/electronic device1502may also include one or more communication interfaces1507and/or network interfaces1513for communicating with other computing/electronic devices. The communication interface(s)1507and the network interface(s)1513may be based on wired communication technology, wireless communication technology, or both.

The computing device/electronic device1502may also include one or more input devices1509and one or more output devices1511. The input devices1509and output devices1511may facilitate user input. Other components1515may also be provided as part of the computing device/electronic device1502.

Data1506and instructions1504may be stored in the memory1503. The processor1501may load and execute instructions1504from the memory1503to implement various functions. Executing the instructions1504may involve the use of the data1506that is stored in the memory1503. The instructions1504are executable to implement one or more of the processes or configurations shown herein, and the data1506may include one or more of the various pieces of data described herein. When the processor1501executes the instructions1504, various portions of the instructions1504amay be loaded onto the processor1501, and various pieces of data1506amay be loaded onto the processor1501.

The memory1503may be any electronic component capable of storing electronic information. The memory1503may be embodied as random access memory (RAM), read only memory (ROM), magnetic disk storage media, optical storage media, flash memory devices in RAM, on-board memory included with the processor, EPROM memory, EEPROM memory, an ASIC (Application Specific Integrated Circuit), registers, and so forth, including combinations thereof.

Alternatively, or in addition to, there may be more than one processor1501a, which may operate in parallel to load instructions1504band data1506band execute the instructions1504busing the data1506b. These instructions1504bmay include performing auto white balance, auto exposure control, and/or auto focusing.