Motion detection based on observing several pictures

A method for motion detection based on observing several pictures is disclosed. Step (A) may compute a first motion score of an area in a target picture by a comparison of the area between the target picture and a first reference picture. Step (B) may compute a second motion score of the area by another comparison of the area between the target picture or a second reference picture and a third reference picture. Step (C) may temporal filter the target picture with the first reference picture based on the first motion score and the second motion score. At least one of the computing of the first motion score, the computing of the second motion score, and the temporal filtering may be controlled by one or more gain settings in a circuit. At least two of the first, the second, and the third reference pictures may be different pictures.

This application relates to U.S. Provisional Application No. 62/097,663, filed Dec. 30, 2014, which is hereby incorporated by reference in its entirety.

FIELD OF THE INVENTION

The present invention relates to motion detection for video temporal filtering generally and, more particularly, to methods and/or apparatus for motion detection based on observing several pictures.

BACKGROUND OF THE INVENTION

Conventional motion detection looks at a local error measure, commonly a sum-of-absolute-differences, between a target picture and a reference picture. Even if no motion exists, such local error measures tend to be non-zero due to noise and changes in scene lightness. Therefore, motion detection commonly detects small differences between the pictures as no motion and detects big differences as motion. Temporal filtering is used to combine a target picture with a motion compensated reference picture, and uses strong filtering where no motion is detected.

It would be desirable to implement motion detection based on observing several pictures.

SUMMARY OF THE INVENTION

The present invention concerns a method for motion detection based on observing several pictures. Step (A) may compute a first motion score of an area in a target picture by a comparison of the area between the target picture and a first reference picture. Step (B) may compute a second motion score of the area by another comparison of the area between the target picture or a second reference picture and a third reference picture. Step (C) may temporal filter the target picture with the first reference picture based on the first motion score and the second motion score. At least one of the computing of the first motion score, the computing of the second motion score, and the temporal filtering may be controlled by one or more gain settings in a circuit. At least two of the first, the second, and the third reference pictures may be different pictures.

The objects, features and advantages of the present invention include providing motion detection based on observing several pictures that may (i) use motion detection between two different pairs of pictures to determine how to apply a temporal filter between a pair of the pictures, (ii) use motion detection between non-adjacent pictures to determine how to temporal filter between adjacent pictures, and/or (iii) detect motion based on motion between several picture pairs.

DETAILED DESCRIPTION OF EMBODIMENTS

Motion detection may be used in many applications, such as security cameras, and/or in many operations, such as motion compensated temporal filtering (e.g., MCTF) a sequence of pictures (or images). For the motion compensated temporal filtering, a filter may adaptively combine one or more reference (or previous) pictures and a target (or current) picture of the sequence based on detected motion in the target picture relative to the reference pictures. The filtering may also decide locally how to combine the multiple pictures (e.g., fields and/or frames) to reduce noise while limiting filter-created artifacts.

Typically, the filter may favor a reference picture more the more the filter determines that no motion exists in a local area relative to the reference picture. For such a filter, motion may mean motion in an absolute sense, if motion exists. In various embodiments, the reference pictures may be pre-transformed per a motion model (e.g., a process used to estimate motion between the pictures). The transformed (motion compensated) reference pictures may be subsequently combined with the target picture. For a motion compensated temporal filtering case, motion generally means motion between the motion compensated reference pictures and the target picture. For a non-motion compensated temporal filtering case, motion generally means motion between the non-compensated reference pictures and the target picture.

Referring toFIG. 1, a block diagram of a camera system100is shown illustrating an example implementation of a camera/recorder system (or apparatus). In some embodiments, the camera system100may be a digital video camera, a digital still camera or a hybrid digital video/still camera. In an example, the electronics of the camera system100may be implemented as one or more integrated circuits. For example, an application specific integrated circuit (e.g., ASIC) or system-on-a-chip (e.g., SOC) may be used to implement a processing portion of the camera system100. In various embodiments, the camera system100may comprise a camera chip (or circuit)102, a lens assembly104, an image sensor106, an audio codec108, dynamic random access memory (e.g., DRAM)110, non-volatile memory (e.g., NAND flash memory, etc.)112, one or more serial interfaces114, an interface116for connecting to or acting as a universal serial bus (e.g., USB) host, an interface for connecting to a removable media118(e.g., SD—secure digital media, SDXC—secure digital extended capacity media, etc.), a wireless interface120for communicating with a portable user device, a microphone122for recording audio, and a speaker124for playing audio. In some embodiments, the lens assembly104and the image sensor106may be part of a separate camera connected to the processing portion of the system100(e.g., via a video cable, a high definition media interface (e.g., HDMI) cable, a USB cable, an ethernet cable, or wireless link).

In various embodiments, the circuit102may comprise a number of modules (or circuits) including, but not limited to, a pulse width modulation (e.g., PWM) module, a real time clock and watchdog timer (RTC/WDT), a direct memory access (e.g., DMA) engine, a high-definition multimedia interface (e.g., HDMI), an LCD/TV/Parallel interface, a general purpose input/output (e.g., GPIO) and an analog-to-digital converter (e.g., ADC) module, an infrared (e.g., IR) remote interface, a secure digital input output (e.g., SDIO) interface module, a secure digital (e.g., SD) card interface, an audio inter-IC sound (e.g., I2S) interface, an image sensor input interface, and a synchronous data communications interface (e.g., IDC SPI/SSI). The circuit102may also include an embedded processor (e.g., ARM, etc.), an image digital signal processor (e.g., DSP), and a video and/or audio DSP. In embodiments incorporating the lens assembly104and image sensor106in the system100, the circuit102may be configured (e.g., programmed) to control the lens assembly104and receive image data from the sensor106. The wireless interface120may include support for wireless communication by one or more wireless protocols such as Bluetooth®, ZigBee®, Institute of Electrical and Electronics Engineering (e.g., IEEE) 802.11, IEEE 802.15, IEEE 802.15.1, IEEE 802.15.2, IEEE 802.15.3, IEEE 802.15.4, IEEE 802.15.5, and/or IEEE 802.20. The circuit102may also include support for communicating using one or more of the universal serial bus protocols (e.g., USB 1.0, 2.0, 3.0, etc.). The circuit102may also be configured to be powered via the USB connection. However, other communication and/or power interfaces may be implemented accordingly to meet the design criteria of a particular implementation.

In various embodiments, programming code (e.g., executable instructions for controlling various processors of the circuit102) implementing a temporal filter with noise-robust and/or slow-motion robust motion detection may be stored in one or more of the memories110and112. When executed by the circuit102, the programming code generally causes the circuit102to receive a sequence of pictures from the sensor106, temporal filter based on measurements if an area is stationary for several pictures, temporal filtering based on motion detection on small and big areas, temporal filter based on comparing down-sampled pictures, and/or temporal filtering of adjacent pictures based on motion detection of non-adjacent pictures.

For noisy image sequences, the differences between pictures, even in stationary areas, may be large since the noise in each picture is different. Moreover, slow motion tends to add only small amounts to motion scores. Therefore, conventional motion detection may fail to correctly detect slow motion and/or motion in noisy sequences of pictures. False positives (e.g., detecting motion where none exists) may result in too-noisy output pictures. False negatives (e.g., not detecting actual motion) may result in temporal artifacts. Various embodiments of the present invention generally contain one or more of the following features that may be used individually or in combination to make temporal filtering based on motion compensation more robust.

Motion detection may be based on observing if the video is stationary or moving for several pictures (or frames or fields). Specifically, for the same location, scores are generally used from multiple picture comparisons. By incorporating extra data into the still or moving decision, the detection may be more robust.

Temporal filtering of adjacent pictures may be based on motion detection of non-adjacent pictures. Adjacent pictures may be combined with a temporal filtering because adjacent pictures are generally more similar to each other than non-adjacent pictures. For slow motion, non-adjacent pictures may exhibit greater motion and, therefore, may exhibit higher motion scores than adjacent pictures. Performing detection on non-adjacent pictures (e.g., a target picture and a non-adjacent reference picture) may provide a more robust detection of slow motion, especially in the presence of noise.

Referring toFIG. 2, a graphical representation140of several motion detections is shown. Consider a sequence of several frames N to N−3 (e.g., reference numbers142to148). A motion detection150generally detects motion between a target frame N (142) and a reference frame N−1 (144). Another motion detection may be used to seek earlier motion. A motion detection152generally detects motion between the reference frame N−1 (144) and a reference frame N−2 (146). Still another motion detection154may detect motion between the reference frame N−2 (146) and a reference frame N−3 (148). Up to all of the detections150,152, and154may be used to filter (156) samples between an area in the target frame N (142) and the area the reference frame N−1 (144). While the example uses two earlier detections (e.g., the detection152and the detection154), any number of detections greater than a single detection may be used. The area may range from a single pixel to many pixels (e.g., 4×4, 8×8, 16×16, 32×32, or 64×64 blocks of pixels).

Referring toFIG. 3, a flow diagram of a motion detection method160is shown in accordance with a preferred embodiment of the present invention. The method (or process)160may be performed by the circuit102. The method160generally comprises a step (or state)162, a step (or state)164, a step (or state)166, a step (or state)168, a step (or state)170, a decision step (or state)172, a step (or state)174, a step (or state)176, a decision step (or state)178, and a step (or state)180. The steps162-180may be implemented in hardware, software, firmware or any combination thereof in an apparatus (or circuit or device). The sequence of the steps is shown as a representative example. Other step orders may be implemented to meet the criteria of a particular application.

In the step162, the circuit102may motion compensate one or more reference pictures (e.g., the frames N−1, N−2, N−3, etc.). The circuit102may compare an area of the target picture (e.g., the frame N) to a spatially co-located area of a reference picture A (e.g., the frame N−1) in the step164to generate a raw score A (e.g., a target motion score). In some embodiments, the reference picture A may not be temporally adjacent to the target picture N (e.g., the reference picture A may be the frame N−2). In other embodiments, the reference picture A may be temporally adjacent to the target picture N (e.g., the reference picture A may be the frame N−1). In the step166, the area of the reference picture A may be compared with the spatially co-located area of another reference picture B (e.g., the frame N−2) to generate another raw score B (e.g., an additional motion score). The area of the reference picture B may be compared in the step168to the spatially co-located area of a reference picture C (e.g., the frame N−3) to generate a raw score C (e.g., another motion score). The circuit102may combine two or three of the three raw scores A, B and/or C in the step170to generate a combined score. The decision step172generally determines if additional detections may be useful in one or more additional areas. If the additional detections may be useful, the steps164-170may be repeated.

In the step174, the circuit102may use the combined score and a gain value, applied by the circuits102and/or106, to temporal filter a target sample in the area of the target picture N with another reference picture E. The reference picture E (e.g., frame N−1 or N+1) may be temporally adjacent to the target picture N. In the step176, the filtered target sample may be stored in one or more of the memories (e.g., the memory110).

A check may be performed in the decision step178to determine if any more target samples exist in the current target picture N. If more target samples have yet to be processed, the method160may move to the next unprocessed target sample and return to the temporal filter process (e.g., the step174). Once all of the target samples in the current target picture N have been processed, the method160may continue in the step180with the target samples in the next picture.

The gain settings in the camera system100may include an analog gain and/or a digital gain in the image sensor106, and/or a digital gain in the circuit102. One or more of such settings may be considered in the temporal filtering. Furthermore, offset settings, exposure settings and/or aperture settings may also be considered in the temporal filtering. The circuit102generally controls the lens assembly104and/or the image sensor106for an automatic exposure operation. Changes in the automatic exposure may change the light levels in the image data received from the sensor106. The gain settings affect the noise in pictures; therefore, any of the steps computing the various scores (e.g., the steps164,166and/or168), combining the scores (e.g., the step170), and/or using the scores for temporal filtering (e.g., the step174) may be controlled based on the gain settings, offset settings, exposure settings and/or aperture settings.

The scores computed in the steps164,166and/or168may be any score that is generally higher when motion exists between pictures. The scores may include, but are not limited to, sum-of-absolute-differences and sum-of-squared-differences. The scores may further be modified based on tone (e.g., brightness and/or color) as described in co-pending U.S. patent application Ser. No. 14/580,867, filed Dec. 23, 2014, which is hereby incorporated by reference in its entirety.

The steps164-168generally show three picture comparisons. In general, more or fewer picture comparisons may be implemented to meet the criteria of a particular application. The combining operations may use lookup tables and/or mathematical transformations to generate the combined motion scores. The step170generally shows combining two or more scores from different pictures.FIGS. 5 and 6may illustrate embodiments of various combination operations. Other comparisons between the target frame N (142) and the reference frames may be implemented.

Referring toFIG. 4, a graphical representation190of several motion detections is shown. Consider a sequence of multiple frames N to N−4 (e.g., reference numbers142to149). As in the representation140, the motion detection150generally detects motion between the target frame N (142) and the reference frame N−1 (144). The detected motion may establish (e.g., the step164inFIG. 3) the raw score A. Another motion detection192may detect motion between the target frame N (142) and the reference frame N−2 (146) to calculate the raw score B. The motion detection192may be a variation of the step166. In various embodiments, a motion detection194may detect motion between the target frame N (142) and the reference frame N−4 (149) to calculate the raw score C. The motion detection194may be a variation of the step168. In some embodiments, the motion detection194may be between the target frame N (142) and the reference frame N−3 (148). In other embodiments, the motion detection194may be between two of the reference frames (e.g., between the reference frame N−3 and the reference frame N−4).

The step170may combine two or three of the raw scores A, B and/or C to calculate the combined score. The circuit102may use the combined score and the gain value in the step174to temporal filter a target sample in the area of the target picture N with the reference picture E. The reference picture E (e.g., frame N−1 or N+1) may be temporally adjacent to the target picture N. In the step176, the filtered target sample may be stored in one or more of the memories (e.g., the memory110). Thereafter, additional target samples and additional target pictures may be filtered.

Referring toFIG. 5, a diagram of an example score combination200by lookup table is shown. A signal202may carry scores from two or more frames to a multi-dimensional lookup table (e.g., LUT)204. An entry (or value) stored in the LUT204at an index formed by the scores may be presented from the LUT204as a combined score in a signal206.

Referring toFIG. 6, a diagram of an example score combination circuit (or module)220using maximum selection and two-dimensional combining is shown. The scores that do not use the target frame N may be received via a signal222by a maximum circuit (or module)226. The scores that use the target frame N may be received by the maximum circuit226and a combine circuit (or module)230via a signal224. The circuit226is generally operational to select a maximum score (or value) among the received scores. The maximum score may be passed in a signal228to the circuit230. The circuit230is generally operational to perform a two-dimensional lookup or mathematical operations on the scores received in the signals224and228to generate and present a combined score in a signal232.

Various embodiments of the circuit230may implement a two-dimensional (e.g., a dimension for the signal224and another dimension for the signal228) lookup. Other embodiments of the circuit230generally select the highest score in the signal228. Some embodiments of the circuit230may transform the maximum score per formula 1 as follows:
Combined_score=((Max_score−SUB)×MUL)  (1)
Where a subtraction value SUB and a multiplication value MUL may be controllable parameters, and where a value Max_score may be the maximum score in the signal228. Still other embodiments may transform the maximum score with the score in the signal224as follows:

If(Max_score<THR) Combined_score=0;else {A=(CUR−SUB)×MUL)Combined_score=max(Min_score, A)}
Where a threshold THR, a minimum score Min_score, the subtraction value SUB and the multiplication value MUL may be controllable parameters. A current value CUR may be the score that uses the target picture N in the signal224. The temporal filtering may combine the target picture N and a reference picture using a blending curve.

Referring toFIG. 7, a diagram240of an example blending curve242is shown. A strength of the temporal filtering (or blending) may be a continuum for one or more filter strengths. The diagram240generally illustrates a range of medium filter strengths and fixed filter strengths. A degree of filtering may depend on the blending curve242.

An example of blending is generally determined as follows:

D=detected motion score; and

Alpha (curve242)=lookup of the value D.

A filtered result (sample) may be calculated by formula 2 as follows:
Result=(Alpha×T)+((1−Alpha)×R)  (2)
In the diagram240, the X axis generally represents the detected motion value D (e.g., the combined motion score of the target frame N). For 8-bit levels of detected motion, the X axis is generally labeled from 0 to 255. The Y axis generally represents an alpha value and ranges from 0 (zero) to 1 (unity). Other ranges of D and alpha may be implemented to meet the criteria of a particular application. Other techniques for determining the value D may also be implemented, such as considering several target samples simultaneously.

Small detected motion values D may be illustrated in the section244. The section244generally results in a low value of alpha per the blending curve242. Medium (or intermediate) detected motion values D may be illustrated in the section246. The section246generally results in a range of values for alpha per the blending curve242. Large detected motion values of D may be illustrated in the section248. The section248generally results in a high value of alpha per the blending curve242.

Where slow or no motion is detected, the value D is small and in the section244. Therefore, the value alpha may be small (and optionally a fixed value). Per formula 2, the small value alpha generally weights the blending to favor the reference sample, or in some cases (e.g., alpha=0.5) averages the reference sample with the target sample. Such blending may be considered a strong filtering. Where medium motion is detected, the value D may be medium and in the section246. Thus, the value alpha may be medium. Per formula 2, the medium value alpha variably weights the blending between the target sample and the reference sample, depending on the level of motion. Such blending may be considered a medium filtering. Where fast motion is detected, the value D may be large and in the section248. Therefore, the value alpha may be large and weights the blending to favor the target sample. Such blending is generally considered a weak filtering. Where the value alpha=1, no filtering is accomplished and the target sample is unchanged.

In various embodiments, the blending curve242may be implemented as one or more LUTs. For example, a single LUT (e.g., LUT204) may store all points of the blending curve242. The value D may be implemented as the combined score value.

In other embodiments, different LUTs may store different blending curves and/or different portions of one or more blending curves. Selection of a particular LUT is generally based on the combined score value. For example, if the combined score is zero, an LUT number 0 may be utilized. If the combined score is greater than zero and less than a threshold T1, an LUT number 1 may be utilized. If the combined score is greater than the threshold T1and less than a threshold T2, an LUT number 2 may be utilized. If the combined score is greater than the threshold T2, an LUT number 3 is generally utilized. Other numbers of LUTs may be implemented to meet the criteria of a particular application.

In some embodiments, the combined score may be a lookup table number. The number of LUTs may be clamped per formula 3 as follows to a maximum value to avoid having too many LUTs:
Table=min(combined score,number of tables−1)  (3)

In various embodiments, the combined score may be used to scale the value D received by the curve242or the LUT204. The scaling may be implemented per formula 4 as follows:
D_used=D_before_multiplication×combined score  (4)

In other embodiments, the combined score may be used to offset the value D received by the curve242or the LUT204. The offsetting may be implemented per formula 5 as follows:
D_used=D_before_offset+combined score  (5)

The functions and structures illustrated in the diagrams ofFIGS. 1-7may be designed, modeled and simulated using one or more of a conventional general purpose processor, digital computer, microprocessor, microcontroller and/or similar computational machines, programmed according to the teachings of the present specification, as will be apparent to those skilled in the relevant art(s). Appropriate software, firmware, coding, routines, instructions, opcodes, microcode, and/or program modules may readily be prepared by skilled programmers based on the teachings of the present disclosure, as will also be apparent to those skilled in the relevant art(s). The software is generally embodied in a medium or several media, for example a non-transitory storage media, and may be executed by one or more of the processors. As used herein, the term “simultaneously” is meant to describe events that share some common time period but the term is not meant to be limited to events that begin at the same point in time, end at the same point in time, or have the same duration.