Patent ID: 12207024

DETAILED DESCRIPTION

Please refer toFIG.2.FIG.2is a block diagram of a motion detection device60according to an embodiment of the present invention. The motion detection device60can be matched with a passive detector62and an external host64to provide preferred smart motion detecting function. The motion detection device60is electrically connected between the passive detector62and the external host64. The passive detector62is used to detect if a specific situation happened, such as a living thing passed by or a door opened, so as to trigger the motion detection device60to analyze if a true event of the specific situation existed, which means the living thing detected by the passive detector62is identified as an expected object. When the true event is determined, the motion detection device60transmits related data for the external host64to determine a security alarm.

In one embodiment, the passive detector62can be a temperature detector, such as an infrared detector, and the motion detection device60can be selectively operated in a sleep mode or a wakeup mode. While a monitoring region is in an usual state, the passive detector62does not detect temperature variation, the motion detection device60stays in a sleep mode; while in an unusual state that the specific situation happens (such like the living thing passed by), the passive detector62detects the temperature variation and generates a triggering signal to switch the motion detection device60from the sleep mode to a wakeup mode.

The motion detection device60can include an image capturing unit66, an operating processor68, a memory70and a lighting unit72. The operating processor68can drive the image capturing unit66to alternatively switch between the sleep mode and the wakeup mode, and further can drive the image capturing unit66to optionally capture monitoring images with low quality and high quality. In one embodiment, the lighting unit72can be actuated only while the image capturing unit66captures image, so as to enhance the image capturing unit66capturing images in a power efficiency manner.

The image capturing unit66may capture a background monitoring image with a low frame rate in the sleep mode, and capture a plurality of monitoring images with a high frame rate in the wakeup mode. The background image could be captured in low quality, wherein the background image is used for auto-exposure control of the image capturing unit66. The monitoring images could comprise a first monitoring image with the low quality and a second monitoring image with the high quality, wherein the first monitoring image is used for the operating processor68to identify if the true event is happened and the second monitoring image is used for the external host64to determine the security alarm. The monitoring images captured by the image capturing unit66can be stored inside the memory70, and further the high quality monitoring image can be transmitted to the external host64.

In this embodiment, the monitoring system utilizes the passive detector62to detect the object passing through the monitoring region for a start, and then utilizes the motion detection device60to analyze whether the passed object conforms to a predetermined condition (i.e., true event). As a view range of the passive detector62has passerby and the specific situation is identified, the motion detection device60is switched to the wakeup mode by the passive detector62and determines whether the passerby is the expected object (which means the human); if so, the motion detection device60actuates the external host64, and the external host64identifies the object within the monitoring images to optionally drive the motion detection device60in a recording mode, to transmit monitoring video, to send out a warning, to shut down the motion detection device60and to awaken another motion detection device60′ electrically connected with the external host64.

Please refer toFIG.3.FIG.3is a flow chart of a motion detection method applied to the motion detection device60according to the embodiment of the present invention. First, step S200and S202are executed to startup the monitoring system, and the passive detector62is utilized to detect the object within the view range. If the passive detector62does not detect the temperature variation, step S204is executed to keep the image capturing unit66in the sleep mode; if the passive detector62detects the temperature variation, step S206is executed that the passive detector62transmits the triggering signal to switch the image capturing unit66from the sleep mode to the wakeup mode. Then, step S208and step S210are executed, the lighting unit72can be actuated in accordance with surrounding illumination and the image capturing unit66captures the first monitoring image (with the low quality), the operating processor68simply analyzes the first monitoring image for determining whether to actuate the external host64.

In one embodiment, the image capturing unit66captures the low quality monitoring image by using partial pixels, such as to divide the pixel array into a plurality of 2×2 pixel blocks and to use only one pixel in each pixel block to capture the image. In another embodiment, the image capturing unit66captures image by all pixels and divides all pixels to several pixel block (such as 2×2 pixel block) so as to combine values in all pixels in each pixel block as a block value and generates the low quality monitoring image by those block values.

In step S210, the operating processor68preferably analyzes a specific region of interest (ROI) within the first monitoring image to determine actuation of the external host64, dimensions of the specific region is smaller than the first monitoring image, so that the operating processor68can rapidly acquire an image analysis result due to less data calculation in ROI; the first monitoring image setting as a low quality monitoring image is helpful to speed up image analysis about the specific region of interest. Position and dimensions of ROI are preferably predefined by the user, for example, a door and a window are situated in the first monitoring image, ROI can cover the pattern of the door to avoid the analysis result from being interfered by left shadow on the window, or ROI can cover edges of window for detecting a thief climbing into the window and also preventing the analysis result from being interfered by the left shadow; the position and dimensions of ROI further may be varied according to the analysis result. However, the operating processor68may analyze an entire region within the first monitoring image to perform the step S210, which depends on design demand. The said image analysis can be executed by identifying a pattern contour within the monitoring image, comparing feature point on the monitoring image, and analyzing intensity variation of the monitoring image optionally.

As the object does not conform to the predetermined condition, suck like the passerby within the monitoring image being the animal instead of the human, step S212is executed to not actuate the external host64, and the image capturing unit66may be passively or actively shut down to back the sleep mode. As the object conforms to the predetermined condition, which means the passerby within the monitoring image may be an unauthorized person, step S214is executed to actuate the external host64and the image capturing unit66starts to capture the second monitoring image with the high quality, and the second monitoring image can be captured as static images format or a continuing video format and can be stored inside the memory70. Next, step S216is executed that the external host64receives the second monitoring image and utilizes image recognition algorithm to precisely analyze the object within the second monitoring image.

The second monitoring image does not conform to a predetermined threshold, which means the object is not the unauthorized person, step S218is executed to shut down the motion detection device60passively or actively for energy economy; the second monitoring image conforms to the predetermined threshold, hence the object is defined as the unauthorized person, step S220is executed that the external host64can switch the motion detection device60into the recording mode, the motion detection device60transmits the monitoring video outwardly for backup, and the other motion detection devices60′ can be simultaneously awaken for overall monitoring. Therefore, the passive detector62cannot directly actuate the external host64while detecting the object, the motion detection device60wakes up by trigger of the passive detector62to capture the first monitoring image, and the external host64is actuated in accordance with the low quality image analysis of the first monitoring image through the motion detection device60.

The motion detection device60can begin to capture the second monitoring image while the external host64is actuated. The external host64has to spend a period of time on waking other motion detection devices, the second monitoring image can record any doubtful event inside the monitoring region before the other motion detection devices are awaken, which means the monitoring system does not miss the doubtful event in a term between a detection timing of the passive detector62and a wakeup timing of the other motion detection devices. The first monitoring image with the low quality is used by the motion detection device60to determine existence of the object, the existence determination is rough computation and may be affected by noise, and the second monitoring image with the high quality is used by the external host64to analyze the accurate motion detection of the object, such as face recognition.

The present invention further provides an exposure adjustment function for preferred operational efficiency of the motion detection device60. Please refer toFIG.4andFIG.5.FIG.4is a flow chart of a motion detection method applied to the motion detection device60according to another embodiment of the present invention.FIG.5is a waveform diagram of a frame rate executed by the image capturing unit66according to the foresaid embodiment of the present invention. In the embodiment, steps having the same numeral as one of the above-mentioned embodiment have the same content, and a detailed description is omitted herein for simplicity. As the motion detection device60is not awaken by the passive detector62, step S205can be executed to periodically switch the image capturing unit66to the wakeup mode in the low frame rate, and the image capturing unit66in the wakeup mode can execute the exposure adjustment and capture a low quality background image. As the motion detection device60is awaken, step S207is executed to transform the image capturing unit66into the wakeup mode in the high frame rate, and later, the image capturing unit66still captures the monitoring image with the low quality to compare with the background image for determining actuation of the external host64.

For example, as shown inFIG.5, the image capturing unit66may execute the exposure adjustment and capture the background image one frame per second (1 fps) while the motion detection device60is not triggered by the passive detector62, which means an exposure parameter of the image capturing unit66can be adjusted and the background image can be established at timing T1, T2, T3and T4. While the passive detector62triggers the motion detection device60into the wakeup mode at timing T5, the motion detection device60may capture the first monitoring images thirty frames per second (30 fps), the latest background image (captured at the timing T4) has the exposure parameter similar to ones of the first monitoring image captured at the timing T5, so that the image capturing unit66in the wakeup mode is not in need of the exposure adjustment, and can immediately acquire the superior monitoring image with suitable exposure parameters.

In conclusion, the motion detection device of the present invention is electrically connected between the passive detector and the external host, and the motion detection device is utilized to actuate the external host while the passive detector triggers the motion detection device switched from the sleep mode to the wakeup mode. As the motion detection device is in the sleep mode, the motion detection device can be awaken in the low frame rate or stay in the sleep mode to adjust the exposure parameter and to capture the background image; as the motion detection device is switched to the wakeup mode, the motion detection device is transformed into the high frame rate to capture the low quality monitoring image. The motion detection device executes the simple image analysis via ROI of the low quality monitoring image for a start for determining whether to actuate the external host; since the motion detection device actuates the external host, the motion detection device captures and stores the high quality monitoring image, and the high quality monitoring image can be used by the external host for the accurate image analysis and execution of related application programs. The motion detection device of the present invention can effectively economize start-up time of the monitoring system without waiting for a wakeup period of the external host and an exposure adjustment period of the motion detection device.

Please refer toFIG.6andFIG.7.FIG.6is a functional block diagram of a smart motion detection device80according to a first embodiment of the present invention.FIG.7is a procedural diagram of the smart motion detection device80according to the first embodiment of the present invention. The smart motion detection device80can include a memory82, a processor84and a sensor array86, which are three separate components or combined as one or two integrated components. The sensor array86can be directly coupled to the memory82and further electrically connected with the processor84. The sensor array86includes a plurality of light detecting pixels arranged in two-dimension manner to capture images. The processor84can be switched between a sleep mode and a wakeup mode, and used to process an image captured by the sensor array86to identify a particular event in the captured images, such as an unexpected object been captured in the captured images.

The image captured by the sensor array86may be pre-stored (i.e., wrote) into the memory82or directly transmitted to the processor84in accordance with modes of the processor84or an alarm signal resulted from motion detection. The memory82can have the image capacity of predefined quantity; when the memory82is full and a new image is prepared to pre-store, a former image can be removed for storing the new image. The image processed by the processor84and the pre-stored image in the memory82can be transmitted to an external storage module88electrically connected with the smart motion detection device80.

As the first embodiment shown inFIG.7, the processor84stays in the sleep mode when the smart motion detection device80is not activated. The sensor array86can include a comparator90adapted to generate the alarm signal when monitoring motion of an object. As the processor84is operated in the sleep mode, the sensor array86continuously or intermittently captures a plurality of images (such as capture five images in every 1 second), and the plurality of images are pre-stored into the memory82; in the meantime, the comparator90reads and compares at least some of the pre-stored images I1with a reference image. When intensity variation between one of the pre-stored images I1and the reference image is smaller than a predefined value, the processor84keeps in the sleep mode and the comparator90reads the next pre-stored image I1for a comparison with the reference image. When the intensity variation is greater than the predefined value, the comparator90can generate the alarm signal utilized to awake the processor84and further to pre-store the image captured by the sensor array86into the memory82. That is, the alarm signal is used to switch the processor84from the sleep mode to the wakeup mode.

There has variety ways for the comparator90to compare the pre-stored images I1and the reference image, for example the comparator90could compare whole image range or only compare partial pixels for the pre-stored images I1and the reference image. The comparator90could compare intensity summation of all pixels or partial pixels, in another way the comparator90could compare intensity of each pixel in whole image range or only partial pixels.

When the processor84is operated in the wakeup mode, a real-time image I2captured by the sensor array86is directly transmitted to the processor84for digital processing and may not be stored into the memory82. The processor84in the wakeup mode may process the real-time image I2and receive the pre-stored image I1in the memory82by turns, or may receive the pre-stored image I1after processing of the real-time image I2. A process of the real-time image I2can precede that of the pre-stored image I1, so the smart motion detection device80is able to focus on an instant situation within the monitoring area. The process of the pre-stored image I1may be executed when the process of the real-time image I2is completed or paused. If an operating capability of the processor84is sufficient for mass data, the real-time image I2and the pre-stored image I1can be processed alternately, hence the smart motion detection device80can show detection results about the current and previous period at the same time.

In some embodiments, the pre-stored images captured by the sensor array86when the processor84is operated in the sleep mode can be pre-stored into the memory82, and the real-time images captured by the sensor array86when the processor84is operated in the wakeup mode can be transmitted to the processor84. In other embodiments, the processor84and the sensor array86can be turned off under a non-working mode; when the smart motion detection device80receives a trigger signal, the sensor array86can capture and send the images to the memory82directly, and then the processor84can send a request to the sensor array86for receiving the captured images. The trigger signal may be an alarm resulted from an external unit or a built-in unit of the smart motion detection device80.

In addition, at least one of an image quality and a frame rate of the sensor array86may be changed when the processor84is operated in the sleep mode or the wakeup mode. For example, as the processor84is in the sleep mode, the sensor array86can capture the low-quality image or capture the image in the low frame rate for comparing with the reference image. Transmission bandwidth and storage capability are economized accordingly. The alarm signal is generated because the intensity variation between the low-quality image (or the image captured in the low frame rate) and the reference image is greater than the predefined value, so that the sensor array86starts to capture the high-quality image or capture the image in the high frame rate for pre-storing into the memory82, and simultaneously the processor84can be switched to the wakeup mode. Then, the pre-stored high-quality image or the pre-stored image captured in the high frame rate in the memory82is transmitted to the processor84operated in the wakeup mode; therefore the smart motion detection device80does not lose image information before the processor84is in the wakeup mode.

Please refer toFIG.8toFIG.11.FIG.8is a functional diagram of the smart motion detection device80′ according to a second embodiment of the present invention.FIG.9is a procedural diagram of the smart motion detection device80′ according to the second embodiment of the present invention.FIG.10is a functional diagram of the smart motion detection device80″ according to a third embodiment of the present invention.FIG.11is a procedural diagram of the smart motion detection device80″ according to the third embodiment of the present invention. In the embodiments, elements having the same numerals as ones of the first embodiment have the same functions, and a detailed description is omitted herein for simplicity.

In a possible embodiment, the smart motion detection device80′ can include the memory82, the processor84, the sensor array86′ and a passive sensor92. The passive sensor92is electrically connected with the processor84and the sensor array86′. The processor84is kept in the sleep mode and the sensor array86′ is shut down when the passive sensor92does not detect any abnormal situation. As the passive sensor92detects the motion of the object, the passive sensor92can generate the alarm signal, and the alarm signal is used to drive the sensor array86′ and switch the processor84from the sleep mode to the wakeup mode. When the processor84is still in the sleep mode, the sensor array86′ can capture and transmit the pre-stored image I1to the memory82. When the processor84is operated in the wakeup mode, the sensor array86′ can capture and transmit the real-time image I2to the processor84, and the pre-stored image I1in the memory82can be transmitted to the processor84accordingly.

The smart motion detection device80may have the non-working mode. The processor84and the sensor array86′ can be turned off under the non-working mode. As the passive sensor92detects the motion of the object and generates the alarm signal, the sensor array86′ is triggered by the alarm signal and starts to capture and send the pre-stored image into the memory82. After that, the processor84can be switched to the wakeup mode and then sends the request to the sensor array86′ for receiving the pre-stored image.

In another possible embodiment, the smart motion detection device80″ can include the memory82, the processor84, the sensor array86″ having the comparator90, and the passive sensor92. The passive sensor92can activate the sensor array86″ when detecting the abnormal situation. The activated sensor array86″ can capture and transmit the pre-stored image I1to the memory82, and the comparator90can compare the pre-stored image I1with the reference image for determining whether to switch on the processor84. The comparator90is utilized to identify the abnormal situation. If the intensity variation between the pre-stored image I1and the reference image is smaller than the predefined value, the abnormal situation may be resulted from noise and the processor84is not switched on. If the intensity variation is greater than the predefined value, the abnormal situation can be defined as someone or something intruding into the monitoring area of the smart motion detection device, so that the processor84is switched to the wakeup mode for recording. As the processor84is operated in the wakeup mode, the real-time image I2captured by the sensor array86″ and the pre-stored image I1in the memory82can be transmitted to the processor84and then to the external storage module88for the digital processing.

Please refer toFIG.12.FIG.12is a flow char of a determining method according to the embodiment of the present invention. The determining method illustrated inFIG.12can be suitable for the smart motion detection devices shown inFIG.6toFIG.11. First, steps S800and S802are executed to start the determining method and to monitor the motion of the object. The said monitoring function can be applied by the sensor array86,86′ and86″ or the passive sensor92. As there is not abnormal situation, step S804is executed to keep the processor84in the sleep mode; as the motion of the object is detected, steps S806and S808are executed to generate the alarm signal for enabling the processor84and capturing the image via the sensor array86,86′ and86″. When the processor84is not operated in the wakeup mode, step S810is executed that the sensor array86,86′ or86″ can produce the pre-stored image I1in the memory82. When the processor84is operated in the wakeup mode, steps S812and S814are executed that the sensor array86,86′ or86″ can produce the real-time image I2, and both the pre-stored image I1and the real-time image I2can be transmitted to the processor84.

After that, step S816is executed that the processor84can analyze the real-time image I2captured by the sensor array86,86′ or86″ when capturing function of the sensor array86,86′ or86″ is activated. When the sensor array86,86′ or86″ is not activated, probably owning to the disappeared object or any other situations, step S818is executed to analyze the pre-stored image I1inside the memory82by the processor84. It should be mentioned that the processor84not only can process the real-time image I2before the pre-stored image I1, but also alternately process the pre-stored image I1and real-time image I2in accordance with the user's actual demand and the sufficient operating capability.

In conclusion, the alarm signal may be generated by the sensor array or the passive sensor (which can be a thermal sensor, an accelerometer or a gyro). The alarm signal is utilized to activate pre-storing operation of the sensor array and mode switching operation of the processor. When the alarm signal is received, the sensor array can be activated to capture the pre-stored image at a first time and the pre-stored image is transmitted to the memory. For waiting a duration of the processor switched from the sleep mode to the wakeup mode, the processor which receives the alarm signal can send a request to the sensor array for the real-time image and the pre-stored image at a second time later than the first time, so that the pre-stored image from the memory are processed later than the first time, and the real-time image is not stored into the memory but directly transmitted to the processor for the digital processing. Comparing to the prior art, the smart motion detection device and the related determining method of the present invention can effectively economize start-up time of the smart motion detection device without waiting for a wakeup period of the processor.

Please refer toFIG.13in conjunction withFIG.14.FIG.13is a block diagram of a monitoring system1300according to an embodiment of the invention.FIG.14is a diagram showing three different operation scenarios of the portions/components of monitoring system1300inFIG.13. As shown inFIG.13, the monitoring system1300comprises three portions, i.e. an analog integrated circuit (IC) portion/component, a digital IC portion/component, and a backend system device. In other embodiments, the analog IC portion/component and digital IC portion/component may be integrated and implemented as an image sensor apparatus1301, wherein the image sensor apparatus1301could be an integrated circuit (IC). That is, such image sensor apparatus1301can be divided into an analog portion (i.e. the analog IC portion/component) and a digital portion (i.e. the digital IC portion/component).

The analog IC portion/component is a collection of analog circuits/components and for example is or comprises an image sensor circuit such as an analog image sensor1305which comprises an event camera1306(or may be referred to as an event sensor) comprising at least one pixel unit (i.e. one or more pixel units such as pixels or sub-pixels) such as active pixel unit(s). It should be noted that in practice an event camera unit/circuit may comprise a pixel unit and be used to report changes in brightness as they occur and stay silent otherwise. That is, the event camera1306is used to detect whether brightness change(s) occur in one or more pixel units.

The digital IC portion/component is a collection of digital circuits/components and is or comprises a digital processing circuit comprising a first image buffer1311and a motion detector such as a smart motion detector (SMD)1312. When the event camera1306transmits the information of pixel image/value's change, the SMD1312can use such pixel-level information to generate frame data, i.e. a frame-level image, and it can detect whether motion occurs. If it is determined that a motion occurs, then the SMD1312can generate an alert signal to the backend system1315. Instead, if no motions occur, the SMD1312does not generate the alert signal. SMD1312can accurately detect whether actual motion occurs and filter out some undesired image fluctuations such as motion images of shaking leaves or shaking grasses (not limited).

When receiving an alert signal sent from the digital processing circuit1310, the backend system1315is arranged to receive the image streams (i.e. frames) from the digital processing circuit1310and for example to start performing a video recording operation. The backend system1315comprises a second image buffer1316and an external processor1317that is externally coupled to the above-mentioned image sensor apparatus, i.e. the analog IC portion/component and digital IC portion/component.

The startup speed/time of the event camera1306(or the analog image sensor circuit1305) is much faster than that of the SMD1312(or digital processing circuit1310) and also much faster than that of the external processor1317(or the backend system device1315). In one embodiment, if no pixel values change, only the event camera1306(or only the analog image sensor circuit1305) is powered on and the other circuits (i.e. the digital processing circuit1310and the backend system device1315) are powered off or left in a power saving mode (such as working under a low operating frequency), to save more power. Once the event camera1306(or the analog image sensor circuit1305) detects that a pixel value changes, the SMD1312is awakened by a trigger signal, sent from the analog image sensor circuit1305, to perform the above-mentioned motion detection. Only when the motion detection indicates that an actual motion occurs, the external processor1317in the backend system device1315is awakened by a trigger signal, sent from the digital processing circuit1310, to perform further image processing and/or the video recording operation. If the event camera1306includes a plurality of pixels, then the “no pixel values change” may refer to “the number of pixels that has pixel values change less than a first specific threshold,” and the “a pixel value changes” may refer to “a number of pixels that has a pixel value change exceed than a second specific threshold”.

If the pixel value changes before the SMD1312is completely wakened, the pixel value(s) sensed by the image sensor circuit1305is/are stored into the first image buffer1311, and then will be transmitted/transferred to the SMD1312once the SMD1312is completely awakened and able to receive the sensed pixel value(s). That is, when the event camera1306is going to send pixel value(s) to the digital processing circuit1310, the pixel value(s) will be stored into the first image buffer1311before the SMD1312is wakened and the first image buffer1311may store a plurality of pixel values from the event camera1306before the SMD1312is wakened. Then, similarly, when it is determined that the frame data formed by the sensed pixel value(s) is associated with an actual motion event before the external processor1317is completely wakened, the frame(s) or image stream(s) processed by the digital processing circuit1310is/are stored into the second image buffer1316, and then will be transmitted/transferred to the external processor1317once the external processor1317is completely wakened and able to receive the sensed frame(s) or image stream(s). Also, when the digital processing circuit1310is going to send frame(s) to the backend system device1315, the frame(s) will be stored into the second image buffer1316before the external processor1317is wakened and the second image buffer1316may store a plurality of frames from the digital processing circuit1310before the external processor1317is wakened. By using the above-mentioned mechanisms and first/second image buffer(s), more power can be saved as well as image lost can be avoided.

As shown inFIG.14, for example, the analog image sensor circuit1305is always awake (i.e. at an awake state or in a normal mode different from a power saving mode) when it is supplied with power, to continuously or periodically detect whether the pixel value(s) of one or more pixel units change. In a first operation scenario (but not limited), if it is determined that no pixel values change, the analog image sensor circuit1305does not send a trigger signal to wake up the digital processing circuit1310and the backend system device1315. Thus, the digital processing circuit1310and the backend system device1315are kept asleep (i.e. at an asleep state or in the power saving mode). In this situation, the analog image sensor circuit1305does not transmit pixel data to the digital processing circuit1310and backend system device1315.

In a second operation scenario (but not limited), if it is determined that at least one pixel value changed, then the analog image sensor circuit1305is arranged to send a trigger signal to wake up the digital processing circuit1310and also send the sensed pixel data to the digital processing circuit1310. In the situation that the startup speed/time of the first image buffer1311is much faster than that of SMD1312, the sensed pixel data can be temporarily stored in the first image buffer1311by using an image freeze operation before the SMD1312is completely awakened. After the digital processing circuit1310completely exits the power saving mode and enters the normal mode, the pixel data sensed by the analog image sensor circuit1305can be directly transmitted to the SMD1312without using the first image buffer1311. In this situation, if it is determined that no actual motions occur, then the digital processing circuit1310does not send an alert signal to wake up the backend system device1315and also does not transmit the generated frames or image streams to the backend system device1315.

It should be noted that the SMD1312can simultaneously receive and process both the stored pixel data of the first image buffer1311and the incoming pixel data once the SMD1312completely enters the normal mode. For example (not limited), the SMD1312can employ a super high frame per second (HFPS) image processing frequency/rate to rapidly process the accumulated pixel values stored in the first image buffer1311and then synchronize the processed pixel values with the analog values of incoming pixel data. For instance, each time when the analog image sensor circuit1305is called to sense pixel units, the SMD1312is arranged to process the accumulated pixel values stored in the first image buffer1311and process the analog values of incoming pixel data in parallel until the first image buffer1311is empty.

In a third operation scenario (but not limited), the first image buffer1311can be used to collect and store the pixel information (values or differences) of pixel units, which are transmitted from the analog image sensor circuit1305, to form and generate data of one or more complete frames. The SMD1312can determine whether an actual motion occurs based on the generated frame data. If it is determined that an actual motion occurs, the digital processing circuit1310is arranged to send an alert signal to wake up the backend system device1315and also transmit the sensed frames or image streams to the backend system device1315. In this situation, the sensed frames or image streams may be temporarily stored in the second image buffer1316by using the image freeze operation before the backend system device1315is completely awakened. When the backend system device1315completely exits the power saving mode and enters the normal mode, sensed frames or image streams can be directly transmitted to the external processor1317without being buffered in the second image buffer1316.

Refer back toFIG.13again. Specifically, to determine whether a pixel value (one or each pixel value) changes, the event camera1306is arranged for sensing/capturing the current pixel value of a corresponding pixel unit to detect whether the pixel value changes. The event camera1306for example senses or captures the current pixel value of the pixel unit (as shown by S13051), and then it calculates a pixel difference Diff between the sensed current pixel value and a reference pixel value of the pixel unit wherein the reference pixel value of such pixel unit (as shown by S13052) can be a previous pixel value of the pixel unit sensed by the event camera at an earlier timing or an average of a plurality of pixel values of the pixel unit sensed by the event camera at an earlier timing. Accordingly, for more or all pixel units, the event camera1306can capture a plurality of current pixel values and then calculate or generate a plurality of pixel differences respectively corresponding to a plurality of pixel units.

Then, the event camera1306determines whether the pixel value changed by comparing the pixel difference Diff with a pixel threshold TH (as shown by S13053). If the pixel difference Diff becomes higher than the pixel threshold TH, the event camera1306can determine that the pixel value changes. Instead, if the pixel difference Diff is not higher than the pixel threshold TH, the event camera1306determines that the pixel value does not change. It should be noted that in this embodiment the pixel value changing means that the pixel value varies significantly while the pixel value not changing means that the pixel value is not varied or varies insignificantly.

When the pixel difference Diff becomes equal to or higher than the pixel threshold TH, the event camera1306is arranged to generate and send a trigger signal to wake up the digital processing circuit1310, send the currently counted value of a counter value N and the information/data associated with the currently captured pixel value to the digital processing circuit1310, update the reference pixel value by using the currently captured pixel value, and reset the counter value N as zero. The counter value N is configured as zero initially. If the SMD1312is not in the normal mode, the currently counted value of the counter value N and the information/data associated with the currently captured pixel value can be temporarily stored in the first image buffer1311. It should be noted that the pixel difference Diff of the pixel unit is calculated and updated each time after an exposure operation, performed upon the pixel unit, is finished to obtain its currently sensed pixel value. When the pixel difference Diff is not higher than the pixel threshold TH, the event camera1306is arranged to increment or accumulate the counter value N by one. In this situation, the event camera1306does not send the trigger signal, the currently counted value of the counter value N and, the information/data associated with the currently captured pixel value.

The value of such counter value N is used to indicate the number of timings of consecutive frames during which the pixel value of a pixel unit does not change. Equivalently, the counter value N, employed by the event camera1306, is arranged to determine a time interval between the pixel value of such pixel unit changing twice. It should be noted that the values of counter values N corresponding to different pixel units may be identical, different, or may be partially different. In one embodiment, a resultant counter value can be selected from the values of counter values N corresponding to different pixel units, and such resultant counter value for example may be a smallest one in all the counter values N corresponding to different pixel units. The selected resultant counter value can be used to indicate a number of timings of consecutive frames during which the pixel values of all the pixel units do not change.

Further, for example (but not limited), the frame rate may be equal to 30 Hz, i.e. 30 frames in one second, and the counter value N may be sequentially accumulated from zero to 30 during one second if it is determined that the pixel value of the pixel unit does not change during such one second. In this situation, if it is then determined that the pixel value changes at a next frame timing, then the counter value N will not be counted to for example 31, and the event camera1306is arranged to generate a trigger signal to wake up the digital processing circuit1310and to transmit the currently counted counter value (i.e.30) to the digital processing circuit1310before the counter value N is reset as zero. Also, at the same time, the event camera1306is arranged to output information/data of the sensed/captured pixel value to the digital processing circuit1310.

The information/data of the sensed/captured pixel value which changes may be transmitted from the analog image sensor circuit1305to the digital processing circuit1310by using at least two different ways. In one embodiment, the event camera1306may send an actual value of the currently captured pixel value to the digital processing circuit1310, and the digital processing circuit1310can directly use the actual value to replace a corresponding value of a pixel unit in a previous frame to generate a current frame. Further, in other embodiments, the event camera1306may send a difference value between the actual value of the actually captured pixel value and the actual value of a previously captured pixel value to the digital processing circuit1310, and the digital processing circuit1310can add such difference value into a corresponding value of a pixel unit in a previous frame to generate a current frame.

Refer toFIG.15.FIG.15is a diagram showing two different operations for updating and generating data of a current frame for the digital processing circuit1310(or SMD1312). For a pixel unit such as an active pixel unit, data of a last previous frame can be stored in the SMD1312as a static frame FS even though some partial circuits within SMD1312is powered off. Such static frame FS is generated by the digital processing circuit1310at an earlier timing when the event camera1306detects the pixel value(s) change(s); that is, the static frame FS is associated with the previous pixel value(s). As shown in a first scenario ofFIG.15, When it is determined that the pixel value of the active pixel unit changes at a current frame timing, the event camera1306in the first scenario transmits the actual pixel value of such active pixel unit to the image buffer1311of digital processing circuit1310so that the image data associated with the change of pixel value can be stored in the image buffer1311. Then, after SMD1312completely enters the normal mode, the SMD1312can generate the current frame FC based on the data of the static frame FS and the actual value of the pixel value. In practice, the SMD1312is arranged to use the actual value to replace the previous value of such active pixel unit in the static frame FS to generate the current frame FC.

Further, as shown in a second scenario ofFIG.15, for the active pixel unit, when it is determined that the pixel value of the active pixel unit changes at the current frame timing, the event camera1306in the second scenario transmits the difference value of previously captured and currently captured pixels corresponding to such active pixel unit to the image buffer1311of digital processing circuit1310. Then, after SMD1312completely enters the normal mode, the SMD1312can generate the current frame FC based on the data of the static frame FS and the difference value. In practice, the SMD1312adds the difference value into the previous pixel value of the same active pixel unit in the static frame FS to generate the current frame FC.

In one embodiment, when the current frame FC is generated, the SMD1312(or digital processing circuit1310) is arranged to determine whether a motion occurs by generating one or more background frames based on the generated current frame FC and the counter value N and then comparing the current frame FS with the background frames. In practice, information/data of a previous background frame can be stored in a memory circuit of the SMD1312, and the information stored in the memory circuit does not vanish even though the SMD1312is powered off. If the SMD1312has been kept in the normal mode, then the SMD1312may perform a recursive moving average operation based on a previous background frame and the current frame FC to generate a current background frame. For example (not limited), the current background frame can be determined by the following equation:

FBi=FBi-1+FC2

wherein FBiindicates data of the generated current background frame at a current frame timing i, FBi−1indicates data of a previous background frame at a previous frame timing i−1 and FC indicates the current frame (or called as a current data frame). Similarly, the next background frame based on the recursive moving average operation can be determined by the following equation:

FBi+1=FBi+FC′2

wherein FC′ is a next data frame. Usually, in a monitoring system, the curve corresponding to the background frames at the different frame timings will be convergenced into a stable background frame after a specific training time period.

However, if the SMD1312just exited the power saving mode and no data of the previous background frame FBi−1can be used to generate the current background frame FBi, then the SMD1312does not execute the recursive moving average operation. in this situation, the SMD1312is arranged to employ the counter value N to generate a weighting value and then use the weighting value, current data frame FC, and a last background frame FBi−Nstored in the memory circuit of SMD1312when the SMD1312becomes powered off. In this situation, the current background frame FBican be determined by the following equation:
FBi=∝×<FBi−N+(1−∝)×FC

wherein ∝ is the weighting value generated based on the counter value N. in a simplified example (not limited), the weighting value ∝ may be equal to

N-1N⁢or⁢NN+1.
For instance, if the value N is equal to 30, then ∝ is equal to 29/30, and the current background frame FBican be determined by the following equation:

FBi=2⁢93⁢0×FBi-k+13⁢0×FC;
and

alternatively, the current background frame FBimay be determined by the following equation:

FBi=3⁢03⁢1×FBi-k+13⁢1×FC.

However, this is not meant to be a limitation of the invention. In other words, the weighting value ∝ can be dynamically determined based on the number of how many consecutive frames which a pixel unit's image/value does not change. The calculation of using the counter value N to directly generate the current background frame FBiwithout executing the recursive moving average operation becomes more simplified since in this situation executing the recursive moving average operation needs recursively calculating data of background frames FBi−1, . . . , FBi−(N−1).

To determine whether a motion occurs in the current frame FC, the SMD1312is arranged to calculate a frame difference between the current background frame FBiand the current data frame FC, and then it compares the frame difference with a motion threshold THM to determine whether the motion occurs. When the frame difference becomes higher than the motion threshold THM, the SMD1312can determine that the motion occurs and then generate an alarm signal to the backend system device1315.

The motion threshold THM for example may be a fixed threshold value or can be dynamically adjusted. In one embodiment, the SMD1312can use a curve fitting operation to dynamically adjust the motion threshold THM according to a previous value of the motion threshold THM, the current background frame FBithe current data frame FC, and the counter value N.

Then, for generating a next background frame FBi+1, the SMD1312for example can execute the recursive moving average operation based on the generated current background frame FBiand a next data frame FC′ when the next data frame FC′ is generated by using the information/data of pixel value sent from the event camera1306.

To make readers more clearly understand the operations mentioned above,FIG.16is provided.FIG.16is a flowchart diagram of the operations of SMD1312that just entered the normal mode according to an embodiment of the invention. The steps are described in the following:Step1605: Start;Step1610: Generate current data frame FC according to the previous data/static current FS and information/data of the pixel image/value changing (e.g. the actual pixel value or pixel difference value);Step1615: Generate a weighting value ∝ according to the counter value N which is received from the event camera1306and buffered in the first image buffer1311;Step1620: Generate a current background frame according to the weighting value ∝, the last background frame stored by the SMD1312, and the current data frame FC;Step1625: calculate a frame difference between the current data frame FC and the current background frame;Step1630: Dynamically update the motion threshold THM according to the previous/original value of motion threshold THM, the last background frame, and the current data frame FC;Step1635: Determine whether the frame difference is equal to or higher than the motion threshold THM; if it is equal to or higher than the motion threshold THM, the flow proceeds to Step1645; otherwise, the flow proceeds to Step1640;Step1640: Does not generate the alarm signal; andStep1645: Generate the alarm signal.

Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.