Patent Description:
In some applications of the image processing technology, a plurality of image streams are generated according to the same image source. For example, after retrieving the images of the environment, an IP camera may generate a plurality of image streams, while some image streams having higher resolution can be displayed as higher resolution frames on a corresponding display module, and some image streams having lower resolution can be displayed on a display module having a lower resolution to provide an instant surveillance mechanism.

However, a plurality of corresponding circuits are required to perform image processing on a plurality of image streams. The cost of the circuits is thus increased.

<CIT> describes an image frame processing method for processing a plurality of input image frames with an image processing device.

<CIT> describes an apparatus generally comprising an input circuit, a storage circuit and an output circuit.

<CIT> describes a method for enhancing user experiences, especially when changing channels, in digital video broadcasting systems.

<CIT> describes an architecture of a vision pipe included in an image signal processor.

<CIT> describes acquiring and displaying images in real-time.

<CIT> descibes an image capturing apparatus and image capturing method.

In consideration of the problem of the prior art, an object of the present invention is to provide a multi-stream image processing apparatus and a multi-stream image processing method to perform process on a plurality of image streams by using a single latter stage circuit with a time-division method to avoid the high cost of disposition of a multiple of latter stage circuits.

These and other objectives of the present invention will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiments that are illustrated in the various figures and drawings.

An aspect of the present invention is to provide a multi-stream image processing apparatus and a multi-stream image processing method to perform process on a plurality of image streams by using a single latter stage circuit with a time-division method to avoid the high cost of disposition of a multiple of latter stage circuits.

Reference is now made to <FIG> is a block diagram of a multi-stream image processing apparatus <NUM> in an embodiment of the present invention. The multi-stream image processing apparatus <NUM> is able to retrieve the images of an environment to generate a plurality of image streams MS (main image stream), SS1 (first sub image stream) and SS2 (second sub image stream) according to the same image source IS and subsequently perform processing on the image streams MS, SS1 and SS2 to generate a processed main image stream EMS, a processed first sub image stream ESS1 and a processed second sub image stream ESS2.

The multi-stream image processing apparatus <NUM> includes a memory module <NUM>, a former stage circuit <NUM>, a latter stage circuit <NUM>, a processing circuit <NUM> and a synchronization circuit <NUM>.

In an embodiment, the memory module <NUM> has different blocks to store different data required to process the plurality of image streams. The memory module <NUM> can be implemented by a memory having a higher speed such as, but not limited to a double data rate synchronous dynamic random access memory (DDR SDRAM).

In an embodiment, the multi-stream image processing apparatus <NUM> may further include a memory controller <NUM>. The memory controller <NUM> can be implemented by such as, but not limited to a memory interface unit (MIU). Other circuit modules in the multi-stream image processing apparatus <NUM>, e.g. the former stage circuit <NUM> and the latter stage circuit <NUM>, can perform read operation and write operation on the memory module <NUM> through the memory controller <NUM> to store data in the memory module <NUM> or retrieve data from the memory module <NUM>.

In an embodiment, the former stage circuit <NUM> is an image signal processor (ISP) configured to generate the plurality of image streams MS, SS1 and SS2 according to the same image source IS. The image source IS can be such as, but not limited to image sensing components in an IP camera. After the image sensing components sense the image, the former stage circuit <NUM> generates the image streams MS, SS1 and SS2 accordingly and stores the image streams MS, SS1 and SS2 having different resolutions in the memory module <NUM> through the memory controller <NUM>.

The image stream MS is a main image stream and the image streams SS1 and SS2 are respectively a first sub image stream and a second sub image stream. The resolution of the main image stream MS is higher than the resolution of the first sub image stream SS1 and the resolution of the second sub image stream SS2. In an embodiment, the resolution of the main image stream MS is <NUM>, The resolution of the first sub image stream SS1 is <NUM>×<NUM>. The resolution of the second sub image stream SS2 is <NUM>×<NUM>. It is appreciated that the number of the sub image streams and the resolution of each of the image streams described above are merely an example. The present invention is not limited thereto. In an embodiment, the number of the sub image stream can be one or more than one.

In an embodiment, the latter stage circuit <NUM> is an encoder configured to read the image streams MS, SS1 and SS2 from the memory module <NUM> through the memory controller <NUM> to perform encoding process and generate the processed main image stream EMS, the processed first sub image stream ESS1 and the processed second sub image stream ESS2 that are finished being encoded. In different embodiments, the encoding process is performed based on such as, but not limited to H264, H265 or other encoding standards.

The synchronization circuit <NUM> is electrically coupled to the former stage circuit <NUM>, the latter stage circuit <NUM> and the processing circuit <NUM> and is configured to, according to the control of the processing circuit <NUM>, balance the speed that the former stage circuit <NUM> stores the image stream to the memory module <NUM> and the speed that the latter stage circuit <NUM> reads the image stream from the memory module <NUM>.

The processing circuit <NUM> is electrically coupled to the memory module <NUM>, the former stage circuit <NUM>, the latter stage circuit <NUM> and the synchronization circuit <NUM>. The processing circuit <NUM> can execute software or firmware executable commands <NUM> to perform the function of the multi-stream image processing apparatus <NUM>. More specifically, the processing circuit <NUM> can retrieve the software or firmware executable commands <NUM> from a storage module (not illustrated) included in the multi-stream image processing apparatus <NUM>, in which the software or firmware executable commands <NUM> include such as, but not limited to firmware of the former stage circuit <NUM>, the latter stage circuit <NUM> and the synchronization circuit <NUM> and other related commands for operating and controlling the former stage circuit <NUM>, the latter stage circuit <NUM> and the synchronization circuit <NUM>. The processing circuit <NUM> further operates and controls the former stage circuit <NUM>, the latter stage circuit <NUM> and the synchronization circuit <NUM> accordingly to perform the storing, reading and encoding of the image streams MS, SS1 and SS2 and generate the processed main image stream EMS, the processed first sub image stream ESS1 and the processed second sub image stream ESS2.

It is appreciated that in an embodiment, the storage module can be implemented by another memory independent from the memory module <NUM>. For example, the storage module can be implemented by such as, but not limited to a optical disk, a random access memory (RAM), a read only memory (ROM), a floppy disk, a hard drive or a CD-ROM. In another embodiment, the storage module can also be implemented together with the memory module <NUM> as a single memory.

Reference is now made to <FIG> at the same time. The detail function of the multi-stream image processing apparatus <NUM> is further described in accompany with <FIG> and <FIG> in the following paragraphs.

<FIG> is a flow chart of a multi-stream image processing method <NUM> in an embodiment of the present invention. The multi-stream image processing method <NUM> can be used in the multi-stream image processing apparatus <NUM> illustrated in <FIG>. In an embodiment, the multi-stream image processing method <NUM> includes the steps illustrated in <FIG>.

Step S210: a plurality of image streams MS, SS1 and SS2 are generated by the former stage circuit <NUM> according to the same image source IS. As described above, image streams MS, SS1 and SS2 include the main image stream MS, the first sub image stream SS1 and the second sub image stream SS2. The resolution of the main image stream MS is higher than the resolution of each of the first and the second sub image streams SS1 and SS2.

Step S220: Within an image frame processing time period, a N-th sub image frame of the sub image stream SS1 is stored by the former stage circuit <NUM> in the current first sub image storage block 114A of the memory module <NUM>, a N-th sub image frame of the sub image stream SS2 is stored by the former stage circuit <NUM> in the current second sub image storage block 116A of the memory module <NUM>, and a N-th main image frame of the main image stream MS is stored by the former stage circuit <NUM> in a main image storage block <NUM> of the memory module <NUM>. N is a positive integer equal to or larger than <NUM>.

Reference is now made to <FIG> at the same time. <FIG> is a timing diagram of the image processing performed by the multi-stream image processing apparatus <NUM> in an embodiment of the present invention.

The time sequence during the processing of the multi-stream image processing apparatus <NUM> includes a plurality of image frame processing time periods, such as the consecutive three image frame processing time periods TN-<NUM>, TN and TN+<NUM> exemplarily illustrated in <FIG>. Each two adjacent image frame processing time periods include a synchronization signal time period therebetween, such as the synchronization signal time period TSN between the image frame processing time periods TN-<NUM> and TN, and the synchronization signal time period TSN+<NUM> between the image frame processing time periods TN and TN+<NUM>. The synchronization signal time periods TSN and TSN+<NUM> respectively correspond to the transmission of the synchronization signals SN and SN+<NUM> and respectively mark the beginning of the image frame processing time periods TN and TN+<NUM>.

In the present embodiment, within the synchronization signal time periods TSN and TSN+<NUM>, both of the former stage circuit <NUM> and the latter stage circuit <NUM> do not perform data processing. The former stage circuit <NUM> and the latter stage circuit <NUM> only perform data processing within the image frame processing time periods TN-<NUM>, TN and TN+<NUM>.

In <FIG>, the time periods corresponding to the processing of the former stage circuit <NUM> is illustrated as blocks having slashes. Since the former stage circuit <NUM> can simultaneously generate the image streams MS, SS1 and SS2, the former stage circuit <NUM> simultaneously stores the N-th sub image frame of the sub image stream SS1 in the current first sub image storage block 114A, stores the N-th sub image frame of the sub image stream SS2 in the current second sub image storage block 116A and stores the N-th main image frame of the main image stream MS in the main image storage block <NUM> after the image frame processing time period TN begins.

Step S230: Within a first sub period TS1 of the image frame processing time period TN, a N-<NUM>-th sub image frame of the sub image stream SS1 stored in a previous first sub image storage block 114B of the memory module <NUM> is read by the latter stage circuit <NUM> and a N-<NUM>-th sub image frame of the sub image stream SS2 stored in a previous second sub image storage block 116B of the memory module <NUM> is read by the latter stage circuit <NUM>. The N-<NUM>-th sub image frame of the sub image stream SS1 and the N-<NUM>-th sub image frame of the sub image stream SS2 are processed by the latter stage circuit <NUM>.

In an embodiment, as illustrated in <FIG>, within the image frame processing time period TN-<NUM>, the N-<NUM>-th sub image frame of the sub image stream SS1 and the N-<NUM>-th sub image frame of the sub image stream SS2 are respectively stored in the previous first sub image storage block 114B and the previous second sub image storage block 116B by the former stage circuit <NUM> through the memory controller <NUM>.

The time periods corresponding to the processing of the latter stage circuit <NUM> is illustrated as blocks having dots. The latter stage circuit <NUM> is operated by using a time-division method to process one image frame in one time period. As illustrated in <FIG>, within the first sub period TS1, the latter stage circuit <NUM> first reads the N-<NUM>-th sub image frame corresponding to the first sub image stream SS1 from the previous first sub image storage block 114B and processes the N-<NUM>-th sub image frame to generate the image frame corresponding to the processed first sub image stream ESS1. Subsequently, within the remained time period within the first sub period TS1, the latter stage circuit <NUM> reads the N-<NUM>-th sub image frame corresponding to the second sub image stream SS2 from the previous second sub image storage block 116B and processes the N-<NUM>-th sub image frame to generate the image frame corresponding to the processed second sub image stream ESS2.

Step S240: Within a second sub period TS2 of the image frame processing time period TN behind the first sub period TS1, the N-th main image frame stored in the main image storage block <NUM> is read by the latter stage circuit <NUM> and the N-th main image frame is processed by the latter stage circuit <NUM>.

In an embodiment, a capacity of the main image storage block <NUM> is larger than or equals to a size of the N-th main image frame. In some embodiments, the operation speed of the latter stage circuit <NUM> may be faster than the operation speed of the former stage circuit <NUM>. As a result, in order to avoid the occurrence of the condition that the speed that the latter stage circuit <NUM> reads data from the main image storage block <NUM> exceeds the speed that the former stage circuit <NUM> stores data to the main image storage block <NUM>, the synchronization circuit <NUM> performs coordination and synchronization between the former stage circuit <NUM> and the latter stage circuit <NUM>.

Reference is now made to <FIG> is a block diagram of the synchronization circuit <NUM> in an embodiment of the present invention.

As illustrated in <FIG>, the synchronization circuit <NUM> includes a first comparison module <NUM>, a second comparison module <NUM> and a synchronization processing module <NUM>.

The first comparison module <NUM> and the second comparison module <NUM> retrieve frame processing information FP1 and FP2 from the former stage circuit <NUM> and the latter stage circuit <NUM> respectively. The frame processing information FP1 includes a former stage frame number F1 and a storing row number RW of the image frame that the former stage circuit <NUM> currently stores. The frame processing information FP2 includes a latter stage frame number F2 and a reading row number RR of the image frame that the latter stage circuit <NUM> currently reads.

In an embodiment, the first comparison module <NUM> performs comparison according to the former stage frame number F1 of the image frame that the former stage circuit <NUM> currently stores and the latter stage frame number F2 of the image frame that the latter stage circuit <NUM> currently reads to determine whether the image frame that the former stage circuit <NUM> currently stores and the image frame that the latter stage circuit <NUM> currently reads are the same image frame. The second comparison module <NUM> performs comparison according to the storing row number RW of the image frame that the former stage circuit <NUM> currently stores and the reading row number RR of the image frame that the latter stage circuit <NUM> currently reads to determine whether the reading row number RR of the image frame that the latter stage circuit <NUM> currently reads exceeds the storing row number RW of the image frame that the former stage circuit <NUM> currently stores.

The synchronization processing module <NUM> further determines the operation condition of the former stage circuit <NUM> and the latter stage circuit <NUM> according to the comparison results of the first comparison module <NUM> and the second comparison module <NUM> to further determines the synchronization strategy being used.

Reference is now made to <FIG> at the same time. <FIG> is a flow chart of a synchronization flow <NUM> performed during the operation of the synchronization circuit <NUM> in an embodiment of the present invention. The operation mechanism of the synchronization circuit <NUM> is further described in detail in accompany with <FIG> and <FIG> in the following paragraphs.

Step S510: As illustrated in <FIG>, the first comparison module <NUM> retrieves the former stage frame number F1 and the latter stage frame number F2 in the frame processing information FP1 and the frame processing information FP2 respectively from the former stage circuit <NUM> and the latter stage circuit <NUM> to perform comparison.

Step S520: According to the first comparison result CR1 of the first comparison module <NUM>, the synchronization processing module <NUM> further determines whether the image frame that the former stage circuit <NUM> currently stores and the image frame that the latter stage circuit <NUM> currently reads are the same image frame.

Step S530: When the image frame that the former stage circuit <NUM> currently stores and the image frame that the latter stage circuit <NUM> currently reads are not the same image frame, e.g. the condition within the first sub period TS1 of the image frame processing time period TN, in which the latter stage circuit <NUM> reads the N-<NUM>-th sub image frame and the former stage circuit <NUM> stores the N-th main image frame, the synchronization processing module <NUM> does not activate the synchronization mechanism of the synchronization circuit <NUM>.

Step S540: When the image frame that the former stage circuit <NUM> currently stores and the image frame that the latter stage circuit <NUM> currently reads are the same image frame, e.g. the condition within the second sub period TS2 of the image frame processing time period TN, in which the latter stage circuit <NUM> reads the N-th sub image frame and the former stage circuit <NUM> also stores the N-th main image frame, the synchronization processing module <NUM> activates the synchronization mechanism of the synchronization circuit <NUM>. The second comparison module <NUM> retrieves the storing row number RW and the reading row number RR in the frame processing information FP1 and FP2 respectively from the former stage circuit <NUM> and the latter stage circuit <NUM> to perform comparison.

Reference is now made to <FIG> at the same time. <FIG> is a diagram of the main image frame <NUM> in an embodiment of the present invention.

Step S550: According to the second comparison result CR2 of the second comparison module <NUM>, the synchronization processing module <NUM> determines whether the storing row number RW of the main image frame <NUM> that the former stage circuit <NUM> currently stores exceeds the reading row number RR of the main image frame <NUM> that the latter stage circuit <NUM> currently reads.

Step S560: When the storing row number RW exceeds the reading row number RR, the content read by the latter stage circuit <NUM> does not exceed the content stored by the former stage circuit <NUM>. As a result, by delivering a latter stage circuit control signal CC1, the synchronization processing module <NUM> allows the latter stage circuit <NUM> to keep reading the N-th main image frame and process the N-th main image frame.

Step S570: On the contrary, when the storing row number RW does not exceed the reading row number RR, the content read by the latter stage circuit <NUM> exceeds the content stored by the former stage circuit <NUM>. As a result, by delivering the latter stage circuit control signal CC1, the synchronization processing module <NUM> stops the latter stage circuit <NUM> from reading the N-th main image frame to avoid the reading of the incorrect data content.

As a result, under the coordination and the synchronization of the synchronization circuit <NUM>, the latter stage circuit <NUM> can read the content of the N-th main image frame in an order to process the N-th main image frame and generate the main image frame corresponding to the processed main image stream EMS. It is appreciated that in the present embodiment, the latter stage circuit <NUM> finishes processing the N-th main image frame before the next synchronization signal time period TSN+<NUM>.

Furthermore, within the image frame processing time period TN+<NUM>, the former stage circuit <NUM> can store the N+<NUM>-th sub image frames of the sub image streams SS1 and SS2 and the N+<NUM>-th main image frame of the main image stream MS by using the method described above.

Further, by using the method described above, the latter stage circuit <NUM> can read and process the N-th sub image frames of the sub image streams SS1 and SS2 first and subsequently reads and processes the N+<NUM>-th main image frame of the main image stream MS.

It is appreciated that in an embodiment, the current first sub image storage block 114A and the second sub image storage block 116A that the former stage circuit <NUM> uses to store the N-th sub image frames of the sub image streams SS1 and SS2 within the image frame processing time period TN become the previous sub image storage blocks in the image frame processing time period TN+<NUM>. The previous first sub image storage block 114B and the previous second sub image storage block 116B in the image frame processing time period TN becomes current sub image storage block in the image frame processing time periodTN+<NUM> and are used to store the N+<NUM>-th sub image frames of the sub image streams SS1 and SS2.

The embodiments described above use the condition that the capacity of the main image storage block <NUM> is larger than or equals to the size of the N-th main image frame as an example. In another embodiment, the capacity of the main image storage block <NUM> can be smaller than the size of the N-th main image frame and the main image storage block <NUM> operates as a ring buffer. In other words, when the part of the main image frame stored by the former stage circuit <NUM> reaches the highest address of the main image storage block <NUM>, the latest data of the image frame further replaces the lowest address of the main image storage block <NUM>. When the part of the main image frame read by the latter stage circuit <NUM> reaches the highest address of the main image storage block <NUM>, the data is kept reading from the lowest address of the main image storage block <NUM>.

Reference is now made to <FIG> and <FIG> at the same time. <FIG> is a block diagram of the synchronization circuit <NUM> in another embodiment of the present invention. <FIG> is a flow chart of a synchronization flow <NUM> performed during the operation of the synchronization circuit <NUM> in another embodiment of the present invention. The embodiment of the main image storage block <NUM> that operates as the ring buffer is further described in detail in accompany with <FIG> and <FIG> in the following paragraphs.

The synchronization circuit <NUM> illustrated in <FIG> is similar to the synchronization circuit <NUM> illustrated in <FIG> and includes the first comparison module <NUM>, the second comparison module <NUM> and the synchronization processing module <NUM>. In the present embodiment, the synchronization circuit <NUM> illustrated in <FIG> further includes a first calculation module <NUM>. The first calculation module <NUM> is configured to calculate a difference DR between the storing row number RW and the reading row number RR according to the frame processing information FP1 and FP2.

Besides the flow including the steps S510 to S570 illustrated in <FIG>, the synchronization mechanism performed by the synchronization circuit <NUM> illustrated in <FIG> further includes the flow illustrated in <FIG>.

Step S810: The first calculation module <NUM> calculates the difference DR between the storing row number RW and the reading row number RR.

Step S820: The synchronization processing module <NUM> determines whether the difference DR exceeds the capacity of the main image storage block <NUM>.

Step S830: When the difference DR does not exceed the capacity of the main image storage block <NUM>, the internal effective content that is not encoded does not fully occupy the capacity of the main image storage block <NUM>. As a result, by using the former stage circuit control signal CC2, the synchronization processing module <NUM> allows the former stage circuit <NUM> to keep storing the N-th main image frame.

Step S840: On the contrary, when the difference DR exceeds the capacity of the main image storage block <NUM>, the speed that the former stage circuit <NUM> stores the main image frame is faster than the speed that the latter stage circuit <NUM> reads the main image frame. If the storing is kept being performed, the data of the main image frame that the latter stage circuit <NUM> has not read yet is going to be replaced. As a result, by using the former stage circuit control signal CC2, the synchronization processing module <NUM> stops the former stage circuit <NUM> from storing the N-th main image frame. The storing of the data of the main image frame is therefore stopped temporarily.

As a result, by using the mechanism described above, the embodiment of the main image storage block <NUM> operating as a ring buffer not only avoids the condition that the incorrect data is read when the speed of the read operation of the latter stage circuit <NUM> is too fast, but also avoids the condition that the data that the latter stage circuit <NUM> has not read is replaced when the speed of the storage operation of the former stage circuit <NUM> is too fast.

Reference is now made to <FIG> at the same time. <FIG> is a timing diagram of the image processing performed by the multi-stream image processing apparatus <NUM> in another embodiment of the present invention.

The time sequence during the processing of the multi-stream image processing apparatus <NUM> includes a plurality of image frame processing time periods, such as the consecutive three image frame processing time periods TN-<NUM>, TN and TN+<NUM> exemplarily illustrated in <FIG>. Each two adjacent image frame processing time periods include a synchronization signal time period therebetween, such as the synchronization signal time period TSN between the image frame processing time periods TN-<NUM> and TN, and the synchronization signal time period TSN+<NUM> between the image frame processing time periods TN and TN+<NUM>. The synchronization signal time periods TSN and TSN+<NUM> respectively correspond to the transmission of the synchronization signals SN and SN-<NUM> to respectively mark the beginning of the image frame processing time periods TN and TN+<NUM>.

In the present embodiment, during the first sub period TS1 in the image frame processing time period TN, the latter stage circuit <NUM> still reads and processes the N-<NUM>-th sub image frames of the sub image streams SS1 and SS2. During the second sub period TS2 in the image frame processing time period TN, the latter stage circuit <NUM> still reads and processes the N-th main image frame.

However, in the present embodiment, besides the feature that the main image storage block <NUM> is implemented by a ring buffer, the image frame processing time period TN further includes a third sub period TS3 before the first sub period TS1. For the N-<NUM>-th main image frame that the image frame processing time period TN-<NUM> corresponds to, the latter stage circuit <NUM> not only processes the N-<NUM>-th main image frame in a latter part of the image frame processing time period TN-<NUM>, but also processes the N-<NUM>-th main image frame in the synchronization signal time period TSN and the third sub period TS3. Identically, the latter stage circuit <NUM> can also process the N-th main image frame in the latter part of the image frame processing time period TN, the synchronization signal time period TSN+<NUM> and the third sub period TS3 in the image frame processing time period TN+<NUM>.

However, for the former stage circuit <NUM>, the N-th sub image frames of the sub image streams SS1 and SS2 and the N-th main image frame of the main image stream MS are still stored from the beginning of the image frame processing time period TN. As a result, within the third sub period TS3, the N-<NUM>-th main image frame is read and the N-th main image frame is stored simultaneously in the main image storage block <NUM>. The advantage of applying the third sub period TS3 is that the latter stage circuit can perform data processing with a slower speed and accomplish a power-saving mechanism.

Reference is now made to <FIG> at the same time. <FIG> is a block diagram of the synchronization circuit <NUM> in yet another embodiment of the present invention. <FIG> is a flow chart of a synchronization flow <NUM> performed during the operation of the synchronization circuit <NUM> in yet another embodiment of the present invention. The embodiment of the main image storage block <NUM> that operates as the ring buffer is further described in detail in accompany with <FIG> in the following paragraphs.

The synchronization circuit <NUM> illustrated in <FIG> is similar to the synchronization circuit <NUM> illustrated in <FIG> and includes the first comparison module <NUM>, the second comparison module <NUM>, the synchronization processing module <NUM> and the first calculation module <NUM>. In the present embodiment, the synchronization circuit <NUM> illustrated in <FIG> further includes a second calculation module <NUM>. The second calculation module <NUM> is configured to operate when the image frame that the former stage circuit <NUM> currently stores and the image frame that the latter stage circuit <NUM> currently reads are different.

The synchronization mechanism performed by the synchronization circuit <NUM> illustrated in <FIG> includes the flow including the steps S510, S520 and S540 to S570 illustrated in <FIG> and the flow including the steps S810 to S840 illustrated in <FIG>. However, in the present embodiment, when the image frame that the former stage circuit <NUM> currently stores and the image frame that the latter stage circuit <NUM> currently reads are determined to be different in the step S520 in <FIG>, e.g. the condition illustrated in the third sub period TS3, the synchronization circuit <NUM> illustrated in <FIG> operates the flow illustrated in <FIG> to activate the synchronization mechanism.

Step S1110: According to the frame processing information FP1 and FP2, the second calculation module <NUM> calculates a difference between the size FR of the N-<NUM>-th main image frame and the reading row number RR of the N-<NUM>-th main image frame. Such a difference stands for the remained data amount in the N-<NUM>-th main image frame that is not read and processed by the latter stage circuit <NUM>. Subsequently, the second calculation module <NUM> calculates a sum SUM of the difference and the storing row number RW of the N-th main image frame. The storing row number RW stands for the data amount in the N-th main image frame that the former stage circuit <NUM> is stored.

Step S1120: the synchronization processing module <NUM> determines whether the sum SUM exceeds the capacity of the main image storage block <NUM>.

Step S1130: When the sum SUM does not exceed the capacity of the main image storage block <NUM>, the speed that the former stage circuit <NUM> performs storage operation does not exceed the speed that the latter stage circuit <NUM> performs read operation. As a result, by using the former stage circuit control signal CC2, the synchronization processing module <NUM> allows the former stage circuit <NUM> to keep storing the N-th main image frame.

Step S1140: On the contrary, when the sum SUM exceeds the capacity of the main image storage block <NUM>, the speed that the former stage circuit <NUM> performs storage operation exceeds the speed that the latter stage circuit <NUM> performs read operation. If the storing is kept being performed, the data of the main image frame that the latter stage circuit <NUM> has not read yet is going to be replaced. As a result, by using the former stage circuit control signal CC2, the synchronization processing module <NUM> stops the former stage circuit <NUM> from storing the N-th main image frame. The storing of the data of the main image frame is therefore stopped temporarily.

As a result, the multi-stream image processing apparatus <NUM> of the present invention can use a single latter stage circuit to process the plurality of image streams generated by the former stage circuit with a time-division method to avoid the cost of disposition of a plurality of latter stage circuits.

Further, by using the time-division method, the memory module only needs to store N-<NUM>-th sub image frame and the N-th sub image frame corresponding to the sub image frames having a lower data amount due to the lower resolution. For the main image frame having a higher data amount due to the higher resolution, the disposition of two storage blocks is not necessary. The space cost of the memory module can be reduced as well.

Furthermore, by disposing the synchronization circuit, the memory module can include the main image storage block implemented by the ring buffer. The capacity can be reduced to further reduce the space cost of the memory module.

Claim 1:
A multi-stream image processing method, comprising:
generating (S210) a plurality of image streams (MS, SS1, SS2) by a former stage circuit according to a same image source, wherein the image streams comprises a main image stream and at least one sub image stream, and the resolution of the main image stream is higher than the resolution of the sub image stream;
within a N-th image frame processing time period, storing (S220) a N-th sub image frame of the sub image stream by the former stage circuit in at least one current sub image storage block of a memory module (<NUM>) and storing a N-th main image frame of the main image stream by the former stage circuit in a main image storage block of the memory module (<NUM>), wherein N is a positive integer equal to or larger than <NUM>;
within a first sub period of the N-th image frame processing time period, reading (S230) a N-<NUM>-th sub image frame of the sub image stream stored in a previous sub image storage block of the memory module (<NUM>) by a latter stage circuit and processing the N-<NUM>-th sub image frame by the latter stage circuit;
within a second sub period of the N-th image frame processing time period behind the first sub period, reading (S240) the N-th main image frame stored in the main image storage block by the latter stage circuit and processing the N-th main image frame by the latter stage circuit;
retrieving (S510) frame processing information of the former stage circuit and the latter stage circuit, wherein the frame processing information comprises a former stage frame number and a storing row number of the image frame that the former stage circuit currently stores, and comprises a latter stage frame number and a reading row number of the image frame that the latter stage circuit currently reads;
determining (S520) whether the former stage frame number and the latter stage frame number are the same; and
when the former stage frame number and the latter stage frame number are the same, activating a synchronization mechanism of a synchronization circuit (<NUM>), wherein the synchronization mechanism is performed based on the storing row number and the reading row number.