Patent ID: 12211188

DETAILED DESCRIPTION

FIG.1is a diagram illustrating an image dewarping device100of an embodiment of the present invention. The image dewarping device100can be applied to correct warped images captured by a wide-angle lens or a fisheye lens. The image dewarping device100includes a camera10, a processor11and a memory12. The camera10is for capturing images. The camera10may include any image capture device. For example, the camera10may have a built-in lens13, such as a wide-angle lens, a fisheye lens, or any lens that may cause image distortion. The camera10may also include an image sensor14, such as a complementary metal oxide semiconductor (CMOS). The processor11is coupled to the camera10for processing the image. The processor11can be any form of processing device, such as a central processing unit, a microprocessor, a logic operation unit, a programmable chip or any other processor that can process images. The memory12is coupled to the processor11for buffering image processing data. The memory12can be any form of memory device, such as random access memory (RAM) or non-volatile memory. The processor11can determine which pixels in the image captured by the camera10are to be stored to the memory12, and further perform coordinate transformation on the pixels stored in the memory12to achieve image dewarping.

FIG.2is a diagram illustrating an image dewarping process. The image captured by a wide-angle lens or fisheye lens is distorted, such as the source image SRC. The processor11can transform the source image SRC into the corrected destination image DST through the coordinate transformation matrix H. Generally, the camera10would first store the entire source image SRC to the memory12, and then the processor11would transform the source image SRC into the corrected destination image DST through the coordinate transformation matrix H. After the destination image DST is produced, the memory space occupied by the source image SRC in the memory12is then released. The coordinate transformation matrix H can be generated by the processor11according to the lens parameters of the lens13. However, the entire source image SRC often occupies a large amount of memory space, resulting in over consumption of resource in the image dewarping device100. This may cause the bottleneck in the processor11accessing the image. Thus, more memory is needed in order to maintain the efficiency of image processing. The image dewarping method proposed in the embodiment can release part of the space of the memory12before the completing the destination image DST transformation, and thus maintaining the efficiency of image processing.

FIGS.3A and3Bare diagrams illustrating the image dewarping process of an embodiment of the present invention. The source image SRC may be an image captured by the lens13with the image sensor14. The source image SRC includes M×N pixels (e.g., 1920×1080 pixels). In practice, the image sensor14can divide the pixels in the source image SRC into N input rows of pixels ROW1˜ROWN, and transfer the N input rows of pixels to the processor11sequentially from top to bottom. In the embodiment, the region of interest ROI can be drawn from the source image SRC according to the lens parameters of the lens13, and an N×2 table can be generated to record the starting and ending coordinates the N input rows of pixels in the region of interest ROI. An example of a table is shown below:

TABLE 1InputStartingEndingRowCoordinateCoordinate100200300...110904101611186710531128191101...539355156554035515655413551565...N00

The number N can be determined by the image resolution. For example, N can be 1080, 2048 and so on. The processor11can determine which input pixels are located in the region of interest ROI according to the first table. As shown in the first table, if the input row of pixels does not include input pixels located in the region of interest ROI, the starting and ending coordinates are both 0. Taking the Input Row540as an example, the 355th-1565th pixels of the Input Row540can be stored to the memory12. Furthermore, the input rows of pixels ROW1˜ROWN are sequentially transferred to the memory12from top to bottom. As shown inFIG.3A, because the input row of pixels ROW1has no input pixels located in the region of interest ROI, the input row of pixels ROW1would not be stored to the memory12. In contrast, the input row of pixels ROWk includes input pixels located in the region of interest ROI, such as the input segment of pixels SECk inFIG.3B, these input pixels located in the region of interest ROI would be stored to the memory12. The input row of pixels ROWn inFIG.3Aalso includes input pixels located in the region of interest ROI, such as the input segment of pixels SECn inFIG.3B. These input pixels located in the region of interest ROI would also be stored to the memory12. Likewise, other input rows of pixels are done by the same process

FIGS.4A and4Bare the continuation ofFIGS.3A and3B.FIGS.4A and4Bare diagrams illustrating the image dewarping process of the embodiment. In the process of image dewarping, it is not necessary to wait until the entire source image SRC is transferred to the memory12before performing image dewarping. In the embodiment, coordinate transformation can be performed simultaneously during the time period when the source image SRC is being transformed. As shown inFIG.4A, the input segments of pixels SECk˜SECn include the input pixels required to generate the output row of pixels OUT1. When the input segments of pixels SECk˜SECn are all stored in the memory12, the processor11can read from the memory12the input pixels corresponding to the output row of pixels OUT1and perform coordinate transformation on the input pixels according to the coordinate transformation matrix H to generate the output row of pixels OUT1. The equation for coordinate transformation is as follows:
pdst=Hpsrc
psrcis the pixel coordinates of the source image SRC, and pdstis the pixel coordinates of the destination image DST.

A schematic diagram illustrating the destination image DST is shown inFIG.4B. The input rows of pixels ROW1˜ROWN are sequentially transferred to the memory12from top to bottom. When the output row of pixels OUT1is generated, the coordinate transformation has been completed for the input pixels of the first input segment of pixels SECk. Also, the input segment of pixels SECk does not correspond to output rows of pixels other than the output row of pixels OUT1. At this time, the memory space occupied by the input segment of pixels SECk can be released. However, the input segment of pixels SECn still includes pixels that are still needed for coordinate transformation, so the input segment of pixels SECn cannot be released from the memory12yet. It can only be released after all the pixels of the input segment of pixels SECn have completed the coordinate transformation. The memory space of the memory12can be released for other applications, thus speeding up image processing. The same method can be applied to generate the output row of pixels OUT2and the remaining output rows of pixels until the coordinate transformation has been completed for the source image SRC and the destination image DST.

In the embodiment, the coordinate transformation matrix H can be obtained according to the internal and external parameters of the lens13:

sp=A[Rt]⁢P=HPH=[h1⁢1h1⁢2h1⁢3h2⁢1h2⁢2h2⁢3h31h3⁢2h3⁢3]

The lowercase p is the image plane coordinates. s is scale. A is the camera parameter matrix, including the focal length. R is the image rotation parameter matrix. t is the image shift parameter matrix. The uppercase P is the world coordinates. The method for obtaining the coordinate transformation matrix H is known to those skilled in the art, and is not be repeated herein.

In addition, one can also find out the coordinates corresponding to the last input pixel of the output row of pixels OUT1according to the pixel coordinates of the output row of pixels OUT1and the coordinate transformation matrix H, so as to determine the timing for performing coordinate transformation for the input pixels corresponding to the output row of pixels OUT1. For example, using the coordinates of the last pixel poutin the output row of pixels OUT1(shown inFIG.4B) and the coordinate transformation matrix H, one can obtain the coordinates corresponding to the input pixel pin(shown inFIG.4A) with the equation:
pin=Hpout

In other words, when the input pixel pinis stored to the memory12, at this time all the pixels required to generate the output row of pixels OUT1are stored in the memory12, and the processor11can start to perform the coordinate transformation to generate the output row of pixels OUT1.

FIG.5is a flowchart of the image dewarping method200. The image dewarping method200includes the following steps:S202: Capture the source image SRC from the camera10;S204: Select the input row of pixels ROWk from the source image SRC;S206: If the input row of pixels ROWk includes the input pixels in the region of interest ROI in the source image SRC, store the input pixels to the memory12to generate an input segment of pixels SECk;S208: When the pixels required to generate the output row of pixels OUT1are all stored in the memory12, read from the memory12the pixels corresponding to the output row of pixels OUT1, and perform coordinate transformation on the pixels to generate the output row of pixels OUT1; andS210: When coordinate transformation has been completed on the pixels, release from the memory12the input segment of pixels (such as input segment of pixels SECk) that does not correspond to other output rows of pixels.

After releasing the memory space occupied by the input segment of pixels SECk, the processor11can wait until all the pixels required to generate the output row of pixels OUT2have been stored in the memory12, and then use the coordinate transformation matrix H to perform coordinate transformation on the pixels to generate output row of pixels OUT2. Then, the input segment of pixels that has completed coordinate transformation and does not correspond to other output rows of pixels can be released from the memory12. The process can be repeated until the coordinate transformation has been completed for the source image SRC and the destination image DST.

The image dewarping method provided by the embodiment does not need to store the entire source image SRC. It can simultaneously transform and output the rows of pixels of the destination image DST while inputting the rows of pixels of the source image SRC to the memory12. It can also the release input segments of pixels before other pixels complete coordinate transformation. This method reduces memory loading and improves image processing performance under limited computing resources.

FIG.6is a diagram illustrating an image dewarping process of another embodiment. The source image SRC may be an image generated by the captured by the lens13with the image sensor14. The source image SRC includes M×N pixels, e.g., 1920×1080 pixels. In practice, the image sensor14can divide the pixels in the source image SRC into N input rows of pixels ROW1˜ROWN, and transfer the N input rows of pixels to the processor11sequentially from top to bottom. In the embodiment, a region of interest ROI may be drawn from the source image SRC according to the lens parameters of the lens13. Each input row of pixels may be divided into P input segments of pixels according to the lens parameters of the lens13and the region of interest ROI and generate P N×2 tables to record the starting and ending coordinates of the input segment of pixels located in the region of interest ROI. P can be an integer greater than 1. For example, if P is equal to 3, that is, each input row of pixels is divided into 3 input segments of pixels, and the coordinates are recorded in 3 tables. An example of the 3 tables is as follows:

TABLE 2InputStartingEndingRowCoordinateCoordinate100200300...110001110011200...118715719119704719120698719...539355719540355719541355719...N00

TABLE 3InputStartingEndingRowCoordinateCoordinate100200300...110904101611186710531128191101...118720119911972011991207201199...539720119954072011995417201199...N00

TABLE 4InputStartingEndingRowCoordinateCoordinate100200300...110001110011200...118120012041191200121512012001221...539120015655401200156554112001565...N00

Tables 2, 3 and 4 corresponding to the input rows of pixels are segmented at the 720th pixel and the 1200th. The number N is determined by the image resolution. For example, the number N can be 1080, 2048 and so on. The processor11can determine which input pixels are located in the region of interest ROI and store in which memory block according to the Tables 2, 3 and 4. If the input segment of pixels does not include input pixels located in the region of interest ROI, both the starting and ending coordinates are represented by 0. Furthermore, the three input segments of pixels of an input row of pixels are stored in three different blocks in the memory12. Taking the Input Row540as an example, the 355th to 719th pixels can be stored in the first memory block; the 720th to 1199th pixels can be stored in the second memory block; the 1200th to 1565th pixels can be stored in the third memory block. In the embodiment, image dewarping can be performed in the same manner as described inFIGS.4A and4B. In other words, it can use a coordinate transformation matrix to complete image dewarping. However, the differences between this embodiment and the aforementioned embodiment are described in the follow paragraphs.

FIGS.7A and7Bare diagrams illustrating releasing the input segments of pixels from the memory12in the image warping process ofFIG.6. As shown inFIG.7A, the input row of pixels ROWk can be divided into three input segments of pixels SECk1, SECk2and SECk3. Also, the input row of pixels ROWn can be divided into three input segments of pixels SECn1, SECn2and SECn3. When the coordinate transformation of the output row of pixels OUT1has been completed, as shown inFIG.7B, the input segment of pixels SECk2still includes the pixels corresponding the output row of pixels OUT2that have not been transformed yet, but the input segments of pixels SECk1and SECk3do not include the pixels corresponding to other output rows of pixels. Therefore, the memory space occupied by the input segments of pixels SECk1and SECk3can be released from the memory12, while the memory space occupied by the input segment of pixels SECk2cannot be released yet.

At another point in time, the input segments of pixels SECn1and SECn3still include the pixels corresponding to the output row of pixels OUT3that have not been transformed. Thus, the memory space occupied by the input segments of pixels SECk1and SECk3cannot be released yet. However, the input segment of pixels SECn2does not include pixels corresponding to other output rows of pixels. Thus the memory space occupied by the input segment of pixels SECn2can be released from the memory12.

FIG.8is a flowchart of the image dewarping method300of another embodiment. The image dewarping method300includes the following steps:S302: Capture the source image SRC from the camera10;S304: Select the input row of pixels ROWk from the source image SRC;S306: Divide the input row of pixels ROWk into P input segments of pixels;S308: If the input row of pixels ROWk includes the input pixels in a region of interest ROI in the source image SRC, store the input pixels to the memory12;S310: When the pixels required to generate the output row of pixels OUT1have all been stored in the memory12, read from the memory12the pixels corresponding to the output row of pixels OUT1, and perform coordinate transformation on the pixels to generate the output row of pixels OUT1; andS312: When coordinate transformation has been completed on the pixels, release from the memory12an input segment of pixels that does not correspond to other output rows of pixels.

After releasing the memory space occupied by the input segment of pixels, the processor11can wait until all the pixels required to generate the output row of pixels OUT2have been stored in the memory12, and then use the coordinate transformation matrix H to perform coordinate transformation on the pixels to generate the output row of pixels OUT2. Then, the input segment of pixels that has completed coordinate transformation and does not correspond to other output rows of pixels can be released from the memory12. The process can be repeated until the coordinate transformation has been completed for the source image SRC and the destination image DST.

The image dewarping method provided by the embodiment does not need to store the entire source image SRC. It can simultaneously transform and output rows of pixels of the destination image DST while inputting the rows of pixels of the source image SRC to the memory12. It can also the release input segments of pixels before other pixels complete coordinate transformation. This method reduces memory loading and improves image processing performance under limited computing resources.

Those skilled in the art will readily observe that numerous modifications and alterations of the device and method may be made while retaining the teachings of the invention. Accordingly, the above disclosure should be construed as limited only by the metes and bounds of the appended claims.