Image processing device and method of correcting images

An image processing device to correct for wobble of an image constructed according to exemplary embodiments of the invention includes an input interface to communicate with an image capture sensor, and a motion vector detector to process first and second images received from the image sensor through the input interface, to detect, in the first image, feature points having feature values higher than a threshold value, and to compare the first image to the second image by using at least parts of the feature points to determine a global motion vector of the first image. The threshold value is adjustable depending on a gain of the image sensor.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from and the benefit of the Korean Patent Application No. 10-2018-0005521, filed on Jan. 16, 2018, which is hereby incorporated by reference for all purposes as if fully set forth herein.

BACKGROUND

Field

Exemplary implementations of the invention relates generally to image capture and processing of images, and more particularly, to image processing devices and methods of correcting images having wobble.

Discussion of the Background

Image capture devices have been increasingly used with the development of electronic cameras and imaging technology, which raises issues of correcting for image wobble (e.g., shaking, blur) when the image is captured by people or environments in which the image capture device may be unintentionally moved, e.g., by the photographer or environmental conditions. Thus, image wobble is generated from various environments, e.g., when a person uses a mobile camera such as a portable digital camera, a smart phone, and the like, due to the person being unsteady, camera shake or the like. In case of an outdoor-installed CCTV, image blur is generated from a weather environment such as rain, wind, and the like, and an in-vehicle black box camera suffers from the image blur generated from a vibration of the vehicle during driving. Commercially available image correction devices or programs suffer from various drawbacks, such as complexity and inaccuracy. Accordingly, there remains a need for imaging processing devices and methods to correct for image wobble in a simple and accurate manner.

SUMMARY

Devices and methods constructed according to the principles and exemplary implementations of the invention are capable of efficiently correcting for wobble of images taken in various environments. For example, devices methods according to exemplary implementations of the invention correct for wobble with a relatively high degree of reliability even in a dark environment

According to one or more exemplary embodiments of the invention, an image processing device to correct for wobble of an image includes: an input interface to communicate with an image capture sensor; and a motion vector detector to process first and second images received from the image sensor through the input interface, to detect, in the first image, feature points having feature values higher than a threshold value, and to compare the first image to the second image by using at least parts of the feature points to determine a global motion vector of the first image, the threshold value being adjustable depending on a gain of the image sensor.

The motion vector detector may be configured to increase the threshold value as the gain of the image sensor increases.

The threshold value may be a first value when the gain of the image sensor is lower than a first level, the threshold value may be a second value higher than the first value when the gain of the image sensor is higher than a second level that is higher than the first level, and the threshold value may increase between the first value and the second value as the gain of the image sensor increases when the gain of the image sensor is between the first level and the second level.

The feature values may be contrast levels.

The first image may include a plurality of sub blocks. The motion vector detector may be configured to detect feature point candidates in the plurality of sub blocks, respectively, and to select one or more of the feature point candidates having feature values higher than the threshold value as the feature points.

The image processing device may further include an image stabilizer to correct the first image in response to a signal from the global motion vector.

The motion vector detector may be configured to generate signals representative of local motion vectors of feature point groups of the first image by comparing them to the second image, and to generate signals representative of the global motion vector based upon the local motion vectors, and wherein the feature point groups are defined by grouping the at least parts of the feature points.

The motion vector detector may be configured to adjust the number of feature points included in each of the feature points groups depending on the gain of the image sensor.

The number of feature points included in each of the feature points may increase as the gain of the image sensor increases.

The number of feature points may be a first value when the gain of the image sensor is lower than a first level, the number of feature points may be a second value higher than the first value when the gain of the image sensor is higher than a second level that is higher than the first level, and the number of feature points may increase between the first value and the second value as the gain of the image sensor increases when the gain of the image sensor is between the first level and the second level.

According to one or more exemplary embodiments of the invention, a method for correcting wobble in an image includes the steps of: receiving first and second images from an image sensor; receiving a gain of the image sensor; detecting, in the first image, feature points having feature values higher than a threshold value; and comparing the first image to the second image by using at least parts of the feature points to generate signals to correct for the wobble of the first image. The threshold value is adjustable depending on the gain of the image sensor.

The threshold value may increase as the gain of the image sensor increases.

The step of comparing the first image to the second image may include: defining feature point groups by grouping the at least parts of the feature points; generating signals representative of local motion vectors of the feature point groups of the first image by comparing to the second image; and generating signals representative of a global motion vector of the first image based upon the local motion vectors.

The number of feature points included in each of the feature points group may be adjusted depending on the gain of the image sensor.

The number of feature points included in each of the feature points may increase as the gain of the image sensor increases.

According to another exemplary embodiment of the invention, an image processing device to correct for wobble in an image includes: an input interface to communicate with an image sensor; and a motion vector detector to receive first and second images from the image sensor through the input interface, and to generate signals representative of a global motion vector of the first image based upon local motion vectors of feature point groups of the first image by comparing them to the second image, the feature point groups being defined by grouping at least parts of feature points included in the first image. The motion vector detector is configured to adjust the number of feature points included in each of the feature point groups depending on a gain of the image sensor.

The number of feature points included in each of the feature point groups may increase as the gain of the image sensor increases.

The feature points included in the first image may have feature values higher than a threshold value, the threshold value may increase as the gain of the image sensor increases.

The first image may include a plurality of sub blocks, and the motion vector detector may be configured to detect feature point candidates in the plurality of sub blocks, respectively, and to select one or more of the feature point candidates having feature values higher than the threshold value as the feature points included in the first image.

The threshold value may increase as the gain of the image sensor increases.

DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENTS

When an exemplary embodiment may be implemented differently, a specific process order may be performed differently from the described order. For example, two consecutively described processes may be performed substantially at the same time or performed in an order opposite to the described order. Also, like reference numerals denote like elements.

In exemplary embodiments, an image processing device, and/or one or more components thereof, may be implemented via one or more general purpose and/or special purpose components, such as one or more discrete circuits, digital signal processing chips, integrated circuits, application specific integrated circuits, microprocessors, processors, programmable arrays, field programmable arrays, instruction set processors, and/or the like.

According to one or more exemplary embodiments, the features, functions, processes, etc., described herein may be implemented via software, hardware (e.g., general processor, digital signal processing (DSP) chip, an application specific integrated circuit (ASIC), field programmable gate arrays (FPGAs), etc.), firmware, or a combination thereof. In this manner, the image processing device, and/or one or more components thereof may include or otherwise be associated with one or more memories (not shown) including code (e.g., instructions) configured to cause the image processing device, and/or one or more components thereof to perform one or more of the features, functions, processes, etc., described herein.

The memories may be any medium that participates in providing code to the one or more software, hardware, and/or firmware components for execution. Such memories may be implemented in any suitable form, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks. Volatile media include dynamic memory. Transmission media include coaxial cables, copper wire and fiber optics. Transmission media can also take the form of acoustic, optical, or electromagnetic waves. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a compact disk-read only memory (CD-ROM), a rewriteable compact disk (CD-RW), a digital video disk (DVD), a rewriteable DVD (DVD-RW), any other optical medium, punch cards, paper tape, optical mark sheets, any other physical medium with patterns of holes or other optically recognizable indicia, a random-access memory (RAM), a programmable read only memory (PROM), and erasable programmable read only memory (EPROM), a FLASH-EPROM, any other memory chip or cartridge, a carrier wave, or any other medium from which information may be read by, for example, a controller/processor.

FIG. 1is a conceptual view for schematically illustrating the distortion in images captured by an image capturing device when the image capturing device is wobbled.

Referring toFIG. 1, various image capturing devices, such as a smart phone, a tablet PC, CCTV, a Web camera (Webcam), a Pan-Tilt-Zoom (PTZ) camera, a camcorder, and a laptop, may capture a subject15and generate an image of the subject, which may be an animate or inanimate object.

A wobble (as used herein means any unintentional movement, vibration, or shaking that distorts an image) may occur when the image capturing device capture the subject15. For example, due to wobble the image capturing device11at a first time t1and the image capturing device12at a second time t2may capture the subject15at different positions even though the subject has not moved. Accordingly, even though the subject15is stationary, the image captured at the first time t1and the image captured at the second time t2may be different from each other. For example, even though the subject15is stationary, as shown inFIG. 1, a feature point of the subject15in the captured image is located at a first position P1of an image at the first time t1, but it is located at a second position P2of an image at the second time t2.

FIG. 2is a block diagram of an exemplary embodiment of an image processing device constructed according to the principles of the invention.

Referring toFIG. 2, an image processing device100is coupled to an image sensor200. The image sensor200generates a plurality of images IMG that are captured in chronological order, the generated images IMG are provided to the image processing device100. The image sensor200may convert optical signals to electrical signals using any known technology such as CMOS (Complemental Metal Oxide Semiconductor), and may digitize the electrical signals to generate each of the images IMG.

The image sensor200is further coupled to a sensor gain controller300. The image sensor200may receive a gain GN provided from the sensor gain controller300. The sensor gain controller300may adjust the gain value GN, in which the electrical signals are converted to the images IMG, to control the brightness of the images IMG. The gain GN of the image sensor200provided by the sensor gain controller300may be adjusted in various manners. For example, the sensor gain controller300may increase the gain GN of the image sensor200to increase the brightness of the image when at least parts of the images to be captured are perceived as relatively dark. For instance, the sensor gain controller300may increase the gain GN of the image sensor200to increase the brightness of the image in response to a user's selection received through, for example, a user interface1400shown inFIG. 14.

In case where the gain GN of the image sensor200increases, the brightness of the images IMG increases, but the noise included in the images IMG may also increase. According to the exemplary embodiments of the invention, the sensor gain controller300provides the image processing device100with a gain GN that is utilized to generate the images IMG. The image processing device100utilizes the gain GN to correct for wobble in the images IMG.

The image processing device100includes an input interface110, a motion vector detector120, an image stabilizer130, and an output interface140.

The input interface110communicates signals between the image processing device100and the image sensor200. The input interface110may receive the images IMG from the image sensor200, and may transfer the received images IMG to the motion vector detector120.

Although not shown inFIG. 2, a buffer memory coupled to the image processing device100and the image sensor200may be further provided. In this case, the image sensor200may store the generated images IMG in the buffer memory, and the input interface110of the image processing device100may access the buffer memory to read the images IMG.

According to the illustrated embodiment, the image processing device100may further include an interface receiving the gain GN of the image sensor200from the sensor gain controller300. The input interface110may transfer the received gain GN to the motion vector detector120.

The motion vector detector120processes the images IMG on the basis of the gain GN. The motion vector detector120detects feature points having feature values higher than a threshold value in an image to be corrected (hereinafter, referred to as “target image TGIMG”), and determines a global motion vector between the target image TGIMG and a comparison image CPIMG by using at least parts of the detected feature points. Here, the threshold value is adjusted depending on the value of the gain GN of the image sensor200.

The comparison image CPIMG may be an image that is captured before capturing the target image TGIMG. For example, the target image TGIMG may be an a-th image of the images IMG generated by the image sensor200in chronological order, and the comparison image CPIMG may be an (a-b)-th image of the images IMG, where a is a positive integer, and b is a positive integer less than a.

The motion vector detector120includes a feature point detector121, a local motion vector calculator122, and a global motion vector calculator123, as is known in the art. The feature point detector121is configured to detect feature points having feature values higher than the threshold value in the target image TGIMG. In an exemplary embodiment, the feature point detector121may divide the target image TGIMG into a plurality of sub blocks, select feature point candidates from each of the plurality of sub blocks, and detect one or more of the feature point candidates having feature values higher than the threshold value as feature points. The local motion vector calculator122may receive the gain GN through the input interface110and may adjust the threshold value depending on the value of the gain GN of the image sensor200.

The local motion vector calculator122is configured to calculate local motion vectors of the feature points by comparing the target image TGIMG to the comparison image CPIMG. For example, the local motion vector calculator122may calculate the local motion vectors of the feature points by comparing each of regions of the target image TGIMG which includes feature points to the comparison image CPIMG.

The global motion vector calculator123is configured to calculate a global motion vector according to the local motion vectors. For example, the average of the local motion vectors may be determined as the global motion vector. However, the exemplary embodiments are not limited thereto. Various manners for calculating the global motion vector according to the local motion vectors may be utilized as is known in the art. For instance, a histogram of the local motion vectors may be calculated, and the global motion vector may be determined according to the calculated histogram.

The image stabilizer130corrects for wobble in the target image TGIMG based on the global motion vector. The output interface140outputs the corrected target image TGIMG′.

FIG. 3is a flowchart of a method of correcting for wobble in an image at the image processing device according to an exemplary embodiment of the invention.

Referring toFIGS. 2 and 3, at step S110, a gain GN of the image sensor200is received from the sensor gain controller300. The images IMG are further received from the image sensor200.

At step S120, feature points having feature values higher than a threshold value are detected in the target image TGIMG. In an exemplary embodiment, the feature point detector121may detect a plurality of feature point candidates, and detect one or more feature point candidates having feature values higher than the threshold value as the feature points.

The feature point candidates may be selected in various manners. In an exemplary embodiment, the target image TGIMG may be divided into a plurality of sub blocks, a point having the highest feature value among points of each sub block may be selected as the feature point candidate, and feature point candidates having a feature value higher than the threshold value among the selected feature point candidates may be detected as the feature points.

In an exemplary embodiment, the feature value of each point may be a contrast level between a corresponding point and points adjacent thereto. A point having the highest contrast level in each sub block, for example, a point positioned on edges of an object in the image, a point positioned on the intersection of the edges, or the like, may be selected as the feature point candidate of the corresponding sub block. A point that is not positioned on the edges, for example, a point positioned on a background in the image, may be selected as the feature point candidate. In this regard, U.S. Pat. No. 9,466,095 is hereby incorporated by reference for all purposes as if fully set forth herein.

The noise included in the target image TGIMG increases as the gain GN of the image sensor200increases. The feature values of the points included in the target image TGIMG may decrease as the noise increases. For example, the sensor gain controller300may increase the gain GN of the image sensor200when the image is perceived as being relatively dark by an illuminance sensor or when a user input commanding the increase of the gain GN is received through a user interface. In this manner, the target image TGIMG is bright, but the contrast levels of the edge of the object included in the target image TGIMG may decrease with increase in the noise. If a point having a relatively low feature value is selected as a feature point and a local motion vector is calculated using the feature point, the possibility that the local motion vector reflects the wobble direction of the image may be reduced. That is, the local motion vector may have low reliability in reflecting the wobble direction of the image. To this end, the low reliability of the local motion vectors may mean that the reliability of a global motion vector calculated based on the local motion vectors is relatively low.

The noise included in the target image TGIMG decreases as the gain GN of the image sensor200decreases. In response to the decrease of the noise, the feature values of points included in the target image TGIMG may increase. For example, the sensor gain controller300may reduce the gain GN of the image sensor200when the image is perceived as relatively bright by the illuminance sensor or when a user input commanding the decrease of the gain GN is received through the user interface. In this manner, the target image TGIMG which is relatively clear may be obtained, and therefore the contrast levels of the edge of the object included in the target image TGIMG may increase. As such, if a relatively large number of local motion vectors are calculated using a relatively large number of feature points and a global motion vector is calculated based on the increased number of the local motion vectors, the reliability of the global motion vector may be improved.

According to the exemplary embodiments and the principles of the invention, points having the feature values higher than the threshold value are detected as the feature points in the target image TGIMG, with the threshold value increasing as the gain GN of the image sensor200increases. According to the increase in the threshold value, the detected feature points may have relatively high feature values, local motion vectors may be calculated by using the feature points having the high feature values. For instance, when perceived as relatively dark, a more definite edge in the target image TGIMG may be adopted as the feature points. Accordingly, the reliability of the local motion vectors may increase, and the reliability of the global motion vector that is calculated based on the local motion vectors may also increase. As a result, the image processing device100may correct for wobble in the image with high reliability.

The gain GN of the image sensor200may be utilized to generate the images IMG and may be a factor that determines the brightness or noise of the images IMG. Accordingly, the feature points may be selected efficiently by adjusting the threshold value according to the gain GN of the image sensor200. If the image capturing device includes an illuminance sensor and the threshold value is adjusted using an illuminance value provided from the illuminance sensor, given that the lens of the image sensor200is located at a physically different position from the illuminance sensor and the gain GN of the image sensor200does not necessarily have to be adjusted according to the illuminance value, the brightness and noise of the images IMG may vary independently of the level of the illuminance value. If feature points are selected using a threshold value that is adjusted according to the illuminance value and local motion vectors are calculated using the feature points, the possibility that the local motion vectors reflect the wobble direction may decrease. On the other hand, in the image processing device100constructed according to the exemplary embodiments, the threshold value is adjusted using the gain GN of the image sensor200, which is a factor that determines the brightness or noise of the images IMG, and thus the feature points may be selected efficiently.

At step S130, the target image TGIMG and the comparison image CPIMG is compared in the Local Motion Vector Calculator by using at least parts of the detected feature points. The local motion vector calculator122may calculate the local motion vectors of the feature points, and may determine the global motion vector according to the local motion vectors, as known in the art.

At step S140, the wobble of the target image TGIMG may be corrected according to the comparison result. The image stabilizer130may correct the wobble of the target image TGIMG on the basis of the global motion vector.

FIG. 4is a flowchart of an exemplary embodiment of step S120ofFIG. 3.FIG. 5Ais a schematic diagram for illustrating feature point candidates that are detected in a target image.FIG. 5Bis a schematic diagram for illustrating feature points selected from the feature point candidates.FIG. 6is a graph illustrating the relationship between a gain of an image sensor and a threshold voltage according to the principles of the invention.

Referring toFIGS. 2 and 4, at step S210, feature point candidates are detected from a plurality of sub blocks of the target image TGIMG. Referring toFIG. 5A, the target image TGIMG is divided into the plurality of sub blocks SBLK. Feature values, such as contrast levels, of points of each sub block may be calculated, and a point having the highest feature value among the calculated feature values may detected as a feature point candidate FPC.

Referring back toFIGS. 2 and 4, at step S220, one or more feature point candidates that have feature values higher than the threshold value are selected as feature points. Referring toFIG. 5B, feature point candidates having feature values equal to or lower than the threshold value are shown as being removed. Feature point candidates having feature values higher than the threshold value are selected as the feature points FP.

The threshold value may increase as the gain GN of the image sensor200increases. Referring toFIG. 6, the threshold value is a first value TV1when the gain GN of the image sensor200is lower than a first level G11, the threshold value increases between the first value TV1and a second value TV2as the gain GN of the image sensor200increases when the gain GN is between the first level G11and a second level G12, the threshold value is the second value TV2when the gain GN of the image sensor200is higher than the second level G12. As such, the threshold value is adaptively adjusted according to the gain GN of the image sensor200.

FIG. 7is a flowchart of a method of comparing a target image and a comparison image according to an exemplary embodiment of the invention.

Referring toFIGS. 2 and 7, at step S310, at least parts of the feature points are selected. This will be described in detail with reference toFIGS. 8 to 11.

At step S320, feature point groups are defined by grouping the selected feature points. Here, the number of the feature points included in the feature point group is adjusted according to the gain GN of the image sensor200. The feature point group may include feature points adjacent to each other.

At step S330, local motion vectors associated to the feature point groups are determined, respectively.

Step S310to step S330may be performed by the local motion vector calculator122. The local motion vector calculator122may select at least parts of the feature points, receive the gain GN of the image sensor200through the input interface110to group the selected feature points, and determine the local motion vectors for the feature point groups, respectively.

At step S340, a global motion vector is determined depending on the determined local motion vectors.

Given that the noise included in the target image TGIMG increases as the gain GN of the image sensor200increases, some of the local motion vectors of the individual feature points may have relatively low reliability. According to the exemplary embodiments and the principles of the invention, the local motion vector is calculated for the feature point group that includes one or more feature points, and the feature points may be grouped such that the number of the feature points included in the feature group increases as the gain GN of the image sensor200increases. Accordingly, each of the local motion vectors may have high reliability in reflecting a wobble direction of the image, and the global motion vector calculated based on the local motion vectors may also have high reliability.

FIG. 8is a flowchart of an exemplary embodiment of step S310ofFIG. 7.FIG. 9is a conceptual view for illustrating processes of calculating the local motion vectors of step S410ofFIG. 8.FIG. 10is a conceptual view for illustrating processes of selecting feature points by using the local motion vectors of step S420ofFIG. 8.

Referring toFIG. 8, at step S410, local motion vectors of the feature points are calculated by comparing the target image TGIMG and the comparison image CPIMG shown inFIG. 2. Referring toFIG. 9, one feature point FP of the target image TGIMG is shown for descriptive convenience. In the target image TGIMG, a rectangle centering on the feature point FP is defined as a motion estimation area MEA. The motion estimation area MEA may have a size of M*N, where M and N each may refer to the number of points (or pixels). In the comparison image CPIMG, a searching area SA is defined. The searching area SA may overlap the motion estimation area MEA and have a size larger than the motion estimation area MEA. For example, the searching area SA may have a size of (M+2L)*(N+2L), where L may refer to the number of points. Each of portions of the searching area SA may be compared to the motion estimation area MEA, and a portion of the searching area SA that is similar to the motion estimation area MEA may be determined as a matched area MTA. A correlation value between each of the portions of the searching area SA and the motion estimation area MEA is calculated, and a portion having the lowest correlation value among the portions of the searching area SA may be determined as the matched area MTA. For example, SAD (Sum of Absolute Difference) between each of the portions of the searching area SA and the motion estimation area MEA may be calculated as the correlation value based on the following equation 1.

Referring to equation 1, x and y are the coordinates of a feature point FP in the motion estimation area MEA, and u and v are the displacement for the searching area SA. It(x+i, y+j) is a function value that takes a certain factor of the coordinate (x+i, y+j) of a current image (or target image), and It-1(x+u+i, y+v+j) is a function value that takes the certain factor of the coordinate (x+u+i, y+v+j) of a previous image (or comparison image). It(x+i, y+j) and It-1(x+u+i, y+v+j) may be one of various functions known in the art. SADx,y(u, v) is Sum of Absolute Difference (SAD) between the motion estimation area MEA and a portion including a coordinate (x+u, y+v) of the searching portion SA. A portion of the searching area SA having the lowest SAD may be determined as the matched area MTA, a corresponding displacement u and v may be determined as a local motion vector of the feature point FP.

That is, the local motion vector of the feature point FP may be determined according to the coordination corresponding to the determined matched area MTA, and local motion vectors of the other feature points included in the target image TGIMG may be calculated in the same way as described above.

Referring back toFIG. 8, at step S420, the calculated local motion vectors are clustered to select at least parts of the feature points. A local motion vector different from plural local motion vectors having similar direction and magnitude may correspond to a motion of a moving object existed in the target image TGIMG, or to an inaccurate motion estimate due to noise. The individual motion of the object or the inaccurate motion estimate may not be related to a wobble direction of the image. For this reason, the plural local motion vectors having similar direction and magnitude may be selected, and a local motion vector different therefrom may not be selected. Referring toFIG. 10, a local motion vector of each feature point of the target image TGIMG is shown. The local motion vectors may be clustered to select a cluster having the largest number of local motion vectors. InFIG. 10, local motion vectors that are excluded from a cluster of the largest number of local motion vectors having similar direction and magnitude are highlighted with dotted circles. The local motion vectors highlighted with dotted circles may be unselected and the remaining local motion vectors may be selected.

FIG. 11is a flowchart of another exemplary embodiment of step S310ofFIG. 7.

Referring toFIG. 11, at step S510, local motion vectors of the feature points are calculated. Step S510may be described in the same way as step S410ofFIG. 8. Hereinafter, overlapping description will be omitted to avoid redundancy.

At step S520, at least parts of the feature points are selected by comparing the calculated local motion vectors to motion data received from a gyro sensor. The image capturing device shown inFIG. 1may further include the gyro sensor configured to detect a physical motion (e.g., wobble) of the image capturing device. The motion data may include data obtained by converting angular velocity measured by the gyro sensor into pixel data of the image. The image processing device100shown inFIG. 2may communicate the gyro sensor to receive the angular velocity, and convert the received angular velocity into the motion data.

According to the illustrated embodiment, local motion vectors in a range similar to direction and magnitude of the motion data may be selected. The local motion vectors that are not in the range similar to the direction and magnitude of the motion data may be unselected from the calculation of the global motion vector.

FIG. 12is a schematic diagram for illustrating an exemplary embodiment of grouping feature points of step S320ofFIG. 7.

Referring toFIG. 12, feature point groups including feature points FP are shown as being divided with dotted boxes. The feature points FP included in one dotted box may form a feature point group. In an exemplary embodiment, each feature point group may include feature points FP adjacent to each other. InFIG. 12, the sub blocks SBLK may be grouped into unit block UNBLK, feature points FP included in the unit block UNBLK may be defined as the feature point group. However, the exemplary embodiments are not limited thereto, and the feature points FP may be grouped in various manners.

According to the exemplary embodiments and the principles of the invention, the local motion vectors are determined for feature point groups, respectively, to determine the global motion vector, the number of the feature points FP included in one feature group increases as the gain GN of the image sensor200increases. In case where the feature points FP included in one unit block UNBLK are defined as the feature point group, the size of the unit block UNBLK may increase as the gain GN of the image sensor200increases. That is, the number of the sub blocks SBLK included in each of the unit blocks UNBLK may be adjusted depending on the gain GN of the image sensor200. Accordingly, each of the local motion vectors may have a high reliability in reflecting a wobble of the image, and thus the global motion vector calculated according to the local motion vectors may also have a high reliability.

FIG. 13is a graph illustrating the relationship between a gain of an image sensor and the number of the feature points included in a feature point group according to the principles of the invention.

Referring toFIG. 13, the number of the feature points included in the feature point group is: a first value X1when the gain GN of the image sensor200is lower than a first level G21; and a second value X2when the gain GN of the image sensor200is higher than a second level G22. The number of the feature points included in the feature point group increases between the first value X1and the second value X2as the gain GN of the image sensor200increases between the first level G21and the second level G22. As such, the number of the feature points included in the feature point group is adaptively adjusted depending on the gain GN of the image sensor200.

In an exemplary embodiment, the local motion vector of the feature point group may be determined as a local motion vector of a feature point having the lowest SAD among the feature points of the feature point group. In this case, given that the SAD for each of the feature points has already been calculated through Equation 1, the local motion vector of the feature point group may be determined in a relatively short time. In another exemplary embodiment, the local motion vector of the feature point group may be determined as an average of the local motion vectors of the feature points included in the feature point group.

In still another exemplary embodiment, SAD of the feature point group may be calculated based on the following equation 2.

Referring to equation 2, SAD of the feature point group SADG(u,v) may be recalculated by averaging SADx,y(u, v) that is calculated for each of k feature points [(x1, y1), (x2, y2), (x3, y3), . . . , (xk, yk)] of the feature point group by taking the displacement u and v as inputs. k is the number of the feature points included in the feature point group. Performing this calculation repeatedly, the displacement u and v that outputs the lowest SADG(u,v) may be determined as the local motion vector of the feature point group.

FIG. 14is a block diagram of an exemplary computer system suitable for implementing exemplary embodiments of the image capturing device.

Referring toFIG. 14, a computer system100includes a lens1100, an image sensor1200, a motor controller1300, a motor1350, a user interface1400, a RAM1500, a storage1600, a display1700, and a digital signal processor1800.

The lens1100receives an optical signal from a subject. The optical signal received through the lens1100is converted into an electrical signal. The image sensor1200may digitize the electrical signal with a proper gain to generate an image. For example, the image sensor1200may include an analog-to-digital converter.

The motor controller1300is configured to control an operation of the motor1350in response to a control of the digital signal processor1800, the motor1350is configured to operate the lens1100.

The user interface1400detects a user's input for controlling an operation of the computer system1000, and generates a corresponding command. The user interface1400may include an input device able to detect information caused by the user's input, such as a key pad, a mouse, a finger scan sensor, a dome switch, a touch pad, a jog wheel, and the like.

The RAM1500may include at least one of various types of memories, such as SRAM (Static RAM), DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), and the like. The RAM1500may be provided as a working memory of the digital signal processor1800. The digital signal processor1800may include a working memory that is seperated from the RAM1500. For example, the RAM1500may temporarily store images received from the image sensor1200, and may provide the digital signal processor1800with the stored images.

The storage1600may include various types of storage devices able to store data even when the power is off, such as a flash memory, a hard disk, and the like.

The display1700displays information processed by the computer system1000in response to the control of the digital signal processor1800. For instance, the display1700displays the image TGIMG′ shown inFIG. 2corrected by the digital signal processor1800.

The digital signal processor1800is configured to control overall operations of the computer system1000. The digital signal processor1800may operate in response to an input command received through the user interface1400, and may perform the functions of the image processing device100and the sensor gain controller300described with reference toFIG. 2. The digital signal processor1800may include a processor for executing one or more applications performing commands causing operations or steps described with reference toFIGS. 2, 3, 4, 7, 8, and11. The digital signal processor1800may load program codes corresponding to the applications and execute the program codes to operate the applications. The program codes and commands may be stored in the storage1600.