Image sensor cropping images in response to cropping coordinate feedback

Reducing consumption of image sensor processor bandwidth includes capturing an image containing subject matter with an image sensor and cropping the image to generate a cropped image. Cropping the image is performed by the image sensor in response to coordinates received from an image sensor processor. The cropped image is sent from the image sensor to the image sensor processor and new coordinates based on a position of the subject matter in the cropped image are determined with the image sensor processor. The new coordinates are then sent to the image sensor.

TECHNICAL FIELD

This disclosure relates generally to image sensors and in particular but not exclusively, relates to anti-shaking technologies.

BACKGROUND INFORMATION

Image stabilization is important to image sensor technologies because it enhances final image quality. Image sensor movement during image capture sometimes distorts final image(s) as a result of the subject matter in the captured frame(s) changing during exposure. In still cameras, movement during image acquisition is particularly problematic at slow shutter speeds or when using lenses with a long focal length. In video cameras, camera shake can cause a noticeable frame-to-frame wiggle in the recorded images. To combat this problem many physical and electronic methods have been employed. Examples of such methods include the use of mechanical camera stability equipment, microelectronic sensors/actuators, and hardware/software to electronically correct images. Sometimes, electronic image stabilization is performed by (1) taking a much larger image than needed during the exposure period, (2) sending the image to the image sensor processor, and (3) cropping the captured frames to a smaller size using the image sensor processor. By doing this, the image sensor will likely retain the intended image in its field of view during acquisition and will be able to crop out differences between each frame. However, sending such large images to the image sensor processor may result in wasted bandwidth usage since much of the image is ultimately removed. Hence, reducing the bandwidth load on the image sensor processor would be desirable.

DETAILED DESCRIPTION

Embodiments of a system and method for image sensors cropping images in response to cropping coordinate feedback are described herein. In the following description, numerous specific details are set forth to provide a thorough understanding of the embodiments. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects.

FIG. 1is a block diagram illustrating an imaging system100including an image sensor101, image sensor processor103, and a display105. During operation, image sensor101captures an image that is larger than an output image that is output to display105. Image sensor101then sends the captured image to image sensor processor103. Image sensor processor103crops the captured image to generate a smaller output image according to an image stabilization algorithm. The output image is then sent to display105. Although imaging system100facilitates image stabilization, it rapidly consumes image sensor processor103bandwidth.FIG. 1includes “1.1W×1.1H” and “1W×1H” labels to note the size of the images in some conventional imaging systems when they are at image sensor101, image sensor processor103, and display105, respectively.

FIG. 2is a block diagram illustrating imaging system200including an image sensor201coupled to receive cropping coordinate feedback211from an image sensor processor203, in accordance with an embodiment of the disclosure. Imaging system200may further include display205. Imaging system200includes image sensor201to capture an image containing subject matter. Image sensor201is configured to crop the image to generate a cropped image207in response to receiving previous coordinates (cropping coordinate feedback211) of the subject matter. Cropped image207may be 1.03W×1.03H of the image captured by image sensor201. Image sensor processor203is coupled to receive cropped image207from image sensor201and configured to determine coordinates of cropped image207based on a position of the subject matter in cropped image207. In one example, subject matter in cropped image207may include a landscape, animal, person, and/or object. Image sensor processor203is coupled to send the coordinates (cropping coordinate feedback211) of the subject matter to image sensor201, and configured to crop cropped image207to generate output image209in response to the coordinates. Output image209may be 1.0W×1.00H of the image captured by image sensor201. This configuration of imaging system200facilitates image stabilization by cropping the captured image to include the relevant subject matter in output image209.

By cropping the image with image sensor201, and then sending cropped image207from image sensor201to image sensor processor203, a reduction in image sensor processor203bandwidth usage is achieved along with mitigation of image shake. Reduction of bandwidth usage may both conserve power and allow image sensor processor203to handle larger images. In one embodiment, image sensor201captures an image that is 1.1 times the width and 1.1 times the height of output image209. In the same or another embodiment, cropped image207is 1.03 times the width and 1.03 times the height of output image209. This image size reduction prior to sending cropped image207to image sensor processor203, may result in a decrease of image sensor processor203bandwidth usage by ˜15%.

In one embodiment, determining the coordinates of cropped image207may include analyzing movement between cropped image207and a previously cropped image. Analyzing movement between cropped image207and the previously cropped image may include determining local movement between cropped image207and the previously cropped image. The previously cropped image may be cropped from a previous image that was captured by image sensor201immediately preceding the capture of the image. Analyzing movement between cropped image207and the previously cropped image may also include determining global movement between cropped image207and the previously cropped image, wherein the previously cropped image was cropped from a previous image that was a first image captured by image sensor201in a series of images that includes the image and the previous image.

In one embodiment, an electrical output may be coupled to receive output image209from image sensor processor203. In one example, the electrical output includes display205. In another example, the electrical output includes non-volatile memory.

FIG. 3illustrates a flow chart300showing a method for reducing consumption of image sensor processor bandwidth, in accordance with an embodiment of the disclosure. The order in which some or all of the process blocks appear in process300should not be deemed limiting. Rather, one of ordinary skill in the art having the benefit of the present disclosure will understand that some of the process blocks may be executed in a variety of orders not illustrated, or even in parallel.

In process block301, an image containing subject matter is captured with an image sensor (e.g. image sensor201). In one embodiment, capturing the image includes capturing an image that is 1.1 times the width and 1.1 times the height of the output image.

In process block303, the image sensor receives coordinates from the image sensor processor. In process block305, the image is cropped by the image sensor (e.g. image sensor201) to generate a cropped image (e.g. cropped image207). Cropping the image is performed by the image sensor in response to the coordinates received from image sensor processor in process block303. In one embodiment, cropping the image to generate a cropped image includes cropping raw image data. In the same or another embodiment, cropping the image to generate a cropped image includes cropping the image to a size that is 1.03 times the width and 1.03 times the height of the output image.

The cropped image is sent from the image sensor (e.g. image sensor201) to the image sensor processor (e.g. image sensor processor203), in process block307. In one embodiment, a 15% bandwidth usage reduction may be achieved by sending the image sensor processor an image smaller than 1.1 times the width and 1.1 times the height of the output image.

In process block309, new coordinates based on a position of the subject matter in the cropped image are determined using the image sensor processor (e.g. image sensor processor203). Determining the new coordinates based on the position of the subject matter may include determining the new coordinates based on movement between the cropped image and a previously cropped image that was cropped by the image sensor processor prior to cropping the cropped image. In one embodiment, determining new coordinates based on movement between the cropped image and the previously cropped image includes determining local movement between the cropped image and the previously cropped image, wherein the previously cropped image was cropped from a previous image that was captured by the image sensor immediately preceding the capture of the image. In another embodiment, determining new coordinates based on movement between the cropped image and the previously cropped image includes determining global movement between the cropped image and the previously cropped image, wherein the previously cropped image was cropped from a previous image that was a first image captured by the image sensor in a series of images that includes the image and the previous image.

Process block311shows that the new coordinates are sent to the image sensor (e.g. image sensor201). In process block313, the cropped image is cropped to generate an output image. Cropping the cropped image is performed by the image sensor processor in response to the new coordinates. Process block315shows that the output image may be output. In one embodiment, the output image is output to a display. In another embodiment, the output image is output to non-volatile memory.

It is worth noting that the steps illustrated in flow chart300may repeat themselves an indeterminate number of times. In one embodiment, capturing a subsequent image containing the subject matter with the image sensor and cropping the subsequent image to generate a subsequent cropped image may occur. Additionally, cropping the subsequent image may be performed by the image sensor in response to the new coordinates received from the image sensor processor.

FIG. 4is a diagram illustrating cropping process400performed by the imaging system100inFIG. 1. Several steps and multiple repetitions of cropping process400are shown. First image401is captured with an image sensor (e.g. image sensor101). First image401is larger than the image that is output (e.g. to display105). Then, first image401is sent to an image sensor processor (e.g. image sensor processor103) where cropping coordinates are determined based on an image stabilization algorithm that analyzes subject matter in the first image. The image sensor processor crops first image401to generate a smaller first output image403. Smaller first output image403is then sent to an electrical output (e.g. display105).

Next, second image405is captured. Second image405is larger than the image that is output (e.g. to display105). Then, second image405is sent to an image sensor processor (e.g. image sensor processor103) where cropping coordinates are determined based on an image stabilization algorithm that analyzes subject matter in the second image. The image sensor processor crops second image405to generate a smaller second output image407. Smaller second output image407is then sent to an electrical output (e.g. display105). These steps repeat themselves until the final image or video is achieved. InFIG. 4these steps repeat themselves a third time, wherein third image409is captured by the image sensor, transferred to the image sensor processor and cropped to smaller third output image411.

By sending a larger captured image (e.g. image401,405, or409) to the image sensor processor, image sensor processor bandwidth is quickly consumed. Therefore it is preferable to transfer a smaller image from the image sensor to the image sensor processor to limit the work of the image sensor processor while ultimately still achieving the same anti-shaking functionality.

FIG. 5is a diagram illustrating cropping process500associated with imaging system200illustrated inFIG. 2, in accordance with an embodiment of the disclosure. Several steps and multiple repetitions of cropping process500are shown. A first image501is captured with an image sensor (e.g. image sensor201). In one embodiment, first image501is 1.1 times the height and 1.1 times the width (1.1W×1.1H) of output image505. The image sensor (e.g. image sensor201) is configured to crop first image501to generate a first cropped image503in response to receiving previous coordinates (e.g. cropping coordinate feedback211) of the subject matter. InFIG. 5, subject matter in first cropped image503consists of a light bulb. In one embodiment, first cropped image503is 1.03 times the width and 1.03 times the height of first output image505. The image sensor processor (e.g. image sensor processor203) is coupled to receive first cropped image503from the image sensor, and configured to determine first coordinates of first cropped image503based on a position of the subject matter (e.g. the light bulb) with first cropped image503. The image sensor processor is coupled to send the first coordinates of the subject matter to the image sensor (e.g. cropping coordinate feedback211), and configured to crop first cropped image503to generate first output image505in response to the first coordinates. Sending the coordinates of the subject matter to the image sensor allows for cropping process500to begin a new cycle. In one embodiment, an electrical output may be coupled to receive first output image505from the image sensor processor. In one example, the electrical output includes a display (e.g. display205). In another example, the electrical output includes non-volatile memory.

Next, second image507is captured with an image sensor (e.g. image sensor201). In one embodiment, second image507is 1.1W×1.1H of output image511. The image sensor (e.g. image sensor201) is configured to crop second image507to generate a second cropped image509in response to receiving first coordinates (e.g. cropping coordinate feedback211) of the subject matter. In one embodiment, second cropped image509is 1.03 times the width and 1.03 times the height of second output image511. The image sensor processor (e.g. image sensor processor203) is coupled to receive second cropped image509from the image sensor, and configured to determine second coordinates of second cropped image509based on a position of the subject matter. The image sensor processor is coupled to send the second coordinates of the subject matter to the image sensor (e.g. cropping coordinate feedback211), and configured to crop second cropped image509to generate second output image511in response to the second coordinates. In one embodiment, an electrical output may be coupled to receive second output image511from the image sensor processor.

The steps of cropping process500may repeat themselves indefinitely until the final image or video is achieved. InFIG. 5, these steps repeat themselves a third time, wherein third image513is captured by the image sensor, cropped by the image sensor to generate third cropped image515, transferred to the image sensor processor and cropped to the W×H third output image517.

In the embodiment depicted inFIG. 5, the subject matter (e.g. light bulb) is centered in image501. This means that opposite edges of output image505are located equidistant from the edges of first image501that run parallel to the edges of first output image505. Previous coordinates (e.g. cropping coordinate feedback211) of the subject matter may be coordinates (with respect to pixels of image501) of the lower left hand corner of first cropped image503. In one embodiment, the location of the lower left hand corner of first cropped image503may be previous coordinates [20, 20] where the X and Y axis are represented by the bottom edge of image501and the left edge of image501respectively, and [0, 0] (X and Y intercept) is represented by the lower left hand corner of image501. The image sensor (e.g. image sensor201) is configured to crop first image501to generate first cropped image503in response to receiving previous coordinates [20, 20]. Cropping may occur by removing sections of image501that do not fit into the area of first cropped image503, where location of first cropped image503is determined by the coordinates of first cropped image503's lower left hand corner (e.g. [20, 20]). The image sensor processor (e.g. image sensor processor203) is coupled to receive first cropped image503from the image sensor, and configured to determine first coordinates of first cropped image503based on a position of the subject matter (e.g. the position of the light bulb). InFIG. 5, since the subject matter (e.g. light bulb) remained in a centered position, the image sensor processor will determine that the first coordinates remain [20,20]. The image sensor processor will then send the first coordinates (e.g. cropping coordinate feedback211) of the subject matter to the image sensor, and crop first cropped image503to generate first output image505in response to the first coordinates.

Second image507is then captured with an image sensor (e.g. image sensor201). The image sensor is configured to crop second image507to generate second cropped image509in response to receiving first coordinates (e.g. cropping coordinate feedback211) of the subject matter. In the depicted example, first coordinates are [20, 20]. Subsequently, the image sensor will crop second image507in the same way it cropped first image501and send the image sensor processor second cropped image509. The image sensor processor will determine second coordinates of second cropped image509based on a position of the subject matter (e.g. light bulb). Here, in second image507, the position of the subject matter has shifted to the right and down. This means that the cropping location should be shifted to the right and down. Accordingly, second coordinates may be coordinates [0, 40]. It should be noted that the subject matter (e.g. light bulb) is still contained in the second image. The image sensor processor will then send the second coordinates of the subject matter to the image sensor, and crop second cropped image509to generate second output image511in response to the second coordinates.

Lastly inFIG. 5, the image sensor will capture third image513. The image sensor will crop third image513in response to receiving second coordinates of the subject matter (e.g. [0, 40]) to generate third cropped image515. Third cropped image515is then sent to the image sensor processor. The image sensor processor will determine third coordinates of third cropped image515based on a position of the subject matter (e.g. light bulb). Here, in third image513, the position of the subject matter has shifted to the left and up. Accordingly, third coordinates may be [10, 30]. The image sensor processor will then send the third coordinates of the subject matter to the image sensor, and crop third cropped image515to generate third output image517in response to the third coordinates.

It has been observed that cropping process500is desirable because global movement (the difference between the first frame captured and the most recent frame captured) during a capture period is rarely more than 10% of output image size (e.g. output images505,511, or517). Additionally, local movement (the difference between two frames in sequence) is rarely greater than 3% of output image size. By first cropping to a 1.03×1.03 image based on the coordinates of the subject matter in the last image captured, the subject matter will still be contained in the cropped image. Therefore the 1.03×1.03 image may be passed to the image sensor processor to conserve image sensor processor bandwidth while the subject matter of the image sensor is still contained in the cropped image.

In one embodiment the image sensor processor calculates cropping coordinates based on global movement and feeds these global movement cropping coordinates back to the image sensor. In one example, the image sensor processor passes to the image sensor coordinates of one corner of a 1.03W×1.03H frame used to crop a 1.1W×1.1H image (e.g. image501,507, or513). In the same or another embodiment, the image sensor processor may first calculate the local movement between a second cropped image and a third cropped image and use the local movement to calculate the global movement of the third cropped image relative to the first cropped image. The global image movement may then be fed back for use by the image sensor. It should be noted that cropping coordinates may also be comprised of edge values of the frame used to crop a captured image (e.g. image501,507, or513), and that the frame used to crop may be rectangular or other geometry.