Auto focus technique in an image capture device

Multiple sets of pixel values representing a captured image of a scene are received, with each set representing an image captured with a corresponding degree of focus. An image processor may identify a region of interest in the captured image, automatically determine the configuration parameters for a lens assembly to provide a desired degree of focus for the region of interest, and generate signals to configure a lens assembly. In an embodiment, the region of interest is a face, the desired degree of focus of the face is determined by computing a rate of variation of luminance of pixels representing the face, and the desired degree is the degree of the image having the maximum degree of focus.

RELATED APPLICATIONS

The present application is related to the following co-pending US applications, which are both incorporated in their entirety herewith:

BACKGROUND

The present disclosure relates generally to the field of image processing, and more specifically to an auto-focus technique in image capture devices such as still picture cameras and video cameras.

2. Related Art

An image capture device (ICD) generally refers to a device which captures images of scenes. When designed to capture a single image of a scene, the device is referred to as a camera. On the other hand, when continuous images of a scene are captured by a device, the device is often referred to as a camcorder. The images can be captured in digital form or analog form. To capture images in digital form, sensors may be arranged in the form of an array, with each sensor in the array forming a digital value representing a point (small portion) of the scene. On the other hand, in analog form, a light-sensitive medium is designed to respond to incident rays to capture the image. In general, the sensors and other media which capture the images are referred to as capturing media.

ICDs also generally contain a lens assembly, which receives light rays representing a scene sought to be captured, and converges the light onto the capturing medium. The lens assembly may contain one or more lenses, which can be configured to obtain various degrees of convergence. In one embodiment, configuration entails moving the absolute and/or relative position of individual lenses (typically along an imaginary axis connecting the lens assembly to the capturing medium) contained in a lens assembly. However, configuration can entail any other changes (e.g., effecting curvature change, etc.) as provided by the present/future underlying technology to effect focusing (described below) of the received light rays to the capturing media.

It is generally required to focus on a scene of interest before capturing the scene. Focusing refers to configuring a lens assembly such that the incident light rays are made to converge on the capturing medium. As a simplified illustration, it may be appreciated that parallel light rays originating at infinite distance would be received in parallel and would converge at a distance equaling a focal length of the lens assembly.

As a scene gets closer, the rays may be less than parallel (i.e., divergent), and thus the distance between the lens assembly and the capturing medium may need to be correspondingly more than the focal length (assuming the focal length of the lens assembly does not change). In general, a resulting captured image is rendered distinct and clear when the focusing is accurate. When the lens assembly is focused less than accurately, the clarity of the captured image is generally correspondingly less.

Several ICDs provide a manual mechanism by which a user can configure lens assembly to obtain a desired focus, typically while viewing the image through the lens assembly. Assuming only the position of the lens needs to be adjusted for focusing, the lens position is changed directly in response to the user operation of the manual mechanism. Once a desired focusing is set, the user then captures the image. Such manual mechanism to configuring lens assembly may be termed as manual focusing.

On the other hand, there has been a general recognized need for auto-focus ICDs. Auto-focusing generally refers to determining the appropriate parameters (e.g., distance in the examples above) to configure the lens assembly, and performing the determined configuration by using appropriate computations and configurations within an ICD.

DETAILED DESCRIPTION

Overview

A processor provided according to an aspect of the present invention receives multiple sets of pixel values, with each set of pixel values representing a corresponding image of a scene captured using a corresponding set of configuration parameters for a lens assembly. The processor identifies a region of interest of the scene, and determines the degree of focus present in the region for each image. A desired set of configuration parameters are then determined based on the degrees of focus and corresponding sets of configuration parameters. The processor then generates signals to configure the lens assembly according to the desired set of configuration parameters.

In an embodiment, the region is determined to be an image portion representing a face within the scene such that a user of a image capture device can capture images focused on faces. However, to the extent other types of objects can be characterized for reliable ‘recognition’ within image capture devices using appropriate image processing mechanisms, auto-focus may be based on such other types of objects as well.

Another aspect of the present invention provides a reliable approach to determine image portions representing a desired object (e.g., skin) in a scene. In one embodiment, the two chrominance components and the sum of the two components are checked to determine whether each of the three values is in a respective range. If the three values are in the respective ranges, the pixel value is deemed to represent skin. Further processing may be performed to recognize the region representing the desired object from such points.

Camera

FIG. 1is a block diagram of a camera used to illustrate an example embodiment in which several aspects of the present invention may be implemented. While the description is provided with respect to a still camera (which facilitates a user to capture a single or a few images of a scene) merely for illustration, it should be appreciated that the features can be implemented in various other image capture devices without departing from the scope and spirit of several aspects of the invention, as will be apparent to one skilled in the relevant arts by reading the disclosure provided herein.

Camera100is shown containing lens enclosure110, lens assembly115, image sensor array120, image processor130, display140, non-volatile memory150, input (I/P) interface160, motor170, digital to analog (D/A) converter180, and RAM190. Only the components as pertinent to understanding of the operation of the example embodiment are included and described, for conciseness and ease of understanding. Each component ofFIG. 1is described in detail below.

Lens enclosure110is shown housing lens assembly115and image sensor array120, and is generally designed to shield extraneous (i.e., other than the light being received via the lens assembly) light from being incident on image sensor array120(in general, capturing medium). Lens assembly115may contain one or more lenses, which can be configured to focus light rays (denoted by arrow105) from a scene to impinge on image sensor array120.

In one embodiment, lens assembly115is moved relative to image sensor array120along an axis (shown as dotted line112) by appropriate operation of motor170to obtain a desired degree of focus when capturing an image received on path105. However, depending on the technology available, various alternative structures/material may be employed for lens assembly and the corresponding controls may be presented on path171to achieve a desired configuration.

D/A converter180receives a digital value from image processor130(on path138) representing a distance by which lens assembly115is to moved, and converts the digital value to an analog signal which is provided to motor170via path187. It should be understood that the digital value represents an example configuration parameter used to configure lens assembly115. However, depending on the implementation of lens assembly115, additional/different parameters may be received as well.

Motor170is coupled to lens assembly115, and operates to adjust the absolute position of lens assembly115, and/or relative positions of individual lenses within lens assembly115, in response to receiving a corresponding analog signal from D/A converter180. Motor170may be coupled to lens assembly115in a manner facilitating movement of either the whole of lens assembly115or individual lenses within it along an axial direction indicated by dotted line112.

Display140displays an image frame in response to the corresponding display signals received from image processor130on path134. Display140may also receive various control signals (not shown) from image processor130indicating, for example, which image frame is to be displayed, the pixel resolution to be used etc. Display140may also contain memory internally for temporary storage of pixel values for image refresh purposes, and is implemented in an embodiment to include an LCD display.

Input interface160provides a user with the facility to provide inputs, for example, to select features such as whether auto-focus is to be enabled/disabled. The user may be provided the facility of any additional inputs, as described in sections below.

Image sensor array120may contain an array of sensors, which together generate digital values representing an image represented by light rays received via lens assembly115. Each sensor may generate a digital value representing the corresponding point (small portion) of the image. The digital value can be in RGB format, with each component value being proportionate the corresponding color intensity and time of exposure (shutter not shown). Image sensor array120forwards the array of digital values as a stream sequentially to image processor130on path122for further processing. In an embodiment, image sensor array120is implemented as a CCD (charge coupled device)/CMOS sensor array.

RAM190stores program (instructions) and/or data used by image processor130. Specifically, pixel values that are to be processed and/or to be user later, may be stored in RAM190via path139by image processor130.

Non-volatile memory150stores image frames received from image processor130via path135. The image frames may be retrieved from non-volatile memory150by image processor130and provided to display140for display. In an embodiment, non-volatile memory150is implemented as a flash memory. Alternatively, non-volatile memory150may be implemented as a removable plug-in card, thus allowing a user to move the captured images to another system for viewing or processing or to use other instances of plug-in cards.

Non-volatile memory150may contain an additional memory unit (e.g. ROM, EEPROM, etc.), which store various instructions, which when executed by image processor130provide various features of the invention described herein. In general, such a memory unit (including RAMs, non-volatile memory, removable or not) from which instructions can be retrieved and executed are referred to as a computer readable medium. It should be appreciated that the computer readable medium can be deployed in various other embodiments, potentially in devices, which are not intended for capturing images, but providing several features described herein.

Image processor130forwards pixel values received on path113to path134to enable a user to view the scene presently pointed by the camera. In addition, when the user ‘clicks’ a button (indicating intent to record the captured image on non-volatile memory150), image processor130causes the pixel values representing the present (at the time of clicking) image to be stored in memory150.

In addition, image processor130may configure lens-assembly115to auto-focus on a scene according to several aspects of the present invention, as described below.

FIG. 2is a flowchart illustrating the manner in which lens assembly115may be configured to achieve auto-focus on a scene. The flowchart is described with respect toFIG. 1, and in relation to image processor130, merely for illustration. However, various features can be implemented in other environments and other components. Furthermore, the steps are described in a specific sequence merely for illustration.

Alternative embodiments in other environments, using other components, and different sequence of steps can also be implemented without departing from the scope and spirit of several aspects of the present invention, as will be apparent to one skilled in the relevant arts by reading the disclosure provided herein. The flowchart starts in step201, in which control passes immediately to step210.

In step210, image processor130receives multiple sets of pixel values. Each received set of pixel values represents an image of a scene captured by camera100using a corresponding set of configuration parameters for the lens assembly. Each set of configuration parameters are designed to focus the lens assembly with different degrees of focus at a corresponding time instance. It is also assumed that each image captures the same scene, but with correspondingly different degrees of focus (illustrated below with respect toFIGS. 3A-3Dbelow).

In an embodiment, each set of pixel values received represents a subsampled version (obtained by subsampling an image captured by image sensor array120) of the image, wherein each pixel value in the subsampled version is generated by averaging the pixel values of a group (e.g., 3×3 array of pixel values) of adjacent pixels image captured by lens assembly115. Control then passes to step230.

In step230, image processor130identifies a region of interest within (i.e., a part of) each image (corresponding to each set of pixel values received in step210). Image processor130may be implemented with appropriate logic (rules, processing logic, etc.) to identify the region of interest. If different types of regions (e.g., face, grass, sky, etc.) are of possible interest, appropriate inputs may be provided to image processor130to identify the specific type of region of interest within the scene. Control then passes to step240.

In step240, image processor130determines a set of configuration parameters (for lens assembly115) that provides a desired degree of focus for the image. In an embodiment described below, the desired degree of focus equals the focus of one of the images having best focus, and the configuration parameters corresponding to such an image are selected. However, alternative techniques can be employed which determine the desired degree of focus based on the degree of focus present in the regions of step230. Control then passes to step260.

In step260, image processor130generates signals to focus lens assembly115using the set of configuration parameters determined in step240. In an embodiment, image processor130provides the configuration parameters as digital values to a D/A converter (such as D/A converter180), which in turn generates a corresponding analog signal to cause a motor (such as motor170) coupled to lens assembly115to adjust the position of lens assembly115to provide the desired degree of focus. Control then passes to step270.

In step270, image processor130enables a user to capture the image. Image processor130may indicate to a user that the lens assembly115has been auto-focused, for example, by displaying a message on display140. The user may then ‘click’ to capture the image. In response to a user's ‘click’, image processor130stores a set of pixel values of a ‘focused’ image captured using the determined set of configuration parameters. Alternatively, the user may merely click once and the auto-focusing described above may be performed along with capturing of the image in response to a single click. Control passes to step299, in which the flowchart ends.

In the description above, although image processor130is described as receiving multiple sets of pixel values, each corresponding to a different configuration of the lens assembly, it must be understood that image processor130may provide the configuration parameters corresponding to each of the configurations (to cause the lens assembly to be configured).

In particular, image processor130may adaptively determine the set of configuration parameters until a desired degree of focus is obtained. In such an adaptive technique, image processor130may receive a single set of pixel values in each iteration, determine if the degree of focus is as desired, and provide a new set of configuration parameters for the next iteration based on the degrees of focus identified for the previous iterations. Image processor130may then repeat such combination of steps until a desired degree of focus is obtained.

As noted above in step210, the set of configuration parameters for lens assembly115determines how focused (degree of focus) a captured image will be. This is clarified with an illustration below.

Degree of Focus

FIGS. 3A through 3Dillustrate the variation of focus with the configuration parameters used for lens assembly115. The figures are illustrated with respect to a point Po and its corresponding image Pi captured by image sensor array120. In each ofFIGS. 3A through 3D, point Po is located at the same distance from image sensor array120, but lens assembly115is shown located at four different positions from the image sensor array. It must be understood thatFIGS. 3A through 3Dare illustrated merely to clarify the term ‘degree of focus’ and its variation with configuration parameters of lens assembly115, and as such are not intended to depict the actual focus phenomenon precisely.

As may be seen fromFIG. 3A, representative light rays (depicted as lines emanating from point Po) converge at a single point Pi which lies on image sensor array120. This configuration of lens assembly115results in the image Pi of point Po being focused on image sensor array120.

InFIG. 3B, lens assembly115is farther away from image sensor array120as compared to the case inFIG. 3A(or equivalently, lens assembly115is configured using a set of parameters such that image of point Po is focused at point ‘A’ on a plane310rather than on image sensor array120). It may be seen fromFIG. 3Bthat the image Pi captured by image sensor array120now falls over an area instead of being a point as inFIG. 3A. This corresponds to a ‘degree of focus’ which is less than optimal focus ofFIG. 3A.

FIG. 3Cillustrates a situation wherein lens assembly115is closer to image sensor array120(as compared to inFIG. 3A). It may be observed that the image of Po is focused on a plane320at a point B, while the corresponding image captured by image sensor array is again an area instead of a point. Again, the degree of focus achieved inFIG. 3Cis less than the optimal case ofFIG. 3A.

Similarly,FIG. 3Dillustrates a situation wherein lens assembly115is even closer to image sensor array120than shown inFIG. 3C. It may be observed that the image of Po is focused on a plane330at a point C, while the corresponding image captured by image sensor array is represented by an even greater area than inFIG. 3C. The degree of focus inFIG. 3Cis thus more than that inFIG. 3D. InFIG. 3Awe have the maximum degree of focus possible.

The manner in which such degree of focus can be digitally characterized and used in auto-focusing is described below with an example.

Digital Characterization of Degree of Focus

It may also be appreciated from the description above that inFIGS. 3B through 3Dthe image of point Po captured by image sensor array120is a ‘diffused’ image (spread over an area, instead of being a point), and as such would be less ‘sharp’ as compared to that inFIG. 3A. Further, the extent of “sharpness” would vary depending on the degree of focus resulting from the particular configuration of lens assembly115. For example, the image captured by image sensor array120inFIG. 3Dwould be less ‘sharp” than inFIG. 3C, since the image is formed over a larger area.

In an embodiment, image processor130computes a rate of variation of luminance across pixel values in a region of interest to determine if a desired degree of focus has been obtained, as described below with an example illustration.

FIGS. 4B and 4Cshow corresponding images of a point420(point420being assumed to correspond to a single pixel) in an object410(in a scene400) as captured by image sensor array120at two example configuration settings respectively of lens assembly115. It is assumed that point420is a ‘white” pixel and has a luminance value 255 (Y value in a YCbCr color space), and pixels to the left and right of point420are ‘black’ pixels each with Y values of 0.

InFIG. 4B, lens assembly115is configured such that point420is not focused (out of focus) on image sensor array120. As a result, the image of point420as captured by lens assembly120falls over several pixels (area) instead of corresponding to a single pixel as might be desired. The image of420is shown inFIG. 4Bas being ‘spread’ over four pixels marked as P, Q, R and S (corresponding to example row and column numbers as shown). All other pixels are assumed to be black for ease of illustration. Pixels P, Q, R, and S may have luminance values less than 255 (of point420), and are assumed, to illustrate the example, to have values 100, 110, 100, and 110 respectively.

Image processor130may compute a rate of variation of luminance of pixel values along a horizontal direction (merely as an example, although any other direction could also be used). Thus, image processor130may note (compute) a variation of luminance values as 0, 100, 110, 0 corresponding to pixels at columns 1 through 4 in row 1.

InFIG. 4C, lens assembly115is configured such that point420is focused on image sensor array120, resulting in a corresponding point image represented by pixel ‘I’ (in row 1, column 2). Thus, image pixel ‘I’ may have a luminance value of 255, and image processor130may note a variation of luminance values as 0, 255, 0 corresponding to pixels at columns 1 through 3 in row 1.

It may be noted that the rate of variation of luminance values in a focused image (as inFIG. 4C) has a higher rate of variation (0, 255, 0) than that (0, 100, 110, 0) in an out-of-focus image (as inFIG. 4B). In general, image processor130may compute a rate of variation of luminance across pixels of an entire row (or column, or any other direction), or across only a subset of pixels in the row (column or direction).

Image processor130, thus, generates ‘rate of change of luminance values’ for each image (corresponding to a configuration setting for lens assembly115), and uses the ‘rate of change of luminance values’ as a measure of the corresponding degree of focus obtained. The rate change can be determined using various high frequency filters, such as Sobel Operator, Laplacian of Gaussian, canny edge detector etc.

As noted above, image processor130, selects a set of configuration parameters such that the ‘rate of change of luminance values’ is maximum within a portion of the images. In an embodiment, the portion corresponds to ‘face’, and face is determined by first determining whether a pixel of an image corresponds to skin. The manner in which pixels corresponding to skin can be identified is described below in further detail.

Identifying Pixels Representing Skin

FIG. 5is a flowchart illustrating the manner in which image processor130identifies pixels representing skin in one embodiment. Again, the flowchart is described with respect toFIG. 1, and in relation to image processor130, merely for illustration. However, various features can be implemented in other environments and other components. Furthermore, the steps are described in a specific sequence merely for illustration. Various alternative embodiments in other environments, using other components, and different sequence of steps can also be implemented without departing from the scope and spirit of several aspects of the present invention, as will be apparent to one skilled in the relevant arts by reading the disclosure provided herein. The flowchart starts in step501, in which control passes immediately to step510.

In step510, image processor130receives a pixel in the form of a luminance value and a pair of chrominance values. In an embodiment, the luminance and chrominance values correspond to the respective values in a YCbCr color space representation (Y corresponding to the luminance value, Cb and Cr corresponding to the pair of chrominance values). Image processor130may receive the pixel value in YCbCr form from an external system (such as image sensor array120), or may internally perform color space conversion of pixel value received in an alternative form (for example, RGB). Control then passes to step520.

In step520, image processor130determines whether the first chrominance value (Cb) lies in a first range of values. Control passes to step540if the value is in the range, else control passes to step590. In an embodiment the first range corresponds to a range of values between 67 and 123 (both inclusive).

In step540, image processor130determines whether the second chrominance value (Cr) lies in a second range of values. Control passes to step560if the value is in the range, else control passes to step590. In an embodiment the second range corresponds to a range of values between 136 and 175 (both inclusive).

In step560, image processor130determines whether the sum of the first chrominance value (Cb) and the second chrominance value (Cr) lies in a third range of value. Control passes to step580if the value is in the range, else control passes to step590. In an embodiment the third range corresponds to a range of values between 220 and 275 (both inclusive).

In step580, image processor130concludes that the pixel represents skin. Control then passes to step599, in which the flowchart ends. In step590, image processor130concludes that the pixel does not represent skin. Control then passes to step599, in which the flowchart ends.

It should be appreciated that the above approach and ranges have been determined based on various experiments and observations. The approach facilitates identifying pixels representing skin with minimal computations (one addition and three comparisons), which facilitates the determination to be performed with minimal processing resources.

It should also be appreciated that approach can be extended to potentially several other objects with appropriate changes to the ranges, as will be apparent to one skilled in the relevant arts by reading the disclosure provided herein.

In one embodiment, to facilitate further processing as regards to auto-focusing, for each pixel in a set of pixels corresponding to an image, image processor130determines whether the pixel represents skin or not in a manner illustrated above, and generates a corresponding ‘skin map’ of the image.

The skin map can be in the form of an array equaling the dimensions of image sensor array120, with each bit of the map indicating whether the output of the corresponding sensor element represents skin or not (as a binary value). Alternatively, the skin map can have the same dimensions as the subsampled image described elsewhere in the present application, for reduced computational complexity.

The skin map may be generated for each frame (or at some intervals) in case of camcorder type devices in which the scene can change during recording. Even in still cameras, the skin map can be computed for every frame received since the degree of focus can have a bearing on the pixel values (used to determine whether the pixels represent skin or not). Alternatively, if it is assumed that the scene does not change, the skin map can be computed based on one of the frames received while attempting auto-focus.

Image processor130may then operate on the skin map as an input, to identify a facial region (an example of a region of interest in one embodiment), as described in detail below. First, the skin map is pre-processed to add additional pixels in a skin map as representing skin and to remove some pixels from being considered skin based on a recognition that points adjacent to skin are likely to be skin and points that are not adjacent to skin are unlikely to be skin.

Pre-processing of Skin Map

FIGS. 6A,6B and7are flowcharts illustrating the manner in the which skin map may be pre-processed prior to determining regions forming a face in one embodiment. Again, the flowchart is described with respect toFIG. 1, and in relation to image processor130, merely for illustration. However, various features can be implemented in other environments and other components. Furthermore, the steps are described in a specific sequence merely for illustration. Various alternative embodiments in other environments, using other components, and different sequence of steps can also be implemented without departing from the scope and spirit of several aspects of the present invention, as will be apparent to one skilled in the relevant arts by reading the disclosure provided herein. The flowchart ofFIG. 6Astarts in step601, in which control passes immediately to step605.

In step605, image processor130receives a (source) skin map specifying which pixels in the image represent skin. The skin map is generated, for example, by image processor130as described above with respect to the flowchart ofFIG. 5. Control then passes to step610.

In step610, image processor130sets a comparison number to a first value. As will be apparent from the description below, the comparison number indicates the number of neighboring pixels representing skin to be present before a subject pixel is also deemed to represent skin. Control then passes to step615.

In step615, image processor130receives a status bit corresponding to a pixel (as a subject pixel) from the skin map received in step605. In an embodiment, the status bit is received as a binary value, wherein a binary 1 (0) specifies that the (first) pixel has been identified as representing skin, and a binary 0 (1) specifies otherwise. Control then passes to step620.

In step620, if the status bit indicates that the pixel represents skin, control passes to step635, else control passes to step625.

In step625, image processor130determines from the skin map the number of pixels neighboring (immediately adjacent to) the subject pixel that also represent skin. Control passes to step630if image processor130determines (from the corresponding values in the skin map) that the number of neighboring pixels representing skin is equal to or greater than the comparison number, else control passes to step635.

It must be noted that the operation of this step may be preformed at a macro-block level (group of pixels, e.g. 3×3 pixel block) instead of at the pixel level described above, i.e., if a predetermined number of macroblocks surrounding a current (subject) macroblock are marked as skin type then the current macroblock is also marked as skin-type macroblock. Each macroblock may initially be marked as skin type of at least a pre-specified number of pixels within the macroblock are indicated to represent skin in the source skin map.

In step630, image processor130updates the skin map to indicate that the pixel represents skin. Control then passes to step635.

In step635, if image processor130determines that all pixels in the skin map have been processed control passes to step645, else control passes to step640. In step640, image processor130receives a status bit of a next pixel from the skin map. Control then passes to step620.

In step645, image processor130stores the updated skin map. Control then passes to step655. In step650, image processor130sets the comparison number to a new value. Control then passes to step655. In step655, image processor130receives a status bit corresponding to a pixel (as a subject pixel) from the skin map received in step605. Control then passes to step660.

In step660, if the status bit indicates that the pixel represents skin, control passes to step665, else control passes to step675. In step665, image processor130determines from the skin map the number of pixels neighboring (immediately adjacent to) the subject pixel that also represent skin. Control passes to step675if image processor130determines (from the corresponding values in the skin map) that the number of neighboring pixels representing skin is equal to or greater than the comparison number, else control passes to step670.

In step670, image processor130updates the skin map to indicate that the pixel does not represent skin. Control then passes to step675.

In step675, if image processor130determines that all pixels in the skin map have been processed control passes to step685, else control passes to step680.

In step680, image processor130receives a status bit of a next pixel from the skin map. Control then passes to step660. In step685, if image processor130determines that pre-processing of the skin has been performed to a desired level of confidence, control passes to step695, else control passes to step690. In an embodiment, a desired level of confidence level is deemed to be reached if a predetermined number of iterations of steps615through675have been performed. However, different approaches (e.g., based on different criteria such as number of pixels added/removed in an iteration) can be undertaken until a desired level of confidence is attained.

In step690, image processor130sets the comparison number to a new value. In an embodiment, the comparison value set in this step is smaller than the comparison values set in the first iteration of steps610and650. Control then passes to step615, in which image processor130receives the status bit for a first pixel again from the skin map, and the operations of the flowchart are repeated.

In step695, image processor130stores the pre-processed (updated) skin map for further processing. Control then passes to step699, in which the flowchart ends.

In the flowchart ofFIG. 6Ball steps are identical to correspondingly (similarly) numbered steps inFIG. 6A, except for the following:a) Step610is not present, and control passes to step615after execution of step605.b) In step625, if image processor130determines that if at least one neighboring pixel is present which is indicated by the skin map as representing skin, control passes to step630, else control passes to step635.c) If in step675, image processor130determines that all pixels in the skin map have been processed, control passes to step695in which the updated skin map is stored, and then to step699in which the flowchart ends.

It may be observed from the flowchart ofFIG. 6Athat several iterations of each of the two loops formed by steps615-620-625-630-635-640-620, and by steps655-660-665-670-675-680-660may be performed, while only one iteration of each of the loops is performed in the flowchart ofFIG. 6B.

The skin maps stored at the end (i.e., step695) of flowcharts ofFIGS. 6A and 6Bare then processed as illustrated inFIG. 7. Merely for reference, the skin maps generated at the end of processing by the flowcharts ofFIGS. 6A and 6Bare respectively referred to as skin map A and skin map B. The flowchart ofFIG. 7starts in step701, in which control passes immediately to step710.

In step710, image processor130receives the updated skin maps (A and B) stored at end (step695) of Flowcharts ofFIGS. 6A and 6B. Control then passes to step720.

In step720, for each pixel location in skin maps A and B, image processor130performs a logical AND operation of the corresponding status bits, and writes the ANDed value to the corresponding pixel location in a “final” skin map. Control then passes to step730.

In step730, image processor stores the ‘final’ skin map for further processing. Control then passes to step799in which the flowchart ends.

It may be observed that the operations of the steps of flowcharts6A,6B and7may add additional pixels in a skin map as representing skin and remove some pixels from being considered skin. In particular, the loop formed by steps615-620-625-630-635-640-620operates to mark ‘non-skin’ pixels as skin-pixels, and the loop formed by steps655-660-665-670-675-680-660operates to remove skin pixels from being considered skin.

As an example, pixels representing eye, although not of face color, would need to be identified (and included) as part of a facial region. The loop formed by steps615-620-625-630-635-640-620may cause addition of ‘skin’ pixels (pixels in the eye region added as ‘skin pixels) to the skin map, and thus enables identification of such pixels also as potentially lying in a facial region.

Noise and other undesirable effects may erroneously cause a pixel otherwise not representing skin to be captured as a ‘skin’ pixel in the skin map prior to the processing ofFIG. 6. The loop formed by steps655-660-665-670-675-680-660may cause such a pixel to be removed from the pixel map. Also, it is noted that that the operation of the steps of flowcharts6A and6B, and the ANDing operation in flowchart ofFIG. 7may be performed to prevent or minimize the probability of two separate but closely spaced skin clusters from merging. At the completion of pre-processing, the ‘final’ skin map (obtained at step730ofFIG. 7) may contain one or more ‘clusters’ of pixels (a group of adjoining/contiguous pixel locations) identified as skin pixels, each cluster potentially representing a facial region. Image processor130may then mark the boundaries of each of such clusters with a bounding rectangle, as described next.

Identifying and Marking Potential Facial Regions

FIG. 8is a flowchart illustrating the manner in which image processor130identifies and marks one or more cluster (group) of pixels as potentially representing a desired facial region in one embodiment. The flowchart is described with respect to a single cluster (termed ‘first’ cluster, merely to simplify the following description), however relevant steps (820-860) in the flowchart may be performed to locate all skin clusters present in the skin map. The flowchart starts in step801, in which control passes immediately to step810.

In step810, image processor130receives a skin map indicating whether each corresponding pixel represents skin. The skin map may be pre-processed, for example, as described above with respect to the flowchart ofFIGS. 6A,6B,7and8. Control then passes to step820.

In step820, image processor130determines different clusters of pixels representing skin by examining the skin map. In general, skin pixels in contiguous locations, reasonably representing dimensions of a face may be viewed as a cluster. Ideally, the number of clusters equals the number of faces (assuming non-overlap of faces in the captured image) in the image. Control then passes to step830.

In step830, image processor130checks whether there are multiple skin clusters present in skin map. Control passes to step840if there is only a single skin cluster, or else to step850. In step840, image processor130concludes that the lone skin cluster represents a face. Control then passes to step899, in which the flowchart ends.

In step850, image processor130checks whether one of the skin clusters is at least one-third (or some reasonably large size) the size of the captured image. Control passes to step860if there is such a skin cluster, or else to step880.

In step860, image processor130concludes that skin cluster with at least one third size represents a face. Control then passes to step899.

In step880, image processor130selects the face of interest based on user input. Thus, image processor130may display a rectangle surrounding each of the faces, and receive an input from user via input interface160indicating which of the rectangles should be used as a basis for auto-focusing. Control then passes to step899.

Having thus identified the facial region of interest, image processor130processes the corresponding pixel values (for example, Y component, as described above) of pixels in the facial region, determines the corresponding degree of focus of the facial region, and provides a set of configuration parameters to lens assembly115to focus (with a desired degree of focus) on the facial region as described in sections above. The operation of the steps described above are briefly illustrated below with an example.

Illustrative Example of Facial Region Identification

FIGS. 9A-9Jprovide an example illustration of the operation of the steps of flowcharts ofFIGS. 6A,6B,7and8. Black areas in the Figures denote skin pixels, and white areas denote non-skin pixels.

InFIG. 9A, a (source) skin map900of a captured image is shown in which three skin clusters910,920and930may be seen.

A first iteration of steps615-620-625-630-635-640-620(FIG. 6A) with comparison number set to 3 (step610) may result in the skin map ofFIG. 9Aupdated to provide the skin map ofFIG. 9B, in which it may be seen that some non-skin (white) pixels in each of clusters910,920and930have been indicated as skin (black) pixels.

A first iteration of steps655-660-665-670-675-680-660(FIG. 6A) with comparison number set to 5 (step650) may result in the skin map ofFIG. 9Bupdated to provide the skin map ofFIG. 9C, in which it may be seen that some skin pixels in clusters910and920have been removed.

A second iteration of steps615-620-625-630-635-640-620(FIG. 6A) with comparison number set to 3 (step610) may result in the skin map ofFIG. 9Cupdated to provide the skin map ofFIG. 9D, in which it may be seen that some non-skin pixels in cluster910have been indicated as skin pixels.

A second iteration of steps655-660-665-670-675-680-660(FIG. 6A) with comparison number set to 5 (step650) may result in skin map ofFIG. 9Dupdated to provide the skin map ofFIG. 9E, in which it may be seen that some skin pixels in cluster910have been removed, and cluster920has been completely removed.

A third iteration of steps615-620-625-630-635-640-620(FIG. 6A) with comparison number set to 2 (step610) may result in the skin map ofFIG. 9Eupdated to provide the skin map ofFIG. 9F, in which it may be seen that some non-skin pixels in cluster910have been indicated as skin pixels.

A third iteration of steps655-660-665-670-675-680-660(FIG. 6A) with comparison number set to 5 (step650) may result in skin map ofFIG. 9Fupdated to provide the skin map ofFIG. 9G.

Operation of steps615-620-625-630-635-640-620(FIG. 6B) may result in the skin map ofFIG. 9Aupdated to provide the skin map ofFIG. 9H, in which it may be seen that some non-skin pixels in clusters910,920and930have been indicated as skin pixels.FIG. 9Hmay be compared withFIG. 9Bwhich was obtained using a comparison number of three.

Operation of steps655-660-665-670-675-680-660(FIG. 6B) with comparison number set to 5 (step650) may result in the skin map ofFIG. 9Hupdated to provide the skin map ofFIG. 9I, where some pixels have been removed.

Finally,FIG. 9Jshows the ‘final’ skin map obtained by ANDing corresponding pixels in the skin maps ofFIGS. 9G and 9I.FIG. 9Jshows two skin clusters910and930. Operation of the steps of the flowchart ofFIG. 8identify (locate) the two skin clusters910and930, and marks the corresponding boundaries940and950.

Image processor130may display the image corresponding to the skin map shown inFIG. 9J, and prompt the user to indicate the desired region of interest. The user may then indicate, via input interface160(shown inFIG. 1), that the desired region of interest is region940containing skin cluster910(which may correspond to a face). Image processor130then focuses on region910by configuring the parameters of lens assembly115in a manner described in sections above, and waits for the user to ‘click’ and captures and stores the focused image. An embodiment of image processor130is described next.

Image Processor

FIG. 10is a block diagram of image processor130in one embodiment. Image processor130is shown containing high frequency computation block1010, image signal processor (ISP) pipeline1020, sub-window logic1030, and central processing unit (CPU)1050. Image processor130may contain other components/blocks also, but are not shown as not being relevant to an understanding of the described embodiment(s). Each component is described in detail below.

ISP pipeline1020receives a stream of pixel values representing an entire image (row wise) on path113. The pixel values may be received directly from image sensor array120(ofFIG. 1). ISP pipeline1020may be implemented as a shift register, and shifts in pixels received on path113, and transfers (stores) the pixels to buffer registers or other internal buffer. ISP pipeline1020may perform various operations on the stored pixels such as optical black restoration (subtracting a black color reference level from each of the pixels), sensor linearization (which removes non-linear effects of image sensor array120), white balance, color correction (transformation of pixel values from one color space to another specific color space), gamma correction, demosaicing (individual R/G/B pixel signals obtained from Bayer color filter array converted to simultaneous R/G/B component values), etc., as suited for the specific environment.

HF computation block1010retrieves a set of pixel values from ISP pipeline1020, and computes a rate of variation of luminance across the pixels, and provides a corresponding high frequency value as a measure of the rate of variation of luminance to image processor130. With respect toFIG. 4Aand the section ‘Digital Characterization of Degree of Focus’ above, CPU1050may specify to HF computation block1010the pixel locations of pixels in object410in image400. HF computation block1010may then retrieve the corresponding pixel values from ISP pipeline1020, and compute a ‘rate of variation’ of luminance value (which may be, for example, a single number) of the pixels in object410, and forward the value to CPU1050. In one embodiment, HF computation block1010is implemented using a Sobel operator noted above, and adding some of the output values of the Sobel operator to generate a single value representing the rate of variation.

Sub-window logic1030receives control inputs from CPU1050specifying dimensions and locations of one or more sub-windows (rectangular areas) in the captured image that are to be subsampled. For each of a group (for example, nine adjacent pixels) of pixel values in the sub-window, sub-window logic1030computes the average of the pixel values in the group, and generates a corresponding single pixel value having the computed average value. The ‘averaged’ pixels thus generated form a subsampled version of portion of the image in the sub-window, and the subsampled version is provided by sub-window logic1030to image processor130. In an embodiment, sub-window logic1030provides a 64×64 pixel wide subsampled version of images captured by image sensor array120, and provides them to image processor130as noted above with respect to step210of the flowchart ofFIG. 2. Operation on such sub-sampled images reduces the computational requirements in CPU1050.

CPU1050performs the logic specified inFIG. 2based on the subsampled images received from sub-window logic1030and high frequency values received from HF computation block1010. In particular, CPU1050determines the region of interest (as described above with respect toFIGS. 5,6A,6B,7and8) and indicates the region to HF computation block1010. The instructions, which cause CPU1050to provide such features, may be received on path135or via139. CPU1050may store skin map in RAM190via path139.

It may be appreciated that CPU1050determines the skin map and region of interest, and can provide the appropriate control values (including identification of the region of interest) to HF computation block1010. Then, for each image of the scene, CPU1050may receive a high frequency measure (representing the degree of focus). CPU1050may send configuration parameters (i.e., number indicating distance) on path138to configure lens assembly115.

CONCLUSION