Patent Description:
Users often experience events which they would like to capture a scene, in photographs or video, and view at a later date and/or time, for example, a child's first steps or words, graduation, or a wedding. Often, these events may be near-static and their occurrence generally predictable (e.g., a wedding, a graduation, a serene landscape, or a portrait) and may be easily captured using an imaging system, e.g., a camera, video recorder, or smartphone. For such moments there may be sufficient time for the imaging system to determine and adjust proper exposure settings to capture the moment event. However, sometimes capturing fast moving scenes with the proper exposure may present a challenge, especially if the scene is temporary (e.g., the scene contains moving objects or the imaging system is subjected to quick panning through a scene having various brightness levels).

Even when the user of the equipment captures an image of a scene at the proper moment or utilizes a multi-shot system, the user must be aware when the event may occur and take into account an imaging system delay for determining focus and exposure. Therefore, the user must be attentive to foresee when such moments will occur and plan accordingly. This can be difficult. Often, at least some portion of the moment or event may have passed without being properly captured. Accordingly, systems and methods to expedite calculating and adjusting exposure of an imaging system would be beneficial.

<CIT> relates to a camera including an electronic flash that emits a first wink pulse of relatively low intensity fired before exposure to produce a range signal indicative of a subject distance and reflectivity, and a second main pulse of relatively high intensity fired during exposure. <CIT> relates to a reference picture array for a time of flight sensor in which different reference pixels have different sensitivity.

<CIT> relates to a camera in which a dimming correction amount is calculated based on a focal length.

<CIT> relates to an exposure control and distance measuring device. A reflection coefficient of an object is detected based on the output of a photosensitive device using the distance of the object from the device. A flash can be controlled based on the determined information.

According to one aspect of the present invention, there is provided an imaging apparatus as defined in claim <NUM>. Preferred features are set out in dependent claims <NUM> to <NUM>.

According to a second aspect of the present invention, there is provided a method for capturing an image as defined in claim <NUM>. Preferred features of the method are set out in dependent claims <NUM> and <NUM>.

According toa third aspect of the present invention, there is provided a computer program produce as defined in claim <NUM>.

The above-mentioned aspects, as well as other features, aspects, and advantages of the present technology will now be described in connection with various embodiments, with reference to the accompanying drawings. The illustrated embodiments, however, are merely examples and are not intended to be limiting. Throughout the drawings, similar symbols typically identify similar components, unless context dictates otherwise.

Various aspects of the novel systems, apparatuses, and methods are described more fully hereinafter with reference to the accompanying drawings. Rather, these aspects are provided so that this disclosure may be thorough and complete, and may fully convey the scope of the disclosure to those skilled in the art. The scope of the disclosure is intended to cover aspects of the systems, apparatuses, and methods disclosed herein, whether implemented independently of, or combined with, any other aspect of the invention. In addition, the scope of embodiments of the invention, including those described herein, is intended to cover such an apparatus or method which is practiced using other structure, functionality, or structure and functionality in addition to or other than the various aspects of the embodiments set forth herein. It should be understood that any aspect disclosed herein may be embodied by one or more elements of a claim.

Although particular aspects are described herein, many variations and permutations of these aspects fall within the scope of the disclosure. Although some benefits and advantages of the preferred aspects are mentioned, the scope of the disclosure is not intended to be limited to particular benefits, uses, or objectives. Rather, aspects of the disclosure are intended to be broadly applicable to various imaging and photographic technologies, system configurations, computational systems, flash systems, and exposure determination systems. The Detailed Description and drawings are intended to be illustrative of the disclosure of embodiments of the invention, rather than limiting.

In photography, when a user is using an imaging system (or camera) in a manual mode the user can actively control what the imaging system is focused on and may select various characteristics (e.g., aperture, shutter speed, "film" speed) that control the exposure. This allows the imaging system to capture an image nearly instantaneously when the user activates a control interface to capture an image. However, when an imaging system is used in an automatic focus ("autofocus") and an automatic exposure mode, before an image is captured the imaging system is configured to determine a correct exposure and perform an autofocus process. Automatically determining the exposure for an image that will be captured using a flash may involve combining information determined from a no-flash exposure of a target object (or scene, herein used interchangeably with "target object" unless indicated otherwise) with information determined from a pre-flash exposure of the target object, before the actual image is captured. Accordingly, an automatic exposure process takes time to perform and results in a delay of capturing an image.

A "no-flash period" is a broad phrase that is used herein to generally refer to a period of time when an imaging system may determine and set an exposure level based on the brightness of the target object with only ambient illumination (for example, sunlight or other light that is not from the imaging system flash). During the no-flash period, a flash of the imaging system does not illuminate the scene and may be in a standby mode or off. The no-flash period may exist any time that the imaging system is active but has not been activated to capture an image.

A "pre-flash period" is a broad phrase that is used herein to generally refer to a period of time during which an imaging system may activate a flash at a level of power that is less than full power to determine and set exposure parameters of the imaging system. In some embodiments, the flash produces illumination of a target object at two or more illumination levels. However, because this process may not be performed until the user activates the imaging system to capture the image, the time required for the imaging system to perform the pre-flash exposure delays the capture of the image. Additionally, because a flash is often of limited power, a target object at a far distance (for example, <NUM>-<NUM>+ meters) from the imaging system may not be adequately illuminated during a pre-flash period. This may cause the imaging system to generate an improper flash exposure or take an extended period of time to determine the exposure.

One exemplary solution for expediting the capture of the desired image may include utilizing alternate or additional sensors for determining the exposure of the imaging system, for example sensors configured to determine distance and reflectance of the target object.

<FIG> illustrates a graphical representation of an example of a timing diagram <NUM> for an automatic exposure control (AEC) system. The timing diagram <NUM> shows an example of an exposure level of a sensor and a flash output level for an imaging system capturing an image when a flash is used. In the timing diagram <NUM>, time increases along the x-axis from left to right. A relative flash output power (of the emitted light) associated with the imaging system increases along the y-axis, that is, a low flash output level near zero of the y-axis to a higher flash output level increasing along the y-axis. The timing diagram <NUM> also indicates a relative amount of exposure, increasing along the y-axis, that is, a low amount of exposure near zero of the y-axis to a higher amount of exposure increasing along the y-axis. The flash line <NUM> indicates the level of the light being emitted by a flash of the imaging system (referred to in this example as a camera), while the exposure line <NUM> indicates the exposure as determined and set by an AEC process or system of the imaging system. The timing diagram <NUM> further includes specific points in time along the x-axis.

As illustrated in <FIG>, the flash line <NUM> starts at or near the no-flash period <NUM> of zero, indicating a flash is not emitting light, for example when the camera is off or when the camera is just being used to view a scene but has not yet been instructed to capture an image. During this period (along the x-axis prior to the time <NUM>), the exposure line <NUM> is at a no-flash exposure level, indicting an exposure setting of the camera that may be higher than at times when the camera is using the flash. At a time prior to t0, the camera may identify a no-flash exposure, for example, the level of brightness of the target object and the level of natural light in the scene and/or the environment without the flash from the camera. The camera is configured with an AEC process that can be used to determine exposure.

At time t0, the flash line <NUM> increases to a constant, pre-flash illumination level during a pre-flash period <NUM>. The pre-flash level may be when the flash is charged to a power level that is lower than its full power level, or controlled to only emit light at a level that is lower than light it emits at its full power level. In some embodiments, the pre-flash level may be a constant illumination level or a varying illumination level that does not increase beyond a given threshold of illumination. As shown in the timing diagram <NUM>, the pre-flash level of the depicted embodiment is maintained at the constant illumination level during the pre-flash period <NUM>. During this pre-flash period, the AEC process may adjust the exposure for the target object based on the brightness of the target object as illuminated by the pre-flash, as indicated by the exposure line <NUM> decreasing at section <NUM> until it reaches an exposure level <NUM>. At time t1, the pre-flash period ends, as indicated by the flash line <NUM> decreasing back to a low level <NUM> near or at a zero value.

The time a camera spends adjusting the exposure during the pre-flash period may be affected by, for example, one or more of the brightness of the environment, the distance of the target object from the camera, and the reflectance of the target object. For example, the time to determine exposure may be less when the target object is far from the camera or when the target object has a low reflectance because the light from the flash of the camera would not greatly increase the brightness of the target object due to the distance and/or the low reflectance of the target object, so a proper flash exposure is not necessary to determine.

At time t2, while the flash line <NUM> has decreased back to near or at the zero value, the AEC process adjusts the exposure to the estimated flash exposure level <NUM>. The estimated flash exposure level <NUM> may be calculated using a no-flash exposure (prior to <NUM>) of the target object and the measured and adjusted exposures identified during the pre-flash period (between t0 and t1). The AEC process may calculate the frame brightness of the illuminated frame during the pre-flash period using information collected from one or more images collected during the pre-flash period. The AEC process may further calculate the brightness of the target object as it would be illuminated during the main flash, which utilizes the flash illuminated during the pre-flash period, but at a higher power level, thus identifying the exposure of the camera for capturing an image. The brightness of the target object may be used to determine the proper exposure to capture an image with the main flash, and the AEC process can set the exposure to the estimated flash exposure level <NUM> at time t2.

At time t3, the main flash is activated at the higher power level during the main flash period <NUM> for a period of time, that is, until time t4, during which an image is captured. At t5, after the image is captured, the AEC process may reset the exposure of the camera to the no-flash exposure setting <NUM> and the flash may be turned off to the near zero level during a no-flash period <NUM>.

With the above described process, the adjustment and calculation of the exposure necessary for the desired target object may take as many as fifteen (<NUM>) or more frames. In many embodiments, the entire process may take half a second or more. As may be seen in timing diagram <NUM>, a majority of time may be spent waiting for the exposure of the camera to be adjusted during the pre-flash period. Accordingly, capturing an image using the AEC process described above may introduce significant latencies into the process of capturing an image of the target object.

Various processes may be used to determine the correct exposure using a pre-flash process, each causing a delay in the time between when a camera receives a user command to take a picture and when the picture is actually taken. <FIG> shows three examples flash timing diagrams (graphs) which may be used in a camera, and illustrates potential delays that are caused by AEC systems and exposure adjustment schemes. The three graphs (timing diagrams similar to that in <FIG>) <NUM>, <NUM> and <NUM>, respectively, are illustrated with "time" increasing along the x-axis from left to right and "light intensity" increasing along the y-axis from bottom to top. The timing diagrams <NUM>, <NUM>, and <NUM> shown include labels corresponding to the flash timing diagram of <FIG>, such that similar periods from <FIG> are shown in the timing diagrams of <FIG>. For example, each timing diagram <NUM>, <NUM> and <NUM> has a no-flash period <NUM>, a pre-flash period <NUM>, a main flash period <NUM>, and another no-flash period <NUM> after the main flash period <NUM>. These examples illustrate significant delays in an imaging process. For the example illustrated in timing diagram <NUM>, the total time to perform a pre-flash exposure determination process is about <NUM> seconds. For timing diagram <NUM>, the total time is about <NUM> seconds. For timing diagram <NUM>, the total time is about <NUM> seconds.

In some embodiments, the autofocus system may rely on similar timing issues of the AEC system described above. Accordingly, the autofocus system may suffer from many deficiencies described above. For example, if the environment is too dark or lowly lit, the autofocus system may not work properly because the environment is too dark. Accordingly, the autofocus system may use the flash of the camera to assist in the focusing of the camera, which may be slow and cause delays in the time from when the camera is commanded to capture an image to the time when the camera actually captures the image, similar to delays described above in relation to the exposure. Autofocus may be further slowed depending on the initial lens position and the type of focusing algorithm used to focus the camera.

Imaging systems may incorporate laser sensors and/or time-of-flight (TOF) systems. These TOF systems may be used to assist the exposure and focusing of the cameras, and significantly reduce the exposure determination process. In various embodiments, TOF systems may be used to: measure distance, measure returned or reflected energy, and/or identify signal-to-noise ratios. The TOF systems can include a light emitter and a light sensor. The light emitter may be configured to emit light, while the light sensor system may be configured to sense a portion of the emitted the light that reflects off of the target object and returns to the light sensor. The time it takes for light emitted from the light emitter and to reflect from a target object to the light sensor can be used to identify the distance of the target object from the TOF system. A TOF system can also be configured to determine the amount of energy of the sensed light (reflected off of the target object) and this may be used to determine reflectance of the target object and exposure information. In some embodiments, the phase difference of the emitted light and the sensed light may be used to determine the distance.

<FIG> is a diagram illustrating an example of an image capture device <NUM>, according to some embodiments of the invention. In this example, the image capture device <NUM> is a camera that includes a time of flight (TOF) system <NUM>, though the image capture device <NUM> may be any device capable of capturing a still or moving image, regardless of format (digital, film, etc.) or type (video camera, still camera, web camera, etc.). The image capture device <NUM> is configured to determine a distance to a target scene or a target object, and to determine exposure (e.g., at least one exposure parameter) of a target scene or a target object using the TOF system <NUM>. For clarity of description, both a target scene and a target object will be referred to as a "target object" in the context of being the subject matter that the camera is focused on. An exposure parameter may be any of various parameters that can determine an exposure or affect the exposure. An example of an exposure parameter is a parameter indicative of an aperture or entrance pupil through which light propagating through a lens towards a sensor passes through (for example, an f/# or physical aperture size). Another example of an exposure parameter is a duration of time a shutter of the camera is open to let light pass to a sensor of the camera (which may be referred to as the "shutter speed"). Another example of an exposure parameter is a parameter for controlling the operation of a sensor of the camera to sense light and capture an image, for example, the "film speed" - a term a person of ordinary skill in the art will understand is a setting that affects the sensitivity of the sensor (a carry-over term from film photography, each film having its sensitivity rated on a relative scale as indicated by its ISO). Another example of an exposure parameter is a parameter indicative of ambient light being reflected by the target object, and which may be used to determine the exposure used to capture an image of the target object. Another example of an exposure parameter is a parameter indicative of light from a light source reflected by the target object. For example, the light (from a light source) may be light produced by a light emitter <NUM> of the TOF system <NUM>. The light emitter <NUM> of the TOF system may be incorporated in the image capture device <NUM> or coupled to the image capture device <NUM>. In some embodiments, the light emitter <NUM> is separate from the image capture device <NUM>, that is, it is not incorporated into or structurally attached to the image capture device <NUM>.

The embodiment of <FIG> illustrates emitted light <NUM> from a light emitter <NUM> propagating along a light path <NUM> that represents the path of light from the light emitter <NUM> to a target object <NUM>. <FIG> also illustrates a reflected light <NUM> which may represent the light or the reflected path of the light that illuminates the target object <NUM> (for example, from light emitter <NUM>) and reflects from the target object <NUM> to a light sensor <NUM> of the TOF system <NUM>. In some embodiments, the image capture device <NUM> may include a clock, a timer, or some other means for determining the amount of time between when the emitted light <NUM> is emitted from the light emitter <NUM> to illuminate the target object <NUM> and when the emitted light <NUM> reflected from the target object <NUM> is sensed by the light sensor <NUM>. In some embodiments, the light emitter <NUM> and the light sensor <NUM> may be two components that are configured to operate together, instead of being part of a single component TOF system <NUM>. While the light emitter <NUM> and the light sensor <NUM> may be two distinct components and/or systems, for the purposes of this disclosure, they will be discussed as forming a TOF system <NUM>. In some embodiments, the TOF system <NUM> may be an integrated TOF system, where the light emitter <NUM> and the light sensor <NUM> are part of a single integrated system.

In an example of its operation, the light emitter <NUM> may emit a pulsed infrared (IR) light. This emitted light <NUM>, which can be characterized (and referred herein) as a light signal(s) or as including a plurality of photons, illuminates the target object <NUM> and reflects from the target object to the light sensor <NUM>. A clock or timer of the TOF system <NUM>, or another component of the image capture device <NUM>, may determine the time it takes between emitting the emitted light <NUM> and sensing the reflected light <NUM> at the light sensor <NUM>. Using this amount of time and the known speed of light, a distance that light travels from the light emitter <NUM> to the target object <NUM> and back to the light sensor <NUM> may be calculated using Equation <NUM>. <MAT> The distance to the target object is half of the distance traveled. Accordingly, a target object <NUM> that is at a location farther away from the camera, compared to target objects that are closer to the camera, will require more time for the emitted light <NUM> to travel from the light emitter <NUM> to the target object <NUM> and back to the light sensor <NUM>.

The TOF system <NUM> may be configured to identify a returned energy from the target object. The returned energy identifies the amount of energy the emitted light has after reflecting off the target object. The greater the amount of energy of the emitted light when sensed by the light sensor of the TOF system <NUM> after reflecting off the target object, the higher the reflectance of the target object. Target object reflectance may be directly associated with how bright or dark a target object appears. Accordingly, for a given light condition and distance, the lower the amount of energy of the light when sensed at the light sensor <NUM>, the darker the appearance of a target object.

The TOF system may be configured to generate a signal to noise (Signal-to-Noise ratio or SNR) that indicates the strength of the return signal (light) at the TOF system after the return signal reflects off the target object. For example, when the return signal received is strong (in relation to background noise or noise introduced by the environment), the SNR is higher. Alternatively, if the return signal received is weaker (in relation to the background noise), then the SNR may be lower. With regards to the reflection of the return signal of the target object, a higher SNR may indicate that the target object has a higher reflectance (e.g., that the target object may be of a color or material that reflects light), while a lower SNR indicate that the target object has a lower reflectance (e.g., that the target object may be of a color or material that absorbs more light). The discussion above may apply to scenarios when the SNR is measured when the reflection is received from the target object at the same distance. However, the SNR may also vary dependent upon the distance of the target object from the TOF system. Accordingly, the same target object may generate different SNR values based on the distance of the target object from the TOF system. As the target object moves further from the TOF system (e.g., the distance gets larger), the SNR will become lower.

<FIG> illustrates a high-level block diagram of one embodiment of an image capture device <NUM> (similar to the image capture device <NUM> of <FIG>) having a set of components including an image processor <NUM> linked to a camera <NUM>, to a flash (or other light source) <NUM>, to a TOF system <NUM>, and to modules for determining automatic exposure correction (AEC module <NUM> and auto-focus (AF) module <NUM>). The image processor <NUM> may also be in communication with a working memory <NUM>, a memory <NUM>, and a device processor <NUM>, which in turn may be in communication with electronic storage module <NUM>, a display <NUM> (for example an electronic or touchscreen display), and a distance/reflectance module <NUM>. In some embodiments, a single processor may comprise both the image processor <NUM> and the device processor <NUM> instead of two separate processors as illustrated in <FIG>. In some embodiments, one or both of the image processor <NUM> and the device processor <NUM> may comprise a clock <NUM>, shown in <FIG> as integrated within the device processor <NUM>. Some embodiments may include three or more processors. In some embodiments, some of the components described above may not be included in the image capture device <NUM> or additional components not described above may be included in the image capture device <NUM>. In some embodiments, one or more of the components described above or described as being included in the image capture device <NUM> may be combined or integrated into any other component of the image capture device <NUM>.

The image capture device <NUM> may be, or may be part of, a cell phone, digital camera, tablet computer, personal digital assistant, laptop computer, personal camera, action camera, mounted camera, connected camera, wearable device, automobile, drone, or the like. The image capture device <NUM> may also be a stationary computing device or any device in which the TOF system <NUM> would be advantageous. A plurality of applications may be available to the user on the image capture device <NUM>. These applications may include traditional photographic and video applications, high dynamic range imaging, panoramic photo and video, or stereoscopic imaging such as 3D images or 3D video.

Still referring to <FIG>, the image capture device <NUM> includes the camera/lens ("camera") <NUM> for capturing images of target objects and/or scenes. The camera <NUM> may include at least one sensor, at least one optical imaging component that focuses light received from the field of view (FOV) of the image capture device <NUM> (for example, the FOV of the camera <NUM>) to the at least one sensor (for example, a CMOS or CCD sensor), the AF module <NUM> coupled to the at least one optical imaging component, and the AEC module <NUM> coupled to the at least one optical imaging component. In some embodiments, the image capture device <NUM> may include more than one camera. The camera <NUM> may be coupled to the image processor <NUM> to transmit a captured image to the image processor <NUM>. In this embodiment, signals to and from the camera <NUM> are communicated through the image processor <NUM>.

The image capture device may include the flash <NUM>. In some embodiments, the image capture device <NUM> may include at least two flashes. The flash <NUM> may include, for example, a flash bulb, a reflector, a geometric light pattern generator, or an LED flash. The image processor <NUM> can be configured to receive and transmit signals from the flash <NUM> to control the flash.

The image processor <NUM> may be further coupled to the TOF system <NUM>. In some embodiments, the TOF system <NUM> may include two components, as described above. In some embodiments, the TOF system <NUM> may include a light emitter <NUM> and a light sensor <NUM>. The light emitter <NUM> may be configured to emit radiation (for example, light) from the TOF system <NUM>. For ease of description, any radiation emitted from the TOF system <NUM> will be referred to as "light" including visible and non-visible radiation. The light is directed at the target object of the image capture device <NUM>. The light sensor <NUM> is configured to sense light emitted by the light emitter <NUM> after the light has reflected from an object. In some embodiments, the light sensor <NUM> may be configured to sense light reflected from multiple target objects of a scene.

As illustrated in <FIG>, the image processor <NUM> is connected to the memory <NUM> and the working memory <NUM>. In the illustrated embodiment, the memory <NUM> may be configured to store the capture control module <NUM>, the distance/reflectance module <NUM>, the operating system <NUM>, the time-of-flight (TOF) module <NUM>, the AEC module <NUM>, and the AF module <NUM>. Additional modules may be included in some embodiments, or fewer modules may be included in some embodiments. These modules may include instructions that configure the image processor <NUM> to perform various image processing and device management tasks. The working memory <NUM> may be used by the image processor <NUM> to store a working set of processor instructions or functions contained in one or more of the modules of the memory <NUM>. The working memory <NUM> may be used by the image processor <NUM> to store dynamic data created during the operation of the image capture device <NUM> (e.g., one or more target object distance measurements or FOV distance measurement arrays, the reflectance of one or more target objects or FOV reflectance measurement arrays, exposure estimates, focus estimates, etc.). While additional modules or connections to external devices or hardware may not be shown in this figure, they may exist to provide other exposure and focus adjustment and estimation options or actions.

As mentioned above, the image processor <NUM> may be configured by or may be configured to operate in conjunction with several modules stored in the memory <NUM>. The capture control module <NUM> may include instructions that control the overall image capture functions of the image capture device <NUM>. For example, the capture control module <NUM> may include instructions that configure the image processor <NUM> to capture raw image data of the target object using the camera <NUM>. The capture control module <NUM> may also be configured to activate the flash <NUM> when capturing the raw image data. In some embodiments, the capture control module <NUM> may be configured to store the captured raw image data in the electronic storage module <NUM> or to display the captured raw image data on the display <NUM>. In some embodiments, the capture control module <NUM> may direct the captured raw image data to be stored in the working memory <NUM>. In some embodiments, the capture control module <NUM> may call one or more of the other modules in the memory <NUM>, for example the distance/reflectance module <NUM>, the TOF module <NUM>, the AEC module <NUM>, or the AF module <NUM>.

The distance/reflectance module <NUM> may comprise instructions that allow the image processor <NUM> or the device processor <NUM> to calculate, estimate, or otherwise determine the distance to and reflectance of the target object or FOV of the image capture device <NUM>. The distance/reflectance module <NUM> may include instructions for using the TOF system <NUM>, the camera <NUM>, and the clock <NUM> to identify the distance of the target object. When identifying the distance to and reflectance of the target object, the distance/reflectance module <NUM> may be configured to determine the distance to the target object. Accordingly, the distance/reflectance module <NUM> may comprise the instructions to emit a light signal via the light emitter <NUM> and sense a reflection of the light signal off the target object via the light sensor <NUM>. The instructions may further instruct the clock <NUM> to measure the time between the emission of the light signal and the sensing of the reflection of the light signal. Based on the amount of time that elapses between when the light signal is emitted by the light emitter <NUM> and when the light signal reflection is sensed by the light sensor <NUM>, the distance/reflectance module <NUM> may comprise instructions to determine the distance the light signal traveled, for example using Equation <NUM> above. The distance/reflectance module <NUM> may further comprise instructions for determining the distances of multiple points in the FOV of the image capture device <NUM> and for forming an array of the distances. The instructions contained therein may include identifying distances (as described above for the target object) for each of a plurality of points or positions within the FOV of the image capture device <NUM> an storing the array in one of the working memory <NUM> or the electronic storage module <NUM>, for example.

Additionally, the distance/reflectance module <NUM> may comprise instructions for determining the reflectance of the target object or an array of points within the FOV of the image capture device <NUM>. As described above in relation to the distance instructions, the distance/reflectance module <NUM> may further comprise instructions for emitting the light signal via the light emitter <NUM> of the TOF system <NUM> and sensing the reflected light signal via the light sensor <NUM>. Based on the energy of the light reflected off of the target object, the distance/reflectance module <NUM> may identify the reflectance of the target object. Additionally, the instructions contained therein may direct the distance/reflectance module <NUM> to identify the reflectance of each of a plurality of point or locations within the FOV of the image capture device <NUM>, and may provide for the storage or display of the identified reflectance values.

In some embodiments, the distance/reflectance module <NUM> may further comprise instructions for generating the offline configuration data described below in reference to <FIG>.

The AEC module <NUM> may comprise instructions that allow the image processor <NUM> or the device processor <NUM> to calculate, estimate, or adjust the exposure of the camera <NUM> and thus of the image capture device <NUM>. The AEC module <NUM> may include the instructions allowing for the exposure estimations described above in reference to <FIG> and below in reference to <FIG>. Accordingly, the AEC module <NUM> may comprise instructions for utilizing the TOF system <NUM> (including both the light emitter <NUM> and the light sensor <NUM>), the camera <NUM>, the clock <NUM>, and the flash <NUM> to identify and/or estimate the no-flash exposure, the pre-flash exposure, and the flash exposure. Additionally, the AEC module <NUM> may include instructions for adjusting the exposure of the camera <NUM> to at least one of the no-flash exposure, the pre-flash exposure, and the flash exposure. In some embodiments, the AEC module may further comprise instructions for illuminating the flash at one of the no-flash, pre-flash, and main flash levels of illumination.

As the brightness of the target object as captured by the image capture device <NUM> is directly related to the exposure of the image capture device <NUM>, the no-flash exposure of the image capture device <NUM> may be identified at any time when the flash is not illuminated but the image capture device <NUM> is turned on. Accordingly, in some embodiments, the AEC module <NUM> may be configured to constantly monitor the exposure of the image capture device <NUM> based on the brightness of the target object. The AEC module <NUM> may be integrated or otherwise communicate with one or more of the capture control module <NUM> and the operating system <NUM>, and instead capture an image according to the methodology described above. However, as described above, the use of the flash <NUM> with the AEC module <NUM> may introduce unnecessary delays in the capture of an image.

Alternatively, the TOF system <NUM> may provide depth and SNR information for different portions of the FOV instead of just a single point. The AF module <NUM> and the AEC module <NUM> may utilize this information from the TOF system <NUM> and employ certain strategies and methods to achieve optimal exposure and focus for target objects at various locations in the FOV. For example, if a portrait of a person was taken, and the person was not standing at the center of the image but rather off-center, such as at the left third of the FOV, the TOF system <NUM> may accurately detect the person's location, (assuming the person is the closest object to the camera). Accordingly, the AF module <NUM> and the AEC module <NUM> may choose to focus and expose on the nearest object, in this case the person.

Still referring to <FIG>, the operating system <NUM> may configure the image processor <NUM> to manage the working memory <NUM> and the processing resources of image capture device <NUM>. For example, the operating system <NUM> may include device drivers to manage hardware resources such as the camera <NUM>, the flash <NUM>, and the TOF system <NUM>. Therefore, in some embodiments, instructions contained in the processing modules discussed above and below may not interact with these hardware resources directly, but instead interact with this hardware through standard subroutines or APIs located in the operating system <NUM>. Instructions within the operating system <NUM> may then interact directly with these hardware components. The operating system <NUM> may further configure the image processor <NUM> to share information with device processor <NUM>. The operating system <NUM> may also include instructions allowing for the sharing of information and resources between the various processing modules of the image capture device.

The AF module <NUM> can include instructions that configure the image processor <NUM> to adjust the focus position of the camera <NUM>. The AF module <NUM> can include instructions that configure the image processor <NUM> to perform focus analyses and automatically determine focus parameters in some embodiments, and can include instructions that configure the image processor <NUM> to respond to user-input focus commands in some embodiments. In some embodiments, the AF module <NUM> may use information from the light emitter <NUM> and the light sensor <NUM> to determine when the target object (or one or more points or positions within the FOV of the image capture device) is at a specific distance and appropriate focus. In some embodiments, the AF module <NUM> may include instructions for identifying and adjusting the focus of the camera <NUM> based on light emitted from the flash <NUM> and received at the light sensor <NUM> from the target object or one or more points or positions within the FOV. In some embodiments, the AF module <NUM> may be configured to receive a command from the capture control module <NUM>, the distance/reflectance module <NUM>, the AEC module <NUM>, the TOF module <NUM>, or from one of the image processor <NUM> or device processor <NUM>.

The AF module <NUM> may be configured to perform a search algorithm only during the pre-flash period and may not perform any functions during the no-flash period. Accordingly, if information from the TOF system <NUM> is provided to the AF module <NUM>, an amount of time taken by the AF module <NUM> to perform the auto-focusing functions can be reduced.

In <FIG>, the device processor <NUM> may be configured to control the display <NUM> to display the captured image, or a preview of the captured image including estimated exposure and focus settings, to a user. The display <NUM> may be external to the image capture device <NUM> or may be part of the image capture device <NUM>. The display <NUM> may also be configured to provide a viewfinder displaying the preview image for the user prior to capture the image of the target object, or may be configured to display a captured image stored in the working memory <NUM> or the electronic storage module <NUM> or recently captured by the user. The display <NUM> may include a panel display, for example, a LCD screen, LED screen, or other display technologies, and may implement touch sensitive technologies. The device processor <NUM> may also be configured to receive an input from the user. For example, the display <NUM> may also be configured to be a touchscreen, and thus may be configured to receive an input from the user. The user may use the display <NUM> to input information that the processor may provide to the distance/reflectance module <NUM> or the TOF module <NUM> or the AEC module <NUM> or the AF module <NUM>. For example, the user may use the touchscreen to select the target object from the FOV shown on the display <NUM> or set or establish the exposure levels and focus settings of the image capture device <NUM>. The device processor <NUM> may receive that input and provide it to the appropriate module, which may use the input to select perform instructions enclosed therein (for example determine the distance or reflectance of the target image at the distance/reflectance module <NUM>, determine the focus of the target image at the AF module <NUM>, etc.).

In some embodiments, the device processor <NUM> may be configured to control the one or more of the processing modules in the memory <NUM> or to receive inputs from one or more of the processing modules in the memory <NUM>. The TOF module <NUM> may be configured to interact with the TOF system <NUM>. The TOF module <NUM> may comprise instructions for applying Equations <NUM> and <NUM>, as described herein, to determine various parameters and values based on measurements and actions performed by the TOF system <NUM>. For example, the TOF module <NUM> may include the equations for determining a distance traveled by the signal emitted by the light emitter <NUM> or including software for interacting with and/or controlling the TOF system <NUM> and the light emitter <NUM> and the light sensor <NUM>. In some embodiments, the TOF module <NUM> may be configured to store or acquire the offline configuration information described below. In some embodiments, the device processor <NUM> or the TOF module <NUM> may select multiple equations for use with the TOF system <NUM> and may determine to use one or more of the equations to identify a desired parameter based on the emitted and sensed light signals.

The device processor <NUM> may write data to the electronic storage module <NUM>, for example data representing captured images. While the electronic storage module <NUM> is represented graphically as a traditional disk device, in some embodiments, the electronic storage module <NUM> may be configured as any storage media device. For example, the electronic storage module <NUM> may include a disk drive, such as a floppy disk drive, hard disk drive, optical disk drive or magneto-optical disk drive, or a solid state memory such as a FLASH memory, RAM, ROM, and/or EEPROM. The electronic storage module <NUM> can also include multiple memory units, and any one of the memory units may be configured to be within the image capture device <NUM>, or may be external to the image capture device <NUM>. For example, the electronic storage module <NUM> may include a ROM memory containing system program instructions stored within the image capture device <NUM>. The electronic storage module <NUM> may also include memory cards or high speed memories configured to store captured images which may be removable from the camera.

Although <FIG> depicts a device <NUM> having separate components to include a processor, imaging sensor, and memory, in some embodiments these separate components may be combined in a variety of ways to achieve particular design objectives. For example, in an alternative embodiment, the memory components may be combined with processor components to save cost and improve performance.

Additionally, although <FIG> illustrates a number of memory components, including the memory <NUM> comprising several processing modules and a separate memory comprising a working memory <NUM>, in some embodiments, different memory architectures may be utilized. For example, a design may utilize ROM or static RAM memory for the storage of processor instructions implementing the modules contained in memory <NUM>. The processor instructions may be loaded into RAM to facilitate execution by the image processor <NUM>. For example, working memory <NUM> may comprise RAM memory, with instructions loaded into working memory <NUM> before execution by the image processor <NUM>. In some embodiments, one or more of the processing modules may be software stored in the memory <NUM> or may comprise a hardware system combined with the software components. Furthermore, functions associated above with one of the image processor <NUM> and the device processor <NUM> may be performed by the other of the image processor <NUM> and the device processor <NUM> or both the image processor <NUM> and the device processor <NUM>, though not described as such above.

In some embodiments, the image processor <NUM> may be further configured to participate in one or more processing operations prior to capturing an image, while capturing an image, and after capturing an image. For example, prior to capturing the image, the image processor <NUM> may be configured to perform one or more of the processes described above (e.g., estimating and adjusting the exposure and the focus of the camera <NUM>). In some embodiments, the image processor <NUM> may be configured to, in conjunction with one or more of the LED flash, the TOF system <NUM>, the distance/reflectance module <NUM>, the TOF module <NUM>, the AEC module <NUM>, and the AF module <NUM>, adjust the exposure and the focus of the image capture device <NUM> (specifically the camera <NUM>). The image processor <NUM> may thus be configured to enable the image capture device <NUM> to capture an image of the target object or FOV with proper settings (exposure and focus) as desired by the user.

In some embodiments, the image processor <NUM> may be involved with and/or control the adjustment and estimation of the exposure and focus of the camera <NUM>. The image processor <NUM> may be configured to control the flash <NUM>, the camera <NUM>, the AEC module <NUM>, the distance/reflectance module <NUM> to establish an estimated flash exposure (as described in relation to <FIG> above). Accordingly, the image processor <NUM> may monitor the brightness of the target object prior to any illumination from the flash <NUM> (monitoring the brightness of the target object <NUM>, as referenced in <FIG>, may include using the camera <NUM> to view the target object and detect or identify the brightness of the target object or the environment without the light from the flash <NUM>). The image processor <NUM> may then control the flash <NUM> to emit a pre-flash level of light and adjust the exposure of the image capture device <NUM> based on commands and inputs received from the AEC module <NUM>. Once a pre-flash exposure is reached, the image processor <NUM> may turn off the flash <NUM> and set the exposure of the image capture device <NUM> to the estimated flash exposure as calculated by the AEC module <NUM>. Then, the image processor <NUM> may activate the flash <NUM> at the main flash light level and capture the image of the target object <NUM>.

Alternatively, or additionally, the image processor <NUM> may be configured to generate an estimated pre-flash exposure prior to performing the steps discussed above. For example, the image processor <NUM> may be configured, via one or more of the TOF module <NUM>, the TOF system <NUM>, and the distance/reflectance module <NUM>, to perform a distance and reflection estimation of the target object or the array of points or positions within the FOV. As described herein, the distance and reflection estimation may be based on the amount of time that elapses between when the light (or light signal) is emitted from the light emitter <NUM> and when the return light signal (after reflecting off the target object or the points or positions within the FOV) is received by the light sensor <NUM>. Before, while, or after the TOF reflectance and estimation is performed, the image processor <NUM> may also be monitoring the brightness of the target object prior to any illumination from the flash <NUM>, as discussed above in relation to the no-flash period, which may involve monitoring the brightness levels as received by the camera <NUM> without any illumination from the flash <NUM> to identify a no-flash exposure of the image capture device <NUM> (only environmental light). The image processor <NUM> may then combine the information received from the TOF system <NUM> (distance and reflectance) with the no-flash exposure to generate a pre-flash exposure estimation. In some embodiments, the pre-flash exposure estimation may involve referencing offline pre-calibration values, which will described in further detail below. In some embodiments, the pre-flash exposure estimate may be accurate enough that the image processor <NUM> may skip the pre-flash exposure adjustment described above and proceed directly to estimating the flash exposure for use during image capture. In some embodiments, the TOF distance estimation may be combined with the pre-flash exposure adjustment prior to estimating the flash exposure for use during image capture with the main flash.

In some embodiments, the image processor <NUM> may use the TOF distance estimates provided by the TOF system <NUM>, the TOF module <NUM>, and the distance/reflectance module <NUM> to improve estimates of the focus of the image capture device <NUM> and reduce the time needed to adjust the focus of the image capture device <NUM> (specifically, the camera / lens ("camera") <NUM>) in response to the flash <NUM> during a pre-flash period where the focus of the camera <NUM> may be adjusted in response to the brightness of the target object as viewed by the camera <NUM>. Similarly as described above for the exposure estimation, the image processor <NUM> may be configured to estimate a pre-flash focus of the camera <NUM>, and may use that estimate in conjunction with a reduced pre-flash focus adjustment and estimation period or instead of the pre-flash focus adjustment and estimation period.

Alternatively, or additionally, the image processor <NUM> may only act in response to instructions from one or more other components or modules of the image capture device <NUM>. For example, the AEC module <NUM> or the AF module <NUM> may issue instructions to other components of the image capture device <NUM> to allow the AEC module <NUM> to calculate the estimated flash exposure based on either of the methods described above (with or without the TOF system inputs) or to allow the AF module <NUM> to calculate the estimated focus as described above. Additionally, statistics may be collected using various hardware (such as an image signal processor (ISP)) based on the image data from the sensor at real time. For example, the collected statistics may be sums and averages of all regions on a certain size grid, such as 64x48. The collected statistics may also include histograms of the image data.

<FIG> is a graph <NUM> that illustrates the relationship between distance between a target object and an imaging device and exposure. The x-axis of the graph <NUM> indicates the distance between the target object and the camera in centimeters (cm), while the y-axis of the graph <NUM> represents the determined "correct" (relative) exposure level for the camera. For example, graph <NUM> shows examples of the pre-flash exposure information verses distance of a target object. <FIG> is a chart <NUM> that is associated with graph <NUM> and illustrates examples of information relating to distance <NUM>, examples of information relating to the exposure <NUM>, and examples of information relating to TOF sensor data, such as signal-to-noise ratio (SNR) <NUM>. The graph <NUM> and chart <NUM> comprise experimental data identified from tests and comparisons of various aspects disclosed. The chart <NUM> includes various columns: distance <NUM> includes the distance between the target object and the camera, three columns corresponding to the LED AEC Estimation data, including exposure <NUM> depicting the pre-flash exposure for the target object, luma <NUM> depicting the pre-flash luma of the target object, and pre-flash time column <NUM> including the amount of time the pre-flash period lasts, in seconds (for example, the time it takes for the camera to reach the exposure of exposure <NUM>). The chart <NUM> also includes columns comprising TOF system data, including a measured distance <NUM>, including the distance between the target object and the camera as measured by the TOF system (in millimeters) and SNR <NUM> indicating the signal-noise ratio (SNR) as identified by the TOF system. The chart <NUM> also has four rows corresponding to different distances (in centimeters). The distances include <NUM>, <NUM>, <NUM>, and <NUM>.

As shown in the graph <NUM>, when the distance between the target object and the camera is small, the pre-flash exposure is lower because when the target object is near the camera, the light from the flash has a greater impact on the brightness of the target object as viewed by the camera, and the pre-flash exposure level is lower. Correspondingly, as the distance between the camera and the target object increases, the exposure levels increase because the light from the flash has a lesser impact on the brightness of the target object as viewed by the camera, and the exposure level must be higher to capture the image at a given brightness level. Thus, as shown in the graph <NUM> and the chart <NUM>, when the target object is a distance of <NUM> from the camera, the pre-flash exposure of the camera may be <NUM>. When the target object is a distance of <NUM> from the camera, the pre-flash exposure of the camera may be <NUM>. Similarly, when the distance is <NUM>, the pre-flash exposure may be <NUM>, and when the distance is <NUM>, the pre-flash exposure may be <NUM>. The luma (brightness) <NUM> of the chart <NUM> indicates the brightness of the target object as viewed by the camera given the pre-flash exposure levels of the exposure <NUM> at a given distance. The luma is the final luma at the end of the pre-flash process, and it corresponds to the pre-flash exposure index. For example, the pre-flash luma for the target object at the distance of <NUM> is <NUM>, while the pre-flash luma for the target object at the distance of <NUM> is <NUM>, at the distance of <NUM> is <NUM>, and at the distance of <NUM> is at <NUM>. Thus, as described above, as the target object is further from the camera, the resulting pre-flash exposure may be higher to obtain the same or similar brightness levels.

The pre-flash time column <NUM> provides the time that elapses while the AEC system adjusts the exposure of the camera to the pre-flash exposure level. As seen by comparing the values of the pre-flash time column <NUM> as they correspond to the various distances, the time decreases as the pre-flash exposure increases. For example, the time is <NUM> seconds when the distance of the target object is <NUM> and the pre-flash exposure if <NUM>, but only <NUM> seconds when the target object is <NUM> from the camera and the pre-flash exposure is <NUM>. This shows that the time lost to the pre-flash exposure adjustment is directly associated with the amount of exposure adjustment performed.

The measured distance <NUM> indicates the distance between the target object and the camera, as determined by the TOF system. By comparing the measured distance <NUM> with the distance <NUM>, one sees that the TOF provides an accurate measurement of the distance between the target object and the camera. In this data, the measured valued via the TOF system is off by under a centimeter (<NUM> from distance <NUM> vs. <NUM> from measure distance <NUM>). Finally, the SNR <NUM> depicts the SNR as identified by the TOF system. The SNR decreases from <NUM> at <NUM> distance between the camera and the target object to <NUM> at <NUM> distance between the camera and the target object.

<FIG> depicts a graph <NUM> illustrating pre-flash exposure verses Signal-Noise Ratio, illustrating the pre-flash exposure of target object, at a given distance, having varying reflectance values, in accordance with an exemplary embodiment. The x-axis of the graph <NUM> indicates colors (or reflectance values) of the target objects, while the y-axis of the graph <NUM> represents the exposure level of the camera. <FIG> is a chart <NUM> illustrating information associated with <FIG>. The graph <NUM> and chart <NUM> comprise experimental data identified from tests and comparisons of various aspects disclosed. The graph <NUM> depicts the pre-flash exposure vs reflectance of a target object. The chart <NUM> includes various columns: object <NUM> includes the color (reflectance) of the target object, three columns corresponding to the LED AEC Estimation data, including pre-flash exposure <NUM> depicting pre-flash exposure information for the target object, luma <NUM> depicting pre-flash luma information of the target object, and pre-flash time <NUM> including information of the amount of time the pre-flash period lasts, in seconds (for example, the time it takes for the camera to reach the exposure of the pre-flash exposure <NUM>). The chart <NUM> also includes columns comprising TOF system data, including a measured distance <NUM>, including the distance between the target object and the camera as measured by the TOF system (in millimeters) and signal-noise ratio (SNR) <NUM> indicating the signal-noise ratio (SNR) as identified by the TOF system. The chart <NUM> also has three rows corresponding to different colors of target objects having different reflectance values (in centimeters). The colors include white, grey, and black.

As shown in the graph <NUM>, when the signal to noise ratio of the target object is greater, the pre-flash exposure value is lower. This may be because, as described above, when the target object has a higher reflectance, the return signal reflecting off the target object is higher, and the target object with a higher reflectance typically needs less exposure than those with lower reflectance. This corresponds to the description above that when the target object is more reflective (e.g., has a higher reflectance), the flash may affect the exposure of the camera.

As shown in the graph <NUM> and the chart <NUM>, the pre-flash exposure <NUM> corresponds with the exposure (or color/material) of the target object in object <NUM>. For example, the white object (having a higher reflectance than the grey or black target objects) has a lower exposure value of <NUM>, while the grey and black target objects have exposure values of <NUM> and <NUM>, respectively. These values conform to the discussion herein that target objects with higher reflectance values may use lower exposure values than target objects with lower reflectance values. Additionally, the luma (brightness) values of luma <NUM> indicate the amount brightness of the target object(s) as viewed by the camera. As shown, the white target object, the black target object, and the grey target object are all at comparable values (<NUM>, <NUM>, and <NUM>, respectively). In some embodiments, an algorithm may be used to manipulate the luma within a reasonable range. For example, the luma values of the white, black, and grey target objects (<NUM>, <NUM>, and <NUM>, respectively) may each be considered reasonable values. Additionally, in the chart <NUM>, as the pre-flash brightness of luma <NUM> increases, the pre-flash time <NUM> decreases. Additionally, as shown in the graph <NUM>, the SNR of each of the target objects in SNR <NUM> reduces as the exposure levels of pre-flash exposure <NUM> increases (exposure level of <NUM> has SNR of <NUM>, while exposures of <NUM> and <NUM> have SNRs of <NUM> and <NUM>, respectively). As shown in Figure <NUM>, the SNR may correlate with the reflectance of the target object (for example, the white target object has a higher SNR than the grey target object, which has a higher SNR than the black target object at approximately the same distances). As shown in Figure <NUM>, the luma and pre-flash time values may have less correlation with the pre-flash exposure than the SNR.

The measured distance <NUM> indicates the distance between the target object and the camera, as determined by the TOF system. As shown in measured distance <NUM>, the target objects of object <NUM> are all within <NUM> of each other (white is measured at <NUM>, grey at <NUM>, and black at <NUM>). Finally, the SNR <NUM> depicts the SNR as identified by the TOF system. The SNR decreases from <NUM> for the white target object to <NUM> for the black target object.

<FIG> depicts a block diagram illustrating a process <NUM>, that an imaging device may be configured to implement, for estimating a flash exposure based on LED flash automatic exposure correction using a camera not utilizing a TOF or laser sensor, as described above. As depicted, the process <NUM> includes four blocks involved in the identification of the estimated flash exposure, described above. The process <NUM> begins with a determination of a no-flash exposure at block <NUM>. The no-flash exposure of block <NUM> may correspond to the discussion above of the identification of the exposure of the camera when the flash is not active at any level. For example, this may occur when the camera is turned on but not emitting light from the flash <NUM>. The no-flash exposure may be identified and stored for later use. Once the no-flash exposure is determined at block <NUM>, the process <NUM> proceeds to blocks <NUM> and <NUM>, which may operate simultaneously.

Block <NUM> includes activating the flash of the camera at a pre-flash level. This may include, for example, controlling the flash to illuminate a target object at less than a full flash level. While the flash is activated at the pre-flash level, the block <NUM> adjusts the exposure of the camera to a pre-flash exposure level. In some embodiments, the pre-flash exposure level may be adjusted so that the luma is within a specified range (for example, as described above, each of <NUM>, <NUM>, and <NUM> may be within the specified range). In some embodiments, the reflectance, the distance of the target object, and the pre-flash exposure level may be adjusted to bring the luma value within the specified range. Such adjustment may be performed by the AEC module described above, or by any other module configured to control the exposure of the camera. During the blocks <NUM> and <NUM>, the camera may not capture any images. Instead, the camera may just monitor the brightness of the target object and/or the environment and adjust its exposure according to the monitored brightness to a target exposure level. Additionally, the pre-flash exposure may be stored for later use. For example, the AEC module <NUM> may use a pre-flash and main flash brightness ratio (some value pre-defined/determined) together with the stored pre-flash exposure level to estimate a desired exposure level on the final flash image. In some embodiments, the main flash may be brighter than pre-flash, (brightness ratio ><NUM>), so the AEC module <NUM> will further lower the exposure level from the pre-flash exposure level so that the final image can be properly exposed. In some embodiments, the adjustment of the exposure of the camera may be performed by an exposure algorithm. Once the pre-flash exposure is determined at block <NUM>, the process <NUM> proceeds to block <NUM>.

Block <NUM> includes estimating the exposure of the camera for the main flash illumination. This may be performed by the exposure algorithm or the AEC module described above. The exposure algorithm may be configured to, utilizing the no-flash information acquired at block <NUM> and the pre-flash exposure information acquired at block <NUM>, compute brightness of the target object (specifically, the brightness of the frame including the target object, as viewed by the camera while illuminated by the LED) via the collected statistics information described above. The exposure algorithm may further extrapolate the brightness of the target object and the scene when illuminated by the main flash driving current (substantially larger than the pre-flash driving current(s)). This extrapolated brightness may represent an estimated brightness level for the target object or scene that is expected when the main flash is illuminated under full current (for example, the main flash driving current). The extrapolated brightness may then be used to set the exposure of the camera to a proper level such that the target or scene is not over or under exposed (for example, so that the capture target or scene is at the proper brightness). Once the estimated flash exposure is determined, the process <NUM> ends. In some embodiments, the identified brightness and/or exposure may be stored for later use.

<FIG> is a block diagram illustrating a process <NUM>, that an imaging device may be configured to implement, for estimating a flash exposure based on a determined distance and reflectance of the target object by a camera utilizing a TOF system or laser sensor in conjunction with an AEC module, for example, as described above. As depicted, the process <NUM> includes four blocks involved in the identification of the estimated pre-flash, though an optional fifth block may be included (though not shown herein). The process <NUM> begins with blocks <NUM> and <NUM> operating simultaneously. Block <NUM> includes estimating a distance to the target object using the TOF system.

In some embodiments, the determination of the distance of the target object may include only a single distance estimation based on the center of the target object or the center of the FOV of the camera. In some embodiments, the distance of the target object may be determined using an array of distance information for various points distributed across the FOV of the camera. For example, the TOF system may emit to and sense light from various locations around the FOV to generate the array of distance information. In some embodiments, the camera may then average the distance information from the array to determine a distance of the image. In some embodiments, the camera may identify the distance image for the target object at a specific location within the FOV (as selected by the user, for example, when selecting a focus point, etc.) based on the distance information from the array at a point in the array corresponding to the location of the target object within the FOV.

An imaging device may be configured to determine reflectance of a target object using the TOF system. The reflectance may be determined based on the returned energy of the light emitted from a TOF emitter and received by a TOF sensor. The greater the returned energy, the greater the reflectance of the target object, as described above. Similar to the distance information above, a determination of the reflectance may include determination of a reflectance of an array of the FOV, where the reflectance of multiple locations within the FOV may be determined and/or calculated.

Simultaneously with the TOF system distance estimations, a camera may be configured to identify the no-flash exposure at block <NUM>. The no-flash exposure at block <NUM> may be similar or identical to the no-flash exposure at block <NUM> described above in relation to <FIG>. The no-flash exposure of block <NUM> may be determined simultaneously with the TOF distance estimation of block <NUM> because the TOF system does not utilize light that effects the brightness of the target object or FOV. Accordingly, light (for example, IR light) emitted by the TOF system does not affect the brightness of the target object or FOV as viewed by the camera during the exposure detection, and thus the no-flash exposure detection may be performed simultaneously with the TOF system distance estimation. In some embodiments, exposure measurements by the camera may be performed simultaneously with any TOF system estimations. Once the blocks <NUM> and <NUM> are complete, the process <NUM> proceeds to block <NUM>.

The camera of <FIG> estimates the pre-flash exposure at block <NUM>. The camera may use the TOF distance estimation identified at block <NUM> with the no-flash exposure of block <NUM> to calculate an initial pre-flash exposure prior to the pre-flash period, as described above. Applying the estimated initial pre-flash exposure before entering the pre-flash period may reduce the amount of time the camera may spend in the pre-flash period, because the exposure of the camera will be at a level closer to proper pre-flash exposure, thus reducing the exposure adjustment necessary during the pre-flash period. As described above, when the camera uses the pre-flash period to adjust a no-flash exposure to reach a pre-flash exposure, the camera may adjust the exposure a great deal dependent upon at least the environmental lighting, the distance between the target object and the camera, and the reflectance of the target object. However, since the blocks <NUM> and <NUM> provide the distance and the reflectance of the target object, the block <NUM> can provide a more accurate estimate of the exposure of the camera needed to capture the image of the target device at the desired brightness level, and the exposure of the camera should need to be adjusted less from the pre-flash exposure estimation level.

In some embodiments, the pre-flash exposure estimation of block <NUM> may be pre-calibrated. In some embodiments, the offline pre-calibration may involve identifying the exposure of the camera when capturing the target object at various distances, thus building a library of exposure values for a target object at various distances. In some embodiments, the offline pre-calibration may comprise identifying the exposure values of the camera when capturing various target objects having a varying reflectance at the same distance, thus building a library of exposure values for a distance with varying reflectance values. In some embodiments, the pre-calibration may provide information that helps the pre-flash exposure estimation of block <NUM> develop more accurate estimates of the pre-flash exposure. In some embodiments, a library or other database may be generated by capturing target objects at various distances with various reflectance values. The library or database may cover all scenarios and may allow the algorithm to accurately identify the correct exposure level in the library or database, given input distance and reflectance or SNR. For example, the pre-calibration information may allow the block <NUM> to better estimate the pre-flash exposure when provided with the TOF distance estimation, as block <NUM> may review the exposure values in the calibration information for the estimated distance and the estimate reflectance. The better estimate may allow the pre-flash exposure estimation of block <NUM> to more accurately estimate the pre-flash exposure value, thus further shortening the amount of time the camera may spend in the pre-flash period.

Once the block <NUM> completes the pre-flash exposure estimation, the process <NUM> proceeds to block <NUM>, where the camera enters the pre-flash period described above (for example, enters blocks <NUM> and <NUM> of process <NUM>). In some embodiments (not shown in this figure), the pre-flash exposure estimation of block <NUM> may provide an accurate pre-flash exposure estimate. Accordingly, the process <NUM> may skip the pre-flash exposure adjustment period and proceed directly to image capture during the main flash period. Thus, in some embodiments, the pre-flash exposure estimation of block <NUM> may comprise the main flash exposure estimation block using the TOF distance and reflection estimate and the no-flash exposure of blocks <NUM> and <NUM>, respectively. Such elimination of the pre-flash period altogether may greatly reduce the latencies introduced by the AEC process described above when not used with the TOF system.

In some embodiments, the camera may use the TOF distance estimation of block <NUM> to better estimate a pre-flash focus. For example, when a camera is preparing to capture an image, the camera may use information from the TOF distance estimation to generate a more accurate pre-flash focus estimate that reduces the amount of time the camera spends focusing on the target object, but the camera may still proceed to the pre-flash period.

<FIG> is a flowchart illustrating an example of a method <NUM> for determining exposure, according to some embodiments. The method <NUM> may start at block <NUM> and proceed to block <NUM>. At block <NUM>, the method <NUM> may emit a light signal toward a target object via a TOF system. The TOF system may comprise the TOF system <NUM>/<NUM> (including both the light emitter and light sensor), as described above in relation to <FIG> and <FIG>. In some embodiments, the TOF system may be part of an image capture device, such as the image capture devices <NUM> and <NUM> shown in <FIG> and <FIG>, respectively. The TOF system may be controlled via a processor (for example, one of image processor <NUM> or device processor <NUM>) or via any of the modules described above in relation to <FIG>. Once the light signal has been generated and emitted toward the target object, the method <NUM> proceeds to block <NUM>.

At block <NUM>, the method <NUM> senses a reflection of the emitted light signal off the target object via the light sensor, for example the light sensor <NUM>/<NUM> as referenced in <FIG> and <FIG>, respectively. In some embodiments, the light sensor of the TOF system may communicate information regarding the received reflection to the processor of the image capture device or may store the information in a memory, such as working memory <NUM> or electronic storage module <NUM>. Alternatively, the information from the light sensor may be communicated to any of the modules in the image capture device <NUM> as shown in <FIG>. Once the reflected light is sensed by the light sensor of the TOF system, the method <NUM> progresses to block <NUM>.

At block <NUM>, the method <NUM> determines a return energy based on the reflection of the emitted light signal as sensed by the light sensor. The return energy may be determined by the light sensor itself or may be determined by one of the processors. Alternatively, or additionally, the return energy may be determined based on the reflection of the emitted light signal by one or more of the modules of the image capture device <NUM> of <FIG>, such as the distance/reflectance module <NUM> or the TOF module <NUM>, for example. Once the method <NUM> determines a reflected energy based on the reflection of the emitted light signal, the method <NUM> proceeds to block <NUM>. At block <NUM>, the method <NUM> measures a time between when the light signal is emitted by the light emitter of the TOF system and when the light sensor of the TOF system senses the reflection of the emitted light signal off of the target object. In some embodiments, the measurement of the time may be performed by the processor or by one of the modules of the image capture device <NUM> of <FIG>, for example, the distance/reflectance module <NUM> or the TOF module <NUM>. In some embodiments, the measurement of the time may involve the clock <NUM>. In some embodiments, the results of the measurement may be stored in the memory or communicated to the processors or any of the associated modules. Once the time has been measured, the method <NUM> progresses to block <NUM>.

At block <NUM>, the method <NUM> determines a distance between the target object and the TOF system based on the measured time. In some embodiments, this determination may be performed by the TOF system itself or one of the processors. In some embodiments, the determination may be made by the distance/reflectance module <NUM> of the image capture device <NUM>. In some embodiments, the determined distance may be stored in one of the memories or may be immediately used by one of the modules or the processors. Once the block <NUM> is complete, the method <NUM> proceeds to block <NUM>. At block <NUM>, the method <NUM> identifies a reflectance of the target object based on the returned energy. In some embodiments, the reflectance may be identified by the distance/reflectance module <NUM> or the TOF system. In some embodiments, one of the processors or one of the other modules may be configured to identify the reflectance of the target object based on the sensed reflection and the identified ambient or no-flash lighting. In some embodiments, the reflectance may be determined based on the returned energy as sensed by the light sensor. In some embodiments, identifying the reflectance may also incorporate one or more other measured, identified, or determined parameter (such as the ambient light exposure, etc.). Once the reflectance is determined at block <NUM>, the method <NUM> proceeds to block <NUM>.

At block <NUM>, the method <NUM> determines an exposure level of the target object based on the determined distance and the identified reflectance. In some embodiments, the determination of the exposure level may be performed by the AEC module <NUM> or one of the processors or one of the other modules of <FIG>. In some embodiments, the exposure level may be determined by the light sensor. In some embodiments, the exposure level may be stored in one of the memories or may be immediately communicated to one of the modules of the image capture device <NUM> of <FIG>. Once the exposure level is determined, the method <NUM> ends at block <NUM>.

In some embodiments, the determined, identified, measured, or generated values or amounts described above may be displayed, for example on the display <NUM>, as referenced by <FIG>, or stored in the working memory <NUM> or the electronic storage module <NUM> or processed by one of the processors.

The various operations of methods described above may be performed by any suitable means capable of performing the operations, such as various hardware and/or software component(s), circuits, and/or module(s). Generally, any operations illustrated in the Figures may be performed by corresponding functional means capable of performing the operations.

As used herein, the term interface may refer to hardware or software configured to connect two or more devices together. For example, an interface may be a part of a processor or a bus and may be configured to allow communication of information or data between the devices. The interface may be integrated into a chip or other device. For example, in some embodiments, an interface may comprise a receiver configured to receive information or communications from a device at another device. The interface (e.g., of a processor or a bus) may receive information or data processed by a front end or another device or may process information received. In some embodiments, an interface may comprise a transmitter configured to transmit or communicate information or data to another device. Thus, the interface may transmit information or data or may prepare information or data for outputting for transmission (e.g., via a bus).

The various illustrative logical blocks, modules and circuits described in connection with the present disclosure may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller or state machine.

In one or more aspects, the functions described may be implemented in hardware, software, firmware, or any combination thereof. A storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Thus, in some aspects computer readable medium may comprise non-transitory computer readable medium (e.g., tangible media). In addition, in some aspects computer readable medium may comprise transitory computer readable medium (e.g., a signal).

The functions described may be implemented in hardware, software, firmware or any combination thereof. If implemented in software, the functions may be stored as one or more instructions on a computer-readable medium. A storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

For example, such a computer program product may comprise a computer readable medium having instructions stored (and/or encoded) thereon, the instructions being executable by one or more processors to perform the operations described herein.

Claim 1:
An imaging apparatus, comprising:
a camera (<NUM>) for capturing an image of a target object;
a time-of-flight system (<NUM>), the time-of-flight system (<NUM>) comprising:
means (<NUM>) for emitting a light signal; and
means (<NUM>) for sensing a reflection of the emitted light signal off the target object;
means for determining a return energy based on the reflection of the emitted light signal;
means for measuring a time between when the light signal is emitted and when the emitted light signal is sensed;
means (<NUM>) for determining a distance between the target object and the time-of-flight system (<NUM>) based on the measured time;
means (<NUM>) for identifying a reflectance of the target object based on the return energy of the emitted light signal;
an auto-exposure control module for:
determining a first exposure level based on the distance to the target object and the reflectance of the target object; controlling a flash (<NUM>) to emit a pre-flash, wherein the camera (<NUM>) captures a first image of the target object when illuminated by the pre-flash at an exposure based on the first exposure level; for
determining a second exposure level based on the first image; and
for controlling the flash (<NUM>) to emit a main flash; and, means
for controlling the camera (<NUM>) to capture a second image of the target object whilst the target object is illuminated by the main flash at an exposure based on the second exposure level.