Patent ID: 12206986

DETAILED DESCRIPTION

An image capture device may capture the image according to image capture configurations of a selected camera mode. For example, a user of the image capture device may select a camera mode to use to capture the image. The selection may be based on one or more criteria including, for example, an amount of background light, a location of a subject relative to the image sensor, or a motion of the subject. Examples of camera modes that may be available for selection include, without limitation, a still mode, still+local tone mapping (LTM) mode, high dynamic range (HDR) mode, and multi-frame noise reduction (MFNR) mode.

Each of those camera modes may be best suited for particular situations. For example, the still+LTM mode may be preferable where there is low to mid (e.g., 100 to 800) ISO, with or without motion, and a low amount of noise. In another example, the HDR mode may be preferable where there is low to mid ISO, the location of the image to capture is somewhere outdoors, motion is detected up to a certain degree (small motion), and low noise. In yet another example, the MFNR mode may be preferable where there is high (e.g., more than 800) ISO, up to a certain noise level, and without too much motion. The highest quality image or video may result from using the most preferable camera mode given the situation.

In many cases, the user of the image capture device may not select the best camera mode to use at a given time or in a given place. For example, the user may not recognize that the background of an image to capture does not have enough light for a selected mode or that an object to be captured within the image has a motion that may not be captured very well using a selected mode. Furthermore, even if the user of the image capture device selects an appropriate camera mode for capturing a first image, that camera mode may not be the best camera mode for capturing a subsequent image. That is, lighting, motion, or other conditions within a location in which the images are captured may change over a short amount of time. If the user does not account for these changes and select a new camera mode, the subsequently-captured image may be low quality.

Implementations of this disclosure address problems such as these using automated camera mode selection systems and techniques. The implementations of this disclosure are described in detail with reference to the drawings, which are provided as examples so as to enable those skilled in the art to practice the technology. The figures and examples are not meant to limit the scope of the present disclosure to a single implementation or embodiment, and other implementations and embodiments are possible by way of interchange of, or combination with, some or all of the described or illustrated elements. Wherever convenient, the same reference numbers will be used throughout the drawings to refer to same or like parts.

FIGS.1A-Dare isometric views of an example of an image capture device100. The image capture device100may include a body102having a lens104structured on a front surface of the body102, various indicators on the front of the surface of the body102(such as LEDs, displays, and the like), various input mechanisms (such as buttons, switches, and touch-screen mechanisms), and electronics (e.g., imaging electronics, power electronics, etc.) internal to the body102for capturing images via the lens104and/or performing other functions. The image capture device100may be configured to capture images and video, and to store captured images and video for subsequent display or playback.

The image capture device100can include various indicators, including the LED lights106and the LED display108. The image capture device100can also include buttons110configured to allow a user of the image capture device100to interact with the image capture device100, to turn the image capture device100on, and to otherwise configure the operating mode of the image capture device100. The image capture device100can also include a microphone112configured to receive and record audio signals in conjunction with recording video. The side of the image capture device100may include an I/O interface114. The camera may also include a microphone116system integrated into the camera housing. The front surface of the camera may include two drainage ports as part of a drainage channel118for the camera audio system. The camera can include an interactive display120that allows for interaction with the camera while simultaneously displaying camera information on a surface of the camera. As illustrated, the image capture device100may include a lens104configured to receive light incident upon the lens and to direct received light onto an image sensor internal to the lens.

The image capture device100includes a camera exterior that encompasses and protects the camera's internal electronics, which are further described in later sections. The camera exterior includes six surfaces (i.e. a front face, a left face, a right face, a back face, a top face, and a bottom face), wherein the exterior surfaces form a rectangular cuboid. Furthermore, both the front and rear surfaces of the image capture device100are substantially rectangular in shape. The image capture device100can be made of a rigid material such as plastic, aluminum, steel, or fiberglass. Additional camera features, such as the features described above, may be affixed to an exterior of the camera. In some embodiments, the camera described herein includes features other than those described below. For example, instead of a single interface button, the camera can include additional buttons or different interface features, such as a multiple microphone openings to receive voice or other audio commands.

Although not expressly shown inFIGS.1A-D, in some implementations, the image capture devices100may include one or more image sensors, such as a charge-coupled device (CCD) sensor, an active pixel sensor (APS), a complementary metal-oxide semiconductor (CMOS) sensor, an N-type metal-oxide-semiconductor (NMOS) sensor, and/or any other image sensor or combination of image sensors.

Although not expressly shown inFIGS.1A-D, in some implementations, the image capture device100may include one or more microphones, which may receive, capture, and record audio information, which may be associated with images acquired by the image sensors.

Although not expressly shown inFIGS.1A-D, the image capture device100may include one or more other information sources or sensors, such as an inertial measurement unit (IMU), a global positioning system (GPS) receiver component, a pressure sensor, a temperature sensor, a heart rate sensor, or any other unit, or combination of units, that may be included in an image capture apparatus.

In some implementations, the image capture device100may interface with or communicate with an external device, such as an external user interface device, via a wired or wireless computing communication link (not shown). The user interface device may, for example, be the personal computing device360described below with respect toFIG.3. Any number of computing communication links may be used. The computing communication link may be a direct computing communication link or an indirect computing communication link, such as a link including another device or a network, such as the internet, may be used. In some implementations, the computing communication link may be a Wi-Fi link, an infrared link, a Bluetooth (BT) link, a cellular link, a ZigBee link, a near field communications (NFC) link, such as an ISO/IEC 23243 protocol link, an Advanced Network Technology interoperability (ANT+) link, and/or any other wireless communications link or combination of links. In some implementations, the computing communication link may be an HDMI link, a USB link, a digital video interface link, a display port interface link, such as a Video Electronics Standards Association (VESA) digital display interface link, an Ethernet link, a Thunderbolt link, and/or other wired computing communication link.

In some implementations, the image capture device100may transmit images, such as panoramic images, or portions thereof, to the user interface device (not shown) via the computing communication link, and the user interface device may store, process, display, or a combination thereof the panoramic images.

In some implementations, the user interface device may be a computing device, such as a smartphone, a tablet computer, a phablet, a smart watch, a portable computer, and/or another device or combination of devices configured to receive user input, communicate information with the image capture device100via the computing communication link, or receive user input and communicate information with the image capture device100via the computing communication link.

In some implementations, the user interface device may display, or otherwise present, content, such as images or video, acquired by the image capture device100. For example, a display of the user interface device may be a viewport into the three-dimensional space represented by the panoramic images or video captured or created by the image capture device100.

In some implementations, the user interface device may communicate information, such as metadata, to the image capture device100. For example, the user interface device may send orientation information of the user interface device with respect to a defined coordinate system to the image capture device100, such that the image capture device100may determine an orientation of the user interface device relative to the image capture device100. Based on the determined orientation, the image capture device100may identify a portion of the panoramic images or video captured by the image capture device100for the image capture device100to send to the user interface device for presentation as the viewport. In some implementations, based on the determined orientation, the image capture device100may determine the location of the user interface device and/or the dimensions for viewing of a portion of the panoramic images or video.

In some implementations, the user interface device may implement or execute one or more applications to manage or control the image capture device100. For example, the user interface device may include an application for controlling camera configuration, video acquisition, video display, or any other configurable or controllable aspect of the image capture device100.

In some implementations, the user interface device, such as via an application, may generate and share, such as via a cloud-based or social media service, one or more images, or short video clips, such as in response to user input. In some implementations, the user interface device, such as via an application, may remotely control the image capture device100, such as in response to user input.

In some implementations, the user interface device, such as via an application, may display unprocessed or minimally processed images or video captured by the image capture device100contemporaneously with capturing the images or video by the image capture device100, such as for shot framing, which may be referred to herein as a live preview, and which may be performed in response to user input. In some implementations, the user interface device, such as via an application, may mark one or more key moments contemporaneously with capturing the images or video by the image capture device100, such as with a tag, such as in response to user input.

In some implementations, the user interface device, such as via an application, may display, or otherwise present, marks or tags associated with images or video, such as in response to user input. For example, marks may be presented in a camera roll application for location review and/or playback of video highlights.

In some implementations, the user interface device, such as via an application, may wirelessly control camera software, hardware, or both. For example, the user interface device may include a web-based graphical interface accessible by a user for selecting a live or previously recorded video stream from the image capture device100for display on the user interface device.

In some implementations, the user interface device may receive information indicating a user setting, such as an image resolution setting (e.g., 3840 pixels by 2160 pixels), a frame rate setting (e.g., 60 frames per second (fps)), a location setting, and/or a context setting, which may indicate an activity, such as mountain biking, in response to user input, and may communicate the settings, or related information, to the image capture device100.

FIG.2is a cross-sectional view of an example of a dual-lens image capture device200including overlapping fields-of-view210,212. In some implementations, the image capture device200may be a spherical image capture device with fields-of-view210,212as shown inFIG.2. For example, the image capture device200may include image capture devices220,222, related components, or a combination thereof, arranged in a back-to-back or Janus configuration. For example, a first image capture device220may include a first lens230and a first image sensor240, and a second image capture device222may include a second lens232and a second image sensor242arranged oppositely from the first lens230and the first image sensor240.

The first lens230of the image capture device200may have the field-of-view210shown above a boundary250. Behind the first lens230, the first image sensor240may capture a first hyper-hemispherical image plane from light entering the first lens230, corresponding to the first field-of-view210.

The second lens232of the image capture device200may have a field-of-view212as shown below a boundary252. Behind the second lens232, the second image sensor242may capture a second hyper-hemispherical image plane from light entering the second lens232, corresponding to the second field-of-view212.

One or more areas, such as blind spots260,262, may be outside of the fields-of-view210,212of the lenses230,232, light may be obscured from the lenses230,232and the corresponding image sensors240,242, and content in the blind spots260,262may be omitted from capture. In some implementations, the image capture device200may be configured to minimize the blind spots260,262.

The fields-of-view210,212may overlap. Stitch points270,272, proximal to the image capture device200, at which the fields-of-view210,212overlap may be referred to herein as overlap points or stitch points. Content captured by the respective lenses230,232, distal to the stitch points270,272, may overlap.

Images contemporaneously captured by the respective image sensors240,242may be combined to form a combined image. Combining the respective images may include correlating the overlapping regions captured by the respective image sensors240,242, aligning the captured fields-of-view210,212, and stitching the images together to form a cohesive combined image.

A small change in the alignment, such as position and/or tilt, of the lenses230,232, the image sensors240,242, or both may change the relative positions of their respective fields-of-view210,212and the locations of the stitch points270,272. A change in alignment may affect the size of the blind spots260,262, which may include changing the size of the blind spots260,262unequally.

Incomplete or inaccurate information indicating the alignment of the image capture devices220,222, such as the locations of the stitch points270,272, may decrease the accuracy, efficiency, or both of generating a combined image. In some implementations, the image capture device200may maintain information indicating the location and orientation of the lenses230,232and the image sensors240,242such that the fields-of-view210,212, stitch points270,272, or both may be accurately determined, which may improve the accuracy, efficiency, or both of generating a combined image.

Optical axes through the lenses230,232may be substantially antiparallel to each other, such that the respective axes may be within a tolerance such as 1%, 3%, 5%, 10%, and/or other tolerances. In some implementations, the image sensors240,242may be substantially perpendicular to the optical axes through their respective lenses230,232, such that the image sensors may be perpendicular to the respective axes to within a tolerance such as 1%, 3%, 5%, 10%, and/or other tolerances.

The lenses230,232may be laterally offset from each other, may be off-center from a central axis of the image capture device200, or may be laterally offset and off-center from the central axis. As compared to an image capture device with back-to-back lenses, such as lenses aligned along the same axis, the image capture device200including laterally offset lenses230,232may include substantially reduced thickness relative to the lengths of the lens barrels securing the lenses230,232. For example, the overall thickness of the image capture device200may be close to the length of a single lens barrel as opposed to twice the length of a single lens barrel as in a back-to-back configuration. Reducing the lateral distance between the lenses230,232may improve the overlap in the fields-of-view210,212.

Images or frames captured by an image capture device, such as the image capture device100shown inFIGS.1A-Dor the image capture device200shown inFIG.2, may be combined, merged, or stitched together, to produce a combined image, such as a spherical or panoramic image, which may be an equirectangular planar image. In some implementations, generating a combined image may include three-dimensional, or spatiotemporal, noise reduction (3DNR). In some implementations, pixels along the stitch boundary may be matched accurately to minimize boundary discontinuities.

FIGS.3A-Bare block diagrams of examples of systems configured for image capture. Referring first toFIG.3A, an image capture device300configured for image capture is shown. The image capture device300includes an image capture device310(e.g., a camera or a drone), which may, for example, be the image capture device100shown inFIGS.1A-D. The image capture device310includes a processing apparatus312that is configured to receive a first image from the first image sensor314and receive a second image from the second image sensor316. The processing apparatus312may be configured to perform image signal processing (e.g., filtering, tone mapping, stitching, and/or encoding) to generate output images based on image data from the image sensors314and316.

The image capture device310includes a communications interface318for transferring images to other devices. The image capture device310includes a user interface320, which may allow a user to control image capture functions and/or view images. The image capture device310includes a battery322for powering the image capture device310. The components of the image capture device310may communicate with each other via the bus324.

The image capture device300may implement some or all of the pipelines for automated camera mode selection described in this disclosure, such as the pipeline400ofFIG.4, the pipeline600ofFIG.6, the pipeline700ofFIG.7, the pipeline800ofFIG.8, the pipeline900ofFIG.9, the pipeline1000ofFIG.10, the pipeline1100ofFIG.11, or a combination thereof. The image capture device300may be used to implement some or all of the techniques described in this disclosure, such as the technique1200ofFIG.12.

The processing apparatus312may include one or more processors having single or multiple processing cores. The processing apparatus312may include memory, such as a random-access memory device (RAM), flash memory, or another suitable type of storage device such as a non-transitory computer-readable memory. The memory of the processing apparatus312may include executable instructions and data that can be accessed by one or more processors of the processing apparatus312. For example, the processing apparatus312may include one or more dynamic random access memory (DRAM) modules, such as double data rate synchronous dynamic random-access memory (DDR SDRAM). In some implementations, the processing apparatus312may include a digital signal processor (DSP). In some implementations, the processing apparatus312may include an application specific integrated circuit (ASIC). For example, the processing apparatus312may include a custom image signal processor.

The first image sensor314and the second image sensor316are configured to detect light of a certain spectrum (e.g., the visible spectrum or the infrared spectrum) and convey information constituting an image as electrical signals (e.g., analog or digital signals). For example, the image sensors314and316may include CCDs or active pixel sensors in a CMOS. The image sensors314and316may detect light incident through a respective lens (e.g., a fisheye lens). In some implementations, the image sensors314and316include digital-to-analog converters. In some implementations, the image sensors314and316are held in a fixed orientation with respective fields of view that overlap.

The image capture device310may include a communications interface318, which may enable communications with a personal computing device (e.g., a smartphone, a tablet, a laptop computer, or a desktop computer). For example, the communications interface318may be used to receive commands controlling image capture and processing in the image capture device310. For example, the communications interface318may be used to transfer image data to a personal computing device. For example, the communications interface318may include a wired interface, such as a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, or a FireWire interface. For example, the communications interface318may include a wireless interface, such as a Bluetooth interface, a ZigBee interface, and/or a Wi-Fi interface.

The image capture device310may include a user interface320. For example, the user interface320may include an LCD display for presenting images and/or messages to a user. For example, the user interface320may include a button or switch enabling a person to manually turn the image capture device310on and off. For example, the user interface320may include a shutter button for snapping pictures. The image capture device310may include a battery322that powers the image capture device310and/or its peripherals. For example, the battery322may be charged wirelessly or through a micro-USB interface.

In some implementations, the image capture device310may include one or more hardware or software components for performing global tone mapping against pixels of an image captured using the image capture device310. The global tone mapping performed using those one or more hardware or software components may integrate color correction operations. For example, those one or more hardware or software components may be used to perform the technique1200described below with respect toFIG.12.

Referring next toFIG.3B, a system330configured for image capture is shown. The system330includes an image capture device340and a personal computing device360that communicate via a communications link350. The image capture device340may, for example, be the image capture device100shown inFIGS.1A-D. The personal computing device360may, for example, be the user interface device described with respect toFIGS.1A-D. The image capture device340includes a first image sensor342and a second image sensor344that are configured to capture respective images. The image capture device340includes a communications interface346configured to transfer images via the communication link350to the personal computing device360.

The personal computing device360includes a processing apparatus362that is configured to receive, using the communications interface366, a first image from the first image sensor, and receive a second image from the second image sensor344. The processing apparatus362may be configured to perform image signal processing (e.g., filtering, tone mapping, stitching, and/or encoding) to generate output images based on image data from the image sensors342and344.

The image capture device340may implement some or all of the pipelines for automated camera mode selection described in this disclosure, such as the pipeline400ofFIG.4, the pipeline600ofFIG.6, the pipeline700ofFIG.7, the pipeline800ofFIG.8, the pipeline900ofFIG.9, the pipeline1000ofFIG.10, the pipeline1100ofFIG.11, or a combination thereof. The image capture device340may be used to implement some or all of the techniques described in this disclosure, such as the technique1200ofFIG.12.

The first image sensor342and the second image sensor344are configured to detect light of a certain spectrum (e.g., the visible spectrum or the infrared spectrum) and convey information constituting an image as electrical signals (e.g., analog or digital signals). For example, the image sensors342and344may include CCDs or active pixel sensors in a CMOS. The image sensors342and344may detect light incident through a respective lens (e.g., a fisheye lens). In some implementations, the image sensors342and344include digital-to-analog converters. In some implementations, the image sensors342and344are held in a fixed relative orientation with respective fields of view that overlap. Image signals from the image sensors342and344may be passed to other components of the image capture device340via the bus348.

The communications link350may be a wired communications link or a wireless communications link. The communications interface346and the communications interface366may enable communications over the communications link350. For example, the communications interface346and the communications interface366may include an HDMI port or other interface, a USB port or other interface, a FireWire interface, a Bluetooth interface, a ZigBee interface, and/or a Wi-Fi interface. For example, the communications interface346and the communications interface366may be used to transfer image data from the image capture device340to the personal computing device360for image signal processing (e.g., filtering, tone mapping, stitching, and/or encoding) to generate output images based on image data from the image sensors342and344.

The processing apparatus362may include one or more processors having single or multiple processing cores. The processing apparatus362may include memory, such as RAM, flash memory, or another suitable type of storage device such as a non-transitory computer-readable memory. The memory of the processing apparatus362may include executable instructions and data that can be accessed by one or more processors of the processing apparatus362. For example, the processing apparatus362may include one or more DRAM modules, such as DDR SDRAM.

In some implementations, the processing apparatus362may include a DSP. In some implementations, the processing apparatus362may include an integrated circuit, for example, an ASIC. For example, the processing apparatus362may include a custom image signal processor. The processing apparatus362may exchange data (e.g., image data) with other components of the personal computing device360via the bus368.

The personal computing device360may include a user interface364. For example, the user interface364may include a touchscreen display for presenting images and/or messages to a user and receiving commands from a user. For example, the user interface364may include a button or switch enabling a person to manually turn the personal computing device360on and off In some implementations, commands (e.g., start recording video, stop recording video, or snap photograph) received via the user interface364may be passed on to the image capture device340via the communications link350.

In some implementations, the image capture device340and/or the personal computing device360may include one or more hardware or software components for performing global tone mapping against pixels of an image captured using the image capture device340. The global tone mapping performed using those one or more hardware or software components may integrate color correction operations. For example, those one or more hardware or software components may be used to perform the technique1200described below with respect toFIG.12.

FIG.4is a block diagram of an example of a camera mode selection and capture pipeline400. In some implementations, the camera mode selection and capture pipeline400may be included in an image capture device, such as the image capture device100shown inFIGS.1A-Dor the image capture device200shown inFIG.2. In some implementations, the camera mode selection and capture pipeline400may represent functionality of an integrated circuit, for example, including an image capture unit, a camera mode selection unit, or a combined camera mode selection and image capture unit.

The camera mode selection and capture pipeline400receives input402and processes the input402to produce output404. The input402may be information or measurements usable to select a camera mode at an automated camera mode selection unit406. For example, the input402may include measurements related to criteria processed by the automated camera mode selection unit406, such as dynamic range, motion, and/or light intensity. The input402may be received using one or more sensors of the image capture device or processor implementing the camera mode selection and capture pipeline400.

The output404may be an image captured using an image capture unit408. The image capture unit408uses the camera mode selected by the automated camera mode selection unit406to capture an image, such as using an image sensor (e.g., the first image sensor314and/or the second image sensor316, or the first image sensor342and/or the second image sensor344). For example, the image captured using the image capture unit408may be an image or a frame of a video. That image or frame may be one of a sequence or series of images or frames of a video, such as a sequence, or series, of frames captured at a rate, or frame rate, which may be a number or cardinality of frames captured per defined temporal period, such as twenty-four, thirty, or sixty frames per second. The output404may be output for display at the image capture device and/or transmitted to another component or device.

The automated camera mode selection unit406includes a parameter calculation unit410and a mode selection unit412. The parameter calculation unit410processes the input402to determine values for the image selection criteria represented within the input402. The parameter calculation unit410outputs those values or data indicative thereof to the mode selection unit412. The mode selection unit412selects a camera mode to use to capture an image based on those values or data. In some implementations, the mode selection unit412may select the camera mode based at least in part on secondary input414.

In some implementations, the automated camera mode selection unit406may include additional units or functionality. In some implementations, the parameter calculation unit410and the mode selection unit412may be combined into one unit. In some implementations, aspects of one or both of the parameter calculation unit410or the mode selection unit412may be separated into multiple units.

FIG.5is a block diagram of an example of a parameter calculation unit500of an automated camera mode selection unit of an image capture device. For example, the parameter calculation unit500may be the parameter calculation unit410of the pipeline400described with respect toFIG.4. The parameter calculation unit500includes a HDR estimation unit502, a motion estimation unit504, and a light intensity estimation unit506. The HDR estimation unit502estimates dynamic range for an image to be captured using first input508. The motion estimation unit504estimates motion for the image to be captured using second input510. The light intensity estimation unit506estimates light intensity for the image to be captured using third input512.

Some or all of the first input508, the second input510, or the third input512may be included within input received at the automated camera mode selection unit that implements the parameter calculation unit500. For example, some or all of the first input508, the second input510, or the third input512may be received within the input402described with respect toFIG.4. In some implementations, some or all of the first input508, the second input510, or the third input512may refer to the same or similar information or measurements.

The HDR estimation unit502uses the first input508to determine whether a scene for the image to be captured is HDR or non-HDR. The HDR estimation unit502determines that the scene for the image to be captured is non-HDR if the first input508indicates that the image sensor of the image capture device will be able to capture the entire information of the scene without resulting in dark or saturated pixels. Similarly, the HDR estimation unit502determines that the scene for the image to be captured is HDR if there are at least a threshold number of dark pixels and a threshold number of bright pixels. Those threshold numbers may be the same or different. As such, the HDR estimation unit502processes the first input508to determine a number of dark pixels and a number of bright pixels.

The motion estimation unit504uses the second input510to determine whether camera motion is detected. For example, the motion estimation unit504can determine whether sensor measurements (e.g., angular speed) indicated within the second input510meets a motion threshold. If the motion threshold is met (e.g., because the angular speed is higher than a value for the motion threshold), the motion estimation unit504determines that camera motion is detected.

The light intensity estimation unit506uses the third input512to determine a light intensity for the scene for the image to be captured. Thus, the light intensity may be estimated based on data representing the scene luminance within the third input512. As such, the light intensity estimation unit506may determine the light intensity for the scene based on a light intensity threshold. For example, the light intensity estimation unit506may determine that the light intensity is low where the scene luminance is below the light intensity threshold. In another example, the light intensity estimation unit506may determine that the light intensity is high where the scene luminance is higher than the light intensity threshold. In yet another example, the light intensity estimation unit506may determine that the light intensity is medium where the scene luminance is neither lower nor higher than the light intensity threshold.

Implementations and examples of automated camera mode selection according to this disclosure may use different inputs for the parameter calculations described with respect toFIGS.4and5and/or perform different processing than as described with respect toFIGS.4and5. In particular,FIGS.6-11are block diagrams of examples of pipelines for different automated camera mode selection techniques according to this disclosure.

Referring first toFIG.6, a pipeline600used to implement a first technique for automated camera mode selection is shown. According to the first technique, a HDR estimation unit602(e.g., the HDR estimation unit502described with respect toFIG.5) receives two input values604and606(e.g., the first input508described with respect toFIG.5). The two input values604and606refer to control points defined using one or more look up tables (LUTs). Each of the two input values604and606is compared to a threshold. In some cases, both of the two input values604and606may be compared to the same threshold.

In particular, the two input values604and606represent control points used to define what a dark pixel is and what a bright pixel is. A first LUT corresponding to the first input value604corresponds to a first curve for the image and a second LUT corresponding to the second input value606corresponds to a second curve for the image. The first curve of the first LUT and the second curve of the second LUT are weighting curves. The weighting curves are exponential functions between 0.0 and 1.0. These curves are vectors (the LUTs) of 256 values. On the other hand, the 64×48 thumbnail is represented using 16 bits. Each pixel value of the thumbnail is transformed to an 8 bits value. This 8 bits value is the index for the LUTs, i.e., Score_bright=LUT_bright[8bits_value] and Score_dark=LUT_dark[8bits_value], for example. That is, each curve (an exponential function) is parametrized by one control point (tuning parameter). The transformation to 8 bits is done for all pixels of the thumbnail and all the scores are summed. These scores are then normalized to get FDarkand FBright.

Thus, the HDR estimation unit602determines whether a scene for the image to be captured is HDR based on where the two input values604and606are with respect to the weighting curves. The HDR estimation unit602outputs an indicator608as a result of that determination. The indicator608is a Boolean in which a first value (e.g., true) indicates that the HDR estimation unit602determined the scene to be HDR and in which a second value (e.g., false) indicates that the HDR estimation unit602determined the scene to be non-HDR.

The motion estimation unit610(e.g., the motion estimation unit504described with respect toFIG.5) receives one input value612(e.g., the second input510described with respect toFIG.5) representing an angular speed of an object detected using a gyroscope or other sensor. The motion estimation unit610determines whether the input value612is higher than a first threshold. If the input value612is higher than the first threshold, the motion estimation unit610determines that camera motion is detected.

If the input value612is not higher than the first threshold, a Sum of Absolute Differences (SAD) is determined between the current and the previous thumbnails. If the SAD is higher than a second threshold, the motion estimation unit610determines that scene motion is detected. If the SAD is not higher than the second threshold, the motion estimation unit610determines that no motion is detected. The motion estimation unit610outputs an indicator614as a result of the one or two determinations. The indicator614is a Boolean in which a first value (e.g., true) indicates that the motion estimation unit610detected some motion and in which a second value (e.g., false) indicates that the motion estimation unit610did not detect motion.

The light intensity estimation unit616(e.g., the light intensity estimation unit506described with respect toFIG.5) receives one input value618(e.g., the third input512described with respect toFIG.5) representing a scene luminance estimated within exposure correction operations. The input value618is compared against a threshold for light intensity. If the input value618is greater than the threshold, the light intensity estimation unit616determines that the light intensity is high. If the input value618is less than the threshold, the light intensity estimation unit616determines that the light intensity is low. If the input value618is neither greater nor less than the threshold, the light intensity estimation unit616determines that the light intensity is medium.

The light intensity estimation unit616outputs an indicator620based on the comparison between the input value618and the threshold. For example, the indicator can have a first value when the light intensity is determined to be high, a second value when the light intensity is determined to be low, and a third value when the light intensity is determined to be medium.

A mode selection unit622(e.g., the mode selection unit412described with respect toFIG.4) selects a camera mode to use to capture an image based on the indicator608, the indicator614, and the indicator620. For example, each of the camera modes that may be selected may be defined to correspond to particular combinations of values of the indicator608, the indicator614, and the indicator620. As such, the particular combination of values of the indicator608, the indicator614, and the indicator620may be used to identify one of the camera modes. As a result, a selected camera mode624is selected. The selected camera mode624may then be used, such as by an image capture unit (e.g., the image capture unit408described with respect toFIG.4), to capture an image.

Referring next toFIG.7, a pipeline700used to implement a second technique for automated camera mode selection is shown. The second technique implemented using the pipeline700represents an extension of the first technique implemented using the pipeline600described with respect toFIG.6. The second technique addresses temporal instability (e.g., output flickering) that may result from using the threshold values to produce the indicator608as output of the HDR estimation unit602and to produce the indicator614as output of the motion estimation unit610. In particular, the second technique introduces fuzzy logic to improve the behavior and the robustness of the automated camera mode selection process, and further introduces temporal smoothing filtering in which inputs are smoothed.

The pipeline700includes an HDR estimation unit702, a motion estimation unit704, a light intensity estimation unit706, and a mode selection unit708(which may, for example, be the HDR estimation unit502, the motion estimation unit504, and the light intensity estimation unit506described with respect toFIG.5, and the mode selection unit412described with respect toFIG.4). The output of each of the HDR estimation unit702, the motion estimation unit704, and the light intensity estimation unit706is expressed as a fuzzy value between 0.0 and 1.0. The mode selection unit708processes the fuzzy values output from each of the HDR estimation unit702, the motion estimation unit704, and the light intensity estimation unit706. In particular, the mode selection unit generally performs fuzzification against the fuzzy values received as input, evaluates the fuzzy values in view of rules, aggregates the output of those rules, and performs defuzzification against the aggregated output.

According to the second technique, the HDR estimation unit702receives two input values710and712(e.g., the first input508described with respect toFIG.5). Processing is performed similar to that described with respect to the HDR estimation unit602of the pipeline600described with respect toFIG.6. In particular, the sum of dark pixels (FDark) is determined and the sum of bright pixels (FBright) is determined, using one or more dedicated weighting functions. The sums FDarkand FBrightare then normalized to determine an output714as a fuzzy value FHDR.

Where an AND condition is used for the HDR detection, such that whether HDR is detected is based on the sum of both of the dark pixels and the bright pixels, the output714is calculated by FDarkand FBright. Alternatively, where an OR condition is used for the HDR detection, such that whether HDR is detected is based on the sum of the dark pixels or the sum of the bright pixels, the output714is calculated as the difference of a first value and a second value, where the first value represents the sum of FDarkand FBrightand the second value represents the product of FDarkand FBright. In some implementations, instead of two sums, a single sum of the mid-tone pixels may instead be determined.

The motion estimation unit704receives one input value716(e.g., the second input510described with respect toFIG.5) representing an angular speed (FGyro) of an object detected using a gyroscope or other sensor. In an implementation, FGyrois the normalized value (between 0 and 1) of the angular speed (input of the motion estimation unit). Processing is performed similar to that described with respect to the motion estimation unit610of the pipeline600. In particular, the motion estimation unit704determines whether the input value716is higher than a threshold (TGyrospeed). If the input value716is higher than TGyroSpeed, the motion estimation unit704determines that camera motion is detected.

However, if the input value716is not higher than TGyrospeed, a SAD is determined between two consecutive thumbnails (e.g., the current and the previous thumbnails). The SAD value is then normalized to produce normalized value FImg. As such, if the input value716is lower than TGyroSpeed, an output718of the motion estimation unit704, FMotion, is expressed as a fuzzy value representing FImg. Otherwise, the output718is expressed as a fuzzy value representing FGyro.

The light intensity estimation unit706receives one input value720(e.g., the third input512described with respect toFIG.5) representing a scene luminance estimated within exposure correction operations. Processing is performed similar to that described with respect to the light intensity estimation unit616of the pipeline600. In particular, the input value720is compared against a threshold for light intensity. The input value720is then normalized to produce an output722. The output722represents a fuzzy value of FLightintensity.

A mode selection unit708selects a camera mode to use to capture an image based on the fuzzy values included within the output714, the output718, and the output722. First, the mode selection unit708fuzzifies the output714, the output718, and the output722. For example, fuzzifying the output714, the output718, and the output722can include identifying each of the output714, the output718, and the output722as one of a small value, a medium value, or a large value (e.g., determining degrees of membership within those size categories). The mode selection unit708then evaluates the fuzzified values (expressed simply as “motion,” “dynamic range,” and “light intensity”) in view of fuzzy inference rules. In some implementations, the fuzzy inference rules may be represented as a three-dimensional decision cube. For example, each of the three axes of the three-dimensional decision cube may represent one of motion, dynamic range, or light intensity.

Examples of the fuzzy inference rules include, without limitation: (1) if motion is small and dynamic range is small and light intensity is small, then mode is MFNR; (2) if motion is small and dynamic range is large and light intensity is small, then mode is MFNR; (3) if motion is small and dynamic range is small and light intensity is medium, then mode is MFNR; (4) if motion is small and dynamic range is large and light intensity is medium, then mode is HDR; (5) if motion is small and dynamic range is small and light intensity is large, then mode is STILL+LTM; (6) if motion is small and dynamic range is large and light intensity is large, then mode is HDR; (7) if motion is large and dynamic range is small and light intensity is small, then mode is STILL; (8) if motion is large and dynamic range is large and light intensity is small, then mode is STILL; (9) if motion is large and dynamic range is small and light intensity is medium, then mode is STILL+LTM; (10) if motion is large and dynamic range is large and light intensity is medium, then mode is STILL+LTM; (11) if motion is large and dynamic range is small and light intensity is large, then mode is STILL+LTM; and (12) if motion is large and dynamic range is large and light intensity is large, then mode is STILL+LTM.

Evaluating the fuzzified values using the fuzzy inference rules includes determining scores for each of the fuzzified values. For example, a small value of motion may have a score of X, where a large value of motion may have a score of Y. In another example, a small value of dynamic range may have a score of M, where a large value of dynamic range may have a score of N. In yet another example, a small value of light intensity may have a score of A, where a medium value of light intensity may have a score of B, and where a large value of light intensity may have a score of C. The scores for each of the three fuzzified values are multiplied to determine a score for a given one of the fuzzy inference rules.

The fuzzy inference rule associated with the highest resulting score may be selected. In some cases, there may be multiple fuzzy inference rules that correspond to a single camera mode. In such a case, the fuzzy inference rule having the highest score for that single camera mode is used instead of the other fuzzy inference rules. The mode selection unit708may then select the camera mode corresponding to the selected fuzzy inference rule.

The mode selection unit708then defuzzifies the fuzzy values used for the selected fuzzy inference rule. For example, defuzzifiying fuzzy values may include plotting a three-dimensional decision cube of those fuzzy values for tuning. For example, the mode selection unit708may include a temporal smoothing unit724. The temporal smoothing unit724processes the fuzzy values corresponding to the selected fuzzy inference rule using temporal smoothing filtering, such as to avoid instabilities. For example, the temporal smoothing unit724can process given fuzzy values as Fi,t=alphai*Fi,t−1+(1−alphai)*Fi,t, where t means time or frame index and i means “Dark”, “Bright”, “Gyro”, “Spatial”, “Histo, and the like, for example. In some implementations, the temporal smoothing unit724may be external to the mode selection unit708.

As a result, a selected camera mode726is selected. The selected camera mode726may then be used, such as by an image capture unit (e.g., the image capture unit408described with respect toFIG.4), to capture an image.

Referring next toFIG.8, a pipeline800used to implement a third technique for automated camera mode selection is shown. The third technique implemented using the pipeline800represents an extension of the second technique implemented using the pipeline700described with respect toFIG.7. The third technique introduces spatial analysis for improving the output of the HDR estimation unit702and the use of exposure values for improving the output of the light intensity estimation unit706. In particular, the third technique uses spatial information in addition to or in place of bright and dark pixel information for HDR detection.

The pipeline800includes an HDR estimation unit802, a motion estimation unit804, a light intensity estimation unit806, and a mode selection unit808(which may, for example, be the HDR estimation unit502, the motion estimation unit504, and the light intensity estimation unit506described with respect toFIG.5, and the mode selection unit412described with respect toFIG.4). The output of each of the HDR estimation unit802, the motion estimation unit804, and the light intensity estimation unit806is expressed as a fuzzy value between 0.0 and 1.0.

According to the third technique, the HDR estimation unit802receives two input values810and812(e.g., the first input508described with respect toFIG.5). Processing is performed similar to that described with respect to the HDR estimation unit702of the pipeline700described with respect toFIG.7. In particular, the sum of dark pixels (FDark) is determined and the sum of bright pixels (FBright) is determined, using one or more dedicated weighting functions. The sums FDarkand FBrightare then normalized to determine an output814as a fuzzy value FHDR.

However, whereas other techniques for automated camera mode selection use the sums FDarkand FBrightto determine whether HDR is detected, the HDR estimation unit802further uses spatial information816for the dark pixels and for the bright pixels to detect HDR. For example, the HDR estimation unit802operates under the principle that a scene with HDR should have a difference of intensity between the center and the border of the scene (e.g., in backlight conditions). As such, the HDR estimation unit802uses the spatial information816to detect differences in the background and in the foreground of the scene. The spatial information816may include, for example, a saliency map or a similar mechanism.

The absolute difference between the average of the background and foreground regions of the scene can be normalized to detect HDR. For example, a value FHistocan be defined as the product of FDarkand FBright. Using the spatial analysis introduced within the HDR estimation unit802, the output814(FHDR) can be determined based on whether the spatial information816(e.g., backlight detection) is used along with or instead of the sums FDarkand FBright. For example, where the spatial information816is used along with the sums FDarkand FBright, the output814can be expressed as the product of FHistoand FSpatial. In an implementation, FSpatialis the normalized value (between 0 and 1) of a pattern difference value, where a pattern is a small 3×3 matrix that is composed of black and white areas. This pattern is applied to the thumbnail to compute the difference between the white areas and the black areas. In another example, where the spatial information816is used instead of the sums FDarkand FBright, the output814can be expressed as the difference between a first value and a second value, where the first value is the sum of FHistoand FSpatialand where the second value is the product of FHistoand FSpatial.

The motion estimation unit804receives one input value818(e.g., the second input510described with respect toFIG.5). Processing is performed similar to that described with respect to the motion estimation unit704of the pipeline700described with respect toFIG.7. In particular, the motion estimation unit804determines whether the input value818is higher than the threshold TGyroSpeed. If the input value818is higher than TGyroSpeed, the motion estimation unit804determines that camera motion is detected. However, if the input value818is not higher than TGyroSpeed, a SAD is determined between two consecutive thumbnails (e.g., the current and the previous thumbnails). The SAD value is then normalized to produce normalized value FImg. As such, if the input value818is lower than TGyroSpeed, an output820of the motion estimation unit804, FMotion, is expressed as a fuzzy value representing FImg. Otherwise, the output820is expressed as a fuzzy value representing FGyro.

The light intensity estimation unit806receives one input value822(e.g., the third input512described with respect toFIG.5). Processing is performed similar to that described with respect to the light intensity estimation unit706of the pipeline700described with respect toFIG.7. In particular, the input value822is compared against a threshold for light intensity. The input value822is then normalized to produce an output824. The output824represents a fuzzy value of FLightIntensity.

The mode selection unit808receives the output814, the output820, and the output824. Processing is performed similar to that described with respect to the mode selection unit708of the pipeline700described with respect toFIG.7. In particular, the mode selection unit808generally performs fuzzification against the output814, the output820, and the output824, evaluates the fuzzy values thereof in view of rules, aggregates the output of those rules, and performs defuzzification against the aggregated output (e.g., using a temporal smoothing unit826). In some implementations, the temporal smoothing unit826may be external to the mode selection unit808. As a result, a selected camera mode828is selected. The selected camera mode828may then be used, such as by an image capture unit (e.g., the image capture unit408described with respect toFIG.4), to capture an image.

Referring next toFIG.9, a pipeline900used to implement a fourth technique for automated camera mode selection is shown. The fourth technique implemented using the pipeline900represents an extension of the first technique implemented using the pipeline600described with respect toFIG.6. The fourth technique allows for the light intensity to be estimated using exposure values or camera ISO and further uses cuboids defined by tuning for the mode selection, such as instead of using fuzzy values.

In particular, the fourth technique uses parameter tuning to define the areas of activation for each camera mode within a three-dimensional decision cube. Values for each of the camera modes correspond to defined, non-overlapping three-dimensional regions within the three-dimensional decision cube. The three-dimensional decision cube may, for example, be the three-dimensional decision cube produced and used in connection with the second technique and/or the third technique, respectfully described above with respect toFIGS.7and8.

The pipeline900includes an HDR estimation unit902, a motion estimation unit904, a light intensity estimation unit906, a mode selection unit908(which may, for example, be the HDR estimation unit502, the motion estimation unit504, and the light intensity estimation unit506described with respect toFIG.5, and the mode selection unit412described with respect toFIG.4), and a parameter tuning unit910.

According to the fourth technique, the HDR estimation unit902receives two input values912and914(e.g., the first input508described with respect toFIG.5). Processing is performed similar to that described with respect to the HDR estimation unit602of the pipeline600described with respect toFIG.6. In particular, the HDR estimation unit902determines whether a scene for the image to be captured is HDR based on where the two input values912and914are with respect to the weighting curves. The HDR estimation unit902outputs an output916as a result of that determination. The output916is a Boolean indicator in which a first value (e.g., true) indicates that the HDR estimation unit902determined the scene to be HDR and in which a second value (e.g., false) indicates that the HDR estimation unit902determined the scene to be non-HDR.

The motion estimation unit904receives one input value918(e.g., the second input510described with respect toFIG.5). Processing is performed similar to that described with respect to the motion estimation unit610of the pipeline600described with respect toFIG.6. In particular, the motion estimation unit904determines whether the input value918is higher than a first threshold. If the input value918is higher than the first threshold, the motion estimation unit904determines that camera motion is detected. If the input value918is not higher than the first threshold, a SAD is determined between the current and the previous thumbnails. If the SAD is higher than a second threshold, the motion estimation unit904determines that scene motion is detected. If the SAD is not higher than the second threshold, the motion estimation unit904determines that no motion is detected. The motion estimation unit904outputs an output920, a Boolean indicator, as a result of the one or two determinations.

The light intensity estimation unit906receives one input value922(e.g., the third input512described with respect toFIG.5). Processing is performed similar to that described with respect to the light intensity estimation unit616of the pipeline600described with respect toFIG.6. In particular, the input value922represents a scene luminance estimated within exposure correction operations. The input value922is compared against a threshold for light intensity. If the input value922is greater than the threshold, the light intensity estimation unit906determines that the light intensity is high. If the input value922is less than the threshold, the light intensity estimation unit906determines that the light intensity is low. If the input value922is neither greater nor less than the threshold, the light intensity estimation unit906determines that the light intensity is medium.

The light intensity estimation unit906produces an output924to indicate the results of the comparison between the input value922and the threshold. However, whereas the input value for the light intensity estimation unit616of the pipeline600is an exposure value, the input value922may be an exposure value or an ISO value. As such, the light intensity estimation unit906may produce the output924based on comparisons between the described thresholds and the ISO value received as the input value922.

The output916, the output920, and the output924may each be represented as a set of three values, each between 0.0 and 1.0. Those values correspond to a location within a three-dimensional region of the three-dimensional decision cube. The parameter tuning unit910receives the output916, the output920, and the output924and determines, based on the values included in each of those, the three-dimensional region of the three-dimensional decision cube to which the output916, the output920, and the output924correspond. Data indicative of that three-dimensional region is then passed to the mode selection unit908, which selects the selected camera mode926as the camera mode that corresponds to that three-dimensional region. The selected camera mode926may then be used, such as by an image capture unit (e.g., the image capture unit408described with respect toFIG.4), to capture an image.

The parameter tuning unit910is also used to update the three-dimensional decision cube used by the mode selection unit908to select the camera mode. That is, over time, the three-dimensional regions of the three-dimensional decision cube may change in size or position, such as based on the inputs received by the pipeline900(e.g., the input402described with respect toFIG.4). The parameter tuning unit910processes these changes to update the three-dimensional decision cube accordingly.

In some implementations, the parameter tuning unit910may not be located before the mode selection unit908in the pipeline900. For example, the mode selection unit908may directly receive the output916, the output920, and the output924. The mode selection unit908may then use the parameter tuning unit910to identify the selected camera mode926. For example, the mode selection unit908can send the three-dimensional index values of the output916, the output920, and the output924to the parameter tuning unit910. The parameter tuning unit910may then to query the three-dimensional decision cube for the selected camera mode926according to those three-dimensional index values.

Referring next toFIG.10, a pipeline1000used to implement a fifth technique for automated camera mode selection is shown. The fifth technique implemented using the pipeline1000represents an extension of the fourth technique implemented using the pipeline900described with respect toFIG.9. The fifth technique improves the temporal smoothing filtering functionality of the automated camera mode selection process by further smoothing the output of the temporal smoothing. In particular, the fifth technique uses temporal smoothing filtering in addition to parameter tuning to select a camera mode based on output from processing units of the pipeline1000.

The pipeline1000includes an HDR estimation unit1002, a motion estimation unit1004, a light intensity estimation unit1006, a mode selection unit1008(which may, for example, be the HDR estimation unit502, the motion estimation unit504, and the light intensity estimation unit506described with respect toFIG.5, and the mode selection unit412described with respect toFIG.4), and a parameter tuning unit1010(which may, for example, be the parameter tuning unit910described with respect toFIG.9).

According to the fifth technique, the HDR estimation unit1002receives two input values1010and1012(e.g., the first input508described with respect toFIG.5). Processing is performed similar to that described with respect to the HDR estimation unit602of the pipeline600described with respect toFIG.6. In particular, the HDR estimation unit1002determines whether a scene for the image to be captured is HDR based on where the two input values1012and1014are with respect to the weighting curves. The HDR estimation unit1002outputs an output1016as a result of that determination. The output1016is a Boolean indicator in which a first value (e.g., true) indicates that the HDR estimation unit1002determined the scene to be HDR and in which a second value (e.g., false) indicates that the HDR estimation unit1002determined the scene to be non-HDR.

The motion estimation unit1004receives one input value1018(e.g., the second input510described with respect toFIG.5). Processing is performed similar to that described with respect to the motion estimation unit610of the pipeline600described with respect toFIG.6. In particular, the motion estimation unit1004determines whether the input value1018is higher than a first threshold. If the input value1018is higher than the first threshold, the motion estimation unit1004determines that camera motion is detected. If the input value1018is not higher than the first threshold, a SAD is determined between the current and the previous thumbnails. If the SAD is higher than a second threshold, the motion estimation unit1004determines that scene motion is detected. If the SAD is not higher than the second threshold, the motion estimation unit1004determines that no motion is detected. The motion estimation unit1004outputs an output1020, a Boolean indicator, as a result of the one or two determinations.

The light intensity estimation unit1006receives one input value1022(e.g., the third input512described with respect toFIG.5). Processing is performed similar to that described with respect to the light intensity estimation unit616of the pipeline600described with respect toFIG.6. In particular, the input value1022represents a scene luminance estimated within exposure correction operations. However, the input value1022may in some cases be an ISO value instead of an exposure value. The input value1022is compared against a threshold for light intensity. If the input value1022is greater than the threshold, the light intensity estimation unit1006determines that the light intensity is high. If the input value1022is less than the threshold, the light intensity estimation unit1006determines that the light intensity is low. If the input value1022is neither greater nor less than the threshold, the light intensity estimation unit1006determines that the light intensity is medium.

The output1016, the output1020, and the output1024may each be represented as a set of three values, each between 0.0 and 1.0. Those values correspond to a location within a three-dimensional region of the three-dimensional decision cube. The parameter tuning unit1010receives the output1016, the output1020, and the output1024and determines, based on the values included in each of those, the three-dimensional region of the three-dimensional decision cube to which the output1016, the output1020, and the output1024correspond. Data indicative of that three-dimensional region is then passed to the mode selection unit1008.

The mode selection unit1008includes a temporal smoothing unit1028(e.g., the temporal smoothing unit724described with respect toFIG.7). The temporal smoothing unit1028processes the values corresponding to the three-dimensional region using temporal smoothing filtering, such as to avoid instabilities. In some implementations, the temporal smoothing unit1028may be external to the mode selection unit1008. The selected camera mode1026may then be used, such as by an image capture unit (e.g., the image capture unit408described with respect toFIG.4), to capture an image.

However, in addition to the temporal smoothing filtering performed against the input to the mode selection unit708, the temporal smoothing unit1028in the pipeline1000also performs temporal smoothing against the output. The temporal smoothing filtering on the output is a kind of median filtering on a window containing past values and works on the last N unsmoothed output, where N is between 1 and 20, to produce a smoothed output. The smoothed output represents the majority mode of the N values of the temporal smoothing filter window.

After a camera mode is selected, and after the search of the three-dimensional decision cube for the three-dimensional region, the selected camera mode1026is added to a buffer of previously unsmoothed camera modes of length N. A histogram of those N values is computer to select the camera mode therefrom having the greatest number of occurrences within those N values. That camera mode is the majority mode used as the smoothed value. In some cases, where two or more camera modes share the majority, the smoothed value is the previous smoothed value, such as to prevent oscillations.

In some implementations, the parameter tuning unit1010may also be used to update the three-dimensional decision cube used by the mode selection unit1008to select the camera mode. That is, over time, the three-dimensional regions of the three-dimensional decision cube may change in size or position, such as based on the inputs received by the pipeline1000(e.g., the input402described with respect toFIG.4). The parameter tuning unit1010processes these changes to update the three-dimensional decision cube accordingly.

In some implementations, the parameter tuning unit1010may not be located before the mode selection unit1008in the pipeline1000. For example, the mode selection unit1008may directly receive the output1016, the output1020, and the output1024. The mode selection unit1008may then use the parameter tuning unit1010to identify the selected camera mode1026. For example, the mode selection unit1008can send the three-dimensional index values of the output1016, the output1020, and the output1024to the parameter tuning unit1010. The parameter tuning unit1010may then to query the three-dimensional decision cube for the selected camera mode1026according to those three-dimensional index values.

Referring next toFIG.11, a pipeline1100used to implement a sixth technique for automated camera mode selection is shown. The sixth technique implemented using the pipeline1100represents an extension of the fifth technique implemented using the pipeline1000described with respect toFIG.10. The sixth technique introduces tuning recommendations for use in the automated camera mode selection process. In particular, the sixth technique introduces tuning recommendations for face scores of 0.0 to 1.0 for local tone mapping.

The pipeline1100includes an HDR estimation unit1102, a motion estimation unit1104, a light intensity estimation unit1106, and a mode selection unit1108(which may, for example, be the HDR estimation unit502, the motion estimation unit504, and the light intensity estimation unit506described with respect toFIG.5, and the mode selection unit412described with respect toFIG.4).

According to the sixth technique, the HDR estimation unit1102receives two input values1110and1112(e.g., the first input508described with respect toFIG.5). Processing is performed similar to that described with respect to the HDR estimation unit602of the pipeline600described with respect toFIG.6. In particular, the HDR estimation unit1102determines whether a scene for the image to be captured is HDR based on where the two input values1112and1114are with respect to the weighting curves. The HDR estimation unit1102outputs an output1116as a result of that determination. The output1116is a Boolean indicator in which a first value (e.g., true) indicates that the HDR estimation unit1102determined the scene to be HDR and in which a second value (e.g., false) indicates that the HDR estimation unit1102determined the scene to be non-HDR.

The motion estimation unit1104receives one input value1118(e.g., the second input510described with respect toFIG.5). Processing is performed similar to that described with respect to the motion estimation unit610of the pipeline600described with respect toFIG.6. In particular, the motion estimation unit1104determines whether the input value1118is higher than a first threshold. If the input value1118is higher than the first threshold, the motion estimation unit1104determines that camera motion is detected. If the input value1118is not higher than the first threshold, a SAD is determined between the current and the previous thumbnails. If the SAD is higher than a second threshold, the motion estimation unit1104determines that scene motion is detected. If the SAD is not higher than the second threshold, the motion estimation unit1104determines that no motion is detected. The motion estimation unit1104outputs an output1120, a Boolean indicator, as a result of the one or two determinations.

The light intensity estimation unit1106receives one input value1122(e.g., the third input512described with respect toFIG.5). Processing is performed similar to that described with respect to the light intensity estimation unit616of the pipeline600described with respect toFIG.6. In particular, the input value1122represents a scene luminance estimated within exposure correction operations. However, the input value1122may in some cases be an ISO value instead of an exposure value. The input value1122is compared against a threshold for light intensity. If the input value1122is greater than the threshold, the light intensity estimation unit1106determines that the light intensity is high. If the input value1122is less than the threshold, the light intensity estimation unit1106determines that the light intensity is low. If the input value1122is neither greater nor less than the threshold, the light intensity estimation unit1106determines that the light intensity is medium.

The output1116, the output1120, and the output1124may each be represented as a set of three values, each between 0.0 and 1.0. Those values correspond to a location within a three-dimensional region of the three-dimensional decision cube. The parameter tuning unit1110receives the output1116, the output1120, and the output1124and determines, based on the values included in each of those, the three-dimensional region of the three-dimensional decision cube to which the output1116, the output1120, and the output1124correspond. Data indicative of that three-dimensional region is then passed to the mode selection unit1108.

The mode selection unit1108includes a temporal smoothing unit1128(e.g., the temporal smoothing unit724described with respect toFIG.7). The temporal smoothing unit1128processes the values corresponding to the three-dimensional region using temporal smoothing filtering, such as to avoid instabilities. The temporal smoothing unit1128also performs temporal smoothing against the output, such as described above with respect to the temporal smoothing unit1028ofFIG.10. In some implementations, the temporal smoothing unit1128may be external to the mode selection unit1108. The selected camera mode1126may then be used, such as by an image capture unit (e.g., the image capture unit408described with respect toFIG.4), to capture an image.

The mode selection unit1108also outputs tuning recommendations1130. The tuning recommendations1130are based on face scores for LTM. A face score is a value from 0.0 to 1.0 and indicates whether there are faces in an image to capture or not. That is, LTM should adapt tuning where a face score is 1.0 indicating that a big face is in the image, but LTM may not need to adapt tuning where the face score is 0.0 indicating no face in the image or a face too small or too out of focus. Where the tuning recommendations1130reflect a higher value, the area of the face can be normalized by a display window, such as to provide a more robust zoom and field of view (e.g., wide versus linear) function. A smoothing operation may also be performed on the tuning recommendations1130to avoid oscillations due to face detection instabilities. For example, the smoothing operation may be expressed as faceScoreSmoothed=alpha*previousFaceScoreSmoothed+(1−alpha)*currentFaceScore.

Further details of implementations and examples of techniques performed using the systems and pipelines described with respect toFIGS.1-11are now described.FIG.12is a flowchart showing an example of a technique1200for automated camera mode selection. The technique1200can be performed, for example, using hardware and/or software components of an image capture system, such as the image capture device100shown inFIGS.1A-Dor the image capture device200shown inFIG.2. For example, the image capture device100or the image capture device200may include one or more software components that process an image captured using an image capture device of the image capture device100or the image capture device200, for example, to perform automated camera mode selection, such as described in one or more of the pipeline400, the pipeline600, the pipeline700, the pipeline800, the pipeline900, the pipeline1000, or the pipeline1100.

In another example, the technique1200can be performed using an integrated circuit. The integrated circuit may, for example, be a field programmable gate array (e.g., FPGA), programmable logic device (PLD), reconfigurable computer fabric (RCF), system on a chip (SoC), ASICs, and/or another type of integrated circuit. An image processor of the integrated circuit includes a camera mode selection unit and/or an image capture unit (e.g., a processor having one or multiple cores) configured to execute instructions to perform some or all of the technique1200.

Although the technique1200is described with respect to a series of operations, the operations comprising the technique1200may be performed in orders other than those described herein. In some implementations, the technique1200may include additional, fewer, or different operations than those described herein.

At1202, inputs corresponding to dynamic range, motion, and light intensity are received. At1204, HDR is estimated based on the dynamic range inputs. Estimating the HDR based on the dynamic range inputs can include using control points of curves for LUTs to compare values of dark and bright pixels to one or more thresholds. At1206, motion is estimated based on the motion input. Estimating the motion can include comparing an angular speed measured with respect to the motion input to one or more thresholds. At1208, light intensity is estimated based on the light intensity input. Estimating the light intensity can include comparing measurements of the scene luminance for the image to capture to one or more thresholds.

At1210, the HDR, motion, and light intensity outputs are temporally smoothed. Temporally smoothing the HDR, motion, and light intensity outputs can include using those outputs to identify a three-dimensional region of a three-dimensional decision cube. For example, the HDR, motion, and light intensity outputs may each be expressed as a set of values indicating a three-dimensional coordinate location. That location can be identified within a three-dimensional region of the three-dimensional decision cube.

At1212, a camera mode is selected. Selecting the camera mode can include identifying a camera mode corresponding to the three-dimensional region of the three-dimensional decision cube identified by the temporal smoothing. At1214, the selected camera mode is used to capture an image. Capturing an image using the selected camera mode can include adjusting settings of an image capture device according to configurations of the selected camera mode.

Some or all of the technique1200may repeat continuously until user input indicating to capture the image is received. For example, a processor or image capture device implementing the technique1200may continuously perform the operations for estimating the HDR, motion, and/or light intensity and/or for performing temporal smoothing until user input indicating to capture the image is received. The user input may, for example, be represented by a user of a device configured for capturing the image interacting with an interface element of the device (e.g., a physical button or a portion of a touch screen).

In some implementations, the technique1200can include selecting the same camera mode for capturing a second image. For example, subsequent to capturing the image using the selected camera mode, user input indicating to capture a second image may be received (e.g., by the user interacting with an interface element to capture the second image). Based on aspects of the scene of the image or a time at which the image was captured, the same camera mode may be selected for capturing the second image.

For example, where the scene of the captured image is similar (e.g., based on a threshold value) to a scene of the second image to capture, the same camera mode can be selected. In another example, where the user input indicating to capture the second image is received within a threshold amount of time (e.g., 1 second) after the first image is captured, the same camera mode can be selected. Selecting the same camera mode in either of these ways prevents additional resources from being spent to determine a camera mode to use when not much has changed since the most recent camera mode selection.

In some implementations, the technique1200can include determining that the scene of the image to capture is dark and selecting a dark setting camera mode in response. For example, one or more of the inputs for the dynamic range, motion, or light intensity may indicate that the image is to be captured during night time or otherwise in a dark area. In such an implementation, an auto-night camera mode may be selected. Selecting the auto-night camera mode may include bypassing or otherwise ignoring aspects of the technique1200that would otherwise be used for selecting a camera mode, for example, the temporal smoothing.

Where certain elements of these implementations may be partially or fully implemented using known components, those portions of such known components that are necessary for an understanding of the present disclosure have been described, and detailed descriptions of other portions of such known components have been omitted so as not to obscure the disclosure.

In the present specification, an implementation showing a singular component should not be considered limiting; rather, the disclosure is intended to encompass other implementations including a plurality of the same component, and vice-versa, unless explicitly stated otherwise herein. Further, the present disclosure encompasses present and future known equivalents to the components referred to herein by way of illustration.

As used herein, the term “bus” is meant generally to denote any type of interconnection or communication architecture that may be used to communicate data between two or more entities. The “bus” could be optical, wireless, infrared, or another type of communication medium. The exact topology of the bus could be, for example, standard “bus,” hierarchical bus, network-on-chip, address-event-representation (AER) connection, or other type of communication topology used for accessing, for example, different memories in a system.

As used herein, the terms “computer,” “computing device,” and “computerized device” include, but are not limited to, personal computers (PCs) and minicomputers (whether desktop, laptop, or otherwise), mainframe computers, workstations, servers, personal digital assistants (PDAs), handheld computers, embedded computers, programmable logic devices, personal communicators, tablet computers, portable navigation aids, Java 2 Platform, Micro Edition (J2ME) equipped devices, cellular telephones, smartphones, personal integrated communication or entertainment devices, or another device capable of executing a set of instructions.

As used herein, the term “computer program” or “software” is meant to include any sequence of machine-cognizable steps which perform a function. Such program may be rendered in any programming language or environment including, for example, C/C++, C#, Fortran, COBOL, MATLAB™, PASCAL, Python, assembly language, markup languages (e.g., HTML, Standard Generalized Markup Language (SGML), XML, Voice Markup Language (VoxML)), as well as object-oriented environments such as the Common Object Request Broker Architecture (CORBA), Java™ (including J2ME, Java Beans), and/or Binary Runtime Environment (e.g., Binary Runtime Environment for Wireless (BREW)).

As used herein, the terms “connection,” “link,” “transmission channel,” “delay line,” and “wireless” mean a causal link between two or more entities (whether physical or logical/virtual) which enables information exchange between the entities.

As used herein, the terms “integrated circuit,” “chip,” and “IC” are meant to refer to an electronic circuit manufactured by the patterned diffusion of trace elements into the surface of a thin substrate of semiconductor material. By way of non-limiting example, integrated circuits may include FPGAs, PLDs, RCFs, SoCs, ASICs, and/or other types of integrated circuits.

As used herein, the term “memory” includes any type of integrated circuit or other storage device adapted for storing digital data, including, without limitation, read-only memory (ROM), programmable ROM (PROM), electrically erasable PROM (EEPROM), DRAM, Mobile DRAM, synchronous DRAM (SDRAM), Double Data Rate 2 (DDR/2) SDRAM, extended data out (EDO)/fast page mode (FPM), reduced latency DRAM (RLDRAM), static RAM (SRAM), “flash” memory (e.g., NAND/NOR), memristor memory, and pseudo SRAM (PSRAM).

As used herein, the terms “microprocessor” and “digital processor” are meant generally to include digital processing devices. By way of non-limiting example, digital processing devices may include one or more of DSPs, reduced instruction set computers (RISCs), general-purpose complex instruction set computing (CISC) processors, microprocessors, gate arrays (e.g., field programmable gate arrays (FPGAs)), PLDs, RCFs, array processors, secure microprocessors, ASICs, and/or other digital processing devices. Such digital processors may be contained on a single unitary IC die, or distributed across multiple components.

As used herein, the term “network interface” refers to any signal, data, and/or software interface with a component, network, and/or process. By way of non-limiting example, a network interface may include one or more of FireWire (e.g., FW400, FW110, and/or other variations), USB (e.g., USB2), Ethernet (e.g., 10/100, 10/100/1000 (Gigabit Ethernet), 10-Gig-E, and/or other Ethernet implementations), MoCA, Coaxsys (e.g., TVnet™), radio frequency tuner (e.g., in-band or out-of-band, cable modem, and/or other radio frequency tuner protocol interfaces), Wi-Fi (802.11), WiMAX (802.16), personal area network (PAN) (e.g., 802.15), cellular (e.g., 3G, LTE/LTE-A/TD-LTE, GSM, and/or other cellular technology), IrDA families, and/or other network interfaces.

As used herein, the term “Wi-Fi” includes one or more of IEEE-Std. 802.11, variants of IEEE-Std. 802.11, standards related to IEEE-Std. 802.11 (e.g., 802.11 a/b/g/n/s/v), and/or other wireless standards.

As used herein, the term “wireless” means any wireless signal, data, communication, and/or other wireless interface. By way of non-limiting example, a wireless interface may include one or more of Wi-Fi, Bluetooth, 3G (3GPP/3GPP2), High Speed Downlink Packet Access/High Speed Uplink Packet Access (HSDPA/HSUPA), Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA) (e.g., IS-95A, Wideband CDMA (WCDMA), and/or other wireless technology), Frequency Hopping Spread Spectrum (FHSS), Direct Sequence Spread Spectrum (DSSS), Global System for Mobile communications (GSM), PAN/802.15, WiMAX (802.16), 802.20, narrowband/Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiplex (OFDM), Personal Communication Service (PCS)/Digital Cellular System (DCS), LTE/LTE-Advanced (LTE-A)/Time Division LTE (TD-LTE), analog cellular, Cellular Digital Packet Data (CDPD), satellite systems, millimeter wave or microwave systems, acoustic, infrared (i.e., IrDA), and/or other wireless interfaces.

As used herein, the term “robot” may be used to describe an autonomous device, autonomous vehicle, computer, artificial intelligence (AI) agent, surveillance system or device, control system or device, and/or other computerized device capable of autonomous operation.

As used herein, the terms “camera,” or variations thereof, and “image capture device,” or variations thereof, may be used to refer to any imaging device or sensor configured to capture, record, and/or convey still and/or video imagery which may be sensitive to visible parts of the electromagnetic spectrum, invisible parts of the electromagnetic spectrum (e.g., infrared, ultraviolet), and/or other energy (e.g., pressure waves).

While certain aspects of the technology are described in terms of a specific sequence of steps of a method, these descriptions are illustrative of the broader methods of the disclosure and may be modified by the particular application. Certain steps may be rendered unnecessary or optional under certain circumstances. Additionally, certain steps or functionality may be added to the disclosed implementations, or the order of performance of two or more steps may be permuted. All such variations are considered to be encompassed within the disclosure.

While the above-detailed description has shown, described, and pointed out novel features of the disclosure as applied to various implementations, it will be understood that various omissions, substitutions, and changes in the form and details of the devices or processes illustrated may be made by those skilled in the art without departing from the disclosure. The foregoing description is in no way meant to be limiting, but rather should be taken as illustrative of the general principles of the technology.