Electronic device for scene evaluation and image projection onto non-planar screens

An electronic device may have one or more imaging sensors. The imaging sensors may be used in combination with an optional mechanical gesture to analyze the lighting conditions of the environment around the electronic device. The electronic device may set auto-exposure, auto-white balance, and auto-focus settings based on the analysis. The electronic device may include a shaped display. The imaging sensors may be used in calibration of a projector for the shaped display and may be used in sending touch inputs associated with the shaped display. The electronic device may be able to capture a photograph during video capture. The electronic device may generate a display screen that identifies which portions of the scene being imaged correspond to the video and which portions correspond to the photograph.

BACKGROUND

The present invention relates to imaging systems and, more particularly, to imaging systems that may use scene evaluation in improving image quality, imaging systems that may be used in supporting shaped displays, and imaging systems that simultaneously capture images having multiple aspect ratios.

Electronic devices such as cellular telephones are often provided with camera sensors. When capturing an image (and when capturing video), the camera sensors may, as examples, perform auto-white balance, auto-exposure, and auto-focus processes. The trio of auto-white balance, auto-exposure, and auto-focus processes may sometimes be referred to as a 3A convergence process. Typically, the 3A convergence process involves capturing a low-resolution preview image and then analyzing that image to determine appropriate imager settings for white balance, exposure, and focus. Once appropriate imager settings are determined, a camera sensor can capture a full-resolution image using the automatically selected settings. The dependence of traditional camera sensors on a single preview image limits image quality, as the image settings derived from that single preview image are often not optimal.

Traditional displays and touch surfaces are rigid (i.e., non-flexible), planar, and rectangular in shape. Display and touch surface designers are therefore unable to provide displays that are flexible, non-planar, and/or non-rectangular (e.g., randomly-shaped displays).

DETAILED DESCRIPTION

Digital camera modules are widely used in electronic devices. An electronic device with a digital camera module is shown inFIG. 1. Electronic device10may be a digital camera, a laptop computer, a display, a computer, a cellular telephone, or other electronic device. Device10may include one or more imaging systems such as imaging systems12A and12B (e.g., camera modules12A and12B) each of which may include one or more image sensors14and corresponding lenses. During operation, a lens focuses light onto an image sensor14. The lens may have fixed aperture. The pixels in image sensor14include photosensitive elements that convert the light into digital data. Image sensors may have any number of pixels (e.g., hundreds or thousands or more). A typical image sensor may, for example, have millions of pixels (e.g., megapixels). In high-end equipment, sensors with 10 megapixels or more are not uncommon. In at least some arrangements, device10may include two (or more) image sensors14, which may capture images from different perspectives. When device10includes two image sensors14, device14may be able to capture stereo images.

Still and video image data from camera sensor14may be provided to image processing and data formatting circuitry16via path26. Image processing and data formatting circuitry16may be used to perform image processing functions such as adjusting white balance and exposure and implementing video image stabilization, image cropping, image scaling, etc. Image processing and data formatting circuitry16may also be used to compress raw camera image files if desired (e.g., to Joint Photographic Experts Group or JPEG format).

In a typical arrangement, which is sometimes referred to as a system on chip or SOC arrangement, camera sensor14and image processing and data formatting circuitry16are implemented on a common integrated circuit15. The use of a single integrated circuit to implement camera sensor14and image processing and data formatting circuitry16can help to minimize costs. If desired, however, multiple integrated circuits may be used to implement circuitry15. In arrangements in which device10includes multiple camera sensors14, each camera sensor14and associated image processing and data formatting circuitry16can be formed on a separate SOC integrated circuit (e.g., there may be multiple camera system on chip modules such as modules12A and12B). Circuitry15conveys data to host subsystem20over path18. Circuitry15may provide acquired image data such as captured video and still digital images to host subsystem20.

Electronic device10typically provides a user with numerous high level functions. In a computer or advanced cellular telephone, for example, a user may be provided with the ability to run user applications. To implement these functions, electronic device10may have input-output devices22such as projectors, keypads, input-output ports, and displays and storage and processing circuitry24. Storage and processing circuitry24may include volatile and nonvolatile memory (e.g., random-access memory, flash memory, hard drives, solid state drives, etc.). Storage and processing circuitry24may also include processors such as microprocessors, microcontrollers, digital signal processors, application specific integrated circuits, etc.

An example of an arrangement for sensor array14is shown inFIG. 2. As shown inFIG. 2, device10may include an array14of pixels28coupled to image readout circuitry30and address generator circuitry32. As an example, each of the pixels in a row of array14may be coupled to address generator circuitry32by one or more conductive lines34. Array14may have any number of rows and columns. In general, the size of array14and the number of rows and columns in array14will depend on the particular implementation. While rows and columns are generally described herein as being horizontal and vertical rows and columns may refer to any grid-like structure (e.g., features described herein as rows may be arranged vertically and features described herein as columns may be arranged horizontally).

Address generator circuitry32may generate signals on paths34as desired. For example, address generator circuitry32may generate reset signals on reset lines in paths34, transfer signals on transfer lines in paths34, and row select (e.g., row readout) signals on row select lines in paths34to control the operation of array14. If desired, address generator circuitry32and array14may be integrated together in a single integrated circuit (as an example).

Signals34, generated by address generator circuitry32as an example, may include signals that dynamically adjust the resolution of array14. For example, signals34may include binning signals that cause pixels28in a first region of array14to be binned together (e.g., with a 2-pixel binning scheme, with a 3-pixel binning scheme, or with a pixel binning scheme of 4 or more pixels) and that cause pixels28in a second region of array14to either not be binned together or to be binned together to a lesser extent than the first region. In addition, signals34may cause pixels28in any number of additional (e.g., third, fourth, fifth, etc.) regions of array14to be binned together to any number of different, or identical, degrees (e.g., 2-pixel binning schemes, 3-or-more-pixel binning schemes, etc.).

Image readout circuitry30may include circuitry42and image processing and data formatting circuitry16. Circuitry42may include sample and hold circuitry, analog-to-digital converter circuitry, and line buffer circuitry (as examples). As one example, circuitry42may be used to measure signals in pixels28and may be used to buffer the signals while analog-to-digital converters in circuitry42convert the signals to digital signals. In a typical arrangement, circuitry42reads signals from rows of pixels28one row at a time over lines40. With another suitable arrangement, circuitry42reads signals from groups of pixels28(e.g., groups formed from pixels located in multiple rows and columns of array14) one group at a time over lines40. The digital signals read out by circuitry42may be representative of charges accumulated by pixels28in response to incident light. The digital signals produced by the analog-to-digital converters of circuitry42may be conveyed to image processing and data formatting circuitry16and then to host subsystem20(FIG. 1) over path18.

As shown inFIG. 3, electronic device10may include one or more camera sensors such as camera sensors14A,14B,14C,14D,14E, and14F. As examples, camera sensor14A may be a primary camera sensor located on a first side (e.g., a front side) of device10and camera sensors14B,14C,14D,14E, and14F may be secondary camera sensors located on second, third, fourth, fifth, and sixth sides, respectively (e.g., rear, top, left, right, and bottom sides, respectively).

At least in arrangements in which electronic device10includes multiple camera sensors14located on at least two different sides of device10, the camera sensors may be used in analyzing and monitoring the environment surrounding device10. Consider, as an example, an arrangement in which device10includes front facing camera sensor14A and rear facing camera sensor14B. In such an arrangement, device10can use camera sensors14A and14B (and/or any other camera sensors present in device10) in studying the scene about to be imaged (or being imaged) by camera sensor14A and in studying the environment around device10. Sensors14A and14B may be used to determine the location of light sources in the environment around device10, to identify characteristics of those light sources (e.g., the color temperature of the light sources, the color profile of the light sources, whether and how the light sources are moving within the environment around device10, the brightness of the light sources, etc.). With this information, device10can automatically optimize imager settings such as exposure, white balance, and focus for the image sensors of device10(e.g., for a primary sensor such as sensor14A), to maximize the quality of images captured by device10.

With some suitable arrangements, device10may utilize a mechanical movement or gesture when obtaining information about its surrounding environment with one or more camera sensors. As an example, in an arrangement in which device10includes a single camera sensor12A, device10may, prior to capturing images, prompt a user of device10to rotate device10(e.g., in a full 360 degree rotation, or in another suitable motion). In this example, the prompt presented by device10may instruct the user to hold device10out at arm's length and spin around in a full circle. While device10is moving (e.g., rotating), device10can use camera sensor12A (and any additional sensors present) to determine the lighting characteristics of its surrounding environment. If desired, sensor12A and any additional sensors present may operate in a high speed image capture mode when device10is utilize a mechanical gesture to analyze its surrounding environment. As another example, in an arrangement in which device10includes a front camera sensor12A and a rear camera sensor12B, a 180 degree rotation may be sufficient. In general (e.g., in single or multi-camera systems), it may be desirable for the motion of device10to include sufficient motion for device10to analyze its surrounding environment (e.g., a full spherical analysis of the environment around device10, a partial spherical analysis of the environment around device10, a partial spherical analysis of the environment around device10including a half-sphere of the region above device10, etc.).

If desired, information on the environment surrounding device10may be cached (i.e., stored) for later use. As an example, when device10determines the lighting characteristics of its surrounding environment, those lighting characteristics may be used to determine appropriate 3A settings for images captured shortly thereafter. In addition, data on the lighting characteristics for a particular location (which may be a location frequented by device10) may be stored and, whenever device10returns to that particular location (as determined by position sensing circuitry24or by user input) the stored data may be used to determine appropriate 3A settings.

In at least some arrangements, device10may combine information from the sensors ofFIG. 3with information from additional sensors such as position sensing circuitry17. As an example, device10may use sensors14A and14B to determine the location of the sun in an outdoor environment. Device10may then use sensors such as an accelerometer to determine that device10is moving within the outdoor environment. Device10may then be able to track the changing position of the sun relative to device10without having to continually determine the position of the sun from the camera sensors14of device10. In such arrangements, device10may periodically reestablish the position of the sun using the camera sensors14of device10, to ensure that any drift errors from the accelerometer do not accumulate over time. Such arrangements may allow device10to automatically and continually adjust its 3A settings (auto-white balance, auto-focus, and auto-exposure).

In general, device10may use any available sensors and inputs such as global positioning system (GPS) circuitry, accelerometers, compasses, magnetometers, clocks, etc. to determine the location of device10and lighting conditions around device10. As examples, GPS circuitry may be used to determine if device10is traveling in a vehicle, accelerometer circuitry may be used to determine if device10is rotating (e.g., is on a carousel or is otherwise being rotated), and the location of the sun relative to device10based on the current time, date, compass settings, and location.

A flowchart of illustrative steps involved in using one or more camera sensors14and/or sensors such as circuitry23in analyzing the environment around device10(e.g., the lighting conditions) and capturing an image are shown inFIG. 4.

In step44, device10may use image sensors such as sensors14A,14B,14C,14D,14E, and14F and/or sensors such as GPS circuitry, accelerometers, clocks, compasses, magnetometers, etc. to identify lighting characteristics of the environment around device10(which is about to be imaged by device10). If desired, step44may involve movement of device10by a user (e.g., a 360 degree, 180 degree, or other mechanical movement of device10). For example, device10may use information from these sensors to determine the locations of light sources in the environment, the brightness of light sources in the environment, the color temperature of light sources in the environment, whether any of the light sources are moving relative to device10, whether the brightness and/or color temperature of any of the light sources are changing, etc.

In step46, device10may use information gather in step46to identify suitable 3A settings for one or more camera sensors and may capture and process one or more images using the camera sensors using those 3A settings. The 3A setting may include one or more of an auto-exposure setting, an auto-white balance setting, and an auto-focus setting. As an example, in an arrangement in which device10determined in step44that the scene to be imaged in step46is an outdoor scene at sunset with the sun behind device10, device10may set its 3A settings to optimize picture quality for any images captured in step46.

If desired, electronic device10may include a shaped display such as display60having a shaped screen62ofFIG. 5(e.g., an input-output device22) and device10may include camera sensors such as camera sensors14G and14H for calibration of shaped display60and/or for sensing touch inputs on shaped display screen62. Camera sensors14G and14H may be any suitable type of camera sensors. As an example, camera sensors14G and14H may be high dynamic range imaging sensors, to ensure sufficient image quality in varied lighting conditions (as may be present in an automobile, as an example). If desired, sensors14G and14H may be sensitive to infrared light, visible light, or infrared light and visible light, as examples. As shown inFIG. 5, shaped display60may be a display having a shaped screen62with any desired shape, including non-rectangular and/or non-planar formats. Shaped display screen62may, if desired, be a flexible display screen, which may facilitate formation of shaped display screen62into a desired shape. As one example, shaped display screen62may be in a dashboard in an automobile and have a non-planar and/or non-rectangular shape.

Device10may include projector48; touch display processor50including warping engine52, dewarping engine54, and image signal processing (ISP) and image computational (ICE) engine56, and processing circuitry58.

With at least some arrangements, the projection of images onto the shaped screen62may be accomplished using warping engine54(in addition or alternatively, projector48may include lens structures that at least partially warp the projected image). In particular, projector48, if driven with non-warped display signals, may require a substantially planar and/or rectangular display screen for proper operation (e.g., for the display to be in focus and to scale across the entire display screen). Therefore, in arrangements of the present invention in which display screen62is non-planar and/or non-rectangular, projector48may be driven with warped display signals that ensure that images projected onto screen62are in focus and to scale across the entirety of the active portion of screen62. In other words, since display screen62is “warped” from a traditional rectangular and planar display screen, the output of projector62may also be warped (using warping engine52) to compensate for the “warping” (from the rectangular and planar norms) of display screen62.

Camera sensors14G and14H may be used in calibrating warping and dewarping engines52and54. Calibration operations may be performed at any desired time. As an example, calibration operations may be performed upon power up of device10. In arrangements in which curved display60may be powered up after (e.g., separate from) powering up device10, calibration operations may be performed upon power up of curved display60. Calibration operations may include projecting a known pattern (such as predetermined grid lines) with projector48, imaging the projected pattern using camera sensors14G and14H, and analyzing the images from sensors14G and14H to determine appropriate warping and dewarping settings for engines52and54. If desired, calibration operations may involve an iterative process that repeats until the projected pattern fits onto curved display screen62correctly (e.g., is in focus and to scale across the entirety of the display screen).

Alternatively or in addition to calibration operations, camera sensors14G and14H may be used in sensing touch inputs on shaped display screen62. In particular, during normal operation sensors14G and14H, dewarping engine54and circuitry56may identify user touch inputs on the surface of display screen62, including the location of the inputs within the display screen62. When touch inputs are detected, circuitry such as circuitry56and58may provide appropriate command information to host subsystem20. If desired, camera sensors14G and14H may be configured to capture stereo images. In such arrangements, camera sensors14G and14H, dewarping engine54, and circuitry56may identify user input in a projected space64in front of the display screen62. In general, curved display60can perform calibration operations and/or identify touch inputs even in arrangements in which display60includes only a single camera sensor such as sensor14G.

A flowchart of illustrative steps involved in calibrating a shaped display such as shaped display60and identifying touch inputs on the shaped display are shown inFIG. 6.

In step64, a known pattern may be projected onto the shaped display. As an example, projector48nay project a predetermined pattern such as an array of parallel lines (e.g., grid lines) onto shaped display screen62.

In step66, one or more image sensors may capture one or more images of the projected pattern. For example, camera sensors14G and14H may capture one or more images of the pattern projected onto screen62in step64.

In step68, differences between the projected pattern (as imaged in step66) and the expected pattern (e.g., the desired appearance of the pattern when projected onto screen62) may be determined and appropriate warping and dewarping settings generated. The generated warping and dewarping settings may be used to configure engines54and56.

As illustrated by dashed lines74, the processes of steps64,66, and68may be repeated. As an example, the processes of steps64,66, and68may be repeated as part of an iterative process that repeats until the differences between the projected pattern (as imaged in step66) and the expected pattern are less than a predetermined threshold. Alternatively or in addition, the processes of steps64,66, and68may be repeated periodically. Such an arrangement may be beneficial in embodiments in which display screen62is flexible and/or subject to warping over time.

In step70, images (e.g., data) may be projected onto display screen62by projector48. The projected data may be appropriately warped by warping engine52such that the projected images are in-focus, to-scale, and fill the active region of display screen62.

In step72, one or more camera sensors such as camera sensors14G and14H may be used to identify touch inputs on the surface of display screen62(or in region64in front of display screen62). As an example, camera sensors14G and14H, together with dewarping engine54and circuitry56and58may be used to identify shadows on the surface of display screen62indicative of a user's finger pressing against a portion of display screen62.

In accordance with various embodiment of the present invention, device10may include a camera sensor14capable of simultaneously capturing images having different aspect ratios and may include a display that facilitates the simultaneous capture of images having different aspect ratios. As an example, camera14may be used in capturing video in a first format (e.g., a widescreen 16:9 format or any other suitable format) and, without interrupting video capture operations, camera14may take a photograph in a second format (e.g., a snapshot in a 4:3 format or any other suitable format. With some suitable arrangements, the simultaneous capturing of images having different aspect ratios may be accomplished with a high speed camera sensor14capable of capturing the photograph in-between adjacent frames of the video. In order to assist a user of device10in simultaneously capturing images having different aspect ratios, device10may include a display (e.g., an input-output device22, a shaped display60, etc.) that displays screen75. Display screen75may display real-time previews of images being captured or images that could be captured by one or more image sensors in device10. Display screen75may include a central region76in the first format (e.g., the widescreen 16:9 format) and additional regions78that, in combination with the central region76, are in the second format (e.g., the 4:3 snapshot format). If desired, the additional regions78may be presented in such a way as to identify to the user that the regions78are not part of the video format but are part of the photograph format. As examples, the additional regions78may be dimmed relative to the central region76and/or the additional regions78may be alpha blended to provide a semi-transparent appearance to regions78. Both regions76and78may be filled with image data of a scene being imaged by camera sensor14of device10.

If desired, display screen75may include a side frame80include one or more user-selectable icons such as icons82A,82B, and82C. As examples, icon82A may be an icon allowing a user to start and stop (or pause) a video record, icon82B may be an icon allowing a user to take a single (or multiple) photograph, and icon82C may be an icon that enables and disables the dimming and/or alpha blending of the additional regions78.

FIG. 8illustrates a simplified block diagram of imager200(e.g., an imager that may analyze its surrounding environment to determined 3A settings, that may support a shaped display, and that may simultaneously capture video and photographs having different aspect ratios). Pixel array201includes a plurality of pixels containing respective photosensors arranged in a predetermined number of columns and rows. The row lines are selectively activated by row driver202in response to row address decoder203and the column select lines are selectively activated by column driver204in response to column address decoder205. Thus, a row and column address is provided for each pixel.

CMOS imager200is operated by a timing and control circuit206, which controls decoders203,205for selecting the appropriate row and column lines for pixel readout, and row and column driver circuitry202,204, which apply driving voltages to the drive transistors of the selected row and column lines. The pixel signals, which typically include a pixel reset signal Vrst and a pixel image signal Vsig for each pixel are sampled by sample and hold circuitry207associated with the column driver204. A differential signal Vrst-Vsig is produced for each pixel, which is amplified by amplifier208and digitized by analog-to-digital converter209. The analog to digital converter209converts the analog pixel signals to digital signals, which are fed to image processor210which forms a digital image.

FIG. 9shows in simplified form a typical processor system300, such as a digital camera, which includes an imaging device such as imaging device200(e.g., an imager that may analyze its surrounding environment to determined 3A settings, that may support a shaped display, and that may simultaneously capture video and photographs having different aspect ratios). Processor system300is exemplary of a system having digital circuits that could include imaging device200. Without being limiting, such a system could include a computer system, still or video camera system, scanner, machine vision, vehicle navigation, video phone, surveillance system, auto focus system, star tracker system, motion detection system, image stabilization system, and other systems employing an imaging device.

Processor system300, which may be a digital still or video camera system, may include a lens such as lens396for focusing an image onto a pixel array such as pixel array201when shutter release button397is pressed. Processor system300may include a central processing unit such as central processing unit (CPU)395. CPU395may be a microprocessor that controls camera functions and one or more image flow functions and communicates with one or more input/output (I/O) devices391over a bus such as bus393. Imaging device200may also communicate with CPU395over bus393. System300may include random access memory (RAM)392and removable memory394. Removable memory394may include flash memory that communicates with CPU395over bus393. Imaging device200may be combined with CPU395, with or without memory storage, on a single integrated circuit or on a different chip. Although bus393is illustrated as a single bus, it may be one or more buses or bridges or other communication paths used to interconnect the system components.

Various embodiments have been described illustrating imaging systems that may use scene evaluation in improving image quality, imaging systems that may be used in supporting shaped displays, and imaging systems that simultaneously capture video and photographs having different ratios.

An imaging system may be used to identify lighting characteristics of its surrounding environment. As an example, an imaging system may use one or more camera sensors together with an optional mechanical gesture to identify the locations and color temperature of light sources and other lighting characteristics of the surrounding environment. After determining the lighting characteristics of the surrounding environment, the imaging system may use the determined lighting characteristics is generating imager settings such as auto-exposure, auto-white balance, and auto-focus settings. The determined lighting characteristics may be stored for later use, which may be especially beneficial if the imaging system frequently captures images in on or more favorite locations.

An imaging system may be used in supporting a shaped display. The imaging system may be used in calibrating warping and dewarping engines for the shaped display, such that images projected into a shaped display screen are properly displayed. In addition or alternatively, the imaging system may be used in identifying user touch input on the shaped display screen and/or user spatial input in front of the shaped display screen.

An imaging system may be capable of capturing video in a first format and, while capturing the video, capturing a photograph in a second format. The imaging system may be incorporated into an electronic device having a display. The display may be configured to display a screen including a preview of the video in the first format and a preview of the photograph in the second format, where some portions of the photograph have been altered to visually distinguish portions of the second format that do not overlap with portions of the first format.