Cinematic image blur in digital cameras based on exposure timing manipulation

Cinematic motion blur and other cinematic effects are enabled during image capture through exposure timing manipulation. The resulting captured images and videos include one or more cinematic effects without the need for a user to impose the cinematic effects post-capture.

BACKGROUND

1. Field of Art

The disclosure generally relates to the field of digital image capture, and more specifically to the manipulation of exposure timing to achieve cinematic image blurring.

2. Description of the Related Art

The advancement of digital video and image encoding has led to increasingly sophisticated image capture techniques. The capture of digital images by an image capture device (hereinafter “camera”) is performed by an image sensor. Many types of image sensors are commonly used in cameras and other image-capturing devices, such as charge-coupled devices (CCDs) and complementary metal-oxide-semiconductors (CMOSs). Image sensors convert light, such as light entering the aperture of a camera through a camera lens, into image information. In this way, a camera can “capture” objects before it by converting the light reflected from the objects and passing through the camera lens into an image.

Modern image sensors are able to capture a large amount of light in a relatively small window. This allows cameras to capture frames with very small exposures, which produces very sharp images. However, such image capture can produce undesirably crisp images when played back in video format. In some applications, such as cinema, it is desirable to have a high amount of motion blur in each image, which requires relatively long exposure times. However, due to the high light sensitivity of modern images sensors, images with long exposure times become over exposed, and are not suitable for viewing. Therefore, a process is needed to allow modern image sensors to achieve motion blur without over exposing the image.

DETAILED DESCRIPTION

Configuration Overview

The example embodiments described herein beneficially allow for the inclusion of various cinematic effects in images during the capture of the images by a camera, as opposed to creating these effects post-image capture. One example of a cinematic effect is the motion blur found in images and video captured by traditional film cameras. This motion blur is a desirable effect, as it is often associated with big-budget theatrical releases. Depending on the intensity of the motion blur, it can also create a retro effect often found in hand-held cameras. Another example of a cinematic effect is color emphasis, where one of the three primary colors (red, blue, or green) is emphasized or deemphasized during image capture. In the case where red is emphasized, it creates an image that looks warmer than it actually is, and may be useful for artificially creating the look of dawn or dusk in an image.

System Architecture

FIG. 1illustrates a block diagram of a camera100including an image capture system enabling exposure timing manipulation according to one example embodiment. The camera100includes an image sensor chip110, a settings register120, and an image store130. The settings register receives cinematic effect data140, and outputs reset160, expose170, and read180signals to the image sensor chip based on the received cinematic effect data140. The image sensor chip receives light150, and based on the reset, expose, and read signals, outputs electrical image data190based on the received light to the image store. For the sake of clarity,FIG. 1depicts only one representative camera, though any possible camera, such as a 3D camera with, for example, a plurality of image sensor chips and settings registers, can support exposure timing manipulation.

The image sensor chip110captures light150incident upon the image sensor for conversion by the camera100into electrical image data190. The image sensor chip is described in greater detail inFIG. 3. The image sensor chip also receives reset160, expose170, and read180signals from the settings register120, which control respectively the reset, expose, and read functionalities of the image sensor chip. In many embodiments, the image sensor chip is a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS), which converts photons incident upon the image sensor chip to stored charges. The image sensor chip also includes circuitry (not illustrated in the embodiment ofFIG. 1) to convert these stored charges into a digital signal that forms the basis for the electrical image data190. In addition, the image sensor chip may also include a lens or filter, such as a Bayer filter, or any other component commonly included in image sensor chips, though these components are omitted from the embodiment ofFIG. 1for the purposes of simplicity.

The settings register120receives cinematic effect data140, for instance from a user of the camera100, and converts the received cinematic effect data into reset160, expose170, and read180signals, which are subsequently transmitted to the image sensor chip110. The settings register is described in greater detail inFIG. 3. In various embodiments, the settings register is contained partially or entirely within the image sensor chip.

The image store130stores electrical image data190produced by the image sensor chip110. The image store may be any type of non-transitory computer-readable storage media, for example a hard drive or flash memory. The image store may store the electrical image data as RAW image data, or it may first convert or compress the data into a known image or video format, such as JPG or MPEG, prior to storing the electrical image data. In various embodiments, for example when the camera is used to capture movies, the image store is only a temporary store, holding a small number of frames that are subsequently transmitted to a main image store (not shown). For the purposes of this application, any reference to an “image store” is meant to include the image store130and any other supplementary image stores as used in embodiments of the invention.

The cinematic effect data140instructs the settings register120to generate the reset160, expose170, and read180signals to produce a particular cinematic effect. For example, the cinematic effect described by the cinematic effect data might be a “1950's hand-held camera” effect, and in response to receiving the cinematic effect data, the settings register generates the reset, expose, and read signals required to produce such an effect. An alternative cinematic effect includes a degree of motion blur (e.g., “90 degree exposure”) to apply to an image. The cinematic effect data might also explicitly specify the exposure times (e.g., “Exposure from 0 to 0.01 seconds, from 0.03 to 0.04 seconds, and from 0.06 to 0.07 seconds”). If exposure settings have been pre-programmed in the settings register, the cinematic effect data might simply specify the pre-programmed setting (e.g., “Exposure pattern 3”). The process by which cinematic effect data becomes reset, expose, and read signals is described in more detail below with respect toFIG. 2.

The reset160, expose170, and read180signals control the reset, expose, and read functionalities of the image sensor chip110, respectively. In general, each of the reset, expose, and read signals comprise a digital pulse train, wherein a first value (e.g., 1) instructs the image sensor to enable that feature, and a second value (e.g., 0) instructs the image sensor to disable that feature, though other configurations may be utilized in other embodiments. Example expose pulse trains are shown inFIG. 4. In various embodiments, certain features (like expose) might have a plurality of “enable” states, and the pulse pattern may then step between more than two values. The reset, expose, and read signals are described in more detail below with respect toFIG. 3.

The electrical image data190is the image data produced by the image sensor chip110and representing the light incident upon the image sensor chip during one or more exposure periods. As described above, the electrical image data190may be produced in a variety of formats, and may include a variety of effects as specified by the cinematic effect data.

FIG. 2illustrates a settings register for supporting exposure timing manipulation according to one example embodiment. The settings register includes a settings register interface module200, a settings register store210, a cinematic effect processor220, a reset pulse pattern module260, an expose pulse pattern module270, and a read pulse pattern module280.

The settings register interface module200receives cinematic effect data from the camera100and sends reset, expose, and read signals produced by the settings register120to the image sensor chip110. When the settings register receives cinematic effect data140, the interface module collects the data, and provides it to the cinematic effect processor220. When the reset260, expose270, and read280pulse pattern modules have generated reset160, expose170, and read180signals, the interface module outputs these signals to the image sensor chip110. In some embodiments, any updates to the firmware or settings of the settings register will be received by the interface module and sent to the settings register store210.

The settings register store210stores data representing various cinematic image effect settings. For example, the settings register store may store pre-determined conversion algorithms for converting cinematic effect data140to reset160, expose170, and read180pulse signals configured to produce a particular cinematic effect associated with the cinematic effect data. The settings register store210may additionally store user preferences. For example, a user may specify, using an interface on the camera (not illustrated in the embodiment ofFIG. 2), an exposure timing manipulation setting for storage in the settings register store and subsequent use. The settings register store may also store firmware for the settings register, which may, for example, define the protocol used for outputting the reset, expose, and read signals via the settings register interface200.

The cinematic effect processor220converts cinematic effect data140into data for use by the reset260, expose270, and read280pulse pattern modules. The cinematic effect processor receives the cinematic effect data from the interface module200, and retrieves a conversion algorithm from the settings register store210based on the cinematic effect data. Based on the retrieved conversion algorithm, the cinematic effect processor determines the appropriate timing and duration for the reset160, expose170, and read180signals, and instructs the reset260, expose270, and read280pulse pattern modules, respectively, to create the appropriate pulse patterns for the rest, expose, and read signals.

In one example, if the cinematic effect data specifies a “180 degree exposure with 4 sub-exposures” effect, the retrieved conversion algorithm first determines a frame duration, an exposure window, and a net exposure time associated with the cinematic effect data, and determines that each frame should have 4 sub-exposures spaced evenly within the first ½ of the frame (corresponding with a 180 degree exposure window). In this example, the cinematic effect processor220determines that the first sub-exposure is ¼ of the length of the net exposure time and starts at the beginning of the frame. It also determines that the other three sub-exposures are the same length and are spread equally throughout the first ½ of the frame duration. Additional cinematic effects and the reset, expose, and read signals associated with such effects are discussed in greater detail below with regards toFIG. 4.

The reset260, expose270, and read280pulse pattern modules generate reset, expose, and read pulse patterns, respectively, based on the instructions of the cinematic effect processor. Each of the modules receives parameters for a pulse pattern from the cinematic effect processor220. For example, the cinematic effects processor220may send a list of initial signal values and signal switch times (indicating when the value of a particular signal is to switch) to each of the reset, expose, and read modules. From these parameters, the reset, expose, and read modules each generate a pulse train signal, and send the generated pulse train signals to the interface module200for outputting as the reset160, expose170, and read180signals, respectively.

Referring next toFIG. 3, it illustrates the operation of an image sensor chip110according to one example embodiment. The image sensor chip includes three pixels, pixel 1310, pixel 2320, and pixel 3330. “Pixel” as used herein represents an area of the image sensor chip110on which incident light is converted into information describing an associated image pixel. The pixels may use a CCD, a CMOS, or any image sensor capable of converting photons of light to an electrical signal. Each pixel receives the light150and outputs electrical image data190representing the light based on the received reset160, expose170, and read180signals. Although not illustrated in the embodiment ofFIG. 3, each pixel includes one or more capacitors configured to store charge produced by the pixel in response to being exposed to light incident upon the pixel. The image sensor chip110may also have a lens or filter (not shown) through which the light passes before reaching the pixels. In general, the image sensor chip110will comprise thousands or millions of pixels arrayed in a grid, but only three are shown here, in this configuration, for clarity.

The reset signal160controls the reset functionality of each pixel. The pixel, upon receiving an enabled reset signal, is configured to discharge any charge stored in the pixel's capacitors, and reset any internal registers storing light and image data at various stages of transition through the pixel. Upon receiving an enabled expose signal, the pixel is configured to expose itself to light incident upon the pixel. For example, upon receiving an enabled expose signal, the pixel begins capturing light, and upon receiving a disabled expose signal, the pixel stops capturing light. Note that because the expose signal may be a pulse train, the pixel may capture light incident upon it during each of several sub-exposures for a particular frame in a short period of time. In some embodiments, the charges produced by these sub-exposures are accumulated inside the capacitors of the pixel. Upon receiving an enabled read signal, the pixel is configured to output information representing the one or more exposures of the pixel. For example, upon receiving an enabled read signal, the pixel outputs information representing the stored charge of the pixel. In this embodiment, the pixel may convert the stored charge into a digital representation of the stored charge using, for example, an analog-to-digital converter (not illustrated in the embodiment ofFIG. 3). The information representing the one or more exposures of the pixels output by each pixel is combined to form the electrical image data190. In some embodiments, the electrical image data is digitally processed by the image sensor chip110prior to being output by the image sensor chip110.

Example Frame Sequences for Enabling Cinematic Effects

FIG. 4illustrates example frame sequences showing various examples of exposure timing manipulation according to one example embodiment. Each sequence is three frames long, and is representative of various exposure timing manipulation schemes. It should be noted that although three-frame sequences are illustrated in the embodiment ofFIG. 4, frame sequences of any length, including a single frame sequence, can benefit from exposure timing manipulation. The horizontal axis indicates the passage of time, with time progressing to the right, and the vertical axis indicates the exposure signal170, where a low value indicates a disabled exposure signal, and a high value indicates an enabled exposure signal. The dotted exposure window, E, represents a traditional film camera exposure window associated with a desired cinematic effect. Though not shown, before capturing a frame, the image sensor chip110may be reset, wherein the reset signal160is enabled for some period of time and then disabled. Also not shown, after capturing a frame, the image sensor chip110may be read out, wherein the read signal180is enabled for some period of time and then disabled.

Sequence400A illustrates a traditional digital camera exposure pattern, contrasted against a traditional film camera 90-degree exposure window, E. Of note, the film camera exposure is much longer than a digital exposure, which creates the appearance of motion blur on the film frame. Due to the heightened sensitivity of the image sensor chip110relative to the film camera, the digital exposure occurs faster. Due to differences in the technology between the film and digital cameras, both capture substantially the same amount of light, even though the exposure times differ. Sequence400A shows three separate frames with substantially identical exposure durations, e. However, in various embodiments, different frames may have different exposure durations to account for variable amounts of light. For example, during the time of the first frame, the camera may be pointed at a very bright light source, so the total exposure time may be relatively short in order to prevent over exposure, and during the time of the second frame, the camera may be pointed at a very dark scene, so the total exposure time may be relatively long in order to capture sufficient light. It should be noted that exposure durations and locations within a frame duration occur may occur entirely within an exposure window E associated with a traditional film camera exposure period.

Sequence400B illustrates a sequence of frames with a modified exposure timing exposure pattern, contrasted against a traditional film camera 90-degree exposure window. Each frame in this sequence has two sub-exposures. Since there are two sub-exposures, each sub-exposure duration se is substantially the same as ½ of the single exposure duration e in the sequence400A. This allows the total exposure time to be substantially equal to the single exposure duration e, resulting in a substantially similar amount of total captured light by the image sensor chip110. In various embodiments, the sub-exposures will have different sub-exposure times (see sequence400C) in order to compensate for variable brightness. Note that the second sub-exposure starts at a time substantially equal to ½ of the exposure window, E. As the sub-exposures occur over a wider range of the frame, this allows each frame to capture more motion, and therefore to have more motion blur. If the sub-exposures were attempting to replicate a traditional film 180-degree exposure window, then the second sub-exposure start time would be substantially equal to ½ of the traditional film 180-degree exposure window duration.

Sequence400C illustrates a sequence of frames with various examples of modified exposure timing exposure patterns, contrasted against a traditional film camera 90-degree exposure window. In this sequence, both modified exposure timings (see frames 1 and 3) and a normal single exposure (see frame 2) are used. The first frame shows an example embodiment exposure pattern wherein the exposure spacing duration, d, between the frames is substantially the same, but the individual sub-exposure durations se are different. In this embodiment, there are three sub-exposures, with the first sub-exposure having a longer duration than the second, and the second having a longer duration than the third, each occurring within the exposure window E. This changes the characteristics of the frame, weighing the image motion at the beginning of the frame more heavily than the image motion at the middle or end of the frame. The second frame has a normal exposure (see sequence400A above). The third frame shows an example embodiment exposure pattern wherein the exposure spacing duration, d, between the frames decreases after each subsequent sub-exposure, but the individual sub-exposure durations, se, are substantially the same. In this embodiment, the exposure timing decreases approximately exponentially. This exposure pattern weighs the image motion at the middle or end of the frame more heavily than the image motion at the beginning of the frame.

Capturing Image Data Using Exposure Timing Manipulation

FIG. 5is a flow chart for an example process of capturing image data using exposure timing manipulation. A camera, e.g., camera100, receives500the cinematic effect data described a particular cinematic effect for use by the camera, such as motion blur. The cinematic effect data may originate from a user (for example, by manual entry of the cinematic effect data with a keypad or other interface), or from a separate computer (not shown) associated with the camera.

An image sensor exposure pattern is determined510by the camera based on the received cinematic effect data. The determined exposure pattern represents a starting time and ending time for a pattern of one or more image sensor sub-exposures within a particular image frame. The determined exposure pattern may be converted into a signal for use by the camera in manipulating the exposure to light of a camera image sensor.

The image sensor is exposed520to light in a pattern based on the determined exposure pattern. The light captured by the image sensor during the exposure pattern is converted into image data and stored530at the camera, for instance in a non-transitory computer-readable storage medium at the camera. Prior to storage, the image data may be manipulated, formatted, or otherwise processed by the image sensor, or by another component of the camera.

Additional Configuration Considerations

A configuration as described herein advantageously enables the inclusion of various cinematic effects in images at the time the images are captured by a camera. For example, effects such as motion blur can be simulated at the time of image capture. Moreover, an intensity of the motion blur can also be controlled to create effects such as a retro effect. Another example of a cinematic effect is color emphasis, where one of the three primary colors (red, blue, or green) is emphasized or deemphasized during image capture. In the case where red is emphasized, it creates an image that looks warmer than it actually is, and may be useful for artificially creating the look of dawn or dusk in an image.