Controlling image capture and/or controlling image processing

A method including selecting one or more portions of a scene; controlling an image sensor to capture, at a relatively wide field of view, a wide-field-of-view image of a scene; controlling the image sensor to capture, at a relatively narrow field of view or narrow fields of view, one or more narrow-field-of-view images of the one or more selected portions of the scene.

TECHNOLOGICAL FIELD

Embodiments of the present invention relate to controlling image capture and/or controlling image processing.

BACKGROUND

It is now common practice for images to be captured, stored and displayed as an array of pixels.

A image sensor may have an array of sensor elements each of which is configured to capture a pixel of the image.

As image sensors use increasingly large numbers of sensor elements, the images captured by the image sensor become larger. The transport routes for the images therefore need to have greater bandwidth and increased memory capacity may be required to store the images.

BRIEF SUMMARY

According to various, but not necessarily all, embodiments of the invention there is provided a method comprising: selecting one or more portions of a scene; controlling an image sensor to capture, at a relatively wide field of view, a wide-field-of-view image of the scene; and controlling the image sensor to capture, at a relatively narrow field of view or narrow fields of view, one or more narrow-field-of-view images of the one or more selected portions of the scene.

According to various, but not necessarily all, embodiments of the invention there is provided an apparatus comprising: at least one processor; and at least one memory including computer program code the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform:

selection of one or more portions of a scene;

control of an image sensor to capture, at a relatively wide field of view, a wide-field-of-view image of a scene;

control of the image sensor to capture, at a relatively narrow field of view or narrow fields of view, one or more narrow-field-of-view images of the one or more portions of the scene.

According to various, but not necessarily all, embodiments of the invention there is provided a method comprising: selecting one or more portions of a scene; storing in a memory for subsequent retrieval a wide-field-of-view data structure recording an image of the scene captured at a relatively wide field of view; and storing in a memory for subsequent retrieval one or more narrow-field-of-view data structures recording one or more images restricted to the one or more portions of the scene.

According to various, but not necessarily all, embodiments of the invention there is provided a method comprising: receiving a wide-field-of-view image; receiving one or more narrow-field-of-view images; forming by a processor a composite wide-field-of-view image using the wide-field-of-view image and the one or more narrow-field-of-view images.

DETAILED DESCRIPTION

FIGS. 1 and 8A to 8Dillustrate a method10of capturing a wide-field-of-view image22of a scene20and capturing one or more narrow-field-of-view images24of one or more selected portions26of the scene20.

Referring toFIG. 1, the Figure illustrates a method10comprising: at block12, selecting one or more portions26of a scene20;

at block14, controlling an image sensor3to capture, at a relatively wide field of view, a wide-field-of-view image22of the scene20; and

at block16, controlling the image sensor3to capture, at a relatively narrow field of view or narrow fields of view, one or more narrow-field-of-view images24of the one or more selected portions26of the scene20.

As will be described later, the method may be a computerized (computer-implemented) method.

InFIG. 8A, an image sensor3is capturing an image of a scene. A scene is a two dimensional or three dimensional arrangement of photographable elements. The captured image has a wide field of view (FOV), it is a wide field of view image22. The field of view is that part of the scene20that is captured by the image sensor3. In this example, the whole of the scene20is captured by the image sensor3and, referring toFIG. 8B, the whole of the image sensor3is used to capture the wide field of view image22.

InFIG. 8C, an image sensor3is capturing one or more narrow field of view images of the same scene20. The one or more captured images are narrow field of view images24. The field of view is that part of the scene20that is captured by the image sensor3. In this example, as illustrated inFIG. 8D, only one distinct portion26of the scene20is captured by each of one or more sub-sets28of the image sensor3. Each of the one or more sub-sets28of the image sensor3are used to capture the one or more narrow field of view images22. The whole of the image sensor3is not used. The narrow-field-of-view images24are restricted to the one or more portions26of the scene20.

Each of the one or more sub-sets28of the image sensor3may be distinct and non-overlapping and are used to capture narrow field of view images22that are distinct and non-overlapping.

Each of the one or more sub-sets28of the image sensor3may have a different size, and may be controlled to have a new size. Thus the captured narrow field of view images22may have different sizes (different fields of view), and may be controlled to have a new size (field of view).

Selecting one or more portions26of a scene20may be achieved by selecting one or more sub-sets28of the image sensor3used to capture one or more narrow field of view images22. The selection of a sub-set28may involve determining a position and a size.

Although the sub-sets28illustrated are rectangular, other shapes may be used.

The image sensor3may comprise an array of sensor elements. The array may, for example, have M rows and N columns of sensor elements. The image sensor may be used at different spatial resolutions. Spatial resolution for a defined sensor area is the ratio of the number of sensor elements (pixels) used to capture an image in that sensor area to the number of sensor elements (pixels) available to capture that image in that sensor area. Maximum spatial resolution occurs when all the available sensor elements (pixels) in a sensor area are used to capture an image. In this case the operational sensor elements are contiguous. Reduced spatial resolution occurs when all the available sensor elements (pixels) in a sensor area are not used to capture an image. In this case the operational sensor elements are not contiguous but at least some operational sensor elements are separated by non-operational sensor elements.

The image sensor3may be operated with a selected spatial resolution to capture, at a relatively wide field of view, a wide-field-of-view image22of the scene20that has the selected spatial resolution. For example, the image sensor3may be operated with a high spatial resolution to capture, at a relatively wide field of view, a wide-field-of-view image22of the scene20that has a high spatial resolution. Alternatively, the image sensor3may be operated with a low spatial resolution to capture, at a relatively wide field of view, a wide-field-of-view image22of the scene20that has a low spatial resolution.

Likewise, a selected sub-set28of the image sensor3may be operated with a defined spatial resolution to capture, at a relatively narrow field of view, a narrow-field-of-view image24that has the defined spatial resolution.

Referring back toFIG. 1, at block16, the method10may control the image sensor3to capture multiple narrow-field-of-view images24at different camera settings. The multiple narrow-field-of-view images24may be for one portion26of the scene20or for multiple portions26of the scene. The multiple narrow-field-of-view images24may be captured simultaneously or sequentially. That is, the narrow-field-of-view images24may be distributed over time and/or space.

Different camera settings may, for example, involve a different one or more of: focus setting when the image was captured; spatial resolution of the image (number of pixels); and rate of repeated capture of a narrow-field-of-view image24for the same portion26of the scene20.

Camera settings associated with different captured images and their portions can also be associated with the captured image, for example as metadata. The associated camera settings can be integrated with an image data structure for the captured image, for example, in header information or the associated camera setting may be stored as a separate data structure that can be linked to the image data structure. The camera settings may then be transferred with and/or stored with the captured image. It is favorable for the post-processing of the image content that camera settings applied during the time of capture t0,t1,t2,t3are recorded accurately for each image. The image size, cropping and other image capture related settings are stored in a format that has common reference parameters enabling the comparison of camera settings between multiple captured images.

In bothFIGS. 2 and 3, a sequence of captured images is illustrated.

Each sequence comprises capture, at time t0, of a wide-field-of-view image22of the scene20and sequential capture at times t1, t2, t3of different narrow-field-of-view images24of the scene20.

At t0, the image sensor3is controlled to capture using a relatively wide field of view, a single wide-field-of-view image24of the scene20.

At t1, the image sensor3is controlled to capture simultaneously, using a relatively narrow field of view or narrow fields of view, multiple narrow-field-of-view images24of the corresponding multiple portions26of the scene20.

This simultaneous narrow field of view capture is repeated at time t2where the same portions26of the scene20are captured simultaneously, using the same relatively narrow field of view or narrow fields of view, to provide, for time t2, multiple narrow-field-of-view images24of the corresponding multiple portions26of the scene20. The same portions26of the scene20may be captured simultaneously at time t2(with the same focus setting or different focus setting) as were captured simultaneously at time t1.

This simultaneous narrow field of view capture is repeated at time t3where the same portions26of the scene20are captured simultaneously, using the same relatively narrow field of view or narrow fields of view, to provide, for time t3, multiple narrow-field-of-view images24of the corresponding multiple portions26of the scene20. The same portions26of the scene20may be captured simultaneously at time t3(with the same or different focus settings) as were simultaneously captured at times t1and t2.

Thus in this example, each portion26of the scene20is captured at times t1, t2, t3with the same field of view and with the same or a different focus setting. If different focus settings are used, then they may all be different to the focus setting used, at time t0, to capture the wide-field-of-view images22or there may be a common setting. It may be possible to vary other camera settings such as exposure or intensity of a flash over the sequence of image captures at t0to t3.

During the period of time (t1-t3), the image sensor3is controlled to capture exclusively at a relatively narrow field of view or narrow fields of view, multiple different narrow-field-of-view images24. That is during this period the image sensor is controlled not to capture, at a relatively wide field of view, a wide-field-of-view image22of a scene20.

Although these Figures disclose at each time t1, t2, t3the capture of two narrow-field-of-view images24, it should be appreciated that more or less narrow-field-of-view images24may be captured. Each narrow-field-of-view image24is associated with a single portion26of the scene20

At each time t1, t2, t3the same number of narrow-field-of-view images24may be captured.

At each time t1, t2, t3the field of view used to capture narrow-field-of-view images24for the same portion26of the scene may remain constant (fixed).

The field of view used to capture narrow-field-of-view images24for different portions26of the scene may be different.

The time period between times t0, t1, t2and t3may be the same.

In this illustrated example, the image sensor3is controlled to capture using exclusively a relatively wide field of view (t0) and is then controlled to capture using exclusively a narrow field(s) of view (t1-t3)

During the period of capture using exclusively a relatively wide field of view, the image sensor3is controlled to perform n separate capture events separated in time by a time Tn. At each capture event, the image sensor3is controlled to capture using a relatively wide field of view, a single wide-field-of-view image24of the scene20. In the illustrated example n=1, however, in other embodiments n may be varied and have a different value.

During the period of capture using exclusively relatively narrow field(s) of view, the image sensor3is controlled to perform m separate capture events separated in time by a time Tm. At each capture event, the image sensor3is controlled to capture simultaneously, using a relatively narrow field of view or narrow fields of view, multiple narrow-field-of-view images24of the corresponding multiple portions26of the scene20. In the illustrated example m=3, however, in other embodiments m may be varied and have a different value. The values of Tn and Tm may be the same or they may be different.

The rate of capture events at the wide field of view can be expressed as n/(n*Tn+m*Tm). The rate of capture events at the narrow field of view can be expressed as m/(n*Tn+m*Tm). The ratio of the rate of capture events at the wide field of view to the rate of capture events at the narrow field of view can be expressed as n/m. In the illustrated example, this ratio is 1/3.

In the illustrated example, the period of capture using exclusively a relatively wide field of view precedes the period of capture using exclusively a relatively narrow field(s) of view, however, the order may be reversed such that the period of capture using exclusively a relatively narrow field(s) of view precedes the period of capture using exclusively a relatively wide field of view.

In the illustrated example, a single cycle of the period of capture using exclusively a relatively wide field of view and the period of capture using exclusively a relatively narrow field(s) of view is illustrated. In some but not necessarily all embodiments, the cycle of the period of capture using exclusively a relatively wide field of view and the period of capture using exclusively a relatively narrow field(s) of view, may be repeated.

In some but not necessarily all embodiments, some parameters of the cycle remain the same when the cycle is repeated and some (or none) may be varied. Parameters of the cycle may, for example, include one or more of: spatial resolution, the position and/or size of the portions26of the scene20, n, Tn, m, Tm, focus, etc.

Referring toFIG. 2, the one or more narrow-field-of-view images24and the wide-field-of-view image22have the same spatial resolution.

The wide-field-of-view image22has a relatively high spatial resolution H and the one or more narrow-field-of-view images24each has a relatively high spatial resolution H. They may, for example, have maximum spatial resolution.

Referring toFIG. 3, the one or more narrow-field-of-view images24and the wide-field-of-view image22have the different spatial resolution.

The wide-field-of-view image22has a relatively low spatial resolution L and the one or more narrow-field-of-view images24each has a relatively high spatial resolution H.

High spatial resolution may, for example, be maximum spatial resolution or greater than 50% of maximum resolution.

Low spatial resolution may, for example, be less than 50% of maximum spatial resolution.

Referring toFIG. 4, the Fig illustrates a method for forming a low spatial resolution (L), wide-field-of-view image22of a scene20and the one or more high spatial resolution (H) narrow-field-of-view images24of portions26of the scene from an original high spatial resolution (H), wide-field-of-view image22of the scene20.

The original high spatial resolution (H), wide-field-of-view image22is processed (e.g. filtered) to obtain a derivative, low spatial resolution (L), wide-field-of-view image22of the scene20.

The original high spatial resolution (H), wide-field-of-view image22is processed (e.g. cropped) to obtain one or more high spatial resolution, narrow-field-of-view images24.

Referring back toFIG. 1, the block12, requires selection of portion(s)26of a scene20. These portions26are captured at block16using a narrow field(s) of view.

The selection may, for example, be performed manually by a user. As an example, a wide field of view image of the scene20may be presented on a viewfinder display. The user may be able to select a portion26on the display by selecting a photographic element in the scene20or by forming a window around a portion of the scene. If the display viewfinder is a touch sensitive display, selection of a photographic element may occur by touching the display whereas a window may be formed by tracing a rectangle on the display. It may be possible to move and re-size the window.

The selection may, for example, be performed semi-automatically. As described in the preceding paragraph, the user may select a portion of the scene, however, the user-selected portion may be a putative portion. The image of the scene20may be analyzed to re-position and/or re-size the putative portion and define the portion26used for defining which narrow-field-of-view images are captured.

Alternatively, the selection may, for example, be automatic. For example, the image of the scene may be processed to select automatically the one or more portions26of a scene20. As an example, a face recognition algorithm may be used to select portions of the image that correspond to human faces. As another alternative, time separated, low spatial resolution images of the scene20may be processed to select automatically the one or more portions26of a scene20that have changed between the images. A window may be defined for each portion. If windows are clustered, they may be combined to form a larger window or if they overlap they may be re-sized so that they do not overlap. The window may in some but not necessarily all embodiment be displayed on a display of a viewfinder and the user may be able to remove, add, re-position and re-size the windows.

FIG. 5illustrates an example of a method21for forming a composite image25. A composite image25of the scene20is formed using a wide-field-of-view image22and one or more narrow-field-of-view images24.

Each of one or more parts23of the wide-field-of-view image22corresponding to the one or more portions26of the scene20are replaced with the corresponding one or more narrow-field-of-view images24.

The process of replacement may utilize camera settings associated with a narrow-field-of-view image, if any. The process of replacement may utilize camera settings associated with the wide-field-of-view image, if any.

The method21enables independent selection, for each part23(portion26), of which narrow-field-of-view image(s)24are used in forming the composite image25.

In some but not necessarily all embodiments, the selection is performed by a user. As an example, the set of multiple different narrow-field-of-view images24in a set27may be presented as a separate menu for each portion26of the scene (part23of the wide-field-of-view image22). The user can select from the menu for a part23, one or more narrow-field-of-view images24in the set27for replacing that part23of the wide-field-of-view image22in the composite image25.

In other embodiments, the selection is performed automatically. Algorithms may be used to select a ‘best’ one or more narrow-field-of-view images24in the set27for replacing a part23of the wide-field-of-view image22in the composite image25. An example of one algorithm is a face recognition algorithm that is programmed to score narrow-field-of-view images24of faces by increasing a score when a smile is present and decreasing the score if one or more eyes are closed. The algorithm then selects, for each part23, the highest scoring narrow-field-of-view image24. Another example of an algorithm is one that determines which of the available narrow-field-of-view images24is sharpest (has the best focus). This may be achieved, for example, by using a filter to identify interest points and selecting the narrow-field-of-view images24that has most interest points.

In some but not necessarily all embodiments, one of the of the multiple different narrow-field-of-view images24in the set27may be selected by a user to form the composite image25. Thus the narrow-field-of-view image used to correspond to a first portion of the scene20, when forming the composite image, may have been captured at a first time and the narrow-field-of-view image used to correspond to a second portion of the scene20, when forming the composite image, may have been captured at a second different time

In other embodiments, a sequence of the of the multiple different narrow-field-of-view images24in the set27may be selected by a user to form the composite image25. Each selected sequence of multiple different narrow-field-of-view images24in the set27is run as a local video sequence within the wide-field-of-view images22. This forms a cinemagraph.

Thus a sequence of narrow-field-of-view images used for a video at a first portion of the scene20in the composite image, may have been captured over a first time and the sequence of narrow-field-of-view images used for a video at a second portion of the scene20in the composite image25may have been captured over a second different time.

Also, a sequence of narrow-field-of-view images used for a video at a first portion of the scene20in the composite image, may have been captured over a first time and a narrow-field-of-view image used for a second portion of the scene20in the composite image25may have been captured at a second different time or during the first time.

FIG. 6illustrates an example of an apparatus2.

In this example, the apparatus comprises a controller9, a user input6, an image sensor3and a display4.

The controller9is configured to received input commands from the user input device6and provide output commands to the display4. In some embodiments, the user input6and the display4may be combined as a touch sensitive display. The display4may be used as a viewfinder.

The controller9is also configured to communicate with the image sensor3.

The controller9is configured to send commands to the image sensor3that control the capturing of an image by the image sensor3.

Capturing an image may comprise creating a data structure recording the image and sending the data structure to the controller9. The controller9may be configured to store in a memory, for subsequent retrieval by the controller9, the data structure. A wide-field-of-view data structure may record a wide-field-of-view image22and a narrow-field-of-view data structure may record a narrow-field-of-view image24.

The controller9may also control settings of the image sensor3such as focus, spatial resolution, n, Tn, m, Tm, the position and size of the portions26of the scene (e.g. definitions of the field of view for the narrow-field-of-view images24).

The apparatus2may be any device that is capable of processing images. The apparatus2may or may not comprise the image sensor3, user input6and display4. It may, for example, be a module for integration into a host electronic device that comprises those components.

The apparatus2may be an electronic device. It may be a hand-portable electronic device. A hand-portable electronic device is an electronic device that is sized to be held in a human hand when in use and is sized to fit in an inside pocket of a jacket.

Implementation of controller9can be in hardware alone (a circuit, a processor etc), have certain aspects in software including firmware alone or can be a combination of hardware and software (including firmware).

The controller9may be implemented using instructions that enable hardware functionality, for example, by using executable computer program instructions in a general-purpose or special-purpose processor that may be stored on a computer readable storage medium (disk, memory etc) to be executed by such a processor.

InFIG. 6, the example of a controller9comprises a processor5that is configured to read from and write to the memory7. The processor5may also comprise an output interface via which data and/or commands are output by the processor and an input interface via which data and/or commands are input to the processor5.

The memory7stores a computer program8comprising computer program instructions that control the operation of the controller9when loaded into the processor5. The computer program instructions8provide the logic and routines that enables the apparatus to perform the methods illustrated in the Figs and described in the preceding paragraphs. The processor5by reading the memory7is able to load and execute the computer program8.

The apparatus2may therefore comprise: at least one processor5; and at least one memory7including computer program code8the at least one memory7and the computer program code8configured to, with the at least one processor5, cause the apparatus2at least to perform:

selection of one or more portions26of a scene20;

control of the image sensor3to capture, at a relatively wide field of view, a wide-field-of-view image22of the scene20; and

control of the image sensor3to capture, at a relatively narrow field of view or narrow fields of view, one or more narrow-field-of-view images24of the one or more portions26of the scene20.

The apparatus2may therefore comprise: at least one processor5; and at least one memory7including computer program code8the at least one memory7and the computer program code8configured to, with the at least one processor5, cause the apparatus2at least to perform:

receiving the wide-field-of-view image22;

receiving the one or more narrow-field-of-view images24;

forming a composite image25of the scene20with the relatively wide field of view using the wide-field-of-view image22and the one or more narrow-field-of-view images24.

Referring toFIG. 7, the computer program8may arrive at the apparatus2via any suitable delivery mechanism1. The delivery mechanism1may be, for example, a non-transitory computer-readable storage medium, a computer program product, a memory device, a record medium such as a compact disc read-only memory (CD-ROM) or digital versatile disc (DVD), an article of manufacture that tangibly embodies the computer program8. The delivery mechanism may be a signal configured to reliably transfer the computer program8. The apparatus2may propagate or transmit the computer program8as a computer data signal.

Although the memory7is illustrated as a single component it may be implemented as one or more separate components some or all of which may be integrated/removable and/or may provide permanent/semi-permanent/dynamic/cached storage.

As used here ‘module’ refers to a unit or apparatus that excludes certain parts/components that would be added by an end manufacturer or a user.

The blocks illustrated in theFIG. 1may represent stages in a method and/or sections of code in the computer program8. The illustration of a particular order to the blocks does not necessarily imply that there is a required or preferred order for the blocks and the order and arrangement of the block may be varied. Furthermore, it may be possible for some blocks to be omitted.