Patent Description:
The art of video capturing includes acquiring and processing of digital image data to produce a digital video stream to, e.g., analyse or display to a user. Acquiring and processing are performed by a so called image pipeline which is an integral part of any camera system. For many camera systems today, the image pipeline is controlled by automated processes which adjust settings used in handling of image data in the image pipeline.

The image pipeline may be seen as including different components, or blocks, that each handles a specific part in the pipeline. Non-limiting examples of parts that can be controlled in this manner are gain, exposure (shutter), white balance (WB), and encoding. From many of these components of the image pipeline, information about image data that is or has been processed by the components may be sent to some kind of controller. The information can for example be general or specific color information or different forms of pixel value distributions. Based on the information, the controller can determine that an adjustment of a setting of the component is needed in order to improve the capturing or processing. For example, if the controller receives information that indicates that a processed image frame is too dark, e.g. by that the number of pixel values below a certain threshold exceeds a limit, the controller may decide to increase the exposure time setting of an exposure time component. Based on next received image information of the same type, the controller may determine further adjustments. A feedback-based process for determining adjustments of image pipeline settings is thereby formed.

However, this process may be problematic to implement with satisfying results. On one hand, it is difficult to predict how much time it takes until a setting has been applied in the image pipeline, depending on current processing load, and therefore one has to wait, in most cases, unnecessarily long time to be sure that the settings have had effect. One the other hand, not waiting long enough may lead to that the setting adjustment is determined based on information about image data for image data not processed with the settings. This introduces an undesired variation in the image characteristics such as color, exposure, gain, etc. So, a dynamically set balance between waiting time and risk of variation must be considered, which is difficult to achieve with satisfying result over time.

To make the problem even more complex, several setting adjustments may be determined, by the controller, at the same time. The settings may be related to different components in the image pipeline. This process introduces a synchronization problem, as discussed in <CIT>, but also makes the above discussed balance even more difficult to achieve. An even longer waiting time must be considered for all settings to have had effect, making the feedback process slower.

Hence, there is a need for improvement within this field.

<CIT> discloses a pipelined video pre-processor that includes a plurality of configurable image-processing modules. The modules may be configured using direct processor control. Image data processed with an updated set of module parameters may be tagged. It can be determined when the updated set of parameters have been applied to every stage in the pipeline.

<CIT> discloses a method of processing images. A photo viewing application is connected to an image processing pipeline of a portable electronic device so that the image processing pipeline can be tuned based on user input received via the photo viewing application. As the user interacts with the application, image modification data is received indicating modification of the image to be displayed, and the photo viewing application causes the displayed image to be adjusted accordingly.

An aim of the present invention is to provide a method of determining adjustment of an image pipeline setting that can be made more time-efficient or more reliable, preferably both.

According to a first aspect, these and other objects are achieved, in full or at least in part, by a method of adjusting settings of one or more components for image data handing in an image pipeline constructed for handling an image data stream comprising image data in the form of image frames. The method is defined in claim <NUM>. The inventive concept relies on the realization of the inventors to use a tag that is associated with a setting and, when the setting has taken effect in the image pipeline, associate feedback of image data, processed with that setting, with the tag. The controller can thereby know which setting that the feedback relates to, and does not need to rely upon an estimation that the setting should have taken effect after a certain time. By the inventive concept, there is no longer a need for time estimation, handling of time count, or wait a certain time, for settings that are determined. Moreover, the adjustment method is very reliable in that the controller will with certainty know which setting that received feedback is related to.

As will be disclosed in more detail later, the controller may comprise units for implementing different functionality. For example, a control unit may be provided for determining setting adjustments, and a driver unit may be provided for transmitting determined settings to the image pipeline.

The image pipeline setting which is adjusted may be included in the one or more settings. Hence, in such an embodiment, the method may comprise determining an adjustment of the one or more settings. Alternatively, the image pipeline setting may be a setting for another component in the image pipeline than the one or more settings takes effect in. For example, the one or more settings may include a gain setting. Based on feedback concerning the light level in image frames processed with the gain setting, an exposure setting may be adjusted instead of the gain setting to, for instance, increase the light level in future image frames. Another non-limiting example is adjustment of an exposure time setting based on feedback provided by feedback from a white balance component. In yet another non-limiting example, feedback providing indication of the (color) intensity information of processed images is used for determining an adjustment for other components handling noise filtering, color correction, gain, or contrast enhancement.

The one or more settings may be transmitted to the image pipeline, i.e. to one or more components thereof, as a data bundle, optionally together with additional data which can include the tag. Alternatively, the tag can be transmitted separately, and the association can in that case be made via a logical connection.

As used herein "image pipeline" means any part of a capturing or processing device for digital imaging that have impact on the capturing/acquiring or processing of image data. The image pipeline can be divided into different components which each handles a different part of the acquiring or processing. Some components may be located in an image sensor part of a camera system, and some components may be located in a processing part of the camera system. An encoder can be seen as an additional component which is also part of the image pipeline. From the image pipeline is output, in the typical case, an image data stream in the form of a video stream with encoded image frames that represent the captured video of a scene.

As used herein "tag" means some type of identification that identifies the one or more settings for the controller. For example, the controller may have sent several data bundles, each containing one or more settings, and each being associated with a tag, wherein the tags are identifiable relative each other. The tag may be semantic, such as a label "A", "B", etc., or numerical, such as "<NUM>", "<NUM>", etc. The tag may comprise or consist of a time stamp received from a timer included in a camera system. The tag may also, or instead, be formed of a value of the one or more settings that the tag is associated with. For example, the tag may comprise an exposure time value, partly or in full, that is included in an exposure setting. Moreover, the tag must not be uniquely identifiable over time, but at least identifiable relative other entities of one or more settings that are active at the same time in the adjustment process. When a first setting is overwritten by second setting in a component of the image pipeline, the first setting can be said to be inactive.

The method may be performed iteratively. In other words, the step of determining an adjustment of the image pipeline setting may be part of the initial step of determining one or more settings in the next cycle of the method. Hence, the adjustment of the image pipeline settings is an iterative process that may adjust different settings based on feedback that is regularly sent from the image pipeline to the controller.

It is noted that transmitting feedback regarding image data is a well-known feature of an image pipeline. As will be evident from the description herein, several (all well-known) types of image data feedback exist and can be used in the present invention.

In one embodiment, the one or more settings comprises a group of settings, meaning that a plurality of settings are formed by the controller to be transmitted to the image pipeline. The group of settings may be seen as an entity of settings. The group of settings may be determined based on image data feedback received in one or more earlier cycles of the method. Preferably, the group of settings should take effect for handling (acquiring and/or processing) of (the) same image data. Since a group of settings may contain settings for different components in the image pipeline, which processes data successively, the application of the group of settings in the image pipeline must be coordinated. Moreover, different settings may require different amount of image data to take effect. In other words, one setting type may take effect over one or two image frames while another setting type may need three image frames to take effect. For these purposes, the method may further comprise determining a timing for applying the group of settings such that each of the settings takes effect for processing of (the) same image data, wherein the group of settings is transmitted to the image pipeline according to the determined timing. Moreover, the tag is transmitted in association with at least one setting in the group of settings.

In other words, a synchronized application of the group of settings is performed.

By (the) same image data is meant the same data package. The size of a data package may depend on many factors, such as how the data flow structure of the image pipeline is formed. In one embodiment, the same image data corresponds to a same image frame, which is the image data acquired by an image capturing device at a given moment (i.e. at the same time).

As implied, the tag may be transmitted in association with one setting or a sub-group of settings within the group of settings.

Hence, in one embodiment, the tag is transmitted together with each of the settings within the group of settings to different image processing components of the image pipeline. In this way, each component receiving a setting will be able to associate feedback of image data processed, using the received setting, with the tag without further actions.

Alternatively, the tag may be transmitted, in association with a first setting of the one or more settings, to an image processing component in the image pipeline. In such an embodiment, the tag may be transmitted to later image processing components via image data that is transferred between components in the image pipeline. The tag may be included in the image data in itself (i.e. via pixel values) or, e.g., as metadata. In one embodiment, the tag is transmitted with the setting that is related to a component located earlier in the image pipeline than the components related to the other settings in the group. Hence, the tag will in that case be transmitted to all the other (later) components together with the image data.

In yet another embodiment, the tag is transmitted in association with each of the settings and, in addition, is transmitted via image data to other components in the image pipeline as well. The tag can thereby be sent in association with feedback from the other components.

A further advantage with an embodiment in which the tag is transmitted via image data is that the component may know, when receiving image data including the tag, that a setting of the same group for a preceding component has taken effect. This can be a trigger, or verification of the set timing, to implement the setting of the current component.

In one embodiment, the tag includes unique data identifying the one or more settings. By unique is meant a tag that provides an identification that will not reoccur for other tags over time. An example of a unique tag is a combination of letters and numbers that is formed by any commercially available algorithm for creating unique sequences.

The feedback comprises image statistics, such as block level data or histogram data.

Non-limiting examples of image pipeline components are: gain, exposure (shutter), pre-processing (e.g. remove effects, such as dead pixels, caused by sensor defects), white balance (WB), post-processing (e.g. noise filtering, contrast enhancement, sharpening), and encoding. Other components of the image pipeline may handle demosaicing, color correction, and tone mapping.

In one embodiment, the one or more settings comprises a plurality of settings that are divided into a first group of settings and a second group of settings. The first group of settings is selected to be applied in processing of image data of a first iteration. The second group of settings is selected to be applied in processing of image data of a second iteration. The first and second group of settings may comprise settings of the same type. For example, the first group of settings may comprise a first gain setting and a first filter setting to be applied in processing of image data of the first iteration, and the second group of settings may comprise a second gain setting and a second filter setting to be applied in processing of image data of the second iteration.

The method may, in this embodiment, further comprise transmitting, in association with the feedback and the tag, information of which iteration the processed image data is of. Hence, the controller receives feedback together with the tag and which iteration the feedback is based on. By the combination of the tag and the iteration information, the controller knows which settings has been active when processing the image data of the feedback.

Alternatively, the plurality of settings may be associated with a plurality of tags, that is one for each group of settings. In that case, feedback sent from a component is associated with the tag from which the controller may know which iteration the tag relates to. A simple way to implement this way of tagging, is to assign different tag characteristics to different iterations. For example, groups of settings for a first iteration may be associated with tags in the form of odd numbers and groups of settings for a second iteration may be associated with tags in the form of even numbers.

Based on the received feedback and the one or more settings associated with the tag in combination with the iteration information, the controller may determine an adjustment of an image pipeline setting to be applied to the same iteration. Hence, adjustments of settings for one iteration, and not other iterations, based on feedback which, with certainty, is based on processed image data of that iteration, is enabled by this embodiment of the invention.

It is noted that this embodiment may be combined with a synchronized application of a group of settings, as disclosed as another embodiment above. In other words, the one or more settings may comprise one or more groups of settings and a timing for applying the settings within each of these groups of settings may be determined such that the settings within a group takes effect for same image data.

By the term iteration is meant, within the context of this application, image pipeline processing cycles that are performed iteratively and with different settings for at least one component in the pipeline. As an example, processing of different iterations may be performed alternately.

A non-limiting example of an image pipeline adapted for different iterations is one that is adapted to produce high dynamic range (HDR) images. Such images are based on a well-known technique of acquiring image data with different exposure times. The image data of different exposure times are processed in different iterations.

In an alternative embodiment, the one or more settings comprise a plurality of settings of the same type but for different iterations. For example, the plurality of settings may include a first filter setting for a first iteration and a second filter setting for a second iteration. The plurality of settings may be transmitted to a filter component of the image pipeline, in association with a tag and iteration information, in accordance to what has been described for embodiments above. The same principles and advantages of the use of a tag and indication information that have been disclosed above, applies to this embodiment as well.

According to a further aspect, there is provided a computer-readable storage medium having computer code instructions stored thereon which, when executed by a processor, causes the processor to carry out the method according to the first aspect. The storage medium may be a non-transitory storage medium.

According to a further aspect, the above and other objects are achieved, in full or at least in part, by a system for adjusting settings of one or more components for image data handling in an image pipeline constructed for handling an image data stream comprising image data in the form of image frames. The system is defined by claim <NUM>.

The system may further comprise an image sensor adapted to acquire the image data. The image sensor may be adapted for any type of imaging, including visual light imaging and thermal imaging. Alternatively, the system may further comprise a ranging sensor, for example a radar unit, that is adapted to acquire the image data.

The image pipeline may be adapted to perform image data processing selected from the group of: gain, exposure, pre-processing, white balance, filtering, post-processing, and encoding.

Hence, it is to be understood that this invention is not limited to the particular component parts of the device described or steps of the methods described as such device and method may vary. It must be noted that, as used in the specification and the appended claim, the articles "a", "an", "the", and "said" are intended to mean that there are one or more of the elements unless the context clearly dictates otherwise. Thus, for example, a reference to "an object" or "the object" may include several objects, and the like. Furthermore, the word "comprising" does not exclude other elements or steps.

The invention will now be described in more detail by way of example and with reference to the accompanying schematic drawings, in which:.

<FIG> provides an overview of an image pipeline <NUM>. The image pipeline <NUM> may be part of a camera system, such as a thermal camera system or a visual light camera system. The camera system may be a monitoring camera system. As the skilled person recognizes, an image pipeline may take various forms and is not limited to the ones exemplified herein. Hence, the present invention may be applied to various forms of image pipelines in a variety of camera applications.

The image pipeline <NUM> is constructed for handling an image data stream <NUM> comprising image data in the form of image frames <NUM>-<NUM>. Between each image frame, intermediate data <NUM> is provided as a buffer and may also include peripheral data such as metadata or supplementary image data. The handling involves image data acquiring, performed in a sensor part <NUM>, and sub-sequent image data processing, performed in a processing part <NUM>. The processing part <NUM> may be implemented as a chip. The sensor part <NUM> and the processing part <NUM> may be implemented in a structurally separated way, meaning that the sensor part <NUM> may perform its operation on a first device (not illustrated) and transmit image data to a second device (not illustrated) for processing by the processing part <NUM>.

The image pipeline <NUM> comprises components for the image data handling. These components are exemplified in <FIG> as an exposure time component <NUM>, a gain component <NUM>, a pre-processing component <NUM>, a white balance (WB) component <NUM>, and a filter component <NUM>. The names indicate their respective mechanism, namely control of exposure time (<NUM>), applying gain (<NUM>), adjusting white balance (<NUM>), and applying filter (<NUM>). The mechanism in each component is controlled by a setting, or a set of settings, that is stored in each component. A storage location for one or more settings of a component is sometimes referred to as a register.

In an image pipeline <NUM>, it is desirable to be able to adjust these settings to adapt the image data to different parameters, such as scene properties (e.g. amount of available light, motion, colors), limitations (or desired decrease) of data transmission, or user-initiated requests of change in image data characteristics. For this purpose, a controller <NUM> is provided, including a control unit <NUM> and a driver <NUM>. The control unit <NUM> and the driver <NUM> are here illustrated as separate devices for illustrative purposes, however it is appreciated that these may be implemented together in a single device. The controller <NUM> may comprise circuitry which is configured to implement the control unit <NUM> and the driver <NUM> and, more specifically, their functionality.

In a hardware implementation, each of the control unit <NUM> and the driver <NUM>, may correspond to circuitry which is dedicated and specifically designed to provide functionality of the parts. The circuitry may be in the form of one or more integrated circuits, such as one or more application specific integrated circuits or one or more field-programmable gate arrays.

In a software implementation, the circuitry may instead be in the form of a processor, such as a microprocessor, which in association with computer code instructions stored on a (non-transitory) computer-readable medium, such as a non-volatile memory, causes the controller <NUM> to carry out (part of) any method disclosed herein. Examples of non-volatile memory include read-only memory, flash memory, ferroelectric RAM, magnetic computer storage devices, optical discs, and the like. In a software case, the control unit <NUM> and the driver <NUM> may thus each correspond to a portion of computer code instructions stored on the computer-readable medium, that, when executed by the processor, causes the controller <NUM> to carry out the functionality of the component.

As indicated in <FIG>, a bidirectional stream for data exchange is provided between the image pipeline <NUM> and the controller <NUM>. From the image pipeline <NUM>, feedback is sent to the control unit <NUM>, and from the driver <NUM>, updated settings for different components are sent to the image pipeline <NUM>. The feedback consists of image statistics that can be seen as data providing a condensed representation of processed image data at different stages (at different components) in the image pipeline <NUM>. Such feedback is typically provided by the processing part <NUM> of the image pipeline <NUM>. However, in some cases, also the sensor part <NUM> of the image pipeline <NUM> may provide feedback to the controller <NUM>. Based on the feedback, the control unit <NUM> decides on adjustment of one or more settings which are sent to the driver <NUM> for transmittal to the image pipeline <NUM>.

Going more into detail, <FIG> illustrates a first embodiment of the invention. The figures illustrate the image pipeline <NUM> at different (consecutive) moments when the image data stream <NUM> is passed through. To emphasize the principles of the invention, only one component, namely the gain component <NUM>, is illustrated and the other components are left out from these figures. The embodiment is not limited to the gain component <NUM>, and may be adapted to work for other components in the image pipeline <NUM> as well. Moreover, the arrows of the figures illustrate in which directions data may be transmitted, and does not necessarily mean data is currently transmitted.

Starting in <FIG>, the image frames <NUM> and <NUM> have passed through the gain component <NUM> and the image frame <NUM> is under processing. The current gain setting <NUM> is G1, meaning that the gain component <NUM> processes the image data stream <NUM> with the setting G1 which, e.g., indicates the amount of gain that is applied to pixel values of the image data in an image pipeline. The image pipeline <NUM> transmit feedback (FDBK) <NUM> to the control unit <NUM>. Here the feedback <NUM> is illustrated as coming from the gain component <NUM>. However, it is understood, that the feedback may generally come from any component(s) in the image pipeline <NUM>. Specifically, a setting may be adjusted based on feedback from the component in the image pipeline <NUM> to which the setting pertains or from any subsequent component in the image pipeline <NUM>. In this case, the feedback may hence come from any component which is subsequent to the gain component <NUM> in the image pipeline <NUM>. The feedback <NUM> includes image statistics, for example block statistics in the form of pixel block average/variance/median values or histogram data. A non-limiting example of block statistics is obtaining a color value for each 16x16 pixel block of the image data. This can be seen as a reduced/condensed representation of the image data. A non-limiting example of histogram data is information on the number of pixels per, e.g., light intensity level. Various types of well-known image statistics, also of other types than block statistics and histogram data, that are suitable for various purposes exist and is readily available for the skilled person.

The control unit <NUM> has, based on earlier received feedback, determined an adjustment of the gain setting from G1 to G2. The purpose of the adjustment may be to change image characteristics, such as noise or light level, of image data that is processed by the gain setting. Another purpose may be that more or less gain is required for adapting the bitrate of a resulting data stream of image data output from the image pipeline. The adjustment may be determined based on user-input or by other parts of the image system, such as by an encoder. These purposes are applicable to the inventive concept in general. It is also noted that the adjustment, in general, is a well-known image pipeline process. However, the addition of a tag, according to the inventive concept, improves known ways and methods of how to adjust settings of the image pipeline and adds further benefits such as enabling a faster adjustment process.

The adjustment may be performed based on target image characteristics which are affected by the setting. The control unit <NUM> may compare the received feedback <NUM> to target image characteristics and adjust the settings on basis thereof. For example, if the received feedback <NUM> indicates that image data of a processed image frame is darker than a target value, the control unit <NUM> may increase the gain from G1 to G2 to increase the light level in future image frames. In that way, the control unit <NUM> may adjust the settings to cause the image characteristics to reach their target values. A target value for the settings itself may be set, wherein the control unit <NUM> aims to reach the target value when adjusting the setting. The target image characteristics and/or the target value for a setting may be set based on user-input or by other parts of the image system, such as by an encoder, for achieving certain characteristics of the image data.

The adjusted gain setting <NUM> (being G2) is sent from the control unit <NUM> to the driver <NUM> for implementation in the image pipeline <NUM>. The adjusted gain setting <NUM> is associated with a tag in the form of an ID number (being "<NUM>" for the case of the adjusted gain setting <NUM>). The association may be implemented by that the tag is sent together with the adjusted gain setting <NUM>, i.e. in the same data package. Moving forward to <FIG>, the driver <NUM> has sent the adjusted gain setting <NUM> to the gain component <NUM> which implements it and uses it for processing of the image frame <NUM>. The ID is transmitted in association with the adjusted gain setting <NUM> to the gain component <NUM>. Moreover, with the adjusted gain setting <NUM> active for processing of image data, second feedback <NUM> is sent from the gain component <NUM> to the control unit <NUM>. This feedback <NUM> is based on the image frame <NUM>, not the previously processed image frames <NUM>, <NUM>. In association with the second feedback <NUM>, the ID is sent to the control unit <NUM>. When receiving the second feedback <NUM>, with the associated ID, the control unit <NUM> knows, from the association between the ID and the setting G2 that the second feedback <NUM> is based on image data processed with the setting G2. The association between different tags (in this case IDs) and settings may be stored in a memory of the control unit <NUM> or in a memory remotely accessible to the control unit <NUM>.

With this information at hand, i.e. the second feedback <NUM> and the knowledge of which gain setting is used for processing of the image data on which the second feedback <NUM> is based, the control unit <NUM> may now make a well-founded determination for the next adjustment of gain setting that is to be applied, namely an adjustment from G2 to G3 as indicated by gain setting <NUM>. Before transmitting this gain setting <NUM> to the driver <NUM> for implementation, the gain setting <NUM> is associated with a tag, in this case being an ID number "<NUM>". A unique tag (ID number in this case) is chosen, at least relative all settings that are active at the moment. In this example, only the gain setting G2 with ID number "<NUM>" is active so any other ID number than "<NUM>" would suffice. However, in a real implementation, hundreds of different settings (for different components) may be active. Preferably, and when also keeping historical information about settings that have been applied (together with their tags), the tag should also be uniquely selected with respect to inactive, but previously implemented, settings.

Moving forward to <FIG>, the further gain setting <NUM>, including G3 and associated with ID number "<NUM>", is sent to the driver <NUM>. The control unit <NUM> receives third feedback <NUM> from the gain component <NUM>. At this moment, the control unit <NUM> would, without the tag, not know if the third feedback <NUM> is based on image data processed with the setting G2 or G3. However, with the tag that is associated with the third feedback <NUM>, namely ID number "<NUM>", the control unit <NUM> knows that the third feedback <NUM> still relates to image data processed with the gain setting G2. In this way, the method provides a time-efficient method for adjustment where the control unit <NUM> does not need to estimate how much time it takes for a setting to have effect, which is complex in itself, and handle different timers for all different settings that are active. Moreover, the control unit <NUM> may, by receiving feedback with a tag having the ID number "<NUM>", know that the previous setting G2 has been overwritten in the gain component <NUM> and hence the control unit <NUM> may set that settings as inactive in the memory of settings. The tag of an inactive setting may be reused to save processing power and memory.

Moving forward, <FIG> illustrate a second embodiment of the present invention. Here, the inventive concept as explained with the example of the first embodiment will now be applied to a more complex scenario. However, the main principles of the inventive concept still apply, that is to tag setting(s) that are implemented in an image pipeline and correspondingly tag feedback determined for image data processed with that settings.

Starting in <FIG>, the image pipeline <NUM> is exemplified with components exposure time component <NUM>, gain component <NUM>, white balance component <NUM>, and filter component <NUM>. Through these components the image data stream <NUM>, with image frames <NUM>-<NUM> flows. At the moment in time which <FIG> illustrates, the different components handle the image data stream <NUM> using exposure time setting <NUM> E1, gain setting <NUM> G1, white balance setting <NUM> W1, and filter balance setting <NUM> F1. In this context, it may be emphasized that the settings do not necessarily correspond to numerical values. In the case of a filter, properties of a certain filter or a certain filter type to apply in the filter component <NUM>, may be included in the filter setting.

All settings <NUM>-<NUM> are associated with a tag in the form of an ID number "<NUM>" and these associations are made as disclosed for the first embodiment. Consequently, feedback <NUM> provided by the image pipeline <NUM>, in the illustrated case from the filter component <NUM>, is associated with the ID number "<NUM>", letting the control unit <NUM> know that the feedback <NUM> relates to image data that is processed with settings associated with that ID number.

A group of settings <NUM>, including an exposure time setting, a gain setting, a white balance setting, and a filter setting, has been determined by the control unit <NUM> and is sent to the driver <NUM>. Moving forward to <FIG>, the driver <NUM> has received the group of settings <NUM>. The driver <NUM> now has an additional functionality, when it comes to a group of settings that is to be implemented in different components in the image pipeline <NUM>, namely synchronizing the implementation of the group of settings such that the settings takes effect for processing of the same image data, preferably the same image frame. In this way, the resulting image stream will change with respect to corresponding characteristics all at once and not one at a time, which can be beneficial both for analytic purposes but also for providing an image stream that is more pleasant to look at for a user. In order to achieve a synchronization, the driver <NUM> determines a timing for applying the group of settings <NUM>. How to decide this timing is straight-forward when knowing parameters such as how the components are ordered and how long it takes for each component to apply a setting. The timing may also be decided based on dynamic parameters, such as current latency, in the image pipeline.

In <FIG>, exposure time setting <NUM> has been applied, meaning that the driver <NUM> has transmitted the setting E2 to the exposure time component <NUM>. Moving forward to <FIG>, gain setting <NUM> has been implemented in a corresponding manner, and takes effect for processing of the image frame <NUM>. As also illustrated in this figure, feedback <NUM> is sent from the gain component <NUM> to the control unit <NUM> with an associated tag of ID number "<NUM>". The control unit <NUM> has started to form a second group of settings <NUM> that is to be filled with further settings.

Next, in <FIG>, a white balance setting <NUM> has been implemented in the white balance component <NUM>. Moreover, third feedback <NUM> is sent from the filter component <NUM> to the control unit <NUM>, and fourth feedback <NUM> is sent from the exposure time component <NUM>. The third feedback <NUM> is associated with ID number "<NUM>" and the fourth feedback is associated with ID number "<NUM>". Accordingly, the control unit <NUM> may separate these two feedbacks as being based on image data being processed with different settings. The control unit <NUM> also knows which settings that have been used, which can be utilized in the analysis. Without the tag of the inventive concept, the separation would not have been possible and the control unit <NUM> would possibly need to discard both feedbacks <NUM>, <NUM> since it does not know what they are based on. The control unit <NUM> is hence provided with a larger amount of, and more reliable, feedback information. In addition, as illustrated in <FIG>, the control unit <NUM> receives fifth feedback <NUM> from the image pipeline in the form of image data that is fully or partly processed. Hence, this is not a condensed form of the image data but could for example correspond to an image frame. The fifth feedback <NUM> is associated with a tag in the form of an ID number "<NUM>" indicating that it has been processed using settings associated with that tag. In some embodiments, it is beneficial for the control unit <NUM> to regularly receive feedback in the form of image data to perform certain analyses.

As can also be noticed in <FIG>, the control unit <NUM> continues to build the second group of settings <NUM>.

Going forward some image frames, <FIG> illustrates a moment where all settings in the group of settings <NUM> have been implemented in the image pipeline <NUM>. At this moment, the control unit <NUM> has finished its determination of the second group of settings <NUM> and sends it to the driver <NUM> for implementation.

The inventive concept may be extended to even more complex scenarios. <FIG> illustrates a scenario where the image pipeline <NUM> is adapted to process image data of two different exposure times, handled in a first iteration <NUM> and a second iteration <NUM>. By the term iterations is meant pipeline processing cycles that are performed iteratively and with different settings for at least one component in the pipeline. In the example of <FIG>, two iterations are performed with different settings for the exposure time component <NUM> and for the gain component <NUM>. The exemplified image pipeline is adapted to produce high dynamic range (HDR) images (also known as wide dynamic range, WDR, images) based on pairs of image data frames that is acquired with different exposure times (in different iterations of the image processing) within the pair. This is a well-known technique for producing HDR images and can be extended to include a base with a larger number of image data frames acquired with different exposure times. The image data frames are handled by the image pipeline <NUM> successively in pairs, meaning that the iterations are performed alternately. Consequently, consecutive image frames, e.g. image frames <NUM>-<NUM> and <NUM>-<NUM>, in the image data stream <NUM> forms an image pair that are to form a common HDR image. The consecutive image frames are acquired with different exposure times. The exposure time is handled by the exposure time component <NUM>.

For this type of application, where image data is processed in iterations, a plurality of settings <NUM> divided into a first group of settings (E2<NUM> + G2<NUM>), for exposure time and gain settings for iteration <NUM>, and a second group of settings (E2<NUM> + G2<NUM>), for exposure time and gain settings for iteration <NUM>, are formed by the control unit <NUM>. By handling the group of settings as sub-groups of settings forming a common group of settings <NUM>, it may be avoided that a setting for a component takes effect between processing of images of the same pair. This advantage is based on a functionality that a component finishes the processing of a current setting before applying a next setting package. In other words, a component in the image pipeline <NUM> will finish processing a complete cycle of iterations, for example both iteration <NUM> and <NUM>, with a current group of settings before applying a next group of settings, even if the next group is received in the middle of the iteration cycle. Hence, the component will not change settings between iteration <NUM> and <NUM> for the example given above.

The plurality of settings <NUM> is associated with a tag in the form of an ID number "<NUM>", for reasons explained in connection to previously disclosed embodiments. Moreover, when providing feedback from the image pipeline <NUM>, e.g. from the gain component <NUM>, regarding image data processed with a setting, information of which iteration that image data is of is also sent in association. To exemplify, the gain component <NUM> in <FIG> may send feedback to the control unit <NUM> including the ID number "<NUM>" and iteration number "<NUM>", indicating that the feedback relates to image data (e.g. image frame <NUM>-<NUM> or image frame <NUM>-<NUM>) of iteration <NUM> processed with settings tagged with <NUM>. The control unit <NUM> receives the feedback together with the tag and the iteration information, and then knows which settings that have been active at the processing. Based on this information, the control unit <NUM> may determine an adjustment of settings (the same or other type of settings) for the same iteration and include with settings of other iterations that may be determined based on other feedback. In the illustrated example, the control unit <NUM> has prepared an adjusted plurality of settings <NUM>. The control unit has adjusted the exposure and gain setting for iteration "<NUM>" based on feedback tagged with the ID number "<NUM>" and iteration number "<NUM>". The control unit has further adjusted the exposure and gain setting for iteration "<NUM>" based on feedback tagged with ID "<NUM>" and iteration number "<NUM>". In other words, the control unit <NUM> is provided with reliable information of the origin of the feedback, also in this complex scenario, and can therefore make a detailed adjustment for settings of a specific iteration rather than making a general adjustment which has effect on all iterations.

It is understood that the above example extends to the case where more than two different exposures are used to form an HDR image. In that case, the plurality of settings may include more than two groups of settings which are used alternately in more than two iterations.

Another more complex scenario, for which the inventive concept also works well, is when having a multi-channel image pipeline processing images from different image sensors. In one example of a multi-channel image pipeline, image data from different sensors, forming different image channels, are processed alternately, similar to the example of different iterations of image data processing given above.

A corresponding embodiment of the inventive method, as in the case with the forming of HDR images, may be applied here. Namely, the one or more settings may comprise a plurality of settings divided into different groups for being applied in processing of image data of different image data channels. A first group may be applied when processing a first image data channel, a second group may be applied when processing a second image data channel, etc. When providing feedback from the image pipeline <NUM>, information of which channel processed image data is of may be associated with feedback of that processed image data and with the tag of plurality of settings.

As evident from the variety of examples provided herein, it is appreciated that a person skilled in the art can modify the above-described embodiments in many ways and still use the advantages of the invention as shown in the embodiments above. For example, feedback may include several tags corresponding to different settings or groups of settings that are sent to different components of the image pipeline. In other words, feedback may include several tags of different group of settings which have been active in parallel in different parts of the image pipeline when processing the same frame. Moreover, the inventive concept may be adapted to be applied for parallel image pipelines, which process image data in parallel, with a common controller determining pluralities of settings for several pipelines. Feedback may in such a case be associated with a tag and, in addition, an indication of which image pipeline the feedback relates to.

Claim 1:
A method of adjusting settings of one or more components of an image pipeline for image data stream handling, said image data stream comprising image data in the form of image frames, the method comprising
determining, by a controller (<NUM>), one or more settings of one or more components in an image pipeline (<NUM>) performing image data processing,
associating, by the controller (<NUM>), the one or more settings with a tag,
transmitting the one or more settings to the one or more components in the image pipeline (<NUM>), wherein the tag is transmitted in association with the one or more settings,
processing image data by the one or more components implementing the one or more settings,
providing feedback from any component in the image pipeline (<NUM>) regarding image data processed by the one or more components implementing the one or more settings, wherein the feedback consists of image statistics as data providing a condensed representation of the processed image data,
transmitting, from the image pipeline (<NUM>) to the controller (<NUM>), the feedback, wherein the tag is transmitted in association with the feedback, and
determining, based on the received feedback and the one or more settings associated with the tag, the updated settings of the one or more image pipeline components transmitting said updated settings to the one or more components in the image pipeline (<NUM>).