Patent Description:
Typically, a video stream is produced by capturing a series of images using an image sensor. Alternatively, the series of images may be rendered by an image rendering engine.

As an image from the series of images is transferred into a video image processing pipeline a plurality of processes will start working with it in order for producing a video frame. As a rule, the processes are finalized before a subsequent image, in the series of images, is transferred to the video image processing pipeline. The so produced video frames are forming a video stream.

This will result in a period of idle video image processing pipeline followed by that the video image processing pipeline is running at full capacity. This will result in uneven power consumption. Hence, the power consumption of the video image processing pipeline varies during production of one video frame. An example of the variation of power consumption of the video image processing pipeline is illustrated in <FIG>. As seen, the characteristics of the power consumption of the video image processing pipeline is similar to a sinus wave with the same frequency as the frame rate. In <FIG> the time span of a video frame is indicated as FT, i.e. frame time. The variation in power consumption of the video image processing pipeline is due to the different power consumption of individual processes of the video image processing pipeline. The power consumption of four different individual processes of a video image processing pipeline building up the power consumption illustrated in <FIG> is illustrated in the <FIG> within the oval insert in the top of the figure as consumption diagrams a)-d).

Varying power consumption of the video image processing pipeline may have different drawbacks. For example, a powering unit powering the video image processing pipeline must be dimensioned according to the peak power consumption. Hence, there is a need for evening out the power consumption of the video image processing pipeline.

In <CIT> examples are disclosed for adjusting a performance state of a graphics subsystem and/or a processor based on a comparison of an average frame rate to a target frame rate. The adjustment of the performance state is made in order to reduce the power consumption in general of the graphics subsystem.

<NPL>, discloses controlling the peak power of a Network-on-Chip based multimode system by controlling the communication bandwidth between components of the system and the memory. The bandwidth allocation is reduced by scaling down the number of virtual channels that are allocated to the communications.

<NPL>, discloses a credit-based peak power control scheme for a Network-on-Chip. The disclosed scheme regulates each flow's injection rate at the sender to minimize performance penalty with two different throttling schemes for real-time traffic and best-effort traffic: a rate-based throttling and an energy-budget based throttling, respectively.

According to a first aspect a video image processing pipeline controller is provided according to claim <NUM>.

Accordingly, the bandwidth available for communication with the system memory is tuned. Some or all of the processing functions of the video image processing pipeline communicating with the system memory will be choked. This will result in that the timing of these processing functions will shift or be prolonged. By, in time, shifting and/or prolonging the active time, of some or all of the processing functions of the video image processing pipeline, the power consumption may be held on a stable level. Hence, the peak power consumption may be lowered. If the bandwidth is reduced too much, the video processing pipeline will not be able to produce the video stream with the target frame rate. Hence, the reduction of bandwidth need to be controlled such that the current frame rate does not drop below the target frame rate. In other word, the controller is configured to reduce the bandwidth as much as possible while maintaining a constant frame rate. By controlling the bandwidth of the one or more memory access channels just right, the above can be achieved. The controlling algorithm uses two variables, the target frame rate and the current frame rate at which the video processing pipeline is producing the video stream. The controlling of the bandwidth in order to reduce the peak power consumption of the video image processing pipeline may run continuously in the background.

By reducing the peak power consumption of the video image processing pipeline the powering unit powering the video image processing pipeline may be dimensioned to deliver less power. For the example of using power over Ethernet, PoE, to power a camera comprising the video image processing pipeline, the PoE installation may be dimensioned to deliver less power than without the present reduction of peak power. Hence, if peak power decreases, the power needed to be delivered by a PoE aggregate may also decrease. Alternatively, the PoE aggregate may be used for powering more cameras.

Instead of dimensioning down the powering unit, the excess power may be used by other processes in a device comprising the video image processing pipeline.

Increasing the available bandwidth to a bandwidth at which the current frame rate does not drop below the target frame rate may comprise increasing the available bandwidth to a smallest previous bandwidth at which the current frame rate did not drop below the target frame rate.

The target frame rate may be an original frame rate. The original frame rate may be read out from the video source producing the video image data for the video image pipeline. The original frame rate may be read out before regulation of the bandwidth.

The video image processing pipeline controller may further be configured to globally control the bandwidth of a plurality of memory access channels.

The video image processing pipeline controller may further be configured to individually control the bandwidth in each of a plurality of memory access channels. In order to achieve as good and even control of the power consumption of the video image pipeline as possible, it may be advantageous to control different memory access channels differently. This may possibly be made using an actual measured or estimated power consumption as feedback.

According to a second aspect a video image processing system is provided. The video image processing system comprises: a video source configured to provide video image data; a video image processing pipeline comprising a plurality of processing functions, wherein each processing function is configured to process the video image data; a system memory, wherein the processing functions of the image processing pipeline are configured to access the system memory via one or more memory access channels; and a video image processing pipeline controller according to the first aspect
The above mentioned features of the video image processing pipeline controller, when applicable, apply to this second aspect as well. In order to avoid undue repetition, reference is made to the above.

The video image processing pipeline may comprise two or more of the following processing functions: image sensor correction function, noise reduction function, image scaling function, gamma correction function, image enhancement function, color space conversion function, chroma subsampling function, compression function, data storage function and data transmission function.

A processing function of the video image processing pipeline may be implemented as a computer software portion run on a general purpose processor or on a graphics processing unit, a field-programmable gate array, a fixed-function application-specific integrated circuit, or an analog circuit.

The image source may be an image sensor. The image source may be an image rendering engine.

The video image processing system may further comprise: an electrical power consuming unit; and an electrical power managing unit configured to monitor electrical power saved by the controlling of the available bandwidth at which the image processing pipeline is allowed to communicate with the system memory, and to distribute at least a fraction of the saved electrical power to the electrical power consuming unit.

The image processing system may be implemented in a camera. The camera may be a monitoring camera.

According to a third aspect a method of reducing a peak power consumption in a video image processing pipeline is provided according to claim <NUM>.

Increasing the available bandwidth to a bandwidth at which the current frame rate does not drop below the target frame rate may comprise increasing the bandwidth to a smallest previous available bandwidth at which the current frame rate did not drop below the target frame rate.

The above mentioned features of the controller and the system, when applicable, apply to this third aspect as well. In order to avoid undue repetition, reference is made to the above.

According to a fourth aspect a non-transitory computer readable recording medium having recorded thereon program code which when executed at a device having processing capabilities is configured to perform the method according to the third aspect.

The above and other aspects of the present invention will now be described in more detail, with reference to appended drawings showing embodiments of the invention. The figures should not be considered limiting the invention to the specific embodiment; instead they are used for explaining and understanding the invention.

As illustrated in the figures, the sizes of layers and regions may be exaggerated for illustrative purposes and, thus, are provided to illustrate the general structures of embodiments of the present invention.

<FIG> illustrates a video image processing system <NUM>. The video image processing system <NUM> comprises a video source <NUM>, a video image processing pipeline <NUM>, a system memory <NUM>, one or more memory access channels <NUM>, and a video image processing pipeline controller <NUM>.

The video source <NUM> is configured to provide video image data. The video image data may be a series of images. The video source <NUM> may be an image sensor. The image sensor is configured to capture the video image data. The image sensor may form part of a video camera. Alternatively, or in combination, the video source <NUM> may be an image rendering engine. The image rendering engine is configured to render the video image data. The image rendering engine may be configured to render photorealistic or non-photorealistic images from a 2D or 3D model by means of a computer program. The video source <NUM> is configured to transfer the video image data to the video image processing pipeline <NUM>.

The video image processing pipeline <NUM> is configured to process the video image data into video frames. The video image processing pipeline <NUM> comprises a plurality of processing functions <NUM>. Each processing function <NUM> is configured to process the video image data. Some of the plurality of processing functions <NUM> may be dependent on each other. Hence, they need to be executed one after another. Some of the plurality of processing functions <NUM> may be independent on each other. Hence, they may be executed in parallel. The processes of the plurality of processing functions <NUM> are typically finalized well before a subsequent image, in the series of images, is transferred to the video image processing pipeline <NUM>.

The video image processing pipeline <NUM> comprises two or more of the following processing functions <NUM>: an image sensor correction function, a noise reduction function, an image scaling function, a gamma correction function, an image enhancement function, a color space conversion function, a chroma subsampling function, a compression function, a data storage function, and a data transmission function. The image sensor correction function may comprise a Bayer filter. The color space conversion function may comprise conversions between different formats, such as RGB, YUV and YCbCr.

A specific processing function <NUM> of the video image processing pipeline <NUM> may be implemented as a computer software portion run on a general purpose processor or on a graphics processing unit, a field-programmable gate array, a fixed-function application-specific integrated circuit, or an analog circuit. Every one of the plurality of processing function <NUM> may be implemented using the same type of implementation. Different ones of the plurality of processing function <NUM> may be implemented using different implementations of the processing function <NUM>. A subset of the plurality of processing function <NUM> may be implemented using the same type of implementation. Accordingly, the processing function <NUM> of the video image processing pipeline <NUM> may be implemented as software, dedicated hardware or firmware, or some combination of dedicated hardware, firmware and/or software.

The system memory <NUM> is a memory used by the processing function <NUM> while processing the video image data into video frames. The system memory <NUM> may be a volatile memory, such as a random-access memory, RAM.

The processing functions <NUM> of the image processing pipeline <NUM> are configured to access the system memory <NUM> via the one or more memory access channels <NUM>. The one or more memory access channels may be direct memory access, DMA, channels. The processing functions <NUM> of the image processing pipeline <NUM> may be configured to access the system memory <NUM> via a single memory access channel <NUM>. The processing functions <NUM> of the image processing pipeline <NUM> may be configured to access the system memory <NUM> via a plurality of memory access channels <NUM>.

The video image processing pipeline controller <NUM> is configured to control a bandwidth of the one or more memory access channels <NUM>. In case of a plurality of memory access channels <NUM>, the video image processing pipeline controller <NUM> may be configured to control the bandwidth of each memory access channel <NUM> individually. Hence, the bandwidth in each memory access channel <NUM> may be controlled independent from the bandwidths in the other memory access channels <NUM>. In order to achieve as good and even control of the power consumption of the video image pipeline <NUM> as possible, it may be advantageous to control different memory access channels differently. This may possibly be made using an actual measured or estimated power consumption as feedback. The actual power consumption may be given by an electrical power managing unit <NUM>. Alternatively, in case of a plurality of memory access channels <NUM>, the video image processing pipeline controller <NUM> may be configured to control the bandwidth of the memory access channels <NUM> globally. Hence, the bandwidths of the plurality of memory access channels may be controlled together controlling a total bandwidth of the one or more memory access channels <NUM>. Hence, the video image processing pipeline controller <NUM> is configured to control the bandwidth at which processing functions <NUM> of the video image processing pipeline <NUM> communicates, over the one or more memory access channels <NUM>, with the system memory <NUM>. The video image processing pipeline controller <NUM> is configured to control the bandwidth such that the bandwidth is reduced. The controlling of the bandwidth is made based on a current frame rate at which the video image processing pipeline <NUM> produces the video stream and a target frame rate of the video image processing pipeline <NUM>. The target frame rate may be stored in a memory <NUM> of the video image processing system <NUM>. The target frame rate may be an original frame rate of the video stream. The original frame rate may be the frame rate at which the video source is producing the series of images. The reduction of the bandwidth is controlled such that it is secured that the current frame rate does not drop below the target frame rate. Hence, the bandwidth may be controlled based on the current frame rate with the original frame rate as target.

By controlling the bandwidth of the one or more memory access channels <NUM> in accordance with this scheme a reduction of a peak power consumption of the video image processing pipeline <NUM> may be achieved. This since a reduction in the bandwidth induces a shift in and/or prolongs the active time, within a frame time, of the processing functions <NUM>. By shifting and/or prolonging the active time the power consumption can be held on a stable level. Hence, the peak power consumption may be lowered. This is schematically illustrated in connection with <FIG>. Within the oval insert in the top of <FIG> the power consumption of the four different individual processes of the video image processing pipeline <NUM> building up the power consumption as illustrated in <FIG> are schematically illustrated after a reduction of the bandwidth according to the above scheme. The reduction of the bandwidth may result in that an execution of a specific processing function <NUM> is spread out in time. This may imply that the power consumption for that specific processing function <NUM> is also spread out in time. This is illustrated in the oval insert in the top of <FIG>, wherein the power consumption of the different processes a)-d) related to the different processing functions <NUM> are spread out in time as compared with a video image processing system run without reducing the bandwidth, the latter is schematically illustrated in <FIG>. Alternatively, or in combination, the reduction of the bandwidth may result in that an execution of a specific processing function <NUM> is shifted in time. This is also illustrated in the oval insert in the top of <FIG>, wherein the power consumption of the processes a) and b) are shifted in time as compared with a video image processing system run without reducing the bandwidth, the latter is schematically illustrated in <FIG>.

As mentioned above, some of the plurality of processing functions <NUM> may be dependent on each other. In the example illustrated in <FIG> processes a) and c) are depended on each other and processes b) and d) are dependent on each other. Hence, they are executed one after another. Further, also as mentioned above, some of the plurality of processing functions <NUM> may be independent on each other. For example, in the example illustrated in <FIG> process a) is independent on processes b) and d). Hence, process a) may be executed in parallel with processes b) and/or d).

The video image processing pipeline controller <NUM> may be configured to tune the bandwidth, at which the processing functions <NUM> of the image processing pipeline <NUM> communicates, over the one or more memory access channels <NUM>, with the system memory <NUM>, until the peak power is minimized.

The video image processing pipeline controller <NUM> may further be configured to decrease the bandwidth, at which the processing functions <NUM> of the image processing pipeline <NUM> communicates, over the one or more memory access channels <NUM>, with the system memory <NUM>, in steps. The bandwidth may be reduced in steps until the current frame rate drops below the target frame rate. In response to the current frame rate drops below the target frame rate, the video image processing pipeline controller <NUM> may be configured to increase the bandwidth to a bandwidth at which the current frame rate does not drop below the target frame rate. This may, for example, be made by increasing the bandwidth to a smallest previous bandwidth at which the current frame rate did not drop below the target frame rate.

The video image processing system <NUM> may further comprise an electrical power consuming unit <NUM> and an electrical power managing unit <NUM>.

The electrical power consuming unit <NUM> may be one or more of an illuminator device configured to illuminate a scene viewed by the video source <NUM>, a pan/tilt motor configured to pan/tilt the video source <NUM>, and a cooling arrangement configured to cool one or more components of the system <NUM>.

The electrical power managing unit <NUM> is configured to monitor the electrical power saved by the controlling of the bandwidth at which the image processing pipeline <NUM> communicates, over the one or more memory access channels <NUM>, with the system memory <NUM>. The electrical power managing unit <NUM> may further be configured to distribute at least a fraction of the saved electrical power to the electrical power consuming unit <NUM>.

In connection with <FIG> a method of reducing a peak power consumption in the video image processing pipeline <NUM> will be discussed. The method comprises the following acts. Storing S402 the target frame rate of the video image processing pipeline <NUM>. The target frame rate may be stored in the memory <NUM> of the video image processing system <NUM>. Reducing S404, based on a current frame rate at which the video image processing pipeline <NUM> produces the video stream and the target frame rate, a bandwidth, at which processing functions <NUM> communicates over the one or more memory access channels <NUM> with the system memory <NUM>. The reducing S404 is performed such that it is secured that the current frame rate does not drop below the target frame rate. The act of reducing S404 may be performed by a video image processing pipeline controller <NUM>. The act of reducing S404 the bandwidth may comprise decreasing the bandwidth in steps until the current frame rate drops below the target frame rate, and, in response to the current frame rate drops below the target frame rate, increasing the bandwidth to a bandwidth at which the current frame rate does not drop below the target frame rate. The act of reducing S404 the bandwidth may further comprise, in response to the current frame rate drops below the target frame rate, increasing the bandwidth to a smallest previous bandwidth at which the current frame rate did not drop below the target frame rate.

For example, the video image processing system may be implemented in a camera <NUM>. This is illustrated in connection with <FIG>. The camera <NUM> may be a monitoring camera. In case the system is implemented in the camera <NUM>, the electrical power consuming unit <NUM> may be one or more of an illuminator device configured to illuminate a scene viewed by the camera <NUM>, a pan/tilt motor configured to pan/tilt the camera <NUM>, and a cooling arrangement configured to cool the camera <NUM>.

Claim 1:
A video image processing pipeline controller configured to control an available bandwidth at which processing functions (<NUM>) of a video image processing pipeline (<NUM>) is allowed to communicate, over one or more memory access channels (<NUM>), with a system memory (<NUM>),
wherein the video image processing pipeline controller is configured to reduce the available bandwidth while securing that the current frame rate does not drop below a target frame rate, at which the video image processing pipeline (<NUM>) produces a video stream, thereby inducing a shift in an active time and/or prolonging the active time, within a frame time, of the processing functions (<NUM>) such that a peak power consumption of the video image processing pipeline (<NUM>) is reduced,
wherein the video image processing pipeline controller is configured to reduce the available bandwidth while securing that the current frame rate does not drop below a target frame rate by decreasing the available bandwidth in steps until a current frame rate drops below the target frame rate, and, in response to the current frame rate drops below the target frame rate, increasing the available bandwidth to a bandwidth at which the current frame rate does not drop below the target frame rate.