Patent Description:
Surveillance of the public using imaging, in particular video imaging, is common in many areas around the world. Areas that may need monitoring are for example banks, stores, and other areas where security is needed, such as schools and government facilities. Other areas that may need monitoring are processing, manufacturing and logistics applications where video surveillance is primarily used to monitor processes.

Captured video images may be complemented with other image data also referred to as overlay objects, such as graphics. The graphics may for example be, an icon, a diagram, different masking graphics such as boxes or circles, or text.

A variety of image processing methods have been proposed in which an image signal acquired by taking images via a video camera etc., is superimposed with other image data such as characters and graphics. In an example, a video post processor, which may be referred to as an image compositor, overlays a graphics overlay onto video frames captured by the camera. In order to do so, the image compositor loads, or in other words reads, a representation of the graphics overlay and adds it to a video frame. The representation is usually a raster format representation of the graphics overlay. The raster format is also known as bitmap format. The raster format represents an image by pixels arranged in a two-dimensional grid, where each pixel includes one or more bits to represent the pixel value. For example, the ARGB32 format has <NUM> bytes per pixel, while the A8 format has <NUM> byte per pixel. Thus, the raster format is an example of an uncompressed format. An advantage of the raster format is that it can be rendered efficiently and fast by, e.g., a graphical processing unit (GPU). However, when the graphics overlay is represented in a raster format, the loading of the graphics overlay for each video frame represents a considerable cost in terms of memory bandwidth.

An object of embodiments herein may thus be to obviate some of the problems mentioned above, or at least reduce the impact of them. Specifically, an object of embodiments herein may be how to reduce the memory bandwidth in connection to overlaying an overlay, such as a graphics overlay, onto a sequence of video frames.

Embodiments herein solve the above problems by controlling the representation format of the overlay based on whether or not the content of the overlay has been updated within a time period, for example within a time period which corresponds to at least the last two video frames.

Although embodiments have been summarized below, the invention is defined by the appended claims.

The inventors have realized that overlays, such as graphic overlays, may also be represented in different compressed formats, such as in a run-length-encoded (RLE) format, in which the image information has been compressed, either lossless or lossy. Either way, a compressed format represents the overlay using less amount of data than the uncompressed format. An advantage of representing the overlay in a compressed format is that it is typically cheaper for the image compositor to load the compressed representation of the overlay compared to the uncompressed representation of the overlay. However, providing the compressed representation of the overlay comes with an overhead cost in terms of an extended processing time. In some cases, when the compressed representation of the overlay is generated from the uncompressed representation of the overlay, there is also an additional memory bandwidth cost since the encoder first has to load the uncompressed representation of the overlay before compressing it. In those cases, the combined memory bandwidth cost for first loading the uncompressed representation of the overlay by the encoder and then loading the compressed representation of the overlay by the image compositor is higher than the memory bandwidth cost for loading the uncompressed representation directly by the image compositor. Notably, this overhead cost appears each time a new compressed representation of the overlay is provided. During time periods when the content of the overlay changes between video frames, new compressed representations of the overlay would have to be provided frequently, thereby giving rise to a high overhead cost.

Embodiments herein make use of the fact that during time periods when the content of the overlay does not change between video frames, it is enough to provide a compressed representation of the overlay once, and then to load the same compressed representation of the overlay for every video frame for the time periods when the content of the overlay does not change. Thus, during periods when the content of the overlay does not change the option of using a compressed version of the overlay is advantageous since it reduces the memory bandwidth at a low overhead cost compared to using an uncompressed representation of the overlay.

Thus, the image-processing device will save memory bandwidth at a low overhead cost by adding the overlay in the compressed representation format when the overlay is not updated frequently.

The various aspects of embodiments disclosed herein, including particular features and advantages thereof, will be readily understood from the following detailed description and the accompanying drawings, in which:.

Embodiments herein may be implemented in an image-processing device. In some embodiments herein the image-processing device may comprise or be an image-capturing device such as a digital camera. <FIG> depicts various exemplifying image-capturing devices <NUM>. The image-capturing device <NUM> may e.g., be or comprise any of a camcorder, a network video recorder, a camera, a video camera <NUM> such as a surveillance camera or a monitoring camera, a digital camera, a wireless communication device <NUM>, such as a smartphone, including an image sensor, or a car <NUM> including an image sensor.

<FIG> depicts an exemplifying video network system <NUM> in which embodiments herein may be implemented. The video network system <NUM> may include an image-capturing device, such as the video camera <NUM>, which can capture and perform image processing on a digital image <NUM>, such as a digital video frame. A video server <NUM> in <FIG> may obtain the video frame, for example from the video camera <NUM> over a network or the like, which is indicated in <FIG> with the double-pointing arrows. In some embodiments herein the image-processing device may comprise or be the video server <NUM>.

The video server <NUM> is a computer-based device that is dedicated to delivering video.

However, in <FIG>, the video server <NUM> is connected over the video network system <NUM>, to the image-capturing device, here exemplified by the video camera <NUM>. The video server <NUM> may further be connected to a video storage <NUM> for storage of video frames, and/or connected to a monitor <NUM> for display of video frames. In some embodiments the video camera <NUM> is connected directly with the video storage <NUM> and/or the monitor <NUM>, as indicated by the direct arrows between these devices in <FIG>. In some other embodiments the video camera <NUM> is connected to the video storage <NUM> and/or the monitor <NUM> via the video server <NUM>, as indicated by the arrows between the video server <NUM> and the other devices.

<FIG> is a schematic view of an imaging system <NUM>, in this case of a digital video camera, such as the video camera <NUM>. The imaging system <NUM> images a scene on an image sensor <NUM>. The image sensor <NUM> may be provided with a Bayer filter, such that different pixels will receive radiation of a particular wavelength region, in a known pattern. Typically, each pixel of the captured image is represented by one or more values representing the intensity of the captured light within a certain wavelength band. These values are usually referred to as colour components, or colour channels. The term "image" may refer to an image frame or video frame including information originating from an image sensor that has captured the image.

After having read the signal of individual sensor pixels of the image sensors <NUM>, different image processing actions may be performed by an image signal processor <NUM>. The image signal processor <NUM> may comprise an image processing part 302a, sometimes referred to as an image processing pipeline, and a video post-processing part 302b.

Typically for video processing the images are comprised in a stream of images, also referred to as a stream of video frames. <FIG> illustrates a video stream <NUM> from the image sensor <NUM>. The video stream <NUM> may comprise multiple captured video frames, such as a first captured video frame <NUM> and a second captured video frame <NUM>. The stream of images may also be referred to as a video sequence.

Image processing may comprise application of overlays (e.g., privacy masks, explanatory text). The image signal processor <NUM> may also be associated with an analytics engine performing object detection, recognition, alarms, etc..

The image processing part 302a may e.g. perform image stabilization, apply noise filtering, distortion correction, global and/or local tone mapping, transformation, and flat-field correction. The video post-processing part 302b may for example crop parts of an image, apply overlays, and comprise the analytics engine. Thus, embodiments disclosed herein may be implemented by the video post-processing part 302b.

Following the image signal processor <NUM> the image may be forwarded to an encoder <NUM>, wherein the information in the video frames is coded according to an encoding protocol, such as H. The encoded video frames are then forwarded to for example a receiving client, exemplified here with the monitor <NUM>, to the video server <NUM>, the storage <NUM>, etc..

As mentioned above, an object of embodiments herein may be how to reduce the memory bandwidth in connection to overlaying an overlay, such as a graphics overlay, onto a sequence of video frames. In embodiments herein an overlay is an image to be overlaid on another image, such as onto a video frame of a video sequence.

<FIG> illustrates a video sequence <NUM> corresponding to the video stream <NUM> of <FIG>. The lower part of <FIG> illustrates content of a video frame <NUM> of the video sequence <NUM>.

In <FIG> the video sequence <NUM> has been processed by overlaying an overlay <NUM> comprising text. Specifically, in <FIG> the video frame <NUM> has been processed by overlaying the overlay <NUM>. In embodiments disclosed herein a content of the overlay <NUM> refers to image content of the overlay <NUM>. Such image content may for example be an icon, a diagram, a bounding box, text, a privacy mask, another video frame from another camera or another channel, etc. Generally, the content of the overlay <NUM> may be static, but in embodiments disclosed herein the content of the overlay <NUM> is updated from time to time. For example, the content of the overlay <NUM> may be static for several sequential video frames, but sometimes it will change. For example, the content of the overlay <NUM> may be static for a first set of sequential video frames. Thus, there may be a first content of the overlay <NUM> to be overlaid with the first set of video frames. Then the content of the overlay <NUM> may change and be static again during a second set of video frames. There may be a second content of the overlay <NUM> to be overlaid with the second set of video frames. The change of the content of the overlay <NUM> may comprise change of text, animation of an icon, updating of a diagram, change of position of a bounding box, etc..

<FIG> illustrates a reference method of how an image-processing device <NUM>, such as the image-capturing device <NUM> or the video server <NUM>, may perform such overlaying. The image-processing device <NUM> may store a representation of the overlay <NUM> in an uncompressed representation format.

The representation of the overlay is in the form of an image and the format of the representation refers to an image format of the representation. Specifically, the overlay <NUM> may be represented by an uncompressed image format in which the image information has not been compressed. Thus, the representation of the overlay in the uncompressed representation is a direct representation of image pixels, such as a raster format or a block-based format. The raster format may also be referred to as a bitmap. In embodiments disclosed herein the representation of the overlay in the uncompressed representation format will be referred to as an uncompressed representation <NUM> of the overlay <NUM>. Thus, in embodiments herein the uncompressed representation <NUM> of the overlay <NUM> is in the compressed representation format. The uncompressed representation <NUM> of the overlay <NUM> may further be referred to as a first representation <NUM> of the overlay <NUM>. The uncompressed representation <NUM> may contain values describing colors for each pixel of an image of the overlay <NUM>. The compressed representation <NUM> may further comprise some metadata about the overlay <NUM> like its color format, size, position in the video frame <NUM>, etc..

The representation of the overlay <NUM> may be stored in one or more buffers or memory areas. In some embodiments herein at least two buffers are used such that writing and reading representations of the overlay <NUM> may be performed independently of each other.

For example, a first buffer may be available for writing a first representation of the overlay <NUM>. The first representation of the overlay <NUM> may comprise a first content which may be new compared to a content of a previous representation of the overlay <NUM>. When the first representation of the overlay <NUM> has been written to the first buffer a signal may be generated that indicates that the first buffer is available for reading and thus for adding the first representation of the overlay <NUM> to the video sequence <NUM>. The signal may further indicate that the first content of the overlay <NUM> has been updated compared to a previous content of the representation of the overlay <NUM>. The signal may indicate that the first buffer is active for reading and then adding the first content of the first buffer to the video sequence <NUM>.

While the first buffer is available for reading a second buffer may be available for writing a second representation of the overlay <NUM>. A content of the second representation of the overlay <NUM> may be updated and thus different from the first content.

When the second representation of the overlay <NUM> has been written to the second buffer a second signal may be generated that indicates that the second buffer is available for reading and thus for adding the second representation of the overlay <NUM> to the video sequence <NUM>. The second signal may further indicate that the second content of the overlay <NUM> has been updated compared to the first content of the first representation of the overlay <NUM>.

The representation of the overlay <NUM> may be rendered by the image-processing device <NUM> from a description of the overlay <NUM>. The rendering may be performed by a Central Processing Unit (CPU) or a Graphic Processing Unit (GPU) <NUM> of the image-processing device <NUM>. The CPU/GPU <NUM> may receive the description of the overlay with details of the overlay <NUM> to be added to the video frame <NUM>. The description of the overlay may be in the form of a text file or a graphics file, detailing the content and visual appearance of the overlay <NUM>. The description of the overlay may be in the form of a two-dimensional vector-based description, such as in scalable vector graphic (SVG) format, or a drawing command or instruction for a vector graphics library. The description of the overlay may also be a 3D graphics-based description in the form of a command, a texture and a shader. In some other embodiments the representation of the overlay <NUM> may be read from a library of rendered overlays.

The image-processing device <NUM> may further comprise an image-compositor <NUM> that overlays the representation of the overlay <NUM>, e.g., overlays the uncompressed representation <NUM> of the overlay <NUM>, such as a bitmap overlay, onto the video frame <NUM>. An image-compositor generally refers to a component which combines two or more images to create one image. In this case, the image compositor <NUM> may combine the uncompressed representation <NUM> of the overlay <NUM> with the video frame <NUM>. Thus, for each video frame for which an overlay is to be added the image-compositor <NUM> will load the representation of the overlay <NUM>.

As mentioned above, since the loading of the representation of the overlay <NUM> is performed for each video frame, regardless of whether the representation of the overlay has been updated since the previous frame or not the loading of the representation of the overlay for each video frame represents a considerable cost in terms of memory bandwidth when the overlay <NUM> is represented in the raster format, e.g., by the bitmap overlay.

Embodiments herein will now be described with reference to <FIG> illustrating an enhanced method for adding the overlay <NUM> to the video sequence <NUM> and parts of the image-processing device <NUM> for performing the method. As mentioned above, overlays, such as graphic overlays, may also be represented in different compressed image formats, such as in the RLE format. A compressed format represents the overlay <NUM> using less amount of data than the uncompressed format. In particular, there may be an encoder, e.g., implemented in software together with the CPU or GPU <NUM>, configured to compress, or in other words encode, the uncompressed format representation of the overlay <NUM> into a compressed format representation of the overlay <NUM>. For example, there may be an RLE encoder, e.g., implemented in software together with the CPU or GPU <NUM>, configured to run-length-encode the raster format representation of the overlay <NUM> into an RLE representation of the overlay. Also, there may be a decoder <NUM> for decoding coded image data, such as the RLE coded image data, before overlaying the representation of the overlay onto the video frames. The image-compositor <NUM> may comprise the decoder <NUM> which may be hardware-implemented. Thus, the uncompressed representation <NUM> of the overlay <NUM>, such as the bitmap overlay, may be rendered in a first representation format of the overlay <NUM> which is uncompressed, while a compressed representation <NUM> of the overlay <NUM>, such as an RLE overlay, may be rendered in a second representation format of the overlay <NUM> which is compressed. The compressed representation <NUM> of the overlay <NUM> may further be referred to as a second representation <NUM> of the overlay <NUM>. Other uncompressed formats may be a block-based format, sometimes also called tiled format. The uncompressed format may be a direct representation of pixels. The CPU or GPU <NUM> may produce the uncompressed format or decode it from an image file in a certain format like Joint Photographic Experts Group (JPEG). Other compressed formats may be a block-based entropy coding format, such as JPEG.

Representing the overlay <NUM> in a compressed format typically reduces the memory bandwidth in connection to the overlay <NUM> being loaded by the image compositor <NUM>. However, it comes with an overhead cost to provide the representation of the overlay <NUM> in the compressed format. For example, there will be an additional processing cost when the CPU or GPU <NUM> provides the RLE-encoded overlay <NUM>, both in the case when the RLE-overlay <NUM> is directly rendered from an overlay description and in the case when the RLE-overlay <NUM> is encoded from the bitmap overlay <NUM>. In the latter of these cases, there is also an additional memory bandwidth cost since the CPU or GPU <NUM> first has to load the bitmap overlay <NUM> before compressing it. Notably, this overhead cost appears each time a new representation of the overlay <NUM> in the compressed format is provided. During time periods when the content of the overlay <NUM> changes between video frames, new compressed representations of the overlay <NUM> would have to be provided frequently, thereby giving rise to a high overhead cost. Therefore, it is advantageous to use the bitmap overlay <NUM> when the content of the overlay <NUM> changes. However, during time periods when the content of the overlay <NUM> does not change between video frames, it is enough to RLE-encode the overlay <NUM> once, and then control the image compositor <NUM> to load the same RLE-encoded overlay for every video frame for the time periods when the content of the overlay <NUM> does not change. Thus, during periods when the content of the overlay <NUM> is not updated, or in other words does not change, for example when there is no change of text, animation of an icon, updating of a diagram, change of position of a bounding box, etc., the option of using an RLE-encoded representation of the overlay <NUM> is advantageous since it reduces the memory bandwidth at a low overhead cost compared to using a raster format representation of the overlay <NUM>.

Therefore, in embodiments disclosed herein when the content of the overlay <NUM> requires to be re-rendered frequently or within a short time interval, e.g., for an Augmented Reality (AR) overlay during pan-tilt-zoom (PTZ) movements or bounding boxes that move as objects move, then the overlay <NUM> may be rendered and applied by the uncompressed format, such as a raster format. Once the content of the overlay <NUM> stays stable for a couple of frames, the overlay may be encoded into the compressed format, such as the RLE format, which may be used by the image compositor <NUM> instead of the raster format. In other words, embodiments herein make use of the fact that during time periods when the content of the overlay <NUM> does not change between video frames, it is enough to encode the overlay <NUM> once, and then control the image compositor <NUM> to load the same encoded overlay <NUM> for every video frame for the time periods when the content of the overlay <NUM> does not change. Thus, during periods when the content of the overlay <NUM> does not change, for example when it does not move, the option of using an encoded version of the overlay <NUM> is cheaper from a memory bandwidth point of view compared to using a raster format representation of the overlay memory.

The encoding of the overlay into the compressed format, such as the RLE format, may take a while, but meanwhile the image compositor <NUM> may use the uncompressed format representation, such as the raster format overlay. Then the memory bandwidth usage will be optimized.

Exemplifying embodiments for adding the overlay <NUM> to the video sequence <NUM> will now be described with reference to <FIG>, <FIG>, <FIG> and with further reference to <FIG>, <FIG>, <FIG>, <FIG> and <FIG>.

<FIG> illustrates a flowchart describing a method performed in the image-processing device <NUM> for adding the overlay <NUM> to the video sequence <NUM>. The overlay <NUM> may be a graphic overlay.

In some embodiments the image-processing device <NUM> is the video camera <NUM>, such as a surveillance camera. The image-processing device <NUM> may also be the video-server <NUM>.

The below actions may be taken in any suitable order, e.g., in another order than the order presented below.

The image-processing device <NUM> determines whether or not a content of the overlay <NUM> is updated within a time period T. In some embodiments disclosed herein a length of the time period T is at least two video frames of the video sequence <NUM>. For example, the length of the time period T may be <NUM>-<NUM> video frames in order to avoid too frequent jumps back and forth between the two representation formats i.e., between adding the uncompressed representation <NUM> and adding the compressed representation <NUM> of the overlay <NUM> to the video sequence <NUM>.

<FIG> schematically illustrates a timeline with a first time t1 when the image-processing device <NUM> receives information about the content of the overlay <NUM>. At a second time t2 which is within the time period T from the first time t1 the image-processing device <NUM> checks whether or not it has received new information about the content of the overlay <NUM>. At a third time t3 which is within the time period T from the second time t2 the image-processing device <NUM> again checks whether or not it has received new information about the content of the overlay <NUM>. The same procedure is repeated at a fourth time t4 which is within the time period T from the third time t3. The image-processing device <NUM> may make further checks of whether or not it has received new information about the content of the overlay <NUM> within the time periods T. The CPU or GPU <NUM> may perform the actions of receiving and checking.

In some embodiments determining whether or not the content of the overlay <NUM> is updated within the time period T comprises obtaining first information <NUM> indicative of a first content of the overlay <NUM> and determining whether or not second information <NUM> indicative of a second content of the overlay <NUM> is obtained within the time period T starting from receiving the first information. The second content may differ from the first content.

As explained above in relation to <FIG>, the representation of the overlay <NUM> may be stored in one or more buffers or memory areas. In some embodiments herein at least two buffers are used such that writing and reading representations of the overlay <NUM> may be performed independently of each other. Therefore, in some embodiments disclosed herein obtaining the first information <NUM> comprises obtaining a first indication of a first buffer comprising the first content of the overlay <NUM> and available for reading the first content and wherein obtaining the second information <NUM> comprises obtaining a second indication of a second buffer comprising the second content of the overlay <NUM> and available for reading the second content of the overlay <NUM>. The overlay data may comprise the first representation of the overlay <NUM> in the uncompressed format, such as the raster format or tiled format.

According to <FIG> the image-processing device <NUM> receives the first information <NUM> indicative of the first content of the overlay <NUM> at the first time t1. Then the image-processing device <NUM> waits during the time period T until the second time t2 and checks whether or not it has received new information about the content of the overlay <NUM>. However, at the second time t2 the image-processing device <NUM> does not receive any new information about the content of the overlay <NUM> nor has the image-processing device <NUM> received such new information during the time period T from the first time t1. As will be explained in more detail below in action <NUM> the image-processing device <NUM> may determine that the content of the overlay <NUM> has not changed during the time period T and that this means that there is a possibility to save memory bandwidth by encoding and loading the content of the overlay <NUM> in a compressed representation instead of repeatedly loading the same uncompressed representation of the overlay <NUM>. Also at the third time t3 there is no new received information about the content of the overlay <NUM>. The image-processing device <NUM> may continue to save memory bandwidth by loading and adding the same compressed representation and not repeatedly loading the same uncompressed representation of the overlay <NUM>.

At the fourth time t4 the image-processing device <NUM> again checks whether or not it has received new information about the content of the overlay <NUM>. At the fourth time t4 the image-processing device <NUM> determines that the content of the overlay <NUM> has been updated and may then determine to stop loading and adding the encoded representation of the overlay <NUM> and start using the uncompressed representation of the overlay <NUM>. Thus, as soon as the content of the overlay <NUM> has been updated the encoded representation of the overlay <NUM> will not be loaded and added any more.

The image-processing device <NUM> determines, based on whether or not the content of the overlay <NUM> is determined to be updated within the time period T, whether or not to provide the compressed representation <NUM> of the overlay <NUM> in the compressed format. The compressed representation format may be used later when adding the overlay <NUM> to the video sequence <NUM> which is described below in action <NUM>. It is determined to provide the compressed representation <NUM> of the overlay <NUM> in response to determining that the content of the overlay <NUM> is not updated within the time period T. It is determined to not provide the compressed representation <NUM> of the overlay <NUM> in response to determining that the content of the overlay <NUM> is updated within the time period T.

The image-processing device <NUM> may determine to provide a specific representation format at the end of the time period T. Further, the encoding of the overlay <NUM> into the compressed format may take a while, but meanwhile the image compositor <NUM> may load and add the uncompressed representation of the overlay <NUM>.

In some other embodiments the updating of the overlay <NUM> is deterministic and may be determined beforehand. An example is provided in <FIG> in which the overlay <NUM> displays the time in hours minutes and seconds. For such deterministic overlays the image-processing device <NUM> may determine whether or not the content of the overlay <NUM> is updated within the time period T beforehand. Thus, the image-processing device <NUM> may also determine which representation format to provide beforehand.

In some embodiments disclosed herein the image-processing device <NUM> computes a compression ratio of the compressed representation <NUM> with respect to the uncompressed representation <NUM> when the image-processing device <NUM> has provided the compressed representation. In embodiments disclosed herein the compression ratio may be defined as a ratio of a first memory size required to store the compressed representation <NUM> of the overlay <NUM> divided by a second memory size required to store the uncompressed representation <NUM> of the overlay <NUM>. For example, if the overlay <NUM> has many varying colour patterns, then the compression ratio will be lower than if the overlay <NUM> has large areas with homogeneous colour.

By computing the compression ratio the image-processing device <NUM> may check that the compressed representation <NUM> actually requires less memory bandwidth than the uncompressed representation <NUM>. If the compression ratio is not high enough, for example if it isn't larger or substantially larger than a threshold compression ratio then the image-processing device <NUM> may continue to load and add the uncompressed representation <NUM> to video data of the video frame <NUM> of the video sequence <NUM> although it has already provided the compressed representation <NUM>.

The image-processing device <NUM> then adds the overlay <NUM> to video data of the video frame <NUM> of the video sequence <NUM>.

In general, adding the overlay <NUM> may comprise to exchange image data in the video frame <NUM> with image data in the representation of the overlay <NUM> or to blend image data in the video frame <NUM> with image data in the representation of the overlay <NUM>.

When the compressed representation <NUM> of the overlay <NUM> in the compressed format has been provided the image-processing device <NUM> adds the compressed representation <NUM> of the overlay <NUM> to video data of the video frame <NUM> of the video sequence <NUM> if the compression ratio of the compressed representation <NUM> of the overlay <NUM> with respect to the uncompressed representation <NUM> of the overlay <NUM> is above a threshold, while if the compression ratio is below the threshold, the image-processing device <NUM> adds the uncompressed representation 421of the overlay <NUM> to video data of the video frame <NUM> of the video sequence <NUM>.

When the compressed representation <NUM> of the overlay <NUM> in the compressed format has not been provided, the image-processing device <NUM> adds the uncompressed representation <NUM> of the overlay <NUM> to video data of the video frame <NUM> of the video sequence <NUM>. Thus, determining to add the uncompressed representation <NUM> or the compressed representation <NUM> is based at least on whether the compressed representation <NUM> has been provided or not.

Adding the overlay <NUM> to the video data of the video frame <NUM> of the video sequence <NUM> may comprise providing a representation of the overlay.

When the representation format to be added is the uncompressed format, the image-processing device <NUM> provides the first representation of the overlay <NUM> in the uncompressed format, such as the bitmap overlay <NUM>. Examples of how to provide, or in other words render, the first representation was described above in relation to <FIG>. For example, the CPU/GPU <NUM> may receive the description of the overlay with details of the overlay <NUM> to be added to the video frame <NUM>. The description of the overlay may be in the form of a text file or a graphics file, detailing the content and visual appearance of the overlay <NUM>. The description of the overlay may be in the form of a two-dimensional vector-based description, such as in SVG format, or a drawing command or instruction for a vector graphics library. The description of the overlay may also be a 3D graphics-based description in the form of a command, a texture and a shader.

In some other embodiments the representation of the overlay <NUM> may be read from a library of rendered overlays.

The image-processing device <NUM> may control the image compositor <NUM> to load the uncompressed representation of the overlay <NUM> in the uncompressed format for adding it to the video data of the video frame <NUM> of the video sequence <NUM> when the compressed representation <NUM> of the overlay <NUM> in the compressed format has not been provided or when the compression ratio is below the threshold.

When the representation format to be added is the compressed format, the image-processing device <NUM> provides the second representation of the overlay <NUM> in the compressed format, such as the RLE overlay <NUM>. For example, the encoder, e.g., implemented in software together with the CPU or GPU <NUM>, may encode the first, uncompressed format, representation of the overlay <NUM> into the second, compressed format, representation of the overlay <NUM>. For example, the RLE encoder may run-length-encode the raster format representation of the overlay <NUM> into the RLE representation of the overlay. In some other embodiments, the image-processing device <NUM> provides the second representation of the overlay <NUM> from a description of the overlay <NUM> or from a library of rendered encoded overlays.

The image-processing device <NUM> may control the image compositor <NUM> to decode the compressed representation of the overlay <NUM> in the compressed format and add the decoded compressed representation of the overlay <NUM> to the video data of the video frame <NUM> of the video sequence <NUM> when the compressed representation <NUM> of the overlay <NUM> in the compressed format has been provided and the compression ratio is above the threshold.

In some embodiments herein the image compositor <NUM> is controlled to read a specific representation out of the first representation and the second representation by sending a control message to the image compositor <NUM> with an address to a register in which the specific representation is stored.

In one example embodiment an RLE decoding, or in other words decompression, comprises browsing a message formed of start pixels, an associated pixel value and an associated number of repetition of the pixel value, and adding the pixel value to a sequence of pixels from the start pixel which is as long as the number of repetition of the pixel value.

The uncompressed format may be a direct representation of image pixels, such as a raster format or a block-based format. The compressed format may be the RLE format or a block-based entropy coding format, such as JPEG.

Providing the compressed representation <NUM> of the overlay <NUM> in the compressed format may comprise any one or more of:.

Further, when the representation format is determined to be the compressed format, then the image-processing device <NUM> may provide the same compressed representation <NUM> of the overlay <NUM> in the compressed format to multiple video frames of the video sequence <NUM> as long as there is no update of the content of the overlay <NUM>. This will save memory bandwidth as there is no need to provide further representations of the overlay <NUM> as long as there is no update of the content of the overlay <NUM>.

Further RLE encoded overlays may also be added to the video frame <NUM>. For example, overlays with static content may be added to the video frame <NUM> and other video frames in the video sequence <NUM>.

Even though embodiments herein have mainly been described using RLE as an encoding format other types of compression of the overlay <NUM> are also possible.

With reference to <FIG>, a schematic block diagram of embodiments of an image-processing device <NUM> is shown. The image-processing device <NUM> corresponds to any of the image-processing devices <NUM> of <FIG> or the image-processing device <NUM>. Thus, the image-processing device <NUM> may comprise or be any of a camera, such as a surveillance camera, a monitoring camera, a camcorder, a network video recorder, and the wireless communication device <NUM>. In particular, the image-processing device <NUM> may be the camera <NUM>, such as a surveillance video camera, or the video-server <NUM>.

As mentioned above, the image-processing device <NUM> is configured to perform the method according to <FIG>.

The image-processing device <NUM> may further comprise a processing module <NUM>, such as a means for performing the methods described herein. The means may be embodied in the form of one or more hardware modules and/or one or more software modules.

The image-processing device <NUM> may further comprise a memory <NUM>. The memory may comprise, such as contain or store, instructions, e.g. in the form of a computer program <NUM>, which may comprise computer readable code units which when executed on the image-processing device <NUM> causes the image-processing device <NUM> to perform the method of adding the overlay <NUM> to the video sequence <NUM>.

The image-processing device <NUM> may comprise a computer and then the computer readable code units may be executed on the computer and cause the computer to perform the method of <FIG>.

According to some embodiments herein, the image-processing device <NUM> and/or the processing module <NUM> comprises a processing circuit <NUM> as an exemplifying hardware module, which may comprise one or more processors. Accordingly, the processing module <NUM> may be embodied in the form of, or 'realized by', the processing circuit <NUM>. The instructions may be executable by the processing circuit <NUM>, whereby the image-processing device <NUM> is operative to perform the methods of <FIG> as described above. As another example, the instructions, when executed by the image-processing device <NUM> and/or the processing circuit <NUM>, may cause the image-processing device <NUM> to perform the methods according to <FIG>.

In view of the above, in one example, there is provided an image-processing device <NUM> for adding the overlay <NUM> to the video sequence <NUM>.

Again, the memory <NUM> contains the instructions executable by said processing circuit <NUM> whereby the image-processing device <NUM> is operative for performing the method according to <FIG>.

<FIG> further illustrates a carrier <NUM>, or program carrier, which comprises the computer program <NUM> as described directly above. The carrier <NUM> may be one of an electronic signal, an optical signal, a radio signal and a computer readable medium.

In some embodiments, the image-processing device <NUM> and/or the processing module <NUM> may comprise one or more of a determining module <NUM>, an adding module <NUM>, and a computing module <NUM> as exemplifying hardware modules. In other examples, one or more of the aforementioned exemplifying hardware modules may be implemented as one or more software modules.

Moreover, the processing module <NUM> may comprise an Input/Output unit <NUM>. According to an embodiment, the Input/Output unit <NUM> may comprise an image sensor configured for capturing the raw video frames described above such as the raw video frames comprised in the video stream <NUM> from the image sensor <NUM>.

According to the various embodiments described above, the image-processing device <NUM> and/or the processing module <NUM> and/or the determining module <NUM> is configured to determine whether or not the content of the overlay <NUM> is updated within the time period T.

The image-processing device <NUM> and/or the processing module <NUM> and/or the determining module <NUM> is further configured to determine, based on whether or not the content of the overlay <NUM> is determined to be updated within the time period T, whether or not to provide the compressed representation <NUM> of the overlay <NUM>. It is determined to provide the compressed representation <NUM> in response to determining that the content of the overlay <NUM> is not updated within the time period T. It is determined to not provide the compressed representation <NUM> in response to determining that the content of the overlay <NUM> is updated within the time period T.

When the compressed representation <NUM> of the overlay <NUM> has been provided, the image-processing device <NUM> and/or the processing module <NUM> and/or the adding module <NUM> is further configured to, add the compressed representation <NUM> of the overlay <NUM> to video data of a video frame <NUM> of the video sequence <NUM> if a compression ratio of the compressed representation <NUM> with respect to the uncompressed representation <NUM> of the overlay <NUM> is above a threshold, while if the compression ratio is below the threshold, the image-processing device <NUM> and/or the processing module <NUM> and/or the adding module <NUM> is further configured to, add the uncompressed representation <NUM> of the overlay <NUM> to video data of the video frame <NUM> of the video sequence <NUM>.

When the compressed representation <NUM> of the overlay <NUM> has not been provided, the image-processing device <NUM> and/or the processing module <NUM> and/or the adding module <NUM> is further configured to, add the uncompressed representation <NUM> of the overlay <NUM> to video data of the video frame <NUM> of the video sequence <NUM>.

The image-processing device <NUM> and/or the processing module <NUM> and/or the computing module <NUM> may be further configured to compute the compression ratio of the compressed representation to the uncompressed representation when the image-processing device <NUM> has provided the compressed representation.

In some embodiments herein the image-processing device <NUM> and/or the processing module <NUM> and/or the adding module <NUM> is further configured to add the overlay <NUM> in the determined representation format to the video data of the video frame <NUM> of the video sequence <NUM> by being configured to:.

In some embodiments herein the image-processing device <NUM> and/or the processing module <NUM> and/or the adding module <NUM> is configured to provide the compressed representation <NUM> of the overlay <NUM> in the compressed format by being configured to do any one or more of:.

The image-processing device <NUM> and/or the processing module <NUM> and/or the determining module <NUM> may further be configured to determine whether or not the content of the overlay <NUM> is updated within the time period T by being configured to receive first information <NUM> indicative of the first content of the overlay <NUM> and determine whether or not second information <NUM> indicative of the second content of the overlay <NUM> is received within the time period T starting from receiving the first information.

The image-processing device <NUM> and/or the processing module <NUM> and/or the determining module <NUM> may further be configured to obtain the first information <NUM> by being configured to obtain the first indication of the first buffer comprising the first content of the overlay <NUM> and available for reading the first content and to obtain the second indication of the second buffer comprising the second content of the overlay <NUM> and available for reading the second content of the overlay <NUM>.

As used herein, the term "module" may refer to one or more functional modules, each of which may be implemented as one or more hardware modules and/or one or more software modules and/or a combined software/hardware module. In some examples, the module may represent a functional unit realized as software and/or hardware.

As used herein, the term "computer program carrier", "program carrier", or "carrier", may refer to one of an electronic signal, an optical signal, a radio signal, and a computer readable medium. In some examples, the computer program carrier may exclude transitory, propagating signals, such as the electronic, optical and/or radio signal. Thus, in these examples, the computer program carrier may be a non-transitory carrier, such as a non-transitory computer readable medium.

As used herein, the term "processing module" may include one or more hardware modules, one or more software modules or a combination thereof. Any such module, be it a hardware, software or a combined hardware-software module, may be a connecting means, providing means, configuring means, responding means, disabling means or the like as disclosed herein. As an example, the expression "means" may be a module corresponding to the modules listed above in conjunction with the figures.

As used herein, the term "software module" may refer to a software application, a Dynamic Link Library (DLL), a software component, a software object, an object according to Component Object Model (COM), a software component, a software function, a software engine, an executable binary software file or the like.

The terms "processing module" or "processing circuit" may herein encompass a processing unit, comprising e.g. one or more processors, an Application Specific integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or the like. The processing circuit or the like may comprise one or more processor kernels.

As used herein, the expression "configured to/for" may mean that a processing circuit is configured to, such as adapted to or operative to, by means of software configuration and/or hardware configuration, perform one or more of the actions described herein.

As used herein, the term "action" may refer to an action, a step, an operation, a response, a reaction, an activity or the like. It shall be noted that an action herein may be split into two or more sub-actions as applicable. Moreover, also as applicable, it shall be noted that two or more of the actions described herein may be merged into a single action.

As used herein, the term "memory" may refer to a hard disk, a magnetic storage medium, a portable computer diskette or disc, flash memory, Random Access Memory (RAM) or the like. Furthermore, the term "memory" may refer to an internal register memory of a processor or the like.

As used herein, the term "computer readable medium" may be a Universal Serial Bus (USB) memory, a DVD-disc, a Blu-ray disc, a software module that is received as a stream of data, a Flash memory, a hard drive, a memory card, such as a MemoryStick, a Multimedia Card (MMC), Secure Digital (SD) card, etc. One or more of the aforementioned examples of computer readable medium may be provided as one or more computer program products.

As used herein, the term "computer readable code units" may be text of a computer program, parts of or an entire binary file representing a computer program in a compiled format or anything there between.

As used herein, the terms "number" and/or "value" may be any kind of number, such as binary, real, imaginary or rational number or the like. Moreover, "number" and/or "value" may be one or more characters, such as a letter or a string of letters. "Number" and/or "value" may also be represented by a string of bits, i.e. zeros and/or ones.

As used herein, the expression "in some embodiments" has been used to indicate that the features of the embodiment described may be combined with any other embodiment disclosed herein.

Claim 1:
A method, performed by an image-processing device (<NUM>, <NUM>), for adding an overlay (<NUM>) to a video sequence (<NUM>), the method comprising:
- determining (<NUM>), based on a representation of the overlay (<NUM>) stored in one or more buffers or memory areas of the image-processing device (<NUM>, <NUM>), whether or not a content of the overlay (<NUM>) is updated within a time period (T);
- determining (<NUM>), based on whether or not the content of the overlay (<NUM>) is determined to be updated within the time period (T), whether or not to provide, by a Central Processing Unit, CPU, or Graphic Processing Unit, GPU, (<NUM>), a compressed representation (<NUM>) of the overlay (<NUM>) in a compressed format to an image compositor (<NUM>), wherein it is determined to provide the compressed representation (<NUM>) of the overlay (<NUM>) in response to determining that the content of the overlay (<NUM>) is not updated within the time period (T), and wherein it is determined to not provide the compressed representation (<NUM>) of the overlay (<NUM>) in response to determining that the content of the overlay (<NUM>) is updated within the time period (T); and
- when the compressed representation (<NUM>) of the overlay (<NUM>) in the compressed format has been provided, decoding and adding (<NUM>), by the image compositor (<NUM>), the compressed representation (<NUM>) of the overlay (<NUM>) to video data of a video frame (<NUM>) of the video sequence (<NUM>) if a compression ratio of the compressed representation (<NUM>) of the overlay (<NUM>) with respect to an uncompressed representation (<NUM>) of the overlay (<NUM>) in an uncompressed format is above a threshold, while if the compression ratio is below the threshold, adding (<NUM>) the uncompressed representation (<NUM>) of the overlay (<NUM>) to video data of the video frame (<NUM>) of the video sequence (<NUM>), and
- when the compressed representation (<NUM>) of the overlay (<NUM>) in the compressed format has not been provided, adding (<NUM>) the uncompressed representation (<NUM>) of the overlay (<NUM>) to video data of the video frame (<NUM>) of the video sequence (<NUM>).