METHOD AND DEVICE FOR ASSESSING PACKET DEFECT CAUSED DEGRADATION IN PACKET CODED VIDEO

Because of the encoding, decoding, and/or transmitting characteristic, the blocks affected by packet defect usually gather in a small spatial/temporal area. The viewers perception of each affected block will influence by other affected block in this small area. The invention proposes using processing means for clustering blocks affected by the packet loss into at least one cluster, for using at least one of spatial and temporal characteristics of the at least one cluster for determining a visibility value of the at least one cluster, for classifying the at least one cluster as belonging into one of at least two different class candidates, wherein each class candidate is associated with a different weight; for weighting the determined visibility value with the weight associated with the class of the at least one cluster, and for assessing the degradation of the video using a sum of the weighted visibility value.

EXEMPLARY EMBODIMENTS OF THE INVENTION

The invention may be realized on any electronic device comprising a processing device correspondingly adapted. For instance, the invention may be realized in a television, a mobile phone, a personal computer, a navigation system or a car video system.

In an embodiment, the invention proposes a new pooling technique of detected artefacts which depends on a spatial-temporal occurrence pattern of the artefacts in the video. The proposed pooling is a “swarm based” pooling which tries to mimic the human visual systems (HVS) different sensibility for artefacts in dependency on the size of connected areas in which artefacts occur as well as the development of such areas over time.

When a lot of blocks affected by packet defect are gathered in a small connected area, viewers cannot tell the exact number of total artefacts but only can give some classification such as “big swarm”, “medium swarm”, or “small swarm” which may refer to spatial size and/or temporal duration. The cumulated overall perception distortion caused by these artefacts deviates from simple summation of level values of all the artefacts.

Therefore, clusters or swarms are proposed as replacement. First, swarm can be defined independent from each other. I.e. there is no constraint that swarms should not be near or adjacent to each other though such swarms may be merged, in particular, for keeping the number of swarms at a level of human perception which allows for identifying each swarm. Viewers are then able to identify and remember the features of the swarm because the scale of the swarm matches the scale of human perception.

Swarms are clusters of blocks directly or indirectly, by error propagation through residual encoding, affect by packet defect, i.e. incomplete retrieval or reception of a packet or unavailability of the entire packet.

In an embodiment, swarms comprise all blocks affected by defect of a certain packet. In another embodiment, one swarm can comprise less but all blocks affected by defect of a certain packet, the remaining blocks being comprised in at least one different swarm. In yet another embodiment, blocks affected by defects of in several packets are comprised in one swarm.

That is, the invention is based on swarms related to and resulting from packet defect and proposes different embodiments for refinement of the swarms. The refinement is achieved by a step of swarm merging, a step of swarm splitting or a combination thereof.

Clustering as proposed creates entities which can be assigned with spatial and temporal characteristics such as size and duration of the entity.

This allows for a new pooling strategy using this characteristics for providing a single value which indicates an overall quality or quality degradation of the video, given the artefacts levels for blocks in the video.

In an embodiment, swarms are classified as being of one of two or more, e.g. five, different swarm types, the different swarm types having different weights of contribution, in pooling, to the overall perception distortion.

A single packet loss or partial defect affects an initial set of macro-blocks which can be subjected to error concealment. The artefacts in the initial set then can propagate to previous and/or following frames as a result of inter-frame prediction of video codec. The initial artefacts in the initial set are predictable as they are a direct result of the defect and/or the error concealment. ArtefactsFIG. 1(a) gives an example of such initial artefacts.

The types of artefacts resulting from propagation to previous and/or following frames as a result of inter-frame prediction of video codec are far more difficult to predict. An example of artefacts resulting from propagation is shown inFIG. 1(b).

The types of the propagated artefacts are only indirectly resulting from the defect and/or the error concealment algorithm and may affect only a fraction of a block. Therefore, they are not always predictable. Fortunately, most codec provides some error control method. E.g. slicing is a common error control method in which several macro-blocks constitute a slice and the spatial prediction reference is restricted to the macro-blocks within the same slice. Error propagation is then terminated at the boundary of each slice in spatial axis. IDR is another exemplary error control method to terminate error propagation in the temporal axis.

With error control methods, the error propagation will be limited in a certain range and guaranteed not to be flooded.

A collection of blocks with visible initial artefacts caused by a single packet defect is called am initial swarm. The initial swarm combined with a collection of the blocks with visible artefacts caused by error propagation of the single packet's defect is called a packet swarm.

In an embodiment, different packet swarms comprising adjacent blocks in a same frame or in a contiguous sequence of frames can be fused or merged. A first situation where two swarms swiand swjmay be merged is exemplarily shown inFIG. 3(a). Similarly, a same block affected in successive frames by different packet defects can cause the corresponding packet swarms swiand swjto be merged as exemplarily shown inFIG. 3(b). Or, the packet swarm comprising an affected block in the succeeding frame at a relative location corresponding to a continuation of a motion as indicated by a motion vector of an affect block in the preceding frame can be combined with the packet swarm of said block in the preceding frame. Furthermore, there is an embodiment where a single swarm swican be split into two or more swarms when parts of it propagate into different directions as exemplarily shown inFIG. 3(b).

Let denote the video sequence V={Fi} where Fiis the ithframe of the video, and Fi={Bij} where Bijis the jthblock of frame Fi.

And let denote P={pm}, is the mthpacket which is lost during transmission. For each lost packet pm, a packet swarm swmcan be defined as a set of blocks. This set includes blocks for which a residual and/or a motion vector is affected by defect in packet pm. and blocks with perceivable artefacts which use block(s) in swmas reference, directly or indirectly, e.g. are predicted using these blocks or using blocks predicted by these blocks.

Let denote ALV(Bij) an artefact level value of block Bij. In an embodiment, the set can be limited to blocks which show perceivable artefacts, e.g. with an artefact level value ALV(Bij) at least as high as a perceptibility threshold th. The artefact level value ALV(swm) of a swarm is result of a pooling of the artefact level values of blocks in the swarm:

If blocks which only show non-perceivable artefacts are not already excluded from the swarm, influence of their artefacts in pooling can be suppressed by appropriate weighting. But as the artefact level value of non-perceivable artefacts is low impact on pooling is limited anyway and suppression can be omitted.

Further, let denote, as exemplarily depicted inFIG. 2(a), SZ(swm) a measure of the size of the minimal rectangle which covers the spatial locations of all the artefact blocks A in swarm swm, e.g. the number of blocks comprised in the minimal rectangle of frame Fk. Let denote, as exemplarily depicted inFIG. 2(b), D(swm) a measure of the maximal temporal distance between blocks in swarm swm, e.g. proportional to the number x−1 of affected frames between an earliest frame Fkand a latest frame Fk+xaffected by the swarm. And let denote V(swm)=SZ(swm)*D(swm) the so-called “volume” of a swarm. These values SZ(swm) and D(swm) can be used for classifying the swarm swm, e.g. assign a class value C(swm) to the swarm swmusing at least one of size and duration of the swarm, the class value C(swm) being associated with a weight coefficient w(C(swm)). The weight coefficients used in an exemplary embodiment was determined using a dataset of videos with mean observer scores determined based on subjective tests.

An embodiment of the proposed invention then determines an overall distortion or artefact level value of the video by weighted summation of the artefact level values of the swarms in the video, wherein each swarm's artefact level value is weighted by the weight coefficient associated with the class value assigned to the swarm using its spatial and/or temporal characteristic:

In an exemplary embodiment, a binary classification of swarms in small swarms and big swarms is realized. To be classified as a big swarm, a swarm lasting longer than a predetermined duration threshold thDspecifying a number of frames, D(swm)>thD, is classified as a big swarm. In a further exemplary embodiment, a swarm with a volume of at least a predetermined number of blocks thV, V(swm)>thV, is classified as a big swarm. In yet a further exemplary embodiment, a swarm with an artefact density, the swarm's artefact level value divided by the swarm's volume at least as high as a predetermined artefact density threshold thA, ALV(swm)/V(swm)>thA, is classified as a big swarm. Even yet further exemplary embodiments combine two of the criteria for classification as a big swarm. An exemplary embodiment using all three criteria further used thD=2 and thV=19, and set w(C(swm))=c0in case of C(swm))=0 and set w(C(swm))=c1in case of C(swm))=1 with c1<>c0, c0and c1being comprised in [0; 1].

The decision of c0and c1is an optimization problem, to maximize the value of the Pearson's sample correlation which is obtained by dividing the covariance of the mean observer score and the predicted score by the product of their standard deviations:

MOS is a sample vector of subjective mean scores assigned to given videos in a data base and PRED(ALV(c1, c1))) is a sample vector of predicted scores derived artefact level values calculated using the given videos in the data base. Pearson's sample correlation is the correlation between these two vectors. Pearson's sample correlation is a suitable measure for determining prediction accuracy.

Solve the optimization problem in an exemplary dataset, the prediction accuracy reaches maximum for c0=0.9, and c1=0.1. wherein the maximum reached is by 10 percent higher than the maximum reachable with a pooling which indiscriminatingly adds up all the artefacts in the blocks of the video.

The exemplary data base comprises six CIF format video contents, which cover a wide range of spatial complexity index and temporal complexity index, namely Foreman, Hall, Mobile, Mother, News, and Paris. The six sequences are encoded using H.264 encoder with two sequence structures, IBBP and IPPP. Group of Picture (GOP) size (i.e. the length between two IDR frames) is 15 frames. A proper fixed quantization parameter is used to prevent the compressed video from visible coding artefacts. Each row of macro-blocks is encoded as an individual slice, and one slice is encapsulated into a RTP packet. To simulate transmission error, loss patterns generated at five packet loss rates (PLRs) [0.1%, 0.4%, 1%, 3%, 5%] are used to generate error bitstream, which is decoded by ffmpeg decoder to generate PVSs (processed video sequences) for viewers to perform subjective scoring as well as for automatic MOS prediction.

A more complex exemplary embodiment uses for classification the following five classes, each with a corresponding different weight:

Imperceptible: “no artefact (or problematic area) can be perceived during the whole video display period”, e.g. all of swarm size, swarm duration and artefact density in the swarm are below corresponding thresholds.

Perceptible but not annoying: “artefact(s) can be perceived occasionally, but don't influence the interested content, or it appears in the background for an instant moment”, e.g. swarm size and swarm duration are below corresponding thresholds.

Slightly annoying: “noticeable artefact appear in the region of interest (ROI), or noticeable artefacts are detected for several instant moments even if they do not appear in the ROI”, e.g. artefact density in the swarm and one of swarm size and swarm duration are below corresponding thresholds.

Annoying: “noticeable artefact appears in ROI for several times or many noticeable artefacts are detected and last for a long time”, e.g. artefact density in the swarm is below a corresponding threshold.

Very annoying: “video content cannot be understood well due to artefacts and the artefacts spread all over the sequence”, e.g. none of e.g. swarm size, swarm duration and artefact density in the swarm is below a corresponding threshold.

There is an exemplary embodiment of the invention where a swarm based pooling strategy is used to evaluate the overall quality of a video which is degraded by packet loss, given the artefact level of all the blocks in the video. In the used pooling strategy, at first the blocks with perceivable artefacts are grouped into clusters, so-called swarms, according to their spatial/temporal locations. Then each swarm is classified and assigned a weight coefficient depending on the classification. Contribution of each swarm to the overall quality degradation is determined by multiplying the sum of the artefact levels of all blocks in the swarm by the assigned weight coefficient. Finally contributions of all the swarms are added up to determine the overall quality degradation.