# Full Reference Video Quality Assessment based on Statistical based Transform Coefficients

Amitesh Kumar Singam, Muhammad Shahid

► To cite this version:

Amitesh Kumar Singam, Muhammad Shahid. Full Reference Video Quality Assessment based on Statistical based Transform Coefficients. 2023. hal-03973193

HAL Id: hal-03973193

https://hal.science/hal-03973193

Preprint submitted on 4 Feb 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Full Reference Video Quality Assessment based on Statistical based Transform Coefficients

1st Amitesh Kumar Singam

IEEE Signal Processing Society

Institute of Electrical and Electronics Engineers

Hyderabad, India

0000-0002-2532-0989

*Abstract*—Now a days, need of Application developers towards developing front end based video applications like Skype or others forced in huge competition between quality of service and experience. Out of all existing approaches, we considered alternative approach based on full reference video quality assessment and moreover, our interest lies towards formulating and melding effective features into one model based on human visualizing characteristics. This research explores the trade offs between quality prediction and complexity towards identifying sparseness of LSSVM model and also involves in feature extraction of h.264 bit stream information extracted at macro block layer towards building up of a machine learning based model for quality Assessment. These features which are expected to have high correlation with the perceptual quality of the videos and We concluded that our proposed model outperformed in terms of performance but fault in encoding process which was traced out due to error concealment should be evaluated and in case of building up of proposed model with SSIM and MS-SSIM metric at frame level sparseness was traced out.

## I. INTRODUCTION

In Research field, due to the extremely huge demand of Quality of Service(QoS) and quality of experience(QoE) between end users and application developers towards providing well developed metric for the better quality resulted in neediness for the robust methods towards assessing the video quality. Therefore services like video calling and live video streaming in mobile or hand-held devices boosted the interest in objective video quality assessment since the availability of reference video is partially available. One of the key features of video services is user experience according to present trending as observed by the end user. For instance, the visual quality within transmission can get degraded while capturing, loading, saving or reproduction and distortions due to packet loss, delay or other issues. Since humans are considered as true judges of video quality due to end users of the video services. In technical terms, the process of validating of video quality by viewers is referred as subjective video quality assessment. However, subjective experiments are often too time-consuming, inconvenient, costly and moreover it must be done under specific recommendations in order to produce standard results but not presently because of awareness of User Experience(UX). Moreover these facts gave rise to neediness of some effective ways of estimating quality of videos based on prediction accuracy.

## II. VIDEO QUALITY ASSESSMENT

In our research work, we carefully employed the specifications recommended by ITU-R BT 500-10 as mentioned in [1] and VQEG Hybrid test plan in [2] which is explained briefly in following sections.

### A. Test Video Sequences

In this paper six different video sequences of CIF and QCIF spatial resolutions were selected in raw progressive format from [3] based on different motion content and including various levels of spatial-temporal complexity recommended by ITU-R P.910 [4]. The estimation of spatial-temporal information is essential in order to find quality of transmitted video sequence is highly dependent on respective whereabouts. The formulations for quantifying spatial and temporal perceptual information of test sequences are shown in following sections.

a) Spatial Information: SI is calculated for each frame of video sequence on luminance component, which leads in obtaining time series(n frames) of spatial information and mathematically expressed as

$$SI = \max_{time}[\text{std}_{space}[\text{sobel}(f_n)]] \quad (1)$$

* sobel(f_n) is each frame at time n which is filtered with sobel filter.

* std_space is standard deviation over pixels.

* max_time is maximum value in time series.

b) Temporal Information: TI is calculated for amount of temporal changes of a video sequence on luminance plane and measurement of temporal perceptual information is computed as

$$TI = \max_{time}[\text{std}_{space}[M_n(i, j)] \quad (2)$$

M(i,j) is motion difference between pixel values in space for sequential frames on the luminance plane. M_n(i, j) as a function of frame(n) and it is expressed as

$$M_n(i, j) = f_n(i, j) - f_{n-1}(i, j)$$

f_n(i, j) is pixel at i^th row and j^th column of a n^th frame in a video sequence and f_{n-1}(i, j) is pixel at i^th row and j^th column of a (n-1) frame in a video sequence. The higher motion content in successive frames of a video sequence leads to higher temporal content. The below scatter plots shows
spatial and temporal perceptual content obtained from above equations, which were computed on luminance plane for all CIF and QCIF test sequences.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Spatial and Temporal information computed for luminance component of selected CIF and QCIF videos</figcaption></figure>

Each of 120 video sequences are of 10 seconds long with fast, medium and slow motion content. In this paper the generation, scaling and encoding process of Test videos sequences were done using JM reference software 16.1 based on H.264 standards from [5]. In Group Of Pictures, intra coded frame contains high quality but gradually reduced for following sequence of P-frames and the occurrence of I frame effect will have huge influence on subjective test. Since the I-frame plays important role in testing video sequences, It has been finalized not to consider initial 10 seconds for quality measurement. With the help of virtual dub, same video sequence was doubled up from 10sec to 20sec without changing temporal and spatial resolutions. Finally after encoding process, all test sequences were cropped last 10 out of 20 seconds long in order to overcome I frame effect.

<table><thead><tr><th>Spatial Resolutions</th><th>Video Sequences</th><th>Bit rates(kbps)</th></tr></thead><tbody><tr><td rowspan="7">CIF @ 30,15 frame rates</td><td>Akiyo</td><td>200,400,600,800,1000</td></tr><tr><td>News</td><td>200,400,600,800,1000</td></tr><tr><td>Foreman</td><td>200,400,600,800,1000</td></tr><tr><td>Crew</td><td>200,400,600,800,1000</td></tr><tr><td>Soccer</td><td>200,400,600,800,1000</td></tr><tr><td>Football</td><td>200,400,600,800,1000</td></tr><tr><td rowspan="7">QCIF @ 30,15 frame rates</td><td>Akiyo</td><td>100,200,300,400,500</td></tr><tr><td>News</td><td>100,200,300,400,500</td></tr><tr><td>Foreman</td><td>100,200,300,400,500</td></tr><tr><td>Crew</td><td>100,200,300,400,500</td></tr><tr><td>Soccer</td><td>100,200,300,400,500</td></tr><tr><td>Football</td><td>100,200,300,400,500</td></tr></tbody></table>

The encoding details of CIF and QCIF video sequences are mentioned in above tabular column. Out of 120 videos, the video sequences encoded at 15fps are temporally up scaled to 30fps by repeat frame method and QCIF videos are spatially up scaled to CIF using Bi-cubic Interpolation method with the help of Virtual dub for 120 test sequences at temporal resolution of 30fps and spatial resolution of 352x288(CIF).

### III. PROPOSED IDEA

Mainly, the proposed method involves in extraction of visual quality relevant bit stream information at macro block level and building of a machine learning based model for quality Estimation. These features were based on high computational complexity towards respective target scores.

#### A. Feature Extraction of H.264 Bitstream Data

The feature extraction of H.264 coded bitstream data was done as following. Firstly the encoded bitstream data of transmitted video sequence within a channel were decoded using a modified version of JM reference software 16.1 in order to generate an XML file of coding features for each video sequence. These XML files contained video information at macroblock level. Secondly a Java program was developed to analyze and process the large XML data in order to extract 17 selected features of the encoded videos at frame level. These features which are expected to have high correlation with the perceptual quality.

1) Test Methodology of LSSVM Model: Radial Basis function was selected as kernel function for realization implicit mapping of give input data into higher dimensional induced kernel space and it provides good efficiency within predication accuracy which results in obtaining better training and testing errors. An optimization algorithm was used for tuning the hyper parameters σ and γ with respect to good performance measure. Grid search method was employed which performs an exhaustive search through a subset of features space in proposed algorithm for solving model selection by finding optimal parameters [6]. This algorithm is guided by performance metric(MSE) and measured by leave one out cross validation in training set. The performance and accuracy of LSSVM model depends on (σ², γ), Where σ² width of kernel and γ is regularization parameter. For each pair of hyper parameters (σ², γ) leave one out cross validation method on training set is performed to estimate the prediction error. Therefore a robust model is obtained by selecting those optimal pair of hyper parameters which gives the lowest MSE.

### IV. CROSS VALIDATION STRATEGY

In order to avoid the over fitting problem, we need to measure the generalization performance of LSSVM model when data approximation is not used during training process.In this paper, we selected leave one out cross validation technique to obtain better generalization ability. k-fold cross-validation has capability of not only minimizing training error but also reduces cross validation error. Cross-validation is a re-sampling strategy used to validate the performance of model by random sub-sampling of the available data. In other words, the original data is subdivided randomly into k folds. In each round of CV, k-1 folds are used for training and the remaining one fold is used for validation, this procedure is repeated for k times with each of the k folds should use exactly once for the model validation. Performance estimation of our proposed model is determined by average of results obtained in k folds for k rounds

#### A. Leave One Out Cross Validation

We employed leave one out cross validation technique for estimating the performance of predictive model, leave one out cross validation method is functions similar to K-fold CV, where K is equal to total number of data points. In each round of LOOCV, single data point is used for validation while
remaining data points are used for training and this procedure is repeated such that each of the all data points should involve once for the model validation.

Given data set $D = [(x_1, y_1),..., (y_n, y_n)]$

Training set $T = [(x'_1, y'_1),..., (x'_n, y'_n)]$

For Mε (1..c), where M is prediction model, (X, Y) is random variable distributed according to given probability distribution function in order to reduce the Mean square error.

$$ \epsilon_m(i) = E_L(Y, f_m(X)) \quad (3) $$

where $E_L$ is expected loss.

Working methodology of LOOCV is explained in following steps

* Step1: randomly permute the data from given data set

* Step2: Split into N-Folds, each fold is equal to one instance of given input data.

* Step3: for each round use N-1 instances for training and remaining for Validating.

$$ MSE_{LOOCV} = \frac{1}{N} \sum_{n=1}^{N} \epsilon_m(n) \quad (4) $$

Where $\epsilon_m$ Average of empirical error and $\epsilon_m(n)$ is empirical error in validation set in each of N rounds for N-instances.

* Step 4: we choose M* to minimize $\epsilon_m$ and retrain using M* on all of data and then we get final prediction function

<figure><img src="image_3.png" /><figcaption>Fig. 2: Architecture Design of Proposed Model for Video quality</figcaption></figure>

## V. BENCHMARK MEASUREMENT

According to VQEG phase II test [7], Performance of Objective Quality prediction Model is evaluated by three aspects. They are prediction accuracy, consistency and monotonicity, these attributes are evaluated by following performance metrics.

**Metric-1:** Pearson linear correlation metrics related to prediction accuracy of proposed prediction model which measures the linearity between a performance of model and the scores. Mathematical model of Pearson linear correlation coefficient metrics is

$$ r_p = \frac{\sum_{i=1}^{N} (\bar{Y}_i - \bar{Y}) * (Y_i - Y)}{\sum_{i=1}^{N} \sqrt{((\bar{Y}_i - \bar{Y})^2 * (Y_i - Y)^2}}} \quad (5) $$

where N denotes total number of video sequences. $\bar{Y}_i, Y_i$ represents estimated and target values of video sequence. Y, $\bar{Y}$ represents mean of target and estimated values respectively.

## VI. EVALUATION BASED ON STATISTICAL BASED TRANSFORM COEFFICIENTS

For instance, it's not per video, we evaluated our proposed model at frame level. Moreover, these features were extracted out of H.264 bit stream data and also These features are expected to have high correlation with corresponding perceptual quality score of the selected video, mainly feature extraction of 120 videos has been processed using JM Reference software Version 16.0. The properties of encoded videos are acquired from bit stream data of H.264/AVC which has been generated as a trace file while encoding process. Rather than using completely decoded frame, our interest lies in reversing the entropy encoding of bit stream. By analyzing three successive Nal, Slice and Macro Block Layer and the following features were extracted.

* Avg QP- Average Quantization Parameter.

* Avg bitrate [kbps]-Average Bits per second

* Inter[%]-Percentage of Inter Coded Macro Blocks

* Intra[%]-Percentage of Intra Coded Macro Blocks

* Skip[%]-Percentage of Skip Coded Macro Blocks

* P16x16[%]-Percentage of Inter Coded Macro Blocks with 16x16 subdivision

* P8x8[%]-Percentage of Inter Coded Macro Blocks with 8x8 subdivision

* P4x4[%]-Percentage of Inter Coded Macro Blocks with 4x4 subdivision

* MVX- Average of Horizontal absolute Motion Displacement

* MVY- Average of Vertical absolute Motion Displacement

* MVDX- Average of the Motion Displacement difference in horizontal direction

$$ MVD_X = |MV_x(i_l, j) - MV_x(i_r, j)| \quad (6) $$

Where $(i_l, j)$ and $(i_r, j)$ positions at left and right edge image or frame

* MVDY- Average of the Motion Displacement difference in vertical direction

$$ MVD_Y = |MV_y(i_l, j) - MV_y(i_r, j)| \quad (7) $$

Where $(i_l, j)$ and $(i_r, j)$ positions at left and right edge image or frame

* Zero MVs[%]-Percentage of Zero absolute Motion vectors

* Zero MVDs[%]-Percentage of Zero Motion vector Difference

* Motion Intensity

$$ MI1 = \sum_{i=0}^{N} \sqrt{MV_{X^2_i} - MV_{Y^2_i}} \quad (8) $$

where N is the total number of macro blocks in each frame. MVX and MVY are the absolute motion vector
of the i-th macro block in Horizontal(X) and Vertical(Y) directions respectively.

* Motion Intensity II

$$MI2 = \sqrt{MV_{X}^{2} - MV_{Y}^{2}} \qquad (9)$$

where MV_X and MV_Y are the average of absolute motion vectors in each frame in X and Y directions respectively.

* InPframes[%]-Percentage of Intra coded macro blocks in P frames.

## VII. EXPERIMENTAL ANALYSIS

Our LSSVM Model is trained and tested with 120x17 input features out of 120 video sequences generated, moreover, input data contains 17 selected features extracted from bit stream information at decoder side of channel as mentioned. Objective scores of 120 video sequences for SSIM and MS-SSIM metrics were selected as target values for our Proposed model and our LSSVM model is trained randomly with 80x17 features with respective 80x1 target values. Trained LSSVM model has been tested with remaining 40x17 features accordingly. The performance of LSSVM model is measured between true and predicted values for selected quality metrics. The prediction accuracy of our proposed models for generated video sequences was evaluated by Pearson correlation coefficient between true and predicted scores after non-linear regression analysis and regression plots are generated for the proposed models and for different quality metrics used. Moreover regression plots illustrate predicted values or residuals after regressing with true values Y of various quality metrics. Below figure illustrates correlation between predicted and target scores of SSIM. Below figure illustrates correlation

<figure><img src="image_4.png" /><figcaption>Fig. 3</figcaption></figure>

between predicted and target scores of MS-SSIM.

MS-SSIM LSSVM: R=0.96554

<figure><img src="image_5.png" /><figcaption>Fig. 4</figcaption></figure>

## VIII. CONCLUSION

We concluded that our metric has good fitness towards proposed method in-terms of correlation coefficient and moreover our proposed metric completely understood Human Visual System. It would be rather interesting to reconnoiter the state of the art in machine learning algorithms which helps in further improvement of LSSVM algorithm in an effective way towards eliminating Sparseness.

## REFERENCES

[1] "ITU-R radio communication sector of itu, recommendation itu-r bt.500-12," 2009, http://www.itu.int/.

[2] "Vqeg hybrid testplan, version 1.2," 2011, ftp://vqeg.its.bldrdoc.gov.

[3] "Test video sequences," 2012, http://http://media.xiph.org/.

[4] "Subjective video quality assessment methods for multimedia applications," September 1999, iTU-T, Recommendation ITU-R P910.

[5] "H.264/avc jm reference software version 16.0, tech.rep, joint video team," 2011, available: http://iphome.hhi.de/suehring/tml/download/.

[6] W. Zhang, C. Li, and B. Zhong, "Lssvm parameters optimizing and non-linear system prediction based on cross validation," in *Fifth International Conference on Natural Computation, ICNC 09*, vol. 1, aug. 2009, pp. 531-535.

[7] "Full reference television phase ii subjective test plans," September 2002, objective Quality Model Evaluation Criteria.