# Intraprediction Complexity Control Algorithm Based on Visual Saliency

Authors: Nana Shan; Wei Zhou; Zhemin Duan
Journal: Journal of Sensors
 (2022)

---

## Abstract

Intraprediction is one of the most complex parts of High-Efficiency Video Coding (HEVC), because it selects the best prediction mode by calculating the cost of every Coding Unit (CU), which provides higher complexity of intracoding. Visual saliency map can show the attention regions of the human eyes, which is generated by certain static and space-time saliency detection method. By analyzing the percentage of coding time for different size CU, and the relation of visual saliency and CU depth, an intraprediction complexity control algorithm based on visual saliency is proposed in this paper. Based on the feature of the video and the target level, the saliency threshold is adapted to determine whether the current CU in the intraprediction processing should be split into smaller CUs or the division processing should be stopped early. Three samples were compared by the proposed algorithm and other algorithm, and the proposed algorithm has better performance in PSNR, BitRate, and coding time. Experimental results show that this algorithm can effectively control the coding complexity of intraprediction with minimal visual loss and can be applied to a number of scenarios, such as real-time video coding.


---

## Body

## 1. Introduction

The number of intraprediction modes in HEVC is increased up to 35, and like interprediction coding, intraprediction also use recursive algorithm to compare all prediction modes of all coding units, and then selects the optimal mode from them, which will cause high coding complexity [1].In order to use the correlation between luma and chroma, [2] presents a convolutional neural network based intrachroma prediction method and cross component linear model. The plane modeling coefficients can be predicted by the neighboring depth pixels, and a certain threshold can be compared with the prediction error [3].Support Vector Machine (SVM) was adopted to get higher precision decision-making in [4]. For different video sequences with different coding complexity, the proposed method employed a fixed decision boundary. An improved CU-level rate control algorithm was proposed in [5]. The gradient detection operator and the Hadamard transform algorithm are used to detect the texture complexity area and the transform compression area, and the scale coefficients adjust the target bits of each CTU. In order to tackle the shortcoming of traditional machine learning algorithms, reference [6] proposed the acceleration properties of different modules. It also provides Heuristic Model Oriented Framework (HMOF) algorithm, which can adapt the properties of different modules. It uses various Convolutional Neural Network (CNN) to choose the best intraprediction mode and it investigates combinations of them [7].However, the above algorithms are based on the characteristics of intraprediction itself, which are based on the statistical results of the coding units. According to the feature of visual saliency, it also can be analyzed from the visual characteristics of the human eyes to reduce the complexity of intraprediction, which can control the complexity of intraprediction with minimal visual loss. In this paper, a complexity control algorithm of intraprediction based on visual saliency is proposed, which can reduce the encoding time within the target complexity range, and the visual loss is unnoticeable. By adapting this method, different coding strategy can be made in different application scenario, which can bring higher compression performance.


## 2. Intraprediction

In the spatial domain, the closer the distance between pixels, the stronger the correlation is between them, so we can use the weighted sum of several adjacent pixels to predict the current pixel. Then in the transmission, it is not necessary to transmit the actual pixel value but only the difference between the actual pixel value and the predicted value. The original pixel value can be recovered by directly adding the difference signal to the predicted value in the decoding process. Because of the strong correlation between adjacent pixels, the residual value is often very small, which can reduce the encoded information and achieve the purpose of video compression. Intraprediction is a process of removing the spatial redundancy of the video, which uses the correlation between the video images in the spatial domain to predict the current pixel according to the pixels that have been coded in the current image. It is an important compression method in video coding, especially when interprediction coding cannot be used. Intraprediction coding has become an important means to ensure the compression rate [8].


### 2.1. Prediction Mode

In HEVC, the prediction template for intraprediction is shown in Figure1, where Rx,y (shown in gray area) on the left and above represents the pixel reconstruction value of the adjacent blocks and Px,y (shown in white area) in the middle represents the pixel prediction value of the current block.Figure 1
Intraprediction template [1].The prediction unit sizes supported by intraprediction are4×4, 8×8, 16×16, 32×32, and 64×64, and each PU size includes 35 prediction modes, as shown in Table 1, which are planar mode, DC mode, and 33 angle prediction modes.Table 1
Number of 35 prediction modes.
No.Mode0Planar mode1DC mode2-3433 angle prediction modes


#### 2.1.1. Planar Mode

As shown in Figure2, the planar mode uses filters in the horizontal and vertical directions and uses the average value as the predicted value of the current pixel, which can better maintain the continuity of the image boundary.Figure 2
Planar mode [1].


#### 2.1.2. DC Mode

The DC prediction is obtained by using the average value of the reference pixels on the left side and the upper side (excluding the upper left corner, the lower left side, and the upper right side) of the current pixel, so it is suitable for a flat image with a large area.


#### 2.1.3. Angle Mode

HEVC specifies 33 angle prediction modes, mode 10 represents the horizontal direction and mode 26 represents the vertical direction, patterns 2-17 are horizontal class patterns and patterns 18-34 are vertical class patterns, as shown in Figure3. Different modes correspond to different offset angles, and the size of the offset angle can be calculated by
(1)θ=arctanx32.Figure 3
Angle prediction mode [1].The 33 angle prediction modes can be divided into two categories: vertical prediction and horizontal prediction. 2-17 is the horizontal prediction. When the prediction direction shifts upward,θ is positive. When the prediction direction is shifted downward, θ is negative. 18-34 is a vertical prediction, and θ is positive when the prediction direction shifts to the left; when the prediction direction is shifted to the right, θ is negative.


### 2.2. Flow of Intraprediction

There are three steps in intraprediction:


#### 2.2.1. Get Reference Pixels

As shown in Figure1, the reference pixels of the current pixel block can be divided into five parts: left, lower left, upper left, upper, and upper right. If some reference pixels do not exist, they will be replaced by reference pixels in adjacent areas. When all reference pixels are unavailable, they will be replaced by fixed values. The calculation method is as shown:
(2)R=1≪BitDepth−1.


#### 2.2.2. Reference Pixel Filtering

When the pixel position is as shown in Figure1, vertical-right mode is shown in equation (3), horizontal-down mode is shown as equation (4), diagonal-down-right mode is shown in equation (5), and the filtering calculation method is
(3)Fx,0=Rx−1,0+2Rx,0+Rx+1,0+2≫2,(4)F0,y=R0,y−1+2R0,y+R0,y+1+2≫2,(5)F0,0=R0,1+2R0,0+R1,0+2≫2,(6)F2N,0=R2N−1,0+3R2N,0+2≫2,(7)F0,2N=R0,2N−1+3R0,2N+2≫2.


#### 2.2.3. Calculate the Predicted Value

The calculation methods of the predicted value are different in different modes. The calculation methods of the three modes are described as follows:(1) Planar Mode. The calculation method of the planar mode predicted pixel is shown:
(8)Px,yH=N−xR0,y+x∙RN+1,0,(9)Px,yV=N−yRx,0+y∙R0,N+1,(10)Px,y=Px,yH+Px,yV+N≫log2N+1.(2) DC Mode.
(11)DCAverage=∑x=1NRx,0+∑x=1NR0,y+N≫log2N+1,(12)P1,1=R1,0+R0,1+2∙DCAverage+2≫2,(13)Px,1=Rx,0+3∙DCAverage+2≫2,(14)P1,y=R0,y+3∙DCAverage+2≫2,(15)Px,y=DCAverage.(3) Angle Mode. For angular prediction modes, each mode is offset in either the horizontal or vertical direction. First of all, the reference pixel projection needs to be mapped into a one-dimensional form according to its angle offset value, and then the predicted value of the current pixel is calculated.


## 2.1. Prediction Mode

In HEVC, the prediction template for intraprediction is shown in Figure1, where Rx,y (shown in gray area) on the left and above represents the pixel reconstruction value of the adjacent blocks and Px,y (shown in white area) in the middle represents the pixel prediction value of the current block.Figure 1
Intraprediction template [1].The prediction unit sizes supported by intraprediction are4×4, 8×8, 16×16, 32×32, and 64×64, and each PU size includes 35 prediction modes, as shown in Table 1, which are planar mode, DC mode, and 33 angle prediction modes.Table 1
Number of 35 prediction modes.
No.Mode0Planar mode1DC mode2-3433 angle prediction modes


### 2.1.1. Planar Mode

As shown in Figure2, the planar mode uses filters in the horizontal and vertical directions and uses the average value as the predicted value of the current pixel, which can better maintain the continuity of the image boundary.Figure 2
Planar mode [1].


### 2.1.2. DC Mode

The DC prediction is obtained by using the average value of the reference pixels on the left side and the upper side (excluding the upper left corner, the lower left side, and the upper right side) of the current pixel, so it is suitable for a flat image with a large area.


### 2.1.3. Angle Mode

HEVC specifies 33 angle prediction modes, mode 10 represents the horizontal direction and mode 26 represents the vertical direction, patterns 2-17 are horizontal class patterns and patterns 18-34 are vertical class patterns, as shown in Figure3. Different modes correspond to different offset angles, and the size of the offset angle can be calculated by
(1)θ=arctanx32.Figure 3
Angle prediction mode [1].The 33 angle prediction modes can be divided into two categories: vertical prediction and horizontal prediction. 2-17 is the horizontal prediction. When the prediction direction shifts upward,θ is positive. When the prediction direction is shifted downward, θ is negative. 18-34 is a vertical prediction, and θ is positive when the prediction direction shifts to the left; when the prediction direction is shifted to the right, θ is negative.


## 2.1.1. Planar Mode

As shown in Figure2, the planar mode uses filters in the horizontal and vertical directions and uses the average value as the predicted value of the current pixel, which can better maintain the continuity of the image boundary.Figure 2
Planar mode [1].


## 2.1.2. DC Mode

The DC prediction is obtained by using the average value of the reference pixels on the left side and the upper side (excluding the upper left corner, the lower left side, and the upper right side) of the current pixel, so it is suitable for a flat image with a large area.


## 2.1.3. Angle Mode

HEVC specifies 33 angle prediction modes, mode 10 represents the horizontal direction and mode 26 represents the vertical direction, patterns 2-17 are horizontal class patterns and patterns 18-34 are vertical class patterns, as shown in Figure3. Different modes correspond to different offset angles, and the size of the offset angle can be calculated by
(1)θ=arctanx32.Figure 3
Angle prediction mode [1].The 33 angle prediction modes can be divided into two categories: vertical prediction and horizontal prediction. 2-17 is the horizontal prediction. When the prediction direction shifts upward,θ is positive. When the prediction direction is shifted downward, θ is negative. 18-34 is a vertical prediction, and θ is positive when the prediction direction shifts to the left; when the prediction direction is shifted to the right, θ is negative.


## 2.2. Flow of Intraprediction

There are three steps in intraprediction:


### 2.2.1. Get Reference Pixels

As shown in Figure1, the reference pixels of the current pixel block can be divided into five parts: left, lower left, upper left, upper, and upper right. If some reference pixels do not exist, they will be replaced by reference pixels in adjacent areas. When all reference pixels are unavailable, they will be replaced by fixed values. The calculation method is as shown:
(2)R=1≪BitDepth−1.


### 2.2.2. Reference Pixel Filtering

When the pixel position is as shown in Figure1, vertical-right mode is shown in equation (3), horizontal-down mode is shown as equation (4), diagonal-down-right mode is shown in equation (5), and the filtering calculation method is
(3)Fx,0=Rx−1,0+2Rx,0+Rx+1,0+2≫2,(4)F0,y=R0,y−1+2R0,y+R0,y+1+2≫2,(5)F0,0=R0,1+2R0,0+R1,0+2≫2,(6)F2N,0=R2N−1,0+3R2N,0+2≫2,(7)F0,2N=R0,2N−1+3R0,2N+2≫2.


### 2.2.3. Calculate the Predicted Value

The calculation methods of the predicted value are different in different modes. The calculation methods of the three modes are described as follows:(1) Planar Mode. The calculation method of the planar mode predicted pixel is shown:
(8)Px,yH=N−xR0,y+x∙RN+1,0,(9)Px,yV=N−yRx,0+y∙R0,N+1,(10)Px,y=Px,yH+Px,yV+N≫log2N+1.(2) DC Mode.
(11)DCAverage=∑x=1NRx,0+∑x=1NR0,y+N≫log2N+1,(12)P1,1=R1,0+R0,1+2∙DCAverage+2≫2,(13)Px,1=Rx,0+3∙DCAverage+2≫2,(14)P1,y=R0,y+3∙DCAverage+2≫2,(15)Px,y=DCAverage.(3) Angle Mode. For angular prediction modes, each mode is offset in either the horizontal or vertical direction. First of all, the reference pixel projection needs to be mapped into a one-dimensional form according to its angle offset value, and then the predicted value of the current pixel is calculated.


## 2.2.1. Get Reference Pixels

As shown in Figure1, the reference pixels of the current pixel block can be divided into five parts: left, lower left, upper left, upper, and upper right. If some reference pixels do not exist, they will be replaced by reference pixels in adjacent areas. When all reference pixels are unavailable, they will be replaced by fixed values. The calculation method is as shown:
(2)R=1≪BitDepth−1.


## 2.2.2. Reference Pixel Filtering

When the pixel position is as shown in Figure1, vertical-right mode is shown in equation (3), horizontal-down mode is shown as equation (4), diagonal-down-right mode is shown in equation (5), and the filtering calculation method is
(3)Fx,0=Rx−1,0+2Rx,0+Rx+1,0+2≫2,(4)F0,y=R0,y−1+2R0,y+R0,y+1+2≫2,(5)F0,0=R0,1+2R0,0+R1,0+2≫2,(6)F2N,0=R2N−1,0+3R2N,0+2≫2,(7)F0,2N=R0,2N−1+3R0,2N+2≫2.


## 2.2.3. Calculate the Predicted Value

The calculation methods of the predicted value are different in different modes. The calculation methods of the three modes are described as follows:(1) Planar Mode. The calculation method of the planar mode predicted pixel is shown:
(8)Px,yH=N−xR0,y+x∙RN+1,0,(9)Px,yV=N−yRx,0+y∙R0,N+1,(10)Px,y=Px,yH+Px,yV+N≫log2N+1.(2) DC Mode.
(11)DCAverage=∑x=1NRx,0+∑x=1NR0,y+N≫log2N+1,(12)P1,1=R1,0+R0,1+2∙DCAverage+2≫2,(13)Px,1=Rx,0+3∙DCAverage+2≫2,(14)P1,y=R0,y+3∙DCAverage+2≫2,(15)Px,y=DCAverage.(3) Angle Mode. For angular prediction modes, each mode is offset in either the horizontal or vertical direction. First of all, the reference pixel projection needs to be mapped into a one-dimensional form according to its angle offset value, and then the predicted value of the current pixel is calculated.


## 3. Visual Saliency

Visual saliency refers to the ability of some elements in the scene to attract people’s visual attention. This ability is due to the fact that the target has some special visual attributes, so it has a strong subjectivity. Visual saliency modeling is the process of quantifying this ability to attract attention.


### 3.1. Characteristics of Visual Saliency

The human visual system can quickly recognize, segment, combine, analyze, and understand objects in the visual scene. The reason why these functions can be performed so efficiently is that the human eye can only process the prominent features in the scene and ignore most of the insignificant background areas. The first stage of vision processing is to transform the scene into various feature representations, such as edges, brightness, angles, colors, and lines. The visual saliency model is based on these characteristics of the visual system [9].


#### 3.1.1. Color Feature

Color is the most obvious contrast feature in an image or video, and areas that are inconsistent with the surrounding colors can easily become attractive places. The common description methods of color feature are directly using local color representation and also using local statistical methods such as color histogram.


#### 3.1.2. Texture Feature

It can represent the changing characteristics of the surface of objects, such as regular decorations on buildings and stripes on animals, which generally have regular distribution. The commonly used methods of texture analysis are the statistical method, structural method, and spectral method.


#### 3.1.3. Shape Feature

The shape of the object, including the edge and contour information of the object, is a very important aspect in the visual feature, and it is a popular research direction, and many edge detection operators have been proposed, such as Sobel operator, Canny operator, and Robert operator. These algorithms have been widely used in image or video processing, but there is still much ways for improvement.


#### 3.1.4. Motion Feature

For video, motion information is more critical information, and in the case of fixed background, moving objects have significant characteristics.


### 3.2. Algorithms of Visual Saliency

The main steps of visual saliency modeling are feature detection, feature comparison to get the visual saliency map, and finally the synthesis of saliency map. Here are some common algorithms:


#### 3.2.1. Saliency Algorithm Based on Image Sparse Representation

This algorithm first converts the format of the input image, divides the converted image of each channel into blocks and performs sparse representation, then calculates the corresponding local saliency and global saliency, and finally synthesizes the saliency map.(1) Format ConversionAccording to equation (16), the image conversion formats
(16)I=R+G+B,RG=R−G,BY=G−R+G/2−minR,G/2.R, G, and B represent the red, green, and blue channels of the input image, respectively, RG and BY represent the red/green and blue/yellow channels, respectively, and I represent the luminance channel.(2) The Sparse RepresentationThe input image is first scaled to2w×2w pixel size, with W being the size of the block. The whole image can be sparsely coded by using the least square shrinkage algorithm.(3) Get the Saliency(17)SLlcxi=1M∑j=1Mρijc,SLcxi=1L∑j=1MSLlcxi,(18)SGlcxi=1N∑j=1Nρijc×11+distxi,xcenter,SGcxi=1L∑j=1MSGlcxi.(4) Synthesizes the Saliency Map(19)SLxi=∑c∈I,RG,BYℵSLcxi,SGxi=∑c∈I,RG,BYℵSGcxi.


#### 3.2.2. Visual Saliency Detection Algorithm Based on Self-Similarity [10]

In this algorithm, the Local Steering Kernel (LSK) of each pixel in the image is used to calculate its local regression kernel matrix, and then the matrix cosine similarity between the current pixel and its surrounding pixels is calculated to obtain the saliency of the current pixel, and finally the saliency map is synthesized.(1) Get the Saliency(20)Si=Pryi=1F,(21)Si=Pryi=1F=pFyi=1Pryi=1pF,(22)Si=p^Fyi=1=GiFi¯−Fi¯∑j=1NGiFi¯−Fj¯,(23)GiFi¯−Fj¯=exp−Fi¯−Fj¯F22σ2=exp−1+ρFi,Fjσ2,(24)ρi=∑l=1Lρfil,fjlfilfjlFiFFjF,(25)Si=1∑j=1Nexp−1+ρFi,Fj/σ2.(2) Process Color ImagesThe algorithm does not directly extract the saliency from each color channel and combine them but defines each color channelC1, C2, and C3 as a different feature matrix Fic1, Fic2, and Fic3, and then defines Fi=Fic1,Fic2,Fic3. Finally, the matrix cosine similarity calculation is adopted.


#### 3.2.3. Detection Algorithm of Human Visual Attention Points

The algorithm gets the saliency map of the image through the following calculation. The saliency map is a gray map between 0 and 255, which represents the probability of each pixel becoming a visual attention point.(1) Feature ExtractionThe image is subsampled to generate three color components and to obtain a multiscale color featureFCi and a multiscale local operating kernel feature FSi.(2) Get the Saliency(26)pFCi=∑j=1MkFCi−FCj∑i=1M∑j=1MkFCi−FCj,(27)pFCiyi=1=1∑j=1NkFCi−FCj,(28)Si=1pFi⋅pFiyi=1=∑i=1M∑j=1MkFCi−FCj∑j=1MkFCi−FCj⋅1∑j=1NkFCi−FCj.(3) Synthesizes the Saliency Map


### 3.3. Application of Visual Saliency in Video Coding

The theories and methods of visual saliency are not only applicable to static images but also to video processing. The following will introduce several application algorithms of visual saliency in video.


#### 3.3.1. Video Coding Algorithm Based on Spatiotemporal Saliency Map [11]

Firstly, the absolute feature difference of the video is calculated by the Gaussian function, and then multiple spatial saliency maps are established, and then they are combined into a global spatial saliency map, and finally the temporal visual saliency map is obtained by using the motion vector of the foreground. The spatiotemporal visual saliency map can be obtained by combining the temporal and spatial visual saliency maps.


#### 3.3.2. Region-of-Interest-Based Video Coding Algorithm [12]

The video coding algorithm introduced in this algorithm is based on Region of Interest (ROI). High quality coding is used for regions of interest, and conversely, less saliency is used for regions of no interest. First, a saliency calculation was performed based on Itti’s visual saliency model [9]. Then, the saliency factor is added to the rate-distortion optimization. And the value of saliency can be flexibly adjusted according to the degree of interest. Then, the complexity of the algorithm can be reduced by reusing some information in the video coding process. Finally, a comparison with other related perceptual coding algorithms is made.


#### 3.3.3. Video Coding Complexity Reduction Algorithm Based on Visual Saliency [13]

Firstly, the algorithm analyzes the complexity of video coding based on two kinds of relationships: one is the relationship between the maximum coding depth and complexity, and the other is the relationship between the maximum coding depth and distortion. Then it uses visual saliency to predict the coding depth, thus reducing the amount of computation for traversal in prediction.


## 3.1. Characteristics of Visual Saliency

The human visual system can quickly recognize, segment, combine, analyze, and understand objects in the visual scene. The reason why these functions can be performed so efficiently is that the human eye can only process the prominent features in the scene and ignore most of the insignificant background areas. The first stage of vision processing is to transform the scene into various feature representations, such as edges, brightness, angles, colors, and lines. The visual saliency model is based on these characteristics of the visual system [9].


### 3.1.1. Color Feature

Color is the most obvious contrast feature in an image or video, and areas that are inconsistent with the surrounding colors can easily become attractive places. The common description methods of color feature are directly using local color representation and also using local statistical methods such as color histogram.


### 3.1.2. Texture Feature

It can represent the changing characteristics of the surface of objects, such as regular decorations on buildings and stripes on animals, which generally have regular distribution. The commonly used methods of texture analysis are the statistical method, structural method, and spectral method.


### 3.1.3. Shape Feature

The shape of the object, including the edge and contour information of the object, is a very important aspect in the visual feature, and it is a popular research direction, and many edge detection operators have been proposed, such as Sobel operator, Canny operator, and Robert operator. These algorithms have been widely used in image or video processing, but there is still much ways for improvement.


### 3.1.4. Motion Feature

For video, motion information is more critical information, and in the case of fixed background, moving objects have significant characteristics.


## 3.1.1. Color Feature

Color is the most obvious contrast feature in an image or video, and areas that are inconsistent with the surrounding colors can easily become attractive places. The common description methods of color feature are directly using local color representation and also using local statistical methods such as color histogram.


## 3.1.2. Texture Feature

It can represent the changing characteristics of the surface of objects, such as regular decorations on buildings and stripes on animals, which generally have regular distribution. The commonly used methods of texture analysis are the statistical method, structural method, and spectral method.


## 3.1.3. Shape Feature

The shape of the object, including the edge and contour information of the object, is a very important aspect in the visual feature, and it is a popular research direction, and many edge detection operators have been proposed, such as Sobel operator, Canny operator, and Robert operator. These algorithms have been widely used in image or video processing, but there is still much ways for improvement.


## 3.1.4. Motion Feature

For video, motion information is more critical information, and in the case of fixed background, moving objects have significant characteristics.


## 3.2. Algorithms of Visual Saliency

The main steps of visual saliency modeling are feature detection, feature comparison to get the visual saliency map, and finally the synthesis of saliency map. Here are some common algorithms:


### 3.2.1. Saliency Algorithm Based on Image Sparse Representation

This algorithm first converts the format of the input image, divides the converted image of each channel into blocks and performs sparse representation, then calculates the corresponding local saliency and global saliency, and finally synthesizes the saliency map.(1) Format ConversionAccording to equation (16), the image conversion formats
(16)I=R+G+B,RG=R−G,BY=G−R+G/2−minR,G/2.R, G, and B represent the red, green, and blue channels of the input image, respectively, RG and BY represent the red/green and blue/yellow channels, respectively, and I represent the luminance channel.(2) The Sparse RepresentationThe input image is first scaled to2w×2w pixel size, with W being the size of the block. The whole image can be sparsely coded by using the least square shrinkage algorithm.(3) Get the Saliency(17)SLlcxi=1M∑j=1Mρijc,SLcxi=1L∑j=1MSLlcxi,(18)SGlcxi=1N∑j=1Nρijc×11+distxi,xcenter,SGcxi=1L∑j=1MSGlcxi.(4) Synthesizes the Saliency Map(19)SLxi=∑c∈I,RG,BYℵSLcxi,SGxi=∑c∈I,RG,BYℵSGcxi.


### 3.2.2. Visual Saliency Detection Algorithm Based on Self-Similarity [10]

In this algorithm, the Local Steering Kernel (LSK) of each pixel in the image is used to calculate its local regression kernel matrix, and then the matrix cosine similarity between the current pixel and its surrounding pixels is calculated to obtain the saliency of the current pixel, and finally the saliency map is synthesized.(1) Get the Saliency(20)Si=Pryi=1F,(21)Si=Pryi=1F=pFyi=1Pryi=1pF,(22)Si=p^Fyi=1=GiFi¯−Fi¯∑j=1NGiFi¯−Fj¯,(23)GiFi¯−Fj¯=exp−Fi¯−Fj¯F22σ2=exp−1+ρFi,Fjσ2,(24)ρi=∑l=1Lρfil,fjlfilfjlFiFFjF,(25)Si=1∑j=1Nexp−1+ρFi,Fj/σ2.(2) Process Color ImagesThe algorithm does not directly extract the saliency from each color channel and combine them but defines each color channelC1, C2, and C3 as a different feature matrix Fic1, Fic2, and Fic3, and then defines Fi=Fic1,Fic2,Fic3. Finally, the matrix cosine similarity calculation is adopted.


### 3.2.3. Detection Algorithm of Human Visual Attention Points

The algorithm gets the saliency map of the image through the following calculation. The saliency map is a gray map between 0 and 255, which represents the probability of each pixel becoming a visual attention point.(1) Feature ExtractionThe image is subsampled to generate three color components and to obtain a multiscale color featureFCi and a multiscale local operating kernel feature FSi.(2) Get the Saliency(26)pFCi=∑j=1MkFCi−FCj∑i=1M∑j=1MkFCi−FCj,(27)pFCiyi=1=1∑j=1NkFCi−FCj,(28)Si=1pFi⋅pFiyi=1=∑i=1M∑j=1MkFCi−FCj∑j=1MkFCi−FCj⋅1∑j=1NkFCi−FCj.(3) Synthesizes the Saliency Map


## 3.2.1. Saliency Algorithm Based on Image Sparse Representation

This algorithm first converts the format of the input image, divides the converted image of each channel into blocks and performs sparse representation, then calculates the corresponding local saliency and global saliency, and finally synthesizes the saliency map.(1) Format ConversionAccording to equation (16), the image conversion formats
(16)I=R+G+B,RG=R−G,BY=G−R+G/2−minR,G/2.R, G, and B represent the red, green, and blue channels of the input image, respectively, RG and BY represent the red/green and blue/yellow channels, respectively, and I represent the luminance channel.(2) The Sparse RepresentationThe input image is first scaled to2w×2w pixel size, with W being the size of the block. The whole image can be sparsely coded by using the least square shrinkage algorithm.(3) Get the Saliency(17)SLlcxi=1M∑j=1Mρijc,SLcxi=1L∑j=1MSLlcxi,(18)SGlcxi=1N∑j=1Nρijc×11+distxi,xcenter,SGcxi=1L∑j=1MSGlcxi.(4) Synthesizes the Saliency Map(19)SLxi=∑c∈I,RG,BYℵSLcxi,SGxi=∑c∈I,RG,BYℵSGcxi.


## 3.2.2. Visual Saliency Detection Algorithm Based on Self-Similarity [10]

In this algorithm, the Local Steering Kernel (LSK) of each pixel in the image is used to calculate its local regression kernel matrix, and then the matrix cosine similarity between the current pixel and its surrounding pixels is calculated to obtain the saliency of the current pixel, and finally the saliency map is synthesized.(1) Get the Saliency(20)Si=Pryi=1F,(21)Si=Pryi=1F=pFyi=1Pryi=1pF,(22)Si=p^Fyi=1=GiFi¯−Fi¯∑j=1NGiFi¯−Fj¯,(23)GiFi¯−Fj¯=exp−Fi¯−Fj¯F22σ2=exp−1+ρFi,Fjσ2,(24)ρi=∑l=1Lρfil,fjlfilfjlFiFFjF,(25)Si=1∑j=1Nexp−1+ρFi,Fj/σ2.(2) Process Color ImagesThe algorithm does not directly extract the saliency from each color channel and combine them but defines each color channelC1, C2, and C3 as a different feature matrix Fic1, Fic2, and Fic3, and then defines Fi=Fic1,Fic2,Fic3. Finally, the matrix cosine similarity calculation is adopted.


## 3.2.3. Detection Algorithm of Human Visual Attention Points

The algorithm gets the saliency map of the image through the following calculation. The saliency map is a gray map between 0 and 255, which represents the probability of each pixel becoming a visual attention point.(1) Feature ExtractionThe image is subsampled to generate three color components and to obtain a multiscale color featureFCi and a multiscale local operating kernel feature FSi.(2) Get the Saliency(26)pFCi=∑j=1MkFCi−FCj∑i=1M∑j=1MkFCi−FCj,(27)pFCiyi=1=1∑j=1NkFCi−FCj,(28)Si=1pFi⋅pFiyi=1=∑i=1M∑j=1MkFCi−FCj∑j=1MkFCi−FCj⋅1∑j=1NkFCi−FCj.(3) Synthesizes the Saliency Map


## 3.3. Application of Visual Saliency in Video Coding

The theories and methods of visual saliency are not only applicable to static images but also to video processing. The following will introduce several application algorithms of visual saliency in video.


### 3.3.1. Video Coding Algorithm Based on Spatiotemporal Saliency Map [11]

Firstly, the absolute feature difference of the video is calculated by the Gaussian function, and then multiple spatial saliency maps are established, and then they are combined into a global spatial saliency map, and finally the temporal visual saliency map is obtained by using the motion vector of the foreground. The spatiotemporal visual saliency map can be obtained by combining the temporal and spatial visual saliency maps.


### 3.3.2. Region-of-Interest-Based Video Coding Algorithm [12]

The video coding algorithm introduced in this algorithm is based on Region of Interest (ROI). High quality coding is used for regions of interest, and conversely, less saliency is used for regions of no interest. First, a saliency calculation was performed based on Itti’s visual saliency model [9]. Then, the saliency factor is added to the rate-distortion optimization. And the value of saliency can be flexibly adjusted according to the degree of interest. Then, the complexity of the algorithm can be reduced by reusing some information in the video coding process. Finally, a comparison with other related perceptual coding algorithms is made.


### 3.3.3. Video Coding Complexity Reduction Algorithm Based on Visual Saliency [13]

Firstly, the algorithm analyzes the complexity of video coding based on two kinds of relationships: one is the relationship between the maximum coding depth and complexity, and the other is the relationship between the maximum coding depth and distortion. Then it uses visual saliency to predict the coding depth, thus reducing the amount of computation for traversal in prediction.


## 3.3.1. Video Coding Algorithm Based on Spatiotemporal Saliency Map [11]

Firstly, the absolute feature difference of the video is calculated by the Gaussian function, and then multiple spatial saliency maps are established, and then they are combined into a global spatial saliency map, and finally the temporal visual saliency map is obtained by using the motion vector of the foreground. The spatiotemporal visual saliency map can be obtained by combining the temporal and spatial visual saliency maps.


## 3.3.2. Region-of-Interest-Based Video Coding Algorithm [12]

The video coding algorithm introduced in this algorithm is based on Region of Interest (ROI). High quality coding is used for regions of interest, and conversely, less saliency is used for regions of no interest. First, a saliency calculation was performed based on Itti’s visual saliency model [9]. Then, the saliency factor is added to the rate-distortion optimization. And the value of saliency can be flexibly adjusted according to the degree of interest. Then, the complexity of the algorithm can be reduced by reusing some information in the video coding process. Finally, a comparison with other related perceptual coding algorithms is made.


## 3.3.3. Video Coding Complexity Reduction Algorithm Based on Visual Saliency [13]

Firstly, the algorithm analyzes the complexity of video coding based on two kinds of relationships: one is the relationship between the maximum coding depth and complexity, and the other is the relationship between the maximum coding depth and distortion. Then it uses visual saliency to predict the coding depth, thus reducing the amount of computation for traversal in prediction.


## 4. A Complexity Control Algorithm for Intraprediction Based on Visual Saliency

In this section, the proportion of coding units of different depths in the optimal mode of intraprediction of the HM-13.0 standard algorithm is analyzed; then, the proportion of the coding time of each depth CU in the whole video coding time is counted, respectively. According to the relationship between the division of coding units in intraprediction and visual saliency, an intraprediction complexity control algorithm based on visual saliency is proposed.


### 4.1. Proportion of each Depth Coding Unit

Correa et al. perform complexity control by dividing each frame in the video into unlimited frames and limited frames. For unlimited frames, it is necessary to perform encoding according to a normal unit division process, and the rate-distortion cost of the encoding unit CU of each depth is calculated in a mode by mode and depth by depth traversal manner. Then the best intraprediction mode is selected by comparing the rate-distortion cost. For limited frames, the maximum CU depth of the limited frames is determined according to the previously encoded unlimited frame, and traversal of all modes and all depths is not required, so that the encoding complexity can be greatly reduced to achieve the purpose of controlling the coding complexity by setting the number of the limited frames, which is based on the requirements of the application [14].The algorithm will alternately arrange two kinds of frames according to the limitation of target complexity and the actual encoding requirements, as shown in Figure 4.Figure 4
Angle prediction mode.Jimenez Moreno et al. proposed an efficient Complexity Control (CC) algorithm [15], which is based on the hierarchical structure of coding units. In the process of video coding, the current coding unit needs to be further divided by considering whether the actual coding time can meet the needs of the target task; the algorithm defines the early termination condition for each size of CU. More importantly, all the parameters can be dynamically generated according to the content of the video, the configuration of the encoding, and the setting of the target complexity, so that the complexity can be effectively controlled. The above algorithm can effectively control the complexity. And algorithm in [14] based on frames. For different video sequences, each frame has its own characteristics, so it should be studied on a smaller coding unit. In [15], the reduction of coding complexity is considered from the perspective of coding time and rate-distortion cost. The visual characteristics of the human eyes are not fully utilized. This paper will analyze the relationship between visual saliency and video coding complexity and propose related algorithms. Since the 64×64 largest coding unit (LCU) in HEVC is the basic unit for video coding, the characteristics of the LCU will be analyzed first. For the quadtree structure, the rate-distortion cost of coding units before and after the division is also compared during intraprediction to determine whether the division is required, as shown in Figure 5. It is required that for each size of coding unit (85 coding units ranging in size from 64×64 to 8×8), and each intraprediction mode of each coding unit (there are 35 prediction modes in each coding unit) is adopted to rate-distortion cost calculation, so the prediction coding of intra- and interframes is the most computationally complex module in video coding. This process is recursive traversal, regardless of the specific characteristics of the video sequence.Figure 5
Quadtree structure.Select a video sequence Cactus with a resolution of1920×1080 and analyze the code stream file formed after compression, then count the depth information of the optimal mode selection as shown in Figure 6. It can be seen that only about 30% of the coding units need to be divided into the maximum depth, while 70% of the coding units do not need to recurse to the maximum depth, and nearly 35% of the optimal modes have a depth of 0 or 1. Therefore, using the algorithms in Figure 6 may result in a large amount of unnecessary computation, which may reduce the overall efficiency of video coding.Figure 6
Percentage of different depth in the best mode.


### 4.2. Characteristics of Visual Saliency

Select test sequence Cactus with a resolution of1920×1080 and count the percentage of the total time for encoding CUs of different depths in the encoding time of the whole intraprediction, as Figure 7 shows. It can be seen that the CU with a coding depth of 3 consumes more time, which is close to half of the total time of intraprediction coding, and the CU with a coding depth of 0 uses less than one tenth.Figure 7
Percentage of different depth in the intraprediction.


### 4.3. Relationship between Coding Unit Depth and Visual Saliency

Figure8 is a certain frame in the video sequence of basketball pass of Class D with 416×240 resolution and its coding unit division by the HM standard algorithm. It can be seen that the area with rich motion or texture in the video has a greater depth. On the contrary, the relatively still or flat area has a smaller depth. Basketball players and basketball in the video are split into smaller units, while most of the static background, such as the ground and walls, are split into 32×32 coding units, and some even maintained 64×64 coding units.Figure 8
Division of coding units.Then generate the visual saliency map of the sequence by using the visual saliency algorithm introduced in reference [10], extract the same frame as Figure 8 to obtain its visual saliency map, and retain the regions with greater saliency. It can be seen from the figure that the greater the brightness of the region, the higher the saliency value, and the more it can highlight the subjective gaze target of the human eyes. From the unit division result it can be seen that the areas with high visual saliency are basically consistent with the areas with small unit division depth. As previously described, the visual characteristics of the human eyes determine that they will pay attention to some regions when watching a video, but they are not sensitive to the changes of the areas they do not pay attention to. Therefore, visual saliency can be combined with the coding process of video to reduce the complexity of video coding Figure 9.Figure 9
Division and visual saliency of basketball pass.


### 4.4. Establishment of Model

Through the above analysis, we combine the visual saliency algorithm with the mode selection algorithm to determine the depth of the best CU, according to the saliency of the frames. If the saliency value is high, the depth of CU is larger, and the optimal mode is selected by traversing and calculating the rate-distortion cost. On the contrary, when the saliency value is low, the division with greater depth will be skipped. A larger coding module is directly adopted, so as to reduce the complexity of video coding. Firstly, the visual saliency algorithm introduced in [10] is adopted to build a saliency model and generate a visual saliency map of the video sequence. Next, threshold value 1 will be set, and when the visual saliency of the coding unit with the depth of 0 is greater than the threshold value 1, the current coding unit is split; otherwise, the depth of the current CU is 0, and the rest of the partitioning and rate-distortion cost comparisons are skipped. Threshold 2 is set in the same manner, and it is determined whether or not the 32×32 CU needs to be split. For the test sequence Cactus, the threshold values under different complexities can be obtained by counting the division results after the test. According to the statistical and the experimental results of the test sequence, the threshold is set by different targets, as shown in Table 2.Table 2
Threshold of different targets.
TargetThreshold 1Threshold 280%0.120.1460%0.080.1140%0.040.09


### 4.5. Characteristics of Visual Saliency

The division process by using the algorithm is shown in Figure10. The dotted line represents the division operation that needs to be determined according to the threshold. If it is greater than the threshold, the division represented by the dotted line is executed. Otherwise, the partition operation is not performed.Figure 10
CU of the proposed algorithm.Figure11 shows the process flow of the algorithm during intraprediction coding, and the detailed description is as follows: (1) firstly, a saliency model of a video sequence is established, the saliency value of each pixel is obtained, and a visual saliency map is formed; (2) a mode selection process of intraprediction for a 64×64 CU, including a DC mode, a planar mode, and 33 angle prediction modes, to obtain an optimal mode of 64×64; (3) calculate an average saliency value of the 64×64 pixels and compare the average saliency value with the threshold value 1, if the average saliency value is greater than the threshold value 1, increase the depth by 1, and go to next step; otherwise, the remaining comparison and calculation steps are skipped, the division of the current CU is terminated in advance, and the depth of the optimal mode of the CU is 0; (4) a mode selection process of intraprediction for a 32×32 CU, including a DC mode, a planar mode, and 33 angle prediction modes, to obtain an optimal 32×32 mode; (5) calculate an average saliency value of 32×32 pixels and compare the average saliency value with the threshold value 2, if the average saliency value is greater than the threshold value 2, increase the depth by 1, and go to next step; otherwise, the remaining comparison and calculation steps are skipped, the division of the current CU is terminated in advance, and the depth of the optimal mode of the CU is 1; (6) a mode selection process of intraprediction for a 16×16 CU, including a DC mode, a planar mode, and 33 angle prediction modes, to obtain a 32×32 optimal mode; (7) an intraprediction mode selection process is performed on an 8×8 CU, including a DC mode, a planar mode, and 33 angle prediction modes, and an 8×8 optimal mode is obtained; (8) calculate and compare the rate-distortion cost of the 64∗64 CU and the sum of the rate-distortion cost of four 32∗32 sub-CUs, the rate-distortion costs of the 32∗32 CU and the sum of the rate-distortion costs of four 16∗16 CU sub-CUs, and the rate-distortion costs of the 16∗16 CU and four 8∗8 sub-CUs to obtain an optimal mode of a current CU; (9) The next 64×64 CU is processed.Figure 11
Division process of the proposed algorithm.


## 4.1. Proportion of each Depth Coding Unit

Correa et al. perform complexity control by dividing each frame in the video into unlimited frames and limited frames. For unlimited frames, it is necessary to perform encoding according to a normal unit division process, and the rate-distortion cost of the encoding unit CU of each depth is calculated in a mode by mode and depth by depth traversal manner. Then the best intraprediction mode is selected by comparing the rate-distortion cost. For limited frames, the maximum CU depth of the limited frames is determined according to the previously encoded unlimited frame, and traversal of all modes and all depths is not required, so that the encoding complexity can be greatly reduced to achieve the purpose of controlling the coding complexity by setting the number of the limited frames, which is based on the requirements of the application [14].The algorithm will alternately arrange two kinds of frames according to the limitation of target complexity and the actual encoding requirements, as shown in Figure 4.Figure 4
Angle prediction mode.Jimenez Moreno et al. proposed an efficient Complexity Control (CC) algorithm [15], which is based on the hierarchical structure of coding units. In the process of video coding, the current coding unit needs to be further divided by considering whether the actual coding time can meet the needs of the target task; the algorithm defines the early termination condition for each size of CU. More importantly, all the parameters can be dynamically generated according to the content of the video, the configuration of the encoding, and the setting of the target complexity, so that the complexity can be effectively controlled. The above algorithm can effectively control the complexity. And algorithm in [14] based on frames. For different video sequences, each frame has its own characteristics, so it should be studied on a smaller coding unit. In [15], the reduction of coding complexity is considered from the perspective of coding time and rate-distortion cost. The visual characteristics of the human eyes are not fully utilized. This paper will analyze the relationship between visual saliency and video coding complexity and propose related algorithms. Since the 64×64 largest coding unit (LCU) in HEVC is the basic unit for video coding, the characteristics of the LCU will be analyzed first. For the quadtree structure, the rate-distortion cost of coding units before and after the division is also compared during intraprediction to determine whether the division is required, as shown in Figure 5. It is required that for each size of coding unit (85 coding units ranging in size from 64×64 to 8×8), and each intraprediction mode of each coding unit (there are 35 prediction modes in each coding unit) is adopted to rate-distortion cost calculation, so the prediction coding of intra- and interframes is the most computationally complex module in video coding. This process is recursive traversal, regardless of the specific characteristics of the video sequence.Figure 5
Quadtree structure.Select a video sequence Cactus with a resolution of1920×1080 and analyze the code stream file formed after compression, then count the depth information of the optimal mode selection as shown in Figure 6. It can be seen that only about 30% of the coding units need to be divided into the maximum depth, while 70% of the coding units do not need to recurse to the maximum depth, and nearly 35% of the optimal modes have a depth of 0 or 1. Therefore, using the algorithms in Figure 6 may result in a large amount of unnecessary computation, which may reduce the overall efficiency of video coding.Figure 6
Percentage of different depth in the best mode.


## 4.2. Characteristics of Visual Saliency

Select test sequence Cactus with a resolution of1920×1080 and count the percentage of the total time for encoding CUs of different depths in the encoding time of the whole intraprediction, as Figure 7 shows. It can be seen that the CU with a coding depth of 3 consumes more time, which is close to half of the total time of intraprediction coding, and the CU with a coding depth of 0 uses less than one tenth.Figure 7
Percentage of different depth in the intraprediction.


## 4.3. Relationship between Coding Unit Depth and Visual Saliency

Figure8 is a certain frame in the video sequence of basketball pass of Class D with 416×240 resolution and its coding unit division by the HM standard algorithm. It can be seen that the area with rich motion or texture in the video has a greater depth. On the contrary, the relatively still or flat area has a smaller depth. Basketball players and basketball in the video are split into smaller units, while most of the static background, such as the ground and walls, are split into 32×32 coding units, and some even maintained 64×64 coding units.Figure 8
Division of coding units.Then generate the visual saliency map of the sequence by using the visual saliency algorithm introduced in reference [10], extract the same frame as Figure 8 to obtain its visual saliency map, and retain the regions with greater saliency. It can be seen from the figure that the greater the brightness of the region, the higher the saliency value, and the more it can highlight the subjective gaze target of the human eyes. From the unit division result it can be seen that the areas with high visual saliency are basically consistent with the areas with small unit division depth. As previously described, the visual characteristics of the human eyes determine that they will pay attention to some regions when watching a video, but they are not sensitive to the changes of the areas they do not pay attention to. Therefore, visual saliency can be combined with the coding process of video to reduce the complexity of video coding Figure 9.Figure 9
Division and visual saliency of basketball pass.


## 4.4. Establishment of Model

Through the above analysis, we combine the visual saliency algorithm with the mode selection algorithm to determine the depth of the best CU, according to the saliency of the frames. If the saliency value is high, the depth of CU is larger, and the optimal mode is selected by traversing and calculating the rate-distortion cost. On the contrary, when the saliency value is low, the division with greater depth will be skipped. A larger coding module is directly adopted, so as to reduce the complexity of video coding. Firstly, the visual saliency algorithm introduced in [10] is adopted to build a saliency model and generate a visual saliency map of the video sequence. Next, threshold value 1 will be set, and when the visual saliency of the coding unit with the depth of 0 is greater than the threshold value 1, the current coding unit is split; otherwise, the depth of the current CU is 0, and the rest of the partitioning and rate-distortion cost comparisons are skipped. Threshold 2 is set in the same manner, and it is determined whether or not the 32×32 CU needs to be split. For the test sequence Cactus, the threshold values under different complexities can be obtained by counting the division results after the test. According to the statistical and the experimental results of the test sequence, the threshold is set by different targets, as shown in Table 2.Table 2
Threshold of different targets.
TargetThreshold 1Threshold 280%0.120.1460%0.080.1140%0.040.09


## 4.5. Characteristics of Visual Saliency

The division process by using the algorithm is shown in Figure10. The dotted line represents the division operation that needs to be determined according to the threshold. If it is greater than the threshold, the division represented by the dotted line is executed. Otherwise, the partition operation is not performed.Figure 10
CU of the proposed algorithm.Figure11 shows the process flow of the algorithm during intraprediction coding, and the detailed description is as follows: (1) firstly, a saliency model of a video sequence is established, the saliency value of each pixel is obtained, and a visual saliency map is formed; (2) a mode selection process of intraprediction for a 64×64 CU, including a DC mode, a planar mode, and 33 angle prediction modes, to obtain an optimal mode of 64×64; (3) calculate an average saliency value of the 64×64 pixels and compare the average saliency value with the threshold value 1, if the average saliency value is greater than the threshold value 1, increase the depth by 1, and go to next step; otherwise, the remaining comparison and calculation steps are skipped, the division of the current CU is terminated in advance, and the depth of the optimal mode of the CU is 0; (4) a mode selection process of intraprediction for a 32×32 CU, including a DC mode, a planar mode, and 33 angle prediction modes, to obtain an optimal 32×32 mode; (5) calculate an average saliency value of 32×32 pixels and compare the average saliency value with the threshold value 2, if the average saliency value is greater than the threshold value 2, increase the depth by 1, and go to next step; otherwise, the remaining comparison and calculation steps are skipped, the division of the current CU is terminated in advance, and the depth of the optimal mode of the CU is 1; (6) a mode selection process of intraprediction for a 16×16 CU, including a DC mode, a planar mode, and 33 angle prediction modes, to obtain a 32×32 optimal mode; (7) an intraprediction mode selection process is performed on an 8×8 CU, including a DC mode, a planar mode, and 33 angle prediction modes, and an 8×8 optimal mode is obtained; (8) calculate and compare the rate-distortion cost of the 64∗64 CU and the sum of the rate-distortion cost of four 32∗32 sub-CUs, the rate-distortion costs of the 32∗32 CU and the sum of the rate-distortion costs of four 16∗16 CU sub-CUs, and the rate-distortion costs of the 16∗16 CU and four 8∗8 sub-CUs to obtain an optimal mode of a current CU; (9) The next 64×64 CU is processed.Figure 11
Division process of the proposed algorithm.


## 5. Experimental Results

In order to analyze the coding performance of the algorithm, the following settings are made in the simulation experiment:(1)
the standard test platform HM-13.0 is used as a benchmark for results comparison(2)
3 standard test sequences, BasketballDrive, BQTerrace, and Cactus, with a resolution of1920×1080 in Class B were used(3)
use encoder _ intra _ main as the configuration file(4)
The objective evaluation criteria were BitRate, PSNR, and coding time(5)
The percentage of the target encoding timeTR to the standard HM-13.0 encoding time is defined as the target complexity CR:(29)CR=TRTHM×100%.(6)
The actual encoding timeTT as a percentage of that standard HM-13. 0 encoding time is defined as the encoding complexity CT:(30)CT=TTTHM×100%.


### 5.1. Objective Quality Assessment

Table3 shows the test results of the algorithm on the HM-13.0 for three standard test sequences of Class B, where encoder_intra_main is used in the configuration mode, and the quantization parameters QP are set to 22, 27, 32, and 37, respectively. Under the given target complexity control, BitRate and Y-PSNR are listed, respectively. The performance of the proposed algorithm is measured with luma PSNR (represented with Y-PSNR) decrease and bitrate (represented with BitRate) increase.Table 3
Result comparing.
LevelQPBasketballDriveCactusBQTerraceY-PSNRBitRateY-PSNRBitRateY-PSNRBitRate100%2241.1149695.5040.57110334.2542.71179577.502739.2921537.3637.8551444.3037.7994493.493237.8211759.6235.5528035.9334.5151367.873735.976697.4633.0615195.2631.5328204.9280%2241.0949859.4140.54110645.6242.65179351.312739.2921831.2037.8351957.9137.7894526.073237.8211921.2635.5328354.6934.5051394.483735.966780.5833.0315327.0131.5328297.7760%2241.0450016.3140.47111627.6342.43182025.372739.2722303.8037.7853099.2537.6997012.683237.8012215.6335.4729018.2834.4052942.213735.946935.8732.9715638.8531.4329071.1140%2240.9950713.2240.40113338.0242.24188418.642739.2423108.0537.7254625.0437.60102796.433237.7612742.735.3929960.6834.2356371.283735.887232.8632.8816117.0731.2330812.71Tables4–7 show the test results of the algorithm on the HM-13.0 for three standard test sequences in Class B. The encoder_intra_main is used in the configuration mode, and the quantization parameter QP is set to 22, 27, 32, and 37, respectively. The percentage of the actual encoding time in the encoding time of the HM-13.0 standard algorithm, when the target complexity control is set, is executed. The experimental results of this algorithm are compared with those of the algorithm described in reference [14] by the change of BitRate and PSNR. The better algorithm will cause less increase of BitRate and less decrease of PSNR. From these tables, the proposed algorithm is better than reference [14] with less PSNR decrease and bitrate increase, while maintaining the same coding time.Table 4
BasketballDrive.
Coding time (%) [14]ΔBitRate (%) [14]ΔPSNR (dB) [14]Coding time (%)proposedΔBitRate (%)proposedΔPSNR (dB) proposed75%+0.2-0.0377%+0.3-0.0259%+1.2-0.2156%+0.7-0.0637%+4.9-0.7739%+2.1-0.12Table 5
Cactus.
Coding time (%) [14]ΔBitRate (%) [14]ΔPSNR (dB) [14]Coding time (%) proposedΔBitRate (%) proposedΔPSNR (dB) proposed76%+0.1-0.0379%+0.3-0.0361%+0.5-0.0759%+1.2-0.1141%+5.7-0.2842%+2.7-0.17Table 6
BQTerrace.
Coding time (%) [14]ΔBitRate (%) [14]ΔPSNR (dB) [14]Coding time (%)proposedΔBitRate (%)proposedΔPSNR (dB) proposed72%+1.8-0.0585%+0.1-0.0667%+2.1-0.0566%+1.4-0.2843%+2.8-0.1440%+4.9-0.48Table 7
Average of BD-rate and BD-PSNR.
LevelBD-rate(%) [14]BD-PSNR(dB) [14]BD-rate(%) proposedBD-PSNR (dB)proposed80%+1.279-0.065+1.139-0.03960%+5.155-0.223+4.707-0.19240%+11.944-0.489+11.073-0.455Tables8–10 list the experimental results of this algorithm compared with the algorithm in [15] when the target complexity is set to about 80% and 70%, respectively. From these tables, the proposed algorithm is better than reference [15] with less PSNR decrease and bitrate increase, while maintaining the same coding time.Table 8
BasketballDrive.
Coding time (%) [15]ΔBitRate (%) [15]ΔPSNR (dB) [15]Coding time (%) proposedΔBitRate (%) proposedΔPSNR (dB) proposed83%+0.3-0.0177%+0.3-0.0274%+0.8-0.0269%+0.5-0.04Table 9
Cactus.
Coding time (%) [15]ΔBitRate (%) [15]ΔPSNR (dB) [15]Coding time (%) proposedΔBitRate (%) proposedΔPSNR (dB)proposed85%+0.2-0.0179%+0.3-0.0375%+0.8-0.0342%+0.7-0.07Table 10
BQTerrace.
Coding time (%) [15]ΔBitRate (%) [15]ΔPSNR (dB) [15]Coding time (%) proposedΔBitRate (%) proposedΔPSNR (dB) proposed80%+0.6-0.0285%+0.1-0.0643%+1.8-0.0671%+0.9-0.18Figure12 is R-D curves of cactus. It can be seen from the figure that the algorithm in this paper can accurately control the intracoding complexity without greatly affecting the coding performance, which are very close to the original R-D curves. The R-D curve with the target complexity of 80% is basically coincident with the HM standard algorithm and slightly deviates from the HM standard algorithm at 60% and 40%.Figure 12
R-D curves of Cactus.


### 5.2. Subjective Quality Assessment

Figure13(a) is a frame in the test video sequence BasketballDrive (1920×1080), and Figure 13(b) is its corresponding saliency map. Figures 13(c) and 13(d), respectively, show the effect comparison after the algorithm is used to divide the unit of a certain frame when the target time is set to 100%, 80%, 60%, and 40%. Figures 13(g) and 13(h) are the result of a certain frame in a test video sequence by using the algorithm for a target time of 80% and 40%, respectively. It can be seen from Figure 13(b) that in this video sequence, the human eyes mainly focus on the basketball players, but do not pay attention to the floor and walls. Therefore, the algorithm strengthens the coding and division of the regions with high visual saliency, simplifies the division of the regions with low visual saliency, and according to different target complexities. This simplified process is controlled by different thresholds, as shown in Figures 13(c) and 13(d). It can be seen from the final coding effect Figure 13 that the algorithm can effectively control the coding complexity, and at the same time, there are almost no obvious visual differences.Figure 13
CU of BasketballDrive.
(a)
Frame of video(b)
Saliency map(c)
100%(d)
80%(e)
60%(f)
40%(g)
80%(h)
40%It can be seen from these figures that the depth of the coding unit is relatively large and the division is relatively detailed in the areas that are easy to be noticed by the human eyes. On the contrary, the depth of the region coding units to which the human eye pays less attention is small. The coding complexity is increased in a large flat area such as a floor, a wall, a river, or an area with less texture. It is not easy to improve the subjective visual quality for the human eyes, while it is helpful to improve the subjective visual quality by using more complex coding for the people and scenery that the human eyes pay attention to and the areas with more complex textures. Therefore, it is entirely possible to reduce the overall encoding complexity by reducing the number of bytes used to encode the regions with lower visual saliency. In this way, the algorithm can use less division to express the video information, so as to achieve the control of coding complexity.


## 5.1. Objective Quality Assessment

Table3 shows the test results of the algorithm on the HM-13.0 for three standard test sequences of Class B, where encoder_intra_main is used in the configuration mode, and the quantization parameters QP are set to 22, 27, 32, and 37, respectively. Under the given target complexity control, BitRate and Y-PSNR are listed, respectively. The performance of the proposed algorithm is measured with luma PSNR (represented with Y-PSNR) decrease and bitrate (represented with BitRate) increase.Table 3
Result comparing.
LevelQPBasketballDriveCactusBQTerraceY-PSNRBitRateY-PSNRBitRateY-PSNRBitRate100%2241.1149695.5040.57110334.2542.71179577.502739.2921537.3637.8551444.3037.7994493.493237.8211759.6235.5528035.9334.5151367.873735.976697.4633.0615195.2631.5328204.9280%2241.0949859.4140.54110645.6242.65179351.312739.2921831.2037.8351957.9137.7894526.073237.8211921.2635.5328354.6934.5051394.483735.966780.5833.0315327.0131.5328297.7760%2241.0450016.3140.47111627.6342.43182025.372739.2722303.8037.7853099.2537.6997012.683237.8012215.6335.4729018.2834.4052942.213735.946935.8732.9715638.8531.4329071.1140%2240.9950713.2240.40113338.0242.24188418.642739.2423108.0537.7254625.0437.60102796.433237.7612742.735.3929960.6834.2356371.283735.887232.8632.8816117.0731.2330812.71Tables4–7 show the test results of the algorithm on the HM-13.0 for three standard test sequences in Class B. The encoder_intra_main is used in the configuration mode, and the quantization parameter QP is set to 22, 27, 32, and 37, respectively. The percentage of the actual encoding time in the encoding time of the HM-13.0 standard algorithm, when the target complexity control is set, is executed. The experimental results of this algorithm are compared with those of the algorithm described in reference [14] by the change of BitRate and PSNR. The better algorithm will cause less increase of BitRate and less decrease of PSNR. From these tables, the proposed algorithm is better than reference [14] with less PSNR decrease and bitrate increase, while maintaining the same coding time.Table 4
BasketballDrive.
Coding time (%) [14]ΔBitRate (%) [14]ΔPSNR (dB) [14]Coding time (%)proposedΔBitRate (%)proposedΔPSNR (dB) proposed75%+0.2-0.0377%+0.3-0.0259%+1.2-0.2156%+0.7-0.0637%+4.9-0.7739%+2.1-0.12Table 5
Cactus.
Coding time (%) [14]ΔBitRate (%) [14]ΔPSNR (dB) [14]Coding time (%) proposedΔBitRate (%) proposedΔPSNR (dB) proposed76%+0.1-0.0379%+0.3-0.0361%+0.5-0.0759%+1.2-0.1141%+5.7-0.2842%+2.7-0.17Table 6
BQTerrace.
Coding time (%) [14]ΔBitRate (%) [14]ΔPSNR (dB) [14]Coding time (%)proposedΔBitRate (%)proposedΔPSNR (dB) proposed72%+1.8-0.0585%+0.1-0.0667%+2.1-0.0566%+1.4-0.2843%+2.8-0.1440%+4.9-0.48Table 7
Average of BD-rate and BD-PSNR.
LevelBD-rate(%) [14]BD-PSNR(dB) [14]BD-rate(%) proposedBD-PSNR (dB)proposed80%+1.279-0.065+1.139-0.03960%+5.155-0.223+4.707-0.19240%+11.944-0.489+11.073-0.455Tables8–10 list the experimental results of this algorithm compared with the algorithm in [15] when the target complexity is set to about 80% and 70%, respectively. From these tables, the proposed algorithm is better than reference [15] with less PSNR decrease and bitrate increase, while maintaining the same coding time.Table 8
BasketballDrive.
Coding time (%) [15]ΔBitRate (%) [15]ΔPSNR (dB) [15]Coding time (%) proposedΔBitRate (%) proposedΔPSNR (dB) proposed83%+0.3-0.0177%+0.3-0.0274%+0.8-0.0269%+0.5-0.04Table 9
Cactus.
Coding time (%) [15]ΔBitRate (%) [15]ΔPSNR (dB) [15]Coding time (%) proposedΔBitRate (%) proposedΔPSNR (dB)proposed85%+0.2-0.0179%+0.3-0.0375%+0.8-0.0342%+0.7-0.07Table 10
BQTerrace.
Coding time (%) [15]ΔBitRate (%) [15]ΔPSNR (dB) [15]Coding time (%) proposedΔBitRate (%) proposedΔPSNR (dB) proposed80%+0.6-0.0285%+0.1-0.0643%+1.8-0.0671%+0.9-0.18Figure12 is R-D curves of cactus. It can be seen from the figure that the algorithm in this paper can accurately control the intracoding complexity without greatly affecting the coding performance, which are very close to the original R-D curves. The R-D curve with the target complexity of 80% is basically coincident with the HM standard algorithm and slightly deviates from the HM standard algorithm at 60% and 40%.Figure 12
R-D curves of Cactus.


## 5.2. Subjective Quality Assessment

Figure13(a) is a frame in the test video sequence BasketballDrive (1920×1080), and Figure 13(b) is its corresponding saliency map. Figures 13(c) and 13(d), respectively, show the effect comparison after the algorithm is used to divide the unit of a certain frame when the target time is set to 100%, 80%, 60%, and 40%. Figures 13(g) and 13(h) are the result of a certain frame in a test video sequence by using the algorithm for a target time of 80% and 40%, respectively. It can be seen from Figure 13(b) that in this video sequence, the human eyes mainly focus on the basketball players, but do not pay attention to the floor and walls. Therefore, the algorithm strengthens the coding and division of the regions with high visual saliency, simplifies the division of the regions with low visual saliency, and according to different target complexities. This simplified process is controlled by different thresholds, as shown in Figures 13(c) and 13(d). It can be seen from the final coding effect Figure 13 that the algorithm can effectively control the coding complexity, and at the same time, there are almost no obvious visual differences.Figure 13
CU of BasketballDrive.
(a)
Frame of video(b)
Saliency map(c)
100%(d)
80%(e)
60%(f)
40%(g)
80%(h)
40%It can be seen from these figures that the depth of the coding unit is relatively large and the division is relatively detailed in the areas that are easy to be noticed by the human eyes. On the contrary, the depth of the region coding units to which the human eye pays less attention is small. The coding complexity is increased in a large flat area such as a floor, a wall, a river, or an area with less texture. It is not easy to improve the subjective visual quality for the human eyes, while it is helpful to improve the subjective visual quality by using more complex coding for the people and scenery that the human eyes pay attention to and the areas with more complex textures. Therefore, it is entirely possible to reduce the overall encoding complexity by reducing the number of bytes used to encode the regions with lower visual saliency. In this way, the algorithm can use less division to express the video information, so as to achieve the control of coding complexity.


## 6. Conclusion

In this paper, an intraprediction unit partition algorithm based on visual saliency is proposed. Firstly, the visual saliency map of each frame of the encoded video is generated according to the visual saliency algorithm, and then the proportion of CUs with different depths in the optimal mode generated by the HM standard algorithm and the relationship between the depth of different coding units and the encoding time are analyzed. There is also the relationship between the depth of coding unit division and its visual saliency, so a unit division model based on visual saliency is established, and the threshold of division is established according to statistical data. Finally, the results of simulation experiments are given. The experimental results show that the algorithm can effectively control the encoding time under the setting of the target complexity. There is only a small degradation in the objective quality, while there is little loss in the subjective quality that is perceptible to the human eye. This method will bring higher video coding efficiency of a number of scenarios, such as real time video coding and social media video.



---

*Source: 5069775-2022-09-29.xml*