# A Rate-Distortion Optimized Coding Method for Region of Interest in Scalable Video Coding

Authors: Hongtao Wang; Dong Zhang; Houqiang Li
Journal: Advances in Multimedia
 (2015)

---

## Abstract

The support for region of interest (ROI) browsing, which allows dropping background part of video bitstreams, is a desirable feature for video applications. With the help of the slice group technique
provided by H.264/SVC, rectangular ROI areas can be encoded into separate ROI slices. Additionally, by imposing certain constraints on motion estimation, ROI part of the bitstream can be decoded
without background slices of the same layer. However, due to the additional spatial and temporal constraints applied to the encoder, overall coding efficiency would be significantly decreased. In this paper, a rate-distortion optimized (RDO) encoding scheme is proposed to improve the coding efficiency of ROI slices. When background slices are discarded, the proposed method uses base layer information to generate the prediction
signal of the enhancement layer. Thus, the temporal constraints can be loosened during the encoding process. To do it in this way, the possible mismatch between generated reference frames and
original ones is also considered during rate-distortion optimization so that a reasonable trade-off between coding efficiency and decoding drift can be made. Besides, a new Lagrange multiplier derivation method is developed for further coding performance improvement. Experimental results demonstrate that the proposed method achieves significant bitrate saving compared to existing methods.


---

## Body

## 1. Introduction

With the rapid development and continuous expansion of mobile communications, mobile internet services are becoming more and more popular. As a result, mobile video applications, such as mobile video broadcasting [1], mobile video conference [2], and mobile video surveillance [3], have become an active research area in recent years. However, due to the fact that mobile devices typically have limited communication bandwidth, constrained power capacity, and various display capabilities, there are several fundamental difficulties in deploying high-quality video service for mobile devices over wireless networks. Among them, one crucial problem for mobile video application is how to browse high-resolution (HR) videos on mobile devices with small screens. Traditional approaches typically downsize the video to achieve the required resolution, which will inevitably cause a loss of perceptual information and a waste of network bandwidth. Visual perception experiments show that small display screen is always a critical factor affecting the browsing experiences. In fact, different parts of a picture do not equally attract people’s attention. People are likely to pay more attention to a certain area, which is called region of interest (ROI), than to other areas of a picture. Thus, it is beneficial to optimize multimedia systems according to ROIs of video content, for example, to make ROI areas have better video quality [4] and to make background areas droppable when needed [5].Scalable video coding extension of the H.264/MPEG-4 AVC video compression standard (H.264/SVC or SVC for short) [6] enables various functionalities to make encoded bitstreams more adaptive to dramatic variation of resource constraints, such as bandwidth, display capability, and power consumption. The base layer of SVC is compatible with H.264/AVC, so that it is easy to meet the requirement for the compatibility when upgrading video broadcasting infrastructures [7]. SVC is capable of encoding original video into different layers. Typically, it generates a high-quality video bitstream that contains one or more substreams, each of which corresponds to a degraded version of the original video signal with lower spatial resolution or lower temporal resolution or lower picture fidelity. Substreams can be extracted at media gateways according to network bandwidth and end device capability.In addition to spatial, temporal, and quality scalabilities, SVC also supports ROI scalability [5]. With the help of the slice group technique (also known as Flexible Macroblock Ordering (FMO) [8]), macroblocks (MBs) in ROI and background areas can be coded into ROI slices and background slices, respectively. A low bitrate substream, which contains ROI slices, can be extracted from the high-quality bitstream without any transcoding operation [6]; therefore SVC can provide ROI browsing functionality for multimedia communication systems. As illustrated in Figure 1, the base layer of a bitstream can be coded with relatively low spatial resolution or modest fidelity to provide basic video quality for devices with small screen or low bandwidth. The enhancement layer can be coded with high resolution or high fidelity. To provide higher video quality, the enhancement layers may be composed of ROI slices and background slices to provide ROI scalability with FMO, which is allowed in the base baseline line profile of H.264/SVC. With the help of spatial scalability, one may choose either the QCIF (Quarter Common Intermediate Format) base layer or the CIF (Common Intermediate Format) enhancement layer according to resource constraints such as network bandwidth, screen size, and power consumption, as illustrated in scenarios (a) and (c). Additionally, ROI scalability introduces a new scenario (scenario (b)), in which both base layer and ROI of enhancement layer are delivered and one may choose between whole scene with low resolution and zoomed ROI area with high resolution for a better experience.Figure 1
Application scenarios for SVC with ROI.To develop a video application system with ROI browsing functionality based on SVC (like the application in Figure1), ROI areas should be detected firstly in each frame, and then they can be coded into ROI slices using FMO. The detection of ROI of a picture has been widely studied with various attention models [9–11]. Given the fact that the ROI of a certain picture may be quite different for different people, it is practical and reasonable for a video application system to allow users to freely select some initial interested objects and then track these objects in subsequent video frames to locate ROI areas. Existing tracking methods can be used to perform the tracking operation. These methods can be categorized into pixel domain (Pel-domain) [12–14] and compressed domain (Com-domain) [15–17] approaches. Generally speaking, the Pel-domain approaches can achieve better tracking accuracy than the Com-domain ones yet with higher complexity. For the simulations in this paper, a color based Monte Carlo tracking technique introduced by Prez et al. [14] is applied for ROI tracking.To support ROI scalability, ROI slices should be self-contained, in other words, decodable in the absence of other slices of the same picture [18]. Thus, during the encoding process, dependencies between different slices, such as dependencies introduced by intraprediction and motion vector prediction, should be prohibited [8]. Besides, in order to provide acceptable visual quality in case background slices are all discarded, several additional temporal constraints are suggested to avoid using background slices in reference pictures to predict current ROI slice. The constrained motion vector method [19] presents an H.264/SVC compatible way where the MBs belonging to current ROI slice should use corresponding ROI slices in reference pictures only as reference. Besides, since the reference frames should be upsampled with a 6-tap interpolation filter for quarter-pel motion estimation (ME), the fractional pixel located within two pixels of the ROI slice boundary must also be ignored during motion estimation. However, this may significantly decrease the coding efficiency. Thus, Bae et al. [20] suggest using half-sample interpolation method for fractional pixel interpolation, where the slice boundary is treated as picture boundary. However, this method has little improvement compared with the constrained motion vector method; therefore, it has not been adopted into the H.264/SVC standard. Generally speaking, the above approaches all make strict truncation of temporal prediction, which leads to significant degradation of coding performance. Fortunately, for SVC application in Figure 1, since the base layer information is available when ROI of enhancement layer is being encoded, it is better to adopt a more flexible method to improve the coding performance of the enhancement layer ROI slice.In this paper, an efficient ROI coding algorithm under the SVC scalable baseline profile is proposed. When the enhancement layer contains ROI slices, information of the base layer is adopted to improve the coding efficiency of them. The framework of the proposed algorithm is illustrated in Figure2. The ROI area of an input picture is coded as a ROI slice by the proposed enhancement layer encoder. It improves the coding efficiency by using rate-distortion optimized (RDO) mode decision, which takes into consideration the error propagation due to the loss of the background slices, instead of directly restricting the motion vectors. A new Lagrange multiplier derivation method, which is associated with the proportion of ROI area, is also derived and used in the RDO model for further improving the ROI slice encoding performance.Figure 2
Architecture of ROI enabled SVC coding system.The remainder of this paper is organized as follows. The proposed ROI coding method is introduced in Section2. Then Section 3 shows some experimental results to verify the benefits of the proposed RDO method and is followed by the conclusion in Section 4.


## 2. ROI Coding for H.264/SVC

In H.264/SVC, a rectangular ROI area of a picture can be coded into a separate slice using FMO technique. However, to support ROI browsing functionality, additional effort should be made to ensure that ROI slices are independent of background data. Several approaches can be used to encode ROI slices. Most of them apply constraints to temporal prediction to enable fully independent decoding of ROI slices, for example, the constrained motion vector method [10, 19] and the half-sample interpolation method [20]. However, in H.264/SVC, such strict constraints may severely degrade the coding efficiency for enhancement layer users. Fortunately, since the corresponding base layer data is always contained in an enhancement layer bitstream, the base layer information is available to be adopted to further improve the coding efficiency of the ROI area of the enhancement layer. In this section, the existing ROI coding approaches are introduced first, and then the proposed RDO based framework for coding ROI slices is presented.


### 2.1. Existing ROI Coding Methods

#### 2.1.1. The Constrained Motion Vector Method

In H.264/SVC, motion estimation (ME) and motion compensation (MC) are performed using motion vectors with the accuracy of quarter-pixel luma samples. If the motion vector represents a fractional pixel position, interpolation is performed to generate predicted signal value.As shown in Figure3, the half-pixel samples are interpolated first from neighboring integer-pixel samples using a 6-tap finite impulse response (FIR) filter [21]. This means that each half-pixel sample is a weighted sum of 6 neighboring integer samples. Once all the half-pixel samples are available, the quarter-pixel samples are interpolated with neighboring half- and/or integer-pixel samples using bilinear interpolation.Figure 3
Interpolation of fractional pixels.The half-pixel sample “b” is interpolated by
(1)b=(20(G+H)-5(F+I)+(E+J)+16)≫5.
And the quarter-pixel sample a is interpolated by
(2)a=(b+G+1)≫1.Since the derivation process of some fractional pixels within two pixels inside the ROI boundary (labeled as “unavailable fractional pixel” in Figure3) depends on pixels out of the ROI region, Hannuksela et al. [19] proposed not to use them as reference during ME/MC; therefore, the dependency between ROI and background can be removed.


#### 2.1.2. The Half-Sample Interpolation Method

To loosen the restriction imposed on the ME process, the half-sample interpolation method [20] modifies the fractional pixel interpolation process to extend the pixels on slice boundaries using the 6-tap interpolation; for example, the half-sample “b” in Figure 3 is generated by
(3)b=(20(G+H)-10I+2J+16)≫5.
In the actual implementation, half-sample interpolation method is only adopted when generating reference signal for ROI areas. Original interpolation method in H.264/SVC is still used to generate reference frames of background slices so that better coding efficiency can be achieved.The above two methods aim to prohibit using samples in the background area of the reference frame during ME/MC, which also prevent a lot of MBs from being coded with proper prediction block located in or overlapped with background area. Therefore the coding efficiency is severely reduced compared with the original coding method.


### 2.2. Proposed RDO Based ROI Coding Framework

In application scenario (b) illustrated in Figure1, when background slices of the enhancement layer are dropped, pixels in them cannot be used as reference to decode the ROI area of enhancement layer, but the base layer information is still available. So the background pixels of the base layer can be reconstructed and used to generate reference frames for ROI areas of enhancement layer using error concealment techniques. However, the mismatch between original and error-concealed reference frames may probably cause severe error propagation; therefore, the coding modes of MBs (which may use the error-concealed blocks as reference) should be selected carefully.LetP and P′ denote the original reference frame (ORF) and the error-concealed reference frame (named as the virtual reference frame, VRF), respectively, where P′ may be generated using base layer information through error concealment techniques. In the proposed RDO framework, the mismatch between P and P′, together with source error introduced by quantization in the encoding loop, is considered as total distortion. The RDO evaluation for mode decision is based on this total distortion. Furthermore, the Lagrange multiplier is modified to take account of the proportion of ROI area for better performance.


#### 2.2.1. Generation of Virtual Reference Frame (VRF)

Figure4 shows the generation of intercoded VRF n at encoder side. Since the background slice is assumed to be discarded, the pixels belonging to the background slice are estimated using the base layer information with the same error concealment method that the decoder uses (in this paper, the well-known BL-skip method [22] is adopted); the pixels belonging to the ROI slice are generated by motion compensation using their own motion vectors and residuals while taking the former VRF n-1 as reference. Then, the VRF n also serves as the reference frame of the following VRFs. The generation of intracoded VRF is similar to that of intercoded VRF except that upsampled textures are directly used to imitate the background slice. Notice that, in actual implementation, the upsampled motion vectors (MVs), residuals, and textures can be easily obtained when calculating the cost of “base layer mode.” Thus, only one additional MC operation for each MB is needed to generate VRF.Figure 4
Generation of virtual reference framen.


#### 2.2.2. Proposed RDO Mode Decision

In the mode decision process of a macroblock, the coding mode with the minimum RD cost is selected:(4)J=D+λR,
where D and R are distortion introduced and bits consumed by the coding mode under consideration, respectively. λ is the Lagrange multiplier.For an MB in a ROI slice, the proposed mode decision scheme considers both the distortion introduced by the difference between the reconstructed MB and the original MB (termed source distortion) and the mismatch between reference MB inP and P′ (termed mismatch distortion). So the RD cost function J for mode decision becomes
(5)J=Ds+Dc+λR,
where Ds stands for the source distortion and Dc is the mismatch distortion.For an MB in a background slice, a basic assumption is that users who receive enhancement layer background slices should also receive ROI slices. Such assumption is reasonable considering ROI slices are more important and thereby are more protected than background slices. Thus,Dc becomes zero, and the cost function for mode decision is now degraded to its original form:
(6)J=Ds+λR.Figure5 depicts in detail the implementation of the proposed RDO mode decision of an MB.(1)
Firstly, given a modem, the best motion vector mvbest for each partition is selected using the original reference frame. And let the corresponding predictor be Pm.(2)
Then, calculate the source distortionDs and the cost bits R through the encoding process of mode m.(3)
If current MB belongs to ROI area, then find the new predictorPm′ from previous VRF with motion vector mvbest and calculate the mismatch error Dc=(Pm-Pm′)2. Note that the following selection should be made: for distortion calculation in mode decision, the previous VRF is used as reference, while for distortion calculation in ME, the previous ORF is still used.(4)
Calculate the RD cost using (5) or (6), and turn to step (1) for the next mode.(5)
Finally, the mode with the minimumJ among all candidate modes is selected as the best mode.Figure 5
The proposed RD cost calculation process for a macroblock.The benefit of the proposed RDO method is illustrated through the rate-distortion (RD) performance comparison in Figures6 and 7 for both spatial and quality SVC. In Figure 6, the spatial SVC bitstream contains a QCIF base layer and a CIF enhancement layer. In Figure 7, the quality SVC bitstream contains a CIF base layer and a CIF enhancement layer. The intraperiod is set to 30. Four pairs of quantization parameter (QP) are chosen for the test: for spatial SVC, QP pairs for QCIF base layer and CIF enhancement layer are (22, 26), (26, 30), (30, 34), and (34, 38), respectively, and, for quality SVC, the QP pairs are (30, 26), (34, 30), (38, 34), and (42, 38), respectively. The original method in H.264/SVC, which uses the ORFs without any constraints on temporal prediction, is simulated as the anchor. Three collections of data are presented in each figure, where “mdrdo” stands for the proposed RDO based mode decision method, and “mv_constrain” and “half-interpolation” stand for the ROI coding methods mentioned in Section 2.1.RD performance for spatial SVC.
(a)
Bitrate savings for enhancement user
(b)
Bitrate savings for ROI userRD performance for quality SVC.
(a)
Bitrate savings for enhancement user
(b)
Bitrate savings for ROI userCoding efficiency in the following two scenarios is considered. The first is that enhancement layer slices are received completely, which means that the quality of full resolution enhancement layer (labeled “Enc full”) should be considered. The second scenario is that background slices are all discarded, which means only the quality of ROI areas (labeled “Dec ROI”) affects the user experience. Average bitrate savings, which are calculated via the excel add-in proposed in VCEG-AE07 [23] (lager Δ-bitrate value means worse performance), of the above three coding methods compared with “orig” method are presented.From Figures6(a) and 7(a), we can see that the coding efficiency of those three methods (for the whole SVC bit-stream) is all lower compared with the “orig.” method, which uses the perfectly decoded reference frames as prediction. However, the proposed method achieves significant improvement compared with the other two methods because of the use of a better reference for coding ROI slices. The average performance gain compared with the one termed “half-interpolation” over the tested sequences is about 5% (Figure 6(a)) and 7% (Figure 7(a)) for spatial and quality SVC, respectively.Though the performances of those methods are inferior compared with original method when enhancement layer slices are all received, however, considering the most important ROI browsing scenario in which all the background slices may be discarded, the RD performance of those methods is much better than that of “orig.” method. As illustrated in Figures6(b) and 7(b), the bitrate saving of the proposed method is up to 50% (about 30% and 4% for spatial and quality SVC on average, resp.) compared with “orig.” method. The lower gain for quality SVC compared with spatial SVC is consistent with the common knowledge that a better concealment quality will be obtained when the base layer and enhancement layer have the same resolution; thus, acceptable quality may be obtained even with “orig.” method for the quality coding configuration (CIF-CIF). Still, the proposed method outperforms “mv_constrain” and “half-interpolation” methods, and the average gain compared with “half-interpolation” is 3% (Figure 6(b)) and 2.5% (Figure 7(b)) for spatial and quality SVC, respectively.


#### 2.2.3. The Selection of Lagrange Multiplier

In RDO optimization, the Lagrange multiplier should be carefully selected to ensure that the most suitable modes are chosen. In this paper, a refined Lagrange multiplier selection method for background and ROI slices is proposed to further improve the RD performance of ROI slices. In H.264/AVC, the Lagrange multiplierλ can be calculated as follows.SupposingR and D in (4) to be differentiable everywhere, the minimum cost J is given by setting its derivative to zero, thus, leading to
(7)λ=-dDdR.
Then, the Lagrange multiplier λ for single layer video coding can be solved through the rate model Rsingle (8) and distortion model Dsingle (9) [24]:
(8)Rsingle=alog⁡2bDsingle,(9)Dsingle=Q212,
where a and b are two constants and Q is the quantization step. According to (8) and (9), the derivative of D and R can be calculated by
(10)dDsingle=Q6dQ,dRsingle=-2aln⁡2·QdQ.
Putting (10) into (7) and letting ln⁡2=12ac, the λ for single layer is finally derived as
(11)λQ=ln⁡212a·Q2=c·Q2,
where c is a constant, which is experimentally suggested to be 0.85 [24], though others proposed 0.68 [25].During the development of H.264/SVC, suchλ is directly used in H.264/SVC reference software Joint Scalable Video Model (JSVM) [26]. However, applying such λ, which is derived for single layer Lagrange multiplier, into multilayer scenario is inappropriate, since the correlation between layers is not considered in the Lagrange multiplier selection. To improve the overall coding performance, an encoder-only optimization contribution for RDO in SVC is presented by Li et al. [27], which is adopted into later JSVM in an optional way. According to this method, the Lagrange multiplier is derived as
(12)λ′=β·(Q+Δ)β·(Q+Δ)+Q·λQ,
where β denotes the resolution ratio of the two layers and (Q+Δ) and Q are the quantization steps for the base layer and enhancement layer, respectively.Similarly, for a specific user who requires base layer slices together with ROI slices, the best Lagrange multiplier for ROI slices can be derived as follows.Let the joint cost be(13)J=ωb·Jb+ωroi·Jroi=ωb·DQ+Δ+λb·RQ+Δ+ωroi·α·DQ+Dc+λroi·RQ+Δ+α·RQ,
where Jb and Jroi are the RD cost functions for base layer and enhancement layer with ROI area, respectively, ωb and ωroi are the contribution weights of base layer cost and ROI area cost, respectively, and ωb+ωroi=1. Similar to [27], the term α denotes the resolution ratio between enhancement layer ROI area and base layer and is introduced as an approximation for bitrate ratio between the ROI slice and base layer slice.The termDc is the mismatch distortion (as described in (5)) introduced by difference between ORF and VRF. Dc is much larger than D(Q) and thus can be regarded as independent with Q. Therefore, put (8) and (9) into (13) and then set the derivative of J to zero; the Lagrange multiplier λroi can be solved as
(14)λroi=Qωbωroi·ln⁡2·Q+Δ2-12aλbdddωbωroi+ln⁡2·αQQ+Δ×12aα·Q+Δ+Q-1.
Considering the base layer Lagrange multiplier λb is determined by single layer λ selection method (11); put λb=c·(Q+Δ)2 into (14) to derive λroi as
(15)λroi=α·(Q+Δ)α·(Q+Δ)+Q·(c·Q2)=α·Q+Δα·Q+Δ+Q·λQ.Note that the derivedλroi is similar to (13). However, the rate ratio has a different form since the area of ROI has been taken into consideration.LetQPb and QPe denote the quantization parameters for base layer and enhancement layer, respectively. Then, according to the relationship between quantization steps and quantization parameters, which has been defined in H.264/AVC standard [28], QPb, QPe, and Q conform to
(16)Q+Δ=2(QPb-12)/6,Q=2(QPe-12)/6.Equation (14) can be simplified according to the quantization parameter difference between base and enhancement layers; namely, ΔQP=QPb-QPe. And finally, the modified Lagrange multiplier λroi can be obtained through the following equation for SVC enhancement layer when ROI is enabled:
(17)λroi=α·2ΔQP/6α·2ΔQP/6+1.Similar to the RDO mode decision part, the performance of the proposed RDO method with modified Lagrange multiplier is shown in Figures8 and 9 for spatial and quality SVC, respectively. The same coding parameters are used. “Enc full” and “Dec ROI” denote the scenarios when background slices are received and discarded, respectively. The proposed “mdrdo” method with Lagrange multiplier λ′ [27] and with the proposed λroi  (17) is simulated separately. And the “mdrdo” method with original single layer Lagrange multiplier (11) is performed as the anchor.RD performance for spatial SVC with newλ.
(a)
Bitrate savings for enhancement user with proposedλ
(b)
Bitrate savings for ROI user with newλRD performance for quality SVC with newλ.
(a)
Bitrate savings for enhancement user with newλ
(b)
Bitrate savings for ROI user with NewλAll the values in Figures8 and 9 are negative, which means that both Lagrange multiplier modification methods have brought benefits to ROI and enhancement layer coding compared with the original Lagrange multiplier method. Compared with λ′, the proposed λroi achieves a better performance for ROI, and the average gain is about 6% for spatial SVC (Figure 8(b)) and 7% for quality SVC (Figure 9(b)). Since λroi is always smaller than λ′, the proposed λroi shifts bits from a background slice to an ROI slice. Thus, for the whole enhancement layer, the proposed λroi leads to trifling loss of coding efficiency compared with λ′. The average loss of “mdrdo + λroi” compared with “mdrdo + λ′” is about 1% for spatial SVC (Figure 8(a)) and 1.5% for quality SVC (Figure 9(a)), respectively. Although the performance of the background area is sacrificed, it is worthwhile given that the performance of semantic-important ROI area is improved.


## 2.1. Existing ROI Coding Methods

### 2.1.1. The Constrained Motion Vector Method

In H.264/SVC, motion estimation (ME) and motion compensation (MC) are performed using motion vectors with the accuracy of quarter-pixel luma samples. If the motion vector represents a fractional pixel position, interpolation is performed to generate predicted signal value.As shown in Figure3, the half-pixel samples are interpolated first from neighboring integer-pixel samples using a 6-tap finite impulse response (FIR) filter [21]. This means that each half-pixel sample is a weighted sum of 6 neighboring integer samples. Once all the half-pixel samples are available, the quarter-pixel samples are interpolated with neighboring half- and/or integer-pixel samples using bilinear interpolation.Figure 3
Interpolation of fractional pixels.The half-pixel sample “b” is interpolated by
(1)b=(20(G+H)-5(F+I)+(E+J)+16)≫5.
And the quarter-pixel sample a is interpolated by
(2)a=(b+G+1)≫1.Since the derivation process of some fractional pixels within two pixels inside the ROI boundary (labeled as “unavailable fractional pixel” in Figure3) depends on pixels out of the ROI region, Hannuksela et al. [19] proposed not to use them as reference during ME/MC; therefore, the dependency between ROI and background can be removed.


### 2.1.2. The Half-Sample Interpolation Method

To loosen the restriction imposed on the ME process, the half-sample interpolation method [20] modifies the fractional pixel interpolation process to extend the pixels on slice boundaries using the 6-tap interpolation; for example, the half-sample “b” in Figure 3 is generated by
(3)b=(20(G+H)-10I+2J+16)≫5.
In the actual implementation, half-sample interpolation method is only adopted when generating reference signal for ROI areas. Original interpolation method in H.264/SVC is still used to generate reference frames of background slices so that better coding efficiency can be achieved.The above two methods aim to prohibit using samples in the background area of the reference frame during ME/MC, which also prevent a lot of MBs from being coded with proper prediction block located in or overlapped with background area. Therefore the coding efficiency is severely reduced compared with the original coding method.


## 2.1.1. The Constrained Motion Vector Method

In H.264/SVC, motion estimation (ME) and motion compensation (MC) are performed using motion vectors with the accuracy of quarter-pixel luma samples. If the motion vector represents a fractional pixel position, interpolation is performed to generate predicted signal value.As shown in Figure3, the half-pixel samples are interpolated first from neighboring integer-pixel samples using a 6-tap finite impulse response (FIR) filter [21]. This means that each half-pixel sample is a weighted sum of 6 neighboring integer samples. Once all the half-pixel samples are available, the quarter-pixel samples are interpolated with neighboring half- and/or integer-pixel samples using bilinear interpolation.Figure 3
Interpolation of fractional pixels.The half-pixel sample “b” is interpolated by
(1)b=(20(G+H)-5(F+I)+(E+J)+16)≫5.
And the quarter-pixel sample a is interpolated by
(2)a=(b+G+1)≫1.Since the derivation process of some fractional pixels within two pixels inside the ROI boundary (labeled as “unavailable fractional pixel” in Figure3) depends on pixels out of the ROI region, Hannuksela et al. [19] proposed not to use them as reference during ME/MC; therefore, the dependency between ROI and background can be removed.


## 2.1.2. The Half-Sample Interpolation Method

To loosen the restriction imposed on the ME process, the half-sample interpolation method [20] modifies the fractional pixel interpolation process to extend the pixels on slice boundaries using the 6-tap interpolation; for example, the half-sample “b” in Figure 3 is generated by
(3)b=(20(G+H)-10I+2J+16)≫5.
In the actual implementation, half-sample interpolation method is only adopted when generating reference signal for ROI areas. Original interpolation method in H.264/SVC is still used to generate reference frames of background slices so that better coding efficiency can be achieved.The above two methods aim to prohibit using samples in the background area of the reference frame during ME/MC, which also prevent a lot of MBs from being coded with proper prediction block located in or overlapped with background area. Therefore the coding efficiency is severely reduced compared with the original coding method.


## 2.2. Proposed RDO Based ROI Coding Framework

In application scenario (b) illustrated in Figure1, when background slices of the enhancement layer are dropped, pixels in them cannot be used as reference to decode the ROI area of enhancement layer, but the base layer information is still available. So the background pixels of the base layer can be reconstructed and used to generate reference frames for ROI areas of enhancement layer using error concealment techniques. However, the mismatch between original and error-concealed reference frames may probably cause severe error propagation; therefore, the coding modes of MBs (which may use the error-concealed blocks as reference) should be selected carefully.LetP and P′ denote the original reference frame (ORF) and the error-concealed reference frame (named as the virtual reference frame, VRF), respectively, where P′ may be generated using base layer information through error concealment techniques. In the proposed RDO framework, the mismatch between P and P′, together with source error introduced by quantization in the encoding loop, is considered as total distortion. The RDO evaluation for mode decision is based on this total distortion. Furthermore, the Lagrange multiplier is modified to take account of the proportion of ROI area for better performance.


### 2.2.1. Generation of Virtual Reference Frame (VRF)

Figure4 shows the generation of intercoded VRF n at encoder side. Since the background slice is assumed to be discarded, the pixels belonging to the background slice are estimated using the base layer information with the same error concealment method that the decoder uses (in this paper, the well-known BL-skip method [22] is adopted); the pixels belonging to the ROI slice are generated by motion compensation using their own motion vectors and residuals while taking the former VRF n-1 as reference. Then, the VRF n also serves as the reference frame of the following VRFs. The generation of intracoded VRF is similar to that of intercoded VRF except that upsampled textures are directly used to imitate the background slice. Notice that, in actual implementation, the upsampled motion vectors (MVs), residuals, and textures can be easily obtained when calculating the cost of “base layer mode.” Thus, only one additional MC operation for each MB is needed to generate VRF.Figure 4
Generation of virtual reference framen.


### 2.2.2. Proposed RDO Mode Decision

In the mode decision process of a macroblock, the coding mode with the minimum RD cost is selected:(4)J=D+λR,
where D and R are distortion introduced and bits consumed by the coding mode under consideration, respectively. λ is the Lagrange multiplier.For an MB in a ROI slice, the proposed mode decision scheme considers both the distortion introduced by the difference between the reconstructed MB and the original MB (termed source distortion) and the mismatch between reference MB inP and P′ (termed mismatch distortion). So the RD cost function J for mode decision becomes
(5)J=Ds+Dc+λR,
where Ds stands for the source distortion and Dc is the mismatch distortion.For an MB in a background slice, a basic assumption is that users who receive enhancement layer background slices should also receive ROI slices. Such assumption is reasonable considering ROI slices are more important and thereby are more protected than background slices. Thus,Dc becomes zero, and the cost function for mode decision is now degraded to its original form:
(6)J=Ds+λR.Figure5 depicts in detail the implementation of the proposed RDO mode decision of an MB.(1)
Firstly, given a modem, the best motion vector mvbest for each partition is selected using the original reference frame. And let the corresponding predictor be Pm.(2)
Then, calculate the source distortionDs and the cost bits R through the encoding process of mode m.(3)
If current MB belongs to ROI area, then find the new predictorPm′ from previous VRF with motion vector mvbest and calculate the mismatch error Dc=(Pm-Pm′)2. Note that the following selection should be made: for distortion calculation in mode decision, the previous VRF is used as reference, while for distortion calculation in ME, the previous ORF is still used.(4)
Calculate the RD cost using (5) or (6), and turn to step (1) for the next mode.(5)
Finally, the mode with the minimumJ among all candidate modes is selected as the best mode.Figure 5
The proposed RD cost calculation process for a macroblock.The benefit of the proposed RDO method is illustrated through the rate-distortion (RD) performance comparison in Figures6 and 7 for both spatial and quality SVC. In Figure 6, the spatial SVC bitstream contains a QCIF base layer and a CIF enhancement layer. In Figure 7, the quality SVC bitstream contains a CIF base layer and a CIF enhancement layer. The intraperiod is set to 30. Four pairs of quantization parameter (QP) are chosen for the test: for spatial SVC, QP pairs for QCIF base layer and CIF enhancement layer are (22, 26), (26, 30), (30, 34), and (34, 38), respectively, and, for quality SVC, the QP pairs are (30, 26), (34, 30), (38, 34), and (42, 38), respectively. The original method in H.264/SVC, which uses the ORFs without any constraints on temporal prediction, is simulated as the anchor. Three collections of data are presented in each figure, where “mdrdo” stands for the proposed RDO based mode decision method, and “mv_constrain” and “half-interpolation” stand for the ROI coding methods mentioned in Section 2.1.RD performance for spatial SVC.
(a)
Bitrate savings for enhancement user
(b)
Bitrate savings for ROI userRD performance for quality SVC.
(a)
Bitrate savings for enhancement user
(b)
Bitrate savings for ROI userCoding efficiency in the following two scenarios is considered. The first is that enhancement layer slices are received completely, which means that the quality of full resolution enhancement layer (labeled “Enc full”) should be considered. The second scenario is that background slices are all discarded, which means only the quality of ROI areas (labeled “Dec ROI”) affects the user experience. Average bitrate savings, which are calculated via the excel add-in proposed in VCEG-AE07 [23] (lager Δ-bitrate value means worse performance), of the above three coding methods compared with “orig” method are presented.From Figures6(a) and 7(a), we can see that the coding efficiency of those three methods (for the whole SVC bit-stream) is all lower compared with the “orig.” method, which uses the perfectly decoded reference frames as prediction. However, the proposed method achieves significant improvement compared with the other two methods because of the use of a better reference for coding ROI slices. The average performance gain compared with the one termed “half-interpolation” over the tested sequences is about 5% (Figure 6(a)) and 7% (Figure 7(a)) for spatial and quality SVC, respectively.Though the performances of those methods are inferior compared with original method when enhancement layer slices are all received, however, considering the most important ROI browsing scenario in which all the background slices may be discarded, the RD performance of those methods is much better than that of “orig.” method. As illustrated in Figures6(b) and 7(b), the bitrate saving of the proposed method is up to 50% (about 30% and 4% for spatial and quality SVC on average, resp.) compared with “orig.” method. The lower gain for quality SVC compared with spatial SVC is consistent with the common knowledge that a better concealment quality will be obtained when the base layer and enhancement layer have the same resolution; thus, acceptable quality may be obtained even with “orig.” method for the quality coding configuration (CIF-CIF). Still, the proposed method outperforms “mv_constrain” and “half-interpolation” methods, and the average gain compared with “half-interpolation” is 3% (Figure 6(b)) and 2.5% (Figure 7(b)) for spatial and quality SVC, respectively.


### 2.2.3. The Selection of Lagrange Multiplier

In RDO optimization, the Lagrange multiplier should be carefully selected to ensure that the most suitable modes are chosen. In this paper, a refined Lagrange multiplier selection method for background and ROI slices is proposed to further improve the RD performance of ROI slices. In H.264/AVC, the Lagrange multiplierλ can be calculated as follows.SupposingR and D in (4) to be differentiable everywhere, the minimum cost J is given by setting its derivative to zero, thus, leading to
(7)λ=-dDdR.
Then, the Lagrange multiplier λ for single layer video coding can be solved through the rate model Rsingle (8) and distortion model Dsingle (9) [24]:
(8)Rsingle=alog⁡2bDsingle,(9)Dsingle=Q212,
where a and b are two constants and Q is the quantization step. According to (8) and (9), the derivative of D and R can be calculated by
(10)dDsingle=Q6dQ,dRsingle=-2aln⁡2·QdQ.
Putting (10) into (7) and letting ln⁡2=12ac, the λ for single layer is finally derived as
(11)λQ=ln⁡212a·Q2=c·Q2,
where c is a constant, which is experimentally suggested to be 0.85 [24], though others proposed 0.68 [25].During the development of H.264/SVC, suchλ is directly used in H.264/SVC reference software Joint Scalable Video Model (JSVM) [26]. However, applying such λ, which is derived for single layer Lagrange multiplier, into multilayer scenario is inappropriate, since the correlation between layers is not considered in the Lagrange multiplier selection. To improve the overall coding performance, an encoder-only optimization contribution for RDO in SVC is presented by Li et al. [27], which is adopted into later JSVM in an optional way. According to this method, the Lagrange multiplier is derived as
(12)λ′=β·(Q+Δ)β·(Q+Δ)+Q·λQ,
where β denotes the resolution ratio of the two layers and (Q+Δ) and Q are the quantization steps for the base layer and enhancement layer, respectively.Similarly, for a specific user who requires base layer slices together with ROI slices, the best Lagrange multiplier for ROI slices can be derived as follows.Let the joint cost be(13)J=ωb·Jb+ωroi·Jroi=ωb·DQ+Δ+λb·RQ+Δ+ωroi·α·DQ+Dc+λroi·RQ+Δ+α·RQ,
where Jb and Jroi are the RD cost functions for base layer and enhancement layer with ROI area, respectively, ωb and ωroi are the contribution weights of base layer cost and ROI area cost, respectively, and ωb+ωroi=1. Similar to [27], the term α denotes the resolution ratio between enhancement layer ROI area and base layer and is introduced as an approximation for bitrate ratio between the ROI slice and base layer slice.The termDc is the mismatch distortion (as described in (5)) introduced by difference between ORF and VRF. Dc is much larger than D(Q) and thus can be regarded as independent with Q. Therefore, put (8) and (9) into (13) and then set the derivative of J to zero; the Lagrange multiplier λroi can be solved as
(14)λroi=Qωbωroi·ln⁡2·Q+Δ2-12aλbdddωbωroi+ln⁡2·αQQ+Δ×12aα·Q+Δ+Q-1.
Considering the base layer Lagrange multiplier λb is determined by single layer λ selection method (11); put λb=c·(Q+Δ)2 into (14) to derive λroi as
(15)λroi=α·(Q+Δ)α·(Q+Δ)+Q·(c·Q2)=α·Q+Δα·Q+Δ+Q·λQ.Note that the derivedλroi is similar to (13). However, the rate ratio has a different form since the area of ROI has been taken into consideration.LetQPb and QPe denote the quantization parameters for base layer and enhancement layer, respectively. Then, according to the relationship between quantization steps and quantization parameters, which has been defined in H.264/AVC standard [28], QPb, QPe, and Q conform to
(16)Q+Δ=2(QPb-12)/6,Q=2(QPe-12)/6.Equation (14) can be simplified according to the quantization parameter difference between base and enhancement layers; namely, ΔQP=QPb-QPe. And finally, the modified Lagrange multiplier λroi can be obtained through the following equation for SVC enhancement layer when ROI is enabled:
(17)λroi=α·2ΔQP/6α·2ΔQP/6+1.Similar to the RDO mode decision part, the performance of the proposed RDO method with modified Lagrange multiplier is shown in Figures8 and 9 for spatial and quality SVC, respectively. The same coding parameters are used. “Enc full” and “Dec ROI” denote the scenarios when background slices are received and discarded, respectively. The proposed “mdrdo” method with Lagrange multiplier λ′ [27] and with the proposed λroi  (17) is simulated separately. And the “mdrdo” method with original single layer Lagrange multiplier (11) is performed as the anchor.RD performance for spatial SVC with newλ.
(a)
Bitrate savings for enhancement user with proposedλ
(b)
Bitrate savings for ROI user with newλRD performance for quality SVC with newλ.
(a)
Bitrate savings for enhancement user with newλ
(b)
Bitrate savings for ROI user with NewλAll the values in Figures8 and 9 are negative, which means that both Lagrange multiplier modification methods have brought benefits to ROI and enhancement layer coding compared with the original Lagrange multiplier method. Compared with λ′, the proposed λroi achieves a better performance for ROI, and the average gain is about 6% for spatial SVC (Figure 8(b)) and 7% for quality SVC (Figure 9(b)). Since λroi is always smaller than λ′, the proposed λroi shifts bits from a background slice to an ROI slice. Thus, for the whole enhancement layer, the proposed λroi leads to trifling loss of coding efficiency compared with λ′. The average loss of “mdrdo + λroi” compared with “mdrdo + λ′” is about 1% for spatial SVC (Figure 8(a)) and 1.5% for quality SVC (Figure 9(a)), respectively. Although the performance of the background area is sacrificed, it is worthwhile given that the performance of semantic-important ROI area is improved.


## 2.2.1. Generation of Virtual Reference Frame (VRF)

Figure4 shows the generation of intercoded VRF n at encoder side. Since the background slice is assumed to be discarded, the pixels belonging to the background slice are estimated using the base layer information with the same error concealment method that the decoder uses (in this paper, the well-known BL-skip method [22] is adopted); the pixels belonging to the ROI slice are generated by motion compensation using their own motion vectors and residuals while taking the former VRF n-1 as reference. Then, the VRF n also serves as the reference frame of the following VRFs. The generation of intracoded VRF is similar to that of intercoded VRF except that upsampled textures are directly used to imitate the background slice. Notice that, in actual implementation, the upsampled motion vectors (MVs), residuals, and textures can be easily obtained when calculating the cost of “base layer mode.” Thus, only one additional MC operation for each MB is needed to generate VRF.Figure 4
Generation of virtual reference framen.


## 2.2.2. Proposed RDO Mode Decision

In the mode decision process of a macroblock, the coding mode with the minimum RD cost is selected:(4)J=D+λR,
where D and R are distortion introduced and bits consumed by the coding mode under consideration, respectively. λ is the Lagrange multiplier.For an MB in a ROI slice, the proposed mode decision scheme considers both the distortion introduced by the difference between the reconstructed MB and the original MB (termed source distortion) and the mismatch between reference MB inP and P′ (termed mismatch distortion). So the RD cost function J for mode decision becomes
(5)J=Ds+Dc+λR,
where Ds stands for the source distortion and Dc is the mismatch distortion.For an MB in a background slice, a basic assumption is that users who receive enhancement layer background slices should also receive ROI slices. Such assumption is reasonable considering ROI slices are more important and thereby are more protected than background slices. Thus,Dc becomes zero, and the cost function for mode decision is now degraded to its original form:
(6)J=Ds+λR.Figure5 depicts in detail the implementation of the proposed RDO mode decision of an MB.(1)
Firstly, given a modem, the best motion vector mvbest for each partition is selected using the original reference frame. And let the corresponding predictor be Pm.(2)
Then, calculate the source distortionDs and the cost bits R through the encoding process of mode m.(3)
If current MB belongs to ROI area, then find the new predictorPm′ from previous VRF with motion vector mvbest and calculate the mismatch error Dc=(Pm-Pm′)2. Note that the following selection should be made: for distortion calculation in mode decision, the previous VRF is used as reference, while for distortion calculation in ME, the previous ORF is still used.(4)
Calculate the RD cost using (5) or (6), and turn to step (1) for the next mode.(5)
Finally, the mode with the minimumJ among all candidate modes is selected as the best mode.Figure 5
The proposed RD cost calculation process for a macroblock.The benefit of the proposed RDO method is illustrated through the rate-distortion (RD) performance comparison in Figures6 and 7 for both spatial and quality SVC. In Figure 6, the spatial SVC bitstream contains a QCIF base layer and a CIF enhancement layer. In Figure 7, the quality SVC bitstream contains a CIF base layer and a CIF enhancement layer. The intraperiod is set to 30. Four pairs of quantization parameter (QP) are chosen for the test: for spatial SVC, QP pairs for QCIF base layer and CIF enhancement layer are (22, 26), (26, 30), (30, 34), and (34, 38), respectively, and, for quality SVC, the QP pairs are (30, 26), (34, 30), (38, 34), and (42, 38), respectively. The original method in H.264/SVC, which uses the ORFs without any constraints on temporal prediction, is simulated as the anchor. Three collections of data are presented in each figure, where “mdrdo” stands for the proposed RDO based mode decision method, and “mv_constrain” and “half-interpolation” stand for the ROI coding methods mentioned in Section 2.1.RD performance for spatial SVC.
(a)
Bitrate savings for enhancement user
(b)
Bitrate savings for ROI userRD performance for quality SVC.
(a)
Bitrate savings for enhancement user
(b)
Bitrate savings for ROI userCoding efficiency in the following two scenarios is considered. The first is that enhancement layer slices are received completely, which means that the quality of full resolution enhancement layer (labeled “Enc full”) should be considered. The second scenario is that background slices are all discarded, which means only the quality of ROI areas (labeled “Dec ROI”) affects the user experience. Average bitrate savings, which are calculated via the excel add-in proposed in VCEG-AE07 [23] (lager Δ-bitrate value means worse performance), of the above three coding methods compared with “orig” method are presented.From Figures6(a) and 7(a), we can see that the coding efficiency of those three methods (for the whole SVC bit-stream) is all lower compared with the “orig.” method, which uses the perfectly decoded reference frames as prediction. However, the proposed method achieves significant improvement compared with the other two methods because of the use of a better reference for coding ROI slices. The average performance gain compared with the one termed “half-interpolation” over the tested sequences is about 5% (Figure 6(a)) and 7% (Figure 7(a)) for spatial and quality SVC, respectively.Though the performances of those methods are inferior compared with original method when enhancement layer slices are all received, however, considering the most important ROI browsing scenario in which all the background slices may be discarded, the RD performance of those methods is much better than that of “orig.” method. As illustrated in Figures6(b) and 7(b), the bitrate saving of the proposed method is up to 50% (about 30% and 4% for spatial and quality SVC on average, resp.) compared with “orig.” method. The lower gain for quality SVC compared with spatial SVC is consistent with the common knowledge that a better concealment quality will be obtained when the base layer and enhancement layer have the same resolution; thus, acceptable quality may be obtained even with “orig.” method for the quality coding configuration (CIF-CIF). Still, the proposed method outperforms “mv_constrain” and “half-interpolation” methods, and the average gain compared with “half-interpolation” is 3% (Figure 6(b)) and 2.5% (Figure 7(b)) for spatial and quality SVC, respectively.


## 2.2.3. The Selection of Lagrange Multiplier

In RDO optimization, the Lagrange multiplier should be carefully selected to ensure that the most suitable modes are chosen. In this paper, a refined Lagrange multiplier selection method for background and ROI slices is proposed to further improve the RD performance of ROI slices. In H.264/AVC, the Lagrange multiplierλ can be calculated as follows.SupposingR and D in (4) to be differentiable everywhere, the minimum cost J is given by setting its derivative to zero, thus, leading to
(7)λ=-dDdR.
Then, the Lagrange multiplier λ for single layer video coding can be solved through the rate model Rsingle (8) and distortion model Dsingle (9) [24]:
(8)Rsingle=alog⁡2bDsingle,(9)Dsingle=Q212,
where a and b are two constants and Q is the quantization step. According to (8) and (9), the derivative of D and R can be calculated by
(10)dDsingle=Q6dQ,dRsingle=-2aln⁡2·QdQ.
Putting (10) into (7) and letting ln⁡2=12ac, the λ for single layer is finally derived as
(11)λQ=ln⁡212a·Q2=c·Q2,
where c is a constant, which is experimentally suggested to be 0.85 [24], though others proposed 0.68 [25].During the development of H.264/SVC, suchλ is directly used in H.264/SVC reference software Joint Scalable Video Model (JSVM) [26]. However, applying such λ, which is derived for single layer Lagrange multiplier, into multilayer scenario is inappropriate, since the correlation between layers is not considered in the Lagrange multiplier selection. To improve the overall coding performance, an encoder-only optimization contribution for RDO in SVC is presented by Li et al. [27], which is adopted into later JSVM in an optional way. According to this method, the Lagrange multiplier is derived as
(12)λ′=β·(Q+Δ)β·(Q+Δ)+Q·λQ,
where β denotes the resolution ratio of the two layers and (Q+Δ) and Q are the quantization steps for the base layer and enhancement layer, respectively.Similarly, for a specific user who requires base layer slices together with ROI slices, the best Lagrange multiplier for ROI slices can be derived as follows.Let the joint cost be(13)J=ωb·Jb+ωroi·Jroi=ωb·DQ+Δ+λb·RQ+Δ+ωroi·α·DQ+Dc+λroi·RQ+Δ+α·RQ,
where Jb and Jroi are the RD cost functions for base layer and enhancement layer with ROI area, respectively, ωb and ωroi are the contribution weights of base layer cost and ROI area cost, respectively, and ωb+ωroi=1. Similar to [27], the term α denotes the resolution ratio between enhancement layer ROI area and base layer and is introduced as an approximation for bitrate ratio between the ROI slice and base layer slice.The termDc is the mismatch distortion (as described in (5)) introduced by difference between ORF and VRF. Dc is much larger than D(Q) and thus can be regarded as independent with Q. Therefore, put (8) and (9) into (13) and then set the derivative of J to zero; the Lagrange multiplier λroi can be solved as
(14)λroi=Qωbωroi·ln⁡2·Q+Δ2-12aλbdddωbωroi+ln⁡2·αQQ+Δ×12aα·Q+Δ+Q-1.
Considering the base layer Lagrange multiplier λb is determined by single layer λ selection method (11); put λb=c·(Q+Δ)2 into (14) to derive λroi as
(15)λroi=α·(Q+Δ)α·(Q+Δ)+Q·(c·Q2)=α·Q+Δα·Q+Δ+Q·λQ.Note that the derivedλroi is similar to (13). However, the rate ratio has a different form since the area of ROI has been taken into consideration.LetQPb and QPe denote the quantization parameters for base layer and enhancement layer, respectively. Then, according to the relationship between quantization steps and quantization parameters, which has been defined in H.264/AVC standard [28], QPb, QPe, and Q conform to
(16)Q+Δ=2(QPb-12)/6,Q=2(QPe-12)/6.Equation (14) can be simplified according to the quantization parameter difference between base and enhancement layers; namely, ΔQP=QPb-QPe. And finally, the modified Lagrange multiplier λroi can be obtained through the following equation for SVC enhancement layer when ROI is enabled:
(17)λroi=α·2ΔQP/6α·2ΔQP/6+1.Similar to the RDO mode decision part, the performance of the proposed RDO method with modified Lagrange multiplier is shown in Figures8 and 9 for spatial and quality SVC, respectively. The same coding parameters are used. “Enc full” and “Dec ROI” denote the scenarios when background slices are received and discarded, respectively. The proposed “mdrdo” method with Lagrange multiplier λ′ [27] and with the proposed λroi  (17) is simulated separately. And the “mdrdo” method with original single layer Lagrange multiplier (11) is performed as the anchor.RD performance for spatial SVC with newλ.
(a)
Bitrate savings for enhancement user with proposedλ
(b)
Bitrate savings for ROI user with newλRD performance for quality SVC with newλ.
(a)
Bitrate savings for enhancement user with newλ
(b)
Bitrate savings for ROI user with NewλAll the values in Figures8 and 9 are negative, which means that both Lagrange multiplier modification methods have brought benefits to ROI and enhancement layer coding compared with the original Lagrange multiplier method. Compared with λ′, the proposed λroi achieves a better performance for ROI, and the average gain is about 6% for spatial SVC (Figure 8(b)) and 7% for quality SVC (Figure 9(b)). Since λroi is always smaller than λ′, the proposed λroi shifts bits from a background slice to an ROI slice. Thus, for the whole enhancement layer, the proposed λroi leads to trifling loss of coding efficiency compared with λ′. The average loss of “mdrdo + λroi” compared with “mdrdo + λ′” is about 1% for spatial SVC (Figure 8(a)) and 1.5% for quality SVC (Figure 9(a)), respectively. Although the performance of the background area is sacrificed, it is worthwhile given that the performance of semantic-important ROI area is improved.


## 3. Simulation Results

To illustrate the overall improvement of the proposed framework, the benefits of the proposed RDO method together with modified Lagrange multiplier are shown in Figures10 and 11 for spatial and quality SVC, respectively. The same coding parameters, as described in Section 2.2, are still used. Now, thanks to the modification of Lagrange multiplier, the proposed “mdrdo + λroi” method achieves significant bitrate saving compared with the coding scheme using the original Lagrange multiplier method (denoted as “orig.”), regardless of the background slices being lost or not. Accordingly, the gain of “mdrdo + λroi” method, which uses the modified Lagrange multiplier, is also increased compared with “mv_constrain” and “half-interpolation” methods which use the original Lagrange multiplier method. For example, the average gain compared with “half-interpolation” method under “Dec ROI” scenario is now 15% (Figure 10(b)) and 12% (Figure 11(b)) for spatial and quality SVC, respectively.RD performance for spatial SVC with proposed RDO framework.
(a)
Bitrate savings for enhancement user of proposed method
(b)
Bitrate savings for ROI user of proposed methodRD performance for quality SVC with proposed RDO framework.
(a)
Bitrate savings for enhancement user of proposed method
(b)
Bitrate savings for ROI user of proposed methodWhen all background slices of enhancement layer are discarded, decoding drift occurs in both “orig.” method and the proposed method. To show the impact of such drift in both cases, visual quality comparison of “mdrdo,” “mdrdo +λroi,” and “orig.” methods is given for decoded ROI. In Figures 12 and 13, the decoded QCIF sized ROI areas under “Dec ROI” scenarios for spatial SVC with a QICF base layer (QP 22) and a CIF enhancement layer (QP 26) are illustrated. The “input” shows the original QCIF ROI area of the input sequence. It can be seen that the “orig.” method, which applies no constraints to temporal prediction, suffers a severe quality degradation, especially at the boundaries of ROI areas (Figures 12(b) and 13(b)). By taking into consideration the error propagation during the mode decision, the proposed “mdrdo” and “mdrdo + λroi” methods can effectively restrict the errors. The error propagation is unnoticeable in Figures 12(c) and 12(d) and 13(c) and 13(d).Visual quality comparison under “Dec ROI” scenario for the “bus” sequence.
(a)
Input
(b)
Orig
(c)
mdrdo
(d)
mdrdo +λroiVisual quality comparison under “Dec ROI” scenario for the “stefan” sequence.
(a)
Input
(b)
Orig
(c)
mdrdo
(d)
mdrdo +λroi


## 4. Conlusions

In this paper, a ROI coding framework based on H.264/SVC is proposed. Firstly, a rate-distortion optimized mode decision method is proposed, which loosens the temporal constraints when coding ROI slices in SVC enhancement layer. The mismatch between reference frames is considered during the RDO process, whenever the background slices are discarded or kept. Then, a new Lagrange multiplier estimation algorithm is derived to improve the coding efficiency of ROI slices. Compared with the existing constraint-based methods, such as the constrained motion vector method and the half-sample interpolation method, experimental results show that the proposed method achieves significant bitrate saving while maintaining both higher objective and subjective video quality.



---

*Source: 489089-2015-01-12.xml*