# A Perception-Driven Transcale Display Scheme for Space Image Sequences

Authors: Lingling Zi; Xin Cong; Yanfei Peng; Pei Yang
Journal: Advances in Multimedia
 (2018)

---

## Abstract

With the rapid development of multimedia technology, the way of obtaining high-quality motion reproduction for space targets has attracted much attention in recent years. This paper proposes a Perception-driven Transcale Display Scheme, which significantly improves the awareness of multimedia processing. This new scheme contains two important modules, transcale description based on visual saliency and perception-driven display of space image sequences. The former concentrates on describing the transcle feature of space targets, including three algorithms, attention region computing, frame rate conversion, and image resolution resizing. On this basis, the latter focuses on high-quality display of space movements under different scales, including three algorithms, namely, target trajectory computing, space transcale display, and space movement display. Extensive quantitative and qualitative experimental evaluations demonstrate the effectiveness of the proposed scheme.


---

## Body

## 1. Introduction

With the rapid development of computer technology, spatial rendezvous and docking needs a large amount of image information between navigation information systems. As an important information form, space image sequences play an important role, such as docking control, docking mechanism design, and track interactive control, and they need to be transmitted between these navigation information systems to measure the relative position, relative velocity, and relative attitude of spacecraft [1–4]. For the spatial image sequence, transcale is its important feature. So, demanding multimedia processing methods are different from those of traditional image processing. At present, the study of space image sequences is still at the initial stage and further exploration is still on the way. Specifically, there are mainly two unresolved problems. The first one is how to achieve the task of high-quality monitoring space targets. For example, the continuous attention to space targets should be strengthened due to the lack of the details of space movements. The second one is how to improve the ability of processing images, such as smooth motion reproduction, accurate trajectory description, and different display effects; all of them should be fine exhibited to enhance the awareness of multimedia processing. To solve the above problems, a transcale display method [5–7] emerged as a new direction for image processing and the challenges are mainly shown as follows.(1) The change of the attention scale: The existing image processing methods only use the inherent characteristics of the image sequences and lack the attention perceived by observers, which needs to be converted from the description of the whole sequences to that of space targets, so as to better capture fine details of space image sequences.(2) The change of the scale of the frame rate: The existing frame interpolation methods have the problem that the details of the interpolation frames are not clear during the process of changing the frame rates, which affects the smoothness of space image sequences. Thereby, new methods need to be studied to obtain the optimal interpolation frames.(3) The change of the resolution scale: Because of the limited screen sizes of different devices, the same image sequence is frequently required to be displayed in the different sizes. However, the important contents are not changed the scales in the existing resizing methods, so new methods need to be proposed to improve the definition of important contents under the guarantee of the global resizing effects.Facing the above challenges, we propose a Perception-driven Transcale Display Scheme (PTDS) to achieve high-quality space motion reproduction. The main contributions have three points. Firstly, we construct a transcale display framework by providing a different perspective, which significantly improves the awareness of multimedia processing. This framework contains two important modules, transcale description based on visual saliency and perception-driven display of space image sequences. Secondly, the module of transcale description is presented to solve the transcale problems, which is the core of PTDS. Finally, the module of perception-driven display is proposed to realize the trajectory description and movement display for space targets under the condition of changing scales. To sum, PTDS could serve for navigation information systems.The rest of the paper is structured as follows. Section2 discusses the framework of PTDS. Sections 3 and 4 describe in detail the formulation of PTDS, namely, transcale description and perception-driven display. Section 5 presents experimental work carried out to demonstrate the effectiveness of PTDS. Section 6 concludes the paper.


## 2. The Framework of PTDS

Figure1 shows the framework of PTDS and it contains two important modules, transcale description and perception-driven display. The first one is to improve the definitions of the important contents perceived by observers under the condition of changing different scales, including the attention scale, the scale of frame rate, and the resolution scale. The way to determine the attention regions from the observers’ viewpoint is naturally becoming a key issue. Recently, the visual saliency technique is more and more widely used in the multimedia field [8–11]. The reasons for using this technique in image processing contain the following three points. Firstly, it can provide the capability of choosing for image description, strengthen the description details of moving targets, so as to priority allocation to achieve the desired image analysis and synthesis of computing resources. Secondly, it can accomplish auxiliary construction of the flexible description scheme by discriminating the saliency regions and non-saliency regions. Finally, it can improve the cognitive ability and decision-making ability for space image sequences. Therefore, we adopt the visual saliency technique to address the above key issue. This module contains the following three parts:Figure 1
The framework of PTDS.(1) Attention region computing: It focuses on the problem of the change of the attention scale and then captures the visual attention regions for space image sequences. The calculation process mainly includes two aspects: spatial attention regions and temporal attention regions.(2) Frame rate conversion: It focuses on the problem of the change of the frame rate scale and improves the motion smoothness of space sequences. The calculation process mainly includes three steps: transform of attention blocks, acquisition of neighboring blocks and prediction of unknown pixels.(3) Image resolution resizing: It focuses on the problem of the change of the resolution scale and improves the definition of space sequences with different resolutions under the guarantee of the global visual effects. The calculation process mainly includes two steps: partition interpolation computing and seam carving operation.On the basis of transcale description, the module of perception-driven display is presented to achieve the task of high-quality spatial motion display under the condition of changing the different scales. And this module is an application of the first module and it could serve for the space navigation information processing system. This module also contains the following three parts:(1) Target trajectory computing: It could adaptively display the motion trajectory of space targets according to the setting of scale parameters. The calculation process mainly includes five steps: attention region computing, frame rate conversion, boundary computing, trajectory coordinate calculation, and motion trajectory display.(2) Space transcale display. It obtains clear motion details under different time scales and spatial scales, and the calculation process mainly includes five steps: key frame computing, display map calculation, thumbnail computing, transcale display, and thumbnail resizing.(3) Space movement display. It realizes the motion overview of the space targets in a narrative way. The calculation process mainly includes four steps: transition map setting, transition pixel calculation, transition region acquisition, and display map calculation.


## 3. The Module of Transcale Description

In this section, we will go over the individual parts of transcale description for space image sequences, including attention region computing, frame rate conversion, and image resolution resizing.


### 3.1. Attention Region Computing

The approach of attention region computing captures the important contents perceived by observers, i.e. visual attention regions, including spatial attention regions and temporal attention regions, and its flow chart is shown in Figure2. Spatial attention regions are obtained according to the characteristics of color and image signature. And temporal attention regions are calculated by adopting visual tracking. Specifically, let I[1,n]=It(1⩽t⩽n) be a space image sequence defined on the 3D space, and we use {T,VA,KT} to represent the attention region, whereTdenotes the collection of time stamp andVA denotes the attention regions containing spatial attention regionVAS and temporal attention regionVAT.KT denotes the transfer mark of attention regions. The computation ofVA is shown in (1)VAIt=VASIt,t=1VASIt∪VATIt,1<t≤nFigure 2
The flowchart of the module of attention region computing.The computation of spatial attention regions is based on the research of biological vision; i.e., human’s visual system is very sensitive to the contrast of the visual signal [12, 13]. We adopt the latest histogram color contrast method [14] to compute the spatial saliency value of any pixel inIt, shown as follows:(2)SalSξxi,yi=∑u=1nearG-Dνc,νuSalξxi,yi∑u=1nearG-Dνc,νu(3)Salξxi,yi=∑j=1,j≠ipnDξxi,yi,ξxj,yj=∑ω=1cnϕωDνc,νωIn (2) and (3), ξxi,yi and ξxj,yj denote any pixel inIt, SalS(ξxi,yi) denotes the spatial saliency value of ξxi,yi, D(ξxi,yi,ξxj,ji) denotes the color distance function, pn denotes the total number of pixels in It, cn denotes the number of distinct pixel colors inIt,νc denotes the color value of ξxi,yi, and φω denotes the probability of νc inIt. In addition, G=∑u=1nearD(νc,νu) in whichnear denotes the number of color that are nearest neighbors of νc.Then we quantifyξxj,yj, which has the larger value of spatial saliency, putting it into the set SALt={ξxj,yj}. Because ξxj,yj may be in a very small, relatively unimportant region, so it is necessary to eliminate such a pixel. There are three elimination rules, shown as follows.Rule 1.
For∀ε, if ξxj±ε,yj∉SALt and ξxj,yj±ε∉SALt, then ξxj,yj is eliminated from SALt, where ε=1,2,...s.Rule 2.
For∀ε′ and ∀ε, if ξxj±ε′,yj∈SALt and ξxj,yj±ε∉SALt, then ξxj,yj is eliminated from SALt.Rule 3.
For∀ε′′ and ∀ε, if ξxj,yj±ε′′∈SALt and ξxj±ε,yj∉SALt, then ξxj,yj is eliminated from SALt.Asε′=1,2,..,(winitial-j), ε′′=1,2,..,(hinitial-j), and winitial andhinitial are the initial resolution of image sequences. On this basis, the boundary values ofVAS(It) can be acquired by using the technology of bounding box.The computation of temporal saliency regions is based on the observations of human attention continuity [15, 16]. According to the obtained spatial saliency regions, the temporal saliency regions are computed using visual tracking. Since the attention regions could vary over time, we useKT to describe the change extent of attention regions, shown as follows:(4)KT=1,VASIt∩VASIt+1VASIt∪VASIt+1>δSR0,otherwisewhere δSR denotes a threshold value. Then timet, which satisfies the above equation, is successively put into the setKTS. In our experiments, we determine the value δSR to be 0.75.We utilize Bayesian target tracking to computeVAT(It). Let γt=[xt,yt,dxt,dyt]T be the state ofVAT(It), in which [xt,yt]T denotes the position vector and [dxt,dyt]T denotes velocity vector. The set of spatial saliency region OS1…k=VATItt=j1…j2 is denoted as the observation set ofVAT(It), in whichj1 andj2 are the adjacent values inKTS and j1<j2. Within the Bayesian inference framework, we estimate posterior probability function p(gk∣OS1…k) to compute target state gk forVAT(It). The calculation of p(gk∣gk-1) includes prediction and updating, shown as follows:(5)pgk∣OS1:k-1=∫pgk∣gk-1pgk-1∣OS1:k-1dgk-1(6)pgk∣OS1:k=pOSk∣gkpgk∣OS1:k-1pOSk∣OSk-1where gk andOSk denote the state and observation value ofk-th frame, respectively, p(gk∣gk-1) denotes the state transition model which can be solved using affine transformation, and pOSk∣gk denotes state measurement model which can be computed using structural local sparse appearance method, shown as follows:(7)pOSk∣gk=∑j=1NDigOk,χi=1M∑d=1kσidwhere d denotes the patch divided by 32×32 pixels inVAS(Im) (in which 1≤m≤k),Ndenotes the number of the divided patches, σi denotes the sparse coding ofd,M denotes the normalization term, and χi denotes the weighted sum of σi, setting set O. AndDig(∙) is the function of diagonal elements computing.


### 3.2. Frame Rate Conversion

To enhance motion smoothness of space image sequences, frame rate conversion is performed, whose core is to compute the high quality intermediate frames, i.e., how to calculate the pixel values of the intermediate frames. Here the partition method is used for calculation. Moreover, for non-attention regions, the values of pixels are unchanged. And for the attention regions, the values of pixels are accurately computed and the main computation steps contain transform of attention blocks, acquisition of neighboring blocks, and prediction of unknown pixels, which is shown in Figure3. Specifically, attention regions in any two consecutive frames, i.e.,VA(It) andVA(It+1), are divided into μ×μ overlapping image blocks, and each block is defined as an attention block,VAB for short. Note that we choose three different values for μ, namely, 8, 16, and 32, and we find that the appropriate value is 16 according to the obtained experiment results. The set ofVABdenotes as VABt={bkt} and the computation procedure for the contained pixels is elaborated as follows.Figure 3
The computation of pixels in attention regions.Firstly, eachVAB is projected using Walsh-Hadamard kernels [17], as shown below, and the results are stored in the temporary set WP=WPj∣1≤j≤ϕ:(8)Qu10=Qu20=1Qu1k=qu1k-1ρkqu1k-1Qu2k=qu2k-1ρkqu2k-1Quk=Qu1k×Qu2ks.t.qu1k-1∈Qu1k,qu2k-1∈Qu2k,ρk∈+1,-1Then the transformed result is assigned a hash value to accelerateVAB projection, setting in the corresponding hash table TBm[hm(bz)](1≤m≤ϕ).Secondly, we establish spatial and temporal expansion rules to obtain the expansion blocks for eachVAB. For ∀bkt(bkt∈VABt), each hash tableTBm needs to be operated according to expansion rules and the corresponding expansion block blt+1(blt+1∈VABt+1) can be produced, putting into the matching set. On this basis, we determine the nearest blocks from the obtained expanding blocks using the freedom searching, shown as follows:(9)bnearn+1=bkn+wαiRiIn (9), Ri∈[-1,1]×[-1,1], α is a constant value and its value is 0.5, ω is a search radius, and bnearn+1 is the best block of bkn. Then for each hash table, the following is used to compare and update the nearest blocks:(10)bnearn+1=bmapn+1,ifdisc⁡bkn,bmapn+1<disc⁡bkn,bnearn+1Finally, the predicted value of the pixels in the attention regions can be computed through a smoothing operation of the obtained nearest blocks (for detailed algorithm, see Algorithm1).Algorithm 1:Frame rate conversion algorithm.
Input: A spatial image sequenceI Output: Changed image sequenceI′ (1) Compute the frame number of I as len. (2) Give the first frame of I as the current frame I1. (3) Give the following frame of I1 as I2 and calculate their attention regions VA1 and VA2 according to the method in Section 3.1. (4) Divide VA1 and VA2 into image block STwith μ×μ. (5) Compute the transformed value of each ST using equation (8) and create hash tables. (6) FOR i=1… the number of hash tables FOR each transformed STin VA1 Obtain the expansion blocks according to spatial and temporal expansion rules; Compute the nearest blocks using the equation (9) and (10); Compute the value of pixels in VA1; END FOR END FOR (7) Merge all the transformed ST. (8) Compute interpolated frame I1′ by combining the attention region with non-attention region. (9) Give the following frame of I2 as I3,I3→I2. (10) Repeat steps (3) to (9) until the frame number of the current frame is (len-1); (11) Synthetic interpolated frames and original frames ReturnI′


### 3.3. Image Resolution Resizing

Image resizing has gained significant importance because of rapid increases in the diversity and versatility of display devices. However, existing methods do not resize results from the viewpoint of the observers. In this section, we present a new resizing method to improve the resizing quality of the important contents perceived by observers. To accomplish this, two main issues need to be addressed. The first is the need to determine the important contents from the observers’ viewpoint. The second is the need to improve the definitions of the important contents during resizing. For the first issue, we adopt the method of attention region computing (see Section3.1) and, for the second one, we introduce a method of partition interpolation with an architecture that is illustrated in Figure 4, containing two steps: partition interpolation computing and seam carving operation. Specifically, we resizeIt fromhinitial×winitial tohnew×wnew and a resizing factor 2s is(11)s=min⁡sH,sW,sH=hnewhinitial,sW=wnewwinitialFigure 4
The method of partition interpolation.The idea of partition interpolation computing is to adopt different calculation approaches for different regions of original image sequences. Naturally, for any pixeli(x,y) in It, the interpolated pixel υp is computed using (12)νp=ARt′′2x+1,2y+1,i∈VAtBRt′2x+1,2y+1,i∈BRtwhereVAt is the simplified symbol ofVA(It), which is computed in Section 3.1, BRt denotes the general region (namely, nonattention region), VAt∪BRt=It, and ARt′′(2x+1,2y+1) and BRt′(2x+1,2y+1) denote the interpolated pixels inVAt andBRt, respectively.The computation ofARt′′(2x+1,2y+1) contains two steps: initial estimation and energy modification. The initial value is calculated using (13)ARt′2x+1,2y+1=0.5ARt2x,2y+0.5ARt2x+2,2y+2,j1<j20.5ARt2x+2,2y+0.5ARt2x,2y+2,j1≥j2In (13),j1 andj2 are the determination function,j1=Z1-3Z2+Z3,j2=-3Z1+Z2+Z4, and Zi(i=1,2,3,4) is computed as follows:(14)Z1=ARt2x,2y+ARt2x+2,2y+2,Z2=ARt2x,2y+2+ARt2x+2,2yZ3=ARt2x+4,2y+ARt2x+2,2y-2+ARt2x,2y+4+ARt2x-2,2y+2Z4=ARt2x+4,2y+2+ARt2x+2,2y+4+ARt2x,2y-2+ARt2x-2,2yFor any pixel in general regions, the following is used to computeBRt′(2x+1,2y+1):(15)BRt′2x+1,2y+1=0.25BRt2x,2y+0.25BRt2x+2,2y+0.25BRt2x,2y+2+0.25BRt2x+2,2y+2Then we use seam carving method to preserve the contents of attention regions. That is to say, the pixels in the different regions are computed through energy function, shown as follows:(16)eHoGARt=max⁡HVAtx,y,ifζl≤x≤ζrandζb≤y≤ζt∂/∂xVAt+∂/∂yVAtmax⁡HVAtx,y,otherwisewhereH(VAt(x,y)) represents the oriented gradient histogram of the pixels in the attention region andζl,ζr,ζt,andζb are the boundary values ofVAt.


## 3.1. Attention Region Computing

The approach of attention region computing captures the important contents perceived by observers, i.e. visual attention regions, including spatial attention regions and temporal attention regions, and its flow chart is shown in Figure2. Spatial attention regions are obtained according to the characteristics of color and image signature. And temporal attention regions are calculated by adopting visual tracking. Specifically, let I[1,n]=It(1⩽t⩽n) be a space image sequence defined on the 3D space, and we use {T,VA,KT} to represent the attention region, whereTdenotes the collection of time stamp andVA denotes the attention regions containing spatial attention regionVAS and temporal attention regionVAT.KT denotes the transfer mark of attention regions. The computation ofVA is shown in (1)VAIt=VASIt,t=1VASIt∪VATIt,1<t≤nFigure 2
The flowchart of the module of attention region computing.The computation of spatial attention regions is based on the research of biological vision; i.e., human’s visual system is very sensitive to the contrast of the visual signal [12, 13]. We adopt the latest histogram color contrast method [14] to compute the spatial saliency value of any pixel inIt, shown as follows:(2)SalSξxi,yi=∑u=1nearG-Dνc,νuSalξxi,yi∑u=1nearG-Dνc,νu(3)Salξxi,yi=∑j=1,j≠ipnDξxi,yi,ξxj,yj=∑ω=1cnϕωDνc,νωIn (2) and (3), ξxi,yi and ξxj,yj denote any pixel inIt, SalS(ξxi,yi) denotes the spatial saliency value of ξxi,yi, D(ξxi,yi,ξxj,ji) denotes the color distance function, pn denotes the total number of pixels in It, cn denotes the number of distinct pixel colors inIt,νc denotes the color value of ξxi,yi, and φω denotes the probability of νc inIt. In addition, G=∑u=1nearD(νc,νu) in whichnear denotes the number of color that are nearest neighbors of νc.Then we quantifyξxj,yj, which has the larger value of spatial saliency, putting it into the set SALt={ξxj,yj}. Because ξxj,yj may be in a very small, relatively unimportant region, so it is necessary to eliminate such a pixel. There are three elimination rules, shown as follows.Rule 1.
For∀ε, if ξxj±ε,yj∉SALt and ξxj,yj±ε∉SALt, then ξxj,yj is eliminated from SALt, where ε=1,2,...s.Rule 2.
For∀ε′ and ∀ε, if ξxj±ε′,yj∈SALt and ξxj,yj±ε∉SALt, then ξxj,yj is eliminated from SALt.Rule 3.
For∀ε′′ and ∀ε, if ξxj,yj±ε′′∈SALt and ξxj±ε,yj∉SALt, then ξxj,yj is eliminated from SALt.Asε′=1,2,..,(winitial-j), ε′′=1,2,..,(hinitial-j), and winitial andhinitial are the initial resolution of image sequences. On this basis, the boundary values ofVAS(It) can be acquired by using the technology of bounding box.The computation of temporal saliency regions is based on the observations of human attention continuity [15, 16]. According to the obtained spatial saliency regions, the temporal saliency regions are computed using visual tracking. Since the attention regions could vary over time, we useKT to describe the change extent of attention regions, shown as follows:(4)KT=1,VASIt∩VASIt+1VASIt∪VASIt+1>δSR0,otherwisewhere δSR denotes a threshold value. Then timet, which satisfies the above equation, is successively put into the setKTS. In our experiments, we determine the value δSR to be 0.75.We utilize Bayesian target tracking to computeVAT(It). Let γt=[xt,yt,dxt,dyt]T be the state ofVAT(It), in which [xt,yt]T denotes the position vector and [dxt,dyt]T denotes velocity vector. The set of spatial saliency region OS1…k=VATItt=j1…j2 is denoted as the observation set ofVAT(It), in whichj1 andj2 are the adjacent values inKTS and j1<j2. Within the Bayesian inference framework, we estimate posterior probability function p(gk∣OS1…k) to compute target state gk forVAT(It). The calculation of p(gk∣gk-1) includes prediction and updating, shown as follows:(5)pgk∣OS1:k-1=∫pgk∣gk-1pgk-1∣OS1:k-1dgk-1(6)pgk∣OS1:k=pOSk∣gkpgk∣OS1:k-1pOSk∣OSk-1where gk andOSk denote the state and observation value ofk-th frame, respectively, p(gk∣gk-1) denotes the state transition model which can be solved using affine transformation, and pOSk∣gk denotes state measurement model which can be computed using structural local sparse appearance method, shown as follows:(7)pOSk∣gk=∑j=1NDigOk,χi=1M∑d=1kσidwhere d denotes the patch divided by 32×32 pixels inVAS(Im) (in which 1≤m≤k),Ndenotes the number of the divided patches, σi denotes the sparse coding ofd,M denotes the normalization term, and χi denotes the weighted sum of σi, setting set O. AndDig(∙) is the function of diagonal elements computing.


## 3.2. Frame Rate Conversion

To enhance motion smoothness of space image sequences, frame rate conversion is performed, whose core is to compute the high quality intermediate frames, i.e., how to calculate the pixel values of the intermediate frames. Here the partition method is used for calculation. Moreover, for non-attention regions, the values of pixels are unchanged. And for the attention regions, the values of pixels are accurately computed and the main computation steps contain transform of attention blocks, acquisition of neighboring blocks, and prediction of unknown pixels, which is shown in Figure3. Specifically, attention regions in any two consecutive frames, i.e.,VA(It) andVA(It+1), are divided into μ×μ overlapping image blocks, and each block is defined as an attention block,VAB for short. Note that we choose three different values for μ, namely, 8, 16, and 32, and we find that the appropriate value is 16 according to the obtained experiment results. The set ofVABdenotes as VABt={bkt} and the computation procedure for the contained pixels is elaborated as follows.Figure 3
The computation of pixels in attention regions.Firstly, eachVAB is projected using Walsh-Hadamard kernels [17], as shown below, and the results are stored in the temporary set WP=WPj∣1≤j≤ϕ:(8)Qu10=Qu20=1Qu1k=qu1k-1ρkqu1k-1Qu2k=qu2k-1ρkqu2k-1Quk=Qu1k×Qu2ks.t.qu1k-1∈Qu1k,qu2k-1∈Qu2k,ρk∈+1,-1Then the transformed result is assigned a hash value to accelerateVAB projection, setting in the corresponding hash table TBm[hm(bz)](1≤m≤ϕ).Secondly, we establish spatial and temporal expansion rules to obtain the expansion blocks for eachVAB. For ∀bkt(bkt∈VABt), each hash tableTBm needs to be operated according to expansion rules and the corresponding expansion block blt+1(blt+1∈VABt+1) can be produced, putting into the matching set. On this basis, we determine the nearest blocks from the obtained expanding blocks using the freedom searching, shown as follows:(9)bnearn+1=bkn+wαiRiIn (9), Ri∈[-1,1]×[-1,1], α is a constant value and its value is 0.5, ω is a search radius, and bnearn+1 is the best block of bkn. Then for each hash table, the following is used to compare and update the nearest blocks:(10)bnearn+1=bmapn+1,ifdisc⁡bkn,bmapn+1<disc⁡bkn,bnearn+1Finally, the predicted value of the pixels in the attention regions can be computed through a smoothing operation of the obtained nearest blocks (for detailed algorithm, see Algorithm1).Algorithm 1:Frame rate conversion algorithm.
Input: A spatial image sequenceI Output: Changed image sequenceI′ (1) Compute the frame number of I as len. (2) Give the first frame of I as the current frame I1. (3) Give the following frame of I1 as I2 and calculate their attention regions VA1 and VA2 according to the method in Section 3.1. (4) Divide VA1 and VA2 into image block STwith μ×μ. (5) Compute the transformed value of each ST using equation (8) and create hash tables. (6) FOR i=1… the number of hash tables FOR each transformed STin VA1 Obtain the expansion blocks according to spatial and temporal expansion rules; Compute the nearest blocks using the equation (9) and (10); Compute the value of pixels in VA1; END FOR END FOR (7) Merge all the transformed ST. (8) Compute interpolated frame I1′ by combining the attention region with non-attention region. (9) Give the following frame of I2 as I3,I3→I2. (10) Repeat steps (3) to (9) until the frame number of the current frame is (len-1); (11) Synthetic interpolated frames and original frames ReturnI′


## 3.3. Image Resolution Resizing

Image resizing has gained significant importance because of rapid increases in the diversity and versatility of display devices. However, existing methods do not resize results from the viewpoint of the observers. In this section, we present a new resizing method to improve the resizing quality of the important contents perceived by observers. To accomplish this, two main issues need to be addressed. The first is the need to determine the important contents from the observers’ viewpoint. The second is the need to improve the definitions of the important contents during resizing. For the first issue, we adopt the method of attention region computing (see Section3.1) and, for the second one, we introduce a method of partition interpolation with an architecture that is illustrated in Figure 4, containing two steps: partition interpolation computing and seam carving operation. Specifically, we resizeIt fromhinitial×winitial tohnew×wnew and a resizing factor 2s is(11)s=min⁡sH,sW,sH=hnewhinitial,sW=wnewwinitialFigure 4
The method of partition interpolation.The idea of partition interpolation computing is to adopt different calculation approaches for different regions of original image sequences. Naturally, for any pixeli(x,y) in It, the interpolated pixel υp is computed using (12)νp=ARt′′2x+1,2y+1,i∈VAtBRt′2x+1,2y+1,i∈BRtwhereVAt is the simplified symbol ofVA(It), which is computed in Section 3.1, BRt denotes the general region (namely, nonattention region), VAt∪BRt=It, and ARt′′(2x+1,2y+1) and BRt′(2x+1,2y+1) denote the interpolated pixels inVAt andBRt, respectively.The computation ofARt′′(2x+1,2y+1) contains two steps: initial estimation and energy modification. The initial value is calculated using (13)ARt′2x+1,2y+1=0.5ARt2x,2y+0.5ARt2x+2,2y+2,j1<j20.5ARt2x+2,2y+0.5ARt2x,2y+2,j1≥j2In (13),j1 andj2 are the determination function,j1=Z1-3Z2+Z3,j2=-3Z1+Z2+Z4, and Zi(i=1,2,3,4) is computed as follows:(14)Z1=ARt2x,2y+ARt2x+2,2y+2,Z2=ARt2x,2y+2+ARt2x+2,2yZ3=ARt2x+4,2y+ARt2x+2,2y-2+ARt2x,2y+4+ARt2x-2,2y+2Z4=ARt2x+4,2y+2+ARt2x+2,2y+4+ARt2x,2y-2+ARt2x-2,2yFor any pixel in general regions, the following is used to computeBRt′(2x+1,2y+1):(15)BRt′2x+1,2y+1=0.25BRt2x,2y+0.25BRt2x+2,2y+0.25BRt2x,2y+2+0.25BRt2x+2,2y+2Then we use seam carving method to preserve the contents of attention regions. That is to say, the pixels in the different regions are computed through energy function, shown as follows:(16)eHoGARt=max⁡HVAtx,y,ifζl≤x≤ζrandζb≤y≤ζt∂/∂xVAt+∂/∂yVAtmax⁡HVAtx,y,otherwisewhereH(VAt(x,y)) represents the oriented gradient histogram of the pixels in the attention region andζl,ζr,ζt,andζb are the boundary values ofVAt.


## 4. The Module of Perception-Driven Display

In this section, we describe in detail the formulation of our proposed the module of perception-driven display, containing three algorithms, namely, target trajectory computing, space transcale display, and space movement display. To some extent, all of them are an application of the module of transcale description and could be directly applied in the navigation information systems.(  1) Target Trajectory Computing. To display the motion trajectories of space targets under different scales, the approach of target trajectory computing is presented and it contains the following five steps:Step 1.
For each frame inIt (1≤t≤n),VAt is computed by utilizing the method of attention region computing.Step 2.
Using the method of frame rate conversion,I[1,n] is changed intoHI[1,N′]={HIt}(1≤t≤N′ and N′>n) in whichN is the number of changed frames andVAt is updated inHIt.Step 3.
The four boundary values are computed for eachVAt, including left boundaryBVL, right boundaryBVR, top boundaryBVT, and bottom boundaryBVB:(17)BVL=minxi∈VAt⁡xi,BVR=maxxi∈VAt⁡xi,BVT=minyi∈VAt⁡yi,BVB=maxyi∈VAt⁡yiStep 4.
For eachVAt, trajectory coordinates are calculated, including the value of horizontal coordinateVArow and the value of vertical coordinateVAcolumn:(18)VArow=BVL+BVR-BVL2,VAcolumn=BVT+BVB-BVT2Step 5.
Motion trajectoryATυ with the different scaleυ is computed using the following formula: (19)ATυ=HI1∪VA1∪VA2∪VA3∪…∪VAN,υ=1HI1∪VA1∪VA3∪VA5∪…∪VA1+2∗N-1/2,υ=2……HI1∪VA1∪VA1+σ∪VA1+2σ∪…∪VA1+σ∗N-1/2,υ=σ(  2) Space Transcale Display. The aim of transcale display is not only to show motion pictures at different time scales but also to display with the different resolution scales. This algorithm mainly includes the following five steps:Step 1.
For eachIt, the key frame Kj(j=1…m) is computed and then the corresponding attention regionVAKj is obtained.Step 2.
The transcale display mapKM is captured by merging the attention regions under the condition of different key frames.Step 3.
For eachVAKj(j=1…m), the corresponding thumbnailVUj is computed by reducing the original image to the same resolution.Step 4.
The transcale display mapKM′ forIt is computed:(20)KM′=KM∪VU1∪VU2∪…∪VUmStep 5.
Using the method of image resolution resizing,VUj could also have high-quality display with the different resolutions.(  3) Space Movement Display. To outline the motion process of space targets, a movement display algorithm is proposed, which focuses on smooth transition of space movement. Specifically, letKp andKq be the continuous key frames inIt and the transition regions of them beKRp(u,v) andKRq(u′,v′), respectively, in whichu,u′∈[1,height],v∈[weight-r+1,weight], v′∈ [1,r],r denotes the transition radius, andheightandweightare the current resolution of key frames. The process of this algorithm consists of the following four steps:Step 1.
BothKRp andKRq are resized to target resolutionheight×2r, and the transition region maps TMapi(i=p,q) are obtained.Step 2.
The value of pixelKR(u,v) in the transition region is computed:(21)KRx,y=c1×TMappx,y+c2×TMapqx,yin whichc1 andc2 are transition parameters,c1=1-c2,c2=v/2r, andx andy are the position coordinates.Step 3.
RepeatSteps 1 and 2 until all the keyframe transition regions are calculated, and then new key frame K′ is recorded.Step 4.
All the obtained key frames are merged to form display mapMD:(22)MD=∑j=1μK′jwhere μ is the total number of key frames.


## 5. Experimental Results and Discussion

In this section, we conduct the quantitative and qualitative experiments to evaluate the performance of PTDS, including the transcale performance (see Sections5.1, 5.2, and 5.3) and the display performance (see Sections 5.4, 5.5, and 5.6). We downloaded space video from Youku and they are segmented the video clips. Then we changed these clips to image sequences and formed our data shown in the Table 1. In all our experiments, we used the MATLAB platform and a PC with a 2.60 GHz Intel(R) Pentium(R) Dual-Core CPU processor with 1.96 GB of main memory.Table 1
Experimental data.
Sequence ID Space image sequence Initial resolution Number of frames S1 Aircraft automatic control 320×240 204 S2 Aircraft automatic homing 320×240 230 S3 Aircraft lift 640×480 238 S4 Aircraft docking 640×480 277 S5 Spacecraft detection 352×240 157 S6 Spacecraft launch 352×240 101 S7 Launch base 320×145 208 S8 Spacecraft flight 576×432 161 S9 Aircraft flight 1024×436 130


### 5.1. Evaluation of Attention Scale

Figure5 shows the obtained attention regions using our method, in which each row exhibits the results of different space image sequences and each column exhibits sample frames of each sequences. In addition, each subfigure consists of three parts: the purple-bordered region in the left marks the attention regions, the blue-bordered region in the upper right marks the spatial attention region, and the green-border region in the lower right marks the temporal attention region. Specifically, the first row shows the results for frames 100, 128, and 134 from S1, and the second row shows the results for frames 119, 121, and 135 from S2. Observing these rows, we can see that the spatial attention regions only capture part of space targets, for example, the result in the first row and first column. This means that both the calculations of spatial attention regions and temporal attention regions are necessary for obtaining the accurate attention regions, especially for image sequences with large motion amplitude. Moreover, the third row shows the results for frame 10, 24 and 30 from S3, and the last row shows the results for frames 8, 45, and 88 from S4. These sequences have a relatively complicated background, which causes great interference to the detection of attention regions. However, the proposed method still obtains good results and these results lay the foundation for transcale display.Figure 5
The obtained attention regions using PTDS from sample frames of different image sequences (from top to down, sequences S1, S2, S3, and S4). In each subfigure, the purple-bordered regions mark the attention regions, the blue-bordered regions mark the spatial attention regions, and the green-border regions mark the temporal attention regions.Figure6 shows the quantitative comparison results for the other methods, including image signature detection (ISD) [18] and global contrast detection (GCD) [14]. Here we adopt the overlap score [19] as a quantitative indicator, which is defined as the following formula:(23)score=RS∩RGRS∪RGwhereRS denotes the obtained attention result andRG denotes the corresponding ground truth bounding box. The larger the overlap scores are, the more accurate the results are. From this figure, we can see the average overlap scores using three methods. It is noted that our method achieves a larger score than those of the other methods, and the results show that the obtained attention regions using our method are closer to the true significant region of original image sequences.Figure 6
Comparison results of the average overlap score.


### 5.2. Evaluation of Frame Rate Scale

The quality of interpolated frames is a key factor to evaluate the change of the scale of frame rate, and in experiments, we removed one in every two consecutive frames of the original image sequence and reconstructed the removed one adopting the different methods, including three-step search (TSS), adaptive rood pattern search (ARPS), Horn and Schunck (H&S), CNF [20], CSH [21] and our proposed method. Figures 7 and 8 show the frame interpolation results for sequence S5 and S6, and from the red-bordered region of each subfigure, the visual differences among algorithms can be determined. TSS and ARPS exhibit a poor interpolation effect, H&S and CNF introduce a suspension effect, CSH produced disappointing effects, and PTDS shows the clear details of spacecraft. From these figures, it is evident that the proposed method shows comparatively better performance in terms of visual quality.Figure 7
Interpolation frame 38 of sequence S5.
(a)
TSS (b)
ARPS (c)
H&S (d)
CNF (e)
CSH (f)
PTDSFigure 8
Interpolation frame 4 of sequence S6.
(a)
TSS (b)
ARPS (c)
H&S (d)
CNF (e)
CSH (f)
PTDSFigure9 shows the average quantitative values of each image sequence by using the different methods. The left part demonstrates the average PSNR values, which is traditional quantitative measure in the term of accuracy, and the right part demonstrates the MSSIM results, which assess the image visibility quality from the viewpoint of image formation, under the assumption of a correlation between human visual perception and image structural information. In this figure, it is obvious that our method could achieve the highest average values.Figure 9
Comparison results of average values in the term of PSNR and MSSIM.


### 5.3. Evaluation of Resolution Scale

We resize the original image sequences by using the different methods, including scaling, the best cropping, improved seam carving (ISC) [22], and our method, and then the obtained results are compared to evaluate the performance on the change of resolution scales.Figure10 shows the comparison results of sequence S7 from 320×145 to 500×230. Figure 10(a) shows the original frames, in which the numbers of frames are 131, 138, 147, 151, 174, 179, 189, 203, and 208. Figure 10(b) shows that using scaling method, the launch base becomes vaguer than before. Figure 10(c) shows that using the best cropping method, the launch base is only partly displayed, resulting in the original information becoming missing. Figure 10(d) shows that using the improved seam carving method, the prominent part of launch base becomes smaller than before, indicating that ISC is not suitable for image enlarging. Figure 10(e) shows that our method clearly displays the prominent objects of the original frames and ensures a global visual effect when the resolution scales are changed. Similarly, resizing results of the sequence S8 are shown in Figure 11.Figure 10
Comparison results of sequence S7 using four methods when the resolution is resized from320×145 to 500×230. (a) The original frames, (b) scaling, (c) the best cropping, (d) ISC, and (e) PTDS.
(a) (b) (c) (d) (e)Figure 11
Comparison results of sequence S8 using four methods when the resolution is resized from576×432 to 600×800. (a) The original frames, (b) scaling, (c) the best cropping, (d) ISC, and (e) PTDS.
(a) (b) (c) (d) (e)


### 5.4. Evaluation of Target Trajectory Computing

Figure12 shows target trajectory of sequence S2 at scales υ=1, υ=10, and υ=20. As can be seen from this figure, as the value ofυ increases, the movement amplitude of space targets becomes greater and the description of the trajectory could be coarser, which clearly characterizes the change of the movement of automatic homing. Similarly, target trajectory of sequence S5 is shown in Figure 13.Figure 12
Target trajectory display of sequence S2.
(a)
υ = 1 (b)
υ = 10 (c)
υ = 20Figure 13
Target trajectory display of sequence S5.
(a)
υ = 1 (b)
υ = 10 (c)
υ = 20Figure14 shows the detailed positions of target trajectories corresponding to Figure 12, which are described in way of computing the position of the center pixel in attention regions. And it can be seen that the larger the scaleυ is, the fewer the description points are and the rougher the position trajectory of center point is and vice versa. Similarly, Figure 15 shows target trajectories corresponding to Figure 13. In short, PDTS exhibits good performance for displaying space target trajectories under different scales, which can lay a foundation for space tasks, such as condition monitoring and motion tracking.Figure 14
Target trajectory of the center pixel corresponding to sequence S2. From left to right,υ=1, υ=10, and υ=20.Figure 15
Target trajectory of the center pixel corresponding to sequence S5. From left to right,υ=1, υ=10, and υ=20.


### 5.5. Evaluation of Space Transcale Display

Figure16 gives transcale displays for sequences S5 and S9, in which the attention regions are shown in the left part of each subfigure. Figure 16(a) shows both the whole process of detecting from near to far and the motion details of the space targets. Similarly, the process of movement from far to near is given in Figure 16(b). From this figure, we can see that the proposed space transcale display algorithm can clearly exhibit the motion process of the space targets at different scales. And by observing the accompanying attention regions, motion details of space targets, such as target pose and operating status, are displayed at different distances and time, which fully reflects the transcale characteristics of the PTDS scheme.Figure 16
Space transcale display of space image sequences.
(a)
S5 (b)
S9Figure17 summarizes the objective evaluation values of space transcale display corresponding to Figure 16, including average gradient, edge intensity, spatial frequency, image definition, and quality score. With the increase of the number of key frames, the values of the above evaluation index are also increasing, indicating that the space transcale display algorithm can obtain high-quality transcale display map.Figure 17
Objective evaluation.


### 5.6. Evaluation of Space Movement Display

Figure18 shows Space movement display for space image sequences, in which a, b, and c are, respectively, S3, S4, and S6. By adopting the narrative diagram, the complete movement of the space targets are clearly exhibited. Figures 18(b), 18(d), and 18(f) are their corresponding transition area details, depicting the comparison results using direct fusion method in the left and our method in the right. For the left part, it is obvious to see a straight line to distinguish between two key frames, while for the right part, smooth and delicate transition areas are displayed to achieve a perfect representation of the movement process.Figure 18
Space movement display for S3, S4, and S6.
(a) (b) (c) (d) (e) (f)


## 5.1. Evaluation of Attention Scale

Figure5 shows the obtained attention regions using our method, in which each row exhibits the results of different space image sequences and each column exhibits sample frames of each sequences. In addition, each subfigure consists of three parts: the purple-bordered region in the left marks the attention regions, the blue-bordered region in the upper right marks the spatial attention region, and the green-border region in the lower right marks the temporal attention region. Specifically, the first row shows the results for frames 100, 128, and 134 from S1, and the second row shows the results for frames 119, 121, and 135 from S2. Observing these rows, we can see that the spatial attention regions only capture part of space targets, for example, the result in the first row and first column. This means that both the calculations of spatial attention regions and temporal attention regions are necessary for obtaining the accurate attention regions, especially for image sequences with large motion amplitude. Moreover, the third row shows the results for frame 10, 24 and 30 from S3, and the last row shows the results for frames 8, 45, and 88 from S4. These sequences have a relatively complicated background, which causes great interference to the detection of attention regions. However, the proposed method still obtains good results and these results lay the foundation for transcale display.Figure 5
The obtained attention regions using PTDS from sample frames of different image sequences (from top to down, sequences S1, S2, S3, and S4). In each subfigure, the purple-bordered regions mark the attention regions, the blue-bordered regions mark the spatial attention regions, and the green-border regions mark the temporal attention regions.Figure6 shows the quantitative comparison results for the other methods, including image signature detection (ISD) [18] and global contrast detection (GCD) [14]. Here we adopt the overlap score [19] as a quantitative indicator, which is defined as the following formula:(23)score=RS∩RGRS∪RGwhereRS denotes the obtained attention result andRG denotes the corresponding ground truth bounding box. The larger the overlap scores are, the more accurate the results are. From this figure, we can see the average overlap scores using three methods. It is noted that our method achieves a larger score than those of the other methods, and the results show that the obtained attention regions using our method are closer to the true significant region of original image sequences.Figure 6
Comparison results of the average overlap score.


## 5.2. Evaluation of Frame Rate Scale

The quality of interpolated frames is a key factor to evaluate the change of the scale of frame rate, and in experiments, we removed one in every two consecutive frames of the original image sequence and reconstructed the removed one adopting the different methods, including three-step search (TSS), adaptive rood pattern search (ARPS), Horn and Schunck (H&S), CNF [20], CSH [21] and our proposed method. Figures 7 and 8 show the frame interpolation results for sequence S5 and S6, and from the red-bordered region of each subfigure, the visual differences among algorithms can be determined. TSS and ARPS exhibit a poor interpolation effect, H&S and CNF introduce a suspension effect, CSH produced disappointing effects, and PTDS shows the clear details of spacecraft. From these figures, it is evident that the proposed method shows comparatively better performance in terms of visual quality.Figure 7
Interpolation frame 38 of sequence S5.
(a)
TSS (b)
ARPS (c)
H&S (d)
CNF (e)
CSH (f)
PTDSFigure 8
Interpolation frame 4 of sequence S6.
(a)
TSS (b)
ARPS (c)
H&S (d)
CNF (e)
CSH (f)
PTDSFigure9 shows the average quantitative values of each image sequence by using the different methods. The left part demonstrates the average PSNR values, which is traditional quantitative measure in the term of accuracy, and the right part demonstrates the MSSIM results, which assess the image visibility quality from the viewpoint of image formation, under the assumption of a correlation between human visual perception and image structural information. In this figure, it is obvious that our method could achieve the highest average values.Figure 9
Comparison results of average values in the term of PSNR and MSSIM.


## 5.3. Evaluation of Resolution Scale

We resize the original image sequences by using the different methods, including scaling, the best cropping, improved seam carving (ISC) [22], and our method, and then the obtained results are compared to evaluate the performance on the change of resolution scales.Figure10 shows the comparison results of sequence S7 from 320×145 to 500×230. Figure 10(a) shows the original frames, in which the numbers of frames are 131, 138, 147, 151, 174, 179, 189, 203, and 208. Figure 10(b) shows that using scaling method, the launch base becomes vaguer than before. Figure 10(c) shows that using the best cropping method, the launch base is only partly displayed, resulting in the original information becoming missing. Figure 10(d) shows that using the improved seam carving method, the prominent part of launch base becomes smaller than before, indicating that ISC is not suitable for image enlarging. Figure 10(e) shows that our method clearly displays the prominent objects of the original frames and ensures a global visual effect when the resolution scales are changed. Similarly, resizing results of the sequence S8 are shown in Figure 11.Figure 10
Comparison results of sequence S7 using four methods when the resolution is resized from320×145 to 500×230. (a) The original frames, (b) scaling, (c) the best cropping, (d) ISC, and (e) PTDS.
(a) (b) (c) (d) (e)Figure 11
Comparison results of sequence S8 using four methods when the resolution is resized from576×432 to 600×800. (a) The original frames, (b) scaling, (c) the best cropping, (d) ISC, and (e) PTDS.
(a) (b) (c) (d) (e)


## 5.4. Evaluation of Target Trajectory Computing

Figure12 shows target trajectory of sequence S2 at scales υ=1, υ=10, and υ=20. As can be seen from this figure, as the value ofυ increases, the movement amplitude of space targets becomes greater and the description of the trajectory could be coarser, which clearly characterizes the change of the movement of automatic homing. Similarly, target trajectory of sequence S5 is shown in Figure 13.Figure 12
Target trajectory display of sequence S2.
(a)
υ = 1 (b)
υ = 10 (c)
υ = 20Figure 13
Target trajectory display of sequence S5.
(a)
υ = 1 (b)
υ = 10 (c)
υ = 20Figure14 shows the detailed positions of target trajectories corresponding to Figure 12, which are described in way of computing the position of the center pixel in attention regions. And it can be seen that the larger the scaleυ is, the fewer the description points are and the rougher the position trajectory of center point is and vice versa. Similarly, Figure 15 shows target trajectories corresponding to Figure 13. In short, PDTS exhibits good performance for displaying space target trajectories under different scales, which can lay a foundation for space tasks, such as condition monitoring and motion tracking.Figure 14
Target trajectory of the center pixel corresponding to sequence S2. From left to right,υ=1, υ=10, and υ=20.Figure 15
Target trajectory of the center pixel corresponding to sequence S5. From left to right,υ=1, υ=10, and υ=20.


## 5.5. Evaluation of Space Transcale Display

Figure16 gives transcale displays for sequences S5 and S9, in which the attention regions are shown in the left part of each subfigure. Figure 16(a) shows both the whole process of detecting from near to far and the motion details of the space targets. Similarly, the process of movement from far to near is given in Figure 16(b). From this figure, we can see that the proposed space transcale display algorithm can clearly exhibit the motion process of the space targets at different scales. And by observing the accompanying attention regions, motion details of space targets, such as target pose and operating status, are displayed at different distances and time, which fully reflects the transcale characteristics of the PTDS scheme.Figure 16
Space transcale display of space image sequences.
(a)
S5 (b)
S9Figure17 summarizes the objective evaluation values of space transcale display corresponding to Figure 16, including average gradient, edge intensity, spatial frequency, image definition, and quality score. With the increase of the number of key frames, the values of the above evaluation index are also increasing, indicating that the space transcale display algorithm can obtain high-quality transcale display map.Figure 17
Objective evaluation.


## 5.6. Evaluation of Space Movement Display

Figure18 shows Space movement display for space image sequences, in which a, b, and c are, respectively, S3, S4, and S6. By adopting the narrative diagram, the complete movement of the space targets are clearly exhibited. Figures 18(b), 18(d), and 18(f) are their corresponding transition area details, depicting the comparison results using direct fusion method in the left and our method in the right. For the left part, it is obvious to see a straight line to distinguish between two key frames, while for the right part, smooth and delicate transition areas are displayed to achieve a perfect representation of the movement process.Figure 18
Space movement display for S3, S4, and S6.
(a) (b) (c) (d) (e) (f)


## 6. Conclusions

In this paper, we focus on the transcale problem of spatial image sequences and propose a novel display scheme which improves the awareness of multimedia processing. The contribution of this scheme manifests in two aspects. On the one hand, space targets sustained attention by using visual saliency technology and then the details of the movement of the targets are enhanced. On the other hand, the motion of spatial image can be smoothly reproduced and effectively showed on the display device with different sizes and resolutions. Experimental results show that the proposed method outperforms the representative methods in terms of image visualization and quantitative measures. In the future, we will further study the transcale characteristics of space image sequences and explore the mutual influence between the different scales. On this basis, we will also establish a robust transcale processing mechanism to better serve spatial rendezvous and docking.



---

*Source: 6045701-2018-10-04.xml*