# An Improved Fractional-Order Optical Flow Model for Motion Estimation

Authors: Bin Zhu; Lianfang Tian; Qiliang Du; Qiuxia Wu; Lixin Shi
Journal: Mathematical Problems in Engineering
 (2018)

---

## Abstract

The Horn and Schunck (HS) optical flow model cannot preserve discontinuity of motion estimation and has low accuracy especially for the image sequence, which includes complex texture. To address this problem, an improved fractional-order optical flow model is proposed. In particular, the fractional-order Taylor series expansion is applied in the brightness constraint equation of the HS model. The fractional-order flow field derivative is also used in the smoothing constraint equation. The Euler-Lagrange equation is utilized for the minimization of the energy function of the fractional-order optical flow model. Two-dimensional fractional differential masks are proposed and applied to the calculation of the model simplification. Considering the spatiotemporal memory property of fractional-order, the algorithm preserves the edge discontinuity of the optical flow field while improving the accuracy of the estimation of the dense optical flow field. Experiments on Middlebury datasets demonstrate the predominance of our proposed algorithm.


---

## Body

## 1. Introduction

Since the optical flow was proposed by Gibson in 1950, many scholars have done research on it. In these studies, the HS optical flow model proposed by Horn and Schunck [1] in 1981 achieved high reliability and accuracy and accordingly began to attract more and more scholarly attention. But the accuracy of the motion estimation with the HS optical flow model would be greatly reduced in the image sequence, which would include complex texture and nonrigid motion; meanwhile the discontinuity of motion estimation is difficult to retain.Many scholars have devoted themselves to the modification or improvement of HS model to increase the accuracy [2], deal with large displacement motion [3], solve the occlusion problem [4], and track nonrigid motion targets [5, 6]. All these models are based on first-order differentiation techniques.Lv et al. [7] added a nonlinear fourth-order diffusive term to the Euler-Lagrange equations of the variational TV model, and the improved model can preserve sharp jump discontinuities, but it would deteriorate when too much noise exists. A high-order total variation minimization method for image deblurring and denoising was proposed by Chan et al. [8]; the model can filter out noise while the edge discontinuity was retained. But details of the edge were neglected.Because of its ability to preserve the texture details of the smooth region while highlighting the image edge feature and its spatiotemporal memory of the target point neighborhood, a fractional differential is applied to many image processing fields [9], such as image denoising [10], image enhancement [11], and motion estimation [12–16]. In [12, 13], the fractional-order smoothing constraint equation was used in the HS optical flow model to preserve the discontinuity of motion estimation, but it does not consider the correlation of the pixel intensity. Although the edge discontinuity of the image can be preserved to a certain extent, the accuracy of the optical flow field estimation is not high. In order to further enhance the accuracy of motion estimation, an improved fractional-order optical flow model is proposed in this paper.The main characteristics of the present paper can be summarized as follows:(1)
Fractional-order was first used in a brightness constraint equation of the HS optical flow model.(2)
A two-dimensional fractional-order differential mask, which was used to compute dual fractional-order differentials, was first proposed and applied to the simplification of the improved fractional-order optical flow model.(3)
Combined with the Euler-Lagrange equation and two-dimensional fractional-order differential masks, the iteration formula of the optical flow field estimation of the improved fractional-order optical flow model was constructed, which is different from any of the former counterparts.


## 2. The Construction of Two-Dimensional Fractional-Order Differential Masks

The mathematical preliminaries of the fractional-order theory used in this paper can be found in [17, 18].In this chapter, six two-dimensional fractional differential masks were presented. There areWxx(μ)(k), Wxy(μ)(k,z), Wxt(μ)(k,p), Wyy(μ)(z), Wyx(μ)(k,z), Wyt(μ)(z,p), where μ is the order of fractional differentials.We defineWxx(μ)(k) as the two-dimensional fractional differential mask of dual derivatives on the x-axis. The two-dimensional fractional derivatives of order μ of a discrete image brightness function I(i,j,t) on the x-axis can be defined as(1)DxμDxμIi,j,t=∑k=-LLWxxμkIi-k,j,t.L is the mask size, when L=2:(2)Wxxμk=-μ-μ+12-μ2-μ-μ-μ+12.Similarly,(3)DxμDyμIi,j,t=∑z=-LL∑k=-LLWxyμz,kIi+z,j+k,t,where k,z∈Z,k≤L,z≤L, when L=2:(4)Wxyμz,k=-μ-μ+12-μ-μ+12-μ-μ+12-μ-μ+12-μ-μ+12-μ-μ+12-μ-μ-μ-μ-μ+12-μ-μ+12-μ8-μ-μ-μ+12-μ-μ+12-μ-μ-μ-μ-μ+12-μ-μ+12-μ-μ+12-μ-μ+12-μ-μ+12-μ-μ+12.Also,(5)DxαDtαIi,j,t=∑p=0T∑k=0LWxtμk,pIi-k,j,tIi,j,t-p,where k,p∈Z, k≤L, P≤T. When T=L=2,(6)Wxtμk,p=1-μ-μ-μ+12-μ-μ-μ-μ+12-μ-μ+12-μ-μ+12-μ-μ+12.What is more, we can getWyy(μ)(z)=WxxT(μ)(k)Wyx(μ)(k,z)=Wxy(μ)(k,z), Wyt(μ)(z,p)=Wxt(μ)(k,p).Normalization should be done in all the mask items. More details about the expansion of dual fractional-order derivatives can be found in [12].


## 3. Improved Fractional-Order Optical Flow Model

### 3.1. The Fractional-Order Optical Flow Model

We substitute the brightness constraint equation of the HS model with the fractional-order Taylor series expansion and get the fractional-order brightness constraint equation as(7)DxαIΓα+1uα+DyαIΓα+1να+DtαIΓα+1=0,where I is the abbreviation of I(i,j,t).Combined with (2.19) and (2.20) in [12], we can get the fractional-order optical flow model in this paper: (8)EU=∫ΩDxαIΓα+1uα+DyαIΓα+1να+DtαIΓα+12+λDβu2+Dβv2dX.


### 3.2. Numerical Algorithm

The Euler-Lagrange equation [19] was applied in this paper to minimize energy function E(U).After simplification, we get the equation as follows:(9)DxαIΓα+1uα-1M+λDxβDxβu+DyβDyβu=0DyαIΓα+1vα-1M+λDxβDxβv+DyβDyβv=0,where (10)M=DxαIΓα+1uα+DyαIΓα+1να+DtαIΓα+1.After a series of computations, the iteration formula for the proposed fractional-order optical flow model can finally be written as(11)∇αIxxu2α-1+∇αIxyuα-1vα+∇αIxtuα-1+λ∑k=-LL∑z=-LLWxyβk,zui-k,j-z=0∇αIyyv2α-1+∇αIyxvα-1uα+∇αIytvα-1+λ∑k=-LL∑z=-LLWyxβk,zvi-k,j-z=0, where u(i-k,j-z), v(i-k,j-z) represented the optical flow vector in the neighborhood of point (i,j), and (12)∇αIxxi,j,t=∑k=-LLWxxαkIi-k,j,tΓ2α+1∇αIxyi,j,t=∑k=-LL∑z=-LLWxyαk,zIi-k,j-z,tΓ2α+1∇αIxti,j,t=∑p=0T∑k=0LWxtαk,pIi-k,j,tIi,j,t-pΓ2α+1∇αIyyi,j,t=∑z=-LLWyyαkIi,j-z,tΓ2α+1∇αIyxi,j,t=∑k=-LL∑z=-LLWyxαz,kIi-k,j-z,tΓ2α+1∇αIyti,j,t=∑p=0T∑z=0LWytαz,kIi,j-z,tIi,j,t-pΓ2α+1.After repeated iteration utilizing the Gauss-Seidel iteration method, the accurate optical flow field can be achieved, and the convergence of the algorithm can be demonstrated by the method mentioned in [13].


## 3.1. The Fractional-Order Optical Flow Model

We substitute the brightness constraint equation of the HS model with the fractional-order Taylor series expansion and get the fractional-order brightness constraint equation as(7)DxαIΓα+1uα+DyαIΓα+1να+DtαIΓα+1=0,where I is the abbreviation of I(i,j,t).Combined with (2.19) and (2.20) in [12], we can get the fractional-order optical flow model in this paper: (8)EU=∫ΩDxαIΓα+1uα+DyαIΓα+1να+DtαIΓα+12+λDβu2+Dβv2dX.


## 3.2. Numerical Algorithm

The Euler-Lagrange equation [19] was applied in this paper to minimize energy function E(U).After simplification, we get the equation as follows:(9)DxαIΓα+1uα-1M+λDxβDxβu+DyβDyβu=0DyαIΓα+1vα-1M+λDxβDxβv+DyβDyβv=0,where (10)M=DxαIΓα+1uα+DyαIΓα+1να+DtαIΓα+1.After a series of computations, the iteration formula for the proposed fractional-order optical flow model can finally be written as(11)∇αIxxu2α-1+∇αIxyuα-1vα+∇αIxtuα-1+λ∑k=-LL∑z=-LLWxyβk,zui-k,j-z=0∇αIyyv2α-1+∇αIyxvα-1uα+∇αIytvα-1+λ∑k=-LL∑z=-LLWyxβk,zvi-k,j-z=0, where u(i-k,j-z), v(i-k,j-z) represented the optical flow vector in the neighborhood of point (i,j), and (12)∇αIxxi,j,t=∑k=-LLWxxαkIi-k,j,tΓ2α+1∇αIxyi,j,t=∑k=-LL∑z=-LLWxyαk,zIi-k,j-z,tΓ2α+1∇αIxti,j,t=∑p=0T∑k=0LWxtαk,pIi-k,j,tIi,j,t-pΓ2α+1∇αIyyi,j,t=∑z=-LLWyyαkIi,j-z,tΓ2α+1∇αIyxi,j,t=∑k=-LL∑z=-LLWyxαz,kIi-k,j-z,tΓ2α+1∇αIyti,j,t=∑p=0T∑z=0LWytαz,kIi,j-z,tIi,j,t-pΓ2α+1.After repeated iteration utilizing the Gauss-Seidel iteration method, the accurate optical flow field can be achieved, and the convergence of the algorithm can be demonstrated by the method mentioned in [13].


## 4. Experimental Results and Analysis

We have conducted our experiments from the following three aspects:(1) the comparison of the performance of previously tested optical flow algorithms; (2) the influence of the order of fractional-order differentials for the algorithm; and (3) the influence of the size of the dual fractional differential mask window for the algorithm. The Middlebury datasets [20] are chosen for evaluating our proposed algorithm for its characteristics of having different structures and attributes, like textures, blurred images, discontinuities in motion, shadows, nonrigid motion, and so on.The angular error (AE) and flow endpoint error (EE) are selected for evaluation. As we are concerned with the performance of the algorithm in the whole image, the AE/EE in the entire image (AVAE/AVEE) are used to visually evaluate our proposed optical model.For convenient description, we call the HS optical flow model the HS model, the model in [12] the FOVOF model, and our proposed model the DFOVOF model. The mask window size is L.Figure1 shows the optical flow field of the HS model, the FOVOF model, and the DFOVOF model in different image sequences. In the DFOVOF model, α=0.8, β=1.4, L=5. Among them, different colors represent different movement directions and different color depth represents different speed of movement. The deeper the color, the higher the speed.Figure 1
Optical flow results of the estimated optical flows by color maps. The first row shows the results of the HS model, the second row shows the results of the HS model, and the third shows the results of the DFOVOF model.
(a)
Yosemite (b)
Grove (c)
Urban (d)
ArmyThe first row in Figure1 shows the acquired optical flow field produced by the traditional HS model. It can be seen that the edge of the image discontinuity is not well reserved and the details of the image texture are neglected.The second row shows the optical flow field obtained by the FOVOF model. The edge discontinuity is reserved to some extent, but the contour of the object is not very clear.In the third row, the optical flow field obtained by the DFOVOF optical flow model is presented. The contour edge is much more obvious; meanwhile the details of image texture can be seen clearly. The DFOVOF model expands the brightness constraint equation in the form of a Taylor fractional-order, and the local brightness correlation of the image is also added for the calculation of the optical flow field, which further reduces the error and improves the accuracy of the model.In order to evaluate the influence of fractional-order parameters on the performance of the model and find out the correlation between the fractional-order and the image characteristics, the AVAE/AVEE of the optical flow field of each image sequence at different fractional orders is estimated, as shown in Figure2. At this time we choose L=2. The smaller the value of AVAE/AVEE, the higher the accuracy of the optical flow field estimation. The first row is the AVAE and AVEE of image sequence “Yosemite”; the second row is the AVAE and AVEE of image sequence “Army.” The third row is the AVAE and AVEE of image sequence “Urban.”Figure 2
The AVAE and AVEE of the optical flow field of three different image sequences at different fractional-order.We can find in Figure2 that the minimum AVAE/AVEE α is distributed in a range of (0.3,0.8) and the minimum AVAE/AVEE β is in a range of (1.3,1.4); the simpler the image texture, the smoother the surface representing the variation of the value of AVEE/AVAE to variable α and β; the more complex the image texture, the steeper the surface; the surface would also increase in incline when there exists a nonrigid motion in image sequences. Thus, it is clear that the algorithm presented here can improve the accuracy of motion estimation, especially for image sequences with complex texture and nonrigid motion. In conclusion, we choose a small α for a complex texture image and choose a relatively large α for an simple texture image sequence. What is more, nonrigid motion would also make the optimal α small. The optimal β changes little in different image sequence.To test the effect of the mask window sizeL on the algorithm’s performance and find out the correlation between mask window length L and image features, we calculate the AVEE of optical flow estimation of six image sequences at different mask window size L which was shown in Table 1, where α=0.8, β=1.3, where (a) is Venus; (b) is dimetrodon; (c) is hydrangea; (d) is Rubberwhale; (e) is grove; and (f) is urban. For comparison, the AVEE of H-S model and FOVOF model were also added to the table. For comparison, the AVEE values for the optical flow field estimation of the H-S model and the FOVOF model are also added to the table.Table 1
The AVEE of six image sequences at different mask window size.
Model Image (a) (b) (c) (d) (e) (f) H-S 0.5494 0.1908 0.2967 0.2268 0.2919 1.3944 FOVOF 0.5334 0.1861 0.2885 0.2071 0.2910 1.1085 DFOVOF L=2 0.5220 0.1848 0.2860 0.2046 0.2874 1.0646 L=3 0.5216 0.1843 0.2854 0.2043 0.2872 1.0603 L=4 0.5204 0.1838 0.2845 0.2024 0.2866 1.0534 L=5 0.5193 0.1826 0.2837 0.1993 0.2860 1.0546 L=6 0.5175 0.1815 0.2825 0.2026 0.2852 1.0573 L=7 0.5183 0.1823 0.2806 0.2034 0.2856 1.0592 L=8 0.5197 0.1831 0.2810 0.2043 0.2863 1.0623We can see in Table1 that the value of AVEE reduces further when applied to the DFOVOF model; in the DFOVOF model the error rate becomes smaller when the mask window size L increases—in general, we get the smallest AVEE value when L is within (2, 7). For all the images, AVEE will decrease first when L rises; after AVEE arrives at its optimal value, it will then increase when L rises. The more complex the image texture is or the more nonrigid the motion is, the smaller the value of L will be.In conclusion, we should choose a large valueL for relatively simple image sequences and choose a small value L for complex texture images and nonrigid motion images.


## 5. Conclusion

We have presented an improved fractional-order optical flow model, which applies fractional-order both in brightness constraint equation and in smoothness constraint equation of the HS model. What is more, we have proposed two-dimensional fractional differential masks and used them to simplify the computation of dual fractional-order derivatives. Experiments show that the model created here improves accuracy further. We have also found the relationships between the values ofα, β, and L and the image content. So different image content should use different α, β, and L, but the algorithm we employ utilizes the same α, β, and L in the whole image. Further studies will focus on the adaptive adjustments of the value of α, β, and L in relation to the image content.



---

*Source: 6278719-2018-05-10.xml*