# An Improved Fractional-Order Variational Optical Flow Model Combining Structure Tensor

Authors: Bin Zhu; Zhaodong Wang; Lianfang Tian; Jinmei Guo; Lingjian Wang; Jameel Bhutto
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

Dealing with problems of illumination changes in optical flow estimation, an improved variational optical flow model is proposed in this paper. The local structure constancy constraint (LSCC) is applied in the data term of the traditional HS (Horn & Schunck) optical flow model to substitute the brightness constancy constraint. The fractional-order smoothness constraint (FSC) is applied in the smoothness term of the HS model. Then, the detailed calculation processes from the optical flow model to the optical flow value are explained. The structure tensor in LSCC is an image feature that is constant in the illumination changes scene. The fractional differential coefficient in FSC can fuse the local neighborhood optical flow vector into the optical flow vector of the target pixel, which can improve the integrity of the motion region with the same motion speed. Combining LSCC with FSC, our improved optical flow model can obtain an accurate optical flow field with clear outline in the illumination abnormity scene. The experimental results show that, compared with other optical flow models, our model is more suitable for the illumination changes scene and can be employed in outdoor motion detection projects.


---

## Body

## 1. Introduction

Motion detection in the image sequence or video is one of the basic tasks of various image processing projects, which has been widely used in the research and engineering practice of 3D reconstruction, moving object segmentation, moving object tracking, video compression, automatic driving, and so on. The variational optical flow method is one of the most commonly used methods of motion detection. The HS optical flow model [1] is the most classical variational optical flow model. It is composed of a brightness constancy constraint and global smoothness constraint. The brightness constancy constraint requires that the intensity of pixels remains constant in the process of motion, while the smoothness constraint supposes that the motion speed of all pixels in the image changes smoothly. However, both constraints are based on ideal assumptions. In practical applications, these two constraints are often violated. Illumination changes in the scene will break the brightness constant constraint and the global smoothness constraint and lead to the blur of the edge between different motion regions.Several other constraints on the data term were proposed to improve the performance of the variational optical flow model, such as gradient constancy constraint [2], Laplacian constancy constraint [3], and Hessian constancy constraint [4], but these constraints depend heavily on the intensity difference. Illumination invariant descriptors such as binary-based [5], real value-based [6, 7], and neighborhood-based [8, 9] were applied to substitute the brightness constancy constraint. But these descriptors fail to identify the edge between different motion regions. The structure tensor [10–12], which can embody the shape feature of the motion objective, is a local structure feature that does not change with the environmental factor. The local structure optical flow model (LSOFM) [13–15], which is established on the structure tensor feature, can improve the robustness in the illumination changes scene. But when a structure includes different motion parts, LSOFM would fail.Because the fractional optical flow vector gradient can fuse the characteristics of the local neighborhood optical flow vector into the optical flow vector of the target pixel, the integrity of regions with the same motion speed is enhanced, so it can preserve the discontinuity of the optical flow field between different motion regions. The fractional-order variational optical flow model (FOVOFM) is proposed by Chen et al. [16], which uses the fractional-order smoothness constraint in the smoothness term of the HS model. The dual fractional order variational optical flow model (DFOVOFM) is proposed by Zhu et al. [17], which the fractional-order derivative is applied both in data term and smoothness term of the HS model. The fractional-order TV-L1 optical flow model was proposed in [18]. The fractional-order-based optical flow model neglects the structural information in image, so error would exist on the edge and some tiny motion regions.An improved variational optical flow model is proposed in this paper. Our model is composed of LSCC and FSC. LSCC cannot only ensure the robustness of the model in scenes of illumination changes but also the contour in the image can be enhanced. FSC cannot only preserve the motion discontinuity between different motion regions but also weaken the influence of illumination changes on optical flow estimation. The model proposed in this paper combines the advantages of the two constraints and solves the illumination change problem and the motion discontinuity problem in optical flow estimation, and different motion parts in a structure can be identified.The contribution of the paper is as follows:(1)
A novel optical flow model combining local structure tensor and fractional-order deviation is designed to improve the robustness of illumination changes(2)
An applicable model simplification method for our model is presented


## 2. Related Work

The research of the optical flow model never stops. An optical flow model using full four-dimensional cost volume was proposed in [19], which applied structural features and stereo matching pipelines to yield high accuracy. To solve the problems caused by untextured areas, motion boundaries, and occlusions in optical flow estimation, post-hoc confidence measures [20] are applied to assess the per-pixel reliability of the flow. Occlusions have remained one of the key challenges. In [21], a symmetric optical flow model was proposed to address occlusion problems of optical flow estimation. In [22], an interpolation correspondence method is proposed to estimate the optical flow, but these methods are usually susceptible to nonrigid motion and large displacements; to solve this problem, robust interpolation of sparse correspondences is proposed in [23].A coarse-to-fine scheme combined with an efficient random search strategy for efficient dense correspondence is proposed to estimate the optical flow in [24]. A coarse-to-fine PatchMatch with sparse seeds to sparse matches is applied to optical flow estimation in [25]. A novel differentiable neighbor-search up-sampling (NSU) module [26] is proposed to improve the coarse-to-fine technology for avoiding the optical flow error in edges, thin bars, and holes. A novel optical flow method related to the frequency domain that uses TV-wavelet regularization is applied to accurate optical flow estimation with the frequency-domain regularization in [27].With the rapid development of artificial intelligence, deep learning-based methods have been widely employed in optical flow estimation. The FlowNet [28] demonstrated that CNNs can be used to estimate the optical flow. In [29], three major improvements were designed for the FlowNet: a subnetwork was elaborated on small displacements, a stacked architecture was applied to warp of the second image with intermediate optical flow, and a schedule of the training data. A new deep network architecture called recurrent all-pairs field transforms is designed to estimate the optical flow in [30]. A deformable cost volume neural network which can estimate the multiscale optical flow in a single high resolution was proposed in [31]. PWC-Net [32], which consists of pyramidal processing, warping, and a cost volume, is designed to estimate optical flow. The network uses the current optical flow estimate to warp the CNN features of the second image and then uses the warped features and features of the first image to construct a cost volume. The drawback of deep learning-based methods is large number of ground truth are needed, and retraining the model is needed for different application scenarios.


## 3. HS Optical Flow Model

The HS model can be written as follows:(1)Eu=Edatau+λEsmoothu,where u=u,v represents the optical flow vector, u is the optical flow value on x axis, v is the optical flow value on y axis, Edatau represents the brightness constraint equation, Esmoothu represents smoothness constraint equation, and λ is a smoothness parameter, and its value is decided by the image quality. In a blurry image with many noises disturbing, the value of λ should be big and vice versa.The brightness constraint equation based on the bright constancy constraint, which supposes that the brightness value of a pixel is constant during the motion, is as follows:(2)Edatau=∫ΩIxu+Iyv+It2dx.The smoothness constraint equation based on the global smoothness constraint, which supposes that the motion speed of all pixels changes smoothly, that is to say, the derivative of the speed is very small, is as follows:(3)Esmoothu=∫Ω∇u2+∇v2dx.


## 4. Local Structure Constraint Equation

The structure tensor which is composed of local geometric features of image is an invariable feature to illumination changes, image rotation, image translation, and image scaling. To increase the robustness of the optical flow model, we use the local structure constancy constraint to substitute the brightness constant constraint.


### 4.1. Image Structure Tensor

The local structure tensor of the two-dimensional image can be expressed as follows:(4)T˜=∇IT∇I=Ix2IxIyIxIyIy2,where I represents the brightness function, ∇=∂x,∂y is the tangent vector along xandy axes, and T˜ is the structure tensor of a pixel. But T˜ is easily be affected by noise, so it is always enhanced by Gaussian filter Gσ:(5)T=Gσ∗T˜=Gσ∗Ix2IxIyIxIyIy2.


### 4.2. Local Structure Constancy Constrain

The local structure constancy constraint (LSCC) is proposed in this paper to enhance the robustness of the optical flow model. LSCC is based on the assumption that the local structure in an image would not change during the motion:(6)Tx,y,t=Tx+u,y+v,t+1,where Tx,y,t represents the structure tensor function from a pixel x,y to t frame. Then, the pixel will move from x,y,t to x+u,y+v,t+1, and u,v is the optical flow value, and Tx+u,y+v,t+1 represents the structure tensor of t+1 frame. Equation (6) means that the structure tensor will not change during the motion.Using the Taylor series expansion and ignoring the minimum term, we can get(7)Txu+Tyv+Tt=0,where Tx, Ty, and Tt are the derivative of the structure tensor function on x,y,andt axes, T=abbc, Tx=∂a/∂x∂b/∂x∂b/∂x∂c/∂x, Ty=∂a/∂y∂b/∂y∂b/∂y∂c/∂y, Tt=∂a/∂t∂b/∂t∂b/∂t∂c/∂t, a=1/l∑x∈Ωx0wIx2, b=1/l∑x∈Ωx0wIxIy, c=1/l∑x∈Ωx0wIy2, Ωx0 represents the local neighborhood of the target point, l is the normalization constant, and w represents the weight function.Then,Edatau can be rewritten as follows:(8)Edatau=∫ΩTxu+Tyv+Tt2dx.


## 4.1. Image Structure Tensor

The local structure tensor of the two-dimensional image can be expressed as follows:(4)T˜=∇IT∇I=Ix2IxIyIxIyIy2,where I represents the brightness function, ∇=∂x,∂y is the tangent vector along xandy axes, and T˜ is the structure tensor of a pixel. But T˜ is easily be affected by noise, so it is always enhanced by Gaussian filter Gσ:(5)T=Gσ∗T˜=Gσ∗Ix2IxIyIxIyIy2.


## 4.2. Local Structure Constancy Constrain

The local structure constancy constraint (LSCC) is proposed in this paper to enhance the robustness of the optical flow model. LSCC is based on the assumption that the local structure in an image would not change during the motion:(6)Tx,y,t=Tx+u,y+v,t+1,where Tx,y,t represents the structure tensor function from a pixel x,y to t frame. Then, the pixel will move from x,y,t to x+u,y+v,t+1, and u,v is the optical flow value, and Tx+u,y+v,t+1 represents the structure tensor of t+1 frame. Equation (6) means that the structure tensor will not change during the motion.Using the Taylor series expansion and ignoring the minimum term, we can get(7)Txu+Tyv+Tt=0,where Tx, Ty, and Tt are the derivative of the structure tensor function on x,y,andt axes, T=abbc, Tx=∂a/∂x∂b/∂x∂b/∂x∂c/∂x, Ty=∂a/∂y∂b/∂y∂b/∂y∂c/∂y, Tt=∂a/∂t∂b/∂t∂b/∂t∂c/∂t, a=1/l∑x∈Ωx0wIx2, b=1/l∑x∈Ωx0wIxIy, c=1/l∑x∈Ωx0wIy2, Ωx0 represents the local neighborhood of the target point, l is the normalization constant, and w represents the weight function.Then,Edatau can be rewritten as follows:(8)Edatau=∫ΩTxu+Tyv+Tt2dx.


## 5. Fractional Smoothness Constraint Equation

The fractional smoothness constraint (FSC) is applied in our model to further improve the performance of optical flow estimation. We use the fractional-order derivative to substitute the integer-order derivative in the smoothness constraint equation of the HS optical flow model:(9)∇u⟶Dαu,∇v⟶Dαv,where Dαu represents α-order derivative of the optical flow vector u on x and y axes, Dαv represents α-order derivative of the optical flow vector v on x and y axes, α is real, α∈1,2, Dαu=Dxαu2+Dyαu2, and Dαv=Dxαv2+Dyαv2.Then,Esmoothu can be designed as follows:(10)Esmoothu=∫ΩDαu2+Dαv2dx.


## 6. Improved Fractional Optical Flow Model

Combining LSCC with FSC, we construct our improved optical flow model; using the structure of the HS model, our optical flow model can be described as follows:(11)Eu=Edatau+λEsmoothu=∫∫ΩTxu+Tyv+Tt2+λDαu2+Dαv2dxdy.Using the Euler–Lagrange equation to minimize the energy function (11), a similar procedure can be seen in [14, 15]; then, we can get(12)Txxi,jui,j+Txyi,jvi,j+Txti,j+λDxα∗Dxαu+Dyα∗Dyαu=0,Tyxi,jui,j+Tyyi,jvi,j+Tyti,j+λDxα∗Dxαv+Dyα∗Dyαv=0,where Txxi,j, Txyi,j, Txti,j, Tyxi,j, Tyyi,j, and Tyti,j are the second-order derivative of the structure tensor at x, y, and t axes:(13)Txx=dTxdx=∂a∂x∂x∂b∂x∂x∂b∂x∂x∂c∂x∂x,Txy=dTxdy=∂a∂x∂y∂b∂x∂y∂b∂x∂y∂c∂x∂y,Tyx=dTydx=∂a∂y∂x∂b∂y∂x∂b∂y∂x∂c∂y∂x,Tyy=dTydy=∂a∂y∂y∂b∂y∂y∂b∂y∂y∂c∂y∂y,Txt=dTxdt=∂a∂x∂t∂b∂x∂t∂b∂x∂t∂c∂x∂t,Tyt=dTydt=∂a∂y∂t∂b∂y∂t∂b∂y∂t∂c∂y∂t,where Dxα∗Dxαu, Dyα∗Dyαu, Dxα∗Dxαv, and Dyα∗Dyαv denote the dual fractional-order derivative of optical flow value u and v on x and y axes, respectively:(14)Dxα∗Dxαu=∑k=−∞0wkαui−k,j+∑k=0∞wkαui−k,j≈∑k=−L−1wkα∇ui−k,j+∑k=1Lwkα∇ui−k,j,Dyα∗Dyαui,j≈∑k=−L−1wkα∇ui,j−k+∑k=1Lwkα∇ui,j−k,where w0α=1 and wkα=1−α+1/kwk−1α, k=1,2,…, L, which denote the size of the neighborhood.Then, we can get(15)Dxα∗Dxαu+Dyα∗Dyαu≈∑i¯,j¯∈χi,jwkij¯αui¯,j¯−ui,j,Dxα∗Dxαv+Dyα∗Dyαv≈∑i¯,j¯∈χi,jwkij¯αvi¯,j¯−vi,j,where χi,j represents a neighborhood of point i,j, kij¯ is equal to maxi¯−i,j¯−j, ui¯,j¯ and vi¯,j¯ are the mean of the optical flow vector in the neighborhood, and ui,j and vi,j are the coordination of the optical flow vector on x and y axes of each pixel in the neighborhood.Then, the optical flow estimation formula is as follows:(16)Txxi,jui,j+Txyi,jvi,j+Txti,j+λ∑i¯,j¯∈χi,jwkij¯αui¯,j¯−ui,j=0,Tyxi,jui,j+Tyyi,jvi,j+Tyti,j+λ∑i¯,j¯∈χi,jwkij¯αvi¯,j¯−vi,j=0,where λ is the smoothness parameter, according to Huerst and Xie [33]. The value of parameter λ is determined by the image quality; when the image is dark, unidentified, blurry, or noisy, λ should be large. Otherwise, λ should be small. In our algorithm, we choose λ∈5.0,10.0 in relatively clear images, and λ∈10.0,15.0 in relatively blurry images. α is the order of fractional derivation in the smoothness term, according to Chen et al. [15], and α=1.3 or 1.4. The accuracy in the case of α=1.3 is relatively higher than in the case of α=1.4, while the latter case needs less iteration times.We can obtain the accurate optical flow value by repeated iteration utilizing the Gauss–Seidel method. The iteration will stop when the accuracy reaches the requirement (the difference of the optical flow value between two adjacent iterations is less than the threshold).


## 7. Experimental Result and Analysis

To testify the superiority and practicability of our optical flow model, Middlebury dataset [34], MPI_Sintel dataset [35], KITTI dataset [36], and an outdoor real scene video are used in experiments. Vertical comparisons are employed to explain the model evolution process; these models include HS model [1], LSCOFM [15], FOVOFM [16], and DFOVOFM [17]. Three widely used high performance models are employed for horizontal comparison: HAST [37], MDP_Flow [38], and PH_Flow [39]. All experiments are run in a PC equipped with Intel Xeon(R) X5600, 3.4 GHZ, and 4 GB memory. The operating system is 64 byte Windows 7, and the software platform is Matlab 10.1.Figures1–3 show the color code map by optical flow estimation of different algorithms in the Middlebury database. They are army sequence, grove sequence, and scheflera sequence, respectively. It can be seen in these figures that, from left to right, we can get more and more clear outlines, which means the optical flow results are getting more and more accurate. But, compared to the GT, some highly-textured thin outlines disappear or become adjoined in the results of our model, such as soldier’s gun in army, the end of branches in Figure 2(e), and the rhizome parts of leaves in Figure 3(e). That is because using the fractional order derivative, the thin parts would be assimilated by the optical flow field of surrounding pixels. Table 1 shows the AEE/AAE of different algorithms in the Middlebury dataset. We can see that, from top to down, the error rate is becoming more and more small, and the error rate of our model is the lowest.Figure 1
Optical flow estimation results of the army sequence in the Middlebury dataset with different algorithms. (a) HS. (b) FOVOFM. (c) DFOVOFM. (d) LSOFM. (e) Ours. (f) GT.
(a)(b)(c)(d)(e)(f)Figure 2
Optical flow estimation results of the grove sequence in the Middlebury dataset with different algorithms. (a) HS. (b) FOVOFM. (c) DFOVOFM. (d) LSOFM. (e) Ours. (f) GT.
(a)(b)(c)(d)(e)(f)Figure 3
Optical flow estimation results of the scheflera sequence in Middlebury dataset with different algorithms. (a) HS. (b) FOVOFM. (c) DFOVOFM. (d) LSOFM. (e) Ours. (f) GT.
(a)(b)(c)(d)(e)(f)Table 1
The AEE/AAE of different algorithms for the Middlebury dataset.
AlgorithmArmyMequnSchefleraWodenGroveUrbanYosemiteTeddyAEEHS0.220.611.010.781.261.430.161.51FOVOFM0.110.210.320.190.891.250.140.64DFOVOFM0.100.220.300.160.670.470.170.51LSOFM0.110.210.260.170.690.460.170.45Ours0.080.170.180.160480.310.180.38AAEHS8.019.1314.212.44.648.214.019.16FOVOFM3.823.615.663.933.244.122.734.48DFOVOFM3.183.215.202.932.942.592.672.33LSOFM2.852.243.203.022.542.572.731.73Ours2.641.982.022.832.342.432.231.32Figures4 and 5 show the color code map by optical flow estimation of different algorithms in the MPI_Sintel dataset. They are Cave3 sequence and Shaman1 sequence, respectively. The results presented show that our method can acquire clear and integral motion outlines under the low contrast scene (Cave3) and illumination insufficient scene (Shaman1). HAST can embody motion details such as the end of the cone in Cave3 and the boy’s hair in Shaman1, but fails in large nontextured regions such as the tail in Cave3 and the gusset parts in Shaman1. Similar situations are shown in PH_Flow and MDP_Flow. The deficiencies of our model also exist in the thin detail parts: the end of the cone in Cave3 and details of boy’s hair are missing in Shaman1. Table 2 shows the AEE/AAE of different algorithms in the MPI_Sintel dataset. We can see in Table 2 that, in sequences with existing large displacement and occlusion problems, the accuracy of our method is lower than its components. When comes to sequences with illumination changes and illumination insufficient, the accuracy of our method is higher than other algorithms.Figure 4
Optical flow results of Cave3 sequence in the MPI_Sintel dataset with different algorithms. (a) HAST. (b) MDP_Flow. (c) PH_Flow. (d) Ours. (e) GT.
(a)(b)(c)(d)(e)Figure 5
Optical flow results of Shama1 sequence in the MPI_Sintel dataset with different algorithms. (a) HAST. (b) MDP_Flow. (c) PH_Flow. (d) Ours. (e) GT.
(a)(b)(c)(d)(e)Table 2
The AEE/AAE of different algorithms in the MPI_SINTEL dataset.
AlgorithmMarket1WallCave3ShamanMarket3Market4Ambush1Ambush3AEEHAST3.648.399.822.412.2040.1243.8714.12PH_Flow3.646.458.051.412.0628.1138.6912.98MDP_Flow0.100.220.300.160.670.470.170.51LSOFM0.110.210.260.170.690.460.170.45Ours0.080.170.180.160480.310.180.38AAEHAST8.019.1314.212.44.648.214.019.16PH_Flow3.823.615.663.933.244.122.734.48MDP_Flow3.183.215.202.932.942.592.672.33LSOFM2.852.243.203.022.542.572.731.73Ours2.641.982.022.832.342.432.231.32The KITTI dataset is a challenging real-world dataset in which most sequences are affected by illumination abnormity, such as illumination changes, illumination uneven, illumination insufficient, and so on. The experimental results are shown in Figures6–8. They are sequence 00004, 00009, and 00011, respectively. We can see that our method obtains clear motion outlines in all image sequences. Despite the illumination uneven conditions, the tree outline can be seen clearly in 00004 and the car outline can be seen clearly in 00009. In sequence 00011 with the illumination normal scene, differences between different algorithms are small. Table 3 shows the AEE/AAE of different algorithms in the KITTI dataset. We can see that, in illumination abnormity scenes, the error rate of our model is the lowest, and the average accuracy of our model for the whole sequences in KITTI is the highest.Figure 6
Visual comparison of 00004 sequence in the KITTI dataset with different methods. (a) HAST. (b) MDP_Flow. (c) PH_Flow. (d) Ours.
(a)(b)(c)(d)Figure 7
Visual comparison of 00009 sequence in the KITTI dataset with different methods. (a) HAST. (b) MDP_Flow. (c) PH_Flow. (d) Ours.
(a)(b)(c)(d)Figure 8
Visual comparison of 00011 sequence in the KITTI dataset with different methods. (a) HAST. (b) MDP_Flow. (c) PH_Flow. (d) Ours.
(a)(b)(c)(d)Table 3
The AEE of TV_L1 and DFOVOF combined with WRMS-based method or our adaptive method in KITTI.
AlgorithmAVE000004000009000011000014000015HAST8.85/15.232.16/5.5318.26/25.928.05/14.975.82/8.039.93/18.74MDP_Flow8.25/14.731.87/5.1316.91/24.257.46/13.895.03/7.298.26/16.53PH_Flow8.57/14.921.94/5.2617.68/25.647.73/14.545.47/7.839.75/17.42Ours8.15/13.651.73/5.1516.65/23.647.76/14.684.79/7.178.18/15.72Figures9–12 show the color code map of optical flow estimation by different algorithms in an outdoor video. We can see that compared to LSOFM, DFOVOFM has less noise affected regions, but the outline is not so clear. Our model achieves better results with the most intact object outline and least noise affected regions than other models. In Figure 10(d), different motion parts in the right sole can be seen, the optical flow field of the first half foot is pink, and the second half is yellow. The motion of one finger in the right hand in Figures 11 and 12 is different from the other part on that hand, and we can see different color regions in Figure 12(d).Figure 9
Inputs and visual comparison of an image sequence from the outdoor video of different algorithms. (a) Input1. (b) Input2. (c) HAST. (d) MDP_Flow.
(a)(b)(c)(d)Figure 10
Visual comparison of an image sequence from the outdoor video of different algorithms. (a) PH_Flow. (b) DFOVOFM. (c) LSOFM. (d) Ours.
(a)(b)(c)(d)Figure 11
Inputs and visual comparison of an image sequence from the outdoor video of different algorithms. (a) Input1. (b) Input2. (c) HAST. (d) MDP_Flow.
(a)(b)(c)(d)Figure 12
Visual comparison of an image sequence from the outdoor video of different algorithms. (a) PH_Flow. (b) DFOVOFM. (c) LSOFM. (d) Ours.
(a)(b)(c)(d)


## 8. Conclusion

An improved variational optical flow model is proposed in this paper to solve problems of illumination changes in optical flow estimation. The model applies LSCC and FSC in the data term and smoothness term of the HS model. Then, the detailed calculation processes from the optical flow model to optical flow value are explained. Experimental results in three public datasets and an outdoor video show that our model is superior to other algorithms in illumination abnormity scenes. The disadvantage of our model is that it cannot embody the motion of thin parts. Experimental results testify the performance of our model. Finally, we have conclusions that our model is more suitable for the illumination changes scene and can be employed in outdoor motion detection projects.



---

*Source: 9960784-2021-07-27.xml*