# An Adaptive Smoothness Parameter Strategy for Variational Optical Flow Model

Authors: Hussain Zaid H. Alsharif; Tong Shu; Bin Zhu; Zeyad Farisi
Journal: Scientific Programming
 (2021)

---

## Abstract

The smoothness parameter is used to balance the weight of the data term and the smoothness term in variational optical flow model, which plays very significant role for the optical flow estimation, but existing methods fail to obtain the optimal smoothness parameters (OSP). In order to solve this problem, an adaptive smoothness parameter strategy is proposed. First, an amalgamated simple linear iterative cluster (SLIC) and local membership function (LMF) algorithm is used to segment the entire image into several superpixel regions. Then, image quality parameters (IQP) are calculated, respectively, for each superpixel region. Finally, a neural network model is applied to compute the smoothness parameter by these image quality parameters of each superpixel region. Experiments were done in three public datasets (Middlebury, MPI_Sintel, and KITTI) and our self-constructed outdoor dataset with the proposed method and other existing classical methods; the results show that our OSP method achieves higher accuracy than other smoothness parameter selection methods in all these four datasets. Combined with the dual fractional order variational optical flow model (DFOVOFM), the proposed model shows better performance than other models in scenes with illumination inhomogeneity and abnormity. The OSP method fills the blank of the research of adaptive smoothness parameter, pushing the development of the variational optical flow models.


---

## Body

## 1. Introduction

Motion detection [1, 2] is a research hotspot of image processing, which is widely applied in motion segmentation [3], target tracking [4], and video [5] surveillance [6]. Optical flow method [7] is one of the most commonly used motion detection methods, which aims to estimate the spatial displacement of each pixel in adjacent image sequences.Variational optical flow algorithm [8] is one of the most popular optical flow methods, which consists of three parts: data term, smoothness term, and smoothness parameter. In order to improve the performance of the variational optical flow algorithm, many improvements have been done to the data and smoothness term to solve the large displacement problem [9], enhance the robustness against noise and illumination changes [10–12], maintain the discontinuity between different motion regions [13], highlight the contour of motion regions [14–16], and deal with occlusion problems [17, 18]. However, little attention is paid to the smoothness parameter selecting method. The smoothness parameter acts as a bridge between the data term and smoothness term that directly affects the final result of optical flow estimation. Small smoothness parameter leads to overfitting, whereas higher value causes underfitting.The optimal smoothness parameter is determined by image quality [19]. Small value of smoothness parameter is assigned to image which has clear outline and high contrast, while large smoothness parameter is assigned to image that has blurry outline and low contrast. In optical flow estimation, many methods determine smoothness parameter through human visual perception which has no exact value, so the smoothness parameter chosen by these methods is inaccurate. Moreover, many existing methods use the same smoothness parameter throughout the whole image, but the image quality in different region may be different and due to this, a mismatch between the smoothness parameter and image content would exist. Although some methods have been proposed for adjusting the smoothness parameter through image content, none of them can solve these problems well.Figure1 shows the illumination inhomogeneous image sequence with different image quality regions and the optical flow estimation results of dual fractional order optical flow model. Figures 1(a) and 1(b) are input image sequences; Figures 1(c) and 1(d) are optical flow estimation results when the smoothness parameter is λ=10 and λ=20, respectively. It can be seen in Figure 1(c) that the contour of the flow field on the bright part is clearer than dark part, whereas in Figure 1(d), the contour of the flow field on the dark part is clearer than bright part, so, if we set λ=10 in bright part of the image and meanwhile λ=20 in dark part of the image, the accuracy would be highest among these three methods.Figure 1
Result of DFOVOFM with different smoothness parameters in illumination inhomogeneous scene. (a) Input 1, (b) input 2, (c)λ=10, and (d) λ=20.
(a)(b)(c)(d)The purpose of this paper is to solve the mismatch problem of smoothness parameter and image content in each part of an image; an adaptive smoothness parameter strategy is designed to select the optimal smoothness parameters for different regions in an image.For this paper, the main contributions are as follows:(1)
A new image segmentation algorithm combining SLIC with LMF is proposed to segment the whole image into several superpixel regions(2)
We would not use the same smoothness parameter in the whole image any more; different smoothness parameters would be assigned to different superpixel region, respectively(3)
Image quality parameters are used to calculate smoothing parameters for the first timeThe rest of the paper is organized as follows. Section2 reviews the work related to smoothness parameter selecting methods. Section 3 introduces the flowchart of the algorithm. Section 4 introduces the variational optical flow model used in this paper. Sections 5 and 6 describe the proposed image segmentation method and image quality parameters, respectively. Section 7 explains how to use image quality parameters to determine smoothness parameter for each superpixel region. The experimental results are discussed in Section 8 and conclusion is presented in Section 9.


## 2. Related Work

The smoothness parameter adjustment strategy for the variational optical flow model was first proposed by Nagel in 1986. He proposed oriented smoothness constrain in [20] which used the variation of gray value to adjust the value of smoothness parameter. However, this method fails in illumination insufficient scenes and illumination changes scenes. In [21], a smoothness weight selection method was proposed; the method uses a blurring operator to calculate the weighted distance, but the spatially varying character needs to be adjusted before it can be applied to variational optical flow model. In [22], a method combining the maximum likelihood function with the cross-validation function was applied to estimate the smoothness parameter, but the method cannot be used in large displacement scenes. A data-driven method based on estimated risk in [23] was used to select the smoothness parameter, but the method is computationally expensive, especially with respect to robust data terms. A Bayesian framework is presented in [24] to estimate the variational model parameters by minimizing the objective function. In [25], an average data constant error criterion based on optimal prediction theory was proposed to dynamically select the smoothness parameter, but the method is limited to linear rigid motion. Kyong et al. applied adaptive convolution kernel function in [26] to address oversegmentation or oversmoothing problem, but when low contrast regions exist, the adaptive strategy would be invalid. In [27], weighted root mean square error criterion was used to estimate the smoothness parameter. The method estimates an initial smoothness parameter by visual perception. Then, search procedures were applied to obtain the optimal parameter around the initial smoothness parameter with the least root mean square error. However, the method has to try many times to find the optimal smoothness parameter, and the same smoothness parameter would be applied in the whole image; when uneven illumination occurs in an image, the method would not be suitable.In recent years, many other improvements for the variational optical flow model were proposed: TV-L1-based methods [28, 29], deep learning-based methods [30–32], and filtering based methods [33–35], but none of them focus on smoothness parameter.


## 3. The Flowchart of the Algorithm

The flowchart of the algorithm is in Figure2: first, the whole image was segmented into several superpixel regions by SLIC and LFM method; then, for each superpixel region, the image quality parameter is calculated; then, a neural network model is designed to use image quality parameter determining the optimal smoothness parameter; the neural network model should be trained by pairs of image quality parameters and optimal smoothness parameter firstly. At last, DFOVOFM and optimal smoothness parameter was used to estimate the optical flow for image sequences.Figure 2
Flowchart of our algorithm.


## 4. The Dual Fractional Order Optical Flow Model

The dual fractional order optical flow model (DFOVOFM) [36] is a fractional order version of the HS model in which the data term and smoothness term of the HS model are reconstructed by fractional order derivative. The predominance of DFOVOFM is that the model integrated the variation characteristics around the target points, which not only retains the edge and texture details, but also eliminates the influence of subtle noise, so as to improve the robustness of the model to environmental changes in the process of optical flow estimation.The DFOVOFM is defined as(1)Eu=∫ΩDxαI⋅uα+DyαI⋅να+DtαI2+λDxβu2+Dyβu2+Dxβv2+Dyβv2dx,where Eu represents the energy function of the optical flow model, u=u,v is the optical flow vector, and u,v are the components of optical flow vector on axis x and axis y, respectively. DxαI, DyαI, and DtαI denote the fractional order derivatives of brightness function I on x,y,t axis. α∈R+ represents the order of fractional order derivative. Dxβu, Dyβu, Dxβv, and Dyβv denote the βth order derivatives of optical flow components u, v on x,y axis, respectively, Ω⊂R3 denotes a neighborhood of point i,j,t, λ is the smoothness parameter, and x=x,y.


## 5. The Superpixel Segmentation

The image quality parameters in a local region are considered to be the same if the RGB values of pixels are same or similar. Therefore, we use the RGB value and position of each pixel to divide the image into several subregions and then calculate the quality parameters of each subregion. Simple Linear Iterative Clustering (SLIC) [37] algorithm with five-dimensional space r,g,b,x,y is applied in this paper for image segmentation.Firstly, the whole image is segmented intoK subregions of equal size; then the coordinate and RGB value of center points are reserved. The distance between two adjacent center points is s=N/K, where N represents the pixel number in the image.Secondly, the Euclidean distance of RGB valuedrgb and position information dxy between the center point and its adjacent points in 2s×2s region is calculated. A weight should be added to dxy because the size of subregion may be different.(2)Ds=drgb+msdxy,where m is a compensation factor; after that, the Ds values of adjacent pixels in the 2s×2s region are ranked from small to large. The new superpixel region is composed by the first s×s pixels; then the center point of every new superpixel region is renewed.We iterate the second procedure until the average distance betweenCnk=xnk,ynk and Cnk+1=xnk+1,ynk+1 is less than threshold Th. Cnk and Cnk+1 represent the coordinates of the center point of the kth iteration and k + 1th iteration, n∈1,s×s. The iteration will terminate when the following equation is true:(3)1s×s∑n=1s×sxnk−xnk+12+ynk−ynk+12<Th.However, according to the conventional SLIC, even if the RGB value is very similar to the center point, the edge points of the superpixel region will be misclassified. In order to correct these erroneously segmented pixels, the local membership function [38] mix is applied to the edge points:(4)mix=Ix−μi2∑i=1nIx−μi2,where mix represent the proportion of pixels belongs to which superpixel region. We reclassify the pixel to the superpixel region of the maximum mix. n is the number of superpixel region around the pixel and n∈2,3,4, Ix is the brightness function of a pixel, and μi is the mean of brightness value of pixels in a superpixel region.Figure3 shows the segmentation result of Figure 1(a). Figures 3(a) and 3(b) are the results of original SLIC and improved SLIC, respectively. It can be seen that some of the edge points of superpixel regions have been classified correctly after processing by the local membership function.Figure 3
Superpixel segmentation results of (a) original SLIC and (b) improved SLIC method.
(a)(b)


## 6. Image Quality Parameter

Image quality parameter is the quantification of human visual perception of an image and it is one of the basic steps in image processing field. Many image processing tasks can be performed well if the image quality is evaluated accurately. Recently, many image quality parameters are proposed such as entropy [39], brightness [40], colorfulness [41], sharpness [42], and contrast [43]. Author in [44] proposed a Color Root Mean Square Enhancement (CRME) method for color image as the extended version of the gray scale Root Mean Enhancement (RME). K. Panetta in [45] proposed a Color Quality Enhancement (CQE) method which measured the color image quality by the combination of colorfulness, sharpness, and contrast.Keeping in consideration the aforementioned image quality parameters and the characteristics of the variational optical flow model, we designed more comprehensive image quality parameters which include CRME, colorfulness, sharpness, and contrast in CQE and signal-to-noise ratio in our work.


### 6.1. CRME

The Color Root Mean Enhancement (CRME) is designed to not only measure the contrast for each color plan, but also measure the difference between the color cube center and the surrounding.(5)CRME=1000k1k2∑i=1k1∑j=1k2logIi,j−∑c=13ζcIc1+Ic2+…+Icn/nlogIi,j+∑c=13ζcIc1+Ic2+…+Icn/n.To calculate CRME, the image is segmented intok1∗k2 blocks, Ii,j is the center of the block i,j, and I1+I2+⋯+In/n is the mean of the intensity in block i,j. ζc is the weight of the color plane, according to NTSC (National Television System Committee) standard, and ζr=0.299, ζg=0.587, and ζb=0.144.


### 6.2. Contrast

Contrast refers to the measurement of different brightness levels between the brightest white and the darkest black in an image. The larger the difference between the brightest and the darkest, the greater the contrast.(6)contrast=∑c=13ζc1k1k2∑l=1k1∑k=1k2logImax,k,l+Imin,k,lImax,k,l−Imin,k,l.To calculate contrast, the image is segmented intok1∗k2 blocks, Imax,k,l and Imin,k,l are the max and min intensity value in block k,l, and ζr=0.299, ζg=0.587, and ζb=0.144.


### 6.3. Sharpness

Sharpness is responsible for the presentation of the details, edges, and textures in the image. For example, in the case of high sharpness, not only are the wrinkles and spots of the face on the picture clearer, but also the bulge or depression of the facial muscles can be identified.(7)sharpness=∑c=13ζc2k1k2∑l=1k1∑k=1k2logImax,k,lImin,k,l.To calculate contrast, the image is segmented intok1∗k2 blocks, Imax,k,l and Imin,k,l are the max and min intensity value in block k,l, and ζr=0.299, ζg=0.587, and ζb=0.144.


### 6.4. Colorfulness

Color is the most sensitive formal element that can cause our common aesthetic pleasure. Color is one of the most expressive elements, because its nature directly affects people’s feelings.(8)colorfulness=2∗logσα2μα0.2∗logσβ2μβ0.2,where α=R−G, β=0.5R+G−B, N is the pixel number of a superpixel region, p∈1,N, μα=1/N∑p=1Nαp,σα2=1/N∑p=1Nαp2−μα2, μβ=1/N∑p=1Nβp,σβ2=1/N∑p=1Nβp2−μβ2.


### 6.5. Signal-to-Noise Ratio (SNR)

SNR is a physical quantity that represents the degree of local image region affected by noise and distortion. SNR is defined as(9)SNR=20log10μδ,where μ indicates the mean of ∇2Ii,j and δ is the corresponding standard deviation.(10)∇2Ii,j=Ix2+Iy2,where Ix and Iy are the gradient of brightness function Ix,y for x−andy− axis, respectively. In general, the higher value of SNR results in better image quality.


## 6.1. CRME

The Color Root Mean Enhancement (CRME) is designed to not only measure the contrast for each color plan, but also measure the difference between the color cube center and the surrounding.(5)CRME=1000k1k2∑i=1k1∑j=1k2logIi,j−∑c=13ζcIc1+Ic2+…+Icn/nlogIi,j+∑c=13ζcIc1+Ic2+…+Icn/n.To calculate CRME, the image is segmented intok1∗k2 blocks, Ii,j is the center of the block i,j, and I1+I2+⋯+In/n is the mean of the intensity in block i,j. ζc is the weight of the color plane, according to NTSC (National Television System Committee) standard, and ζr=0.299, ζg=0.587, and ζb=0.144.


## 6.2. Contrast

Contrast refers to the measurement of different brightness levels between the brightest white and the darkest black in an image. The larger the difference between the brightest and the darkest, the greater the contrast.(6)contrast=∑c=13ζc1k1k2∑l=1k1∑k=1k2logImax,k,l+Imin,k,lImax,k,l−Imin,k,l.To calculate contrast, the image is segmented intok1∗k2 blocks, Imax,k,l and Imin,k,l are the max and min intensity value in block k,l, and ζr=0.299, ζg=0.587, and ζb=0.144.


## 6.3. Sharpness

Sharpness is responsible for the presentation of the details, edges, and textures in the image. For example, in the case of high sharpness, not only are the wrinkles and spots of the face on the picture clearer, but also the bulge or depression of the facial muscles can be identified.(7)sharpness=∑c=13ζc2k1k2∑l=1k1∑k=1k2logImax,k,lImin,k,l.To calculate contrast, the image is segmented intok1∗k2 blocks, Imax,k,l and Imin,k,l are the max and min intensity value in block k,l, and ζr=0.299, ζg=0.587, and ζb=0.144.


## 6.4. Colorfulness

Color is the most sensitive formal element that can cause our common aesthetic pleasure. Color is one of the most expressive elements, because its nature directly affects people’s feelings.(8)colorfulness=2∗logσα2μα0.2∗logσβ2μβ0.2,where α=R−G, β=0.5R+G−B, N is the pixel number of a superpixel region, p∈1,N, μα=1/N∑p=1Nαp,σα2=1/N∑p=1Nαp2−μα2, μβ=1/N∑p=1Nβp,σβ2=1/N∑p=1Nβp2−μβ2.


## 6.5. Signal-to-Noise Ratio (SNR)

SNR is a physical quantity that represents the degree of local image region affected by noise and distortion. SNR is defined as(9)SNR=20log10μδ,where μ indicates the mean of ∇2Ii,j and δ is the corresponding standard deviation.(10)∇2Ii,j=Ix2+Iy2,where Ix and Iy are the gradient of brightness function Ix,y for x−andy− axis, respectively. In general, the higher value of SNR results in better image quality.


## 7. The Calculation of Smoothness Parameter

A neural network model [46–48] is designed to obtain smoothness parameter using image quality parameters.


### 7.1. The Neural Network Model

The input of the model is composed of five nodes which represent five image quality parameters and the output of the model has only one node (the corresponding smoothness parameter). After numerous attempts, we finally determined the number of hidden layers and the number of nodes in each layer: five hidden layers: the first layer has 10 nodes, the second layer has 20 nodes, the third layer has 20 nodes, the fourth layer has 10 nodes, and the fifth layer has 5 nodes. The sigmoid function was chosen as the active function. Softmax was chosen as the loss function.We initialize the weight parameters using the Gaussian distribution with mean value of 0 and variance of 0.01, and then the weight parameters are adjusted by gradient descent method and backpropagation function. The self-built database is used as the training sample, the batch normalization training model is used, and the random deactivation is used to avoid the overfitting of the model. The momentum value is selected as 0.99, the weight penalty item is set as 0.01, and the learning rate is 0.02.


### 7.2. Self-Construction Dataset

Twelve classes of database are used to construct the training set that are Middlebury database, the MPI_Sintel_final database, MPI_Sintel_clean database, KITTI database, and image sequences of eight outdoor videos with different conditions. It is well-known that higher number of training samples results in higher accuracy of the model. In our model, by increasing the number of samples, the accuracy of model also increases smoothly until it reaches 98.5%, but the accuracy increases slightly when the number of samples exceeds 300. Due to this, we use 300 samples for each class. For each sample, i.e., image sequence, the image quality parameters are calculated and the optimal smoothness parameter is obtained by [27]; then they are used to train the neural network model. Table 1 shows some of the training data.Table 1
Image quality parameters and OSP of optical flow dataset.
DatasetSeqCRMESharpContrastColorsnospMidArmy75768586833.4Backyard86838372768.3MPI_fMarket4776876666711.2Cave3433258636510.5MPI_cMarket495939293922.8Cave364539491933.2KITTI0001568646767739.600022635369586511.3Outdoor videoVideo176678372788.7Video2627485635010.4Video3634676516715.7Video4654179372321.5Video5573653357221.3Video6473256275324.5Video7372436214125.6Video8382337161427.3


## 7.1. The Neural Network Model

The input of the model is composed of five nodes which represent five image quality parameters and the output of the model has only one node (the corresponding smoothness parameter). After numerous attempts, we finally determined the number of hidden layers and the number of nodes in each layer: five hidden layers: the first layer has 10 nodes, the second layer has 20 nodes, the third layer has 20 nodes, the fourth layer has 10 nodes, and the fifth layer has 5 nodes. The sigmoid function was chosen as the active function. Softmax was chosen as the loss function.We initialize the weight parameters using the Gaussian distribution with mean value of 0 and variance of 0.01, and then the weight parameters are adjusted by gradient descent method and backpropagation function. The self-built database is used as the training sample, the batch normalization training model is used, and the random deactivation is used to avoid the overfitting of the model. The momentum value is selected as 0.99, the weight penalty item is set as 0.01, and the learning rate is 0.02.


## 7.2. Self-Construction Dataset

Twelve classes of database are used to construct the training set that are Middlebury database, the MPI_Sintel_final database, MPI_Sintel_clean database, KITTI database, and image sequences of eight outdoor videos with different conditions. It is well-known that higher number of training samples results in higher accuracy of the model. In our model, by increasing the number of samples, the accuracy of model also increases smoothly until it reaches 98.5%, but the accuracy increases slightly when the number of samples exceeds 300. Due to this, we use 300 samples for each class. For each sample, i.e., image sequence, the image quality parameters are calculated and the optimal smoothness parameter is obtained by [27]; then they are used to train the neural network model. Table 1 shows some of the training data.Table 1
Image quality parameters and OSP of optical flow dataset.
DatasetSeqCRMESharpContrastColorsnospMidArmy75768586833.4Backyard86838372768.3MPI_fMarket4776876666711.2Cave3433258636510.5MPI_cMarket495939293922.8Cave364539491933.2KITTI0001568646767739.600022635369586511.3Outdoor videoVideo176678372788.7Video2627485635010.4Video3634676516715.7Video4654179372321.5Video5573653357221.3Video6473256275324.5Video7372436214125.6Video8382337161427.3


## 8. Experimental Result and Analysis

Three public databases: Middlebury [49], MPI_Sintel [50], KITTI [51] and several outdoor videos are used in these experiments. Our proposed algorithm is compared with DFOVOFM [36], TV_L1 [52], HAST [53], MDP_Flow [54], PH_Flow [55], and WRMS based smoothness parameter selection method [27]. All the simulation results are performed on MATALB 10.0 with system configuration of Windows 7, Intel 3.3 GHZ, and 16 GB RAM.The intention of these experiments is to show the superiority of our OSP method.In Figure4 and Table 2, we applied DFOVOFM with different fixed smoothness parameter and our OSP method to show the relationship between the image content and optimal smoothness parameter, and our OSP method has the highest accuracy. In Figure 5 and Table 3, we compared DFOVOFM + OSP with other classical optical flow models; we can see our model is excellent in illumination uneven scenes, illumination insufficient scenes, and illumination changes scenes.Figure 4
Visual comparison of different image sequences from different database with different smoothness parameter setting. (a) Army in Middlebury. (b) Grove in Middlebury. (c) Market_4 in MPI_Sintel. (d) Ambush_3 in MPI_Sintel. (e) Image sequence in outdoor video V1. (f) Image sequence in outdoor video V2.
(a)(b)(c)(d)(e)(f)Table 2
The AEE/AAE of DFOVOFM with different smoothness parameters in Middlebury.
AlgorithmArmyMequnSchefleraWodenGroveUrbanYosemiteTeddyAEEλ=30.080.190.270.160.520.400.160.40λ=50.100.170.250.180.610.360.200.37λ=80.100.190.230.200.640.300.220.43λ=100.110.210.240.250.690.340.300.47λ=150.130.230.280.260.730.420.350.53λ=200.130.250.300.320.780.510.400.58OSP0.080.160.190.150.450.290.150.35AAEλ=32.672.132.282.752.342.682.251.46λ=52.692.032.162.932.392.562.361.33λ=82.732.152.043.142.432.452.481.52λ=102.852.242.203.222.542.572.731.73λ=152.922.362.343.452.652.722.921.95λ=203.042.532.463.622.772.863.052.16OSP2.631.981.962.712.262.422.131.25OSP is the result of our own algorithm, and other bold values are the lowest one (using the fixed smoothness parameter) for each sequence.Figure 5
Visual comparison of different image sequences from MPI_Sintel database of different algorithm. (a) Market_1. (b) Cave_3. (c) Shaman_1. (d) Ambush_1. (e) Market_3.
(a)(b)(c)(d)(e)Table 3
The AEE/AAE of DFOVOFM with different smoothness parameters in MPI_SINTEL.
AlgorithmAEEs0–10s10–40s40+CleanHAST6.8020.6833.02246.33MDP_Flow5.8370.6402.60339.46PH_Flow4.3880.7122.85540.23DFOVOFM + OSP5.2360.6342.93542.87FinalHAST9.1851.6145.85654.78MDP_Flow8.4451.4205.44950.51PH_Flow7.4321.3605.42846.03DFOVOFM + OSP7.941.3375.47348.15Bold values are the result of our OSP method.In Figure6 and Table 4, we compared DFOVOFM + OSP with DFOVOFM + WRMS, TV-L1 + OSP, and TV-L1 + WRMS; we can see DFOVOFM + OSP is the best among these four algorithms.Figure 6
Visual comparison of different image sequences with different algorithm. (a) 00004 sequence. (b) 00009 sequence. (c) 00011 sequence. (d) 00014 sequence. (e) 00015 sequence.
(a)(b)(c)(d)(e)Table 4
AEE/AAE of DFOVOFM with different smoothness parameters in KITTI.
AlgorithmAVE000004000009000011000014000015TV-L1 + WRMS8.85/15.232.16/5.5318.26/25.928.05/14.975.82/8.039.93/18.74TV-L1 + OSP8.25/14.731.87/5.1316.91/24.257.46/13.895.03/7.298.26/16.53DFOVOFM + WRMS8.57/14.921.94/5.2617.68/25.647.73/14.545.47/7.839.75/17.42DFOVOFM + OSP8.13/13.851.83/5.0416.45/23.347.24/13.464.86/7.038.05/15.46Bold values are the result of our OSP method.In Figures7 and 8, we demonstrate that, compared to other algorithms, the DFOVOFM + OSP is excellent in outdoor environment.Figure 7
Visual comparison of an image sequence from outdoor video V3mmc5 of different algorithms. (a) Input 1. (b) Input 2. (c) HAST. (d) MDP_Flow. (e) PH_Flow. (f) TV_L1 + WRMS. (g) DFOVOFM + WRMS. (h) TV_L1 + OSP. (i) DFOVOFM + OSP. (j) GT.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)Figure 8
Visual comparison of an image sequence from outdoor video V4mmc5 of different algorithm. (a) Input 1. (b) Input 2. (c) HAST. (d) MDP_Flow. (e) PH_Flow. (f) TV_L1 + WRMS. (g) DFOVOFM + WRMS. (h) TV_L1 + OSP. (i) DFOVOFM + OSP. (j) GT.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)Figure4 shows the optical flow estimation results for DFOVOFM with different smoothness parameters. From up to down in Figure 4, there are Army and Grove in Middlebury, Market_4 and Ambush_3 in MPI_Sintel, and two outdoor videos V1 and V2mmc5. First three columns from left to right represent different smoothness parameters having value λ equal to 3, 10, and 20, whereas fourth, fifth, and sixth column represent the WRMS based smoothness parameter selection method, our proposed method, and Ground Truth. It can be seen from Figure 4 that performance of each database changes by varying the value of λ. We get better results in Middlebury database when λ=3. However, MPI_Sintel final database performs better when λ=10, while results in outdoor videos are better when λ = 20; we can see that different image quality would have different optimal smoothness parameter. Apparently, it can be clearly seen that our proposed method achieves better results than existing methods for all databases. Our method shows clearer contour than other methods such as gun in the soldier’s hand in sequence Army, the end of the branches in Grove, Monster’s tail in Market_4, and the contour giant head and waving stick in Ambush_1. In the case of the outdoor videos V1, V2, the result of our method has most complete outline and least error regions caused by illumination changes. Table 2 shows the error rate (AEE/AAE, Average Endpoint Error/Average Angular Error) for the DFOVOFM with different smoothness parameters in Middlebury database. The red with bold font letter indicates the minimal error rate for fixed λ while black with bold font letter shows the error rate of our method. It can be seen in Table 2 that the value of λ affects the accuracy of the optical flow estimation and the error rate of our method is the lowest.Figure5 shows the optical flow estimation results for different algorithms in MPI_Sintel database. Each row from top to bottom represents different image sequences that are Market_1, Cave_3, Shaman_1, Ambush_1, and Market_3, whereas each column from left to right shows result of different models that are HAST, MDP_Flow, PH_Flow, our DFOVOFM + OSP, and Ground Truth. It can be seen in Figure 5, compared to other algorithms, the predominance of our method is the robustness in the illumination changes and insufficient illumination scenes, such as Shaman_1; we can see intact contour and bright color for our optical flow estimation result but in other algorithms, the gusset parts are missing. Meanwhile, illumination changes happened on the right side of the running girl in Market_1 which caused big optical flow estimation error in the results of HAST, MDP_Flow, and PH_Flow and our method is the most accurate one. However, our algorithm has a drawback that it does not show the movement of tinny details and some texture details are missing that can be seen in the end of the cone in Cave_3 and the beard of the black man in Ambush_1.Table3 shows the average AEE/AAE of different algorithms in MPI_Sintel database, where s0–10, s10–40, and s40+ represent the largest displacement that are less than 10 pixels, between 10 and 40 pixels, and exceeding 40 pixels, respectively. We can see in Table 3 that the accuracy of our method is higher than HAST while little lower than MDP_Flow and PH_Flow, because some sequences in MPI_Sintel include large displacement and occlusion problems due to which the accuracy of our method is slightly less than its components.Figure6 shows the optical flow estimation results for different algorithms in KITTI database. Each row from top to bottom represents different image sequences that are 00004, 00009, 00011, 00014, and 00015 which include illumination inhomogeneous scenes, whereas each column from left to right shows result of different models that are TV_L1 + WRMS, DFOVOFM + WRMS, TV_L1 + OSP, and DFOVOFM + OSP. We can see that results of both TV_L1 + OSP and DFOVOFM + OSP are better than DFOVOFM + WRMS and TV_L1 + WRMS in all five image sequences. Table 4 shows the AEE of the four optical flow estimation methods for 00004, 00009, 00011, 00014, and 00015 image sequences in KITTI database; AVE means the average of AEE of 50 image sequences. It can be justified that our OSP methods have higher accuracy than WRMS methods.Figures7 and 8 show the experimental results of the outdoor videos V3 (illumination changes and inhomogeneous regions) and V4 (illumination changes and insufficient regions). First two columns of these figures are input image sequences while remaining columns from left to right show the results of HAST, MDP_Flow, PH_Flow, TV_L1 + WRMS, DFOVOFM + WRMS, TV_L1 + OSP, DFOVOFM + OSP, and manual segmentation result, respectively. Results obtained by HAST technique as depicted in Figures 7(c) and 8(c) have relatively lesser error regions, but the object outline is not intact and several parts of human body are missing. For MDP_Flow method, we can get better intact human body in Figures 7(d) and 8(d), but the results contain more error regions meantime. We can see that DFOVOFM model achieves better results than other models while DFOVOFM + OSP produces even more better results with most intact object outline and least error regions. Comparing WRMS methods in Figures 7(f), 7(g), 8(f), and 8(g) with our OSP methods in Figures 7(h), 7(i), 8(h), and 8(i), we can see our OSP method achieves better optical flow estimation.


## 9. Conclusion

An adaptive smoothness parameter method for variational optical flow model is proposed in this work to solve a low matching rate problem of smoothness parameter and the image content. The main idea of this method is to assign different smoothness parameters for subregions with different image quality parameters. These subregions can be obtained by combining SLIC with LMF while smoothness parameters are calculated by image quality parameters (CRME, contrast, sharpness, colorfulness, and signal to noise ratio) and self-designed neural network model. Simulation results validate that, in comparison to the WRMS method, our method achieves better results especially in scenes with different image quality regions. Furthermore, combining with DFOVOFM, compared to other optical flow estimation methods, the method shows better performance in illumination abnormal scenes such as illumination changes, illumination insufficiency, and illumination inhomogeneity.



---

*Source: 7594636-2021-12-29.xml*