# Optical Flow Estimation with Occlusion Detection
## 1 Introduction  
The concept of optical flow, describing the apparent motion between images or video frames, arises from the studies of biological visual systems. In certain applications, it is desirable to identify optical flow everywhere in the whole image, such as video segmentation [1], three-dimensional reconstruction [2], and video compression [3]. Occlusion destroys the consistency constraint in optical flow estimation, for example, the assumption that the pixel characteristics are unchanged before and after motion. In addition, the consistency constraint may cause unavoidable errors in the occlusion areas. Therefore, correctly distinguishing the two areas and using different optimization strategies can improve the accuracy of optical flow estimation. The methods of optical flow estimation can be roughly divided into three categories: the variational methods [4], the matching methods [5] and the approximate nearest neighbor fields (ANNF) methods [6]. In addition to the above three methods and the mutual integration [7], the Convolutional Neural Network (CNN)-based methods are obtaining considerable attention in recent years. Flownet [8], Flownet2.0 [9] and SPyNet [10] are all the end-to-end convolutional architectures. DM-CNN [11], MC-CNN [12], PatchBatch [13] and many others CNN-based methods are of matching-based architectures. Besides [PERSON] et al. [14], [PERSON] et al. [15] and [PERSON] et al. [16] apply segmentation results to assist optical flow estimation. However, most of the above methods do not deal with occlusion directly, instead they treat them as the outliers and suppress the influence of occlusion by introducing complex penalty terms, such as the Lorentzian potentials [17] and the Charbonnier potentials [18]. This is the compromise using the consistency constraints in the whole image. For the occlusion area, this is the wrong optimization function, so it will lead to the undesirable results. Because it is a global optimization function, ambiguity may even affect the optical flow in the non-occluded areas. Owing to the lack of feature information in the occluded areas, the smoothing term is often used to deduce the optical flow in the occluded area, but it results in over-smoothing optical flow. The image-adaptive, isotropicdiffusion [19], anisotropic diffusion [20], and isotropic diffusion with non-quadratic regularizes [17] are proposed to deal with over-smoothing, but they do not account for occlusions. [PERSON] et al. [21] proposed the jointly estimate disparity and occlusions method, but its huge computation cost is intolerable in most applications. Therefore, in practice, a three-step-approach [22] is proposed, with the occlusion-unaware optical flow firstly estimated, then the occlusion areas identified and finally the optical flow corrected. Complex penalty terms cannot fundamentally eliminate the influence of occlusion. Bidirectional optical flows in joint optimization would result in huge computational redundancy. In addition, the three-step-approach relies heavily on the quality of occlusion detection. Therefore, in this paper, we propose a more robust optical flow algorithm with occlusion detection, and the overall flow of the proposed method is shown in Figure 1. The contributions of this proposed research include two aspects. For occlusion detection, we mine occlusion information from multiple directions, and propose a regional-based fusion strategy. Matching can provide the most basic occlusion detection results, which do not depend on optical flow. However, because it often only uses the simple features, it may appear false positive phenomenon, which may be invalid in edge areas or illumination changes. Warped image is the intermediate result of the coarse to fine warping algorithm, from which we found the trace of occlusion creatively. Through motion analysis, we find that the ghosting artifacts are the  
Figure 1: Algorithm overview. Given adjacent frames, we first compute matching results and edge detection results.