## 3 Framework of Dynamic-DSO
### Direct Visual Odometry
#### 3.1.1 Front-End  
The front-end part handles the following:  
Candidate point extraction: DSO extracts candidate points from the first frame in initialization process and the keyframes selected in frame tracking process. These candidate points will be continuously tracked during the movement of the camera and participate in the pose solving process.  
In dynamic environments, if points in the dynamic area and in the static region participate in the calculation and optimization of photometric error together, it will break the optimization direction, resulting in the decline of pose estimation and mapping accuracy. Dynamic-DSO uses the method described in Section 4.1 to extract candidate points in dynamic environments.  
Frame tracking: The current frame adapts the nearest reference keyframe for motion tracking. The constant velocity model and the traditional two frame image alignment method are used to achieve the course-to-fine tracking on the multi-scale pyramid model, and the pose of current frame is obtained by minimizing the photometric error between current frame and reference keyframe.  
In Section 4.2, we apply image semantic information to the traditional pyramid motion tracking model to reduce the impact of dynamic objects on tracking accuracy.  
Candidate point tracking: The process exists in both keyframes and non-keyframes. For an immature point obtained from the keyframe, DSO projects it to the current frame according to the relative pose and the initial inverse depth, and calculates the direction of epipolar line. The position of the projection point with the smallest photometric error in the current frame is obtained by epipolar search, then the best matching is obtained by minimizing the photometric error with the Gauss-Newton method. After obtaining the best match, the inverse depth and co-variance of the immature point are calculated to constrain the search interval of subsequent frames, as described in [11].  
For dynamic environments, since the prior information of dynamic objects has been obtained in Dynamic-DSO, if the best match position of the immature point is in the dynamic area of current frame, the inverse depth of this point will not be recovered. Then the point will be set as the outer point, as described in Section 4.3.  
Keyframe Creation: Similar to ORB-SLAM2, we take many keyframes (around 5-10 keyframes per second), then sparsify them afterwards through the early marginalization of the redundant keyframes. Three terms are calculated to determine if a new keyframe is required: mean square optical flow (\(f:=\left(\frac{1}{n}\Sigma_{i=1}^{n}\left\|p-p^{\prime}\right\|^{2}\right)^{ \frac{1}{2}}\)), mean flow without rotation (\(f_{t}:=\left(\frac{1}{n}\Sigma_{i=1}^{n}\left\|p-p^{\prime}\right\|^{2}\right) ^{\frac{1}{2}}\)) and relative brightness factor between two frames (\(a:=\left\|\log(e^{j-a_{t}}t_{j}t_{i}^{-1})\right\|\)). When \(f\), \(f_{t}\) and \(a\) satisfies the following relationship, the current frame is selected as a new keyframe.  
\[w_{f}f+w_{f_{t}}f_{t}+w_{a}a>T_{kf} \tag{1}\]  
where \(w_{f},w_{f_{t}}\) and \(w_{a}\) are the weighting terms.  
Marginalization: The step decides which frames and points should be marginalized. A keyframe will be marginalized if the number of points which are visible in the latest frame is less than 5%. If the number of keyframes exceeds \(N_{f}\) (always fixed at 7), then the keyframe, which is far from current frame and close to any other keyframes should be marginalized.