# Estimating Optical Flow with Streaming Perception and Changing Trend Aiming to Complex Scenarios
## 3 Method
### Training Loss  
In order to correct the parameters in the network more effectively, this paper designed a combined loss for the proposed structure. It consists of three different types of cost functions, including motion cost, angle cost, and trend cost. The first part of the loss is called motion cost. It takes the Smooth L1 Loss [28] between the estimation of optical flow and the ground truth of optical flow, which improves the problem of not being smooth enough when it tends to zero compared to the L1 Loss. In addition, it is less sensitive to outliers when the input value is large compared to the L2 Loss. The loss function of the first part is defined as:  
\[\mathcal{L}_{i}^{Motion}=\parallel Smooth_{L_{1}}\Big{(}f_{i}^{gt}-f_{i}^{ pred}\Big{)}\parallel_{1} \tag{1}\]  
\[Smooth_{L_{1}}(x)=\left\{ \begin{matrix}0.5x^{2},&if\ |x|<1\\ |x|-0.5&\ otherwise\end{matrix} \right. \tag{2}\]  
where the \(\mathcal{L}_{i}^{Motion}\) denotes the momentary displacement loss of the ith optical flow frame; \(f_{i}^{gt}\) and \(f_{i}^{pred}\) denote the ground truth and estimation of the ith optical flow frame respectively. The second part of the loss is angle cost introduces the directional differences between the estimation and ground truth. L1 Loss used in RAFT only takes the absolute value of the difference between the optical flow estimation and ground truth. It apparently ignores the error information in the moving direction or offset angle of each pixel. To address this drawback, the loss function in this paper additionally introduced the angle cost proposed by [PERSON], [PERSON], et al. [29] to assist the motion cost:  
\[\mathcal{L}_{i}^{Angle}=\sum_{j,k}\Big{[}1-2\sin^{2}\Big{(}yaw-\frac{\pi}{4} \Big{)}\Big{]}_{j\times k} \tag{3}\]  
where yaw is the offset angle from the predicted estimation of optical flow \(f_{i}^{pred}(j,k)\) to the ground truth optical flow \(f_{i}^{gt}(j,k)\) in pixel-wise, and its calculation formula is referred to in [29]. The third part of the loss is trend cost, specifically designed for the fused optical flow features with the purpose of comparing the trend of the estimated optical flow with the trend of the ground truth optical flow. After this, the former can be regressed towards the latter as much as possible. When calculating the pixel-wised trend of the optical flow along the two dimensions (height, width) of the images, to increase the penalty for the case where the trend of the estimation is exactly opposite to the trend of the ground truth during network learning, the cost function of this part is designed as a Differential L2 Loss, which is defined as:  
\[\mathcal{L}_{i}^{Trend}=\sum_{j,k}\Biggl{[}\sqrt{\Big{(}\Gamma_{i}^{gt}-\Gamma _{i}^{pred}\Big{)}^{2}}\Biggr{]}_{j\times k} \tag{4}\]  
where \(\mathcal{L}_{i}^{Trend}\) denotes the continuous inter-frame trend loss from \(i\) to \(i-1\) frame of optical flow, which obviously is calculated from \(i=1\), and the initial value is set to 0; \(j\) and \(k\) represent the number of rows and columns in the image matrix. On this basis, considering that the optical flow trend can have significant differences even in two continuous moments at the positions where the moving objects completely leave (i.e., the contour of the moving object) or some new moving objects appear (especially at the boundary of the image). The most important impact on the global error for this case includes two situations of trend changing: abrupt changes in the optical flow state (from the static state to the dynamic state, or vice versa) and reversal of the optical flow's direction. To regulate the sensitivity to the emergence of new moving objects, we introducedhyperparameters \(\delta\) and \(\mu\). These parameters establish the weights of trend loss in the total loss. As the optimal combinations of these two hyperparameters, grid search could be a solution, and the details of this experiment are also tabled in Section 4.3.