# Selection of Key Frames for 3D Reconstruction in Real Time
## 3 Materials and Methods
### Motion Requirements  
In the last section, we introduced a quality threshold to neglect blurry frames and label the corresponding inertial data. Subsequently, we defined a metric to identify inertial data captured when no motion took place. The underlying assumption was that, while the user does not move during the recording, the camera produces redundant frames containing the same image information as the frames before. Thus, we aimed to reduce the number of redundant frames by means of the IMU. The optical flow determines pixel motion while holding the camera in hand with approximately no motion. Optical flow denotes the motion of pixels in a certain area between two consecutive frames. Motion occurs due to camera motion or moving objects. There are different approaches that measure the optical flow. A widely used algorithm to calculate the optical flow is the Lucas-Kanade (LK) algorithm [8], which exhibits a good performance and a low error rate and is easy to implement [15]. The LK algorithm applies to sparse scenes that define a window of n = mxm pixels around a point of interest, so-called feature points. A feature detection algorithm, such as the scale-invariant feature transform (SIFT) [16] or Harris corner detection [17], finds those points. The method assumes that the motion within such a patch is constant. However, if the motion is too fast, the direction of the intensity change is not trackable anymore because the feature point moves out of the window and violates the constant brightness assumption. Pixel motion is described by a 2D vector as \(\overset{\rightarrow}{V}=\left[u,v\right]^{T}\) between two consecutive frames. The main assumption is that the intensity of a pixel does not change between two consecutive frames, captured in a period \(\Delta\)t. Obviously, \(\Delta\)t must be small enough. The assumption provided by [PERSON] and [PERSON] [18] with the first order Taylor series yields  
\[\text{I(x,y,t)}=\text{I(x}+\Delta\text{x,y}+\Delta\text{y,t}+\Delta\text{t}) \tag{3}\]  
where I(x,y,t) denotes the intensity of a pixel, \(\Delta\)x, \(\Delta\)y represent the change in image coordinates, and \(\Delta\)t represents the change in time.  
Motion Threshold  
The main purpose was to define the minimum amount of motion in terms of inertial data according to the optical flow. Therefore, we used the LK algorithm, which was implemented in the OpenCV open-source computer vision library [19], paired with the Shi-Tomasi corner detection [20] to detect the corresponding feature points. While recording, the camera was in a static position at approximately no motion to investigate the optical flow. The optical flow per frame was averaged. First, we extracted the Euclidean distance from each vector \(\overset{\rightarrow}{V}\) as:  
\[d=\sqrt{u^{2}+v^{2}} \tag{4}\]  
and calculated the mean Euclidean distance \(\overline{d}=\frac{1}{N}\sum_{i=1}^{N}d_{i}\) of the vectors in each frame. Since our motion requirement is to distinguish between motion and no motion, we wanted to specify a threshold to label frames either as _bad_ to filter redundant images, since they were not recorded during motion, or as _good_, due to the opposite case.