Estimating the Subjective Video Stability of First-Person Videos

First-Person Videos (FPVs) captured by body-mounted cameras are usually too shaky to watch comfortably. Many approaches, either software-based or hardware-based, are proposed for stabilization. Most of them are designed to maximize stability of videos. However, according to our previous work [1], FPVs need to be carefully stabilized to maintain their First-Person Motion information (FPMI). To stabilize FPVs appropriately, we propose a new video stability estimator Viewing Experience under “Central bias + Uniform” model (VECU) for FPVs on the basis of [1]. We first discuss stability estimators and their role in applications. Based on the discussion and our application target, we design a subjective test using real scene videos with synthetic camera motions to help us to improve the human perception model proposed in [1]. The proposed estimator VECU measures the absolute stability and the experimental results show that it has a good interval scale and outperforms existing stability estimators in predicting subjective scores. Introduction As wearable cameras, such as GoPro or Pivothead, become popular, many people start to capture and share their own FirstPerson Videos (FPVs). FPVs allow viewers to revisit recorders’ life-logs in the First-Person Perspective. The interesting parts of FPVs are not only the objects recorded in the video, but also the recorder’s reactions reflected by their First-Person Motions. However, there is a problem with FPVs: most of them are too shaky for humans to watch comfortably. To solve this problem, both software-based and hardwarebased approaches can be applied. Hardware-based approaches are mainly based on gimbals. Software-based approaches [2–10] estimate the camera motion in either 2D or 3D and then use filters to smooth the computed camera motion and re-project frames to create a stabilized sequence. Nevertheless, these approaches treat the FPVs as normal shaky videos and try to stabilize them as much as possible, which damages the First-Person Motion Information (FPMI) with high probability [1]. When the FPMI is damaged, the viewing experience is limited, which degrades FPVs to normal videos. To appropriately stabilize a FPV, it is necessary to have an accurate estimator of its stability. Quality estimators (QEs) characterize the quality of images or videos, which is useful for many applications. For a stability estimator, two relevant applications are algorithm optimization and product benchmarking [11]. Usually, QEs are classified into No-Reference, Reduced-Reference and Full-Reference estimators, but they can also be categorized by their purpose [11]. Using this principle, we partition stability estimators in 3 ways, which are: (1) Whether the estimator measures the relative or absolute stability of the video, (2) Whether the estimator is based on an objective model or a perceptually-motivated model, (3) Whether the estimator measures the ordinal scale or interval scale of stabilities of different videos. Relative or Absolute: Some applications are only interested in comparing relative video stability or relative stabilization performance of different algorithms. In this situation, the relative measurement is sufficient. The Motion-Vector-based Mean Squared Error (MV-MSE) [12] is an example of such an estimator. MV-MSE measures the MSE between the intentional motion and the original motion. The intentional camera motion can be computed in many ways. For example, in [12], the original motion is transformed into the frequency domain and low-pass filtered to generate the intentional motion. However, no matter how accurate the computed intentional motion is, MV-MSE can only measure the instability of the original motion with respect to the intentional motion. The results in [1] indicate that the subjective stability of intentional motions strictly depends on their amplitude and frequency characteristics, which means the intentional motions themselves are seldom fully stable. As a result, it is hard for MV-MSE to measure the absolute stability of videos. Some applications want to know the absolute stability of videos, for example the video stabilization systems. Based on their criteria of “stable”, they can stabilize the video to the degree they want. Several works [6–10] for video stabilization techniques implicitly generate stability estimators as side-products. When creating the smooth camera path, they perform optimization processes which minimize the sum of squared values of the second derivative of motion curves. This objective function can be thought of as a stability estimator. Smaller values indicate higher stability. Some other objective functions are also adopted. For example, in [7], the mean square value of the third derivative of motion curves is also used. In [6], the difference of simplified affine motion parameters between adjacent frames is used. Objective or Perceptually-motivated: Applications that need fast computation or fair comparison prefer to use objective estimators. These estimators mainly use objective information of the video. As a result, these estimators are not favorably biased toward any stabilization techniques. However, the scores of these estimators may not have high correlation with subjective scores. A good example is the Inter-frame Transformation Fidelity (ITF), which was proposed in [13] and used by many works [14–17] for comparing performance of different stabilization methods. First, the Peak Signal-to-Noise Ratio (PSNR) between adjacent frames are computed. The mean value of PSNR of all adjacent pairs is the ITF of the video, which can be computed easily and quickly. It is believed that the PSNR does not have a high correlation with subjective quality scores [18, 19], which means ITF is objective. For applications that are interested in specific problems of video stability, perceptually-motivated estimators are more useful. In our previous works [1,20], we focused on stabilizing FPVs while preserving their First-Person Motion Information (FPMI). In [20], we first explored the stabilization technique by considering the human perception model. Then in [1], we carefully modeled the perception model and proposed a video stability estimator, the Viewing Experience (VE) score. By modeling the smooth pursuit eye movements of a viewer, the VE score estimates the fraction of the FPV that the viewer can watch comfortably. [1] showed that VE scores correctly characterized the rank order of 3 versions of stabilized videos for several scenes. Also, the stabilization algorithm based on VE scores has similar stabilization performance with Microsoft Hyperlapse [8, 21] and preserves much more FPMI. The estimators from stabilization techniques [6–10] are somewhere between these two categorizations. They consider that viewers prefer smooth motion but do not explore the perception model. As a result, they do not have high correlation with subjective scores and are also not fair enough for comparing different algorithms. Ordinal scale or Interval scale: The results in [1] show that VE scores based on this model well reflect human subjective scores. However, they only have similar rank to subjective scores, which means they have a good ordinal scale [22]. For example, if the subjective scores of three version of the videos have the relationship: SJ1 < SJ2 < SJ3, then our VE score is: V E1 <V E2 < V E3. It is hard to use V Ei to accurately predict SJi, which means V Ei and SJi do not have similar interval scales [22]. An estimator that has a good interval scale is more valuable. For example, watching videos with First-Person Motion may cause dizziness or motion sickness. It is useful to measure the impact of these side-effects, which is possible if the estimator can precisely predict the subjective video stability. In this paper we aim to find a such video stability estimator than can help us predict the subjective video stability and guide us to appropriately stabilize FPVs. To achieve this target, the estimator needs to measure the absolute stability and be based on a perceptually-motivated model, which is satisfied by our precious VE score. For this reason, we only need to design a subjective test to improve the VE score to have a good interval scale by considering saliency models and refining the basic human perception model. In the next section, we briefly review the human perception model in our VE score and point out the parts need improving. In section 3, we introduce the subjective test including the video sources and test design. The experimental results are shown in section 4. We conclude our work and summarize interesting findings in section 5. Viewing experience score Our proposed video stability measurement [1] is called the viewing experience (VE) score, which measures the fraction of frames of the FPV that viewers can watch comfortably. In this section, we first review the motivation and the mathematical model of our VE score. After that we discuss the current limitation of our VE score, which inspires this work. Human perception on stability The motivation of our VE score is that we perform different eye movements in real life and when we are watching videos. Our proposed VE score measures the video stability based on our mathematical model of eye movements when we are watching videos. In real life, we perform vestibulo-ocular reflex/eye movement [23] to physically compensate the unintentional motion of our body to retain our visual target fix at the center of the retina. This eye movement uses the information from our internal-ears and the reaction time is around 0.01 second [23]. However, the eye movement we are using is called smooth pursuit eye movement (SPEM) and saccade [24]. The saccade helps us to catch our visual target and the SPEM uses and only uses the visual clues to assist us to smoothly track the catched target. The time our visual system needs for perform a saccade is 0.15 to 0.2 second [25], which is much longer than the ve