Subjective evaluation of distortions in first-person videos

First-person videos (FPVs) recorded by wearable cameras have different characteristics compared to mobile videos. Video frames in FPVs are subject to blur, rotation, shear and fisheye distortions. We design a subjective test that uses actual captured images with real distortions, synthetic distortions or a combination of both. Results indicate shear is less sensitive to content than rotation. For fisheye, personal preference and content dependence affect the subjective results. The performance of 7 noreference (NR) quality estimators (QEs) and our QE, local visual information (LVI) [1], are evaluated based on subjective results. We propose two mapping functions for rotation and shear that improve the ability of LVI and 4 NR QEs to accurately predict the subjective scores. Introduction As wearable cameras gain increasingly popularity, the socalled first-person videos (FPVs) grow to be an important video source over networks [2]. FPVs have different attributes compared to mobile videos. When recording mobile videos, people can see the ongoing scene in the screen, so they can be aware of how to fit the scene into the camera shot and intentionally control the video to be less shaky. However, FPVs are passive videos, and often lack real-time feedback about the field of view. As a result, the wearer rarely adjusts the camera to be stable or horizontal. Recently, the increased volume of uploading, transmission and storage of FPVs has given rise to a variety of applications. Human Activities in FPVs are classified in [3]. The detection of “snap points” was proposed to extract frames that are worthy to be captured and stored [4]. The ego-sampling method was developed to sample useful frames for faster video browsing [5]. Because of the capture process of FPVs, quality degradations are not limited to transmission or post-processing, and the resulting distortions in frames have not been subjectively evaluated. Motion blur and geometric distortions are two major distortions in FPVs [1]. Motion blur mainly arises from fast motion of the camera. When the camera keeps changing its positions in the scan time of one frame, the captured scene is blurred. Geometric distortions can be classified into 3 categories: rotation, shear and fisheye. Rotation results from head or body rotation. Wearers regularly move their bodies and shake their heads, and rarely are aware whether or not the camera is kept horizontal while recording videos. Shear is caused by camera panning. When the camera changes its positions in the scan time of one frame, the top rows of the frame are not vertically align with the bottom rows. For example, architecture in a sheared image is visually skewed. Fisheye images are captured by wearable cameras with ultra wide-angle lens (i.e. Gopro, Looxie Camera, Mobius). Instead of capturing a rectilinear image, the content appears to be convex. Fisheye is one of the lens distortions, called barrel transformation. The transformation warps the image to be bent; the magnification decreases from center to margins [6]. Most existing subjective tests [7, 8, 9] explored quality degradations starting from an undistorted image, which is considered to be the reference image. By synthetically adding distortions (i.e. Gaussian blur, JPEG, JPEG 2000, noise) to the reference image with different degradation levels, a series of distorted images of the same content but different severity are created. To design a subjective test for FPVs, however, applying each type of distortions separately fails to consider two issues: 1. Degradations in actual images captured by a camera may be subtly different than those created synthetically using a model [10]. In our test, images with real, synthetic or real plus synthetic distortions are evaluated. The real images are extracted from frames in FPVs. 2. Many blurry images are subject to geometric distortions (i.e. rotation, shear or fisheye) simultaneously in FPVs. The question is what is the visual impact on the overall quality when one image has multiple distortions. Multiply-distorted images have been evaluated in the subjective test, but only for blur, JPEG and noise [11]. In addition, these distortions, like blur, JPEG or noise, all have pixel-to-pixel correspondence, whereas geometric distortions do not. When constructing images for the subjective test for different amounts of geometric distortions, we keep the same content in the image center to maintain a consistent focus of attention. Actual images captured by a wearable camera have no reference image, so full-reference (FR) quality estimators (QEs) [12, 13] and reduced-reference (RR) QEs [14, 15] cannot be used to measure their quality, but no-reference (NR) QEs [16, 17, 18] are commonly content dependent and show very different behaviors for different distortions applied to the same image. Using our subjective test, we evaluate the behaviors of different NR QEs for actual captured images and multiply-distorted images. We design a subjective test to evaluate motion blur and geometric distortions in FPVs. Section 2 compares existing QEs with our QE, local visual information (LVI) [1]. The strategy of the quality estimation in LVI cannot be categorized into FR, RR or NR methods. In section 3, our subjective test evaluates actual captured images with real distortions, synthetic distortions or a combination of both using the paired comparison method. The types of distortions include motion blur, rotation, shear and fisheye. In section 4, we proposed two mapping functions for rotation and shear to compute the overall quality of images with blur and geometric distortions. Personal preferences and content dependence in fisheye are discussed. Comparison of LVI with existing QEs FR QEs and RR QEs often use a reference image to estimate the quality of a distorted image with pixel-to-pixel correspondence. They cannot be used for any image for which there is no corresponding reference. NR QEs uses only a single image without any reference to compute a quality score. In [1], we introduced a QE, LVI, that does not fit in the typical categorization of QEs into FR, RR and NR [19]. LVI estimates the quality of a test image based on a pseudo-reference image. Unlike in FR QEs, the pseudo-reference image and the test image are not necessarily pixel-aligned but obey the constraint that they share overlapping content. For example, the two images could be captured by a camera in the same scene but different camera angles. The geometric relationship between the two images are usually modeled by a homography. Since the quality of the pseudo-reference image is unknown, LVI is a relative quality estimation instead of an absolute quality score as computed by FR or NR QEs. LVI primarily measures the amount of relative blur of the test image based on the pseudoreference. When the test image is more blurred than the pseudoreference image, its LVI score would be in the range from 0 to 1. Otherwise, its LVI score would be larger than 1, which indicates the test image is sharper. If the two images have too few matching feature points, we then set the LVI score of the test image to be 0. LVI is insensitive to rotation and shear [1]. To make LVI be able to measure geometric distortions, we need to include rotation and shear into its quality estimation. We design a subjective test to evaluate multiply-distorted (blur with geometric distortions) images. The subjective results are used to develop the rotation and the shear mapping functions for LVI. The mapping functions take the rotation or shear and the LVI score as inputs to compute the overall quality score of an image with simultaneous blur and geometric distortions. Subjective Test Our test employs the paired comparison method for still images containing both actual and synthetic distortions that are typical of images extracted from FPVs. Each pair of test images are simultaneously displayed on two monitors side by side. Viewing distance is kept to be around 3 times the height of test images. 9 videos have been recorded by a Pivothead camera (frame rate: 30fps, resolution: 1920×1080) including “billiards”, “eating”, “flight”, “bell tower”,“winter Hovde Hall”, “parking lot”, “autumn Hovde Hall”, “apartment building” and “parking garage”. 4 distortions including motion blur, rotation, shear and fisheye, are evaluated. Test images are either real frames from the 9 videos or created by adding synthetic distortions to selected frames. Because synthetic shear and fisheye change the image size compared to the original image, all test images are cropped to be 1600×900 to remove marginal regions with little content. Test Method In a paired comparison subjective test, the subject needs to indicate his or her preference among the two images according to their visual quality. 50 subjects including 43 males and 7 females participated in the test. All pairs of images are displayed in random order. To improve the efficiency of paired comparison, we use the “square design” in [20]. Given that we have n stimulus, a full comparison needs 0.5n(n− 1) pairs. By using the “square (a) (b) (c)