Subjective and Objective Quality Assessment for Volumetric Video Compression

Volumetric video is becoming easier to capture and display with the recent technical developments in the acquisition, and display technologies. Using point clouds is a popular way to represent volumetric video for augmented or virtual reality applications. This representation, however, requires a large number of points to achieve a high quality of experience and needs compression before storage and transmission. In this paper, we study the subjective and objective quality assessment results for volumetric video compression, using a state-of-the-art compression algorithm: MPEG Point Cloud Compression Test Model Category 2 (TMC2). We conduct subjective experiments to find the perceptual impacts on compressed volumetric video with different quantization parameters and point counts. Additionally, we find the relationship between the state-of-the-art objective quality metrics and the acquired subjective quality assessment results. To the best of our knowledge, this study is the first to consider TMC2 compression for volumetric video represented as coloured point clouds and study its effects on the perceived quality. The results show that the effect of input point counts for TMC2 compression is not meaningful, and some geometry distortion metrics disagree with the perceived quality. The developed database is publicly available to promote the study of volumetric video compression. Introduction Capturing and displaying volumetric video are becoming easier with the developing technology for augmented reality (AR), mixed reality (MR), and virtual reality (VR) applications. Recent technological developments made the acquisition of volumetric video feasible [1–3]. The acquired volumetric video can easily be used within various AR and VR applications using the head-mounted displays (HMD). Thus, within both industry and the scientific community, the interest to the effective acquisition and display of volumetric video is increasing [3–5]. A popular way of volumetric video representation is using point clouds (PC). As PCs are getting used more commonly [4,6], the interest and need for creating a content delivery chain for PCs increase. In this context, both compression and quality assessment are crucial aspects to improve and spread the use of PCs and volumetric videos. Although there are both subjective [6–14] and objective [15–17] studies on the quality assessment of PCs, the quality assessment of coloured PC compression for acquired volumetric video is still very rare. In fact, to the best of our knowledge, this study is the first to consider TMC2 compression for volumetric video represented as coloured PCs and study its effects on the perceived quality. In this work, we subjectively and objectively evaluate the quality perception for the volumetric video compression scenario using a state-of-the-art PC compression method: MPEG Point Cloud Compression Test Model Category 2 [18] (denoted as TMC2). The contributions of this paper are threefold: (i) The effects of the TMC2 compression method on the perceived quality were analysed for the first time through subjective experiments, (ii) the relationship between the human observers’ perception and the prediction performance of the state-of-the-art objective quality metrics was studied, and (iii) an initial subjective quality database was created to stimulate further research in this subject. Detailed information on the related work, the volumetric video quality database created, subjective experiments conducted, objective quality estimation, and results is presented in the following sections. Related Work A 3D model can be represented using PCs without the need of connectivity information and a texture atlas. Instead, the colour information can be provided with each point. However, for proper visualisation, the number of points should be high, and this increases the size of PCs. Compression is necessary to store and transmit the PCs. Three compression approaches are mainly considered for point cloud compression (PCC): octree-based compression [4, 19], graph-based compression [10], and video-based compression [18]. A few point-based objective measures were proposed to evaluate the geometry distortion in PCC: point-to-point [15], pointto-plane [16], and plane-to-plane [17]. Although these metrics are the state-of-the-art and used in many recent studies [6, 9–13] and in MPEG standardisation activities [4,18], the performance of these metrics was found to be not sufficient to predict the visual quality for different types of contents [12, 13]. To evaluate these objective quality metrics and to understand the effect of visualisation, many subjective quality assessment studies were conducted. Most of these studies concentrate on computer-generated PCs without colour information [6, 8, 9, 12–14], and all of these studies used Gaussian noise or octree pruning as distortions. Some of these studies compared subjective test methodologies [9] and different visualisation strategies [14] which did not yield statistically significant difference. In another study, it is found that visualising PCs as raw PCs and reconstructed surfaces yield statistically significant difference [12]. Coloured PCs were considered only in a few studies. Zhang et al. [7] considered only Gaussian noise either in the location or colour of the points. Mekuria et al. [4] conducted a subjective evaluation for the validation of the proposed codec. Javaheri et al. [10] considered an octree-based and a graph-based compression method and compared geometry-only point-based objective Figure 1. Sample rendered images for PCs at different quality levels: uncompressed reference (leftmost) and compressed PCs where i=1 (centre-left), i=2 (centre), i=3 (centre-right), and i=4 (rightmost). Figure 2. Sample 2D rendered versions of the two selected volumetric videos; (left) ‘Matis’ and (right) ‘Rafa’. quality metrics. Torlig et al. [11] also considers an octree-based method and evaluates the quality of compressed PCs both subjectively and objectively. It is found that projection-based objective quality metrics (i.e. traditional image quality metrics) are better correlated to human viewers’ scores compared to the point-based objective PC quality metrics. Although these studies focus on octree-based or graph-based compression methods, to the best of our knowledge, video-based compression methods have not been studied yet. In this work, we evaluate the video-based TMC2 PCC method both subjectively and objectively. Volumetric Video Quality Database To create a volumetric video quality database, we use two volumetric videos generated by the V-SENSE research group using an affordable set-up with 12 synchronised RGB cameras [3]: Two different people playing with a football as our uncompressed source videos (named ‘Matis’ and ‘Rafa’). Sample 2D rendered versions of these two selected models are presented in Fig. 2. These PCs were selected to have 4 different sampling frequencies (i.e. point counts): p ∈ {62K,127K,250K,495K} to understand the effect of input point counts on the compression, the subjective quality perception, and objective quality estimation. The PCs were compressed using the state-of-the-art MPEG Point Cloud Compression Test Model Category 2 (TMC2) method [18]. This compression method is based on the H.265/HEVC standard [20]. The PC location and colour information are mapped to image pixels and encoded. To see the effect of compression parameters, 4 different quantization parameters (QP) were selected. TMC2 has two different quantization parameters: geometry QP (gQP) and texture QP (tQP). During initial trials, the effect of geometry QP was found to be harsher compared to that of texture. In order to keep the reduction in the perceived quality balanced, a ratio between gQP and tQP was found and was kept the same for these 4 quality levels. The following gQP and tQP values were selected in a pilot study done: (gQP, tQP) ∈ {(17,20),(30,35),(37,43),(41,48)}. The PCs were then encoded with these gQPi and tQPi values for each quality level i ∈ [1,4], using TMC2. In total, 32 different processed video sequences (PVS) were generated for 2 contents × 4 point counts× 4 QPs. Together with the subjective data collected, this volumetric video quality database is made publicly available1. Visualisation for subjective experiment For visualisation of PCs, each point was replaced with small elliptical planes (i.e. splats) to remain true to the data representation of PCs. The size of these splats was arranged so that there would be as little holes as possible (ideally none – see the centreright image in Fig. 3), without causing the model to look swollen. Sample 2D rendered images are presented in Fig. 3 which show the differences in visualisation with respect to splat size. For this purpose, the PCs were loaded to Unity (version 2017.4.1f1), and the scene was rendered and recorded using Unity Recorder tool. The volumetric videos were placed on the origin and rotated on their z-axis to avoid inter-subject variation due to interaction. The objects’ heights were set to 0.5 units, and a camera with perspective projection and 60-degrees field of view was placed 1.143 units away. The videos were comprised of 198 frames, which took 6.6 seconds in 30 frames/second. Subjective Quality Assessment As ‘quality’ is directly related to human perception, subjective quality assessment is the best method to assess multimedia quality. Using single or double stimulus direct rating methods [21] is the most common way to conduct subjective tests. However, very small differences may not be captured by these tests. Pairwise comparisons (PWC) method is easier for subjects to decide; however, it is very hard to obtain accurate and meaningful results when the preference ratios approach 100% (or 0%) [22]. Hence, PWC experiments are not suitable for large quality differences. In this study, we conduct two subjective experiments to analyse both large and small difference