SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We set the field of view (FoV) to the  $110^{\circ}$  following [12, 33]. We use pre-trained Swin Transformer [23] (base version) as our feature extraction backbone. The input viewpoint size  $H \times W$  is fixed to  $224 \times 224$ . The number of viewpoint sequences  $N$  is set to 3 and the length of each sequence  $M$  is set to 5. We set the coordinates of  $N$  starting points to be  $(0^{\circ}, 0^{\circ})$ . The reduced dimension  $D$  is 128 and the number of GRU modules is set to 6. The number of CA operations  $n$  is 4. We set  $\gamma = 0.7$  and  $\beta = 100$  as decreasing factor and scale factor values respectively.
For a fair comparison, we randomly split  $80\%$  ODIs of each dataset for training, and the remaining  $20\%$  is used for testing following [53, 18, 12, 17]. To eliminate bias, we run a random train-test splitting process ten times and show the median result. We train 300 epochs with batch size 4 on CVIQD [35], OIA [11], IQA-ODI [46], and MVAQD [18] datasets without the authentic scanpath data. Respectively, we compare our RPS with two advanced learning-based scanpath prediction methods ScanGAN360 [24] and ScanDMM [32] on JUFE [12] and JXUFE [33] datasets which have the authentic scanpath data. For optimization, we use Adam [21] and the learning rate is set to  $1 \times 10^{-5}$  in the training phase. We employ MSE loss to train our model. We use Spearman rank-ordered correlation (SRCC) and Pearson linear correlation (PLCC) as the evaluation metrics.
4.2 Comparing with the State-of-the-art Methods
We conduct a comparative analysis of Assessor360 with eight FR methods and thirteen NR methods. The quantitative comparison results are presented in Table 1, demonstrating significant performance
Table 2: Cross-dataset validation SRCC and PLCC results of SOTA methods. These models (except WS-PSNR [37] and WS-SSIM [61]) are trained on CVIQD [35], OIQA [11] and MVAQD [18] datasets (80% set) and tested on three other datasets (full set).
Method	CVIQD	OIQA	MVAQD	CVIQD	MVAQD
OIQA	IQA-ODI	MVAQD	CVIQD	IQA-ODI	MVAQD	CVIQD	OIQA	IQA-ODI
SRCC
WS-PSNR	0.5027	0.4360	0.7225	0.7638	0.4360	0.7225	0.7638	0.5027	0.4360
WS-SSIM	0.5442	0.5032	0.7930	0.6625	0.5032	0.7930	0.6625	0.5442	0.5032
MC360IQA	0.4189	0.7114	0.0296	0.7044	0.5687	0.4081	0.0373	0.0025	0.0486
VGCN	0.2361	0.2875	0.2452	0.6932	0.3873	0.4682	0.4650	0.6227	0.3921
Assessor360	0.4597	0.8610	0.5640	0.8430	0.8751	0.6417	0.8756	0.7765	0.8646
PLCC
WS-PSNR	0.4701	0.5468	0.6962	0.7895	0.5468	0.6962	0.7895	0.4701	0.5468
WS-SSIM	0.4363	0.5941	0.6246	0.6536	0.5941	0.6246	0.6536	0.4363	0.5941
MC360IQA	0.4295	0.7872	0.0404	0.7368	0.5930	0.4238	0.0430	0.0202	0.0646
VGCN	0.2582	0.3127	0.2467	0.5929	0.3551	0.2419	0.3420	0.4642	0.3870
Assessor360	0.5332	0.9032	0.5824	0.8636	0.9137	0.6565	0.7232	0.7287	0.8541
Table 3: Cross-dataset validation SRCC results of SOTA methods. These models are trained on MVAQD [18] (full set) and tested on CVIQD [35] and OIAQ [11] all distortion data except MP-BOIQA (removing AVC on CVIQD).
Testing set	MC360IQA	MP-BOIQA	Assessor360
OIQA	0.2542	0.5043	0.6658
CVIQD	0.4749	0.7992	0.8994
Table 4: Quantitative comparison of using different viewpoint sequence generation methods on OIQA [11] and MVAQD [18].
Generation Method	OIQA	MVAQD
SRCC	PLCC	SRCC	PLCC
Random Generation	0.9461	0.9444	0.9359	0.9543
ScanGAN360	0.9705	0.9670	0.9493	0.9694
ScanDMM	0.9652	0.9634	0.9558	0.9612
RPS (Ours)	0.9802	0.9747	0.9607	0.9720
improvements across all datasets against the state-of-the-art methods. Note that these performance enhancements are obtained without training with the 2D IQA dataset, as employed in VGCN [43]. Specifically, Assessor360 outperforms VGCN by  $3\%$  in terms of SRCC on the OIQA dataset and shows a  $1.2\%$  increase in PLCC on the CVIQD dataset. Additionally, our method exhibits a notable improvement of up to  $1.6\%$  in SRCC compared to AHGCN [14].
4.3 Cross-Dataset Evaluation
To assess the generalization capability of our method, we perform cross-validation on the CVIQD [35] and OIQA [11] datasets and compare it against two widely used state-of-the-art methods: MC360IQA [36] and MP-BOIQA [17]. All models are trained on the MVAQD [18] dataset and subsequently tested on the CVIQD and OIQA datasets. The results, presented in Table 3, demonstrate that our method exhibits superior generalization performance compared to the other models. It is worthy to note that, there exists a significant domain gap between the CVIQD and MVAQD datasets. The CVIQD dataset primarily focuses on degradations such as JPEG, H.264/AVC, and H.265/HEVC, while the MVAQD dataset concentrates on JPEG, JP2K, HEVC, WN, and GB. This discrepancy in focus poses a challenge for MC360IQA and MP-BOIQA, as they use the same perspective viewpoints to evaluate different distorted ODIs, leading to poor generalization. In contrast, our proposed RPS overcomes this domain gap effectively by generating different viewpoint sequences for ODIs with different distortions. It consistently performs well, showcasing its effectiveness in addressing the challenges posed by diversely distorted ODIs.
Furthermore, to verify that the performance presented in Table 1 is not due to the overfitting, we retrain MC360IQA, VGCN [43] and Assessor360 on CVIQD, OIAQ and MVAQD datasets, using  $80\%$  of the dataset for training and the remaining  $20\%$  for testing. We then select the model weights with the highest performance on the testset for cross-dataset testing for fair generalization comparisons. The results shown in the Table 2 clearly demonstrate the strong generalization of our method compared to VGCN and MC360IQA. Notably, during instances of training on the MVAQD dataset and subsequent testing on the IQA-ODI [46] dataset, our approach outperforms MC360IQA in terms of SRCC by an approximate margin of 0.8, and surpasses VGCN in SRCC by 0.5. Concurrently, when training on the CVIQD dataset, characterized by fewer distortion types, and testing on the MVAQD dataset, which encompasses more distortion types (JP2K, WN, GB), our approach attains elevated PLCC values of 0.5 and 0.3, respectively, as compared to MC360IQA and VGCN.
Table 5: Quantitative comparison of different generation methods (RPS vs ScanGAN360 [24] and ScanDMM [32]) with metrics of scanpath prediction task on JUFE [12] and JXUFE [33] datasets with authentic scanpaths.
Published Time	Generation Method	JUFE [12]	JXUFE [33]
LEV↓	DTW↓	REC↑	LEV↓	DTW↓	REC↑
-	Random Baseline (lower bound)	35.21	1707.45	0.38	35.08	1695.93	0.38
TVCG22	ScanGAN360 [24]	32.53	1448.65	1.07	31.89	1427.55	1.14
CVPR23	ScanDMM [32]	31.23	1434.36	1.21	31.48	1438.29	1.12
-	RPS w/o DSP (Ours)	29.54	1471.82	2.14	29.99	1463.38	1.94
-	RPS (Ours)	29.48	1454.03	2.21	29.66	1422.85	2.07
-	Human Baseline (upper bound)	23.85	1309.29	3.78	26.73	1302.15	2.88
Figure 3: Visual comparison of the generated viewpoint positions for different methods on ODIs with four distortion types in JUFE. The brighter the area, the more viewpoints are generated in that area.
4.4 Effectiveness of Recursive Probability Sampling
As mentioned in Section 1, there exist many learning-based scanpath prediction methods [24, 47, 32]. They seem to be able to assist in constructing viewpoint sequences. However, they are hardly introduced to OIAQA task. In this section, we first perform the quantitative comparison of the model with RPS and two advanced 360-degree scanpath prediction methods, namely ScanGAN360 [24] and ScanDMM [32] on the datasets without real observed scanpath. Subsequently, we compare the position and sequential order of viewpoints generated by the three methods with the ground-truth (GT) scanpaths by the metrics of scanpath prediction task and visualization to further validate the superiority of RPS on JUFE [12] and JXUFE [33] datasets with real observed scanpath.
Quantitative comparison of performance. We replace the proposed RPS method with ScanGAN360 and ScanDMM to generate sequences of viewports. The model was trained and tested using these viewpoint sequences on OIQA [11] and MVAQD [18] datasets, maintaining the same length and number of viewpoints for a fair comparison. Table 4 shows the quantitative results, demonstrating that viewpoints generated from RPS yield superior performance compared to ScanGAN360 and ScanDMM. Additionally, training using viewpoints generated from these methods outperforms those generated using random schemes, highlighting the crucial role of suitable viewpoint sequences in the OIQA task.
Moreover, we conduct experiments by replacing original VGCN [43] sampling methods with RPS in VGCN. We use RPS to sample the same number of viewpoints as [43] to train VGCN on IQA-ODI [46] and MVAQD datasets. The results presented in Table 7 demonstrate a substantial performance enhancement for VGCN achieved by the viewpoint sampled through RPS, resulting in an increase of 0.07 in SRCC for MVAQD. Additionally, this indicates that the viewpoint sampled with RPS closely aligns with human observations.
Table 6: Quantitative comparison of different starting point positions on MVAQD [18] dataset.
Position (latitude, longitude)	SRCC	PLCC
(0°, 0°), (0°, 0°), (0°, 0°)	0.9607	0.9720
(60°, 0°), (60°, 0°), (60°, 0°)	0.9106	0.9312
(0°, 120°), (0°, 0°), (0°, -60°)	0.9599	0.9660
(60°, 120°), (60°, 0°), (60°, -60°)	0.9174	0.9455
Table 8: Ablation studies of each component in proposed Assessor360 on MVAQD [18] dataset.
Method	Para (M)	SRCC	PLCC
Assessor360 w/o MFA	88.53	0.8514	0.9171
Assessor360 w/o DAB	88.86	0.8437	0.8779
Assessor360 w/ GAP	88.69	0.9393	0.9587
Assessor360 w/ GRU	89.28	0.9607	0.9720
Table 7: Quantitative comparison of using original VGCN sampling method and proposed RPS on IQA-ODI [46] and MVAQD [18] datasets.
Method	IQA-ODI	MVAQD
SRCC	PLCC	SRCC	PLCC
VGCN	0.8117	0.8823	0.8422	0.9112
VGCN-RPS	0.8382	0.8883	0.9122	0.9273
Table 9: Quantitative comparison of using GT sequences and sequences generated by RPS on JUFE [12] dataset. Starting Point (SP).
Viewport
Sequence	Good SP	Bad SP
SRCC	PLCC	SRCC	PLCC
RPS (Ours)	0.6623	0.6365	0.5044	0.4946
GT Sequence	0.7158	0.7013	0.5400	0.5377
Comparison of viewpoint sampling positions and order. Since each ODI in JUFE and JXUFE comprises 30 and 22 scanpaths, respectively, we employ the three methods to generate 30 and 22 sequences, each comprising 20 viewpoints, for each ODI in JUFE and JXUFE, respectively. For each GT sequence, we sample 20 viewpoints from 300 viewpoints with equal time intervals. We adopt Levenshtein distance (LEV) and dynamic time warping (DTW) metrics, as used in ScanGAN360 and ScanDMM, to evaluate the position and sequential order of the viewpoints. Furthermore, the viewing behaviors and patterns can be evaluated using the recurrence measure (REC) [24]. Each metric is calculated by comparing each generated sequence against all the GT sequences and computing the average, resulting in the final value. Besides, to establish an upper bound for each metric, we compute the Human Baseline by averaging the results of comparing each GT sequence against all other GT sequences [41]. Similarly, we establish a lower bound by randomly sampling viewpoints from the ODI, referred to as the Random Baseline.
The quantitative comparison results are reported in Table 5. The proposed unlearnable method outperforms learning-based models regarding the LEV and REC metrics. Specifically, it achieves a LEV improvement of 1.75 and 1.82 compared to ScanDMM [32], and a REC improvement of 1.14 and 0.93 compared to ScanGAN360 [24]. Additionally, our method achieves comparable performance to ScanDMM and ScanGAN360 on the DTW metric for both datasets. Figure 3 presents the positions of generated viewpoints for different generation methods. The panorama is divided into 72 regions, with each region spanning  $15^{\circ}$  degrees of latitude and  $30^{\circ}$  degrees of longitude. It can be observed that the positions generated by the learning-based method are relatively dispersed, while the positions generated by RPS are concentrated near the equator. The concentration gradually decreases with increasing latitude, aligning closely with the ground truth (GT) position.
4.5 Ablation Studies
Impact of the initialization of the starting point. For the ODI without real scanpath data, we initialize the coordinates of  $N$  starting points to be  $(0^{\circ}, 0^{\circ})$  followed by [33]. We conduct experiments to assess the impact of different initial starting points positions on the final performance. Results shown in Table 6 reveals that the model's performance displays relatively minor fluctuations during longitudinal shifts, yet experiences a substantial decline when changing larger latitudes (from points less frequented by humans as starting positions). This divergence arises due to the model's misalignment with authentic browsing behaviors (humans tend to start their panoramic observations closer to the equator). It further emphasizes that simulating actual scanpaths assists in more accurate image quality evaluation.
Effectiveness of MFA and TMM. We exclude the multi-scale feature aggregation (MFA) and distortion-aware Block (DAB) to assess their effectiveness in MVAQD [18]. Specifically, for the previous experiment, we utilize the final stage of the Swin Transformer [23] for feature output. The results, shown in Table 8, indicate that MFA and DAB substantially contribute to the performance, with only a minor increase in the network parameters. The GRU [5] module in TMM is replaced with global average pooling (GAP) for predicting the sequence quality score. The effectiveness of the GRU module in representing temporal transition information is demonstrated in Table 8, and it can assist the model in predicting more accurate quality scores.
Figure 4: Performance of different number and length of viewpoint sequence on MVAQD [18].
Impact of the number and length of the viewpoint sequence. We test  $N = 1,3,5$  three different numbers of viewpoint sequences with varying sequence lengths on MVAQD [18] dataset. The findings, shown in Figure 4, reveal that as the sequence length increases, there is a decreasing trend in model performance across all three numbers of sequences. This suggests that an excessive number of viewpoints may introduce redundant information, potentially disrupting the training process of the network. Furthermore, our experiments demonstrate that incorporating multiple viewpoint sequences, the model can capture a broader range of perspectives of the scene, thereby better reflecting the rating process of ODIs and achieving improved robustness.
4.6 Proximate to Human-Observed Performance
We conduct a comparative analysis between the performance achieved using ground-truth (GT) sequences and pseudo sequences generated by RPS on the JUFE dataset [12]. In the JUFE dataset, the GT sequences are annotated based on whether they originate from good or bad starting points (details in Supplementary Materials). Therefore, for each starting point, we use RPS to generate those sequences with the same number and length of sequences compared to GT sequences. Then, we apply the generated sequences and GT sequences as the input of our proposed network. The results shown in Table 9 highlight a close gap between the contributions of GT sequences and our generated sequences. This result emphasizes the significance of proximity to human observation in enhancing the model's capabilities. Meanwhile, there is still a large exploration space for future methods to better incorporate human's sequences in OIAQA task.
This paper introduces a novel multi-sequence network named Assessor360 for BOIQA based on a realistic assessment procedure. Specifically, we design Recursive Probability Sampling (RPS) to generate viewpoint sequences based on the semantic scene and the distortion. Additionally, we propose Multi-scale Feature Aggregation (MFA) with Distortion-aware Block (DAB) to combine distorted and semantic features of viewpoints. Temporal Modeling Module (TMM) is introduced to learn the temporal transition of viewpoints. We demonstrate the high performance of Assessor360 on multiple OIQA datasets and validate the effectiveness of RPS by comparing it with two advanced learning-based models used for scanpath prediction. Limitation is that the transition direction and distance in RPS are fixed, resulting in equally spaced distances between viewpoints. However, we have confidence that our analyses and the proposed pipeline can provide long-term valuable insights for future OIQA task.
Acknowledgments. This work was partly supported by the National Natural Science Foundation of China (Grant No. 61991451) and the Shenzhen Science and Technology Program (JSGG20220831093004008). The author would like to thank Xiangjie Sui at Jiangxi University of Finance and Economics for his inspiration.
[1] Marc Assens Reina, Xavier Giro-i Nieto, Kevin McGuinness, and Noel E O'Connor. Saltinet: Scan-path prediction on 360 degree images using saliency volumes. In Proceedings of the IEEE International
[56] Weixia Zhang, Kede Ma, Jia Yan, Dexiang Deng, and Zhou Wang. Blind image quality assessment using a deep bilinear convolutional neural network. IEEE Transactions on Circuits and Systems for Video Technology, 30(1):36-47, 2018.
[57] Weixia Zhang, Guangtao Zhai, Ying Wei, Xiaokang Yang, and Kede Ma. Blind image quality assessment via vision-language correspondence: A multitask learning perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14071-14081, 2023.
[58] Xuelei Zheng, Gangyi Jiang, Mei Yu, and Hao Jiang. Segmented spherical projection-based blind omnidirectional image quality assessment. IEEE Access, 8:31647-31659, 2020.
[59] Wei Zhou, Jiahua Xu, Qiuping Jiang, and Zhibo Chen. No-reference quality assessment for 360-degree images by analysis of multifrequency information and local-global naturalness. IEEE Transactions on Circuits and Systems for Video Technology, 32(4):1778-1791, 2021.
[60] Yu Zhou, Yanjing Sun, Leida Li, Ke Gu, and Yuming Fang. Omnidirectional image quality assessment by distortion discrimination assisted multi-stream network. IEEE Transactions on Circuits and Systems for Video Technology, 32(4):1767-1777, 2021.
[61] Yufeng Zhou, Mei Yu, Hualin Ma, Hua Shao, and Gangyi Jiang. Weighted-to-spherically-uniform sSIM objective quality evaluation for panoramic video. In IEEE International Conference on Signal Processing, pages 54-57, 2018.