Assessor360: Multi-sequence Network for Blind Omnidirectional Image Quality Assessment
Authors: Tianhe Wu
Shuwei Shi
Haoming Cai
Mingdeng Cao
Jing Xiao
Yinqiang Zheng
Yujiu Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: Blind omnidirectional image quality assessment
Multi-sequence network
Viewport sequence

Abstract: Blind Omnidirectional Image Quality Assessment (BOIQA) aims to objectively assess the human perceptual quality of omnidirectional images (ODIs) without relying on pristine-quality image information. It is becoming more significant with the increasing advancement of virtual reality (VR) technology. However, the quality assessment of ODIs is severely hampered by the fact that the existing BOIQA pipeline lacks the modeling of the observer's browsing process. To tackle this issue, we propose a novel multi-sequence network for BOIQA called Assessor360, which is derived from the realistic multi-assessor ODI quality assessment procedure. Specifically, we propose a generalized Recursive Probability Sampling (RPS) method for the BOIQA task, combining content and details information to generate multiple pseudo viewport sequences from a given starting point. Additionally, we design a Multi-scale Feature Aggregation (MFA) module with a Distortion-aware Block (DAB) to fuse distorted and semantic features of each viewport. We also devise Temporal Modeling Module (TMM) to learn the viewport transition in the temporal domain. Extensive experimental results demonstrate that Assessor360 outperforms state-of-the-art methods on multiple OIQA datasets. The code and models are available at https://github.com/TianheWu/Assessor360.

Assessor360: Multi-sequence Network for Blind Omnidirectional Image Quality Assessment
Tianhe Wu' Shuwei Shi Haoming Cai Mingdeng Cao?,
Jing Xiao", Yingiang Zheng', Yujiu Yang't
Shenzhen International Graduate School, Tsinghua University
The University Tokyo University of Maryland, College Park Pingan Group
{wth22, ssw20) (ar2000mmall...mmmmmm tsinghua. edu. en, cmd@g ecc u-tokyo ac. jp hmcai lauum.edd,, ziaojing66 @pingan com. en, yqzheng@ai, u totkkoo.eee ac sang-yujiu@ss tsinghua... edu.cn
Abstract
Blind Omnidirectional Image Quality Assessment (BOIQA) aims objectively assess the human perceptual quality of omnidirectional images (ODIs) without relying pristine-quality image information. is becoming more significant with the increasing advancement of virtual reality (VR) technology. However, the quality assessment of ODIs is severely hampered by the fact that the existing BOIQA pipeline lacks the modeling of the observer's 's browsing process. To tackle this issue, propose novel multi multi-sequence network for BOIQA called Assessor360, which is derived from the realistic multi-assessor ODI quality assessment procedure. Specifically, propose a generalized Recursive Probability Sampling (RPS) method for the BOIQA task, combining content and details information to generate multiple pseudo viewport sequences from given starting point. Additionally, we design a Multi-scale Feature Aggregation (MFA) module with Distortion-awarr aware Block (DAB) to fuse distorted and semantic features of each viewport. We also devise Temporal Modeling Module (TMM) to learn the viewport transition in the temporal domain. Extensive experimental results demonstrate that Assessor360 outperforms sfaee-of-the-ar methods on multiple OIQA datasets. The code and models are available https://github..om com/Tianhe Wu/Assessor366.
Introduction
With the development of VR-related techniques, viewers can enjoy a realistic and immersive ex- perience with head-mounted displays (HMDs) [3, 20] by perceiving 360-degree omnidirectional information. However, the acquired omnidirectional image, also named panorama, is not always of
high quality [12, 60]. Degradation may be introduced in any image processing [44, 58, 42], leading to low-quality content that may be visually unpleasant for users. Consequently, developing suitable quality metrics for panoramas holds considerable importance, as they can utilized to guide research in omnidirectional image processing and maintain high-quality visual content.
Generally, most ODIs stored in the equirectangular projection (ERP) format, which exhibits considerable geometric deformation at different latitudes. This distortion can have a negative impact on quality assessment. Therefore, as shown Figure (a), many researchers [36, 17, 43, 58, 14, 60] explore viewport-based methods by projecting the original ERP image into many undeformed viewports (actual content observed by users) and aggregate their features with 2D IQA method the ODI quality score. However, this conventional viewport-based pipeline lacks the modeling of the
"Tianhe Wu and Shuwei Shi contribute equally to this work.
t Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Conerrbing diffenet peopaciio versoots
Virgger quilty Iegrosnaaioon
Soor pradicion
Pesture
Rectilnar
Quality Represstation
Project
20 I0A Methoö
Pesture
Approgats
Quality scorre
Peetuno
Oestdinntionl Imoge
Diffeeser. Pergective Vieerports
Ordinasy Viewport based Fipeline
Olserving srdl peoseving the quality of condinctionll imrge
Recaling the quility of aleerced compparts
Quality scare reting
Matttpple
Scanpoothe
Scaepath
Score 1
Obssve
Rocill quality
Average
Quality
spor
Scmpath n
&
Multtale Subjects
(evaluatos)
Orandinctiod Image
(b) Reslistic ODI Quality Procedure
Illustraiion of the existing ordinary viewport-based pipeline (a) and the realistic omnidirec- tional image quality assessment procedure (b). Plenty of existing viewport-based methods follow pipeline (a) which is inconsistent with authentic assessment procedure (b), causing the predicted quality score to be far from the human perceptual quality score.
Figure I:
observer's browsing process, causing the predicted quality to be far from the human perceptual quality, especially when the ODI containing non-uniform distortion (such as stitching) [33, 12, 36, 61, 37]. In fact, during the process of observing ODIs, viewports are sequentially presented to viewers based on their browsing paths, forming viewport sequence.
While the viewport sampling techniques proposed by Sui [l.[33] and Zhang et (53) can generate viewports with a fixed sequential order for ODIs, their methods lack the ability to generate versatile sequences. Specifically, their methods will produce identical sequences for a given starting point, and the sequential order of viewports will remain constant across different ODIs. This behavior is inconsistent with the observations of multiple evaluators in realistic scenarios, leading to an inability to provide subjectively consistent evaluation results. To address this issue, logical approach is to employ the scanpath prediction model for generating pseudo viewport sequences on the ODI without authentic scanpath. However, current scanpath prediction methods [47, 24, 32, 1, 34] developed for undistorted ODIs and mainly focus on high-level regions. As evaluators are required to provide accurate quality score for ODI, their scanpaths are distributed across both low-quality and high-detail regions. This makes it potentially unsuitable for direct application in BOIQA task where all ODIs are distorted.
To move beyond these limitations, inspired by the realistic multi- multi-assessor ODI quality assessment procedure, shown in Figure 1 (b), we first propose multi-sequence network called Assessor360 for BOIQA, which can simulate the authentic data scoring process to generate multiple viewport sequences (corresponding to multiple scanpaths). Specifically, we propose Recursive Probability Sampling (RPS) to generate multiple pseudo viewport sequences, combining semantic scene and local distortion characteristics. In particular, based Equator-guided guided Sampled Probability (ESP) and petails-guidee Sampled Probability (DSP), RPS will generate different viewport sequences for the same starting point (details in Section 3.2). Furthermore, we develop Multi-scale Feature Aggregation (MFA) with Distortion-aware Block (DAB) to effectively fuse viewport semantic and distorted features for accurate quality perception. The Temporal Modeling Module (TMM) is devised by applying GRU [5] module and MLP layers to learn the viewports temporal transition information in a sequence and regress the aggregated features the final score. Extensive experiments demonstrate the superiority and effectiveness of proposed Assessor360 on multiple OIQA datasets (MVAQD [18], OIQA [11], CVIQD [35], IQA-ODI [46], JUFE [12), JXUFE [33]). We summarize our contributions into four points:
We propose Assessor360 for BOIQA, which can leverage multiple different viewport sequences to assess the ODI quality. To our knowledge, Assessor360 is the first pipeline to simulate the authentic data scoring process in ODI quality assessment.
We propose an unlearnable method, Recursive Probability Sampling (RPS) that can combine semantic scene and local distortion characteristies to generate different viewport sequences for given starting point.

---

We design Multi-scale Feature Aggregation (MFA) and Distortion Block (DAB) to characterize the integrated features of viewports and devise a Temporal Modeling Module (TMM) to model the temporal correlation between viewports.
We apply our Assessor360 to two types of OIQA task datasets: one with real observed scanpath data and the other without. Extensive experiments show that our Assessor360 largely outperforms state-of-the-ar methods.
2 Related Work
Omnidirectional Image Quality Assessment. Similarly to traditional 2D IQA, according to the reference information availability, OIQA can also be divided into three categories: full-reference (FR), reduced-rrfferecce (RR), and no-reference (NR) OIQA, also known as blind OIQA [2]. Due to the structural characteristics of the panorama and the complicated assessment process, OIQA has not matured as much as 2D-IQA [48, 22, 13, 19, 15, 29J. Some researchers extend the 2D image quality assessment metries to the panorama, such S-PSNR [51]. CPP-PSNR [52], and WS-PSNR [37]. However, these methods are not consistent with the Human Visual System (HVS) and they are poorly consistent with perceived quality [60, 43]. Although WS-SSIM [61] and S-SSIM [4] consider some impacts of HVS, the availability of non- non-distortion reference ODIs severely hinders their applications in authentic scenarios.
Therefore, some deep learning- based BOIQA methods [60, 43, 36, 16, 53, 46) are devised to achieve better capabilities. Due the geometric deformation present in ODIs in ERP format, many existing BOIQA methods [59, 46, 60, 17] follow a similar pipeline: sampling viewports in a particular way and simply regressing their features the quality score. MC360IQA [36] first maps the sphere into a cubemap, then employs CNN to aggregate each cuberna plane feature and regress them to a score. ST360IQ [16] samples tangent viewports from the salient parts and uses VIT [10] to estimate the quality of each viewport. VGCN [43] migrates the graph convolution network and pre-trained DBCNN [56] to establish connections between different viewports. However, these methods ignore the vital effect of the viewport sequences generated in multiple observers' browsing process, which have been demonstrated by Sui el al.[33) and Fang el al.[ on the perception of the ODI quality.
Viewport Sampling Strategies in OIQA. Existing viewport sampling strategies can be categorized into three modes: 1) Uniformly sampling without sequential order. Zhou et al.[59] and Fang er al.[12] uniformly extract viewports over the sphere. Jiang et [l.[17] and Sun et [3]] use cube map projection (CMP) and rotate the longitude to obtain several viewport groups. This sampling pattern will cover the full areas, whether they are significant non-significant. 2) Crucial region sampling without sequential order. Xu el al.[43] leverage 2D Gaussian Filter [45] to acquire a heatmap and generate viewports with corresponding locations. Tofighi et [16[ apply ATSal [6] predict salient regions of the panorama, which gives help to sampling viewports. This mode incorporates HVS, where most of the sampled viewport might be observed in the browsing process. Nevertheless, both of the above methods only focus on the image content and do not concern the effect of the sequential order between viewports in the quality assessment process of panoramas. 3) Sampling viewports along the fixed direction. Sui et [33] first considers the effect of sequential order between viewports in the browsing process. They default observers rotate their perspective in a specific direction along the equator. Zhang et [53] further introduce ORB detection to capture key viewports and follow Sui et al. [33] default sampling direction extract viewports. In fact, different evaluators will produce various scanpaths when viewing a panorama. Even if the same evaluator observes the same ODI twice, the scanpath will be different. However, their sampling methods cause the sequential order of the viewport to be fixed, and even for the different image contents, the sequential order of sampled viewports is the same. This creates a significant gap with people's s actual browsing process, leading to unreasonable modeling of the viewport sequence.
3 Method
3.1
Overall Framework
As shown in Figure 2 (a), the proposed Assessor360 framework consists of Recursive Probability Sam- pling (RPS) scheme, Multi-scale Feature Aggregation (MFA) strategy, and Temporal Modeling Mod- ule (TMM). Given a degraded ODI I, to be consistent with the authentic multi-assessor assessment, we initialize N starting points X (w,,,, ) where u, E [-90", ,90% and v; E [-180", 180%] are the corresponding latitude and longitude coordinates. Then, we apply the proposed RPS strategy G with parameters O, to generate multiple viewport sequences S = {(V)
>==)i=1, where V denotes (()))))
M 1N
3

---

Cerending wwwraccieappt sxparos RPS nebsdd
NGLL Xod Eatza Agrraizo MIAI
Terpori Yooking Modiin MndeiiTMM:
Input ODE
Stage
Scoce
OnT sceen
take
Stage
DAB
D Distortion- Distortion-wwvare Block (DAB)
Becuniiee Procability
Sampling
Stage
Conor
Stage
Charse Atveter
Strting points X
Archttecture of proposed A ssessor360
Tetsill-guiiee Sumpled Probability (DSP)
Calmilaa
Select Nomi Vievpst
Spilt
Salrage
Vi
Veta
patzbes
oveeligeed neghbar
I
Equator guited Sampled Prebabiiiy (ESP)
Sumoo
Chicslating it neightr pelches courtinoss
chave prishitinn
of dr) oo daae)) (C
Recusive Probability Sumpling (RPS)
116 -a of -a
Coardincto grobobiilty mop
V--1
Figure 2: Architecture of proposed Assessor360 for BOIQA (a). Our Assessor300 consists of Recursive Probability Sampling (RPS) scheme (b), Multi-scale Feature Aggregation (MFA) strategy, and Temporal Modeling Module (TMM).
each viewport, M denotes the length of each sequence. Next, to perceive the semantic scene and distortion in each viewport, multi-scale features are extracted from multi-stage layers of our method for quality assessment. We use MFA FF with parameters O, to represent this process. The features aggregated from Hx RLx1xD In this case, C' is the viewport's original dimensions, H and W are the height and width of the viewport, and D denotes the embedding dimension. Then, we use TMM H with parameters θ) model each sequence viewport temporal transition information and predict the final viewport sequence quality score. Finally, we average all predicted scores of cach sequence as the ODI's quality 2z. Overall, the whole process can be described follows:
1 IN (L
Q = N
3.2 Recursive Probability Sampling
Recursive Probability Sampling (RPS) strategy adaptively generate the probability of scene transition direction based on prior knowledge of semantic context and degraded features in ODL mainly consists of Equator-guided Sampled Probability (ESP) and Details-guided Sampled Probability (DSP). The viewport sequence is generated by selecting a certain starting point and sampling viewports based on generated probabilities.
Preprocessing for generating probabilities. As illustrated in [47], transition direction and distance are two important factors for locating the next viewport position. We first follow the theory of Moore neighborhood [28] and define K neighbor (K 8) transition directions from the center coordinate of the viewport. Following [31, 33], the transition distance (Au, Ae) set to (24", 24%), avoiding sampling overly overlapped viewports. In details, given a current position x (u', v.), we first generate the corresponding viewport V; from by rectilinear projection [49] R. Then, we uniformly split V. into K + overlapped patches, including K neighbor patches pt = (P!)
K j== and one central
2 patch Pt with height H and width W As shown in Figure 2 (b), the K direction sampled coordinates
X Resihbor { can be calculated by the central patch Pf coordinate x². and (Au, Av). During the browsing process, assessors are not only drawn to the high-level scenario but also focus on low-level texture and details regions [35, 11] to give a reasonable quality score. Therefore, according to the generalized prior content information and pixel-level details metric, we present ESP and DSP for each direction sampled coordinate in x neighbor Then, we choose the next sampled viewport V++1
position z based on them.
Equator-guided Sampled Probability (ESP). Since the equator entails more scene information (1]. we introduce the prior equator bias [9] M to constrain the sampled viewport near the equator. Concretely, the prior equator bias obeys a Gaussian distribution with a mean of and standard deviation of 0.2 in latitude. Regions near the equator have higher sampled weights and regions close

---

Algorithm Viewport Sequence Generation (RPS Algorithm)
Input: N starting points {a; | an ODI I; rectilinear projection R: ESP calculation function (e'; DSP calculation function Fasp selecting function [()())
Output: A set of N length M viewport sequences {st (V))) 1
1: for = 1 - N do
2: Initialize the current coordinate x: 2;
3: for 1
do
4:
Generate viewport by the current coordinate V x- R(x,I)
5: Split V to obtain overlapped neighbor patches P and calculate sampled coordinate X'
6: Calculate ESP and DSP Peop f- Fesp(A), (x), Pdsp + Fds
7:
Pasp)) 8: Sequentially gather generated M viewports {V; Ar viewport sequence 8;
Generate next viewport coordinate x +- [22'AAgregalee
9: Gather generated N viewport sequences {s as the output
to the poles have relatively low sampled weights. We apply the softmax function to M to obtain the coordinate probability map Mp where each coordinate corresponds to sampled probability. Then we take X neighbor sampled probabilities {p') from Mp. Meanwhile, due to the inhibition of return (IOR) [27] where regions that have been fixated by the eyes have lower probability of being fixated again in the near future, we multiply 2 with a decreasing factor 7 where k is the index of the viewport coordinate has been generated. Finally, we apply softmax function to calculate ).
Pesp ( Overall, the ESP calculation function Fes)(At) can be defined as:
Fe) = Softmax [Z
(1,....7...... 1}K
(2)
Details-guided Sampled Probability (DSP). To efficiently measure the texture complexity of each patch, we use pixel-level information entropy E. The mathematical expression is:
H' W'
E(P!) = ΣΣ
((P)=>>
pPP)(m, log, p (P,
m== ==1
where [mm, ra] is the gray value of each pixel at position (m, n) in the patch Pt and p(-) is the probability of occurrence of each gray value. After calculating the entropy of each neighbor patch, we normalize it with the softmax function to obtain Pdsp = (P'spp j for X neighbor The DSP calculation function (P) can be formulated as:
Fasp (P) = Softmax(E(P'))
(4)
Generating viewport sequences. Ultimately, we multiply the ESP and DSP with a scale factor 3 to get the integrated probability. We use the softmax function to obtain the final probability distribution and select the next viewport position r!(1 according to which can be written as:
att1 (S(X"|/oftmax(pp
Peep B))
(5)
where [(X'lp) is the selecting function based on the set of probability p. We can recursively generate viewport Vi+1 with z++1 and keep performing the RPS strategy until the number of viewports in the sequence reaches the desired length. The whole generation algorithm G is shown in Algorithm 1.
3.3 Multi-scale Feature Aggregation
To represent the semantic information and distortion pattern of each viewport, which are assumed as two key factors of quality assessment, we aggregate multi-scale features of the viewport. We first extract features from each stage in pre-trained Swin Transformer [23, 30]. The outputs of the first two stages F1, F2 are more sensitive to the distortion pattern, while the outputs of the last two stages Fa. F4 tend to capture the abstract features (details Supplementary Materials). Before fusing multi-scale features, we first employ four 1 x 1 convolution layers to reduce the feature dimension of the output to D. Then, to further emphasize the local degradation, we devise and apply the Distortion-aware Block (DAB) which includes a 3 x 3 convolution layer following Channel Attention (CA) operations the reduced shallow features F,, F2. This operation can help the model to better perceive the distortion pattern in the channel dimension, achieving distortion-awart capacity. Finally, concatenate and integrate these features with Global Average Pooling (GAP) and multiple 1 x convolution layers to get the quality-related representation. After that, each aggregated feature F fuse will be sent to TMM for viewport sequence quality assessment.
5

---

Table 1: Quantitative comparison of the state-of-the-ar methods and proposed Assessor360. The best are shown in bold, and the second best (except ours) are underlined. Two baselines w/ ERP and w/ CMP mean that we replace input viewport sequences generated by RPS with ERP and CMP.
MVAQD
OIQA
IQA-ODI
CVIQD
Type
Method
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
PSNR
0.8150
0.7591
0.3929
0,3893
0.4018
0.4890
0.8015
0.8425
SSIM [39]
0.8272
0.7202
0.3402
0.2307
0.5014
0,5686
0.6737
0.7273
MS-SSIM [40]
0.8032
0.7136
0,5750
0,5084
0.7434
0,8389
0.9218
0.9272
FR-IQA
WS-PSNR [37]
0.8152
0.7638
0.3829
0.3678
0.3780
0.4708
0.8039
0.8410
methods
WS-SSIM [61]
0.8236
0.5328
0.6020
0.3537
0.5325
0.7098
0.8632
0.7672
VIF [54]
0.8687
0.8436
0,4284
0.4158
0.7109
0.7696
0,9502
0.9370
DISTS [8]
0.7911
0.7440
0.5740
0.5809
0.8513
0.8723
0.8771
0.8613
LPIPS [55]
0.8048
0.7336
0,5844
0.4292
0.7355
0.7411
0.8236
0.8242
NIQE [26]
0.6785
0.6880
0,8539
0,7850
0.6645
0,5637
0,9337
0,8392
BRISQUE [25]
0.8408
0.8345
0.8213
0.8206
0.8171
0,8651
0.8269
0.8199
PaQ-2-PiQ [50]
0.3251
0.3643
0,1667
0.2102
0.0201
0,0419
0.7376
0.6500
MANIQA [48]
0.5531
0,5718
0.4555
0.4171
0.2642
0.2776
0.6013
0.6142
MUSIQ [19]
0.5436
0.6117
0.3216
0,3087
0.0565
0.0983
0.3483
0.3678
CLIP-IQA [38]
0.5862
0.4941
0.2330
0,2531
0.0927
0.1929
0.4884
0.4347
LIQE [57]
0.6837
0.7539
0.7634
0.7419
0.8551
0,9020
0.8594
O. 0.8086
NR-IQA
SSP-BOIQA [58]
0.7838
0,8406
0.8650
0.8600
0.8614
0.9077
methods
MP-BOIQA [17]
0.8420
0,8543
0,9066
0.9206
0,9235
0.9390
MC3601QA [36]
0.6605
0.6977
0.9071
0.8925
0.8248
0,8629
0.8271
0.8240
SAP-net [46]
0,9036
0.9258
VGCN [43]
0.8422
0.9112
0,9515
0.9584
0.8117
0.8823
0,9639
0.9651
AHGCN [14]
0,9647
0,9682
0,9617
0.9658
baseline w/ ERP
0.9076
0,9240
0.8961
0.8857
0.9098
0.9196
0,9330
0.9485
baseline w/ CMP
0.8966
0.9324
0,9216
0.9170
0.9105
0.9122
0,9390
0.9412
Assessor360
0.9607
0.9720
0.9802
0.9747
0.9573
0.9626
0.9644
0.9769
3.4 Temporal Modeling Module
The browsing process of ODI naturally leads to the temporal correlation. The recency effect indicates that users are more likely to evaluate the overall image quality affected by the viewports they have recently viewed, especially during prolonged exploration periods. To model this relation, we introduce the GRU module to learn the viewport transition. Due to the fact that the last token encodes the most recent information and the representation at the last time step involves the whole temporal relationships of sequence, we use MLP layers regress the last feature output by the GRU module to the sequence quality score.
4 Experiments
4.1 Implementation Details
We set the field of view (FoV) to the 110º following [12, 33]. We use pre-trained Swin Transformer [23] (base version)
feature extraction backbone. The input viewport size H x W is fixed to
224 x 224. The number of viewport sequences N is set to 3 and the length of each sequence M is set to 5. We set the coordinates of N starting points be (0P, 09). The reduced dimension D is 128 and the number of GRU modules is set to 6. The number of CA operations n is 4. We set γ = 0.7 and B = 100 as decreasing factor and scale factor values respectively.
For fair comparison, we randomly split 80% ODIs of each dataset for training, and the remaining 20% is used for testing following [53, 18, 12, 17]. To eliminate bias, we run a random train-test splitting process ten times and show the median result. We train 300 epochs with batch size on CVIQD [35], OIQA [11], IQA-ODI [46], and MVAQD [18] datasets without the authentic scanpath data. Respectively, we compare our RPS with two advanced learning- based scanpath prediction methods ScanGAN360 [24] and ScanDMM [32] on JUFE [12] and JXUFE [33] datasets which have the authentic scanpath data. For optimization, we use Adam [21] and the learning rate is set to 1 x 10-5 in the training phase. We employ MSE loss to train model. We use Spearman orde-ed correlation (SRCC) and Pearson linear correlation (PLCC) as the evaluation metrics.
4.2 Comparing with the State-of-the-art Methods
We conduct a comparative analysis of Assessor360 with eight FR methods and thirteen NR methods. The quantitative comparison results are presented in Table 1, demonstrating significant performance

---

Table 2: Cross-dataset validation SRCC and PLCC results of SOTA methods. These models (except WS-PSNNR [37] and WS-SSIM [61]) are trained on CVIQD [35], OIQA [11] and MVAQD [18] datasets (80% set) and tested on three other datasets (full set).
CVIQD
OIQA IQA-ODI
MVAQD
Method
OIQA
IQA-ODI
MVAQD
CVIQD
MVAQD
CVIQD
OIQA
IQA-ODI
SRCC
WS-PSNR
0.5027
0,4360
0.7225
0.7638
0.4360
0.7225
0,7638
0.5027
0,4360
WS-SSIM
0.5442
0,5032
0.7930
0.6625
0.5032
0.7930
0.6625
0.5442
0.5032
MC360IQA
0.4189
0.7114
0.0296
0.7044
0.5687
0,4081
0.0373
0.0025
0,0486
VGCN
0.2361
0,2875
0.2452
0.6932
0.3873
0.4682
0.4650
0.6227
0.3921
Assessor360
0.4597
0.8610
0.5640
0,8430
0.8751
0.6417
0.8756
0.7765
0.8646
PLCC
WS-PSNR
0.4701
0.5468
0.6962
0.7895
0.5468
0.6962
0.7895
0.4701
0.5468
WS-SSIM
0.4363
0.5941
0.6246
0.6536
0.5941
0.6246
0.6536
0,4363
0.5941
MC360IQA
0.4295
0.7872
0.0404
0.7368
0.5930
0.4238
0.0430
0.0202
0.0646
VGCN
0.2582
0.3127
0.2467
0.5929
0.3551
0.2419
0.3420
0,4642
0.3870
Assessor360
0.5332
0.9032
0.5824
0.8636
0.9137
0.6565
0.7232
0.7287
0.8541
Table 3: Cross-dataset validation SRCC results Table 4: Quantitative comparison of using differ- of SOTA methods. These models are trained ent viewport sequence generation methods OIQA MVAQD [18] (full set) and tested on CVIQD [11] and MVAQD [18].
[35] and OIQA [11] all distortion data except MP-BOIQA (removing AVC on CVIQD).
OIQA
MVAQD
Generation Method
SRCC
PLCCC
SRCC
PLCCC
Random Generation
0.9461
0.9444
0,9359
0.9543
Testing
MC3601QA
MP-BOIQA
Assessor60
ScanGAN360
0.9705
0.9670
0,9493
0.9694
OIQA
0,2542
0.5043
0.6658
ScanDMM
0.9652
0.9634
0.9558
0.9612
CVIQD
0.4749
0.7992
0.8994
RPS (Ours)
0,9802
0.9747
0,9607
0.9720
improvements across all datasets against the state-of-the-ar methods. Note that these performance enhancements obtained without training with the 2D IQA dataset, employed in VGCN [43]. Specifically, Assessor360 outperforms VGCN by 3% in terms of SRCC on the OIQA dataset and shows 1.2% increase in PLCC on the CVIQD dataset. Additionally, method exhibits notable improvement of up to 1.6% in SRCC compared to AHGCN [14].
4.3 Cross-Datasel Evaluation
To assess the generalization capability of our method, we perform cross- volsdvvlliotion the CVIQD [35] and OIQA [11 1] datasets and compare it against two widely used state-of-the-art methods: MC360IQA [36] and MP-BOIQA [17]. All models are trained on the MVAQD [18] dataset and subsequently tested on the CVIQD and OIQA datasets. The results, presented in Table 3, demonstrate that our method exhibits superior generalization performance compared to the other models. It is worthy to note that, there exists a significant domain gap between the CVIQD and MVAQD datasets. The CVIQD dataset primarily focuses on degradations such as JPEG, H.264/AVC, and H.265/HEVC. while the MVAQD dataset concentrates on JPEG, JP2K, HEVC, WN, and GB. This discrepancy in focus poses a challenge for MC360IQA and MP-BOIQA, as they use the same perspective viewports to evaluate different distorted ODIs, leading to poor generalization. In contrast, our proposed RPS overcomes this domain gap effectively by generating different viewport sequences for ODIs with different distortions. It consistently performs well, showcasing its effectiveness in addressing the challenges posed by diversely distorted ODIs.
Furthermore, to verify that the performance presented in Table is not due to the overfitting, we retrain MC360IQA, VGCN [43] and Assessor360 on CVIQD, OIQA and MVAQD datasets, using 80% of the dataset for training and the remaining 20% for testing. We then select the model weights with the highest performance on the testset for cross-dataset testing for fair generalization comparisons. The results shown in the Table 2 clearly demonstrate the strong generalization of our method compared to VGCN and MC360IQA. Notably, during instances of training on the MVAQD dataset and subsequent testing on the IQA-ODI [46] dataset, our approach outperforms MC360IQA in terms of SRCC by an approximate margin of 0.8, and surpasses VGCN SRCC by 0.5. Concurrently, when training on the CVIQD dataset, characterized by fewer distortion types, and testing on the MVAQD dataset, which encompasses more distortion types (JP2K, WN, GB), our approach attains elevated PL.CC values of 0.5 and 0.3, respectively, as compared to MC3601QA and VGCN.
7

---

Table 5: Quantitative comparison of different generation methods (RPS vs ScanGAN360 [24] and ScanDMM [32]) with metrics of scanpath prediction task on JUFE [12] and JXUFE [33] datasets with authentic scanpaths.
JUFE [12]
JXUFE [33]
Published Time
Generation Method
LEV
DTW4
REC1
LEV
DTW,
RECT
Random Baseline (lower hound)
35.21
1707.45
0.38
35.08
1695.93
0.38
TVCG22
ScanGANN60 [24]
32.53
1448.65
1.07
31.89
1427.55
1.14
CVPR23
ScanDMM [32]
31.23
1434.36
1.21
31.48
1438.29
1.12
RPS w/o DSP (Ours)
29.54
1471.82
2.14
29.99
1463.38
1.94
RPS (Ours)
29.48
1454.03
2.21
29.66
1422.85
2.07
Human Baseline (upper bound)
23.85
1309.29
3.78
26.73
1302.15
2.88
Gaesiia Nur
ScandAN36e
ScanOMM
16
16
Stithing
ScanGAR116
ScanDMM
RPS (Ours)
CT
Brightess discontiniity
ScanGAM360
ScanDWN
RPS (Ours)
Gaussian noise
ScanGAN360
ScanDMM
(Ours)
10
Figure 3: Visual comparison of the generated viewport positions for different methods on ODIs with four distortion types in JUFE. The brighter the area, the more viewports generated in that area.
4.4 Effectiveness of Recursive Probability Sampling
As mentioned in Section there exist many learning-based scanpath prediction methods [24, 47, 32]. They seem to be able to assist in constructing viewport sequences. However, they are hardly introduced OIQA lask. this section, we first perform the quantitative comparison the model with RPS and two advanced 360-degree scanpath prediction methods, namely ScanGAN360 [24] and ScanDMM [32] on the datasets without real observed scanpath. Subsequently, we compare the position and sequential order of viewports generated by the three methods with the ground-truth (GT) scanpaths by the metries of scanpath prediction task and visualization to further validate superiority of RPS on JUFE [12] and JXUFE [33] datasets with real observed scanpath.
Quantitative comparison of performance. We replace the proposed RPS method with Scan- GAN360 and ScanDMM to generate sequences viewports. The model was trained and tested using these viewport sequences OIQA [11] and MVAQD [18] datasets, maintaining the same length and number of viewports for a fair comparison. Table shows the guantitative results, demonstrating that viewports generated from RPS yield superior performance compared ScanGAN360 and ScanDMM. Additionally, training using viewports generated from these methods outperforms those generated using random schemes, highlighting the crucial role of suitable viewport sequences in the OIQA task.
Moreover, we conduct experiments by replacing original VGCN [43] sampling methods with RPS in VGCN. We use RPS to sample the same number of viewports as [43] to train VGCN on IQA-ODI [46] and MVAQD datasets. The results presented in Table 7 demonstrate a substantial performance enhancement for VGCN achieved by the viewport sampled through RPS, resulting an increase of 0.07 in SRCC for MVAQD. Additionally, this indicates that the viewport sampled with RPS closely aligns with human observations.

---

Table 6: Quantitative comparison of different start- ing point positions on MVAQD [18] dataset.
Table 7: Quantitative comparison of using origi- nal VGCN sampling method and proposed RPS on IQA-ODI [46] and MVAQD [18] datasets.
Position (latitude, longitude)
SRCC
PLCC
IQA-ODI
MVAQD
(0,, 0%), (0%, 0%), (0")0")
0.9607
0,9720
Method
(60". 0"), 160", 0%), (60"
0.9106
0.9312
SRCC
PLCC
SRCC
PLCC
(0", 120"), (0" "0") (0" )0")
60%)
0.9599
0,9660
VGCN
0.8117
0.8823
0.8422
0.9112
(60", 120"), (60", 0"), (60', 600)
0.9174
0,9455
VGCN-RPS
0.8382
0.8883
0.9122
0.9273
Table 8: Ablation studies of each component in proposed Assessor360 on MVAQD [18] dataset.
Table 9: Quantitative comparison of using GT sequences and sequences generated by RPS on JUFE [12] dataset. Starting Point (SP).
Method
Para (M)
SRCC
PLCC
Asscssor360 w/o MFA
88.53
0,8514
0.9171
Viewport
Good SP
Bad SP
Assessor360 w/o DAB
88.86
0,8437
0,8779
Sequence
SRCC
PLCC
SRCC
PLCC
Assessor360 w/ GAP
88.69
0,9393
0.9587
RPS (Ours)
0.6623
0.6365
0.5044
0.4946
Assessor360 w/ GRU
89.28
0.9607
0.9720
GT Sequence
0.7158
0.7013
0,5400
0.5377
Comparison of viewport sampling positions and order. Since each ODI in JUFE and JXUFE comprises 30 and 22 scanpaths, respectively, we employ the three methods to generate 30 and 22 sequences, each comprising 20 viewports, for each ODI in JUFE and JXUFE, respectively. For each GT sequence, we sample 20 viewports from 300 viewports with equal time intervals. We adopt Levenshtein distance (LEV) and dynamic time warping (DTW) metrics, as used in ScanGAN360 and ScanDMM, evaluate the position and sequential order of the viewports. Furthermore, the viewing behaviors and patterns can be evaluated using the recurrence measure (REC) [24]. Each metric is calculated by comparing each generated sequence against all the GT sequences and computing the average, resulting in the final value. Besides, to establish an upper bound for each metric, we compute the Human Baseline by averaging the results of comparing each GT sequence against all other GT sequences [41]. Similarly, we establish a lower bound by randomly sampling viewports from the ODI, referred to as the Random Baseline.
The quantitative comparison results are reported in Table 5. The proposed unlearnable method outperforms learning-baac models regarding the LEV and REC metrics. Specifically, achieves LEV improvement of 1.75 and 1.82 compared to ScanDMM [32], and a REC improvement of 1.14 and 0.93 compared to CA360 [24]. Additionally, our method achieves comparable performance to ScanDMM and ScanGAN360 on the DTW metric for both datasets. Figure 3 presents the positions of generated viewports for different generation methods. The panorama is divided into 72 regions, with each region spanning 15 degrees of latitude and 30 degrees of longitude. It can be observed that the positions generated by the learning- based method are relatively dispersed. while the positions generated by RPS are concentrated near the equator. The concentration gradually decreases with increasing latitude, aligning closely with the ground truth (GT) position.
4.5 Ablation Studies
Impact of the initialization of the starting point. For the ODI without real scanpath data, initialize the coordinates of N starting points to be (0', 09) followed by [33]. We conduct exper- iments assess the impact of different initial starting points positions on the final performance. Results shown in Table 6 revels that the model's performance displays relatively minor fluctuations during longitudinal shifts, yet experiences substantial decline when changing larger latitudes (from points less frequented by humans as starting positions). This divergence arises due to the model's s misalignment with authentic browsing behaviors (humans tend to start their panoramic observations closer to the equator). It further emphasizes that simulating actual scanpaths assists in more accurate image quality evaluation.
Effectiveness of MFA and TMM. We exclude the multi-scale feature aggregation (MFA) and distortion-aware Block (DAB) to assess their effectiveness in MVAQD [18]. Specifically, for the previous experiment, we utilize the final stage of the Swin Transformer [23] for feature output. The results, shown in Table 8, indicate that MFA and DAB substantially contribute to the performance, with only a minor increase in the network parameters. The GRU [5] module in TMM is replaced with global average pooling (GAP) for predicting the sequence quality score. The effectiveness of the GRU module in representing temporal transition information is demonstrated in Table 8, and it can assist the model in predicting more accurate quality scores.

---

Excessive viewports hamper performance
Excessive view bemper perfommnce
0.96
0.95
SRCC
0.93
0.92
0.91
0.90
Viewport Seqaence
0.92
Viewport Sequencc
Viewpooit Seqpessoss
Viesgoet Sequences
0.59
Viewport Sequenecs
0.91
Viewport Sequeiies
10
12
14
2
10
12
The Length of Sequence
The Length of Sequence
Figure 4: Performance of different number and length of viewport sequence on MVAQD [18].
Impact of the number and length of the viewport sequence. We test N 1, three different numbers of viewport sequences with varying sequence lengths on MVAQD [18] dataset. The findings, shown in Figure 4, reveal that as the sequence length increases, there is decreasing trend in model performance across all three numbers of sequences. This suggests that an excessive number of viewports may introduce redundant information, potentially disrupting the training process of the network. Furthermore, our experiments demonstrate that incorporating multiple viewport sequences, the model can capture broader range of perspectives of the scene, thereby better reflecting the rating process of ODIs and achieving improved robustness.
4.6 Proximate to Human-Observed Performance
We conduct a comparative analysis between the performance achieved using ground-truth (GT) sequences and pseudo sequences generated by RPS on the JUFE dataset [12]. In the JUFE dataset, the GT sequences are annotated based on whether they originate from good or bad starting points (details Supplementary Materials). Therefore, for each starting point, we use RPS to generate those sequences with the same number and length of sequences compared to GT sequences. Then, we apply the generated sequences and GT sequences as the input of our proposed network. The results shown in Table 9 highlight close gap between the contributions of GT sequences and our generated sequences. This result emphasizes the significance of proximity to human observation in enhancing the model's capabilities. Meanwhile, there is still large exploration space for future methods to better incorporate human's 's sequences in OIQA task.
5 Conclusion
This paper introduces novel multi-sequence network named Assessor360 for BOIQA based on a realistic assessment procedure. Specifically, we design Recursive Probability Sampling (RPS) to generate viewport sequences based on the semantic scene and the distortion. Additionally, propose Multi-scale Feature Aggregation (MFA) with Distortion-awarr Block (DAB) to combine distorted and semantic features of viewports. Temporal Modeling Module (TMM) is introduced to learn the temporal transition of viewports. We demonstrate the high performance of Assessor360 on multiple OIQA datasets and validate the effectiveness of RPS by comparing it with two advanced learning-bast models used for scanpath prediction. Limitation that the transition direction and distance in RPS are fixed, resulting in equally spaced distances between viewports. However, have confidence that our analyses and the proposed pipeline can provide long-term valuable insights for future OIQA task.
Acknowledgments. This work was partly supported by the National Natural Science Foun- dation of China (Grant No. 61991451) and the Shenzhen Science and Technology Program (ISGG2022083 1093004008). The author would like to thank Xiangjie Sui at Jiangxi University of Finance and Economics for his inspiration.
References
[1] Mare Assens Reina, Xavier Giro-i Nieto, Kevin McGuinness, and Noel E CoCnnor Saltinet: Scan-path prediction on 360 degree images using saliency volumes. In Proceedings of the IEEE International
10

---

Conference on Computer Vision Workshops, pages 2331-2335, 2017.
[2] Sehastian Bosse, Dominique Maniry, Klaus-Robert Müller, Thomas Wiegand, and Wojciech Samek. Deep neural networks for no-reference and full-reference image quality assessment. IEEE Transactions on Image Processing, 27(1):206-219, 2017.
[3] Meixu Chen, Yize Jim, Todd Goodall, Xiangxu Yu, and Alan Conrad Bovik. Study of 3d virtual reality picture quality. IEEE Journal of Selected Topics in Signal Processing, 14(1):89-102, 2019.
[4] Sijia Chen, Yingxue Zhang, Yiming Li, Zhenzhong Chen, and Zhou Wang. Spherical structural similarity index for objective omnidirectional video quality assessment. In IEEE International Conference on Multimedia and Expo, pages 1-6, 2018.
[5] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation. arXiv preprint arXiv: |406. 1078, 2014.
[6] Yasser Dahou, Marouane Tliba, Kevin McGuinness, and Noel O'Connor. Atsal: An attention based architecture for saliency prediction in 360 videos. In Puttern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10-15, 2021, Proceedings, Part III. pages 305-320, 2021.
Xin Deng, Hao Wang, Mai Xu, Yichen Guo, Yuhang Song, and Li Yang. Lau-net: Latitude adaptive upscaling network for omnidirectional image super- super-resolution. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9189-9198, 2021.
[8] Keyan Ding, Kede Ma, Shiqi Wang, and Ecro P Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE Transactions Pottern Analysis and Machine Intelligence, 4(5):2567-2581, 2020.
Yasser Abdelaziz Dahou Djilali, Kevin McGuinness, and Noel O'Connor Simple baselines can fool 360deg salieney metrics. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 3750-3756, 2021.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, and et al. Weissenbom. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint or ar 2020.
[11] Huiyu Duan, Guangtao Zhai, Xiongkuo Min, Yucheng Zhu, Yi Fang, and Xiaokang Yang. Perceptual quality assessment of omnidirectional images. In 2018 IEEE International Symposium Circuits and Systems, pages 1-5, 2018.
[12] Yuming Fang. Liping Huang. Jiebin Yan, Xuelin Liu, and Yang Liu. Perceptual quality assessment of omnidirectional images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 580-588, 2022.
[13] Yuming Fang. Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. Perceptual quality assessment of smartphone photography. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3677-3686, 2020.
[14] Jun Fu, Chen Hou, Wei Zhou, Jiahua Xu, and Zhibo Chen. Adaptive hypergraph convolutional network for no-reference 360 360-degree image quality assessment. In Proceedings of the 30th ACM International Conference on Multimedia, pages 961-969, 2022.
[15] Jinjin Gu, Haoming Cai. Chao Dong. Jimmyy S Ren, and al. Timofte. Ntire 2022 challenge perceptual image quality assessment. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 951-967, 2022.
[16] Nafisch Jabbari Tofighi, Mohamed Hedi Elfkir, Nevrez Imamoglu, Cagri Ozcinar, Erkut Erdem, and Aykut Erdem. St360iq: No-reference onnidirectional image quality assessment with spherical vision transformers. arXiv, 2023.
[17] Hao Jiang, Gangyi Jiang, Mei Yu. Ting Luo, and Haiyong Xu. Multi-angle projection based blind omni- directional image quality assessment. IEEE Transactions on Circuits and Systems for Video Technology, 32(7):4211 1-2233, 2021.
[18] Hao Jiang, Gangyi Jiang, Mei Yu, Yun Zhang. You Yang. Zongju Peng. Fen Chen, and Qingbo Zhang Cubemap- Cubemap-based perception-drig driven blind quality assessment for 360-degree images. IEEE Transactions on Image Processing. 30:2364 2377,, 2021.
11

---

[19] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5148-5157, 2021.
[20] Hak Gu Kim, Heoun-Tase Tack Lim, and Yong Man Ro. Deep virtual reality image quality assessment with human perception guider for onnidirectional image. IEEE Transactions on Circuits and Systems for Video Technology, 30(4):917-5) 2019.
[21] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: S.Ki::112.66880 2014.
[22] Shanshan Lao, Yuan Gong, Shuwei Shi, Sidi Yang. Tianhe Wu, Jiahao Wang. Weihao Xia, and Yujiu Yang. Attentions help cnns see hetter; Attention-based hybrid image quality assessment network. Proceedigss of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pages 1149, 2022.
[23] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[24] Daniel Martin, Ana Serrano, Alexander W Bergman, Gordon Wetzstein, and Belen Masia. Scangan360: generative model of realistic scanpaths for 360 images. IEEE Transactions on Visualization and Computer Graphics, 28(5):2003-22 2013, 2022.
[25] Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing, 21(12):4695-4708 2012.
[26] Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20(3):209-212, 2012.
[27] Michael I Posner, Yoav Cohen, et al. Components of visual orienting. Attention and performance X: Control of language processes, 32:531-555 1984,
[28] Pratibha Sharma, Manoj Diwakar, and Niranjan Lal. Edge detection using moore neighborhood. Interna- tional Journal of Computer Applications, 61(3), 2013.
[29] Shuwei Shi. Qingyan Bai, Mingdeng Cao, Weihao Xia, Jiahao Wang, Yifan Chen, and Yujiu Yang. Region- adaptive deformable network for image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 324-333, 2021.
[30] Shuwei Shi, Jinjin Gu, Liangbin Xie, Xintao Wang. Yujiu Yang, and Chao Dong. Rethinking alignment in video super resouticsolutio transformers. Advances in Neural Information Processing Systems, 35:36081 -60693,, 2022.
[31] Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala. Diego Gutierrez, Belen Masia, and Gordon Wetzstein. Saliency in vr: How do people explore virtual environments? IEEE Transactions on Visualization and Computer Graphics, 24(4):1633-1644 2018.
[32] Xiangjie Sui, Yuming Fang, Hanwei Zhu, Shiqi Wang, and Zhou Wang. Scandmm: A deep markov model of scanpath prediction for 360 images. IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[33] Xiangjie Sui. Kede Ma. Yiru Yao, and Yuming Fang. Perceptual quality assessment of omnidirectional images as moving camera videos. IEEE Transactions on Visualization and Computer Graphics, 28(8):0022 3034, 2021.
[34] Wanjie Sun, Zhenzhong Chen, and Feng Wu. Visual scanpath prediction using ior-roi recurrent mixture density network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(6):2101-22 2118, 2019.
[35] Wei Sun, Ke Gu, Guangtao Zhai, Siwei Ma. Weisi Lin, and Patrick Le Calle. Cvigd: Subjective quality evaluation of compressed virtual reality images. In 2017 IEEE International Conference on Image Processing, pages 3450-3454, 2017.
[36] Wei Sun, Xiongkuo Min, Guangtao Zhai, Ke Gu, Huiyu Duan, and Siwei Ma. Mc360iqa: A multi- channel can for blind 360-degree image quality assessment. IEEE Journal of Selected Topics in Signal Processing. 14(1):64 77, 2019.
12

---

[37] Yule Sun, Ang Lu, and Lu Yu. Weiggheehoospheriiallyyyamoorrr quality evaluation for omnidirectional video. IEEE Signal Processing Letters, 249)):108-1112, 2017.
[38] Jianyi Wang. Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2555-2563, 2023.
[39] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-6) 612, 2004.
[40] Zhou Wang, Eero P Simoncelli, and Alan C. Bovik. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pages 1402, 2003.
[41] Chen Xia, Junwei Han, Fei Qi, and Guangming Shi. Predicting human saccadic scanpaths based on iterative representation learning. IEEE Transactions Image Processing, 28(7):3502 3515, 2019.
[42] Lianghin Xie, Xintao Wang, Shuwei Shi, Jinjin Gu, Chao Dong, and Ying Shan. Mitigating artifacts in real-world video super-resolution models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2956-2964, 2023.
[43] Jiahua Xu, Wei Zhou, and Zhibo Chen. Blind omnidirectional image quality assessment with viewport oriented graph convolutional networks. IEEE Transactions on Circuits and Systems for Video Technology. 31(5)112441173) 7337, 2020.
[44] Mai Xu, Chen Li, Zhenzhong Chen, Zulin Wang, and Zhenyu Guan, Assessing visual quality of omnidi- rectional videos. IEEE Trunsactions on Circuits and Systems for Video Technology, 29(12):3516-35)) 2018.
[45] Mai Xu, Yuhang Song, Jianyi Wang, Mingl aang Qiao, Liangyu Huo, and Zulin Wang. Predicting head movement in panoramic video: A deep reinforcement learning approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, (111))269332708, 2018.
[46] Li Yang, Mai Xu, Xin Deng, and Bo Feng. Spatial attention-based non-reference perceptual quality prediction network for omnidirectional images. In 2021 IEEE International Conference on Multimedia and Expo, pages 1-6, 2021.
[47] Li Yang, Mai Xu, Yichen Guo, Xin Deng, Fangyuan Gao, and Zhenyu Guan. Hierarchical bayesian Istm for head trajectory prediction on omnidirectional images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7563-7580 2021.
[48] Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao. Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 1191-1200, 2022.
[49] Yan Ye, Elena Alshina, and J Boyce. Jvet-g 1003: Algorithm description of projection format conversion and video quality metries in 360lib version 4, Joint Video Exploration Team, 2017.
[50] Zhemqiang Ying, Haoran Niu, Praful Gupta, Dhruv Mahajan, Deepti Ghadiyaram, and Alan Bovik. From patches to pictures 2paq-2-piq): Mapping the perceptual space of picture quality. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 3575-3585, 2020.
[51] Matt Yu, Haricharan Lakshman, and Bernd Girod, A framework to evaluate omnidirectional video coding schemes. 2015 IEEE International Symposiun on Mixed and Augmented Reality, pages 31- 36-3,,70 2015.
[52] Vladyslav Zakharchenko, Kwang Pyo Choi, and Jeong Hoon Park. Quality metrie for spherical panoramic video. In Optics and Photonics for Information Processing X. volume 9970, pages 57-65, 2016.
[53] Chaofan Zhang and Shiguang Liu. No-reference omnidirectional image quality assessment based on joint network. In Proceedings of the 30th ACM International Conference on Multimedia, pages 943 951, 2022.
[54] Lin Zhang, Ying Shen, and Hongyu Li. Vsi: A visual saliency-induced index for perceptual image quality assessment. IELE Transactions on Image processing, 22710414270-4211,, 2014.
[55] Richard Zhang, Phillip Isola, Alexii A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metrie. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018
13

---

[56] Weixia Zhang, Kede Ma, Jia Yan, Dexiang Deng, and Zhou Wang. Blind image quality assessment using deep bilinear convolutional neural network. IEEE Transactions Circuits and Syxtems for Video Technology. 30(1):36-77, 2018.
[57] Weixia Zhang, Guangtao Zhai, Ying Wei, Xiaokang Yang. and Kede Ma. Blind image quality assessment vision-lange languae correspondence: multitask learning perspective. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14071-1408) 2023.
[58] Xuelei Zheng, Gangyi Jiang, Mei and Hao Jiang. Segmented spherical projection bracction-based based blind omnidirectional image quality assessment. JEEE Access, 8:31647-31659, 2020.
[59] Wei Zhou, Jiahua Xu. Qiuping Jiang, and Zhibo Chen, No-reference quality assessment for 360-degree images by analysis of multifrequency information and local-global naturalness. IEEE Transactions on Circuits and Systems for Video Technology, (2(4):177--199) 2021.
[60] Yu Zhou, Yanjing Sun, Leida Li, Ke Gu, and Yuming Fang. Omnidirectional image quality assessment by distortion discrimination assisted multi-stream network. IEEE Transactions Circuits and Systems for Video Technology, (2(4):76671777 2021.
[61] Yufeng Zhou, Mei Yu, Hualin Ma, Hua Shao, and Gangyi Jiang. Weighted to pabherically-unffone ssim objective quality evaluation for panoramic video. In IEEE International Conference Signal Processing, pages 54 57, 2018.
14


Reviews:
[{"review_id": "2TOMIYyAVo", "reviewer": "Reviewer_7uSg", "summary": "This paper proposed a novel blind omnidirectional image quality assessment framework based on the modeling of the observer’s browsing process, which is called Assessor360. In detail, Assessor360 designs a novel viewpoint sampling methods named “Recursive probability sampling (RPS)” based on content-guided-sampled probability and detailed-guided sampled probability. In addition, the authors utilize the Multi-scale Feature Aggregation (MFA) to fuse the shallow layers’ feature of Swin transformer which is enhanced by proposed distortion-aware block (DAB) and deep layer’s feature which can capture abstract feature.", "questions": "Please refer above comments", "limitations": "none", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clear and the performance is promising. ", "weaknesses": "In general, novelty and insight of the paper is limited to some extent. Below are some questions:\n1.   The main contribution of Assessor360 is the human browsing related viewpoint sampling methods, which combine CSP and DSP. Why is sampling via the equator bias named content-guided? Generally speaking, it may be easier to understand if object information or semantic information is used as content guidance. In DSP, the author did not visualize the entropy map of pixels to verify the motivation of this detail-oriented sampling\n2.\tIn section3.3, the author did not explain how the semantic information was extracted? Or the deep layer’s feature of swin transformer can represent the semantic information? Without constraints on deep layer’s features or adding conditions, it is difficult to understand that deep features have extracted semantic information.\n3.\tRPS can be understood as a plug-and-play sampling method. Has the author tried other methods, such as VGCN, with the RPS sampling method to verify the effect?\n4.\tIt would be better for the authors to try and analysis some other time series modeling modules in TMM, such as transformer, etc.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer #1 for his/her insightful comments. Our responses are as follows:\n\n**`R1-Q1`:** Thanks for the questions. For your `first question`, we agree that the name \"content-guided\" may be commonly used when the method is guided by the object information and semantic information in the traditional 2D scenario. However, in the context of panoramic scene explored in our paper, **\"content-guided\" refers to the bias in information density across different latitudes of the panorama**. A common observation is that content information becomes denser as we move closer to the equatorial region of the panoramic image in most instances [9]. At the same time, It is also indicated in [30] that users tend to fixate around the equator of the panoramas, with very few fixations in latitudes far from it, which can influence the final quality assessment results. To model this prior knowledge, we use equator bias as a guidance prior to calculate the probabilities for scene transitions in the RPS strategy. Hence, we refer to it as Content-guided Sampled Probability (CSP). To further prove this phenomenon, we use the \"Spatial Information (SI)\" metric to conduct the statistic experiment to show the information density distribution across different latitudes of the panorama. The data presented in the table supports the conclusion that the density of information within the panorama exhibits a gradual decline from the equator towards the poles.\n|Latitude|CVIQD|OIQA|IQA-ODI|MVAQD|\n|-|-|-|-|-|\n|54~90|387|2210|3174|814|\n|18~54|950|2386 |5177|1290|\n|-18~18|1560|2576|7019|1604|\n|-54~-18|1010|2463|6480|1496|\n|-90~-54|894|2177|5979|1246|\n\nFor the `second question`, in image processing, information entropy is a metric used to measure the texture complexity of an image [7].  Therefore, our motivation is based on the observation that regions with denser texture information in an image tend to have higher information entropy, making them more attractive to human observers. In our DSP strategy, we assign higher probabilities to the corresponding transformation directions in order to emphasize these visually appealing areas. We visualize the entropy map of pixels to verify the motivation of this detail-oriented sampling in the `attached file Figure A`. The visualization proves that entropy can accurately capture regions with high detail and the sampling direction guided by entropy aligns consistently with the actual observational direction.\n\n**`R1-Q2`:** Thanks for the question. In our approach, we load the pre-trained weights of the Swin Transformer on the ImageNet-1K dataset and utilize the deep layers' features as abstract semantic information. When extracting features from deep networks, especially hierarchical architectures, the deep layers tend to capture more abstract representations, representing the overall semantic information of the image [24]. **Indeed, the usage of deep features from hierarchical backbone as semantic information is a common practice in the field of Image Quality Assessment [22, 50, 12, 53, 45, R1].** HyperIQA [R1] also adopts a similar approach, utilizing deep features from hierarchical backbone to represent semantic information, while using shallow features to capture local distortion information. To further demonstrate this characteristic, we visualize the features at different stages of the Swin Transformer in `attached file Figure B`. The results of the feature map clearly shows that the features of the first two stages primarily focus on local distortions. As the receptive field increases in the latter two stages, the feature map places more emphasis on the viewport semantic information.\n\nAdditional References:\n\n[R1] Su, Shaolin, et al. \"Blindly assess image quality in the wild guided by a self-adaptive hyper network.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n**`R1-Q3`:** Thanks for the question. First, we do **NOT** argue that RPS can be understood as a plug-and-play sampling method in our paper. The key components of our method involve two main steps:  RPS performed in a specific order and then using the TMM to model the sequential relationships among the sampled viewports. **The combination of these two steps is essential for achieving the desired effect (see main paper Table 5 and L292--L295).** However, the possibility of achieving better performance with viewports obtained solely through RPS sampling, compared to previous sampling methods, remains a valid point. In order to delve deeper into this matter, we further conduct experiments by replacing their sampling methods with RPS in VGCN. We use RPS to sample the same number of viewports as [40] to train VGCN. The results presented in the table below demonstrate a substantial performance enhancement for VGCN achieved by the viewport sampled through RPS, resulting in an increase of +0.07 in SRCC for MVAQD. Additionally, this indicates that the viewport sampled using RPS closely aligns with human observations.\n|Method|IQA-ODI||MVAQD||\n|-|-|-|-|-|\n||SRCC|PLCC|SRCC|PLCC|\n|VGCN|0.8117|0.8823|0.8422|0.9112|\n|VGCN-RPS|0.8382|0.8883|0.9122|0.9273|\n\n**`R1-Q4`:** Thanks for your suggestion. We replace the GRU of our method with Transformer, LSTM and Temporal Pooling to conduct the experiment in the dataset. The results are displayed in the table below, highlighting GRU as the optimal performer. Our selection of GRU is justified by the following reasons: **(1)** In contrast to Temporal Pooling, the adaptive GRU exhibits superior temporal representation capabilities. **(2)** In comparison with LSTM and Transformer, GRU boasts a reduced parameter count, facilitating efficient training with limited data and shorter sequences while mitigating the risk of overfitting.\n|Method|MVAQD||OIQA||\n|-|-|-|-|-|\n||SRCC|PLCC|SRCC|PLCC|\n|w/ Temporal Pooling|0.8942|0.9410|0.9566|0.9579|\n|w/ LSTM|0.9261|0.9544|0.9701|0.9649|\n|w/ Transformer|0.9368|0.9613|0.9761|0.9711|\n|w/ GRU|0.9607|0.9720|0.9802|0.9747|"}, {"review_id": "zz5dmTMu3m", "reviewer": "Reviewer_hPRU", "summary": "A blind quality assessment model is proposed for 360-degree images.", "questions": "1. How to initialize the starting points (the total number and positions), and does the initialization affect the final performance?\n2. How to interpret Fig. 4?", "limitations": "N.A.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The problem is interesting and practical.\n2. The attempt to model the observer’s browsing process is a plus.\n3. The results look promising.", "weaknesses": "1. The recursive probability sampling is a little ad hoc to the reviewer. It is mostly handcrafted, which is not optimized to mimic the observer's browsing process. This weakens the motivation of the proposed method.  \n2. No validation set is used for the hyperparameter setting, making the results look like overfitting (surpassing full-reference models by a large margin).\n3. Since no validate set is available, more cross-dataset results are necessary to justify the rationality of the proposed model design.\n4. Two baselines with ERP and cube map projection as direct input should be compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer #2 for his/her comments. Our responses are as follows:\n\n**`R2-Weakness 1`**: Thanks for your comment. We respectfully **DISAGREE** with your opinion that the handcrafted RPS weakens our motivation. \n\nFirstly, it is true that previous OIQA methods have relied on handcrafted viewport sampling from ODIs [40,50], followed by subsequent processing. Our approach leverages the content distribution of panoramic images and incorporates human visual fixation characteristics to simulate the human browsing process, making it more flexible and reasonable. **Handcrafted methods for viewport sampling from panoramic images have indeed become a common practice in the field and do not weaken our motivation.**\n\nSecondly, deep learning-based scanpath prediction methods have not been extensively used in the OIQA task for several reasons: **(1) Limited dataset size**: OIQA datasets typically have a small amount of data, making it challenging to use deep neural networks to simulate observers' browsing process. This can lead to overfitting due to the large parameters of the pipeline. **(2) Lack of scanpath labels**: Many commonly used OIQA datasets lack scanpath labels, making it difficult to perform supervised training for scanpath prediction models. **(3) Difficult to optimize**: The table presented below illustrates that existing optimizable scanpath prediction models have a substantial quantity of parameters. This constitutes a minimum of 21% of the parameters within our OIQA model. Moreover, the current OIQA dataset contains a notably limited amount of data, posing challenges to optimizing the model with an increased parameter load, particularly when constrained by limited hardware resources. \n|Method|Params (M)|Param ratio: Sampling Method / Assessor360|\n|-|-|-|\n|DeepGaze III|78.9|88%|\n|SaltiNet|103.6|116%|\n|ScanGAN|33.9|38%|\n|ScanDMM|18.7|21%|\n|RPS|0.0|0.0%|\n\nFinally, our method exhibits strong generalization and universality, as evident from the additional table on generalization experiments. Compared to other handcrafted sampling methods, RPS achieves superior performance, and simply replacing the sampling method in VGCN with RPS leads to significant improvements (see **`R1-Q3`**). This substantiates the effectiveness of our RPS strategy.\n\n**`R2-Weakness 2 and R2-Weakness 3`**: Thanks. In addition to the cross-dataset experiment, we retrained MC360IQA, VGCN, and our method, using 80% of the dataset for training and the remaining 20% for testing. We then select the model weights with the highest performance on the test set for cross-dataset testing for fair generalization comparisons. The SRCC values are shown in the table below (see **`R3-Weakness 4`** for more results). \n|Method|CVIQD|||OIQA|||MVAQD|||\n|-|-|-|-|-|-|-|-|-|-|\n||OIQA|IQA-ODI|MVAQD|CVIQD|IQA-ODI|MVAQD|CVIQD|OIQA|IQA-ODI|\n|MC360IQA|0.4189|0.7114|0.0296|0.7044|0.5687|0.4081|0.0373|0.0025|0.0486|\n|VGCN|0.2361|0.2875|0.2452|0.6932|0.3873|0.4682|0.4650|0.6227|0.3921|\n|Assessor360|0.4597|0.8610|0.5640|0.8430|0.8751|0.6417|0.8756|0.7765|0.8646|\n\nThese experimental results illustrate the outstanding generalization performance of our model, indicating that it is not just outperforming the FR model due to overfitting. In fact, there are hardly any deep learning-based FR-OIQA quality assessment methods. As a result, the FR methods listed in our paper are primarily proposed for 2D-IQA scenarios. They calculate directly on the ERP format ODI, which exhibits geometric deformation at different latitudes. It will create a gap between the predicted scores and the GT scores. WS-PSNR and WS-SSIM face challenges in aligning with the HVS and exhibit limited capability in characterizing image distortion features. Our approach incorporates the ODI browsing process and possesses a robust capacity to depict distortion, thereby enabling it to attain exceptional performance in both fitting and generalization capabilities.\n\n**`R2-Weakness 4`**: Thanks for your suggestion.  We will involve these two baselines in the paper.\n|Method|CVIQD||OIQA||IQA-ODI||MVAQD||\n|-|-|-|-|-|-|-|-|-|\n||SRCC|PLCC|SRCC|PLCC|SRCC|PLCC|SRCC|PLCC|\n|w/ ERP|0.9330|0.9485|0.8961|0.8857|0.9098|0.9196|0.9076|0.9240|\n|w/ Cube map|0.9390|0.9412|0.9216|0.9170|0.9105|0.9122|0.8966|0.9324|\n\n**`R2-Q1`**:  Different initializations (total number and positions) of the starting point can indeed impact the final performance. For the ODI without real scanpath data, we initialize the coordinates of N starting points to be (0, 0) followed by [32]. In our experiments, we found that increasing the number of starting points might potentially lead to better performance (see Figure 5), but it would also introduce additional computational overhead. We carefully considered the **trade-off** between the number of input viewports and performance and settled on selecting N=3. We also conducted tests to assess the impact of different initial positions on the final performance. Humans tend to start their panoramic observations closer to the equator. The model's performance displays relatively minor fluctuations during longitudinal shifts, yet experiences a substantial decline when changing larger latitudes (from points less frequented by humans as starting positions). This divergence arises due to the model's misalignment with authentic browsing behaviors.\n|Position (latitude, longitude)|SRCC|PLCC|\n|-|-|-|\n|(0, 0), (0, 0), (0, 0)|0.9607|0.9720|\n|(60, 0), (60, 0), (60, 0)|0.9106|0.9312|\n|(0, 120), (0, 0), (0, -60)|0.9599|0.9660|\n|(60, 120), (60, 0), (60, -60)|0.9174|0.9455|\n\n**`R2-Q2`**: We depict the spatial distribution of 15 GT scanpaths within the ODI, with each scanpath encompassing the coordinates of 20 visited viewports. The brighter the area in the figure, the higher the frequency of visits in this area. The purpose of visualizing this representation is to facilitate an intuitive comparison of the accuracy of sampling locations obtained through different sampling methods against the GT."}, {"review_id": "5wHVshRYV5", "reviewer": "Reviewer_rvUB", "summary": "This manuscript proposes a novel multi-sequence network for BOIQA, which is inspired by the actual multi-assessor omnidirectional image quality assessment and introduces a generalized recursive probability sampling method for BOIQA tasks. In addition, a multi-scale feature aggregation module with distortion-aware blocks is designed. Extensive experimental results demonstrate that the proposed method achieves good performance on multiple OIQA datasets. However, there are still some problems in this manuscript.", "questions": "1.\tThe proposed model uses a pre-trained Swin Transformer, but the paper does not specify which kind of training model it is, whether it is tiny, small, or base.\n2.\tIn the training phase, is the learning rate always kept at 1e-5?\n3.\tHow are images tested during the testing phase? While choosing different starting point of each sequence, will the result be different? If the result will be different, how to obtain the final score?\n4.\tThere is a typo, in lines 214 and 215, “SRCC and PLCC stand for Spearman rank-ordered correlation and Pearson linear correlation.” “coefficient” is missing. And the English usage and technical expression should be further polished.\n5.\tIt is suggested that the numbers of the figures and tables in the paper be sorted according to the order of introduction.\n", "limitations": "The proposed RPS has a limitation: the transition direction and distance are fixed, resulting in equally spaced distances between viewports. Design a more effective generation method is author’s future work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "With the development of VR-related techniques, there is an increasing need for omnidirectional image quality evaluation. However, the existing methods lack the modeling of the observer’s browsing process, causing the predicted score to be far from the human perceptual quality score. This paper proposes the first pipeline to simulate the authentic data scoring process in ODI quality assessment, and has certain innovation. Meanwhile, the logic of this manuscript is rigorous.", "weaknesses": "In my opinion, the most contribution of this paper is the recursive probability sampling strategy, while other modules are common. And some details are missing as follows:\n1.\tThe input of the proposed model is in the form of sequences, but the Swin Transformer is for processing single images, hence how does the Transformer process multiple images in the sequence? \n2.\tIn the recursive probability sampling process, how is the starting point of each sequence selected?\n3.\tThe ODI’s quality is averaged by all predicted scores of each sequence, have you considered using weights to fuse these scores? How is the result?\n4.\tThere are few comparative methods and experiments for cross-dataset testing.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer #3 for his/her comments and the appreciation of our work. To the concerns expressed by Reviewer #3 in `Weaknesses` and `Questions`, our response is as follows:\n\n**`R3-Weakness 1`**: Thanks for your question. We utilize the Swin Transformer to process each viewport within the sampled sequence. Rather than processing them simultaneously, we independently extract features for each viewport and subsequently pass them into the Temporal Modeling Module (TMM) for temporal relationship modeling.\n\n**`R3-Weakness 2`**: Thanks for your question. For the ODI without real scanpath data, we initialize the coordinates of N starting points to be (0, 0) followed by [32]. These statements can be found in L203--L204 of the main text. In our experiments we found that increasing the number of starting points might potentially lead to better performance (see Figure 5), but it would also introduce additional computational overhead. We carefully considered the trade-off between the quantity of input viewports and performance and settled on selecting N=3. (More details can be seen in **`R2-Q1`**)\n\n**`R3-Weakness 3`**: Thanks for your question. We adopt a direct averaging approach based on the annotation process of the existing OIQA dataset [11,34]. During the data annotation process, multiple annotators provide ratings for each panoramic image, and the final mean opinion score (MOS) of a panoramic image is obtained by directly averaging the scores from all the annotators. Alternatively, if our focus shifts towards solely fitting the image quality score based on the model, we can conduct corresponding experiments involving the utilization of weighted fusion to amalgamate these scores. The experimental results presented in the table below reveal that there is not much difference between the two regression methodologies. The average approach is more stable when the number of scores is small (we only use three starting points).\n|Method|CVIQD||OIQA||IQA-ODI||MVAQD||\n|-|-|-|-|-|-|-|-|-|\n||SRCC|PLCC|SRCC|PLCC|SRCC|PLCC|SRCC|PLCC|\n|Score-weight fusion|0.9815|0.9821|0.9651|0.9668|0.9528|0.9620|0.9183|0.9621|\n|Score-average|0.9644|0.9769|0.9802|0.9747|0.9573|0.9626|0.9607|0.9720|\n\n**`R3-Weakness 4`**: Thanks for your question. We conduct more cross-dataset testing to demonstrate the generalization capability of our method. We retrained MC360IQA, VGCN and our method, using 80% of the dataset for training and the remaining 20% for testing. We then select the model weights with the highest performance on the testset for cross-dataset testing for fair generalization comparisons.\n\nThe SRCC results are shown in the following table:\n|Method|CVIQD|||OIQA|||MVAQD|||\n|-|-|-|-|-|-|-|-|-|-|\n||OIQA|IQA-ODI|MVAQD|CVIQD|IQA-ODI|MVAQD|CVIQD|OIQA|IQA-ODI|\n|WS-PSNR|0.5027|0.4360|0.7225|0.7638|0.4360|0.7225|0.7638|0.5027|0.4360|\n|WS-SSIM|0.5442|0.5032|0.7930|0.6625|0.5032|0.7930|0.6625|0.5442|0.5032|\n|MC360IQA|0.4189|0.7114|0.0296|0.7044|0.5687|0.4081|0.0373|0.0025|0.0486|\n|VGCN|0.2361|0.2875|0.2452|0.6932|0.3873|0.4682|0.4650|0.6227|0.3921|\n|Assessor360|0.4597|0.8610|0.5640|0.8430|0.8751|0.6417|0.8756|0.7765|0.8646|\n\nThe PLCC results are shown in the following table:\n|Method|CVIQD|||OIQA|||MVAQD|||\n|-|-|-|-|-|-|-|-|-|-|\n||OIQA|IQA-ODI|MVAQD|CVIQD|IQA-ODI|MVAQD|CVIQD|OIQA|IQA-ODI|\n|WS-PSNR|0.4701|0.5468|0.6962|0.7895|0.5468|0.6962|0.7895|0.4701|0.5468|\n|WS-SSIM|0.4363|0.5941|0.6246|0.6536|0.5941|0.6246|0.6536|0.4363|0.5941|\n|MC360IQA|0.4295|0.7872|0.0404|0.7368|0.5930|0.4238|0.0430|0.0202|0.0646|\n|VGCN|0.2582|0.3127|0.2467|0.5929|0.3551|0.2419|0.3420|0.4642|0.3870|\n|Assessor360|0.5332|0.9032|0.5824|0.8636|0.9137|0.6565|0.7232|0.7287|0.8541|\n\nThe results in the above table clearly demonstrate the strong generalization of our method compared to VGCN and MC360IQA. Notably, during instances of training on the MVAQD dataset and subsequent testing on the IQA-ODI dataset, our approach outperforms MC360IQA in terms of SRCC by an approximate margin of 0.8, and surpasses VGCN in SRCC by 0.5. Concurrently, when training on the CVIQD dataset, characterized by fewer distortion types, and testing on the MVAQD dataset, which encompasses more distortion types (JP2K, WN, GB), our approach attains elevated PLCC values of 0.5 and 0.3, respectively, as compared to MC360IQA and VGCN.\n\n**`R3-Q1`**: Thanks for your question. We use the **base** version of the pre-trained Swin Transformer.\n\n**`R3-Q2`**: Thanks for your question. Yes, it is always kept at 1e-5. In contrast to the three-stage training complexity of VGCN, our approach is less intricate. Maintaining learning rate to 1e-5 training can achieve good performance and generalization.\n\n**`R3-Q3`**: In the testing phase, the model's hyperparameters and the quantity of viewports sampled using RPS are maintained at the same values as those employed during the training phase. Initially, employ RPS to generate N scanpaths from the coordinate (0, 0), subsequently forwarding these scanpaths to the network for inference. \n\nChoosing different starting points will generate different viewports, and the results will definitely be different (for detailed analysis, see **`R2-Q1`**). Indeed, some distortions (such as stitching) and non-uniform distortion can lead to substantial discrepancies in quality scores assigned by observers situated at different starting points [12, 32]. Therefore, **for the same image in the JUFE and JXUFE datasets, two MOS scores are calculated as labels for the two starting points**. If the necessity arises to restrict an image to a singular quality score, the viable approach is to compute the average quality score.\n\n**`R3-Q4`**: Thanks for your suggestion. We will make the revisions and improve the English usage and expression according to your suggestions. \n\n**`R3-Q5`**: Thanks for your suggestion. We will make the necessary revisions according to your feedback to make the paper appear more standardized.\n\n"}, {"review_id": "cEgb7CIvB9", "reviewer": "Reviewer_BfkF", "summary": "This paper introduces a BOIQA model called Assessor360, which takes into consideration the observer's browsing behavior. Specifically, RPS method is proposed to combine content and detailed information to generate multiple pseudo-viewport sequences from a given starting point. It also uses an MFA module with DAB to fuse distorted and semantic features. Finally, the viewport transition in the temporal domain is learned using TMM.", "questions": "It would be interesting to compare the proposed method with the latest language-guided IQA models CLIPIQA[2] and LIQE[3].\n\n[1] Wang, J., Chan, K. C., & Loy, C. C. (2022). Exploring CLIP for Assessing the Look and Feel of Images. AAAI 2023.\n\n[2] Zhang, W., Zhai, G., Wei, Y., Yang, X., & Ma, K. (2023). Blind image quality assessment via vision-language correspondence: A multitask learning perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14071-14081).", "limitations": "Not applicable.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The pseudo multi-seqeuence generation is an interesting idea to mimic human behaviors during quality assessment.\n+ The combination of MFA and DAB is reasonable in theory, and effective in practice.\n+ Rich experiments are conducted on multiple datasets.", "weaknesses": "- To implement the TMM module, alternative design choices may be explored, such as LSTM, Transformer decoder, etc.\n- L180-L182, the claims that the first two stages are more sensitive to distortion patterns and the latter capture abstract features are not verified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer #4 for his/her comments and the appreciation of our work. To the concerns expressed by Reviewer #4 in `Weaknesses` and `Questions`, our response is as follows:\n\n**`R4-Weakness 1`**: Thanks for your suggestion. We replace the GRU of our method with Transformer, LSTM and Temporal Pooling to conduct the experiment in the dataset. The results are displayed in the table below, highlighting GRU as the optimal performer. Our selection of GRU is justified by the following reasons: (1) In contrast to Temporal Pooling, the adaptive GRU exhibits superior temporal representation capabilities. (2) In comparison with LSTM and Transformer, GRU boasts a reduced parameter count, facilitating efficient training with limited data and shorter sequences while mitigating the risk of overfitting.\n|Method|MVAQD||OIQA||\n|-|-|-|-|-|\n||SRCC|PLCC|SRCC|PLCC|\n|w/ Temporal Pooling|0.8942|0.9410|0.9566|0.9579|\n|w/ LSTM|0.9261|0.9544|0.9701|0.9649|\n|w/ Transformer|0.9368|0.9613|0.9761|0.9711|\n|w/ GRU|0.9607|0.9720|0.9802|0.9747|\n\n**`R4-Weakness 2`**: Thanks for your comments. In our approach, we load the pre-trained weights of the Swin Transformer on the ImageNet-1K dataset and utilize the shallow layers' features as local distortion information and deep layers' features as abstract semantic information. When extracting features from deep networks, especially hierarchical architectures, the deep layers tend to capture more abstract representations, representing the overall semantic information of the image [24]. Indeed, the usage of deep features from hierarchical backbone as semantic information is a common practice in the field of Image Quality Assessment [22, 50, 12, 53, 45, R1]. HyperIQA [R1] also adopts a similar approach, utilizing deep features from hierarchical backbone to represent semantic information, while using shallow features to capture local distortion information. To further demonstrate this characteristic, we visualize the features at different stages of the Swin Transformer in `attached file Figure B`. The results of the feature map clearly shows that the features of the first two stages primarily focus on local distortions. As the receptive field increases in the latter two stages, the feature map places more emphasis on the viewport semantic information.\n\nAdditional References:\n\n[R1] Su, Shaolin, et al. \"Blindly assess image quality in the wild guided by a self-adaptive hyper network.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n**`R4-Q1`**: Thanks for your suggestion. The results of comparing our methods with ClipIQA and LIQE are shown in the tabel below.\n|Method|CVIQD||OIQA||IQA-ODI||MVAQD||\n|-|-|-|-|-|-|-|-|-|\n||SRCC|PLCC|SRCC|PLCC|SRCC|PLCC|SRCC|PLCC|\n|ClipIQA+|0.4148|0.3333|0.1202|0.0933|0.0597|0.0695|0.3504|0.2621|\n|ClipIQA (resnet50)|0.4884|0.4347|0.2330|0.2531|0.0927|0.1929|0.5862|0.4941|\n|LIQE|0.8594|0.8086|0.7634|0.7419|0.8551|0.9020|0.6837|0.7539|\n|Assessor360|0.9644|0.9769|0.9802|0.9747|0.9573|0.9626|0.9607|0.9720|\n\nFrom the results in the above table, it can be seen that the performance of LIQE on the four OIQA datasets is much higher than that of ClipIQA. However, they calculate directly on the ERP format ODI, which exhibits geometric deformation at different latitudes. It will create a gap between the predicted scores and the GT scores. Our approach incorporates the ODI browsing process and possesses a robust capacity to depict distortion, thereby enabling it to attain exceptional performance in both fitting and generalization capabilities.\n"}]

Author rebuttal:
Dear Reviewers and ACs:

We sincerely appreciate all the reviews. They give positive and high-quality comments on our paper with a lot of constructive feedback.
We would like to emphasize that our work proposes a novel, simple and effective new pipeline called Assessor360. We appreciate your recognition of our pioneering effort in attempting to simulate the real panorama quality assessment process, aligning it with the evaluation performed by actual human raters. Our novelty and contribution can be summarized as following:

(1) It is crucial to emphasize that our method can simulate the browsing process of human observers on OIQA datasets, especially on those lack real observed scanpaths through our carefully designed sampling strategy, and has achieved state-of-the-art performance. The paradigm proposed is indeed novel for the community. We first demonstrate the significance of simulating the real scoring process for the OIQA task through our proposed method. In fact, under certain distortions (stitching), observers will give very different quality scores according to their observing process. This is also the primary reason why certain OIQA methods that neglect the sequence of observations yield subpar performance on the JUFE dataset. Future OIQA research can benefit from our findings.

(2) We propose a novel and efficient sampling method called RPS, which sequentially samples salient viewports. It is the first time a viewport sampling method leverages both scene characteristics and local texture and distortion properties to mimic the observer's real evaluation process. Through our approach, we compare RPS with deep learning-based sampling methods. Our method outperforms the deep learning-based methods in terms of performance while requiring fewer parameters.

(3) It is worth noting that ours is the first method to address the OIQA task using a novel multi-sequence pipeline. In the actual data annotation process, we obtain the final label by averaging evaluation scores from multiple different observers who have observed the panorama. Unlike previous approaches that simply model a fixed observation order and directly regress them to the final label, our multi-sequence based method effectively captures this complex process.

We have supplemented our work with additional experiments and visual analyses to incorporate the insightful suggestions of the reviewers:

According to Reviewer 7uSg’s concerns, we visualize the entropy map of pixels to verify the motivation of our proposed detail-oriented sampling in Figure A of the attached file.

According to Reviewer 7uSg and Reviewer BfkF’s questions, we visualize the four-stage feature maps generated by Swin Transformer.

According to Reviewer 7uSg and Reviewer BfkF’s concerns, we add the analytical experiment on TMM with different modules in the rebuttal area.

According to Reviewer 7uSg's question, we add the experiments on the effects of RPS in VGCN.

According to Reviewer hPRU and Reviewer rvUB's concerns, we add more cross-dataset testing experiments in the rebuttal area.

According to Reviewer hPRU and Reviewer rvUB's questions, we have conducted a thorough analysis of the impact of starting points on the final performance and supplementary experiments to support our findings in the rebuttal area.

According to Reviewer hPRU and Reviewer BfkF's suggestions, we add more methods to compare with our results in the rebuttal area.

According to Reviewer rvUB's suggestions, we will make the necessary revisions according to your feedback to make the paper appear more standardized.

In the final version, we will improve other minor points of Reviewer 7uSg, Reviewer hPRU, Reviewer rvUB, and Reviewer BfkF. Thank you all for the valuable suggestions.

Thanks,

Paper 4933 Authors.





Meta-review: Reviewers raised issues and offered suggestions regarding novelty, alternatives to sampling and time series modeling, generalization, and comparisons with baseline methods in the initial reviews. Rebuttals provide additional visualizations and experiments requested by the reviews. The reviewers generally agree that the rebuttal is convincing and are optimistic about the paper. Most reviews appreciate the idea of modeling observers' browsing behavior for quality assessment. Moreover, the RPS sampling method is novel, and the combination of MFA and DAB is effective. Experiments show that the proposed method surpasses the STOA performance and significantly outperforms the compared methods.
Decision: Accept (poster)