Title: Assessor360: Multi-sequence Network for Blind Omnidirectional Image Quality Assessment
Venue: NeurIPS 2023
Accepted: yes
Keywords: Blind omnidirectional image quality assessment

Multi-sequence network

Viewport sequence
Abstract: Blind Omnidirectional Image Quality Assessment (BOIQA) aims to objectively assess the human perceptual quality of omnidirectional images (ODIs) without relying on pristine-quality image information. It is becoming more significant with the increasing advancement of virtual reality (VR) technology. However, the quality assessment of ODIs is severely hampered by the fact that the existing BOIQA pipeline lacks the modeling of the observer's browsing process. To tackle this issue, we propose a novel multi-sequence network for BOIQA called Assessor360, which is derived from the realistic multi-assessor ODI quality assessment procedure. Specifically, we propose a generalized Recursive Probability Sampling (RPS) method for the BOIQA task, combining content and details information to generate multiple pseudo viewport sequences from a given starting point. Additionally, we design a Multi-scale Feature Aggregation (MFA) module with a Distortion-aware Block (DAB) to fuse distorted and semantic features of each viewport. We also devise Temporal Modeling Module (TMM) to learn the viewport transition in the temporal domain. Extensive experimental results demonstrate that Assessor360 outperforms state-of-the-art methods on multiple OIQA datasets. The code and models are available at https://github.com/TianheWu/Assessor360.

Assessor360: Multi-sequence Network for Blind Omnidirectional Image Quality Assessment

Abstract

Blind Omnidirectional Image Quality Assessment (BOIQA) aims to objectively assess the human perceptual quality of omnidirectional images (ODIs) without relying on pristine-quality image information. It is becoming more significant with the increasing advancement of virtual reality (VR) technology. However, the quality assessment of ODIs is severely hampered by the fact that the existing BOIQA pipeline lacks the modeling of the observer's browsing process. To tackle this issue, we propose a novel multi-sequence network for BOIQA called Assessor360, which is derived from the realistic multi-assessor ODI quality assessment procedure. Specifically, we propose a generalized Recursive Probability Sampling (RPS) method for the BOIQA task, combining content and details information to generate multiple pseudo viewport sequences from a given starting point. Additionally, we design a Multi-scale Feature Aggregation (MFA) module with a Distortion-aware Block (DAB) to fuse distorted and semantic features of each viewport. We also devise Temporal Modeling Module (TMM) to learn the viewport transition in the temporal domain. Extensive experimental results demonstrate that Assessor360 outperforms state-of-the-art methods on multiple OIQA datasets. The code and models are available at https://github.com/TianheWu/Assessor360.

1 Introduction

With the development of VR-related techniques, viewers can enjoy a realistic and immersive experience with head-mounted displays (HMDs) [3, 20] by perceiving 360-degree omnidirectional information. However, the acquired omnidirectional image, also named panorama, is not always of high quality [12, 60]. Degradation may be introduced in any image processing [44, 58, 42], leading to low-quality content that may be visually unpleasant for users. Consequently, developing suitable quality metrics for panoramas holds considerable importance, as they can be utilized to guide research in omnidirectional image processing and maintain high-quality visual content.

Generally, most ODIs are stored in the equirectangular projection (ERP) format, which exhibits considerable geometric deformation at different latitudes. This distortion can have a negative impact on quality assessment. Therefore, as shown in Figure 1 (a), many researchers [36, 17, 43, 58, 14, 60] explore viewport-based methods by projecting the original ERP image into many undeformed viewports (actual content observed by users) and aggregate their features with 2D IQA method as the ODI quality score. However, this conventional viewport-based pipeline lacks the modeling of the

(evaluators)

Multiple

Scanpaths

Average

Score

1

Score

𝑛

Quality

score

Ordinary Viewport-based Pipeline

Quality Representation

Quality

score

Omnidirectional Image

Rectilinear

Project

𝑛

Different Perspective Viewports

2D IQA

Method

Feature

1

Feature

2

Feature

𝑛

Aggregate

Generating different perspective viewports

Viewport quality representation

Score prediction

(a)

Observing and perceiving the quality of omnidirectional image

Recalling the quality of the observed viewports

Quality score rating

Observe

Scanpath

1

Scanpath

𝑛

Omnidirectional Image

Recall quality

(b) Realistic ODI Quality Assessment Procedure

observer's browsing process, causing the predicted quality to be far from the human perceptual quality, especially when the ODI containing non-uniform distortion (such as stitching) [33, 12, 36, 61, 37]. In fact, during the process of observing ODIs, viewports are sequentially presented to viewers based on their browsing paths, forming a viewport sequence.

While the viewport sampling techniques proposed by Sui et al. [33] and Zhang et al. [53] can generate viewports with a fixed sequential order for ODIs, their methods lack the ability to generate versatile sequences. Specifically, their methods will produce identical sequences for a given starting point, and the sequential order of viewports will remain constant across different ODIs. This behavior is inconsistent with the observations of multiple evaluators in realistic scenarios, leading to an inability to provide subjectively consistent evaluation results. To address this issue, a logical approach is to employ the scanpath prediction model for generating pseudo viewport sequences on the ODI without an authentic scanpath. However, current scanpath prediction methods [47, 24, 32, 1, 34] are developed for undistorted ODIs and mainly focus on high-level regions. As evaluators are required to provide an accurate quality score for an ODI, their scanpaths are distributed across both low-quality and high-detail regions. This makes it potentially unsuitable for direct application in BOIQA task where all ODIs are distorted.

To move beyond these limitations, inspired by the realistic multi-assessor ODI quality assessment procedure, shown in Figure 1 (b), we first propose a multi-sequence network called Assessor360 for BOIQA, which can simulate the authentic data scoring process to generate multiple viewport sequences (corresponding to multiple scanpaths). Specifically, we propose Recursive Probability Sampling (RPS) to generate multiple pseudo viewport sequences, combining semantic scene and local distortion characteristics. In particular, based on Equator-guided Sampled Probability (ESP) and Details-guided Sampled Probability (DSP), RPS will generate different viewport sequences for the same starting point (details in Section 3.2). Furthermore, we develop Multi-scale Feature Aggregation (MFA) with Distortion-aware Block (DAB) to effectively fuse viewport semantic and distorted features for accurate quality perception. The Temporal Modeling Module (TMM) is devised by applying GRU [5] module and MLP layers to learn the viewports temporal transition information in a sequence and regress the aggregated features to the final score. Extensive experiments demonstrate the superiority and effectiveness of proposed Assessor360 on multiple OIQA datasets (MVAQD [18], OIQA [11], CVIQD [35], IQA-ODI [46], JUFE [12], JXUFE [33]). We summarize our contributions into four points:

We propose Assessor360 for BOIQA, which can leverage multiple different viewport sequences to assess the ODI quality. To our knowledge, Assessor360 is the first pipeline to simulate the authentic data scoring process in ODI quality assessment.

We propose an unlearnable method, Recursive Probability Sampling (RPS) that can combine semantic scene and local distortion characteristics to generate different viewport sequences for a given starting point.

We design Multi-scale Feature Aggregation (MFA) and Distortion-aware Block (DAB) to characterize the integrated features of viewports and devise a Temporal Modeling Module (TMM) to model the temporal correlation between viewports.

We apply our Assessor360 to two types of OIQA task datasets: one with real observed scanpath data and the other without. Extensive experiments show that our Assessor360 largely outperforms state-of-the-art methods.

2 Related Work

Omnidirectional Image Quality Assessment. Similarly to traditional 2D IQA, according to the reference information availability, OIQA can also be divided into three categories: full-reference (FR), reduced-reference (RR), and no-reference (NR) OIQA, also known as blind OIQA [2]. Due to the structural characteristics of the panorama and the complicated assessment process, OIQA has not matured as much as 2D-IQA [48, 22, 13, 19, 15, 29]. Some researchers extend the 2D image quality assessment metrics to the panorama, such as S-PSNR [51], CPP-PSNR [52], and WS-PSNR [37]. However, these methods are not consistent with the Human Visual System (HVS) and they are poorly consistent with perceived quality [60, 43]. Although WS-SSIM [61] and S-SSIM [4] consider some impacts of HVS, the availability of non-distortion reference ODIs severely hinders their applications in authentic scenarios.

Therefore, some deep learning-based BOIQA methods [60, 43, 36, 16, 53, 46] are devised to achieve better capabilities. Due to the geometric deformation present in ODIs in ERP format, many existing BOIQA methods [59, 46, 60, 17] follow a similar pipeline: sampling viewports in a particular way and simply regressing their features to the quality score. MC360IQA [36] first maps the sphere into a cubemap, then employs CNN to aggregate each cubemap plane feature and regress them to a score. ST360IQ [16] samples tangent viewports from the salient parts and uses ViT [10] to estimate the quality of each viewport. VGCN [43] migrates the graph convolution network and pre-trained DBCNN [56] to establish connections between different viewports. However, these methods ignore the vital effect of the viewport sequences generated in multiple observers' browsing process, which have been demonstrated by Sui et al. [33] and Fang et al. [12] on the perception of the ODI quality.

Viewport Sampling Strategies in OIQA. Existing viewport sampling strategies can be categorized into three modes: 1) Uniformly sampling without sequential order. Zhou et al. [59] and Fang et al. [12] uniformly extract viewports over the sphere. Jiang et al. [17] and Sun et al. [36] use cube map projection (CMP) and rotate the longitude to obtain several viewport groups. This sampling pattern will cover the full areas, whether they are significant or non-significant. 2) Crucial region sampling without sequential order. Xu et al. [43] leverage 2D Gaussian Filter [45] to acquire a heatmap and generate viewports with corresponding locations. Tofighi et al. [16] apply ATSal [6] to predict salient regions of the panorama, which gives help to sampling viewports. This mode incorporates HVS, where most of the sampled viewport might be observed in the browsing process. Nevertheless, both of the above methods only focus on the image content and do not concern the effect of the sequential order between viewports in the quality assessment process of panoramas. 3) Sampling viewports along the fixed direction. Sui et al. [33] first considers the effect of sequential order between viewports in the browsing process. They default observers rotate their perspective in a specific direction along the equator. Zhang et al. [53] further introduce ORB detection to capture key viewports and follow Sui et al. [33] default sampling direction to extract viewports. In fact, different evaluators will produce various scanpaths when viewing a panorama. Even if the same evaluator observes the same ODI twice, the scanpath will be different. However, their sampling methods cause the sequential order of the viewport to be fixed, and even for the different image contents, the sequential order of sampled viewports is the same. This creates a significant gap with people's actual browsing process, leading to unreasonable modeling of the viewport sequence.

3 Method

3.1 Overall Framework

As shown in Figure 2 (a), the proposed Assessor360 framework consists of Recursive Probability Sampling (RPS) scheme, Multi-scale Feature Aggregation (MFA) strategy, and Temporal Modeling Module (TMM). Given a degraded ODI I , to be consistent with the authentic multi-assessor assessment, we initialize N starting points X = { ( u i , v i ) } N i =1 , where u i ∈ [ -90 ◦ , 90 ◦ ] and v i ∈ [ -180 ◦ , 180 ◦ ] are the corresponding latitude and longitude coordinates. Then, we apply the proposed RPS strategy G with parameters Θ g to generate multiple viewport sequences S = {{V i j } j M =1 } N i =1 , where V denotes

Distortion-aware Block (DAB)

Conv

3 × 3

Conv

1 × 1

GELU

1

×𝟏× 𝑪

𝟐

𝑯×𝑾×𝑪

Conv

1 × 1

GAP

1

×𝟏×𝑪

Channel Attention (CA)

×𝒏

Recursive Probability

Sampling

Starting points

𝐈𝐧𝐩𝐮𝐭: 𝐎𝐃𝐈

c

Stage 1

Stage 2

Stage 3

Stage 4

Conv

Conv

Conv

Conv

DAB

DAB

Conv

Conv

Conv

c

c

GRU

GRU

MLP

𝑠 1

score

𝑠2

score

𝑠𝑛

score

ODI score

average

take last token

Recursive Probability

Sampling (RPS)

Calculate

Entropy

Split

𝒆𝟏

𝒆𝟐

𝒆𝟑

𝒆𝟒

𝒆𝟓

𝒆𝟔

𝒆𝟕

𝒆𝟖

Softmax

Coordinate probability map

8

overlapped neighbor

patches

Generate position

chosen probabilities

𝑝𝑑𝑠𝑝

1

Softmax

Select Next

Viewport

(𝑢𝑐

𝑡 +Δ𝑢

,

𝑣𝑐

𝑡 +Δ𝑣)

(𝑢𝑐

𝑡

,

𝑣𝑐

𝑡 +Δ𝑣)

(𝑢𝑐

𝑡 -Δ𝑢

,

𝑣𝑐

𝑡 +Δ𝑣)

(𝑢𝑐

𝑡 +Δ𝑢

,

𝑣𝑐

𝑡 -Δ𝑣)

(𝑢𝑐

𝑡 +Δ𝑢

,

𝑣𝑐

𝑡

)

(𝑢𝑐

𝑡

,

𝑣𝑐

𝑡 -Δ𝑣)

(𝑢𝑐

𝑡 -Δ𝑢

,

𝑣𝑐

𝑡 -Δ𝑣)

(𝑢𝑐

𝑡 -Δ𝑢

,

𝑣𝑐

𝑡

)

Calculating

8

neighbor patches coordinates

Equator-guided Sampled Probability (ESP)

𝑝𝑑𝑠𝑝

2

𝑝𝑑𝑠𝑝

8

Details-guided Sampled Probability (DSP)

Softmax

𝑃 𝑥

1

𝑃 𝑥

2

𝑃 𝑥

8

𝑝𝑒𝑠𝑝

1

𝑝𝑒𝑠𝑝

2

𝑝𝑒𝑠𝑝

8

෤

𝑝1

෤

𝑝2

෤

𝑝8

(b)

𝑠1

Viewport Sequences

𝑠2

𝑠𝑛

Generating various viewport sequences by RPS method

Multi-scale Feature Aggregation (MFA)

Temporal Modeling Module (TMM)

(a) Architecture of proposed Assessor360

一一一一一一

PEP

DL

Vt

Pt

Pt

P

neighbor

一一一一

Vt+1

Vt-1

each viewport, M denotes the length of each sequence. Next, to perceive the semantic scene and distortion in each viewport, multi-scale features are extracted from multi-stage layers of our method for quality assessment. We use MFA F with parameters Θ f to represent this process. The features are aggregated from R H × W × C to R 1 × 1 × D . In this case, C is the viewport's original dimensions, H and W are the height and width of the viewport, and D denotes the embedding dimension. Then, we use TMM H with parameters Θ h to model each sequence viewport temporal transition information and predict the final viewport sequence quality score. Finally, we average all predicted scores of each sequence as the ODI's quality Q I . Overall, the whole process can be described as follows:

3.2 Recursive Probability Sampling

Recursive Probability Sampling (RPS) strategy can adaptively generate the probability of scene transition direction based on prior knowledge of semantic context and degraded features in ODI. It mainly consists of Equator-guided Sampled Probability (ESP) and Details-guided Sampled Probability (DSP). The viewport sequence is generated by selecting a certain starting point and sampling viewports based on generated probabilities.

Preprocessing for generating probabilities. As illustrated in [47], transition direction and distance are two important factors for locating the next viewport position. We first follow the theory of Moore neighborhood [28] and define K neighbor ( K = 8 ) transition directions from the center coordinate of the viewport. Following [31, 33], the transition distance (∆ u, ∆ v ) is set to ( 24 ◦ , 24 ◦ ) , avoiding sampling overly overlapped viewports. In details, given a current position x t c = ( u t c , v t c ) , we first generate the corresponding viewport V t from I by rectilinear projection [49] R . Then, we uniformly split V t into K +1 overlapped patches, including K neighbor patches P t = {P t i } K i =1 and one central patch P t c with height H 2 and width W 2 . As shown in Figure 2 (b), the K direction sampled coordinates X t neighbor = { x t i } K i =1 can be calculated by the central patch P t c coordinate x t c and (∆ u, ∆ v ) . During the browsing process, assessors are not only drawn to the high-level scenario but also focus on low-level texture and details regions [35, 11] to give a reasonable quality score. Therefore, according to the generalized prior content information and pixel-level details metric, we present ESP and DSP for each direction sampled coordinate in X t neighbor . Then, we choose the next sampled viewport V t +1 position x t +1 c based on them.

Equator-guided Sampled Probability (ESP). Since the equator entails more scene information [7, 31], we introduce the prior equator bias [9] M to constrain the sampled viewport near the equator. Concretely, the prior equator bias obeys a Gaussian distribution with a mean of 0 and a standard deviation of 0 . 2 in latitude. Regions near the equator have higher sampled weights and regions close

Algorithm 1 Viewport Sequence Generation (RPS Algorithm)

Input: N starting points { x i } N i =1 ; an ODI I ; rectilinear projection R ; ESP calculation function F esp ( X ) ; DSP calculation function F dsp ( P ) ; selecting function Γ( X| ˜ p )

Output: A set of N length M viewport sequences { s i = {V t } t M =1 } N i =1 ;

1: for i = 1 → N do

2: Initialize the current coordinate x ← x i

3: for t = 1 → M do

4: Generate viewport by the current coordinate V ← R ( x, I )

5: Split V to obtain overlapped neighbor patches P and calculate sampled coordinate X

6: Calculate ESP and DSP p esp ←F esp ( X ) , p dsp ←F dsp ( P )

7: Generate next viewport coordinate x ← Γ( X| Aggregate( p esp , p dsp ))

8: Sequentially gather generated M viewports {V t } t M =1 as a viewport sequence s i

9: Gather generated N viewport sequences { s i } N i =1 as the output

to the poles have relatively low sampled weights. We apply the softmax function to M to obtain the coordinate probability map M p where each coordinate corresponds to a sampled probability. Then we take X t neighbor sampled probabilities { p i x } K i =1 from M p . Meanwhile, due to the inhibition of return (IOR) [27] where regions that have been fixated by the eyes have a lower probability of being fixated again in the near future, we multiply p k x with a decreasing factor γ where k is the index of the viewport coordinate has been generated. Finally, we apply softmax function to calculate p esp = { p i esp } K i =1 . Overall, the ESP calculation function F esp ( X ) can be defined as:

Details-guided Sampled Probability (DSP). To efficiently measure the texture complexity of each patch, we use pixel-level information entropy E . The mathematical expression is:

where P t i [ m,n ] is the gray value of each pixel at position ( m,n ) in the patch P t i , and p ( · ) is the probability of occurrence of each gray value. After calculating the entropy of each neighbor patch, we normalize it with the softmax function to obtain p dsp = { p i dsp } K i =1 for X t neighbor . The DSP calculation function F dsp ( P ) can be formulated as:

Generating viewport sequences. Ultimately, we multiply the ESP and DSP with a scale factor β to get the integrated probability. We use the softmax function to obtain the final probability distribution and select the next viewport position x t +1 c according to it, which can be written as:

where Γ( X| ˜ p ) is the selecting function based on the set of probability ˜ p . We can recursively generate viewport V t +1 with x t +1 c and keep performing the RPS strategy until the number of viewports in the sequence reaches the desired length. The whole generation algorithm G is shown in Algorithm 1.

3.3 Multi-scale Feature Aggregation

To represent the semantic information and distortion pattern of each viewport, which are assumed as two key factors of quality assessment, we aggregate multi-scale features of the viewport. We first extract features from each stage in pre-trained Swin Transformer [23, 30]. The outputs of the first two stages F 1 , F 2 are more sensitive to the distortion pattern, while the outputs of the last two stages F 3 , F 4 tend to capture the abstract features (details in Supplementary Materials). Before fusing multi-scale features, we first employ four 1 × 1 convolution layers to reduce the feature dimension of the output to D . Then, to further emphasize the local degradation, we devise and apply the Distortion-aware Block (DAB) which includes a 3 × 3 convolution layer following n Channel Attention (CA) operations to the reduced shallow features ˆ F 1 , ˆ F 2 . This operation can help the model to better perceive the distortion pattern in the channel dimension, achieving distortion-aware capacity. Finally, we concatenate and integrate these features with Global Average Pooling (GAP) and multiple 1 × 1 convolution layers to get the quality-related representation. After that, each aggregated feature F fuse will be sent to TMM for viewport sequence quality assessment.

3.4 Temporal Modeling Module

The browsing process of ODI naturally leads to the temporal correlation. The recency effect indicates that users are more likely to evaluate the overall image quality affected by the viewports they have recently viewed, especially during prolonged exploration periods. To model this relation, we introduce the GRU module to learn the viewport transition. Due to the fact that the last token encodes the most recent information and the representation at the last time step involves the whole temporal relationships of a sequence, we use MLP layers to regress the last feature output by the GRU module to the sequence quality score.

4 Experiments

4.1 Implementation Details

We set the field of view (FoV) to the 110 ◦ following [12, 33]. We use pre-trained Swin Transformer [23] (base version) as our feature extraction backbone. The input viewport size H × W is fixed to 224 × 224 . The number of viewport sequences N is set to 3 and the length of each sequence M is set to 5 . We set the coordinates of N starting points to be (0 ◦ , 0 ◦ ) . The reduced dimension D is 128 and the number of GRU modules is set to 6 . The number of CA operations n is 4 . We set γ = 0 . 7 and β = 100 as decreasing factor and scale factor values respectively.

For a fair comparison, we randomly split 80% ODIs of each dataset for training, and the remaining 20% is used for testing following [53, 18, 12, 17]. To eliminate bias, we run a random train-test splitting process ten times and show the median result. We train 300 epochs with batch size 4 on CVIQD [35], OIQA [11], IQA-ODI [46], and MVAQD [18] datasets without the authentic scanpath data. Respectively, we compare our RPS with two advanced learning-based scanpath prediction methods ScanGAN360 [24] and ScanDMM [32] on JUFE [12] and JXUFE [33] datasets which have the authentic scanpath data. For optimization, we use Adam [21] and the learning rate is set to 1 × 10 -5 in the training phase. We employ MSE loss to train our model. We use Spearman rank-ordered correlation (SRCC) and Pearson linear correlation (PLCC) as the evaluation metrics.

4.2 Comparing with the State-of-the-art Methods

We conduct a comparative analysis of Assessor360 with eight FR methods and thirteen NR methods. The quantitative comparison results are presented in Table 1, demonstrating significant performance

improvements across all datasets against the state-of-the-art methods. Note that these performance enhancements are obtained without training with the 2D IQA dataset, as employed in VGCN [43]. Specifically, Assessor360 outperforms VGCN by 3 % in terms of SRCC on the OIQA dataset and shows a 1.2 % increase in PLCC on the CVIQD dataset. Additionally, our method exhibits a notable improvement of up to 1.6 % in SRCC compared to AHGCN [14].

4.3 Cross-Dataset Evaluation

To assess the generalization capability of our method, we perform cross-validation on the CVIQD [35] and OIQA [11] datasets and compare it against two widely used state-of-the-art methods: MC360IQA [36] and MP-BOIQA [17]. All models are trained on the MVAQD [18] dataset and subsequently tested on the CVIQD and OIQA datasets. The results, presented in Table 3, demonstrate that our method exhibits superior generalization performance compared to the other models. It is worthy to note that, there exists a significant domain gap between the CVIQD and MVAQD datasets. The CVIQD dataset primarily focuses on degradations such as JPEG, H.264/AVC, and H.265/HEVC, while the MVAQD dataset concentrates on JPEG, JP2K, HEVC, WN, and GB. This discrepancy in focus poses a challenge for MC360IQA and MP-BOIQA, as they use the same perspective viewports to evaluate different distorted ODIs, leading to poor generalization. In contrast, our proposed RPS overcomes this domain gap effectively by generating different viewport sequences for ODIs with different distortions. It consistently performs well, showcasing its effectiveness in addressing the challenges posed by diversely distorted ODIs.

Furthermore, to verify that the performance presented in Table 1 is not due to the overfitting, we retrain MC360IQA, VGCN [43] and Assessor360 on CVIQD, OIQA and MVAQD datasets, using 80% of the dataset for training and the remaining 20% for testing. We then select the model weights with the highest performance on the testset for cross-dataset testing for fair generalization comparisons. The results shown in the Table 2 clearly demonstrate the strong generalization of our method compared to VGCN and MC360IQA. Notably, during instances of training on the MVAQD dataset and subsequent testing on the IQA-ODI [46] dataset, our approach outperforms MC360IQA in terms of SRCC by an approximate margin of 0.8, and surpasses VGCN in SRCC by 0.5. Concurrently, when training on the CVIQD dataset, characterized by fewer distortion types, and testing on the MVAQD dataset, which encompasses more distortion types (JP2K, WN, GB), our approach attains elevated PLCC values of 0.5 and 0.3, respectively, as compared to MC360IQA and VGCN.

Brightness discontinuity

Stitching

ScanGAN360

ScanDMM

RPS (Ours)

GT

0

-0

2

4

4

2

4

6

8

10

0

2

4

6

8

10

0

2

4

6

8

10

0

2

4

6

8

10

ScanGAN360

ScanDMM

RPS (Ours)

GT

0

0

-0

2

4

4

4

2

4

6

8

10

0

2

4

6

8

10

0

4

6

8

10

0

2

4

6

810

ScanGAN360

ScanDMM

RPS (Ours)

GT

-0

0

0

-0

2

2

4

4

2

4

6

8

10

0

2

4

6

10

0

2

4

6

8

10

2

4

6

8

10

ScanGAN360

ScanDMM

RPS (Ours)

GT

0

0

0

-0

2

4

4

4

4

0

2

4

6

8

10

2

4

6

8

10

0

2

4

6

8

10

0

2

4

6

8

10

4.4 Effectiveness of Recursive Probability Sampling

As mentioned in Section 1, there exist many learning-based scanpath prediction methods [24, 47, 32]. They seem to be able to assist in constructing viewport sequences. However, they are hardly introduced to OIQA task. In this section, we first perform the quantitative comparison of the model with RPS and two advanced 360-degree scanpath prediction methods, namely ScanGAN360 [24] and ScanDMM [32] on the datasets without real observed scanpath. Subsequently, we compare the position and sequential order of viewports generated by the three methods with the ground-truth (GT) scanpaths by the metrics of scanpath prediction task and visualization to further validate the superiority of RPS on JUFE [12] and JXUFE [33] datasets with real observed scanpath.

Quantitative comparison of performance. We replace the proposed RPS method with ScanGAN360 and ScanDMM to generate sequences of viewports. The model was trained and tested using these viewport sequences on OIQA [11] and MVAQD [18] datasets, maintaining the same length and number of viewports for a fair comparison. Table 4 shows the quantitative results, demonstrating that viewports generated from RPS yield superior performance compared to ScanGAN360 and ScanDMM. Additionally, training using viewports generated from these methods outperforms those generated using random schemes, highlighting the crucial role of suitable viewport sequences in the OIQA task.

Moreover, we conduct experiments by replacing original VGCN [43] sampling methods with RPS in VGCN. We use RPS to sample the same number of viewports as [43] to train VGCN on IQA-ODI [46] and MVAQD datasets. The results presented in Table 7 demonstrate a substantial performance enhancement for VGCN achieved by the viewport sampled through RPS, resulting in an increase of 0.07 in SRCC for MVAQD. Additionally, this indicates that the viewport sampled with RPS closely aligns with human observations.

Comparison of viewport sampling positions and order. Since each ODI in JUFE and JXUFE comprises 30 and 22 scanpaths, respectively, we employ the three methods to generate 30 and 22 sequences, each comprising 20 viewports, for each ODI in JUFE and JXUFE, respectively. For each GT sequence, we sample 20 viewports from 300 viewports with equal time intervals. We adopt Levenshtein distance (LEV) and dynamic time warping (DTW) metrics, as used in ScanGAN360 and ScanDMM, to evaluate the position and sequential order of the viewports. Furthermore, the viewing behaviors and patterns can be evaluated using the recurrence measure (REC) [24]. Each metric is calculated by comparing each generated sequence against all the GT sequences and computing the average, resulting in the final value. Besides, to establish an upper bound for each metric, we compute the Human Baseline by averaging the results of comparing each GT sequence against all other GT sequences [41]. Similarly, we establish a lower bound by randomly sampling viewports from the ODI, referred to as the Random Baseline.

The quantitative comparison results are reported in Table 5. The proposed unlearnable method outperforms learning-based models regarding the LEV and REC metrics. Specifically, it achieves a LEV improvement of 1.75 and 1.82 compared to ScanDMM [32], and a REC improvement of 1.14 and 0.93 compared to ScanGAN360 [24]. Additionally, our method achieves comparable performance to ScanDMM and ScanGAN360 on the DTW metric for both datasets. Figure 3 presents the positions of generated viewports for different generation methods. The panorama is divided into 72 regions, with each region spanning 15 ◦ degrees of latitude and 30 ◦ degrees of longitude. It can be observed that the positions generated by the learning-based method are relatively dispersed, while the positions generated by RPS are concentrated near the equator. The concentration gradually decreases with increasing latitude, aligning closely with the ground truth (GT) position.

4.5 Ablation Studies

Impact of the initialization of the starting point. For the ODI without real scanpath data, we initialize the coordinates of N starting points to be ( 0 ◦ , 0 ◦ ) followed by [33]. We conduct experiments to assess the impact of different initial starting points positions on the final performance. Results shown in Table 6 revels that the model's performance displays relatively minor fluctuations during longitudinal shifts, yet experiences a substantial decline when changing larger latitudes (from points less frequented by humans as starting positions). This divergence arises due to the model's misalignment with authentic browsing behaviors (humans tend to start their panoramic observations closer to the equator). It further emphasizes that simulating actual scanpaths assists in more accurate image quality evaluation.

Effectiveness of MFA and TMM. We exclude the multi-scale feature aggregation (MFA) and distortion-aware Block (DAB) to assess their effectiveness in MVAQD [18]. Specifically, for the previous experiment, we utilize the final stage of the Swin Transformer [23] for feature output. The results, shown in Table 8, indicate that MFA and DAB substantially contribute to the performance, with only a minor increase in the network parameters. The GRU [5] module in TMM is replaced with global average pooling (GAP) for predicting the sequence quality score. The effectiveness of the GRU module in representing temporal transition information is demonstrated in Table 8, and it can assist the model in predicting more accurate quality scores.

Excessive viewports hamper

performance

Excessive viewports hamper

performance

Impact of the number and length of the viewport sequence. We test N = 1 , 3 , 5 three different numbers of viewport sequences with varying sequence lengths on MVAQD [18] dataset. The findings, shown in Figure 4, reveal that as the sequence length increases, there is a decreasing trend in model performance across all three numbers of sequences. This suggests that an excessive number of viewports may introduce redundant information, potentially disrupting the training process of the network. Furthermore, our experiments demonstrate that incorporating multiple viewport sequences, the model can capture a broader range of perspectives of the scene, thereby better reflecting the rating process of ODIs and achieving improved robustness.

4.6 Proximate to Human-Observed Performance

We conduct a comparative analysis between the performance achieved using ground-truth (GT) sequences and pseudo sequences generated by RPS on the JUFE dataset [12]. In the JUFE dataset, the GT sequences are annotated based on whether they originate from good or bad starting points (details in Supplementary Materials). Therefore, for each starting point, we use RPS to generate those sequences with the same number and length of sequences compared to GT sequences. Then, we apply the generated sequences and GT sequences as the input of our proposed network. The results shown in Table 9 highlight a close gap between the contributions of GT sequences and our generated sequences. This result emphasizes the significance of proximity to human observation in enhancing the model's capabilities. Meanwhile, there is still a large exploration space for future methods to better incorporate human's sequences in OIQA task.

5 Conclusion

This paper introduces a novel multi-sequence network named Assessor360 for BOIQA based on a realistic assessment procedure. Specifically, we design Recursive Probability Sampling (RPS) to generate viewport sequences based on the semantic scene and the distortion. Additionally, we propose Multi-scale Feature Aggregation (MFA) with Distortion-aware Block (DAB) to combine distorted and semantic features of viewports. Temporal Modeling Module (TMM) is introduced to learn the temporal transition of viewports. We demonstrate the high performance of Assessor360 on multiple OIQA datasets and validate the effectiveness of RPS by comparing it with two advanced learning-based models used for scanpath prediction. Limitation is that the transition direction and distance in RPS are fixed, resulting in equally spaced distances between viewports. However, we have confidence that our analyses and the proposed pipeline can provide long-term valuable insights for future OIQA task.

Acknowledgments. This work was partly supported by the National Natural Science Foundation of China (Grant No. 61991451) and the Shenzhen Science and Technology Program (JSGG20220831093004008). The author would like to thank Xiangjie Sui at Jiangxi University of Finance and Economics for his inspiration.

References

Marc Assens Reina, Xavier Giro-i Nieto, Kevin McGuinness, and Noel E O'Connor. Saltinet: Scan-path prediction on 360 degree images using saliency volumes. In Proceedings of the IEEE International

Conference on Computer Vision Workshops , pages 2331-2338, 2017.

Sebastian Bosse, Dominique Maniry, Klaus-Robert Müller, Thomas Wiegand, and Wojciech Samek. Deep neural networks for no-reference and full-reference image quality assessment. IEEE Transactions on Image Processing , 27(1):206-219, 2017.

Meixu Chen, Yize Jin, Todd Goodall, Xiangxu Yu, and Alan Conrad Bovik. Study of 3d virtual reality picture quality. IEEE Journal of Selected Topics in Signal Processing , 14(1):89-102, 2019.

Sijia Chen, Yingxue Zhang, Yiming Li, Zhenzhong Chen, and Zhou Wang. Spherical structural similarity index for objective omnidirectional video quality assessment. In IEEE International Conference on Multimedia and Expo , pages 1-6, 2018.

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 , 2014.

Yasser Dahou, Marouane Tliba, Kevin McGuinness, and Noel O'Connor. Atsal: An attention based architecture for saliency prediction in 360 videos. In Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10-15, 2021, Proceedings, Part III , pages 305-320, 2021.

Xin Deng, Hao Wang, Mai Xu, Yichen Guo, Yuhang Song, and Li Yang. Lau-net: Latitude adaptive upscaling network for omnidirectional image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9189-9198, 2021.

Keyan Ding, Kede Ma, Shiqi Wang, and Eero P Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(5):2567-2581, 2020.

Yasser Abdelaziz Dahou Djilali, Kevin McGuinness, and Noel E O'Connor. Simple baselines can fool 360deg saliency metrics. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops , pages 3750-3756, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, and et al. Weissenborn. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Huiyu Duan, Guangtao Zhai, Xiongkuo Min, Yucheng Zhu, Yi Fang, and Xiaokang Yang. Perceptual quality assessment of omnidirectional images. In 2018 IEEE International Symposium on Circuits and Systems , pages 1-5, 2018.

Yuming Fang, Liping Huang, Jiebin Yan, Xuelin Liu, and Yang Liu. Perceptual quality assessment of omnidirectional images. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 580-588, 2022.

Yuming Fang, Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. Perceptual quality assessment of smartphone photography. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3677-3686, 2020.

Jun Fu, Chen Hou, Wei Zhou, Jiahua Xu, and Zhibo Chen. Adaptive hypergraph convolutional network for no-reference 360-degree image quality assessment. In Proceedings of the 30th ACM International Conference on Multimedia , pages 961-969, 2022.

Jinjin Gu, Haoming Cai, Chao Dong, Jimmy S Ren, and et al. Timofte. Ntire 2022 challenge on perceptual image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 951-967, 2022.

Nafiseh Jabbari Tofighi, Mohamed Hedi Elfkir, Nevrez Imamoglu, Cagri Ozcinar, Erkut Erdem, and Aykut Erdem. St360iq: No-reference omnidirectional image quality assessment with spherical vision transformers. arXiv , 2023.

Hao Jiang, Gangyi Jiang, Mei Yu, Ting Luo, and Haiyong Xu. Multi-angle projection based blind omnidirectional image quality assessment. IEEE Transactions on Circuits and Systems for Video Technology , 32(7):4211-4223, 2021.

Hao Jiang, Gangyi Jiang, Mei Yu, Yun Zhang, You Yang, Zongju Peng, Fen Chen, and Qingbo Zhang. Cubemap-based perception-driven blind quality assessment for 360-degree images. IEEE Transactions on Image Processing , 30:2364-2377, 2021.

Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5148-5157, 2021.

Hak Gu Kim, Heoun-Taek Lim, and Yong Man Ro. Deep virtual reality image quality assessment with human perception guider for omnidirectional image. IEEE Transactions on Circuits and Systems for Video Technology , 30(4):917-928, 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Shanshan Lao, Yuan Gong, Shuwei Shi, Sidi Yang, Tianhe Wu, Jiahao Wang, Weihao Xia, and Yujiu Yang. Attentions help cnns see better: Attention-based hybrid image quality assessment network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 1140-1149, 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10012-10022, 2021.

Daniel Martin, Ana Serrano, Alexander W Bergman, Gordon Wetzstein, and Belen Masia. Scangan360: A generative model of realistic scanpaths for 360 images. IEEE Transactions on Visualization and Computer Graphics , 28(5):2003-2013, 2022.

Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing , 21(12):4695-4708, 2012.

Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making a 'completely blind' image quality analyzer. IEEE Signal Processing Letters , 20(3):209-212, 2012.

Michael I Posner, Yoav Cohen, et al. Components of visual orienting. Attention and performance X: Control of language processes , 32:531-556, 1984.

Pratibha Sharma, Manoj Diwakar, and Niranjan Lal. Edge detection using moore neighborhood. International Journal of Computer Applications , 61(3), 2013.

Shuwei Shi, Qingyan Bai, Mingdeng Cao, Weihao Xia, Jiahao Wang, Yifan Chen, and Yujiu Yang. Regionadaptive deformable network for image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 324-333, 2021.

Shuwei Shi, Jinjin Gu, Liangbin Xie, Xintao Wang, Yujiu Yang, and Chao Dong. Rethinking alignment in video super-resolution transformers. Advances in Neural Information Processing Systems , 35:36081-36093, 2022.

Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Belen Masia, and Gordon Wetzstein. Saliency in vr: How do people explore virtual environments? IEEE Transactions on Visualization and Computer Graphics , 24(4):1633-1642, 2018.

Xiangjie Sui, Yuming Fang, Hanwei Zhu, Shiqi Wang, and Zhou Wang. Scandmm: A deep markov model of scanpath prediction for 360° images. IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Xiangjie Sui, Kede Ma, Yiru Yao, and Yuming Fang. Perceptual quality assessment of omnidirectional images as moving camera videos. IEEE Transactions on Visualization and Computer Graphics , 28(8):30223034, 2021.

Wanjie Sun, Zhenzhong Chen, and Feng Wu. Visual scanpath prediction using ior-roi recurrent mixture density network. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(6):2101-2118, 2019.

Wei Sun, Ke Gu, Guangtao Zhai, Siwei Ma, Weisi Lin, and Patrick Le Calle. Cviqd: Subjective quality evaluation of compressed virtual reality images. In 2017 IEEE International Conference on Image Processing , pages 3450-3454, 2017.

Wei Sun, Xiongkuo Min, Guangtao Zhai, Ke Gu, Huiyu Duan, and Siwei Ma. Mc360iqa: A multi-channel cnn for blind 360-degree image quality assessment. IEEE Journal of Selected Topics in Signal Processing , 14(1):64-77, 2019.

Yule Sun, Ang Lu, and Lu Yu. Weighted-to-spherically-uniform quality evaluation for omnidirectional video. IEEE Signal Processing Letters , 24(9):1408-1412, 2017.

Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 2555-2563, 2023.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing , 13(4):600-612, 2004.

Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 , volume 2, pages 1398-1402, 2003.

Chen Xia, Junwei Han, Fei Qi, and Guangming Shi. Predicting human saccadic scanpaths based on iterative representation learning. IEEE Transactions on Image Processing , 28(7):3502-3515, 2019.

Liangbin Xie, Xintao Wang, Shuwei Shi, Jinjin Gu, Chao Dong, and Ying Shan. Mitigating artifacts in real-world video super-resolution models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 2956-2964, 2023.

Jiahua Xu, Wei Zhou, and Zhibo Chen. Blind omnidirectional image quality assessment with viewport oriented graph convolutional networks. IEEE Transactions on Circuits and Systems for Video Technology , 31(5):1724-1737, 2020.

Mai Xu, Chen Li, Zhenzhong Chen, Zulin Wang, and Zhenyu Guan. Assessing visual quality of omnidirectional videos. IEEE Transactions on Circuits and Systems for Video Technology , 29(12):3516-3530, 2018.

Mai Xu, Yuhang Song, Jianyi Wang, MingLang Qiao, Liangyu Huo, and Zulin Wang. Predicting head movement in panoramic video: A deep reinforcement learning approach. IEEE Transactions on Pattern Analysis and Machine Intelligence , 41(11):2693-2708, 2018.

Li Yang, Mai Xu, Xin Deng, and Bo Feng. Spatial attention-based non-reference perceptual quality prediction network for omnidirectional images. In 2021 IEEE International Conference on Multimedia and Expo , pages 1-6, 2021.

Li Yang, Mai Xu, Yichen Guo, Xin Deng, Fangyuan Gao, and Zhenyu Guan. Hierarchical bayesian lstm for head trajectory prediction on omnidirectional images. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(11):7563-7580, 2021.

Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 1191-1200, 2022.

Yan Ye, Elena Alshina, and J Boyce. Jvet-g1003: Algorithm description of projection format conversion and video quality metrics in 360lib version 4. Joint Video Exploration Team , 2017.

Zhenqiang Ying, Haoran Niu, Praful Gupta, Dhruv Mahajan, Deepti Ghadiyaram, and Alan Bovik. From patches to pictures (paq-2-piq): Mapping the perceptual space of picture quality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3575-3585, 2020.

Matt Yu, Haricharan Lakshman, and Bernd Girod. A framework to evaluate omnidirectional video coding schemes. In 2015 IEEE International Symposium on Mixed and Augmented Reality , pages 31-36, 2015.

Vladyslav Zakharchenko, Kwang Pyo Choi, and Jeong Hoon Park. Quality metric for spherical panoramic video. In Optics and Photonics for Information Processing X , volume 9970, pages 57-65, 2016.

Chaofan Zhang and Shiguang Liu. No-reference omnidirectional image quality assessment based on joint network. In Proceedings of the 30th ACM International Conference on Multimedia , pages 943-951, 2022.

Lin Zhang, Ying Shen, and Hongyu Li. Vsi: A visual saliency-induced index for perceptual image quality assessment. IEEE Transactions on Image processing , 23(10):4270-4281, 2014.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 586-595, 2018.

Weixia Zhang, Kede Ma, Jia Yan, Dexiang Deng, and Zhou Wang. Blind image quality assessment using a deep bilinear convolutional neural network. IEEE Transactions on Circuits and Systems for Video Technology , 30(1):36-47, 2018.

Weixia Zhang, Guangtao Zhai, Ying Wei, Xiaokang Yang, and Kede Ma. Blind image quality assessment via vision-language correspondence: A multitask learning perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14071-14081, 2023.

Xuelei Zheng, Gangyi Jiang, Mei Yu, and Hao Jiang. Segmented spherical projection-based blind omnidirectional image quality assessment. IEEE Access , 8:31647-31659, 2020.

Wei Zhou, Jiahua Xu, Qiuping Jiang, and Zhibo Chen. No-reference quality assessment for 360-degree images by analysis of multifrequency information and local-global naturalness. IEEE Transactions on Circuits and Systems for Video Technology , 32(4):1778-1791, 2021.

Yu Zhou, Yanjing Sun, Leida Li, Ke Gu, and Yuming Fang. Omnidirectional image quality assessment by distortion discrimination assisted multi-stream network. IEEE Transactions on Circuits and Systems for Video Technology , 32(4):1767-1777, 2021.

Yufeng Zhou, Mei Yu, Hualin Ma, Hua Shao, and Gangyi Jiang. Weighted-to-spherically-uniform ssim objective quality evaluation for panoramic video. In IEEE International Conference on Signal Processing , pages 54-57, 2018.

Peer reviews:
comments: SUMMARY:
This paper introduces a BOIQA model called Assessor360, which takes into consideration the observer's browsing behavior. Specifically, RPS method is proposed to combine content and detailed information to generate multiple pseudo-viewport sequences from a given starting point. It also uses an MFA module with DAB to fuse distorted and semantic features. Finally, the viewport transition in the temporal domain is learned using TMM.

STRENGTHS:
+ The pseudo multi-seqeuence generation is an interesting idea to mimic human behaviors during quality assessment.
+ The combination of MFA and DAB is reasonable in theory, and effective in practice.
+ Rich experiments are conducted on multiple datasets.

WEAKNESSES:
- To implement the TMM module, alternative design choices may be explored, such as LSTM, Transformer decoder, etc.
- L180-L182, the claims that the first two stages are more sensitive to distortion patterns and the latter capture abstract features are not verified.

QUESTIONS:
It would be interesting to compare the proposed method with the latest language-guided IQA models CLIPIQA[2] and LIQE[3].

[1] Wang, J., Chan, K. C., & Loy, C. C. (2022). Exploring CLIP for Assessing the Look and Feel of Images. AAAI 2023.

[2] Zhang, W., Zhai, G., Wei, Y., Yang, X., & Ma, K. (2023). Blind image quality assessment via vision-language correspondence: A multitask learning perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14071-14081).

LIMITATIONS:
Not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript proposes a novel multi-sequence network for BOIQA, which is inspired by the actual multi-assessor omnidirectional image quality assessment and introduces a generalized recursive probability sampling method for BOIQA tasks. In addition, a multi-scale feature aggregation module with distortion-aware blocks is designed. Extensive experimental results demonstrate that the proposed method achieves good performance on multiple OIQA datasets. However, there are still some problems in this manuscript.\n\nSTRENGTHS:\nWith the development of VR-related techniques, there is an increasing need for omnidirectional image quality evaluation. However, the existing methods lack the modeling of the observer’s browsing process, causing the predicted score to be far from the human perceptual quality score. This paper proposes the first pipeline to simulate the authentic data scoring process in ODI quality assessment, and has certain innovation. Meanwhile, the logic of this manuscript is rigorous.\n\nWEAKNESSES:\nIn my opinion, the most contribution of this paper is the recursive probability sampling strategy, while other modules are common. And some details are missing as follows:\n1.\tThe input of the proposed model is in the form of sequences, but the Swin Transformer is for processing single images, hence how does the Transformer process multiple images in the sequence? \n2.\tIn the recursive probability sampling process, how is the starting point of each sequence selected?\n3.\tThe ODI’s quality is averaged by all predicted scores of each sequence, have you considered using weights to fuse these scores? How is the result?\n4.\tThere are few comparative methods and experiments for cross-dataset testing.\n\nQUESTIONS:\n1.\tThe proposed model uses a pre-trained Swin Transformer, but the paper does not specify which kind of training model it is, whether it is tiny, small, or base.\n2.\tIn the training phase, is the learning rate always kept at 1e-5?\n3.\tHow are images tested during the testing phase? While choosing different starting point of each sequence, will the result be different? If the result will be different, how to obtain the final score?\n4.\tThere is a typo, in lines 214 and 215, “SRCC and PLCC stand for Spearman rank-ordered correlation and Pearson linear correlation.” “coefficient” is missing. And the English usage and technical expression should be further polished.\n5.\tIt is suggested that the numbers of the figures and tables in the paper be sorted according to the order of introduction.\n\nLIMITATIONS:\nThe proposed RPS has a limitation: the transition direction and distance are fixed, resulting in equally spaced distances between viewports. Design a more effective generation method is author’s future work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
A blind quality assessment model is proposed for 360-degree images.

STRENGTHS:
1. The problem is interesting and practical.
2. The attempt to model the observer’s browsing process is a plus.
3. The results look promising.

WEAKNESSES:
1. The recursive probability sampling is a little ad hoc to the reviewer. It is mostly handcrafted, which is not optimized to mimic the observer's browsing process. This weakens the motivation of the proposed method.  
2. No validation set is used for the hyperparameter setting, making the results look like overfitting (surpassing full-reference models by a large margin).
3. Since no validate set is available, more cross-dataset results are necessary to justify the rationality of the proposed model design.
4. Two baselines with ERP and cube map projection as direct input should be compared.

QUESTIONS:
1. How to initialize the starting points (the total number and positions), and does the initialization affect the final performance?
2. How to interpret Fig. 4?

LIMITATIONS:
N.A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a novel blind omnidirectional image quality assessment framework based on the modeling of the observer’s browsing process, which is called Assessor360. In detail, Assessor360 designs a novel viewpoint sampling methods named “Recursive probability sampling (RPS)” based on content-guided-sampled probability and detailed-guided sampled probability. In addition, the authors utilize the Multi-scale Feature Aggregation (MFA) to fuse the shallow layers’ feature of Swin transformer which is enhanced by proposed distortion-aware block (DAB) and deep layer’s feature which can capture abstract feature.\n\nSTRENGTHS:\n1. The motivation is clear and the performance is promising.\n\nWEAKNESSES:\nIn general, novelty and insight of the paper is limited to some extent. Below are some questions:\n1.   The main contribution of Assessor360 is the human browsing related viewpoint sampling methods, which combine CSP and DSP. Why is sampling via the equator bias named content-guided? Generally speaking, it may be easier to understand if object information or semantic information is used as content guidance. In DSP, the author did not visualize the entropy map of pixels to verify the motivation of this detail-oriented sampling\n2.\tIn section3.3, the author did not explain how the semantic information was extracted? Or the deep layer’s feature of swin transformer can represent the semantic information? Without constraints on deep layer’s features or adding conditions, it is difficult to understand that deep features have extracted semantic information.\n3.\tRPS can be understood as a plug-and-play sampling method. Has the author tried other methods, such as VGCN, with the RPS sampling method to verify the effect?\n4.\tIt would be better for the authors to try and analysis some other time series modeling modules in TMM, such as transformer, etc.\n\nQUESTIONS:\nPlease refer above comments\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}