Semantic-Aware and Quality-Aware Interaction Network for Blind Video Quality Assessment

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-28 09:06:14

Abstract: Current state-of-the-art video quality assessment (VQA) models typically integrate various perceptual features to comprehensively represent video quality degradation.  These models either directly concatenate features or fuse different perceptual scores while ignoring the domain gaps between cross-aware features, thus failing to adequately learn the correlations and interactions between different perceptual features. To this end, we analyze the independent effects and information gaps of quality- and semantic-aware features on video quality. Based on an analysis of the spatial and temporal differences between two aware features, we proposed a semantic-**A**ware and quality-**A**ware **I**nteraction **Net**work (**A$^2$INet**) for blind VQA (BVQA). For spatial gaps, we introduce a cross-aware guided interaction module to enhance the interaction between semantic- and quality-aware features in a local-to-global manner. Considering temporal discrepancies, we design a cross-aware temporal modeling module to further perceive temporal content variation and quality saliency information, and perceptual features are regressed into quality score by a temporal network and a temporal pooling.  Extensive experiments on six benchmark VQA datasets show that our model achieves state-of-the-art performance, and ablation studies further validate the effectiveness of each module. We also present a simple video sampling strategy to balance the effectiveness and efficiency of the model. The code for the proposed method will be released.

Main paper: 
 introduction: 
 The goal of video quality assessment (VQA) is to enable the model to perceive the visual quality of videos and produce results consistent with human subjective opinions, making it a popular research topic in multimedia [21, 45] . Blind VQA (BVQA) models evaluate video quality in the absence of reference videos, so huge efforts for BVQA have been devoted and a variety of deep learning-based models have been proposed [47, 59] .Figure 1 : Visualize the spatial feature maps of semantic-and quality-aware, alongside the temporal distribution of features on two high-quality videos (HV1 and HV2) and two lowquality videos (LV1 and LV2). MOS is the mean opinion score, higher values mean better subjective visual quality. (a) Three representative frames, and the semantic-and quality-aware feature maps are presented for the frames boxed in red. We use ResNet-50 [13] pre-trained on the ImageNet dataset [7] and the KoNIQ-10k dataset [15] to generate the semantic-and quality-aware feature maps. (b) The temporal distribution of each aware feature. Frame entropy and frame score for each frame of the feature map are used to measure information content and image quality, respectively.Given that human judgments of video quality are influenced by multiple perceptual factors working in concert, recent top models [47, 56] that adopt multiple networks to extract perceptual features, resulting in superior performance over models using a single network [23, 53] . These models concatenate features or fuse quality scores from different branches to comprehensively represent video quality. However, domain gaps between different aware features remain under-studied in existing work, hindering the full utilization of the advantages of multiple features in VQA and further constraining the perceptual ability of the models.Subjective studies are instructive for the design of objective VQA models, and previous studies [43, 47] indicate that visual content and distortion artifacts play primary roles in human judgments of video quality. Visual content primarily pertains to content composition and motion information, which dominate human preferences for video content and are referred to as semantic-aware. Distortion artifacts such as spatial degradation and temporal flicker are introduced into video due to imperfections in capture equipment or processing algorithms, which is denoted as quality-aware. Taking two sets of videos with varying quality levels, as depicted in Figure 1 , as examples, we analyze the roles of semantic-and quality-aware features in VQA. Two aware features focus on distinct spatial perceptual information by comparing videos with varying qualities from Figure 1 (a). The semantic-aware features focus on objects and are robust to quality degradation, while the qualityaware features are sensitive to quality degradation and exhibit a stronger response in low-quality frames. Thus, combining two aware features spatially can better understand the quality degradation in context. Two aware features exhibit different temporal characteristics by observing Figure 1(b) . High-quality videos (blue and red solid curves) have slower fluctuations in temporal content information and higher frame-level quality. The opposite is true for low-quality videos (green and yellow dashed curves). Temporal content variations and overall quality emerge as key factors in discerning video quality degradation. In summary, crossaware features exhibit perceptual gaps across spatial and temporal dimensions. We argue that simply concatenating features makes it difficult to comprehend the intricate connections between different perceptual features and video quality.To address these challenges, we propose an effective and efficient semantic-Aware and quality-Aware Interaction Network (A 2 INet) for BVQA. To mitigate representation differences and enhance spatial perception, we propose a cross-aware guided interaction (CAGI) module that uses cross-aware guided instance normalization (CGIN) to perceive the gaps between frame-level features with semanticand quality-aware features, and then implements global interaction through a global self-attention layer. Motivated by Figure 1 (b), we design a cross-aware temporal modeling (CATM) module by capturing information with significant quality degradation and content variation to enhance the perception for temporal distortion. Finally, the video quality score is obtained through temporal modeling and temporal pooling for the perceptual features. We conduct extensive experiments to verify that the proposed BVQA model achieves stateof-the-art (SOTA) results on the LIVE-Qual [11] , CVD2014 [33] , LIVE-VQC [37] and KoNViD-1k [14] datasets, with respective improvements of 7.25%, 2.43%, 4.92% and 3.95% in Spearman rank order correlation coefficient (SRCC). The main contributions of this paper are as follows, 1) As far as we know, this is the first attempt to explore differences in various perceptual features and employ cross-aware feature learning to enhance the perceptual ability of models for BVQA.2) We design a CAGI module comprising a CGIN and a global self-attention layer, it uses a local-to-global manner to facilitate interaction on two aware features.3) We introduce a CATM to perceive temporal distortions from the perspectives of quality saliency and content variation. 4) Extensive experiments have verified the advantages of the proposed BVQA model, and ablation studies have demonstrated the effectiveness of each module. 
related work: 
 The success of the BVQA model hinges on effective perceptual feature extraction and temporal modeling to represent video quality degradation across spatial and temporal dimensions.BVQA models based on hand-crafted features. Classic BVQA methods [6, 19, 25, 32, 35, 40] are designed to use hand-crafted features for describing visual quality degradation. Nevertheless, these hand-crafted features only emphasize low-level edges and texture information, falling short of capturing high-level semantic information. Subsequent work [20, 41] combined hand-crafted features with semantic features extracted from pre-trained ResNet-50 [13] models, yielding promising results.BVQA models based on deep features. Deep learning-based models are further divided into fixed backbone [22, 58] and endto-end training [44, 59] methods. Fixed backbone-based methods extract features from video using feature extractors pre-trained on other tasks. Li et al. [23, 24] used a pre-trained ResNet-50 to extract content-aware features. Related researches [3, 26] follow the work of [23, 24] using a single 2D network to capture features. Madhusudana et al. [31] proposed self-supervised learning to train the feature encoder. Many researches [22, 43, 51, 52, 58] have designed multiple backbone networks and concatenated features to capture various information from distorted videos, and obtained better results than a single network. Zhang et al. [56] incorporated five features related to visual perception and achieved SOTA performance.For end-to-end training methods, early models [4, 53] utilized single convolutional neural networks (CNN) to extract perceptual features. Recently, Wu et al. [44, 45] proposed FAST-VQA, which takes video fragments as inputs and trains variants of the video swin Transformer tiny (SwinT-3D) [29] for BVQA. DisCoVQA [46] trained SwinT-3D with sparse frames as input. Yuan et al. [55] designed a video transformer with a multi-path temporal network and sparse attention blocks for capturing different distortions. Sun et al. [38] fine-tuned ResNet-50 and incorporated motion features to represent video quality. ZoomVQA [59] trained the image quality assessment (IQA) and VQA branches separately. Wu et al. [47] used inflated-ConvNext [28] and FAST-VQA models to perceive the quality of aesthetic and technical perspectives, obtaining video quality scores through simple weighted fusion.Overall, existing models usually concatenate features or fuse branch scores to predict video quality. We analyze spatial gaps in different aware features to narrow the feature gaps and enhance mutual perception through a CAGI module. Since this paper focuses on exploring more effective ways of combining different aware features for BVQA rather than feature extraction, the proposed model is designed as a fixed backbone method.Temporal distortion primarily occurs in the form of flicker, jitter, and scene transitions, leading to video quality degradation. Previous studies have shown that temporal modeling is crucial for BVQA.Li et al. [23, 24] modeled temporal relationships frame-by-frame using gated recurrent units (GRUs) [5] , employing both min pooling and soft-weighted average pooling to aggregate frame-level scores. [22, 56] referenced this temporal modeling and temporal pooling method. Chen et al. [4] introduced multi-level GRUs to fuse motion information of different frequencies. Chen et al. [3] designed a pyramid temporal aggregation module that fuses shortterm and long-term memory of frame-level features. Telili et al. [39] utilized bidirectional long short-term networks (Bi-LSTM) to capture temporal correlations between the previous and next frames. Ying et al. [52] used InceptionTime [17] for temporal modeling due to its faster and easier training. Li et al. [26] proposed a hierarchical Transformer that integrates frame-level and clip-level quality to derive video-level quality scores by stacking several divide and conquer Transformer layers. Wu et al. [46] designed a temporal content Transformer to learn the relationships among frame contents. In addition to modeling the long-term dependencies of video frames through temporal networks, some related works [22, 38, 44, 46-48, 58, 59 ] employed 3D networks, such as SlowFast [9] , Swin-Transformer (Swin-3D) [29] and TimeSformer [1] , to extract local spatio-temporal information.In general, an effective temporal modeling module assists the model in extracting long-term dependencies and perceiving temporal distortions in videos. Existing models usually perform temporal modeling on concatenated features and do not fully exploit the gaps in representing video quality over time among different features. Based on the analysis of Figure 1 (b), we propose the CATM module to perceive the temporal content variation and quality saliency information of the video, which fully considers the temporal characteristics of two aware features in videos with varying quality. 
proposed method: 
 Figure 2 depicts the framework of the proposed semantic-aware and quality-aware interaction network (A 2 INet) for BVQA. The distorted video is first inputted into a feature extraction module to extract spatial quality-aware and semantic-aware features (Section 3.1). Subsequently, the cross-aware guided interaction (CAGI) module is employed to perceive local perceptual gaps between the two aware features and achieve global interaction (Section 3.2). Next, the fused features are fed into a cross-aware temporal modeling (CATM) module to further capture quality-aware and semanticaware long-term dependencies, and aggregate frame-level features through temporal pooling to estimate the video-level quality score (Section 3.3). Each module is introduced in detail below.Based on previous subjective experiments regarding human preferences in visual quality [43, 47, 48] , we extract spatial quality-aware features and spatial (or motion) semantic-aware features to represent video quality. The proposed model is designed a dual-branch architecture, comprising a quality-aware network and a semanticaware network, for extracting two aware features. A distorted video V = {V (ℎ, 𝑤, 𝑡)} ∈ R 𝐻 ×𝑊 ×𝑇 is treated as a collection of framelevel images, where 𝐻 × 𝑊 represents the spatial resolution of the video, and 𝑇 is the video length.We employ an IQA model [57] pre-trained on multiple IQA datasets as the quality-aware network. Given that the visual perception system is a hierarchical structure [27, 49] , features 𝑭𝑞1 𝑡 ∈ R 64× 𝐻 2 × 𝑊 2 , 𝑭 𝑞2 𝑡 ∈ R 128× 𝐻 4 × 𝑊 4 , 𝑭 𝑞3 𝑡 ∈ R 256× 𝐻 8 × 𝑊 8 and 𝑭 𝑞4 𝑡 ∈ R 512× 𝐻 16 × 𝑊16 are extracted from four bottlenecks of the qualityaware network, where 𝑡 represents the 𝑡-th frame. Then, the spatial global average pooling and global standard deviation pooling are applied for the features of each bottleneck. The pooled features are concatenated into features 𝒇 𝑞 𝑡 ∈ R 1×1920 at the 𝑡-th frame, and the quality-aware features of the distorted video are represented as features𝑭 𝒒 = 𝒇 𝑞 𝑡 ∈ R 𝑇 ×1920 .For the other branch, we use the pre-trained ResNet-50 [13] on the ImageNet dataset [7] as the semantic-aware network. Similarly, we extract semantic-aware features from the distorted video, denoted as 𝑭 𝑠 = 𝒇 𝑠 𝑡 ∈ R 𝑇 ×7680 . To align the feature dimensions of the two aware features, we pass the features both through a fully connected (FC) layer and obtain features 𝑭 𝑠 𝑓 and 𝑭 𝑞 𝑓 ∈ R 𝑇 ×𝐷 .We propose the CAGI, which comprises a cross-aware guided instance normalization (CGIN) and a global self-attention layer, to perceive local gaps and achieve global interaction between features. 1) CGIN. An adaptive instance normalization is proposed for style transfer [16] by combining the mean and variance of the features from content image 𝑰 𝑋 with the features from style image 𝑰 𝑌 , which is defined asEQUATIONwhere 𝑿 and 𝒀 ∈ R 𝐵×𝐶 ×𝐻 ×𝑊 are the features of input images 𝑰 𝑋 and 𝑰 𝑌 respectively, 𝐵, 𝐶, 𝐻 , and 𝑊 represent the batch size, the feature channel, the height, and width of feature map, respectively, 𝜇 (•) and 𝜎 (•) calculate across spatial dimensions independently for each channel and each sample of 𝑿 and 𝒀 [10, 30] .Inspired by this module, we use feature guided instance normalization (FGIN) to interact the cross-aware features, FGIN (𝑿, 𝒀 ) = 𝛾 𝑠 (𝒀 )𝑿 -𝜇 (𝑿 )EQUATIONwhere 𝛾 𝑠 (•) and 𝜃 𝑠 (•) denote both an FC layer, 𝛾 𝑠 (𝒀 ) and 𝜃 𝑠 (𝒀 ) are treated as affine parameters to scale and shift the normalized features 𝑿 , thereby facilitating the interaction between the two features. However, the process of deforming 𝑿 to 𝒀 by Eq. (2) leads to the loss of the original information of 𝑿 . For this reason, we further modify Eq. (2), as followsEQUATIONwhere 𝛾 1 is a constant and is set to 1. As shown in Eq. (3), we add normalized features 𝑿 in Eq. (3), which prevents the network from losing information 𝑿 during feed-forward. Finally, the proposed CGIN is formulated as follows,CGIN 𝑭 𝑞 𝑓 , 𝑭 𝑠 𝑓 = 𝑓 𝑭 𝑞 𝑓 , 𝑭 𝑠 𝑓 ⊕ 𝑓 𝑭 𝑠 𝑓 , 𝑭 𝑞 𝑓 = 𝑭 𝑞𝑠 ⊕ 𝑭 𝑠𝑞 (4)We narrow the perceptual gaps between the cross-aware features by deforming them towards each other, and then concatenate 𝑭 𝑞𝑠 and 𝑭 𝑠𝑞 to form the feature 𝑭 𝑝𝑒 ∈ R 𝑇 ×2𝐷 .2) Global self-attention layer. After perceiving local gaps on cross-aware features, a global self-attention layer [42, 50] is designed to enhance the global interaction between frames.Similar to the self-attention mechanism, the feature 𝑭 𝑝𝑒 generates query 𝑭 𝑝𝑒 𝑄 , key 𝑭 𝑝𝑒 𝐾 and value 𝑭 𝑝𝑒 𝑉 ∈ R 𝑇 ×2𝐷 matrices through three linear projections 𝑾 𝑄 , 𝑾 𝐾 and 𝑾 𝑉 ∈ R 2𝐷 ×2𝐷 to encode temporal information of the 𝑭 𝑝𝑒 ,EQUATIONwhere 𝑀 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 (•) means a Softmax function, "T" represents the transpose operator. The values of 𝑭 𝑝𝑒 𝑎𝑚 reflect the correlation of two elements between frames. Finally, the 𝑭 𝑝𝑒 𝑎𝑚 is directly multiplied by the 𝑭 𝑝𝑒 𝑉 , and 𝑭 𝑝𝑒 with residual links [42] is added to obtain the feature 𝑭 𝑒 .The CATM module is designed to capture temporal distortions associated with video quality degradation and to regress features into video-level quality scores.Previous research has shown that frames with severe quality degradation may have a greater impact on the quality of the whole video [54] . Additionally, factors such as camera shake and scene transitions [46] also influence video quality. By integrating these theoretical findings with the insights from Figure 1 , we develop the quality saliency perception and content variation perception blocks to enhance the perception of temporal quality, as depicted in Figure 3 . Before input to the CATM module, feature 𝑭 𝑒 is reduced to the 𝐷-dimension through an FC layer, and maximum pooling and standard deviation pooling are utilized to aggregate the saliency and variation information of feature 𝑭 𝑒 along the temporal dimension. Subsequently, features 𝑭 𝑒 𝑚𝑎𝑥 and 𝑭 𝑒 𝑠𝑡𝑑 ∈ R 1×𝐷 are fed into a lightweight network that consists of two FC layers, one ReLU function and one Sigmoid function to obtain features 𝑭 𝑚𝑎𝑥 and 𝑭 𝑠𝑡𝑑 . Then, 𝑭 𝑒 is multiplied by 𝑭 𝑚𝑎𝑥 and 𝑭 𝑠𝑡𝑑 after passing through a temporal convolution layer to derive the temporal quality saliency and content variation information, respectively. Finally, 𝑭 𝑞𝑡 and 𝑭 𝑠𝑡 are concatenated into 𝑭 𝑧 ∈ R 𝑇 ×2𝐷 and input to the temporal network to build long-term dependencies.Similar to previous work [23] , we first crop the feature 𝑭 𝑧 by using a FC layer, and the reduced featureF 𝑧 = f𝑧 𝑡 ∈ R 𝑇 ×128 is obtained. Then, the featureF 𝑧 is input into GRUs to capture long-term dependencies. And a FC layer is used to regress the output features 𝑯 𝑡 = {𝒉 𝑡 } ∈ R 𝑇 ×32 of GRUs into frame-level scores 𝒒 = {𝑞 𝑡 } ∈ R 𝑇 ×1 . We adopt a subjectively-inspired temporal pooling strategy [36] to integrate frame-level scores 𝒒 to a videolevel score 𝑄 𝑝 .1) Sampling strategy. Existing fixed backbone-based methods usually extract features for full-resolution videos, resulting in models with high complexity that increases with the resolution of the video. In this end, we propose a simple video sampling strategy based on the characteristics of semantic-aware features and qualityaware features to balance the effectiveness and efficiency of the models. On one hand, as observed from Figure 1 (a), semantic-aware features focus on object and contextual semantics but has little effect on quality degradation. On the other hand, quality-aware features are sensitive to local quality degradation but lacks understanding of global semantic content. Thus, we resize each frame to preserve the original global semantics and randomly crop each frame into four 768×768 patches to ensure the original local quality. The resized frame is set to 𝑚𝑖𝑛 (𝐻,𝑊 ) = 540 while maintaining the aspect ratio, and we strictly aligned the sample areas to ensure raw temporal variations during patch sampling [18, 44] , where 𝑚𝑖𝑛 (•) denotes the minimum operation. Finally, the imresized video and the cropped video are fed into the semantic-aware and qualityaware networks, respectively. And, the features of four patches are averaged as frame-level quality-aware features.2) Optimization. During training, the proposed model is optimized using mean absolute error (MAE) loss L 𝑀 and rank loss L 𝑅 as objective functions. The MAE loss measures the distance between the predicted score 𝑄 𝑝 and the mean opinion score (MOS) 𝑄 𝑚 , denoted as:EQUATION)where 𝑖 represents the 𝑖-th video from the mini-batch. The differentiable rank loss function is calculated as:EQUATIONwhere Q𝒔 𝑖 𝑗 = 𝑄 𝒔 𝑖 -𝑄 𝒔 𝑗 , 𝒔 ∈ {𝑚, 𝑝}, and 𝜙 (•) is defined as :EQUATIONFinally, the training loss function is represented by:EQUATION)where 𝜂 is the parameter used to balance the two losses. 
experiments 4.1 experimental settings: 
 1) Implementation details. In the experiment, the dimension parameter 𝐷 is set to 1024, and the balance parameter 𝜂 is set to 1.During training, we freeze the weights of the quality-and semanticaware networks. To compare with the VQA model based on the 3D network, we replace the spatial semantic-aware network with the SlowFast [9] pre-trained on the action recognition dataset [2] as the motion semantic-aware network. Only the fast features from the last layer of the network are used as motion semantic features, with the feature dimension of 𝑭 𝒔 being 512. As a distinction, we use S+S to represent the combination of spatial quality-aware and spatial semantic-aware, and S+M to represent the combination of spatial quality-aware and motion semantic-aware. Moreover, we set up two video input modes, full resolution and preprocessing, which are abbreviated as Proposed FR and Proposed PR. For Proposed FR, we take the whole video as the input. For Proposed PR, we sample 128 frames from the video at the same interval and process them as input by the preprocessing described in Section 3.4. The experiments are conducted on PyTorch [34] with single RTX 3090 GPU. The batch size is set to 8, and Adam optimizer with an initial learning rate of 2 × 10 -5 is used for training learnable parameters.2) Compared methods. The performance of the proposed model is compared with sixteen VQA models, including two models combining hand-crafted and deep features (RAPIQUE [41] and CNN-VQM [20] ), and fourteen deep learning-based VQA models (including eight fixed backbone models, VSFA [23] , GSTVQA [3] , CoINVQ [43] , PVQ [52] , DCVQE [26] , Li et al. [22] , CONVIQT [31] and HVS-5M [56] , and six end-to-end training models, FAST-VQA [44] , SimpleVQA [38] , FasterVQA [45] , ZoomVQA [59] , DisCoVQA [46] and VQT [55] ). Note that all models are run with the source code released by the authors and are not trained with additional VQA datasets. Two RTX 3090 GPUs are used for training HVS-5M [56] , FAST-VQA [44] , FasterVQA [45] , ZoomVQA [59] and SimpleVQA [38] . 3) Six benchmark VQA datasets. Six VQA datasets with mean opinion scores (MOS) are used as benchmark datasets to test the performance of different models, including five small-scale datasets (CVD2014 [33] , LIVE-Qual [11] , LIVE-VQC [37] , YT-UGC [43] and KoNViD-1k [14] ) and one large-scale dataset (LSVQ [52] ), their details are listed in Table 1 . The LSVQ dataset comprises a train subset (LSVQ train ), and two test subsets (LSVQ test and LSVQ 1080p ), containing 28056, 7182 and 3573 videos, respectively. For YT-UGC dataset, we follow [22] and 1142 videos are selected for experiments.4) Evaluation criteria. For each VQA dataset, we follow the settings of [38, 44] and the videos in each dataset are partitioned into training and test sets. Specifically, we train each model on the training set and verify the performance of the model on the test set. Pearson linear correlation coefficient (PLCC) and Spearman rank-order correlation coefficient (SRCC) are used as evaluation criteria to quantify the correlation between predicted scores and subjective judgments (i.e., MOS values), where SRCC reflects the monotonicity of BVQA models, and PLCC is utilized to evaluate the accuracy of BVQA models. Following [12] , a nonlinear logistic function maps the predicted scores to the same scale as the MOS before calculating PLCC. The train-test splits are repeated ten times to avoid performance bias, and the median results are reported.Table 2 presents the comparison results of the proposed models and sixteen BVQA models on five small-scale datasets. In Table 2 , the weighted performance of the proposed FR(S+M) model surpasses that of the SOTA model [56] by 2.91% and 3.39% in terms of PLCC and SRCC, respectively. The proposed FR(S+M) model outperforms prior fixed backbone methods on four small-scale datasets, achieving 7.25%, 2.43%, 4.92% and 3.95% improvements over the second-best results in terms of SRCC on the LIVE-Qual, CVD2014, LIVE-VQC, and KoNViD-1k datasets. It is worth mentioning that the proposed FR(S+S) model, which does not consider local spatio-temporal information, still shows competitiveness. The Table 2 : Results on the five benchmark VQA datasets. "F" and "B" represent feature type and backbone network, "H" and "D" stand for hand-crafted and deep features, "2D" and "3D" denote 2D and 3D backbone networks. "W.A." shows the weightedaverage performance over all datasets, and weights are proportional to database-sizes. The best and second best results for fixed backbone methods are highlighted in red bold and blue bold respectively. The best result for end-to-end training methods is underlined. Italics indicate data sourced from original references. "-" indicates that the results are not available. proposed FR(S+S) model improves the LIVE-Qual, CVD2014 and KoNViD-1k datasets by 6.45%, 1.65% and 1.84% in terms of SRCC, respectively. One possible reason for the weaker performance of the proposed FR(S+S) model on the LIVE-VQC dataset is that local spatio-temporal information plays a significant role in this dataset. Despite freezing the feature extractors, the weighted performance of our proposed models still outperforms that of all end-to-end training methods. In summary, the proposed model achieves SOTA performance by analyzing domain gaps across aware features and enhancing the perception of two aware features in space and time through CAGI and CATM, rather than directly concatenating features. Furthermore, the weighted performance of the proposed PR models is only slightly lower than that of the proposed FR models. We will further compare their complexity in Section 4.6.In addition to analyzing the superior performance of the proposed model, we draw several other conclusions. First, the performance of models based on 3D feature extractors is superior to that of models based on 2D feature extractors, suggesting that local spatiotemporal information is more crucial than spatial features for VQA tasks. Although models [3, 23, 26] temporally model frame-level features via temporal network, the separate extraction of spatial and temporal features fails to capture intrinsic spatio-temporal information. Second, models based on multiple feature extractors outperform those based on a single feature extractor. The utilization of multiple feature extractors enables the capture of various perceptual features, providing a richer representation for video quality degradation and yielding superior performance. Lastly, the performance of fixed backbone-based methods is comparable to that of end-to-end training methods in small-scale datasets. End-to-end training-based methods fine-tune the backbone network to extract features related to visual quality while using only part of the video data (clips or sparse frames) as input. Fixed backbone-based methods use feature extractors pre-trained from other tasks, yet they have access to more comprehensive video data. As a result, both methods have similar prediction performance on small datasets.Next, we conduct intra-and cross-dataset validation to verify the generalization ability of the proposed model. The models are trained on the LSVQ train subset and validated on the LSVQ test and LSVQ 1080p (intra-dataset validation). For cross-dataset validation, the optimal trained model is tested on LIVE-VQC and KoNViD-1k datasets. We choose VSFA [23] GSTVQA [3] , PVQ [52] , Li et al [22] , CON-VIQT [31] and HVS-5M [56] as comparison models, as they freeze the backbone network during training. Table 3 presents the comparative results. The proposed PR(S+M) and FR(S+M) models have better generalization performance. Although the proposed FR(S+M) model has poor cross-dataset validation performance on the LIVE-VQC dataset, it demonstrates excellent performance in both intra-and cross-dataset validation. The proposed PR(S+S) and FR(S+S) models outperform all VQA models based on the 2D backbone network, and even surpass PVQ [52] , which includes a 3D feature extractor. The performance of the proposed PR(S+S) even outperforms the proposed FR(S+S) with performance by training on a large amount of video data on the LSVQ test dataset. Overall, the proposed models have good generalization performance.To illustrate the correlation between predicted scores with subjective scores (i.e. MOS), we first visualize the scatter plots of the proposed FR (S+S) and PR (S+M) models on the LSVQ test dataset in Figure 4 . We observe that most of the scatter points cluster around the red line, indicating that the scores predicted by the models are consistent with the subjective scores.Then, we further demonstrate one successful and one failed video prediction case of the proposed FR (S+M) model in Figure 5 . The proposed model accurately predicts video scenes with obvious semantic information but fails to predict video examples where the semantic content is not apparent. One possible explanation is that the proposed method utilizes a fixed backbone network to extract semantic features and relies on the presence of specific semantic scenarios. Overall, the quantitative results illustrate the effectiveness of the proposed model and provide inspirations for us to better handle these scenarios in the future. In this section, the whole video is utilized as input for ablation experiments to assess the effectiveness of each module. Ablation on quality-aware and semantic-aware features. To quantitatively analyze the independent performance and mutual gains of quality-aware and semantic-aware features, we report the PLCC and SRCC results of spatial quality-aware, spatial semanticaware, motion semantic-aware features and their combinations in Table 4 . Note that all features are fed directly into a temporal network and a temporal pooling as described in Section 3.3. First, we found that the quality-aware feature typically takes a prominent role in VQA. Second, combining different aware features brings gains and improves the prediction performance of the model. Moreover, the combinations of quality-aware with semantic-aware (spatial or motion) yield superior results compared to combining two semantic-aware features, owing to the fact that they have more complementary information in distorted videos.Table 4 : Ablation experiments on spatial quality-aware (SQ), spatial semantic-aware (SS), motion semantic-aware (MS) features and combinations (+) using three small-scale datasets. We have respectively bolded the best-performing features and combinations. Ablation on CAGI module. We compare three schemes that combine quality-aware features with semantic-aware features, including direct concatenation (Concat), concatenation followed by multilayer perceptron fusion (Concat+MLP), and the CAGI module. Among them, the MLP consists of an FC layer, ReLU function, and dropout layer. Then, the fused features are input into an FC layer and a temporal network as described in Section 3.3. For a full comparison, we use variants of the CNN and Transformer architectures to extract each perceptual feature and compose eight qualitysemantic aware combinations for analysis. Table 5 presents the performance comparison of the three schemes. Intuitively, we observe that each feature combination through CAGI exhibits higher performance than direct concatenation and is more efficient than simple MLP fusion, proving that CAGI effectively enhances the correlation between the two aware features. Furthermore, CAGI is compatible for most dual-branch architectures.Ablation on CGIN. We further investigate the impact of three instance normalizations on the proposed model, including adaptive instance normalization (AdaIN) [16] , feature guided instance normalization (FGIN) [10] , and the proposed CGIN. The comparison results are listed in Table 6 . From Table 6 , we observe that the proposed CGIN obtains higher prediction accuracy, which implies that CGIN is more helpful for the interaction of two aware features. 7 lists the performance results of different model designing on three datasets. Based on the qualitative and quantitative analysis of the two aware features, we meticulously designed each module, and Table 7 proves that each module has certain gains for the proposed model.In this subsection, we compare the FLOPs and running times on GPU of the proposed models with existing models, and plot the performance curves of FLOPs and running times on videos with different resolutions in Figure 6 . Average results of ten video samples (10 seconds, 30 frames per second) as final test time, and experiments are executed on a single 3090 GPU. It can observe that the FLOPs and running time of the existing models increase with the rise in video resolution. Although HVS-5M [56] exhibits good prediction performance, the computational complexity is unacceptable. When video resolution exceeded 540P, the HVS-5M run for more than 100s and 2160P video could not be tested on a single 3090 GPU (540P, 134.54s; 720P, 224.08s; 1080P, 388.82s; 1440P, 687.7s). The proposed FR(S+S) and FR(S+M) models achieve SOTA performance with computational complexity between VSFA [23] and GSTVQA [3] . The proposed PR(S+S) and PR(S+M) models reduce redundant information from the input video through a simple sampling strategy (as described in Section 3.4), achieving the lowest computational complexity while maintaining performance with the whole video as input (see Table 2 ).  
conclusion: 
 In this paper, we have presented the semantic-Aware and quality-Aware Interaction Network (A 2 INet) for blind video quality assessment (BVQA). Based on the analysis of the domain gaps between semantic-and quality-aware features, we design a cross-aware guided interaction module to enhance the interaction between the two aware features and propose a cross-aware temporal modeling module to perceive temporal distortion from the perspectives of content variation and quality saliency. Experimental results show that the proposed model outperforms the state-of-the-art performance on six benchmark VQA datasets, and ablation studies further verify the effectiveness of each component. Additionally, we demonstrate a simple video sampling strategy to balance the effectiveness and efficiency of the proposed model.  
 
 Appendix:  


Decision: All reviewers appreciated clarifications in the authors' rebuttal, and the paper is now more toward "borderline accept". The main remaining concern is about the novelty of the approach leading to "borderline accept".

Reviews: review: paper_summary: paper_summary: The paper introduces A2INet, a semantic-Aware and quality-Aware Interaction Network for blind video quality assessment (BVQA). It addresses the domain gaps between semantic and quality features by using a cross-aware guided interaction module and a cross-aware temporal modeling module. A2INet enhances feature interaction and models temporal distortions to improve BVQA.
strength_weakness: strength_weakness: 1. The logic is clearly expressed, facilitating understanding.

2. The experimental results indeed demonstrate some of the article's advantages.
limitations: limitations: 1. I am concerned about the role of semantic features, as this might influence the genuine evaluation of quality features; I think we should assess the various roles played by quality information and semantic information before passing through the Cross-aware Guided Instance Normalization module.  If there is any misunderstanding, I welcome the authors to refute it.

2. Does the module's outcome solely focus on global features while neglecting local ones?

3. The majority of the formulas are missing punctuation.

4. Wherein lie the advantages of using semantic features in conjunction with quality-aware features, as opposed to interacting with global and local features?

5. What are the advantages of using a cross-perception guided interaction module, as claimed by the authors, compared to this method[1] of training the interaction between semantic-aware and quality-aware features using contrastive learning?

6. Please analyze the computational complexity and the resulting computational cost of the proposed method.

[1] Zhao K, Yuan K, Sun M, et al. Quality-aware pre-trained models for blind image quality assessment[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 22302-22313.
paper_summary: paper_summary: This paper explored the interaction between semantic awareness and quality awareness, and proposed a new BVQA model that combines the advantages of both types of features.
strength_weakness: strength_weakness: This paper effectively integrates semantic-aware and quality-aware features, addressing the common issue of simply concatenating them. The experiments are ample and rigorous, validating the proposed model's effectiveness. Most importantly, it reduces computational complexity.
limitations: limitations: 1. Line 130-133 “High-quality videos (blue and red solid curves) have slower fluctuations in temporal content information ...... The opposite is true for low-quality video”. This poses a problem. As illustrated in Fig.1, there is a high frame-to-frame similarity in the two high-quality videos, leading to minimal entropy fluctuations. In contrast, the two low-quality videos show considerable motion, resulting in more pronounced entropy fluctuations. These fluctuations seem more tied to the video content rather the video quality. While I concur with the conclusion that temporal content variations and overall quality are critical in identifying video quality degradation (line 134), the provided examples are inappropriate.  In fact, using the videos in supplementary material Fig.3, the HV1 and the LV3 better support the conclusion.
2. The paper frequently mentions "narrowing representation differences." (line 142, 218, 377 etc.). However, semantic-aware and quality-aware features are inherently different by nature and are supposed to be different. What we need to do is combine the different advantages of these two types of features (which is indeed what the authors have done), rather than narrowing their differences. Please consider revising the wording.
3. In Fig. 1, Fst and Fqt are reversed. Additionally, the figure shows Fst and Fqt being added element-wise, whereas the text on line 435 mentions “concatenation”, which is inconsistent. Besides, it is recommended to place the overall output Q of the model outside the module.
4. The blue-green blocks in Fig. 3 is not labeled.
5. The full name of the model is needed only once in the abstract and the body of the text. Line 281and line 916 are redundant.
6. Fig.4, as well as the Fig.3 in supplementary material could be better with scatterplots fit lines’ slopes and intercepts, to illustrate the performance differences across datasets.
paper_summary: paper_summary: By analyzing the independent impact and information gap of quality and semantic perception features on video quality, this paper proposes a semantic perception and quality perception interaction network for blind video quality assessment (BVQA).
strength_weakness: strength_weakness: The method proposed in this paper is relatively novel and technically correct.
limitations: limitations: 1. The paper has formatting errors. For example, the Tables and Figures should be placed at the top.
2. Please demonstrate the prediction performance of the method that combines spatial quality aware, spatial semantic aware, and motion semantic aware.
3. What are the preprocessing operations mentioned in the implementation details? For the sake of fairness, the preprocessing of the input of the comparison method should also be consistent with the preprocessing of the input of the proposed method.
4. Add the experimental results of the end-to-end comparison methods in Table 3.
5. Add ablation experiments on the loss function.