CLiF-VQA: Enhancing Video Quality Assessment by Incorporating High-Level Semantic Information related to Human Feelings

Authors: yachun  mi
yan  shu
yu  li
chen  hui
puchao  zhou
shaohui 2024 liu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 08:59:08

Abstract: Video Quality Assessment (VQA) aims to simulate the process of perceiving video quality by the Human Visual System (HVS). Although subjective studies have shown that the judgments of HVS are strongly influenced by human feelings, it remains unclear how video content relates to human feelings. The recent rapid development of Vision-Language pre-trained models (VLM) has established a solid link between language and vision. And human feelings can be accurately described by language, which means that VLM can extract information related to human feelings from visual content with linguistic prompts. In this paper, we propose CLiF-VQA, which innovatively utilizes the visual linguistic capabilities of VLM to introduce human feelings features based on traditional spatio-temporal features to more accurately simulate the perceptual process of HVS. In order to efficiently extract features related to human feelings from videos, we pioneer the exploration of the consistency between Contrastive Language-Image Pre-training (CLIP) and human feelings in video perception. In addition, we design effective prompts, i.e., a variety of objective and subjective descriptions closely related to human feelings, as prompts. Extensive experiments show that the proposed CLiF-VQA exhibits excellent performance on several VQA datasets. The results show that introducing human feelings features on top of spatio-temporal features is an effective way to obtain better performance.

Main paper: 
 introduction: 
 With the rapid advancement of technology, the threshold of video production has been significantly lowered, enabling an increasing number of users to create and upload videos to various online platforms. However, User-Generated Content (UGC) videos often have annoying distortion because of the absence of professional filming equipment and skills. Moreover, compression techniques [20, 22, 68, 74] and copyright protection processing [21] can also damage the quality of UGC videos. Therefore, Video Quality Assessment (VQA) of in-the-wild videos is increasingly important for major video platforms to filter out and enhance low-quality videos.The lack of raw information and the diversity of distortion types in in-the-wild videos present a significant challenge for VQA research. Fortunately, there are many subjective experiments that provide high-quality datasets [16, 32, 38, 43, 48, 56, 70] , which are labeled according to human mean opinion scores (MOS). With the benefit of these datasets, the current VQA methods can perform supervised training on them to fit the MOS as best as possible. Traditional VQA methods [1, 5, 8, 25, 33, 41, 46, 53] are successful in predicting the quality of perceptual videos, which model spatial and temporal distortions using handcrafted features. However, the hand-crafted features have a low correlation with human perception, so its outcomes are not always reliable. In recent years, with the advancement of deep learning techniques, VQA methods [28, 29, 49, 59, 70] based on deep neural networks (DNNs) can extract more complex and abstract features related to video quality and achieve superior performance than traditional methods. However, most deep learning approaches focus on the effect of spatial and temporal video distortion on video quality, without adequately considering the relationship between video quality factors and human feelings. Research has shown that human judgment is always influenced by how the brain feelings [39, 72] . As shown in Fig. 1 , two videos with the same objective quality but different content have different subjective quality scores. Considering that all the Figure 1 : Validating the impact of human feelings on HVS for VQA and the relevance of CLIP with human feelings in video perception. Two videos with the same quality captured in the same scene using the same equipment. CLIP show high consistency with human perception in video perception. We selected 10 subjects to perform VQA on the two videos and took their mean value as the video quality score. current datasets used for VQA are labeled based on HVS, incorporating human feelings into VQA enables the model to achieve better consistency with HVS. Although some recent studies [9, 29, 57, 60] have demonstrated that video content can indeed influence the human judgment of the video quality, these investigations predominantly extract high-level abstract features that are not directly related to human intuitive perception to assess the impact of video content on HVS. Therefore, we believe that these features do not effectively capture the true human feelings of video quality, thus limiting the effectiveness of these methods in practical applications.Extracting features that capture human feelings from videos presents a significant challenge. This is largely due to the complexity and subtlety of feelings conveyed through videos, which are not easily encapsulated by straightforward data features. It is well known that human feelings can be accurately described through language. Therefore, if we can deduce the feelings a video induces through its associated linguistic expressions, we may open new pathways to tackle the intricacies of such nuanced feature extraction, thereby potentially resolving the challenges previously delineated. Fortunately, the recent advancements in Vision-Language pre-trained models (VLM) have significantly propelled the field forward. These models have not only established a crucial link between linguistic and visual information but have also equipped us with the ability to interpret the feelings elicited by video content through an analysis of language expressions. Specifically, Contrastive Language-Image Pre-training (CLIP) [44] , endowed with a rich vision language prior, has demonstrated robust zero-shot predictive capabilities across multiple tasks. Furthermore, CLIP has the ability to perceive image and video quality to some extent [37, 55] . Nonetheless, a pending challenge in video perception is whether CLIP has good agreement with human feelings. If so, this implies that by conducting an indepth analysis of the linguistic descriptions related to videos, we can capture the feelings intended to be conveyed by the videos with greater precision, thereby making unprecedented strides in the understanding and interpretation of these feelings.In this paper, to address the above difficulty, we verify through extensive experiments that CLIP has a high degree of consistency with human feelings in video perception. Further, we propose a novel model (denoted as CLiF-VQA) to enhance video quality assessment (VQA) by incorporating high-level semantic information related to human feelings. Our model innovatively utilizes the visual language capabilities of CLIP to extract features from visual content that are relevant to human feelings. In addition, it captures low-level-aware features by using the Video Swin Transformer model [36] to reflect spatial and temporal distortions in video frames, providing a comprehensive framework for assessing video quality that is highly consistent with human perception. Specifically, we use a set of objective (e.g., bright, blurry, noisy, colorful, etc.) and subjective (e.g., pleasant, boring, fearful, etc.) descriptions that are closely related to human feelings as prompts. The cosine similarity between the visual content and the text prompts is then computed thereby predicting the score corresponding to each prompt. Further, we design a semantic feature extractor (SFE), which extracts high-level semantic feature maps corresponding to descriptions from multiple regions of the video frame. Finally, we fuse the low-level-aware and high-level semantic features to obtain the video quality score.Our contributions can be summarized as follows:• We validate for the first time that CLIP is highly consistent with human feelings in video perception. • We propose CLiF-VQA, which for the first time incorporates features related to human feelings in VQA.Extensive experiments demonstrate that CLiF-VQA achieves the best performance on multiple VQA datasets. • We design some efficient objective and subjective descriptions that are related to human feelings. These prompts enable us to extract from the video rich features related to subjective and objective human feelings. • We design a zero-shot advanced semantic feature extractor (SFE) based on CLIP. It extracts semantic features by sliding over multiple regions of a video frame and splices the same semantic features according to their relative positions to obtain semantic feature maps of the video frame. 
related work 2.1 classical vqa methods: 
 Classical VQA methods employ handcrafted features to capture specific types of distortions in the video for quality prediction. Early VQA often apply Image Quality Assessment (IQA) algorithms [12, 27, 40, 42, 67, 69] to obtain frame-level features, and then combine with temporal dimension information to obtain video quality scores. For example, V-CORNIA [66] extends the IQA algorithm CORNIA [69] to VQA to obtain frame-level quality scores, and combines these scores through temporal pooling. However, this method does not fully consider the connection between the spatio-temporal information of the video and how they affect the video quality [2, 23, 45, 46] . Natural video statistics (NVS) can take into account both spatio and temporal information, thus it is applied to address the previous problem. V-BLIINDS [46] extracts spatio-temporal statistical features of frame-differences in the video DCT domain and predicts crude frame quality scores using NIQE [42] . TLVQM [25] considers two levels of features, first computing low complexity features for each frame to extract frame-level statistical features related to motion, and then computing high complexity features related to spatial distortion for representative frames. VIDEVAL [53] applies various handcrafted features to detect and measure the distortions and reduces the computational complexity by reducing the feature dimensions.Recently, deep learning-based VQA Methods [3, 28, 29, 34, 49, 58-62, 65, 70, 73, 75, 76] have gradually achieved better performance than classical methods. Rather than relying on handcrafted features, deep VQA methods employ convolutional neural networks (CNN) [10, 11, 13, 14, 47, [50] [51] [52] or Transformer models [7, 35, 36] to extract complex and abstract features that are relevant to video quality aspects. For example, VSFA [29] extracts spatial features of video frames using ResNet-50 [13] pre-trained on ImageNet [6] , and then models the temporal features using GRU [4] . Similar to the architecture of VSFA, while GST-VQA [3] applies VGG-16 [47] to extract spatial features of videos. To better capture the spatio-temporal information of the video, some works [34, 49, 57, 70, 71, 73] adopt 3D-CNN. For example, V-MEON [34] adopts a multi-task framework which utilizes 3D-CNN to extract spatio-temporal features to predict the quality of the video. Other studies [49, 57, 70, 73] combine both 2D-CNN and 3D-CNN to capture the spatial and temporal features of video, and then integrate the two features for quality prediction. Recently, VQA methods [58] [59] [60] using the transformer structure have achieved better results relative to CNN. DisCoVQA [60] uses Video Swin Transformer [36] to extract multi-level spatiotemporal features and improves the learning efficiency of the model by temporal sampling of the features. Similarly, FAST-VQA [58] and FasterVQA [59] obtain fragments by spatial-temporal grid minicube sampling (St-GMS) and then feed the fragments into a modified Video Swin Transformer [36] . Although deep learning-based VQA methods can extract complex high-level semantic features, these features are not directly related to the human point of view. Two recent works [63, 64] attempt to address this issue. Specifically, MaxVQA [64] captures a variety of quality factors that can be observed by humans through a modified vision-language foundation model CLIP and can jointly evaluate multiple specific quality factors and overall perceptual quality scores. Several studies [30, 57, 60] have noted that aesthetic factors [15, [17] [18] [19] 31] of visual content affect video quality assessment. Inspired by this, Dover [63] assesses video quality from both aesthetic and technical perspectives, so it relatively well models the human process of perceiving quality. 
clip for video perception: 
 CLIP, as shown in Fig. 2 , demonstrates excellent zero-shot prediction ability in vision-language tasks. Not only that, it also shows some perceptive ability in IQA and VQA [37, 55] . However, it is not verified whether it still has good perceptual ability on linguistic prompts related to human feelings. The study in this section represents a pioneering effort to ascertain the degree to which the CLIP's video perception aligns with human feelings, thereby ensuring the extraction of human affective features from videos with the highest possible fidelity. Specifically, in order to fully extract video features while avoiding quality loss due to resizing and cropping, we extract semantic features from multiple regions on all video frames by means of sliding window (Details in Sec. 4.1). Then we compute the mean of all the feature values corresponding to a specific prompt as the feature of the video for that prompt. Prompt Design. We apply multiple objective descriptions related to quality factors (e.g., bright, contrast, etc.) and multiple subjective descriptions related to human feelings (e.g., interesting, exciting, etc.) as prompts. For details see Sec. 4.1. Relative to the antonym prompts strategy [55] , our design can extract richer features related to human feelings from videos. Here, we refer to HVS's perception of video quality factors and content as human objective and subjective feelings respectively. Further, we explore the correlation between CLIP and human objective feelings and subjective feelings. Perception of Objective Feelings. We explore the performance of CLIP on four objective descriptions (bright, contrast, noisy, colorful) related to video quality factors, as shown in Fig. 3 . Specifically, we first process the video corresponding to a certain description, and then extract the semantic features that correspond to the description. It can be seen that CLIP is able to accurately perceive changes in video quality factors. This shows that CLIP has a good consistency with human objective feelings in the perception of video quality factors. Perception of Subjective Feelings. Furthermore, we explore the relationship between CLIP and human subjective feelings in video content perception. In particular, we conduct experiments on four subjective descriptions (interesting, exciting, depressing, fearful) that reflect the subjective feelings that video content brings to humans. As shown in Fig. 4 . The results show that CLIP is highly consistent with human judgments in perceiving video content. Performance of CLIP in VQA. The experiments above demonstrate that CLIP has highly consistent results with humans in perceiving both the quality and content of the video separately. However, it remains to be verified whether CLIP is still effective when both objective and subjective descriptions are used as prompts. Therefore, we conduct further experiments to explore CLIP's performance in video quality perception when using both objective and subjective descriptions that can reflect human feelings. Specifically, we use multiple objective and subjective descriptions as prompts, as shown in Fig. 2 . We adopt the architecture of the classical VQA model VSFA [29] to input the feature vectors of the frames into the GRU for regression and time pooling operation to get the quality scores. We conduct our experiments on KoNViD-1k dataset [16] , and in addition to processing the comparison with the VSFA model, we also compare with two feature extraction methods (VGG-16 [47] and EfficientNet-V2 [51] ) widely used in VQA. And we evaluate performance on SROCC, KROCC and PLCC metrics, as shown in Fig. 5 . The results demonstrate that using both objective and subjective descriptions can achieve better results, compared to using a single description. Furthermore, it can be observed that relying on only features related to human feelings surpasses CNN extracted features in VQA. The results also confirm the validity of the prompts we designed. 
the proposed approach: 
 In this section, we introduce the proposed CLiF-VQA, which consists of the semantic feature extraction module 4.1 and the spatial feature extraction module 4.2, as shown in Fig. 6 . First, we employ the semantic feature extraction module to extract high-level semantic features that are related to human feelings. Then lowlevel-aware features are extracted using spatio-temporal feature extraction module. Finally we fuse these two features through a regression module thus obtaining the video quality score.In order to effectively extract features that can reflect human feelings, we first design some objective descriptions and subjective descriptions related to human feelings as prompts of CLIP, as shown in Fig. 2 . The prompts 𝑃 designed in this paper contains two types of descriptions: objective 𝑝 𝑜𝑏 and subjective 𝑝 𝑠𝑢𝑏 :EQUATIONThe adjectives and nouns that delineate feelings within objective descriptions are denoted as 𝐴 𝑜𝑏 and 𝑁 𝑜𝑏 respectively. Objective descriptions are categorized into the following two forms depending on whether they use adjectives or nouns:EQUATION)Subjective descriptions exclusively employ adjectives 𝐴 𝑠𝑢𝑏 to convey feelings:EQUATIONIn addition, due to the limitation of the visual encoder of CLIP on the input size, we can only extract the semantic information of a small region in the video frame. In order to be able to obtain as much semantic information as possible contained in the video frames, we extracted features from multiple regions of the video frames by sampling them multiple times at different locations, as shown in Fig. 6(b) . This avoids the loss of video quality caused by resizing and excessive cropping.Specifically, assuming the video has 𝑇 frames, we perform a sampling operation on the video frames 𝐼 𝑡 (𝑡 = 1, 2, ...,𝑇 )to obtain 𝑚 × 𝑛 image blocks:EQUATIONwhere 𝑏𝑡 represents the block obtained by sampling in the i-th row and j-th column.Given any visual input 𝑏𝑡 and text prompt 𝑃, the vision 𝐸 𝑣 and text 𝐸 𝑡 encoders of CLIP encode them to achieve a consistent Then, calculate the cosine similarity between the visual content and prompts to predict the score for each dimension:EQUATIONA normalization procedure is applied to the acquired cosine similarity score to standardize its range:𝑣 𝑖,𝑗 𝑡 = 𝑒𝑥𝑝 (𝑠 𝑖,𝑗 𝑡,𝑙 ) 𝑛 1 +𝑛 2 𝑘=1 𝑒𝑥𝑝 (𝑠 𝑖,𝑗 𝑡,𝑘 ) | 𝑛 1 +𝑛 2 𝑙=1 (7)where the feature values of 𝑣 𝑖,𝑗 𝑡 are the same number and one-to-one correspondence with the number of descriptions.Then the splicing operation (SFRP) is performed on all the features 𝑣 𝑖,𝑗 𝑡 based on the relative position to obtain the semantic feature map 𝑀 𝑡 of frame 𝐼 𝑡 :EQUATION𝑀 𝑡 contains 𝑟 feature maps, each corresponding to a description. Further, we perform the global average pooling operation (𝐺𝑃 𝑎𝑣𝑔 ) and the global max pooling operation (𝐺𝑃 𝑚𝑎𝑥 ) on 𝑀 𝑡 to obtain the universal features and distinctive features as shown Fig. 6(a) . The outputs of (𝐺𝑃 𝑎𝑣𝑔 ) and (𝐺𝑃 𝑚𝑎𝑥 ) are two r-dimensional feature vectors 𝑓 𝑎𝑣𝑔 𝑡 and 𝑓 𝑚𝑎𝑥 𝑡 , respectively.EQUATION𝑓 𝑎𝑣𝑔 𝑡 and 𝑓 𝑚𝑎𝑥 𝑡 are then concatenated as the semantic feature vectors 𝑓 𝑡 of the video frame 𝐼 𝑡 :EQUATION)where ⊕ is the concatenation operator and 𝑓 𝑡 is a feature vector of length 2 × (𝑛 1 + 𝑛 2 ).Next, we perform a concatenation operation on the semantic features {𝑓 𝑡 } 𝑇 𝑡 =1 of all the video frames thereby obtaining the semantic feature maps 𝑀 𝑠 of the video:EQUATIONwhere ⊕ here is not exactly the same as the concatenation of ⊕ in Eq. 10. Here, the feature vectors {𝑓 𝑡 } 𝑇 𝑡 =1 are concatenated along the channel dimension, so that the dimension of the obtained 𝑀 𝑠 is [2 × (𝑛 1 + 𝑛 2 ),𝑇 ]. Each feature map corresponds to a description, and the feature maps here are divided into two types, namely the feature map with global average pooling operation (𝐺𝑃 𝑎𝑣𝑔 ) and the feature map with global max pooling operation (𝐺𝑃 𝑚𝑎𝑥 ).After extracting the semantic feature maps of the video, we use a multi-layer perceptron (MLP) to obtain the feature vectors 𝐹 𝑠 corresponding to the descriptions. The MLP is composed of two fully connected layers and the activation function is GELU:EQUATIONwhere 𝐹 𝑠 is a 2 × (𝑛 1 + 𝑛 2 ) dimensional feature vector.In VQA, the spatio-temporal features of the video play a very important role in estimating the overall video quality. Since low-level information is easily affected by distortion, extracting the lowlevel-aware features of the video can effectively capture the spatiotemporal distortion of the video. In our approach, to diminish the computational complexity, the video is sampled employing the grid mini-patch sampling approach [58] . First, the video frame 𝐼 𝑡 (𝑡 = 1, 2, ...,𝑇 ) is segmented into 𝑁 ×𝑁 grids of equal size:EQUATIONwhere 𝑔𝑡 denotes the grid of the i-th row and j-th column, and W and H are the height and width of the video.A random patch sampling is then performed on each 𝑔 where 𝑆𝑎𝑚𝑝𝑙𝑖𝑛𝑔𝑡 represents the random sampling operation on the grid of the i-th row and j-th column of the t-th frame. The sampling operation samples the same position on different video frames to ensure temporal continuity.Then splice 𝑀𝑃𝑡 (1 ≤ 𝑖, 𝑗 ≤ 𝑁 ) according to their original positions thereby obtaining the sampled map 𝑆 𝑡 of the video frame 𝐼 𝑡 . The same operation is performed on all frames of the video to obtain the sampled fragments𝑉 𝑓 = [ 𝑆 𝑡 | 𝑇 𝑡 =1]. The video fragments 𝑉 𝑓 are then fed into a modified Video Swin Transformer Tiny [36] and non-linear layers to obtain local quality maps 𝑀 𝑓 𝑖𝑛𝑎𝑙 .EQUATIONFinally we flatten 𝑀 𝑓 𝑖𝑛𝑎𝑙 to obtain the spatio-temporal feature vector 𝐹 𝑓 of the video.EQUATIONAfter extracting the semantic and spatio-temporal features of the video through the semantic feature extraction module and spatiotemporal feature extraction module, we need to map these features to the quality scores via a regression model. First, We concatenate the semantic features 𝐹 𝑠 and spatial features 𝐹 𝑓 to get the overall features 𝐹 𝑣 of the video:EQUATIONThen we design a regression head with two fully connected layers to predict the quality score of the video:EQUATIONThe loss function used to optimize the proposed models consists of two parts: the monotonicity-induced loss and linearity-induced loss. Given m predicted quality scores Q = { q1 , q2 , ..., q 𝑚 } and m ground-truth subjective quality scores 𝑄 = {𝑞 1 , 𝑞 2 , ..., 𝑞 𝑚 }.Specifically, the monotonicity-induced loss predicts the monotonicity of the video quality scores by introducing additional order constraints. The monotonicity-induced loss function is defined as follows:𝐿 𝑚𝑜𝑛 = 1 𝑚 2 𝑚 ∑︁ 𝑖=1 𝑚 ∑︁ 𝑗=1 𝑚𝑎𝑥 (0, 𝑞 𝑖 -𝑞 𝑗 -𝑓 (𝑞 𝑖 , 𝑞 𝑗 ) • ( q𝑖 -q𝑗 )) (19)where𝑓 (𝑞 𝑖 , 𝑞 𝑗 ) = 1 if 𝑞 𝑖 ≥ 𝑞 𝑗 , otherwise 𝑓 (𝑞 𝑖 , 𝑞 𝑗 ) = -1.In contrast, the goal of the linearity-induced loss is to compute the linear relationship between the predicted quality score and ground-truth subjective quality score. The linearity-induced loss function can be denoted as:𝐿 𝑙𝑖𝑛 = (1 - 𝑚 𝑖=1 ( q𝑖 -â)(𝑞 𝑖 -𝑎) √︃ 𝑚 𝑖=1 ( q𝑖 -â) 2 𝑚 𝑖=1 (𝑞 𝑖 -𝑎) 2)/2 (20) where 𝑎 = 1 𝑚 𝑚 𝑖=1 𝑞 𝑖 and â = 1 𝑚 𝑚 𝑖=1 q𝑖 . Finally, the total loss function 𝐿 is obtained by combining the two loss functions 𝐿 𝑚𝑜𝑛 and 𝐿 𝑙𝑖𝑛 above:EQUATIONwhere 𝛼 and 𝛽 represent the weights of monotonicity-induced loss and linearity-induced loss. 
experiments 5.1 experimental setups: 
 5.1.1 Datasets. We test the model on four datasets including LSVQ [70] , KoNViD-1k (1200 videos) [16] , LIVE-VQC (585 videos) [48] , and YouTube-UGC (1067 videos) [56] . Specifically, we pre-train CLiF-VQA on LSVQ 𝑡𝑟𝑎𝑖𝑛 , a subset of LSVQ containing 28,056 videos. Intra-dataset testing is performed on two subsets of LSVQ, LSVQ 𝑡𝑒𝑠𝑡 (7400 videos) and LSVQ 1080𝑝 (3600 videos). We perform cross-dataset testing on KoNViD-1k and LIVE-VQC. Further, we fine-tune the model on KoNViD-1k, LIVE-VQC, and YouTube-UGC. It should be noted that YouTube-UGC contains 1500 videos, but only 1067 videos are available to us.Criteria. Spearman Rank Order Correlation Coefficient (SROCC) and Pearson Linear Correlation Coefficient (PLCC) are used as evaluation Metrics. Specifically, SROCC is used to measure the prediction monotonicity between predicted scores and true scores by ranking the values in both series and calculating the linear correlation between the two ranked series. In contrast, PLCC evaluates prediction accuracy by calculating the linear correlation between a series of predicted scores and true scores. And higher SROCC and PLCC scores indicate better performance.5.1.3 Implementation Details. we employ PyTorch framework and an NVIDIA GeForce RTX 3090 card to train the model in all experimental implementations. In the semantic feature extraction module, we sample each frame of the video 9 times and then perform zero-shot feature extraction with CLIP, as shown in Fig. 6(b ).In the spatio-temporal feature extraction module, we use Video Swin Transformer Tiny [36] , pre-trained on the Kinetics-400 [24] dataset, as the backbone. During training, the initial learning rate of Swin Transformer backbone is set to 0.000075, and the initial learning of other parts is set to 0.00075. We set the batch size to 12 and use the AdamW optimizer with a weight decay rate of 0.05.We compare the proposed method with the following methods:• Classical Methods: BRISQUE [40] , TLVQM [25] , VIDEVAL [53] , RAPIQUE [54] . • Classical + Deep Learning Methods: CNN+TLVQM [26] ,CNN+VIDEVAL [53] . • Deep Learning Methods: VSFA [29] , PVQ [70] , BVQA [28] ,GST-VQA [3] , CoINVQ [57] , FAST-VQA [58] , FasterVQA [59] , DOVER [63] . Compared with some classical methods, CLiF-VQA achieves a significant improvement in performance on all test datasets. In addition, CLiF-VQA achieves better results compared to FAST-VQA and FasterVQA, which focus only on low-level-aware features of the video. This suggests that the introduced human feelings features can well complement the spatial features, thus improving the prediction accuracy. In addition, CLiF-VQA performs better on both intradataset testing and cross-dataset testing compared to the current state-of-the-art DOVER, with an average improvement of 1.02% and 0.85% on SROCC and PLCC.After pre-training on LSVQ, we fine-tune CLiF-VQA on three small datasets (LIVE-VQC, KoNViD-1k, YouTube-UGC), as shown in Tab. 2. As before, all experiments are conducted under 10 train-test splits.As can be seen, CLiF-VQA achieves unprecedented performance on all three datasets. Relative to the current best performance, CLiF-VQA improved by an average of 2.19% and 1.71% on SROCC and PLCC, respectively. The results further illustrate the effectiveness of introducing human feelings in VQA.The performance of the model on two sets of videos when using only spatio-temporal features as well as using both spatio-temporal and semantic features. We validate the effectiveness of the three modules that make up CLiF-VQA. As shown in Tab. 4, CLiF-VQA has acceptable performance when only semantic features related to human feelings are extracted, and the performance of CLiF-VQA is further improved when the regression head is introduced. When only low-level-aware features of the video are extracted, CLiF-VQA performs better than when only semantic features are extracted. However, the performance did not improve significantly after further introducing the regression head. When features related to human feelings are introduced on top of the spatial features, the performance of the model improves significantly, and it is further improved by introducing the regression head. In addition, we further compare the performance of the model on videos that elicit different feelings in humans when using only spatial features and when using both spatial and semantic features, as shown in Fig. 7 . We choose two sets of videos that have very different MOS, but have similar quality scores when predicted using only spatial features. After we further introduce semantic features related to human feelings, the predicted quality scores are closer to the real MOS. These experimental results illustrate the validity of human feelings we introduced in VQA. The results demonstrate that objective descriptions have a greater impact on the performance improvement of CliF-VQA compared to subjective descriptions. And the optimal results can be obtained by using both objective and subjective descriptions. 
conclusion: 
 In this paper, we first analyze that human feelings have a significant impact on video quality assessment (VQA). Further, We validate for the first time that CLIP is highly consistent with human feelings in video quality perception. Extensive experiments demonstrate that CLIP not only has good consistency with human feelings, but also can achieve satisfactory results in VQA by using only the features related to human feelings extracted by CLIP. Motivated by these findings, we propose CLiF-VQA, a method that extracts features related to human feelings and low-level-aware features of the video.Then, the quality score of the video is obtained by aggregating the two features. Experimental results demonstrate that the proposed CLiF-VQA outperforms existing methods on multiple VQA datasets. 
limitation: 
 Our model is not end-to-end, and since considering CLIP's in training would lead to slow computation, we perform CLIP feature extraction separately. There is room for optimizing the prompts we design, such as increasing the number of prompts as well as and selecting better prompts. 
 
 Appendix:  


Decision: Borderline rating from the reviewers, which are all quite confident, but looking at their background, most are at the start of their scientific careers, so not sure that the confidence level should be so high. Examined the paper myself and I agree with the most senior reviewer's recommendation.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose a video quality assessment method, called CLiF-VQA, which augments traditional spatio-temporal feature assessment by incorporating a multimodal large language model, CLIP, to extract high-level semantic information related to human emotions. Through extensive experimental validation on multiple datasets, CLiF-VQA shows highly consistent results with humans in video quality perception, with a promising performance improvement over existing methods.However, this paper lacks a detailed discussion of the potential limitations in practical applications and the specific advantages and disadvantages of the method.However, this paper lacks a detailed discussion of the potential limitations in practical applications and the specific advantages and disadvantages of the method. However, this paper lacks a detailed discussion of the potential limitations and the specific advantages and disadvantages of the method.
strength_weakness: strength_weakness: （1）	The method pioneers the exploration of the consistency between CLIP and human feelings in video perception, offering a new dimension to VQA by considering both objective and subjective aspects.
（2）	The approach demonstrates improved performance over existing methods by combining traditional spatio-temporal features with high-level semantic features.
（3）	Figures and tables are relevant and well-designed, providing clear visual support to the text. For example, Figure 1 effectively illustrates the impact of human feelings on VQA.
（4）	The experimental setup is robust, involving extensive testing on multiple datasets, including LSVQ, KoNViD-1k, LIVE-VQC, and YouTube-UGC.
limitations: limitations: （1）	While the integration of human feelings is innovative, the practical implications and potential limitations in various real-world applications are not thoroughly discussed. 
（2）	Some figure captions could be more descriptive to provide better context. For example, the caption for Figure 3 could include more details about the significance of the results shown.
（3）	The discussion of experimental results could be expanded to include more insights into the limitations and potential areas for improvement of the proposed method.
paper_summary: paper_summary: This paper proposes a new VQA approach, CLiF-VQA, by integrating FAST-VQA and CLIP feeling modules for better Video Quality Assessment performance. It claims that human feelings on videos will also affect the final quality scores of videos, and hence proposes to include "feeling-prompts" into the VQA model. The CLiF-VQA suggests better accuracy than previous models.
strength_weakness: strength_weakness: 1. It is clearly motivated.
2. The writing is good, well understandable; the equations are well-defined.
3. The evaluation is sufficient, proving effectiveness of CLiF-VQA.
limitations: limitations: 1. CLIP is not an MLLM. Please try to revise this misleading name.
2. Please try to discuss the limitations of this work.
paper_summary: paper_summary: The paper proposes a novel method called CLiF-VQA to enhance video quality assessment by incorporating high-level semantic information related to human feelings. The key contributions are:
Validating for the first time that Contrastive Language-Image Pre-training (CLIP) is highly consistent with human feelings in video perception.
Proposing CLiF-VQA, which for the first time incorporates features related to human feelings in VQA, achieving state-of-the-art performance on multiple VQA datasets.
Designing effective prompts, including objective and subjective descriptions closely related to human feelings, as linguistic prompts for CLIP.
Developing a zero-shot advanced semantic feature extractor (SFE) based on CLIP to extract semantic features from videos.
strength_weakness: strength_weakness: (1) The methodology of incorporating human feelings features into VQA is innovative and practical, leveraging the capabilities of MLLMs like CLIP.
(2) Extensive experiments demonstrate the effectiveness of CLiF-VQA, achieving significant improvements over existing methods on multiple datasets.
(3) The use of CLIP and the designed prompts provide insights into how linguistic expressions can be utilized to capture feelings elicited by video content.
(4) The ablation studies validate the contributions of different components of CLiF-VQA and demonstrate the importance of incorporating human feelings features.
limitations: limitations: (1) The implementation details, especially regarding the hyperparameters and training procedures, are not provided in sufficient detail to fully reproduce the results.
(2) More thorough comparisons and discussions with related works, especially recent ones, could strengthen the evaluation of CLiF-VQA.
(3) Further ablation studies exploring the impact of different design choices (e.g., prompt selection) could provide additional insights.
(4) The theoretical analysis of why incorporating human feelings improves VQA could be expanded to deepen the understanding of the proposed approach.
(5) As far as I konw, the model metrics is lower than other experiments (e.g the dover's metrics on KoNViD-1k)
paper_summary: paper_summary: This paper explores the application of CLIP in the field of video quality assessment.
The paper utilizes CLIP to extract high-level semantic information and employs prompts related to human feeling, which enhances the performance of existing VQA.
strength_weakness: strength_weakness: The article explores the ability of CLIP to effectively extract human feeling features and does some subjective testing.
Meanwhile, I think there are some shortcomings in the novelty, theory, and experimentation of the paper, and these shortcomings lead me to not recognize the innovation of the paper. Please refer to **Limitations**.
limitations: limitations: Major issue:
In fact, the paper lacks theoretical explanations as well as sound logical descriptions. I will show my point by listing these aspects as follows:
(1) The paper does not explain the correlation between subjective quality scores and human feeling, including not even defining human feeling. This means that even though CLIP extracts human feeling features, it does not effectively represent video quality. 
(2) When there is some correlation between subjective quality and subjective perception, the novelty of the article will be severely lacking. Because, the pipeline of the paper is almost the same as [1]. 
(3) In Figure 5, the results of “Only subjective” is completely inconsistent with subjective quality, does this mean that human feeling is not correlated with subjective quality.
(4) In Figure 6, the high-level semantics will be lost after the video frames are divided into blocks.
(5) The ablation experiments did not even use ViT (pre-trained in ImageNet) + spatial-temporal as a comparison to illustrate that the gain comes from CLIP rather than semantic features.
Based on these shortcomings, I think the novelty of the paper is limited.

Other issue:
**Introduction:**
(1) Lines 112-115 "It’s pretty obvious that human ... different subjective quality scores.", This sentence is difficult to understand because different content has different subjective qualities but they have the same objective quality, couldn't this be due to poor performance of the objective model.
(2) Lack of definition of "human feeling", including Figure 1.

**Related Work:**
(1) The paper uses CLIP to extract high-level semantic features, yet there is no information describing CLIP in related work (or Sections 3 or 4)
(2) What is the difference between the proposed method and MaxVQA [1]. As far as I know, MaxVQA also uses a combination of CLIP + Fast-VQA.

**Section 3:**
The writing is confusing. 
First, the first description of Figure 3 precedes Figure 2. 
Second, what is the difference between the "+" in the third and fourth columns of figure 3, and what do the numbers mean? 
Where does the data source for Figure 4 come from, and where does the human feeling labeling come from.
In Figure 5, the KROCC is not defined and it is clear from Figure 5 that the objective description is more facilitative. Also, CLIP uses ViT as a backbone network, while the comparison method is based on CNN.

**Proposed method:**
The pipeline of the proposed method is very similar to MaxVQA, except for the addition of human feeling prompts. However, the paper does not have a reliable theoretical or experimental proof that human sensory cue words have a role in video quality.

**Experiments:**
(1) The comparison with MaxVQA is missing, due to the fact that both the proposed method and MaxVQA are based on multi-modal VQA models, while the comparison method is based on unimodal.
(2) Lines 687-689, "Limited by our video memory (24G), we set the batch size to 12 when reproducing FAST-VQA [46] as well as DOVER [51], instead of the 16 used in the original paper", changing the hyperparameters of the comparison method due to memory limitations is a very imprecise experimental setup. To my knowledge, setting the batch size of DOVER [2] to 12 still cannot run on RTX 3090.

[1] Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. 2023. Towards Explainable In-the-Wild Video Quality Assessment: A Database and a Language-Prompted Approach. In ACM MM. 1045–1054.
[2] Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. 2023. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In ICCV. 20144–20154.