Vulnerabilities in Video Quality Assessment Models: The Challenge of Adversarial Attacks
Authors: Aoxiang Zhang
Yu Ran
Weixuan Tang
Yuan-Gen Wang
Venue: NeurIPS 2023 spotlight (2023)
Keywords: video quality assessment
adversarial attack
black-box
just noticeable difference

Abstract: No-Reference Video Quality Assessment (NR-VQA) plays an essential role in improving the viewing experience of end-users. Driven by deep learning, recent NR-VQA models based on Convolutional Neural Networks (CNNs) and Transformers have achieved outstanding performance. To build a reliable and practical assessment system, it is of great necessity to evaluate their robustness. However, such issue has received little attention in the academic community. In this paper, we make the first attempt to evaluate the robustness of NR-VQA models against
adversarial attacks, and propose a patch-based random search method for black-box attack. Specifically, considering both the attack effect on quality score and the visual quality of adversarial video, the attack problem is formulated as misleading the estimated quality score under the constraint of just-noticeable difference (JND). Built upon such formulation, a novel loss function called Score-Reversed Boundary Loss is designed to push the adversarial video’s estimated quality score far away from its ground-truth score towards a specific boundary, and the JND constraint is modeled as a strict $L_2$ and $L_\infty$ norm restriction. By this means, both white-box and black-box attacks can be launched in an effective and imperceptible manner. The source code is available at https://github.com/GZHU-DVL/AttackVQA.

Vulnerabilities in Video Quality Assessment Models: The Challenge of Adversarial Attacks
Ao-Xiang Zhang
Yu Ran
Weixuan Tang"
Yuan-Gen Wang"
Guangzhou University, China
{zax, ranyu>@e gzwu. edu. cn, {tweix, @gngyg) eetizzhu.edu....
Abstract
No-Reference Video Quality Assessment (NR-VQA) plays an essential role improving the viewing experience of end-users. Driven by deep learning, recent NR-VQA models based on Convolutional Neural Networks (CNNs) and Trans- formers have achieved outstanding performance. To build a reliable and practical assessment system, it is of great necessity to evaluate their robustness. However, such issue has received little attention in the academic community. In this paper, we make the first attempt to evaluate the robustness of NR-VQA models against adversarial attacks, and propose patch-based random search method for black-box attack. Specifically, considering both the attack effect on quality score and the visual quality adversarial video, the attack problem formulated as misleading the estimated quality score under the constraint of just-noticeable difference (JND). Built upon such formulation, novel Ioss function called Reversed Boundary Loss is designed to push the adversarial video's estimated quality score far away from its ground-truth score towards a specific boundary, and the JND constraint is modeled strict Le and Li norm restriction. By this means, both white-box and black-box box attacks can be launched an effective and imperceptible manner. The source code is available at https://github..co. http::/gtttubbcco///HHU http://hhhbbbmm///////UUDLLDtAt
Introduction
In recent years, the service of "we-media" has shown an explosive growth. It is reported that Facebook can produce approximately billion video views per day [T] Nevertheless, the storage and transmission of such volumes of video data pose a significant challenge to video service providers [2]. To address this challenge. video coding employed to balance the tradeoff between coding efficiency and video quality. Therefore, Video Quality Assessment (VQA) has become prominent research topic. In general, according to whether pristine video applied as a reference, VQA can divided into Full-Reference VQA (FR-VQA), Reduced- Referenceee VQA (RR-VQA), and No-Reference VQA (NR-VQA), Recently, Convolutional Neural Networks (CNNs) and Transformers have made great progress in NR-VQA. However, according to recent studies, Deep Neural Networks (DNNs) show vulnerability against adversarial examples [19] To build a reliable and practical assessment system, it is of great necessity evaluate the robustness of the NR-VQA models against adversarial attacks.
The threat of adversarial attacks has been studied the task of image classification. Szegedy et al. 4 first found that an image injected with a small amount of imperceptible adversarial perturbations can mislead DNNs with high confidence. Moosav-DezFool perturbations can achieve attack effect on different samples. Athalye el al. [6] showed that adversarial examples can be applied in physical world. Since then, numerous adversarial attacks on CNNs and Transformers have been proposed
al. (5) suggested that universal
9 24 32)
denotes corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Areme
Aome
NIVEA
home
NIVEA
NIVEA
Original
l2 =
255, L = 5/225 1 8 255, L. 255
Figure 1: The original video frame and the perturbed ones constrained by the L2 and Los norm.
Compared with its rapid development in the classification task, there have been few investigations of adversarial attacks in Quality Assessment (QA), including Image Quality Assessment (IQA) and Video Quality Assessment (VQA). Two pioneering works [44, 50 were the closest to this topic in scope. 44 first investigated the adversarial attack on IQA models. It optimized the perturbations with propagated gradients by NR-IQA models. Such perturbations were added to the original image to yield adversarial images. Multiple adversarial images could be generated under the constraint of different Lagrange multipliers, and those adversarial images added with the most perceptually invisible perturbation were selected according to human subjective experiments. This can be understood as a constraint limiting the perturbations below the just- nust-noticeable difference (JND) threshold. (50) designed universal perturbation the fixed perturbation trained by given IQA/VQA model to increase its output scores. Despite 44] and [50] have made early attempt of adversarial attack on TQA/VQA model, there still remain some important issues to be solved. Firstly, since human subjective experiments are both time-consuming and inteesin the test set of 44 includes only 12 images, which is too small to be practical. Secondly, (50) aims to merely increase the estimated quality score outputted by the target model, which is not in line with the definition of adversarial attack. Thirdly, [50] cannot control the visual quality during optimization, and thus the adversarial videos suffer from obvious artifacts. Fourthly, the adversarial attacks in both 44 and [50] are ineffective in more practical black scenarios.
To address the above problems, this paper comprehensively investigates the robustness of NR-VQA models by adversarial attacks. Firstly, adversarial attack on NR-VQA model is mathematically modelled. The target NR- VQA model takes in the adversarial video and outputs the estimated quality score. Meanwhile, in a specific direction. The optimization goal of such attack problem is to mislead the estimated quality score by means of pulling in the distance between the estimated and anchor quality scores under the constraint of restricting the distortion between the adversarial and original videos below the JND threshold. Specifically, the Rcore-Reversed Boundary Loss is designed push the adversarial video's estimated quality score far away from its MOS towards a specific boundary. And limiting the L2 and Lo norm of the perturbations below rather strict threshold can be interpreted as a kind of JND constraint. In this paper. unless otherwise specified, the L2 norm indicates the pixel-level L2 norm averaged on the whole video. An example of applying the constraint of L2 and L norm is given in Fig. [ It can be observed that by means of limiting the L2 and Lx norm of perturbations within rather strict threshold of 3/255 and 5/255, the perturbations are imperceptible to human eyes, which can be considered to be below the JND threshold. The contributions of this work are summarized as follows:
define the anchor quality score as far away from its mean opinion score (MOS)
Adversarial attacks on NR- VQA models are formulated as misleading the estimated quality score by means of pulling the distance between the estimated and anchor quality scores under the JND constraint. To the best of our knowledge, this is the first work to apply adversarial attacks to investigate the robustness of NR- VQA models under black-box setting.
A novel loss function called
Boundary Loss is proposed to push the adver- sarial video's s estimated quality score far away from its MOS towards a specific boundary. By means of minimizing such loss function, the generated adversarial video can effectively disable the NR-VQA models.
Adversarial attacks on NR-VQA models are implemented under white-box and black-box settings. Furthermore, a patch-based random search method is designed for black-box attack, which can significantly improve the query efficiency in terms of both the spatial and temporal domains.
2

---

2 Related work
2.1 NR-VQA models
With the rapid development of deep learning techniques, NR- VQA models based on CNNs and Transformers have received tremendous attention. Li et al. [25] extracted the 3D shearlet transform features from video blocks. Afterwards, logistic regression and CNN were simultancously applied to exaggerate the discriminative parts of the features and predict the quality score. Li et al. [26)
utilized ResNet [27] to extract features for each frame within video, which were fed into GRU (8] to construct the long-term dependencies in temporal domain. You and Korhonen (29 employed LSTM
30
to predict quality score based on the features extracted from small video cubic clips. Zhang et al.
40
reorganized the connections of human visual system, and comprehensively evaluated the quality
score from spatial and temporal domains. Li et al. [] transferred the knowledge from IQA to VQA,
and applied Flsw
(34] to obtain the motion patterns of videos.
Xing et al. (35] were the first to introduce Transformer 38) into NR- VR-VQA to implement space-time
attention. Wu al. 33 designed a spatial-tempoaa distortion extraction module, wherein the video Swin Transformer
22 was utilized to extract multi-level spatial-temporal features. Li and Yang [37] proposed: a hierarchical Transformer strategy, wherein one Transformer was utilized to update the frame-level quality embeddings, and another Transformer was utilized to generate the video-level quality embeddings. To reduce the computational complexity of Transformers, Wu et al. [41] segmented the video frames into smaller patches, which were partially concatenated into fragments and fed into video Swin Transformer 42
2.2 Adversarial attacks in classification
According to the knowledge of the attacker, adversarial attacks can be roughly divided into white-box and black-box attacks. general, most white-box attacks rely on the propagated gradients of the target model. Goodfellow et al. [16] suggested that DNNs were susceptible to analytical perturbations due to their linear behaviors, and proposed the Fast Gradient Sign Method (FGSM). On this basis, Kurakin et al. [17] launched the attack via several steps, and clipped the pixel values within the e-neighbourhood after each step. Similarly, Madry et al. (20] proposed a multi-step variation of FGSM called Projected Gradient Descent (PGD), wherein the updated adversarial example was projected on the Loo norm in each PGD iteration. Carlini and Wagner [3] formulated the adversarial attack as optimization problem with different objective functions, which aimed to find adversarial examples with minimum perturbations. Papernot al. [8 created adversarial examples by iteratively modifying the most significant pixel based on the adversrrial saliency map.
However, in real-world scenarios, the gradients the model may not be accessible to the attacker. Therefore, the black-box setting is a more practical attack scenario. Papernot et al. (21] performed a black-bor box attack by training a local surrogate model, and then used this surrogate model to generate adversarial examples that could mislead the target model. Chen et al. [22) proposed the Zeroth Order Optimization (ZOO) based attack to estimate the gradients of the target model. To improve the query efficiency of ZOO, Ilyas et al. [23] replaced the conventional finite differences method with natural evolution strategy in gradient estimation. Guo et al. (24) proposed a simple black-box attack, which iteratively injected perturbations that could lead to performance degradation of target model in greedy way. Xiao el al. [43 proposed AdvGAN to generate perturbations via feed-forwar process, which could accelerate the generation of adversarial examples in the inference stage.
3 Proposed method
3.1 Problem formulation
In this part, the adversarial attack on NR-VQA model is mathematically modelled as
argmin L (f (II 1 (_=))==)))) ((I))))))))) s.t. D ((I (I ) JND, (1) Tade
where X denotes the number of frames within a video, and (1a4) I== and { adversarial video and original video, respectively. f()) denotes the estimated quality score outputted
denote the
3

---

Algorithm 1 White-box attack on NR-VQA model
1: Require: Estimated score by target NR-VQA model f (-), anchor score b (-), number of frames
optimized in one round T, number of iterations in one round K, and step size
2: Input: Original video [Im]===
3: Output: Adversarial video {Iad) 1z=1
4: for to X by T do
// The optimization process contains [X/T] rounds.
5:
[Ar)=tT +- Initialize the perturbations from a discrete set {-1 (-1/255, 0, 1/255) 2+7 [Iav] 1z < (I+ + + +Ax)= 12t
6:
If Initialize the adversarial video.
7:
for l = to K do
I1 One round contains K iterations. //Compute the direction.
V (Igh) Card (s ((IeN))ttT) (L)))=
8:
9:
[I T4T
+- [I #4T Jæ +
// Update the adversarial video.
Jæ
1 +- Limit the pixel-level Le and Lox norm of perturbations #+T
10:
by the target NR-VQA model, and b(-) denotes the anchor quality score which is far away from its MOS in a specific direction. L(.) is the loss function that measures the distance between the estimated and anchor quality scores. The closer the distance between the estimated and anchor quality scores, the further the distance between the estimated quality score and its MOS, and the stronger the effect of adversarial attack on misleading the estimated quality score. D(.) is the distortion between the original and ad adversarial videos. Restricting the distortion below the JND threshold indicates that the perturbations are imperceptible human eyes. By means of minimizing the loss function under the JND constraint, the adversarial attack on NR-VQA model can be launched in an effective and undetectable manner.
To implement the adversarial attack according to Eq. (T. it is essential to design the loss function C. Although Mean Squared Error (MSE) Loss has been widely applied in regression problem, it is suitable for adversarial attack on NR-VQA model. The reason is that a well-performed adversarial attack NR-VQA model should possess the capability of leading the estimated quality score far away from its MOS in a specific direction, i.e., misleading the NR-V AR-VQA model to assign high-quality score low-quality video, and vice versa. Although applying MSE Loss may push the estimated quality score away from the MOS, it cannot control the leading direction. In this paper, a loss function called Score-Reversed Boundary Loss (Cart) is proposed, which is formulated
Larb (( )==1) - bb -1)))
(1mm)=-))
(2)
where
0, ( is high quality,
(Ie (Im)
boundary
(3)
1, if ( is low quality,
where the high or low quality of video in Eq. 3 can be determined by its MOS in practice, and the boundaries of and are regarded as the anchor quality scores.
Besides, the perturbation distortion D between the original and adversarial videos is another issue to be addressed. In general, limiting D below the JND threshold indicates that such perturbations are invisible to human eyes. However, JND is empirically determined by many factors, including brightness, contrast between the background, and individual visual sensitivity. Therefore, some fields convert it into certain objective metrics for simple modeling, such as PSNR in video coding [TO], SSIM in image/video enhancement (T]) and Lp norm adversarial attacks [13. Zhang et al. 44 introduced human subjective experiments to judge whether the perturbation distortion is below the JND threshold. However, such human subjective experiments are both time-consumin consuming and labor-intensive. In this paper, we refer to the works in adversarial examples to construct the
JND constraint 112 13 [4]. Considering a pair of original and adversarial videos, the problem of constraining the distortion below the JND threshold can be converted into the problem of restricting their pixel-level T.2 and Toxe norm to lower than a rather strict threshold. Consequently, the optimization based on subjective JND can be automatically performed by the objective metries.

---

Frame
Frame
NR-VQA model
Query
Liiresdl
pertabetion
Query
sontrbetion
Estiattl quilit sooms
Query
Quary
Univesel
penubetion
Figure 2: Illustration of the proposed patch-based random search method for black-box attack.
3.2 White-box adversarial attack
In the case of white-box attack, the parameters of the target model are accessible to the attacker. Therefore, the steepest descent method can be applied to minimize the -evre-Reversed Boundary Loss in Eq. (2). The process of generating an adversarial video has [X/T] rounds, wherein T denotes the number of frames optimized in one round together. At the beginning of each round, the adversarial videos are generated by means of adding the original videos with initialized perturbations denoted as { 1=+7 whose element independently sampled from a discrete set {-1/255, 0, 1/255]. Afterwards, K iterations
applied in this round to optimize the perturbations, and the detailed steps of one iteration are as follows. Firstly, the obtained adversarial videos are fed into the target NR-V VQA model, and the steepest descent direction of Larb be calculated optimize the perturbations. Secondly, the projected gradient descent method is utilized to limit the pixel-level L.2 and To norm of the perturbations within 1/255, which can be regarded pseudo-code of generating adversarial example for NR-V VQA model under white-box setting is given Algorithm |
kind of striet JND constraint. The
3.3 Black-box adversarial attack
Compared with white-box attack, black-box attack is a more practical attack scenario, wherein only the final output of the model can be queried optimize the adversarial examples. Note that the number of queries is an important evaluation metric in black-box attack. However, black-box attack on NR-VQA model pose great challenges from the following two aspects. Firstly, in most existing outstanding black-box attacks, the number of queries is positively correlated with the resolution of the video frame. As NR- VQA models are widely applied to 1080p and 2160p high-resolution datasets such as LIVE-VQC (15), YouTube-UGC [5), and LSVQ [66. its query resources may be quite expensive. Secondly, different frames within one video have different content characteristics. Therefore, to obtain better attack effect, each frame should be generated with specific perturbation. However, one video may consist of hundreds of frames, which can lead larger number of queries.
To reduce the query resources from the spatial and temporal perspective under black-box attack, we design patch-based random search method, as illustrated in Fig. 2 In this part, the height and width of a video frame are denoted as H and W, respectively. The height and width of a patch are denoted as h and w, respectively. Each frame is divided into red, green, and blue channels, and each channel is further split into -oneoolalpppinggg H W. patches. In this way, each frame contains L
x 3 patches.
Like the white-box attack, we suppose that the process of generating an adversarial video has |X/T) rounds. In one round, N queries are performed. The detailed steps of the optimization process in the n-th query are as follows. Firstly, for each of the T frames, patches, which are located in the same positions within these T frames, are randomly selected to be perturbed. Note that in each round of the attack, each of the patches would be selected once and only once, and all patches would be selected in N queries. Therefore, can obtain N x Z
W x 3. For the convenience of
5

---

Algorithm 2 Black-box attack on NR-VQA model
1: Require: Estimated score by target NR-V VQA model f (-), anchor score b (-), number of frames optimized one round T, number of queries one round N, perturbation magnitude y, size of video frame H x W, and size of patch h x
2: Input: Original video [_
3: Output: Adversarial video {Iadv] 1z=1
4: for = to X by T do
// The optimization process contains [X/T] rounds.
5:
[1= 2+T f 1
II Initialize the adversarial video.
R t Lrb (f
(I att)
(L))
6:
7:
for n t N do
II One round contains queries.
8:
Select Z patches to perturb in one frame. Р> - Encode information of the positions of the selected patches
Z max (LL x J .1)
9:
10:
m, + Generate universal perturbation map from discrete set {-7, +7}
11:
for operation E {-++} do
12:
Jadv
x+T
Selected patches are injected with m,, according given operation
{ ({
iadv Jz } ((L))
R' Lerb
I/ Evaluate the attack effect.
14:
if R' < R then
15:
x [=
x+T
Tadv x+T
II Update the adversarial video.
R R
17:
break
illustration, the information of the positions of the selected patches is encoded into a vector denoted as p xxx11, where K = the frame, and the element equals to 1 if such patch is perturbed, and 0 otherwise. Note that each of the T frames optimized in one round shares the same Рд- Secondly, for these T x Z patches be perturbed, universal perturbation map m,, is generated, wherein its dimension is the same as the dimension of the patch, and its element independently sampled from a discrete set {-7, +7}. Thirdly, the selected T' x Z patches are subtracted or added with the universal perturbation map, and the generated adversarial video fed into the target NR-VQA model. Such perturbations would be Boundary Loss in Eq. (2) decreases, and abandoned otherwise. Following
3. Each element in РN, corresponds a specific patch within
kept if the the above three steps, an adversarial video with X frames can be generated. The pseudo-code of generating adversarial example for NR-VQA model under black-box setting is given in Algorithm) 2
By means of applying the patch-based random search method, the constraint of JND can be formulated under Loo or pixel-level L2 norm. Firstly, the Loc norm between the original and adversarial videos y. which the maximum modification range of each element in the adversarial video. Secondly, the frame-level L2 norm between the original and adversarial videos can be formulated as
N
HxWx3xI3 Σ PP.ll = =111 E P.lll S N
(4)
==1
n=1
where the last inequality holds as different patches are selected once and only once in different rounds, and the last equality holds when all patches are injected with perturbations. And y is set as 5/255 in black-box attack. Therefore, the pixel-level L2 norm can be deduced that
x X H W X3
H E
W
x x3x h
L2
(5)
H W
x
Experiments
4.1 Experimental setup
NR-VQA models and datasets: Four representative NR-VQA models are tested in the experiments, including VSFA (26), MDTVSFA (39), TiVQA 449 and BVQA-2022 (31). The experiments

---

Table 1: Performance evaluations under white-box setting with L2 norm constraint. Here, the performance before attack is marked in gray values within parentheses.
(a) Performance evaluations on VSFA 26 and MDTVSFA
39
Metrie
SRCC
PLCC
R defined in Eq. 6
Model
VSFA
MDTVSFA
VSFA
MDTVSFA
VSFA MDTVSFA
KoNViD-Ik
-0.7198 (0.7882)
-0.7380 (0.8003) -0.8135 (0.8106)
-0.8212 (0.8074) -0.2507
-0.0368
VQC
-0.7518 (0.7665)
-0.7825 (0.7908) -0.7322 (0.7506)
-0.7729 (0.8091)
-0.4963
-0.3121
YouTube-UGG
-0,7071 (0.7492)
-0.7223 (0.7683) -0,8018 (0.772.))
-0.7939 (0.7950)
-0.3585
-0.2883
LSsQ--Test
-0.6723 (0.7865)
-0.6623 (0.8136) -0.6765 (0.7972)
-0.7906 (0.8125) 0.2692
0.4252
-0.7877 (0.7040)
-0.7573 (0.7114) -0.7830 (0.6868)
-0.7885 (0.7009) 0.0527
0.2589
(b) Performance evaluations TiVQA 49 and BVQA-2022 31
Metric
SRCC
PLCC
R defined Eq.
Model
TiVQA
BVQA-2022
TiVQA
BVQA-2022 TiVQA BVQA-2022
KoNviD-1kk
-0.7212 (0.8046)
-0.7371 (0.8427)-0.820))
(0.8377)
-0.8233
(0.8469) -0.2472
-0.0579
LIVE-VOC
-0.6643 (0.8179)
-0.6705 (.88698)-0.738 (0.8067)
-0.6834 (0.8457)
-0.2144
-0,0042
YouTube-UGC
-0.6179 (0.7907)
-0.6696 (0.8145)-0.7356 (0.8177)
-0.6177 (0.8500)
0.0574
0.1118
LeVQ-Test
-0.7457 (0.8331)
-0.7679
(0)8613) (0.8343)
-0.8063 (0.8531)
0.0248
0.0693
LSVQ-Test 1080p -0.7372 (0.7478) -0.7088 (0.7875)-0.7249 (0.7670) -0.7569 (0.7799) 0.1979
0,2037
Table 2: Performance evaluations under white-box setting with Loo norm constraint. Here, the performance before attack is marked in gray values within parentheses.
(a) Performance evaluations on VSFA
26 and MDTVSFA
39
Metric
SRCC
PLCC
R defined in Eq. 6
Model
VSFA
MDTVSFA
VSFA
MDTVSFA
VSFA MDTVSFA
KoNViD-1k
-0.7155
(0.7882)
-0.7296 (0.8003) -0.8139 (0.8106)
-0.8219 (0.8074) -0.2521
-0.0419
LIVE-VOR VQC
-0.7526 (0.7665)
-0.7766 (0.7908) -0.7323 (0.7506)
-0.7720 (0.8091)
-0.4968
-0,3129
YouTube-UGGC
-0.7013 (0.7492)
-0.7238 (0.7683) -0.8025 (0.7721)
-0.7946 (0.7950)
-0.3669
-0.2974
Les
-0.6601 (0.7865)
-0.6569 (0.8136) 0.6824 (0.7972)
-0.7949 (0.8125)
0.2551
0.4086
LSVQ-Test-1080p
-0.7864 (0.7040)
-0.7438 (0.7114) +0.7833 (0.6868)
-0.7878 (0.7009)
0.0528
0.2533
(b)
Performance evaluations on TiVQA 49 and BVQA-2022 31
Metric
SRCC
PLCC
R defined in Eq. 6
Model
TiVQA
BVQA-2022
TiVQA
BVQA-2022 TIVQA BVQA-2022
KoNViD-Ik
-0.7419 (0.8046)
-0.7265 (0.8427) -0.8230 (0.8377)
-0.8224 (0.8469) -0.2521
-0.0708
LIVE-VOC
-0.6833 (0.8179)
-0.6712 (0.8698)-0.7425 (0.8067)
-0.6867 (0.8457)
-0.2214
-0.0229
YouTube-UGC
-0.6034 (0.7907)
-0.6920 (0.8145)-0.7199 (0.8177)
-0.6889 (0.8500)
0.0479
0.0817
L.SVQ-Test
-0.7591 (0.8331)
-0.7769
(0.8613)-0.7335 (0.8343)
-0.8093 (0.8531)
0.0178
0.0533
LSVQ-Test-1080p-0.77 (0.7478) -0.7128 (0.7875)-0.7442 (0.7670)
-0.7613 (0.7799)
0.1875
0.1876
conducted on mainstream video datasets including KoNViD-Ik [48] LIVE-VOC 15) YouTube- UGC [55, and LSVQ [6. Specifically, LSVQ is further divided into two datasets according to their resolutions. Therefore, five datasets are applied in the experiments. For each of these five datasets, 50 videos randomly selected to evaluate the effectivvness of adversarial attacks.
Evaluation metrics:
The performance of NR-V NR-VQA model is evaluated by Spearman Rank order Correlation Coefficient (SRCC) and Pearson's Linear Correlation Coefficient (PLCC). Specifically, SRCC is used measure the monotonic relationship between the predicted quality scores and the ground-truth quality scores, while PLCC is used to measure the accuracy of the predictions. Note that both SRCC and PLCC are evaluated on batch of videos. However, in real attack scenario, the attacker may merely launch the adversarial attack SRCC and PLCC could be rather small, and thus these two metrics may not accurately reflect the attack effect. Therefore, follow [4] to define metric a single video, which is calculated as
few samples. In this case, the change range of
7

---

Table 3: Performance evaluations under black-box setting. Here, the performance before attack is marked in gray values within parentheses.
(a) Performance evaluations on VSFA
26 and MDTVSFA
39
Metric
SRCC
PLCC
R defined in Ey. 6
Model
VSFA
MDTVSFA
VSFA
MDTVSFA
VSFA MDTVSFA
KoNViD-Ik
-0.0305
(0.7882)
0.0261 (0.8003) 0.0586 (0.8106)
0.1235 (0.8074)
1.6573
1.7587
LIVE-VQC
-0.1605 (0.7665)
-0.0074 (0.7908)-0000) -0.0132 (0.7506)
0.0807 (0.8091)
1.3233
1.4391
YouTube-UGG
-0,2231 (0.7492)
-0.0646 (0.7683)-0010177 (0.772.1)
0.0532 (0.7950)
1.5499
1,5080
L.SVQ-Test
0.2386 (0. (867
-0.0706 (0.8136) 0.2413 (0.7972)
-0.0384 (0.8125)
2.3545
2.1352
LSVQ-Test-1080p
0.2351 (0.7040)
-0.1339 (0.7114) (0.6868)
-0.0288 (0.7009)
2.1839
1.8592
(b) Performance evaluations TiVQA 49 and BVQA-2022 31
Metric
SRCC
PLCC
R defined in Eq. 6
Model
TiVQA
BVQA-2022
TIVQA
BVQA-2022
TiVQA BVQA-2022
KoNViD-1k
-0,4992 (0.8046) 0.1978 (0.8427) -0,4099 (0.8377)
0.2215 (0.8469)
1.2458
2.0480
LIVE-VQC
-0,0762 (0.8179) 0.4571 (0.8698) 0.0452 (0.8067)
0.4778 (0.8457)
1.3234
2,2061
UGG
-0.1357 (0.7907) 0.3951 (0.8145) -0.0087 (0.8177)
0.4306 (0.8500)
1.3478
2.1072
LSVQ-Test
-0,3343
(0.8331) 0.3355 (0.8613) -0.2161 (0.8343)
0.4240 (0.8531)
1.8086
2.2269
LSVQ-Test- 1080p -0.1414 (0.7478) 0,3917 (0.7875) -0.0343 (0.7670) 0.4678 (0.7799)
1.3327
2.0796
the average logarithmic ratio between the expected change in quality prediction and the actual change in quality prediction caused by adversarial video as
x log (F((L)))) ((((L))))) b ((Tm) (1=))=))
R
S
(Idv)] (( Jar=1
where the definition of f and b can be referred to Section 3.1 S denotes the number of videos in the test set. A higher value of R indicates better model robustness. With the above three metrics, the prediction accuracy and robustness against adversarial attacks can be comprehensively evaluated.
Implementation details: We download the authors' source code [26] 39 49 (1], re-train the models
and produce adversarial videos the testing set. Since the estimated quality scores" scale among different VQA models is different, we need to rescale the boundary. The rescaling details can be referred to in the appendix. For specific dataset, the median of the MOS of the selected 50 videos is regarded as the threshold to decide whether a video is of high quality or low quality in Eq. (3) As for white-box attack, Adam optimizer [47] is applied, wherein the step size B is initialized as 3 10" K and T are set to 30 and 1. The pixel-level L2 norm of perturbations is constrained within 1/255. As for black-box attack, N, T, and y are to 300, 1, 5/255. Both h and set 56.
4.2 Performance evaluations under white-box setting
In this part, the performance of different NR-VQA models is evaluated under white-box setting, and the results under both pixel-level Le and Loo norm constraint are given in Tables and respectively. can be observed that the State-of-the-Art (SOTA) NR- VQQ models are completely disabled by adversarial attack. Specifically, such attack under both pixel-level Le and Loe norm constraint bring similar attack effects and lead to performance degradation of 194% and 195% on SRCC averaged on five datasets, demonstrating the vulnerabilities of these NR-VQA models. Mcanwhile, the R values on all datasets approximate to 0, indicating that the adversarial attack with our proposed Boundary Loss can push the estimated quality score towards the target boundary. Note that in white-box setting, the pixel-level L2 and Les norm of perturbations are constrained within 1/255 and 3/255, indicating that the degree of perturbation is kept extremely low level.
4.3 Performance evaluations under black-box setting
In this part, the performance of different NR-VQA models is evaluated under black-box setting, and the results given in Table 3 It can be observed that the SRCC and PLCC of the attacked
8

---

Original
Under white-box setting
Under black-box setting
Figure 3: The original video frames and the corresponding adversarial video frames.
Darmardd qualty sours
Sune Rumoad au
Barmand
022
-46
08
Sha Spane Endt Los
MOs
21
25
n&B
(a) White-hox.
(b) Black-box.
(c)
White-box.
(d) Black-box.
Figure 4: The attack effect on estimated quality scores ((a) and (b)) and the query efficiency with increasing iterations and number of queries ((c) and (d)).
VSFA, MDTVSFA, and TiVQA are around O, indicating that attack effect on these models is quite remarkable. Although the attack effect BVQA-2023 is less significant, it still brings performance degradation of 0.6449 on SRCC and 0.6254 on PLCC on the KoNViD-1 dataset. Such results indicate that the current VQA systems not robust against adversarial attacks, and can be severely disturbed in the black-box setting. Note that according to Eq. (5) the theoretical upper bound of the pixel-level T.2 norm and Too norm of the proposed patch-based random search method are both 5/255. Interestingly, in practical deployment, we find that only about 3/5 patches are injected with perturbations, and the pixel-level L2 norm is around 3/255. In Fig. the examples of the original video frames and the corresponding adversarial video frames under both white-box and black-box settings are presented, which indistinguishable from their original counterparts by human eyes.
4.4 Ablation study
In this part, ablation studies are conducted from four aspects. Firstly, the effectiveness of the proposed
Revrr-Reversed Boundary Loss verified under black-box and white-box settings. The experiments are conducted via attacking VSFA on the KoNViD-1K dataset, and the results are given in Fig. 4
From Fig. 4(a) and 4bb) can be seen that minimizing the satisfied attack effect. Specifically, in the white-box setting, the quality scores estimated by the target NR-VQA model basically reach the target boundary. From Fig. and 4(d) it can be observed that compared with the MSE Loss, the Reversdd Boundary Loss shows superiority in the query efficiency. In particular, the white-box setting, the adversarial attack with the Score-Reverse Boundary Loss can completely disable the NR- RRRVQA model with just few iterations.
Boundary Loss achieves

---

Table 4: Performance evaluations of SRCC for patch- basch and pixel- baxel-baedd methods.
Table 5:
Performance evaluations of SRCC with
different values of T.
KoNViD-
LSVQ Test
KoNViD- Ik
LSVQ-Tes Test
n-th query
T
patch
pixel
patch
pixel
White
Black
White
Black
100
0,5390
0.7850
0.4392 0.7793
1
-0,7003
-0.0305
-0.6594
0.2386
300
-0.0305 0.7831
0.2386 0.7747
2
-0,6972
0.1318
-0,6667
0.2579
500
0.7828
0.7738
4
-0.6747
0.2121
-0,6715
0.3171
1000
0.7823
0.7575
8
-0.6964
0.2747
-0,6768
0.3482
Table 6: Performance evaluations of motion em-
Table 7: Performance evaluations of motion em- beddings on VSFA.
beddings on BVQA-2022,
Motion
KoNviD-Ik
LSVQ-Test
Motion
KoNViD-1k
LevQ-Test
embeddings
SRCC
PLCC
SRCC
PLCC
embeddings
SRCC
PLCC
SRCC
PLCCC
-0.1779 -0.1465
0.1917
0.2641
X
-0.0305
0.0586
0,2386
0.2413
0.1978 0.2215
0.3355
0.4240
0.1858 0.2687
0.4358
0.4536
Secondly, our patch-based random search method is compared with the pixel-based random search method SimBA [24] under black-box setting. The experiments are conducted via attacking VSFA on the KoNViD-IK Ik and LSVQ datasets, and the results are given in Table [ We can see that compared with the pixel-based method, our method shows significant improvements in both attack effect and query efficiency. Specifically, our method merely consumes 300 queries to completely disable the NR- VRA model, while the pixel-based method barely has any attack effect with 1000 queries.
Thirdly, the number of frames optimized in one round, i.e., T, is investigated under both black-box and white-box settings. The experiments are conducted via attacking VSFA on the KoNViD- and LSVQ datasets, and the results are given in Table 5 From the perspective of the white-box setting, different values of 7 have little impact on the attack effect. However, in the black-box setting, smaller T corresponds stronger attack effect. The reason is that different frames within one video have different characteristics, and thus optimizing the perturbations with smaller T' can bring the potential of their individual differences full play. Considering the attack effect,, T is set in our method.
Fourthly, the impact of motion embeddings on the robustness of the model is explored under black- box setting. Note that EVQA-2022 applies motion embeddings, while VSFA does not. In Table 6 the original B BVQA- 2022 and the B BVQA-2022 without motion embeddings are compared. In Table 7 the original VSFA and the VSFA with motion embeddings are compared. It can be observed that the motion embeddings play an important role in improving the robustness of NR-VQA models against adversarial attacks. Specifically, the motion embeddings bring SRCC performance improvement of 0.2598 for VQQ-22022 and 0.2068 for VSFA. One possible reason is that the motion embeddings correspond to the inter-frame information in temporal domain. Therefore, the adversarial impact of perturbations added within frames may be weakened, resulting in an improvement of robustness.
Conclusions
In this paper, we fully investigate the robustness of NR-VQA models against adversarial attacks, and propose the Rever-Reeerreed Boundary Loss for the attack problem. Extensive experiments have been conducted under both white-box and black-box settings, and the following conclusions can be drawn. Firstly, the existing SOTA NR-VQA models are vulnerable adversarial attacks, which brings great challenges to build reliable and practical assessment systems. Secondly, the attack effect and query efficiency are influenced by different components, including the Rever-ee Boundary Loss, the patch-based random search method, and the number of frames optimized in one round. Thirdly, motion embeddings have a positive impact on increasing the robustness of NR- VQA models, which may be the key for developing robust NR-VQA models. These findings present a promising direction for NR-V VQA. Future work may include the following aspects. Firstly, desired to develop the learning-based attack method, i.e., apply generator to produce perturbations to fool the NR-VQA models. Secondly, more characteristics, such as the information in the temporal domain, should be explored to improve the robustness of NR-VQA models.
10

---

Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grants 122 and 62002075, and in part by the Guangdong Basic and Applied Basic Research Foundation under Grant 2023A151501 1428.. The authors acknowledge the Network Center of Guangzhou University for providing HPC computing resources.
References
[1] 99Firms. Facebook video
[2] Y. Mao, M. Wang, S. Wang, and S. Kwong, "High efficiency rate control for versatile vidco coding based on composite cauchy distribution," IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 4. pp. 2371 1-244,,4, 2021.
[3] N. Carlini and D. Wagner, "Towards evaluating the robustness of neural networks." in IEEE Symposium on Security and Privacy. pp. 19-57,2017,
[4] C. Szegedy, W. Zaremba. I. Sutskever, J. Bruna, D. Erhan, J. Goodfellow, and R. Fergus, "Intriguing properties of neural networks." arXiv preprint arXiv:1312.6199, 2013.
2013.
[5] S. M. Moosavi-Dezfooli,, A. Fawzi, O. Fawzi, and P. Frossard, "Universal adversarial perturbations," in IEFF Conference on Computer Vision and Pattern Recognition, pp- 1765-1773, 2017.
[6] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok, "Synthesizing robust adversarial examples." in Interna- tional Conference on Machine Learning, pp. 293, 2018.
C. Laidlaw and S. Feizi, "Functional adversarial attacks," in Advances in Neural Information Processing Systems, pp. 10408- 10418, 2019.
[8] K. Mahmood, R. Mahmood, and M. V. Dijk, "On the robustness of vision transformers adversarial examples," in International Conference on Computer Vision, pp. 7838-7847, 2021.
[9] Z. Wei, J. Chen, Goldblum, Wu, Goldstein, and Y.-.. Jiang. "Towards transferable adversarial attacks on vision transformers," in AAAI Conference on Artificial Intelligence, vol. 36, pp. 2668-2678 2022.
[10] X. Yang. W.S. Ling, Z.K. Lu, E.P. Ong, and S.S. Yao, "Just noticeable distortion model and applications in video coding," " Signal Processing: Image Communication, vol. 20, no. 7, pp. 662-680, 2005.
[11] X. Dong and J. Wen, "A pixel-based outlier-free motion estimation algorithm for scalable video quality enhancement." Frontiers of Computer Science, vol. 9 pp. 729-740, 2015.
[12] J. Rony, L. G. Hafemann, I., Oliveira, B. Ayed, R. Sabourin, and E, Granger, "Decoupling direction and norm for efficient gradient-based Lo adversarial attacks and defenses," in IEEE Conference Computer Vision and Pattern Recognition, pp. 4322-4330, 2019.
[13] R. Pinot, R. Ettedgui, G. Rizk, Y. Chevaleyre, and J. Atif, "Randomization matters how to defend against strong adversarial attacks," " in International Conference on Machine Leariing, pp. 7717-7727, 2020.
[14] F. Croce and M. Hein, "Minimally distorted adversarial examples with fast adaptive boundary attack," in International Conference on Machine Learning, pp. 2196-2205, 2020.
[15] Z. Sinno and A. C. Bovik, "Large-scale study of perceptual video quality." IEEE Transactions on Image Processing, vol. 28, no. 2, pp. 612-627, 2019.
[16] J. Goodfellow, Shlens, and Szegedy, "Explaining and harnessing adversarial examples," " in Interna- tional Conference on Learning Representations, pp. 1-11, 2014.
[17] A. Kurakin, I. J. Goodfellow, and Bengio, "Adversarial examples the physical world," International Conference on Learning Representations, pp. 1-11, 2017.
[18] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, "The limitations of deep lcarning in adversarial settings," in IEEE European Symposium on Security and Privacy, pp. 372-387, 2016.
[19] N. Papernot, P. McDaniel, X. Wu. S. Jha, and A. Swami, "Distillation as a defense to adversarial per- turbations against deep neural networks," in IEEE Symposiun on Security and Privacy, pp. 582-597, 2016.

---

[20] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards deep learning models resistant to adversarial attacks." " In International Conference on Learning Reprexentations, pp. 1-23, 2018.
[21] N. Papernot, P. McDaniel, J. Goodfellow, Jha 7. B. Celik, and A. Swami, "Practical black-hox attacks against machine learning." in ACM Asia Conference on Computer and Communications Security, pp. 506-519, 2017.
[22] P. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J Hsieh, "Z00: Zeroth order optimization based black- box attacks to deep neural networks without training substitute models," in ACM Workshop on Artificial Intelligence and Security, pp. 15-26, 2017.
[23] A. Ilyas, L. Engstrom, A. Athalye, and J. Lin, "Black-box adversarial attacks with limited queries and information." in International Conference on Machine Learning, pp. 2137-2146, 2018.
[24] C. Guo, Gardner, You, A. G. Wilson, and K. Weinberger, "Simple black-hox adversarial attacks," in International Conference on Machine Learning, pp. 2484-2493, 2019.
[25] Y. L. L. Po, C. Cheung, X. Xu, L. Feng, F. Yuan, and K. Cheung, "No-reference video quality assessment with 3D shearlet transform and convolutional neural networks," IEEE Transactions on Circuits and Systems for Video Technology, vol. 26, no. 6. pp. 1044-1057, 2016.
[26] D. Li, T. Jiang, and M. Jiang. "Quality assessment of wn-the-wild videos," in ACM International Conference on Multimedia, pp.
2019.
[27] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in IEEE Conference on Computer Vision and Pottern Recognition, pp. 770-778, 2016.
[28] K. Cho, B. Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio, "Learning phrase repre- sentations using RNN encoder-decoder for statistical machine translation," arXivpreprint arXiv: 140.::::1100111111 2014.
[29] J. You and J. Korhonen, "Deep neural networks for no-reference video quality assessment," in IEFE International Conference on Image Processing, pp. 2349-2353, 2019.
[30] S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997.
[31] B. Li, W. Zhang, M. Tian, G. Zhai, and X. Wang, "Blindly assess quality of in-the-wild videos via quality-awar aware pre-training and motion perception," IEEE Transactions on Cireuits and Systems for Video Technology, vol. 32, no. 9. pp. 5944-5958, 2022.
[32] Y. Wang, Wang, Z. Yin, R. Gong, Wang, A. Liu, and X. Liu, "Generating transferable adversarial examples against vision transformers," in ACM International Conference on Multimedia, pp. 5181-5190, 2022.
[33] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, "A convnet for the 2020s," in IEEE Conference on Computer Vision and Pattern Recognition, pp. 11976-1 1196,, 2022.
[34] C. Feichtenhofer, H. Fan, J. Malik, and Hc, "SlowFast networks for video recognition," " in International Conference on Computer Vision, pp. 6201 6201-6210, 2019.
[35] F. Xing, Y.-G. Wang, H. Wang, L. Li, and G. Zhu, "QtarVQA: Space-time attention for video quality assessment," in IEEF International Conference on Image Processing, pp. 2326-2330, 2022.
[36] H, Wu, C, Chen, L, Liao, J, Hou, W, Sun, Q. Yan, and W. Lin, "DisCoVQA: Temporal inttortionccnnenn transformers for video quality assessment," JEEE Transactions on Circuits and Systems for Video Technology. to be published, doi: 10. 11111TTTT...223.22499999
[37] 7. Li and I.. Yang, "DCVQE: A hierarchical transformer for vidco quality assessment," in Asian Conference on Computer Vision, pp. 2562-2579,
2022. 2022.
[38] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
[39] D. Li, T. Jiang, and M. Jiang, "Unified quality assessment of in-the-wild vidcos with mixed datascts training," International Journal of Computer Vision, vol. 129, no. 4, pp. 2238-1257, 2021.
[40] A Zhang, Y.-G. Wang, W. Tang, L. Li, and S. Kwong, "HVS revisited: A comprehensive video quality assessment framework." arXivpreprint arXiv:2210.04158. 2022.
12

---

[41] H. Wu, C. Chen, J. Hou, L. Liao, A. Wang, W. Sun, Q. Yan, and W. Lin, "FAST-VQA: Efficient end-to- end video quality assessment with fragment sampling." in Earopean Conference Computer Vision, pp. 538-554, 2022.
[42] Z. Liu, J. Ning. Y. Cao, Y. Wei, Z. Zhang, Lin, and H. Hu, "Video swin transformer." in IEEE Conference on Computer Vision and Puttern Recognition, pp. 3202-3211,
[43] C. Xiao, Li, J.-Y. Zhu, W. He, M. Liu, and D. Song, "Generating adversarial examples with adversarial networks," in International Joint Conference on Artificial Intelligence, pp. 5905-3911 2018.
[44] W. Zhang, D. Li, X. Min, G. Zhai, G. Guo, X. Yang. and K. Ma, "Perceptual attacks of no-reference image quality models with -oope--o- in Advances Neural Information Processing Systems, pp. 2916-2929, 2022.
[45] Y. Wang. Inguva, and B. Adsumilli, "YouTube UGC dataset for video compression research," IEEE International Workshop on Multimedia Signal Processing, pp. 1-5,201 2019.
[46] Z. Ying, Mandal, Ghadiyaram, A. C. Bovik, "Patch-VQ: 'Pttciing Up" the video quality problem," in IEEE Conference on Computer Vision and Pattern Recognition, pp. 14019-14029, 2021.
[47] D. Kingma and J. Ba, "Adam: method for stochastic optimization." arXiv preprint 1xXvv11412.6980 2014.
[48] V. Hosu, Hahn, M. Jenadelch, Lin, Men, Szirányi, Li, and D. Saupe, "The konstanz natural video database (KoNViD-Ik)" in International Conference on Quality of Multimedia Experience (QoMEX), 1-6, 2017.
[49] A.-X Zhang and Y.-G. Wang, "Texture information boosts video quality assessment," in IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2050-2054, 2022.
[50] F. Shumitskaya, A. Antsiferova, and D. Vatolin. "Universal perturbation attack on differentiable no- reference image- and video-quality metrics," in British Machine Vision Conference, pp. 1-12, 2022.
13

---

A
Rescaling boundary
Due to the differences of the estimated quality scores' scale among different NR-VQA models, we rescale the boundaries of 0 and to the scale of the estimated quality scores for different NR-VQA models as
boundary mOS.mean
Scaled Scaled_bounde boundary
ESt.st + EST.mean,
(7)
MOS.std
where MOS.mean
mean and MOS.stc denote the mean and variance of the normalized MOS the selected 50 videos, respectively. EST.mean and EST.std denote the mean and variance of the estimated quality scores in the selected 50 videos, respectively. Note that in our proposed adversarial attack, the boundary is set according to a specific original video's MOS and pre-defined threshold determined by the median of MOS within the batch. In practice, these two conditions may not be available to the attacker, and the solutions are follows. The first condition is obtain specific video's ' MOS. reasonable to manually annotate the video, as did in the adversarial attack in classification task in the case of unaware of the ground-truth label. As an alternative approach, we can also apply a well-trained VQA model to obtain the estimated quality score, which can be regarded as the approximated MOS. The second condition is to obtain the threshold. We can apply the approaches in the first condition to obtain the approximate MOS for each video within the dataset, and then calculate the median value as the threshold. Besides, considering the workload for processing huge number of videos within the batch, the threshold can also be set to the average of the upper bound and lower bound of the scoring range for the dataset.
B Ablation study of perturbation on few frames
The adversarial effect with respect to the number of frames to be perturbed is investigated under both black-box and white-box settings. The experiments are conducted via attacking VSFA on the KoNViD-1k and LSVQ-Test datasets, and the results are given in Tables 8 and respectively. It can be observed that the proportion of perturbed frames decreases, the attack effect is degraded under both white-box and black-box settings. Specifically, due to the difficulties of black-box attack, there barely any attack effect when only 1/10 of frames are perturbed. By contrast, under the white-box setting, the NR-V SQ-VQA model can still be fooled when a small number of frames are perturbed.
Table 8: Performance evaluations for different
Table 9: Performance evaluations for different
ratios of perturbed frames under white-box case.
ratios of perturbed frames under black-bo -ox case.
KoNViD-IK 1k
LSVQ- Test
KoNViD- 1k
LSVQ- Test
Ratio
Ratio
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
SRCC
PLCC
0.7882
0.8106
0.7865
0.7972
0.7882
0.8116
0.7865
0.7972
1/10
-0.2692
-0.2332
-0.2262
-0.1813
1/10
0.6836
0.7511
0.7097
0.7378
1/5
-0.5980
-0.4896
-0.4985
-0.4373
1/5
0.6199
0.7127
0.6597
0.7113
1/2
-0.6318
-0.7585
-0.6069
-0.5992
1/2
0.4287
0.5447
0.5050
0.6013
all
-0.7198
-0.8135
-0.6723
-0.6765
all
-0.0305
0.0586
0.2386
0,2413
C Limitation and potential societal impact
In this paper. we mainly attack the VQA models in the spatial domain, which makes it difficult to achieve ideal attack effect against models with rich temporal information, such as BVQA-2022. Meanwhile, aptimization-basel attack strategies need to generate specific perturbations for each video, which have high requirements on computing resources. Secondly, according to our work, people could launch deliberate attack on VQA models, which may reduce the users' confidence of experiencing the VQA systems. In spite of this, we believe that in order to build a reliable and practical assessment system, it is of great necessity to evaluate their robustness. Therefore, this paper can raise awareness of vulnerability existing VQA models, which greatly outweighs its risk.
14


Reviews:
[{"review_id": "g6K0Lv0i8Z", "reviewer": "Reviewer_C8CZ", "summary": "The adversarial attacks against video quality assessment models are proposed in both white box and black box scenarios.   ", "questions": "1. Why $f_d(\\cdot)$ needs to take the test video as input?\n2. What are the motivations of Eq. (3)?\n3. What does the universal perturbation mean in the black box attack? ", "limitations": "N.A.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The problem is important and practical. \n2. The attempt to avoid subjective testing is a good try.\n3. The attacks seem to work as in Tables 1 and 2.", "weaknesses": "1. The authors claim that \"the problem of constraining the distortion below the JND threshold can be converted into the problem of restricting their pixel-level L2 norm below a rather strict threshold.\" A simple thought experiment would reveal that this is not true. Even if the threshold is set to $1/255$, we can simply use this budget to modify only a local region of pixels to be above JND.\n2. The proposed method resembles those presented in the image/video classification literature and thus lacks novelty.\n3. Why not use $\\ell_\\infty$-norm as the primary JND constraint?\n4. The JND constraint should be more mathematically or experimentally justified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We address your mentioned weakness and questions below.\n\n**Regarding the association of JND constraint in our work with $L_2$ and $L_\\infty$ norm:** Thanks for the insightful comment. The authors agree that in extreme situations, only applying $L_2$ norm may lead to modifications exceeding the JND in a local region, and $L_\\infty$ norm may be more appropriate to model the JND constraint.\nNote that in our original manuscript, both $L_2$ and $L_\\infty$ norm are utilized to restrict the perturbations below the JND threshold\nunder black-box setting. Specifically, in the optimization process, each frame is divided into small patches, which are added with a universal perturbation map once and only once whose element's value is $-\\gamma$ or $+\\gamma$.\nTherefore, the $L_\\infty$ norm of perturbations\nare mathematically guaranteed to be no greater than $\\gamma$.\nIn the case of white-box setting, we leverage the Projected Gradient Descent (PGD) to restrict the pixel-level $L_2$ norm. According to the comment, we supplement the experiments of\napplying PGD to restrict the $L_\\infty$ norm, and the results are given in Table 1.\nIt can be seen that the restrictions of $L_2$ or $L_\\infty$ bring similar attack effect under white-box setting. As suggested by the reviewer, we will emphasize the role of $L\\infty$ norm in our JND constraint throughout the paper.\n\nTable 1. Performance evaluations on VSFA.\n||SRCC||PLCC||\n|-|-|-|-|-|\n||KoNViD-1k|LIVE-VQC|KoNViD-1k|LIVE-VQC|\n|$L_2$ constraint|-0.7003|-0.7610|-0.7938|-0.7332|\n|$L_\\infty$ constraint|-0.7190|-0.7474|-0.8135|-0.7322|\n|Without attack|0.7882|0.7665|0.8106|0.7506|\n\nBesides, three experiments are conducted to justify the JND constraint in our method.\nFirstly, $L_\\infty$ norm is utilized to measure the visual quality of the adversarial videos. In the experiments of the original manuscript, the values of $L_\\infty$ norm under both white-box and black-box settings are given, which are 3/255 and 5/255, respectively. Secondly, we design a set of subjective experiments to verify whether the JND constraint can be held in our method. According to [1], the details of human subjective experiments are as follows. (1) *Data collection:* We collect 24 videos from the publicly available dataset KoNViD-1k. Then, these videos are divided into two groups for white-box and black-box attacks, respectively. In this way, one group of videos contains 12 original videos and 12 adversarial videos. (2) *Test methodology:* We recruit 15 human subjects. For each yes-no trial, subjects are shown an adversarial video and its corresponding original video, and then asked to select the original video.\nIn total, we collect 180 opinions for both white-box and black box settings.\nResults show that the probability of correctly selecting the original video is 48.3% and 53.3% under white-box and black-box settings, respectively. Such results are close to random guessing, demonstrating the imperceptibility of the perturbations. Thirdly, in the supplementary PDF, the example frames of the adversarial videos are given, wherein no modification traces can be seen.\n\n[1] W. Zhang et al., ``Perceptual attacks of no-reference image quality models with human-in-the-loop,'' in *NeurIPS,* 2022.\n\n**Regarding the novelty of the paper:** Thanks for the valuable comment. \nAdversarial attacks have been widely studied in classification task. However, there have been few investigations of adversarial attacks in Video Quality Assessment (VQA). Their main differences are: (1) The adversarial classification aims to fool the target model to predict a wrong label, which is a discrete value. In contrast, the adversarial VQA aims to fool the target model to output the score towards the direction farthest away from its MOS, which is a continuous value.\n(2) Due to the distinctions between different tasks, existing loss functions, such as the cross-entropy applied in adversarial classification are incapable of misleading the predicted score towards the expected direction.\n\nWe clarify the novelties of this work again from the perspective of imperceptibility, effectiveness, and efficiency as follows.\n(1) Adversarial attacks on NR-VQA models are first formulated as misleading the estimated quality score under the JND constraint. (2) A novel loss function called Score-Reversed Boundary Loss is designed to push the adversarial video's estimated quality score towards a specific boundary. (3) A patch-based random search method is designed for black-box attack, which can significantly improve the query efficiency in the spatial and temporal domains.  (4) This is the first work to perform the black-box adversarial attack on VQA models. Comprehensive experiments confirm the effectiveness of the proposed method. \n\n**Regarding the motivations of Eq. (3) and the input of $f_d(.)$ :** Thanks for the valuable comment. The motivations of Eq. (3) in the original manuscript are as follows.\nA well-performed adversarial attack on NR-VQA model should possess the capability of leading the estimated quality score far away from its MOS in a specific direction, i.e., misleading the NR-VQA model to assign high score to low-quality video, and vice versa. Therefore, $f_d(.)$ takes the test video as input and outputs the specific boundary based on the MOS of the test video. Afterwards, the perturbations are optimized based on the distance between the estimated quality score outputted by the target model and the specific boundary.\n\n**Regarding the universal perturbation in the black-box attack:** Thanks for the valuable comment. To reduce the query resources in the black-box attack, one universal perturbation map is added to all selected patches in all frames that are optimized in one query together. Specifically, the dimension of the universal perturbation is the same as the dimension of the selected patch, and the element of universal perturbation map is independently sampled from a discrete set $\\\\{-\\gamma,+\\gamma\\\\}$."}, {"review_id": "CA5VETD7Il", "reviewer": "Reviewer_mjyw", "summary": "The paper studies the adversarial robustness of No Reference Video Quality Assessment Algorithms. The paper introduces a new loss function, \"Score-Reversed Boundary Loss,\" and demonstrates its usefulness in designing adversarial attacks for NR VQA algorithms. Overall the paper is well written, contains adequate evaluations, and a decent amount of ablation studies. ", "questions": "Can strict L2 norms always result in upholding JND constraints?", "limitations": "1. The JND constraint is not based on the principles of Vision Science. \n", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Introduces the new loss function \"Score-Reversed Boundary Loss\" and demonstrate its usefulness in designing attacks for NR VQA algorithms\n2. Thorough experimentation of the proposed approach on multiple NR VQA algorithms. \n3. Decent amount of ablation study", "weaknesses": "1. Video Quality Assessment is a Perceptual Science, and JND is motivated by Vision Science principles. However, the authors base their work on the claim that strict L2 norms would be sufficient to uphold JND constraints. The work does not guarantee that the adversarial videos are under the JND constraint for a \"Human Eye.\" In this work, human eyes have not verified the adversarial videos are indeed under the JND constraint and the attacks are effective and imperceptible. The authors should have conducted some human study to verify their JND claim of using a strict L2 norm.\n2. Also, the hyperparameter T=1 set for black-box attack gives the highest scores but potentially results in some sort of temporal flickering. Something that can be verified only by human eyes and not an L2 norm constraint. \n\nMinor Typo :\n1. Line 178 pathes -> patches?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We address your mentioned weakness, questions, and limitations below. \n\n**Regarding JND constraint based on the principles of Vision Science:** Thanks for the valuable comment. In our work, the JND constraint is designed based on the principles of Vision Science, and the design philosophy is as follows.\nDue to the properties of retinal visual cells, the human visual system (HVS) has limited resolution and can only perceive changes larger than a certain threshold [1]. The JND can be regarded as the threshold, which represents the visibility of the HVS. Therefore, it is useful to introduce the JND threshold into the orientated signal processing systems [2]. However, JND is not a fixed value but can be influenced by many factors, including brightness, contrast between the background, and individual visual sensitivity [3]. Therefore, some fields convert it into some objective metrics for simple modeling, including PSNR in video coding [4], SSIM in image/video enhancement [5], and $L_p$ norm in adversarial attacks [6]. In our attack under black-box setting, both $L_2$ and $L_\\infty$ norm are utilized to restrict the perturbations below the JND threshold. In the case of white-box setting, we leverage the Projected Gradient Descent (PGD) to restrict the pixel-level $L_2$ norm. As suggested by the reviewer, we will emphasize the reasons of utilizing $L_p$ norm for modeling the JND constraint based on the principles of Vision Science in the text.\n\n\n[1] N. Jayant et al., ``Signal compression based on models of human perception,'' *Proceedings of the IEEE,* 1993.\n\n[2] J. Wu et al., ``A survey of visual just noticeable difference estimation,'' *Frontiers of Computer Science*, 2019.\n\n[3] J. Rovamo et al., ``Modelling contrast sensitivity as a function of retinal illuminance and grating area,'' *Vision Research,* 1994.\n\n[4] X. Yang et al., ``Just noticeable distortion model and its applications in video coding,'' *Signal Processing: Image Communication,* 2005.\n\n[5] X. Dong et al., ``A pixel-based outlier-free motion estimation algorithm for scalable video quality enhancement.'' *Frontiers of Computer Science,* 2015.\n\n[6] R. Pinot et al., ``Randomization matters how to defend against strong adversarial attacks,'' in *International Conference on Machine Learning,* 2020. \n\n**Regarding whether strict $L_2$ norm always upholds JND constraint:** Thanks for the insightful comment. The authors acknowledge that in extreme situations, only applying $L_2$ norm may lead to modifications exceeding the JND in a local region. \nTo address the issue, $L_\\infty$ norm can be introduced to model the JND constraint, which can limit each pixel's maximum modification value.\nNote that in our original manuscript, both $L_2$ and $L_\\infty$ norm are utilized to optimize the perturbations under black-box setting, which can more comprehensively uphold the JND constraint. Specifically, in the optimization process, each frame is divided into small patches, which are added with a universal perturbation map once and only once whose element's value is $-\\gamma$ or $+\\gamma$.\nTherefore, the $L_\\infty$ norm of perturbations\nis mathematically guaranteed to be no greater than $\\gamma$.  In the case of white-box setting, we leverage the Projected Gradient Descent (PGD) to restrict the pixel-level $L_2$ norm.\nWe supplement the experiments of\napplying PGD to restrict the $L_\\infty$ norm, and the results are given in Table 1. It can be seen that the restrictions of $L_2$ or $L_\\infty$ bring similar attack effect under white-box setting. In the experiments of the original manuscript, the values of $L_\\infty$ norm under both white-box and black-box settings are given, which are 3/255 and 5/255, respectively. As suggested by the reviewer, we will clarify the relation between $L_2$ norm and JND constraint throughout the paper.\n\nTable 1. Performance evaluations on VSFA.\n||SRCC||PLCC||\n|-|-|-|-|-|\n||KoNViD-1k|LIVE-VQC|KoNViD-1k|LIVE-VQC|\n|$L_2$ constraint|-0.7003|-0.7610|-0.7938|-0.7332|\n|$L_\\infty$ constraint|-0.7190|-0.7474|-0.8135|-0.7322|\n|Without attack|0.7882|0.7665|0.8106|0.7506|\n\n**Regarding the human subjective experiments:** Thanks for the valuable comment. As suggested by the reviewer, we design a set of subjective experiments to verify whether the JND constraint can be held in our method. According to [1], the details of human subjective experiments are as follows. (1) *Data collection:* We collect 24 videos from the publicly available dataset KoNViD-1k. Then, these videos are divided into two groups for white-box and black-box attacks, respectively. In this way, one group of videos contains 12 original videos and 12 adversarial videos. (2) *Test methodology:* We recruit 15 human subjects to participate in the subjective experiments. For each yes-no trial, subjects are shown an adversarial video and its corresponding original video, and then asked to select the original video.\nIn total, we collect 180 opinions for both white-box and black box settings.\nResults show that the probability of correctly selecting the original video is 48.3% and 53.3% under white-box and the black-box settings, respectively. Such results are close to random guessing, demonstrating that the adversarial videos generated by our method can achieve satisfied imperceptibility. In the supplementary PDF, the example frames of the adversarial videos are given, wherein no modification traces can be seen.\n\n[1] W. Zhang et al., ``Perceptual attacks of no-reference image quality models with human-in-the-loop,'' in *Advances in Neural Information Processing Systems,* 2022.\n\n**Regarding the minor typo in line 178:** Thanks for the kind comment. We have corrected it in the text."}, {"review_id": "Zj0MBrZ5LW", "reviewer": "Reviewer_xq3Y", "summary": "This paper proposes a patch-based random search method for white-box attacks and black-box attacks for NR-VQA models. A Score-Reversed Boundary Loss is proposed to push the adversarial video's estimated quality score far away from its MOS towards a specific boundary. And the robustness of NR-VQA models is evaluated.", "questions": "- What is the patch selection strategy for Line 8 in Algorithm 2? What is the generated strategy for $\\mathbf{m}_n$ in Line 10 in Algorithm 2? And what is the visibility of the perturbation of the attacked images?\n- Whether four NR-VQA models are strictly trained on the training set, and the samples to be attacked are selected in the test set? The re-trained progress is not mentioned in the paper.", "limitations": "No limitation and societal impact is discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper investigates the robustness of SOTA NR-VQA models against adversarial attacks.\n\n2. The Score-Reversed Boundary Loss gives a useful guide for the decrease of  SROCC and PLCC.", "weaknesses": "- The comparsion with existed methods should be contrained, like [Shumitskaya, BMVC, 2022].\n\n[Shumitskaya, BMVC, 2022] Ekaterina Shumitskaya, Anastasia Antsiferova, and Dmitriy S Vatolin. Universal perturbation attack on differentiable no-reference image- and video-quality metrics. In 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022. BMVA Press, 2022. URL https://bmvc2022.mpi-inf.mpg.de/0790.pdf.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We address your mentioned weakness, questions, and limitations below. \n\n**Regarding the comparison with [1]:** Thank you very much for reminding us of [1] which has been neglected in searching for the related works. We will include this paper in the related work in the revised manuscript.\nThe differences between our work and [1] mainly come from three aspects.\nFirstly, [1] applies white-box attack, while our method considers the more practical attack scenario of black-box attack.\nSecondly, the goals of adversarial attack on VQA models are different.\n[1] aims to merely increase the estimated quality score outputted by the target model. By contrast, our method aims to mislead the score towards the direction farthest away from its MOS and thus can cause more intense disturbance on VQA models, which is more in line with the requirement of adversarial attack.\nThirdly, [1] does not consider the visual quality during optimization, and thus the adversarial videos suffer from obvious artifacts. By contrast, to obtain satisfied imperceptibility, our method introduces the JND constraint during optimization. As a result, from the supplementary PDF, the adversarial videos are indistinguishable from their original counterparts by human eyes.\n \n[1] E. Shumitskaya et al., ``Universal perturbation attack on differentiable no-reference image- and video-quality metrics,'' in *British Machine Vision Conference*, 2022.\n\n**Regarding the patch selection strategy for Line 8 in Algorithm 2:** Thanks for the valuable comment. In Algorithm 2, the height and width of a video frame are denoted as $H$ and $W$, respectively. The height and width of a patch are denoted as $h$ and $w$, respectively. Therefore, a three-channel frame can be divided into $\\lfloor \\frac{H}{h} \\rfloor \\times \\lfloor \\frac{W}{w} \\rfloor \\times 3$ non-overlapping patches. All these patches within one frame would be selected once and only once in $N$ queries during optimization. Specifically, Line 8 in Algorithm 2 indicates that in each query, $ \\max \\left( \\lfloor \\lfloor \\frac{H}{h} \\rfloor \\times \\lfloor \\left. \\left. \\frac{W}{w} \\rfloor \\times 3 / N \\rfloor \\right. \\right. , 1 \\right)$ patches within one frame would be selected for perturbation.\n\n\n**Regarding $\\mathbf{m}_n$ in Algorithm 2:** Thanks for the valuable comment. In Algorithm 2, $\\mathbf{m}_n$ refers the universal perturbation in the $n$-th query.\nNote that one universal perturbation map is added to all selected patches in all frames that optimized in one query together.\nBy this means, the query resources can be largely reduced. Specifically, the dimension of the universal perturbation is the same as the dimension of the selected patch, and the element of universal perturbation map is independently sampled from a discrete set $\\\\{ -\\gamma,  +\\gamma \\\\}$.\n\n**Regarding the visibility of the perturbation of the attacked videos:** Thanks for the valuable comment. In the supplementary PDF, we select some example frames of the generated adversarial videos under both white-box and black-box settings, wherein no modification traces can be seen.\n\n**Regarding the re-trained process of the NR-VQA models:**\nThanks for the valuable comment. In the re-trained process, we download the authors' source code, re-train the model and produce adversarial videos according to the following settings.\nFor a specific dataset, the videos are randomly split into a training set, validation set, and testing set.\nThe VQA models are trained on the training set, and validated on the validation set.\nAfterwards, with the well-trained VQA models, the adversarial videos are generated on the testing set.\nAs suggested by the reviewer, we will add the description of the re-trained progress in the experimental setup.\n\n**Regarding the limitation and potential societal impact of our work:** Thanks for the valuable comment. The limitation and societal impact of our work are as follows. Firstly, our method mainly attacks the VQA models in the spatial domain, which makes it difficult to achieve an ideal attack effect against models with rich temporal information, such as BVQA-2022. Meanwhile, optimization-based attack strategies need to generate specific perturbations for each video, which have high requirements on computing resources. Secondly, according to our work, people could launch deliberate attack on VQA models, which may reduce the users’ confidence of experiencing the VQA systems. However, we believe that in order to build a reliable and practical assessment system, it is of great necessity to evaluate their robustness. Therefore, this paper can raise awareness of vulnerability in existing VQA models, which greatly outweighs its risk. As suggested by the reviewer, we will clarify our work's limitations and potential societal impact in the text for a clear illustration.\n"}, {"review_id": "mdYAVobJvI", "reviewer": "Reviewer_NoLq", "summary": "This paper studies the adversarial robustness of NR-VQA models. A new loss function called Score-Reversed Boundary Loss is designed to push the  estimated quality score of adversarial video far away from its MOS towards a specific boundary. It also proposes a patch-based random search method for black-box attack. The impercetibility of adversarial video is constrained by a L2-constraint.", "questions": "1, Is it possible use other FR-IQA as the fidelity measure as in [42]?\n\n2, Is it possible to only perturb a few frames (not all of them as done in the proposed method) to fool an NR-VQA models?", "limitations": "Not applicable.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ It establishes the first work on evaluating the adversarial robustness of NR-VQA models with reasonable motivations.\n+ Both white-box and black-box attack methods are designed.\n+ Rich experiments are conducted on mainstreaming VQA datasets to evaluate the adversarial robustness of multiple NR-VQA models.\n+ Insightful analysis of the adversarial robustness of specific design choices of NR-VQA models are provided.\n", "weaknesses": "- The definitons of f_{e} and f_{d} are a bit confusing. In my view, the parameters of NR-VQA models are not changed during the adversarial attack, so it would be better to simply define a NR-VQA model as f, then f(I) and f(I_{adv}) will the estiamted quality of the original and perturbed video, respectively. It's better to denote the disturbed score using another symbol without subscript. \n\n- The disturbed score is actually a target score the median MOS of videos from a VQA dataset is set as the threshold to determine whether a video is of high quality or low quality. However, there won't be a dataset with ground-truth MOS before the adversarial attack in may cases. It is not clear how to set the boundary given an arbitrary video with or without MOS in practice.\n\n- The L2-norm constraint is used to ensure the invisibility of perturbation. Why not using other L-norm contrainst is not explained. \n\n-  Do the authors rescale the quality predictions of all NR-VQA models within the range [0, 1] using nonlinear mapping?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We address your mentioned weakness and questions below. In this response, the performance of VQA models is evaluated by SRCC, PLCC, and $R$. Due to the limited space in the response, only SRCC is listed in the following Tables. Note that all these three metrics exhibit similar results in our experiment. \n\n**Regarding the definitons of $f_e$ and $f_d$ :** Thanks for the valuable comment. To avoid misunderstanding, $f_e$ is modified as $f$, which denotes the estimated quality score of the target VQA model. And $f_d$ is modified as $b$, which denotes the disturbed quality score.\n\n**Regarding how to set the boundary in practice:** Thanks for the insightful comment. In our proposed adversarial attack, the boundary is set according to a specific original video's MOS and a pre-defined threshold determined by the median of MOS within the batch. In practice, these two conditions may not be available to the attacker, and the solutions are as follows. The first condition is to obtain a specific video's MOS. It is reasonable to manually annotate the video, as it did in the adversarial attack in classification task in the case of unaware of the ground-truth label. As an alternative approach, we can also apply a well-trained VQA model to obtain the estimated quality score, which can be regarded as the approximated MOS. The second condition is to obtain the threshold. We can apply the approaches in the first condition to obtain the approximate MOS for each video within the dataset, and then calculate the median value as the threshold.\nBesides, considering the workload for processing huge number of videos within the batch, the threshold can also be set to the average of the upper bound and lower bound of the scoring range for the dataset.\n\n**Regarding using other $L_p$ norm to ensure the invisibility of perturbation:** Thanks for the insightful comment. In our original manuscript, both $L_2$ and $L_\\infty$ norm are utilized to ensure the imperceptibility of perturbation under black-box setting. Specifically, in the optimization process, each frame is divided into small patches, which are added with a universal perturbation map once and only once whose element's value is $-\\gamma$ or $+\\gamma$.\nTherefore, the $L_\\infty$ norm of perturbations is mathematically guaranteed to be no greater than $\\gamma$.\nIn the case of white-box setting, we leverage the Projected Gradient Descent (PGD) to restrict the pixel-level $L_2$ norm.\nWe supplement the experiments of\napplying PGD to restrict the $L_\\infty$ norm, and the results are given in Table 1.\nIt can be seen that the restrictions of $L_2$ or $L_\\infty$ bring similar attack effect under white-box setting. In the experiments of the original manuscript, the values of $L_\\infty$ norm under both white-box and black-box settings are given, which are 3/255 and 5/255, respectively. Besides, in the supplementary PDF, the example frames of the adversarial videos are given, wherein no modification traces can be seen. As suggested by the reviewer, we will more clearly illustrate the application of different $L_p$ norm in ensuring the invisibility of perturbation in the paper.\n\nTable 1. Performance evaluations on VSFA.\n||SRCC||\n|-|-|-|\n||KoNViD-1k|LIVE-VQC|\n|$L_2$ constraint|-0.7003|-0.7610|\n|$L_\\infty$ constraint|-0.7190|-0.7474|\n|Without attack|0.7882|0.7665|\n\n**Regarding rescaling the boundaries for different NR-VQA models:** Thanks for the valuable comment. We rescale the boundaries of 0 and 1 to the scale of the estimated quality scores for different NR-VQA models as follows.\nFirstly, given a video dataset, we sample a video batch to launch adversarial attack.\nThen, we normalize each sampled video's  MOS into [0,1], and then calculate the mean ($MOS.mean$) and variance ($MOS.std$) according to all normalized MOS within the batch.\nAfterwards, given this video batch and a target NR-VQA model, we obtain the estimated quality scores, and then calculate the mean ($EST.mean$) and variance ($EST.std$) according to all estimated scores within the batch.\nFinally, the boundaries can be rescaled from the scale of the normalized MOS to the scale of the estimated quality score as\n$$scaled\\ boundary=\\frac{boundary-MOS.mean}{MOS.std}*EST.std+EST.mean$$\nwhere the boundary equals 0 or 1. For clear illustration, we will clarify the rescale strategy in the experimental setup.\n\n**Regarding using other FR-IQA as the fidelity measure:** Thanks for the valuable comment. As suggested by the reviewer, the attack effect using other FR-IQA as the fidelity measure is verified under the white-box setting. The experiments are conducted via attacking VSFA, and the results are given in Table 2. It can be observed that using other FR-IQA as the fidelity measure brings a similar attack effect to our method.\n\nTable 2. Performance evaluations on VSFA.\n||SRCC||\n|-|-|-|\n|FR-IQA|KoNViD-1k|LIVE-VQC|\n|Chebyshev|-0.7120|-0.7409|\n|SSIM|-0.7100|-0.7527|\n|LPIPS|-0.6930|-0.7418|\n|DISTS|-0.6989|-0.7501|\n|None|-0.7003|-0.7610|\n\n**Regarding only perturbing a few frames to fool an NR-VQA model:** Thanks for the valuable comment. Adversarial effect with respect to the number of frames to be perturbed is investigated. The experiments are conducted via attacking VSFA, and the results are given in Tables 3 and 4. It can be observed that there is barely any attack effect when only 1/10 of frames are perturbed under black-box setting. By contrast, under the white-box setting, the NR-VQA model can still be fooled when a small number of frames are perturbed.\n\nTable 3. Performance evaluations under white-box setting.\n||SRCC||\n|-|-|-|\n|Perturbation ratio|KoNViD-1k|LIVE-VQC|\n|0|0.7882|0.7665|\n|1/10|-0.2692|-0.1872|\n|1/5|-0.5980|-0.4757|\n|1/2|-0.6318|-0.6374|\n|all|-0.7003|-0.7610|\n\nTable 4. Performance evaluations under black-box setting.\n||SRCC||\n|-|-|-|\n|Perturbation ratio|KoNViD-1k|LIVE-VQC|\n|0|0.7882|0.7665|\n|1/10|0.6836|0.6403|\n|1/5|0.6199|0.5604|\n|1/2|0.4287|0.2441|\n|all|-0.0305|-0.1605|"}]

Author rebuttal:
According to reviewers C8CZ, mjyw, xq3Y, and NoLq, the examples of the original video frames and the corresponding adversarial video frames are presented in the supplementary PDF under both white-box and black-box settings.

Meta-review: The paper received uniformly positive ratings from the reviewers. The authors' rebuttal further clarifies various concerns of the reviewers. All reviewers are convinced with its technical novelty and contributions. The AC reads the reviews and discussions and recommends to accept the paper.
Decision: Accept (spotlight)