SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By means of applying the patch-based random search method, the constraint of JND can be formulated under  $L_{\infty}$  or pixel-level  $L_{2}$  norm. Firstly, the  $L_{\infty}$  norm between the original and adversarial videos is  $\gamma$ , which is the maximum modification range of each element in the adversarial video. Secondly, the frame-level  $L_{2}$  norm between the original and adversarial videos can be formulated as
$$
\sqrt {H \times W \times 3 \times L _ {2} ^ {2}} = \| \sum_ {n = 1} ^ {N} \gamma \mathbf {p} _ {n} \| _ {2} = \gamma \| \sum_ {n = 1} ^ {N} \mathbf {p} _ {n} \| _ {2} \leq \gamma \sqrt {N \times Z \times h \times w}, \tag {4}
$$
where the last inequality holds as different patches are selected once and only once in different rounds, and the last equality holds when all patches are injected with perturbations. And  $\gamma$  is set as 5/255 in black-box attack. Therefore, the pixel-level  $L_{2}$  norm can be deduced that
$$
L _ {2} \leq \sqrt {\frac {\gamma^ {2} \times N \times Z \times h \times w}{H \times W \times 3}} = \sqrt {\frac {\gamma^ {2} \times \frac {H}{h} \times \frac {W}{w} \times 3 \times h \times w}{H \times W \times 3}} = \gamma . \tag {5}
$$
NR-VQA models and datasets: Four representative NR-VQA models are tested in the experiments, including VSFA [26], MDTVSFA [39], TiVQA [49], and BVQA-2022 [31]. The experiments are
Table 1: Performance evaluations under white-box setting with  $L_{2}$  norm constraint. Here, the performance before attack is marked in gray values within parentheses.
(a) Performance evaluations on VSFA [26] and MDTVSFA [39].
Metric	SRCC	PLCC	R defined in Eq. (6)
Model	VSFA	MDTVSFA	VSFA	MDTVSFA	VSFA	MDTVSFA
KoNViD-1k	-0.7198 (0.7882)	-0.7380 (0.8003)	-0.8135 (0.8106)	-0.8212 (0.8074)	-0.2507	-0.0368
LIVE-VQC	-0.7518 (0.7665)	-0.7825 (0.7908)	-0.7322 (0.7506)	-0.7729 (0.8091)	-0.4963	-0.3121
YouTube-UGC	-0.7071 (0.7492)	-0.7223 (0.7683)	-0.8018 (0.7721)	-0.7939 (0.7950)	-0.3585	-0.2883
LSVQ-Test	-0.6723 (0.7865)	-0.6623 (0.8136)	-0.6765 (0.7972)	-0.7906 (0.8125)	0.2692	0.4252
LSVQ-Test-1080p	-0.7847 (0.7040)	-0.7573 (0.7114)	-0.7830 (0.6868)	-0.7885 (0.7009)	0.0527	0.2589
(b) Performance evaluations on TiVQA [49] and BVQA-2022 [31].
Metric	SRCC	PLCC	R defined in Eq. (6)
Model	TiVQA	BVQA-2022	TiVQA	BVQA-2022	TiVQA	BVQA-2022
KoNViD-1k	-0.7212 (0.8046)	-0.7371 (0.8427)	-0.8204 (0.8377)	-0.8233 (0.8469)	-0.2472	-0.0579
LIVE-VQC	-0.6643 (0.8179)	-0.6705 (0.8698)	-0.7352 (0.8067)	-0.6834 (0.8457)	-0.2144	-0.0042
YouTube-UGC	-0.6179 (0.7907)	-0.6696 (0.8145)	-0.7356 (0.8177)	-0.6177 (0.8500)	0.0574	0.1118
LSVQ-Test	-0.7457 (0.8331)	-0.7679 (0.8613)	-0.7898 (0.8343)	-0.8063 (0.8531)	0.0248	0.0693
LSVQ-Test-1080p	-0.7372 (0.7478)	-0.7088 (0.7875)	-0.7249 (0.7670)	-0.7569 (0.7799)	0.1979	0.2037
Table 2: Performance evaluations under white-box setting with  $L_{\infty}$  norm constraint. Here, the performance before attack is marked in gray values within parentheses.
(a) Performance evaluations on VSFA [26] and MDTVSFA [39].
Metric	SRCC	PLCC	R defined in Eq. (6)
Model	VSFA	MDTVSFA	VSFA	MDTVSFA	VSFA	MDTVSFA
KoNViD-1k	-0.7155 (0.7882)	-0.7296 (0.8003)	-0.8139 (0.8106)	-0.8219 (0.8074)	-0.2521	-0.0419
LIVE-VQC	-0.7526 (0.7665)	-0.7766 (0.7908)	-0.7323 (0.7506)	-0.7720 (0.8091)	-0.4968	-0.3129
YouTube-UGC	-0.7013 (0.7492)	-0.7238 (0.7683)	-0.8025 (0.7721)	-0.7946 (0.7950)	-0.3669	-0.2974
LSVQ-Test	-0.6601 (0.7865)	-0.6569 (0.8136)	-0.6824 (0.7972)	-0.7949 (0.8125)	0.2551	0.4086
LSVQ-Test-1080p	-0.7864 (0.7040)	-0.7438 (0.7114)	-0.7833 (0.6868)	-0.7878 (0.7009)	0.0528	0.2533
(b) Performance evaluations on TiVQA [49] and BVQA-2022 [31].
Metric	SRCC	PLCC	R defined in Eq. (6)
Model	TiVQA	BVQA-2022	TiVQA	BVQA-2022	TiVQA	BVQA-2022
KoNViD-1k	-0.7419 (0.8046)	-0.7265 (0.8427)	-0.8230 (0.8377)	-0.8224 (0.8469)	-0.2521	-0.0708
LIVE-VQC	-0.6833 (0.8179)	-0.6712 (0.8698)	-0.7425 (0.8067)	-0.6867 (0.8457)	-0.2214	-0.0229
YouTube-UGC	-0.6034 (0.7907)	-0.6920 (0.8145)	-0.7199 (0.8177)	-0.6889 (0.8500)	0.0479	0.0817
LSVQ-Test	-0.7591 (0.8331)	-0.7769 (0.8613)	-0.7935 (0.8343)	-0.8093 (0.8531)	0.0178	0.0533
LSVQ-Test-1080p	-0.7351 (0.7478)	-0.7128 (0.7875)	-0.7442 (0.7670)	-0.7613 (0.7799)	0.1875	0.1876
conducted on mainstream video datasets including KoNViD-1k [48], LIVE-VQC [15], YouTube-UGC [45], and LSVQ [46]. Specifically, LSVQ is further divided into two datasets according to their resolutions. Therefore, five datasets are applied in the experiments. For each of these five datasets, 50 videos are randomly selected to evaluate the effectiveness of adversarial attacks.
Evaluation metrics: The performance of NR-VQA model is evaluated by Spearman Rank order Correlation Coefficient (SRCC) and Pearson's Linear Correlation Coefficient (PLCC). Specifically, SRCC is used to measure the monotonic relationship between the predicted quality scores and the ground-truth quality scores, while PLCC is used to measure the accuracy of the predictions. Note that both SRCC and PLCC are evaluated on a batch of videos. However, in real attack scenario, the attacker may merely launch the adversarial attack on a few samples. In this case, the change range of SRCC and PLCC could be rather small, and thus these two metrics may not accurately reflect the attack effect. Therefore, we follow [44] to define a metric on a single video, which is calculated as
Table 3: Performance evaluations under black-box setting. Here, the performance before attack is marked in gray values within parentheses.
(a) Performance evaluations on VSFA [26] and MDTVSFA [39].
Metric	SRCC	PLCC	R defined in Eq. (6)
Model	VSFA	MDTVSFA	VSFA	MDTVSFA	VSFA	MDTVSFA
KoNViD-1k	-0.0305 (0.7882)	0.0261 (0.8003)	0.0586 (0.8106)	0.1235 (0.8074)	1.6573	1.7587
LIVE-VQC	-0.1605 (0.7665)	-0.0074 (0.7908)	-0.0132 (0.7506)	0.0807 (0.8091)	1.3233	1.4391
YouTube-UGC	-0.2231 (0.7492)	-0.0646 (0.7683)	-0.1017 (0.7721)	0.0532 (0.7950)	1.5499	1.5080
LSVQ-Test	0.2386 (0.7865)	-0.0706 (0.8136)	0.2413 (0.7972)	-0.0384 (0.8125)	2.3545	2.1352
LSVQ-Test-1080p	0.2351 (0.7040)	-0.1339 (0.7114)	0.2840 (0.6868)	-0.0288 (0.7009)	2.1839	1.8592
(b) Performance evaluations on TiVQA [49] and BVQA-2022 [31].
Metric	SRCC	PLCC	R defined in Eq. (6)
Model	TiVQA	BVQA-2022	TiVQA	BVQA-2022	TiVQA	BVQA-2022
KoNViD-1k	-0.4992 (0.8046)	0.1978 (0.8427)	-0.4099 (0.8377)	0.2215 (0.8469)	1.2458	2.0480
LIVE-VQC	-0.0762 (0.8179)	0.4571 (0.8698)	0.0452 (0.8067)	0.4778 (0.8457)	1.3234	2.2061
YouTube-UGC	-0.1357 (0.7907)	0.3951 (0.8145)	-0.0087 (0.8177)	0.4306 (0.8500)	1.3478	2.1072
LSVQ-Test	-0.3343 (0.8331)	0.3355 (0.8613)	-0.2161 (0.8343)	0.4240 (0.8531)	1.8086	2.2269
LSVQ-Test-1080p	-0.1414 (0.7478)	0.3917 (0.7875)	-0.0343 (0.7670)	0.4678 (0.7799)	1.3327	2.0796
the average logarithmic ratio between the expected change in quality prediction and the actual change in quality prediction caused by adversarial video as
$$
R = \frac {1}{S} \sum_ {i = 1} ^ {S} \log \left(\frac {\left| f \left(\left(\left\{\mathbf {I} _ {x} \right\} _ {x = 1} ^ {X}\right) _ {i}\right) - b \left(\left(\left\{\mathbf {I} _ {x} \right\} _ {x = 1} ^ {X}\right) _ {i}\right) \right|}{\left| f \left(\left(\left\{\mathbf {I} _ {x} \right\} _ {x = 1} ^ {X}\right) _ {i}\right) - f \left(\left(\left\{\mathbf {I} _ {x} ^ {\mathrm {a d v}} \right\} _ {x = 1} ^ {X}\right) _ {i}\right) \right|}\right), \tag {6}
$$
where the definition of  $f$  and  $b$  can be referred to Section 3.1,  $S$  denotes the number of videos in the test set. A higher value of  $R$  indicates better model robustness. With the above three metrics, the prediction accuracy and robustness against adversarial attacks can be comprehensively evaluated.
Implementation details: We download the authors' source code [26; 39; 49; 31], re-train the models and produce adversarial videos on the testing set. Since the estimated quality scores' scale among different NR-VQA models is different, we need to rescale the boundary. The rescaling details can be referred to in the appendix. For a specific dataset, the median of the MOS of the selected 50 videos is regarded as the threshold to decide whether a video is of high quality or low quality in Eq. (3). As for white-box attack, Adam optimizer [47] is applied, wherein the step size  $\beta$  is initialized as  $3 \times 10^{-4}$ .  $K$  and  $T$  are set to 30 and 1. The pixel-level  $L_{2}$  norm of perturbations is constrained within 1/255. As for black-box attack,  $N$ ,  $T$ , and  $\gamma$  are set to 300, 1, 5/255. Both  $h$  and  $w$  are set to 56.
4.2 Performance evaluations under white-box setting
In this part, the performance of different NR-VQA models is evaluated under white-box setting, and the results under both pixel-level  $L_{2}$  and  $L_{\infty}$  norm constraint are given in Tables 1 and 2, respectively. It can be observed that the State-of-the-Art (SOTA) NR-VQA models are completely disabled by adversarial attack. Specifically, such attack under both pixel-level  $L_{2}$  and  $L_{\infty}$  norm constraint bring similar attack effects and lead to performance degradation of  $194\%$  and  $195\%$  on SRCC averaged on five datasets, demonstrating the vulnerabilities of these NR-VQA models. Meanwhile, the  $R$  values on all datasets approximate to 0, indicating that the adversarial attack with our proposed Score-Reversed Boundary Loss can push the estimated quality score towards the target boundary. Note that in white-box setting, the pixel-level  $L_{2}$  and  $L_{\infty}$  norm of perturbations are constrained within 1/255 and 3/255, indicating that the degree of perturbation is kept on an extremely low level.
4.3 Performance evaluations under black-box setting
In this part, the performance of different NR-VQA models is evaluated under black-box setting, and the results are given in Table 3. It can be observed that the SRCC and PLCC of the attacked
Under white-box setting
Under black-box setting
Figure 3: The original video frames and the corresponding adversarial video frames.
(a) White-box.
Figure 4: The attack effect on estimated quality scores ((a) and (b)) and the query efficiency with increasing iterations and number of queries ((c) and (d)).
(b) Black-box.
(c) White-box.
(d) Black-box.
VSFA, MDTVSFA, and TiVQA are around 0, indicating that the attack effect on these models is quite remarkable. Although the attack effect on BVQA-2022 is less significant, it still brings performance degradation of 0.6449 on SRCC and 0.6254 on PLCC on the KoNViD-1k dataset. Such results indicate that the current VQA systems are not robust against adversarial attacks, and can be severely disturbed in the black-box setting. Note that according to Eq. (5), the theoretical upper bound of the pixel-level  $L_{2}$  norm and  $L_{\infty}$  norm of the proposed patch-based random search method are both 5/255. Interestingly, in practical deployment, we find that only about 3/5 patches are injected with perturbations, and the pixel-level  $L_{2}$  norm is around 3/255. In Fig. 3, the examples of the original video frames and the corresponding adversarial video frames under both white-box and black-box settings are presented, which are indistinguishable from their original counterparts by human eyes.
In this part, ablation studies are conducted from four aspects. Firstly, the effectiveness of the proposed Score-Reversed Boundary Loss is verified under black-box and white-box settings. The experiments are conducted via attacking VSFA on the KoNViD-1k dataset, and the results are given in Fig. 4. From Fig. 4(a) and 4(b), it can be seen that minimizing the Score-Reversed Boundary Loss achieves satisfied attack effect. Specifically, in the white-box setting, the quality scores estimated by the target NR-VQA model basically reach the target boundary. From Fig. 4(c) and 4(d), it can be observed that compared with the MSE Loss, the Score-Reversed Boundary Loss shows superiority in the query efficiency. In particular, in the white-box setting, the adversarial attack with the Score-Reversed Boundary Loss can completely disable the NR-VQA model with just a few iterations.
Table 4: Performance evaluations of SRCC for patch-based and pixel-based methods.
n-th query	KoNViD-1k	LSVQ-Test
patch	pixel	patch	pixel
100	0.5390	0.7850	0.4392	0.7793
300	-0.0305	0.7831	0.2386	0.7747
500	-	0.7828	-	0.7738
1000	-	0.7823	-	0.7575
Table 6: Performance evaluations of motion embeddings on BVQA-2022.
Motion embeddings	KoNViD-1k	LSVQ-Test
SRCC	PLCC	SRCC	PLCC
×	-0.1779	-0.1465	0.1917	0.2641
✓	0.1978	0.2215	0.3355	0.4240
Table 5: Performance evaluations of SRCC with different values of  $T$  .
T	KoNViD-1k	LSVQ-Test
White	Black	White	Black
1	-0.7003	-0.0305	-0.6594	0.2386
2	-0.6972	0.1318	-0.6667	0.2579
4	-0.6747	0.2121	-0.6715	0.3171
8	-0.6964	0.2747	-0.6768	0.3482
Table 7: Performance evaluations of motion embeddings on VSFA.
Motion embeddings	KoNViD-1k	LSVQ-Test
SRCC	PLCC	SRCC	PLCC
×	-0.0305	0.0586	0.2386	0.2413
✓	0.1858	0.2687	0.4358	0.4536
Secondly, our patch-based random search method is compared with the pixel-based random search method SimBA [24] under black-box setting. The experiments are conducted via attacking VSFA on the KoNViD-1k and LSVQ datasets, and the results are given in Table 4. We can see that compared with the pixel-based method, our method shows significant improvements in both attack effect and query efficiency. Specifically, our method merely consumes 300 queries to completely disable the NR-VQA model, while the pixel-based method barely has any attack effect with 1000 queries.
Thirdly, the number of frames optimized in one round, i.e.,  $T$ , is investigated under both black-box and white-box settings. The experiments are conducted via attacking VSFA on the KoNViD-1k and LSVQ datasets, and the results are given in Table 5. From the perspective of the white-box setting, different values of  $T$  have little impact on the attack effect. However, in the black-box setting, smaller  $T$  corresponds to stronger attack effect. The reason is that different frames within one video have different characteristics, and thus optimizing the perturbations with smaller  $T$  can bring the potential of their individual differences to full play. Considering the attack effect,  $T$  is set to 1 in our method.
Fourthly, the impact of motion embeddings on the robustness of the model is explored under black-box setting. Note that BVQA-2022 applies motion embeddings, while VSFA does not. In Table 6, the original BVQA-2022 and the BVQA-2022 without motion embeddings are compared. In Table 7, the original VSFA and the VSFA with motion embeddings are compared. It can be observed that the motion embeddings play an important role in improving the robustness of NR-VQA models against adversarial attacks. Specifically, the motion embeddings bring SRCC performance improvement of 0.2598 for BVQA-2022 and 0.2068 for VSFA. One possible reason is that the motion embeddings correspond to the inter-frame information in temporal domain. Therefore, the adversarial impact of perturbations added within frames may be weakened, resulting in an improvement of robustness.
In this paper, we fully investigate the robustness of NR-VQA models against adversarial attacks, and propose the Score-Reversed Boundary Loss for the attack problem. Extensive experiments have been conducted under both white-box and black-box settings, and the following conclusions can be drawn. Firstly, the existing SOTA NR-VQA models are vulnerable to adversarial attacks, which brings great challenges to build reliable and practical assessment systems. Secondly, the attack effect and query efficiency are influenced by different components, including the Score-Reversed Boundary Loss, the patch-based random search method, and the number of frames optimized in one round. Thirdly, motion embeddings have a positive impact on increasing the robustness of NR-VQA models, which may be the key for developing robust NR-VQA models. These findings present a promising direction for NR-VQA. Future work may include the following aspects. Firstly, it is desired to develop the learning-based attack method, i.e., apply a generator to produce perturbations to fool the NR-VQA models. Secondly, more characteristics, such as the information in the temporal domain, should be explored to improve the robustness of NR-VQA models.
This work was supported in part by the National Natural Science Foundation of China under Grants 62272116 and 62002075, and in part by the Guangdong Basic and Applied Basic Research Foundation under Grant 2023A1515011428. The authors acknowledge the Network Center of Guangzhou University for providing HPC computing resources.
[41] H. Wu, C. Chen, J. Hou, L. Liao, A. Wang, W. Sun, Q. Yan, and W. Lin, “FAST-VQA: Efficient end-to-end video quality assessment with fragment sampling,” in European Conference on Computer Vision, pp. 538–554, 2022.
[42] Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin, and H. Hu, "Video swim transformer," in IEEE Conference on Computer Vision and Pattern Recognition, pp. 3202-3211, 2022.
[43] C. Xiao, B. Li, J.-Y. Zhu, W. He, M. Liu, and D. Song, "Generating adversarial examples with adversarial networks," in International Joint Conference on Artificial Intelligence, pp. 3905-3911, 2018.
[44] W. Zhang, D. Li, X. Min, G. Zhai, G. Guo, X. Yang, and K. Ma, “Perceptual attacks of no-reference image quality models with human-in-the-loop,” in Advances in Neural Information Processing Systems, pp. 2916-2929, 2022.
[45] Y. Wang, S. Inguva, and B. Adsumilli, "YouTube UGC dataset for video compression research," in IEEE International Workshop on Multimedia Signal Processing, pp. 1-5, 2019.
[46] Z. Ying, M. Mandal, D. Ghadiyaram, and A. C. Bovik, "Patch-VQ: 'Patching Up' the video quality problem," in IEEE Conference on Computer Vision and Pattern Recognition, pp. 14019-14029, 2021.
[47] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2014.
[48] V. Hosu, F. Hahn, M. Jenadeleh, H. Lin, H. Men, T. Szirányi, S. Li, and D. Saupe, “The konstanz natural video database (KoNViD-1k)," in International Conference on Quality of Multimedia Experience (QoMEX), pp. 1-6, 2017.
[49] A.-X. Zhang and Y.-G. Wang, "Texture information boosts video quality assessment," in IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2050-2054, 2022.
[50] E. Shumitskaya, A. Antsiferova, and D. Vatolin. "Universal perturbation attack on differentiable no-reference image- and video-quality metrics," in British Machine Vision Conference, pp. 1-12, 2022.
A Rescaling boundary
Due to the differences of the estimated quality scores' scale among different NR-VQA models, we rescale the boundaries of 0 and 1 to the scale of the estimated quality scores for different NR-VQA models as
$$
\text {S c a l e d} = \frac {\text {b o u n d a r y} - \text {M O S . m e a n}}{\text {M O S . s t d}} * \text {E S T . s t d} + \text {E S T . m e a n}, \tag {7}
$$
where MOS.mean and MOS.std denote the mean and variance of the normalized MOS in the selected 50 videos, respectively. EST.mean and EST.std denote the mean and variance of the estimated quality scores in the selected 50 videos, respectively. Note that in our proposed adversarial attack, the boundary is set according to a specific original video's MOS and a pre-defined threshold determined by the median of MOS within the batch. In practice, these two conditions may not be available to the attacker, and the solutions are as follows. The first condition is to obtain a specific video's MOS. It is reasonable to manually annotate the video, as it did in the adversarial attack in classification task in the case of unaware of the ground-truth label. As an alternative approach, we can also apply a well-trained VQA model to obtain the estimated quality score, which can be regarded as the approximated MOS. The second condition is to obtain the threshold. We can apply the approaches in the first condition to obtain the approximate MOS for each video within the dataset, and then calculate the median value as the threshold. Besides, considering the workload for processing huge number of videos within the batch, the threshold can also be set to the average of the upper bound and lower bound of the scoring range for the dataset.
B Ablation study of perturbation on a few frames
The adversarial effect with respect to the number of frames to be perturbed is investigated under both black-box and white-box settings. The experiments are conducted via attacking VSFA on the KoNViD-1k and LSVQ-Test datasets, and the results are given in Tables 8 and 9, respectively. It can be observed that as the proportion of perturbed frames decreases, the attack effect is degraded under both white-box and black-box settings. Specifically, due to the difficulties of black-box attack, there is barely any attack effect when only 1/10 of frames are perturbed. By contrast, under the white-box setting, the NR-VQA model can still be fooled when a small number of frames are perturbed.
Table 8: Performance evaluations for different ratios of perturbed frames under white-box case.
Ratio	KoNViD-1k	LSVQ-Test
SRCC	PLCC	SRCC	PLCC
0	0.7882	0.8106	0.7865	0.7972
1/10	-0.2692	-0.2332	-0.2262	-0.1813
1/5	-0.5980	-0.4896	-0.4985	-0.4373
1/2	-0.6318	-0.7585	-0.6069	-0.5992
all	-0.7198	-0.8135	-0.6723	-0.6765
Table 9: Performance evaluations for different ratios of perturbed frames under black-box case.
Ratio	KoNViD-1k	LSVQ-Test
SRCC	PLCC	SRCC	PLCC
0	0.7882	0.8106	0.7865	0.7972
1/10	0.6836	0.7511	0.7097	0.7378
1/5	0.6199	0.7127	0.6597	0.7113
1/2	0.4287	0.5447	0.5050	0.6013
all	-0.0305	0.0586	0.2386	0.2413
C Limitation and potential societal impact
In this paper, we mainly attack the VQA models in the spatial domain, which makes it difficult to achieve an ideal attack effect against models with rich temporal information, such as BVQA-2022. Meanwhile, optimization-based attack strategies need to generate specific perturbations for each video, which have high requirements on computing resources. Secondly, according to our work, people could launch deliberate attack on VQA models, which may reduce the users' confidence of experiencing the VQA systems. In spite of this, we believe that in order to build a reliable and practical assessment system, it is of great necessity to evaluate their robustness. Therefore, this paper can raise awareness of vulnerability in existing VQA models, which greatly outweighs its risk.