How does Watermarking Affect Visual Language Models in Document Understanding?
Authors: Chunxue Xu
Yiwei Wang
Bryan Hooi
Yujun Cai
Songze Li
Venue: COLM 2025 (2025)
Keywords: VLMs
Document Understanding
Robustness
TL;DR: Investigation of the robustness of VLM for document understanding task

Abstract: Visual Language Models (VLMs) have become foundational models for document understanding tasks, widely used in the processing of complex multimodal documents across domains such as finance, law, and academia. However, documents often contain noise-like information, such as watermarks, which inevitably leads us to inquire: Do watermarks degrade the performance of VLMs in document understanding? To address this, we propose a novel evaluation framework to investigate the effect of visible watermarks on VLMs performance.
We takes into account various factors, including different types of document data, the positions of watermarks within documents and variations in watermark content.
Our experimental results reveal that VLMs performance can be significantly compromised by watermarks, with performance drop rates reaching up to 36\%. We discover that \emph{scattered} watermarks cause stronger interference than centralized ones, and that \emph{semantic contents} in watermarks creates greater disruption than simple visual occlusion. Through attention mechanism analysis and embedding similarity examination, we find that the performance drops are mainly attributed to that watermarks 1) force widespread attention redistribution, and 2) alter semantic representation in the embedding space. Our research not only highlights significant challenges in deploying VLMs for document understanding, but also provides insights towards developing robust inference mechanisms on watermarked documents.

Published as a conference paper at COLM 2025
How does Watermarking Affect Visual Language Models in Document Understanding?
Chunxue Xu', Yiwei WangS, Bryan Hooit, Yujun Cai', Songze Lit*
Southeast University, China University of California, Merced, USA
4 National University of Singapore, Singapore
The University of Queensland, Australia
Abstract
Visual Language Models (VLMs) have become foundational models for document understanding tasks, widely used in the processing of complex multimodal documents across domains such as finance, law, and academia. However, documents often contain noise-like information, such as water- marks, which inevitably leads us to inquire: Do watermarks degrade the perfor- mance of VLMs in document understanding? To address this, we propose a novel evaluation framework to investigate the effect of visible watermarks on VLMs performance. We takes into account various factors, including different types of document data, the positions of watermarks within doc- uments and variations watermark content. Our experimental results reveal that VLMs performance can be significantly compromised by wa- termarks, with performance drop rates reaching up to 36%. We discover that scattered watermarks cause stronger interference than centralized ones, and that semantic contents in watermarks creates greater disruption than simple visual occlusion. Through attention mechanism analysis and em- bedding similarity examination, we find that the performance drops are mainly attributed to that watermarks 1) force widespread attention redistri- bution, and 2) alter semantic representation in the embedding space. Our research not only highlights significant challenges in deploying VLMs for document understanding, but also provides insights towards developing robust inference mechanisms on watermarked documents.
1
Introduction
The rapid advancement of Visual Language Models (VLMs) has transformed document un- derstanding capabilities, enabling sophisticated processing of complex multimodal content without relying on traditional Optical Character Recognition (OCR) technologies. Unlike conventional document processing systems that convert text content into machine-readable characters (Srihari, 1986; Hwang et al., 2021; Hong et al., 2022), modern VLMs directly process original document images, seamlessly integrating visual features with textual in- formation. This OCR-free approach eliminates potential recognition errors in documents with complex layouts, non-standard fonts, and visual noise (Kim et al., 2022; Liu et al., 2024b; Hu et al., 2024), while preserving critical layout and semantic structures that enhance performance in tasks such as question answering, information retrieval, and cross-page analysis.
However, this advancement introduces new robustness challenges that remain largely unex- plored. While considerable research has examined the vulnerability of VLMs to adversarial inputs and visual noise in general contexts, the specific impact of common document modi- fications, particularly watermarks on VLMs performance, represents critical gap in our understanding. Unlike Large Language Models (LLMs), where robustness mechanisms against harmful queries relatively well-established (Mei et al., 2024), VLMs' multimodal
*Corresponding Author.

---

Published as a conference paper at COLM 2025
VLMs can well understand the non- watermarked PDFs (in-domain)
VLMs are generally trained on non-wabermareed PDFs
Query
Answer
>>>
duriag inderence
VLMs misundorsland watemarkedd PDFs (out- of-domain)
Trrining
PDFs without watermarks
Query
Answer
Figure 1: After being fine-tuned on unwatermarked document data, the LVLM provides correct responses to question with unwatermaked document but exhibits obvious errors when responding to watermarked one.
nature creates unique vulnerability surfaces where image features can potentially bypass or undermine existing safeguards.
Watermarking represents one of the most prevalent forms of visual modification in profes- sional documents, serving as standard method for copyright protection and document authentication (Kankanhalli et al., 1999). As illustrated in Figure 1, even a simple watermark can dramatically alter VLMs' document interpretation, causing the model to produce incorrect responses despite no changes to the underlying informational content. This ob- servation raises fundamental questions about VLMs reliability in real-world document processing scenarios, where watermarked materials are the norm rather than the exception.
To address this critical knowledge gap, we introduce comprehensive evaluation frame- work specifically designed to assess VLMs robustness against watermarks in document understanding tasks. Our approach considers the unique characteristics of document-based question answering across diverse content types, including text-heavy documents, charts, and tables (Suri et al., 2024). By systematically embedding visible watermarks with varying properties into clean documents, we generate a watermarked dataset and the original clean dataset. Each dataset entry comprises document image accompanied by corresponding question. Our evaluation is conducted on four state-of-the-art VLMs. The model perfor- mance is assessed across the two datasets, providing insights into the model's sensitivity to visible watermarks. The primary objectives of our study are as follows: To analyze the impact of visible watermarks the performance of VLMs for document understanding. To identify the key characteristics of visible watermarks that most significantly influence model robustness.
With the proposed framework, we perform numerous controlled experiments to find that increasing watermark size does not necessarily amplify the interference effect, contradicting our initial hypothesis. Further analysis reveals that: (1) For watermarks with the same total area, distributing them across multiple locations results significantly stronger interference than placing them in a single fixed position; (2) By examining watermarked documents that successfully cause perturbation, we observe that effective watermarks are often located near the region containing the standard answer, suggesting that obstruction of key content leads to interference; (3) To validate this 'occlusion' reason, we replace text watermark 'MARK' with rectangular mask to improve the probability of covering key information and repeat the experiments. However, the interference effect is notably weaker compared to text-based watermarks, highlighting the importance of semantic information in watermarks rather than 'occlusion' and provide insights for designing more robust watermarking strategies.
We analyze the impact of watermark position and content from two perspectives: attention mechanisms and semantic similarity in embedding space. Our findings show that: (1) The
Documents exist in various formats, such DOC, PDF, and PPT, however, images provide standardized representation.
2

---

Published as a conference paper at COLM 2025
most disruptive watermark position causes the widest redistribution of attention weights. (2) The most disruptive watermark content results the lowest semantic similarity between the embeddings of watermarked and original documents.
This paper makes several contributions to the literature: First, we systematically study and evaluate the vulnerabilities of VLMs for document understanding for the first time. Second, we propose a new framework to evaluate the impact of visual watermarks on VLMs performance. Third, we analyze how watermark position and content affect VLMs performance, and use visualization to reveal the underlying causes.
2 Related Works
2.1 Visual Language Models for Document Understanding
The field of visual document understanding focuses understanding structure in visual documents and extracting key information to convert structured or semi-structured forms into machine-readable formats. A visual document is document that contains variety of visual and text elements. These documents integrate a variety of visual elements, including paragraphs, charts, tables, etc. Text elements (such as paragraphs and lists) and visual elements (such as charts and tables) together constitute the semantics of the documents, QA one of the key tasks in visual document understanding, which involves answering natural language questions based on the context information of visual documents(Ding et al., 2024).
With the rapid development of visual document understanding requirements, VLMs tech- nology for visual document understanding is introduced: Qwen-VL (Bai et al., 2023) retains the Q-Former architecture while replacing the original language model with a larger one. In order to extend the capability of VLMs to visual document understanding, mPLUU-DocOwl (Ye et al., 2023a) proposes a modular model based on mPLUG-Owl (Ye et al., 2023b) for document understanding without OCR. These comprehensive approaches highlight the potential of VLMs in handling complex visual language tasks (Li et al., 2024).
2.2 Robustness of Visual Language Models
With the widespread application and impressive performance of VLMs in multimodal QA and reasoning tasks, their robustness has garnered increasing attention in recent years. Zhao etal. (2023) evaluate the adversarial robustness of VLMs and find that multimodal generation increases security risks and that an attacker can deceive the model by manipulating visual input. Qiu et al. (2024) observe that the multimodal model is not robust image and text perturbation, especially to image perturbation. In the perturbation method tested, character-level perturbation causes the most serious distribution deviation of text, and zoom blur is the most serious deviation of image data. Lee et al. (2024) introduce VHELM to evaluate the robustness of VLMs through variety of test scenarios, such as introducing typos and testing robustness to sketches and out-of-distribution images. Agarwal et al. (2025) introduce MV AVTamperBench, designed to evaluate the robustness of VLMs against video tampering effects such rotation, occlusion, substitution, and repetition.
Significant progress has been achieved in robustness research of VLMs. In this work, we focus on how visual watermarks influence VLMs performance when applied to document understanding, identifying potential risks in visual question answering scenarios.
Methodology
In this section, we describe our methodology for evaluating the impact of watermarks VLMs in document understanding tasks. We first formalize the document understanding task and define performance metrics to quantify model accuracy. Following this, detail our evaluation datasets and experimental setup, including the selection of models and watermark parameters.
3

---

Published as a conference paper at COLM 2025
3.1 Document Understanding Task and VLMs Performance Metric
We use visual question answering (VQA) as our primary task. Because VQA is the most essential and widely used task in document understanding scenarios. (Li et al., 2023). The VQA task is defined as follows:
Given user question 9 and document image as visual context, the system is tasked with generating an answer. Each question is accompanied by a corresponding ground truth gt, which serves as the reference for evaluation. Our evaluation dataset Deval consists of representative VQA data, with each sample structured as (g, For a given VLM model f, which takes the question-image pair (q, i) as input and generates an answer, the answer accuracy over D eval is defined as:
Acc(f, def 1
Σ
1(f(g,)),gt,
(Deval
(gAg())Deval
where 1(.) is the indicator function, which takes the value of 1 if the model's generated answer i)) matches gt, and 0 otherwise.
To evaluate the impact of the watermark on the model's performance, we inject watermark w into each input image in Deval, resulting in a perturbed evaluation dataset, denoted as D' Deval' which consists of samples in the form of (q, i + w, gt).
To quantify this impact, we adopt the Performance Drop Rate (PDR) metric (Zhu et al., 2024), which measures the percentage decrease answer accuracy with respect to the user question 9. The PDR of VLM f under the influence of the watermark is defined as:
PDR(f,D D Peval D'vva) def Acclf,Deval - Acc(f, D Deval)
Moc(f,Deval
The PDR metric directly captures the degradation in model performance when processing watermarked documents compared to original documents, providing clear measure of how significantly watermarks interfere with the model's document understanding capabilities. A higher PDR value indicates a stronger disruptive effect of watermarks on the model.
3.2 Experimental Setting
Evaluation Dataset We construct our datasets by randomly sampling 100 images for each of the three document types - text, chart, and table - from the public MPLUG-DocOwLa dataset (Hu al., 2024) and the LongDocURL dataset available on Hugging Face (Dengchao, 2024). These images are categorized into four classes: Texts, ChartS, ChartM, and TableS, corresponding to text document images, chart document images, and table document images, along with their respective question types (single choice or multiple response).
For each image, we design set of questions based its content, providing four answer choices for the model to select from. The number of correct answers is determined by the category of the image: images in the ChartM class have questions with multiple correct answers, whereas images in the TextS, ChartS, and TableS classes have questions requiring only single correct choice. The resulting four datasets are as follows:
TextS Set: Each instance this dataset consists of a text document image paired with a corresponding single-choice question. Each question presents four answer options, and the groundtruth
single correct option among them.
ChartS Set: Each instance in this dataset consists of chart document image paired with a corresponding single-choice question. Each question includes four answer choices, with only one correct answer designated as the groundtruth.
ChartM Set: Each instance in this set consists of chart document image paired with a multiple-response question. Each question provides four answer choices, and the groundtruth comprises more than one correct options among them.
4

---

Published as a conference paper at COLM 2025
Different Positions of Watermarks
liee
Scattered Position
Center Position
Top-lett Position
Different Content of Watermarks
MARK
MARK
MARK
Word "MAK"
Symbo!
Ractangular Mask
Figure 2: Position and content exploration. (a) For position exploration, the watermark placed SCATTERED in five fixed positions, in CENTER and TOP-LEFT corner of document image; (b) For content exploration, we set the watermark content to text 'MARK', '##l", and fill rectangular box where the watermark located with a rectangular MASK.
TableS Set: Each instance in this set consists of a table document image paired with single-choice question. Each question contains four possible answers, with only one correct option specified as the groundtruth.
For text-based documents, 50% are from real estate and finance, and 50% from law-related studies. Chart and table documents are extracted from academic papers in computer science, physics, and related fields. This diverse collection of document types and question formats enables us to comprehensively evaluate how watermarks affect VLMs across different docu- ment understanding scenarios, from simple text extraction to complex chart interpretation and tabular data analysis.
Evaluation Models. We select 4 representative tie-oofttee-rtt VLMs for our evaluation: MPLUG-DocOwia (Hu et al., 2024) designed specifically for document understanding tasks, employing a specialized architecture for processing document layouts and mixed content types.
INTERNVL-2.5 (Chen et al., 2025) excels in real-world understanding benchmarks and is capable handling high-resolution images and complex document layouts. LLA VA-v1.5 (Liu et al., 2024a) features attention mechanism that dynamically focuses key parts of text or images while processing document content.
QwEN2.5-VI. (Bai et al., 2025) demonstrates strong performance in document understanding tasks, particularly in structured data extraction and long document understanding. This selection covers range of model architectures and specializations, allowing us to identify both general trends and model-specific behaviors in response to watermarked documents.
Experimental Parameters. We primarily investigate the impact of watermark position and content on the interference effect. All watermarks are programmatically added to the document images using the Python Pillow library. To ensure experimental consistency and isolate the effects of other watermark properties, We set watermark color as black and set the watermark transparency 0.5 to ensure that the existence of the watermarks does not affect the visibility of the document to humans,
keep several key parameters fixed: (1)
5

---

Published as a conference paper at COLM 2025
Table 1: PDR(%) for different WATERMARK POSITIONS, Note: The pink numbers represent the highest PDR value across different watermark positions.
TaxiS
CHARTS
CHARMM
TAKES
CenTre SCATFRRDD Tow Centra Scamrmm TOSCAPT CeNTER Scarmard Tonurr][inN SCATFERDD Toe-LAF
MFLUG-DDcCwwz
12
14
1
11
15
21
21
28
21
28
16
2
2
25
36
28
10
11
ILAVA-11.-70
9
23
16
22
17
7
AVG
12
10 14 22
14
but also has certain impact on the performance of the model, which is more in line with the realistic scene of document watermarks; (2) We fix the watermark angle degrees to eliminate orientation as a variable; (3) Additionally, in each experimental condition, the proportion watermarks in the total area of the document image is distributed between 10% and 80%. Experiments watermark angle and opacity are included in the appendix.
Considered Watermark Properties. Our experiments focus on two primary dimensions of watermark properties, examining how each affects VLMs performance (see Figure 2):
(1) Watermark Position: We systematically investigate how watermark placement influences VLMs performance by positioning watermarks in three distinct configurations. The CENTER placement positions the watermark at the center of the document image. The TOP-LEFT configuration places the watermark in the top-left corner of the document. The SCaTTERED approach distributes watermarks across five fixed positions throughout the document.
(2) Watermark Content: We evaluate how different types of watermark content affect VLMs performance using three variants. The MARK content consists text-based watermark containing the word 'MARK'. The '###' uses symbol-basee watermark with the pattern '#H#'. The MASK variant employs a rectangular mask placed the same location as the text-based watermark.
For both dimensions, we carefully control all variables except the one being explored. This ensures that any observed differences in performance can be directly attributed to the specific watermark property under investigation. The total watermark area, color, transparency, and other factors remain consistent across experimental conditions within each dimension.
4 Impact of Document Watermarks on VLMs Performance
Table 1 and Table show VLMs PDR(f, Deval/ D ) datasets with watermarks of different positions and contents. In general, the highest PDR can reach 36,, indicating that VLMs are susceptible to the interference of document watermarks when processing document understanding tasks.
What's more, we investigate the impact of watermark color and area ratio on model perfor- mance. Results (see appendix) show that: (1) Color change doesn't significantly affect VLMs performance. This observation can be attributed how modern VLMs process textual and visual information. These models primarily focus on high-level semantic understanding rather than low-level pixel variations.
(2) A larger watermark area ratio does not necessarily increase interference. This is because when the watermark area is too large, the semantic information carried by the watermark becomes diluted, making ineffective in interfering with document information.
4.1 Impact of WATERMARK POSITION
4.1.1 PDR across Watermark Positions
We can observe from Table that the interference effect of watermarks is significantly influenced by their position.

---

Published as a conference paper at COLM 2025
9.
9.3
Tokon Number
Token Number
Token Number
94
200
200
93
02
300
300
300
01
100
700
100
700
350
100
300
0.
Token Number
Token Number
Token Number
(8) CENTER
SCATTER
(c) TOP-LEFT
Figure 3: Heatmap of attention wight variation between watermarked multimodal input sequences and original one. Brighter points indicate the model's attention weights for the corresponding tokens have changed, with higher brightness indicating greater changes. The yellow dashed line indicates the region where the attention weight changes most significantly.
(1) Position Effect. Watermarks of SCATTERED configuration consistently result in higher PDR values across most testing cases compared to those of the CENTER and TOP-LEFT configurations. In 11/16 cases, the SCATTERED position yields the highest PDR among the three locations, demonstrating its dominant impact. For MPLUG-DocOwLz
the CHARTM dataset, SCATTERED watermarks achieve 24% PDR compared to 15% for CENTER and 9% for TOP-LEFT placements. This pattern is particularly evident in the CHARTS and CHARTM datasets, where SCATTERED watermarks consistently lead to the highest PDR values across all tested models.
(2) Dataset Vulnerability. The overall PDR levels vary significantly across datasets. The CHARTM dataset exhibits the highest PDR values, where reaches up to 36% (INTERNVI.-2.5 of Scattered configuration). This suggests that the CHARTM dataset, which by nature allows for multiple correct answers, already presents a challenge to model performance. The presence of watermarks exacerbates this issue, further reducing the model's prediction accuracy.
(3) Model Robustness. Different VLMs exhibit varying sensitivity to watermarks. QWEN2.5-VL consistently demonstrates low PDR values, with PDR values even dropping to the TExIS and TABLES datasets, indicating its strong robustness against watermarks. In contrast, INTERNVL.22.5 and MPLJU-DocOOwa show higher vulnerabilityy to watermark interference, especially on chart-based documents.
4.1.2 Attention Mechanism Analysis
To further explore the underlying reasons why watermarks with different positions produce different interference effects, we calculate the variation in attention weights generated by MPLUG-DocOWEL model between the watermarked multimodal input sequence and the non- watermarked input sequence, and use it to produce the attention weight variation heatmap (see Figure 3). The same analysis also conducted for QWEN2.5-VL. and LLA LLAVA-v1.5, with results in the appendix. The visualization reveals distinct patterns: the model adjusts atiention weights across the entire image under the Scattered setting (see Figure 3(b),, while attention changes are concentrated in the middle (see Figure 3(a)) and beginning portions (see Figure (()) of the sequence under the CENTER and TOP-LEFT settings, respectively.
These attention variations can be attributed to the VLM's processing architecture. Before performing multimodal fusion, the model divides the image into different patches. When watermarks are embedded in different positions, they affect different patches in the in- put sequence, altering the embedding vectors at these specific locations. These changes subsequently induce corresponding modifications in the attention weights at these positions.
The key insight from these results that SCATTERED watermarks disrupt the model's attention across multiple regions, forcing widespread atlention redistribution that compromises the
7

---

Published as a conference paper at COLM 2025
Table 2: PDR(%) for different WATERMARK CONTENT. Note: The purple numbers represent the highest PDR value across different watermark content.
TExTS
CHARTS
CHARTM
TAmeS
"MARK" "9U4"
MASK
'MARK" "Maa' MARK
'MARK' "AeE" MaSK "MARK" "PHF" MASK
MFLUG-DecOwwz
8
16
11
6
1
28
10
33
32
25
17
26
LLAVA-1.5 5-77
3
13
QwiN2.5.V1.00
AVG
15
model's ability to maintain coherent document understanding. In contrast, when watermarks are concentrated in single location (CENTeR or TOR-LEFT), the model primarily adjusts its attention in that affected region while preserving its processing of information elsewhere. This localized adaptation allows the model's semantic underslanding to remain largely intact, resulting in lower performance degradation compared to the SCATTERED configuration.
4.2 Impact of WATERMARK CONTENT
After examining position effects, we now investigate how different watermark content types affect VLMs performance. While position determines where interference occurs, content characteristics may influence how the model processes and interprets the watermarked information. We test three distinct watermark content: text-based ('MARK'), symbol- based ('###),, ), and visual mask (MASK) to understand their varying impacts on document understanding capabilities.
4.2.1 PDR across Watermark Content
As shown in Table 2, watermark content significantly influences interference effects across models and datasets.
(1) Content Effect. When the watermark content is either 'MARK' or '###', the PDR values of VLMs are generally higher compared to the MASK watermarks. From a statistical perspective, the average PDR across all models and catasets for the 'MARK' setting reaches 14%, for the '###' setting 13%, and for the MASK setting only 9%. In most cases, the PDR under the 'MARK' setting are highly similar to that under the '###'' setting, while the MASK setting exhibits a noticeably lower and more varied interference effect. These results highlight that different watermark content types impact document understanding tasks varying degrees, with explicit textual and symbolic watermarks being the most disruptive across multiple models and datasets.
typesimpact
(2) Model Robustness. Different VLMs show varying sensitivity patterns to watermark content. INTERNVL-2.5 and LLAVA- are more susceptible to interference from '###" watermarks on the TexIS dataset, whereas they are more affected by 'MARK' watermarks on other datasets. Similarly, the QwEN2.5-V L model shows sensitivity watermarks only on the CHARTS and CHARTM datasets, with the highest PDR in the '## configuration. Additionally, the MPLUG-DocOwL2 model exhibits significant sensitivity to MASK watermarks on the TexIS dataset, with its PDR values for 'MARK' and 'H##"' watermarks being substantially lower than those for MASK.
4.2.2 Semantic Interference Analysis
The experimental results indicate that the impact of watermarks on model performance is not solely due to their obstruction of document information. If the primary effect were merely occlusion, the MASK watermarks should outperform the 'MARK' and '###' watermarks. However, our findings do not fully support this assumption. Therefore, we further analyze the influence of watermark content from the perspective of semantic alterations in the cross- modal embedding of input sequence.
8

---

Published as a conference paper at COLM 2025
Original MARK Symbol # Mask
Cooeranoo Ennoediing Cosine Saiilafit
0 0.86
0.74
0.65
0.6
0.5-
0.46
0.4 MARK
Symbolag
Mirk
Figure 4:
Cross-modal embedding sine similarity of watermarked input se- quences with the original one.
Figure 5: Dimentional t-SNE visualization of in- put cross-modal embeddings with watermarks in different content.
Cross-modal Embedding Cosine Similarity. Cosine similarity evaluates the similarity be- tween two embedding vectors in the semantic space by measuring the directional similarity between them (Zhu al., 2025; Salton et al., 1975). The higher the cosine similarity score is, the more similar the two embedding vectors are. We compute the cosine similarity between the cross-modal embedding vectors obtained from the watermarked and non-watermarked input sequence using the MPLUUGDocOOwL model (see Figure 4). The results reveal that the 'MARK' watermark leads to the lowest cosine similarity with the original input (0.46), whereas the MASK watermark results in the highest similarity (0.86). This observation further confirms that although the MASK watermark maximizes the occlusion of key infor- mation, it induces minimal changes in the cross-modal embeddings' semantic information. Consequently, this can explain why the average PDR value under the MASK setting is the lowest.
Dimentional t-SNE Visualization for Cross-modal Embedding, t-SNE (Hinton & Roweis, 2002) can be used to compare the distribution characteristics of moos- embeddings before and after watermarking. If the addition of watermarks causes significant change in the embedding distribution, indicates that watermarks have affected the model's semantic understanding. As shown in Figure 5, the projections of cross-modal embeddings (generated from MPLUG-DocOwLa) with 'MARK' and '###' watermarks the two-dimensional space exhibit noticeable deviation from the original input data points, whereas embedding with MASK watermarks remain highly consistent with the original one, showing almost significant change. This phenomenon suggests that 'MARK' and '#H#' watermarks introduce additional explicit information, altering semantic structure of the input content, which in turn leads to substantial shift in the cross-modal embedding. In contrast, although MASK watermarks visually obscure part of the input content, they don't add new semantic information. As result, the model can maintain relatively stable embedding distribution during cross-modal alignment. This also explains why the PDR values for 'MARK' and 'HHW' watermark settings are more similar, while the PDR for the MASK watermark differs certainly from the other two.
5 Conclusion
In this paper, we establish framework based on VQA datasets to systematically evalu- ate the robustness of VLMs in document understanding task for the first time. Through comprehensive experiments on four state-of-the-art VLMs, we observe that watermarks in documents can introduce varying degrees of interference model performance. Our study provides new insights into robustness evaluation for VLMs in document understanding tasks, highlighting the need for further research in assessing and enhancing the robustness of VLMs in multimodal document processing.

---

Published as a conference paper at COLM 2025
References
Amit Agarwal, Srikant Panda, Angeline Charles, Bhargava Kumar, Hitesh Patel, Priyaranjan Pattnayak, Taki Hasan Rafi, Tejaswini Kumar, and Dong-Kyu Chae. Mvlamperbench: Evaluating robustness of vision-language models, 2025. URL https://arxiv.cr orggabs// 2972.19794.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023. URL https://arxiv org/ wwt.:/r///////////o /b/b////s. 12966.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https: wowg/bb///////// 13923.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng. Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025. URL htttts///ariiiw.or//////1112
Dengchao. Longdocurl dataset. http::///weiiiigaaeec/aaaaa/t/tp 2024.
Yihao Ding, Jean Lee, and Soyeon Caren Han. Deep learning based visually rich document content understanding: A survey, 2024. URL https: http:///aaxxxvvvaba/sss 01287.
Geoffrey E Hinton and Sam Roweis. Stochastic neighbor embedding. Advances in neural information processing systems, 15, 2002.
Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, and Sungrae Park. Bros: A pre-trained language model focusing on text and layout for better key information extraction from documents, 2022. URL https: http://aaxvvvvv/eoo www.:orgg//////11/ abs//108.. 64539.
Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-ddcowlD High-resolution compressing for ocr-free multi-page document understanding. 2024. URL http::/://///xxxxoo////a///////
03420.
Wonseok Hwang, Jinyeong Yim, Seunghyun Park, Sohee Yang, and Minjoon Seo. Spatial dependency parsing for semi-structured document information extraction, 2021. URL
https://arxiv orgo abs 00642.
M.S. Kankanhalli, Rajmohan, and K.R. Ramakrishnan. Adaptive visible watermarking of images. In Proceedings IEEE International Conference on Multimedia Computing and Systems, volume 1, pp. 508-568855 573 vol.1, 1999. doi: 10.1109/MMCS.199 MMC5.1999..79266..
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer, 2022. URL https: //tp///axvvvv/////////eee/eaavvv org/ abs/2111, 15664.....
Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Jos- selin Somerville Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, and Percy Liang. Vhelm: A holistic evaluation of vision language models, 2024. URL https: //arxiv. abs///10./2200////
Xin Li, Yunfei Wu, Xinghua Jiang, Zhihao Guo, Mingming Gong, Haoyu Cao, Yinsong Liu, Deqiang Jiang, and Xing Sun. Enhancing visual document understanding with contrastive learning in large visual-language models. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15546,1555,, June 2024.
10

---

Published as a conference paper at COLM 2025
Zekun Li, Baolin Peng, Pengcheng He, and Xifeng Yan. Evaluating the instloction-followiing robustness of large language models to prompt injection, 2023. URL https:///rxiv..org abs/2308. 10819.
models
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024a. URL htttps///ariiii org/ orggabs/2333 hatp//bb/...0/////
Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document, 2024b.
URL http::///aaxkiv.ogaaab////////// 34903//033/23300477..... 04473.
Lingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi, Jiayi Mao, and Xueqi Cheng. "not aligned" is not" malicious": Being careful about hallucinations of large language models' jailbreak. COLING 2025, 2024.
Jielin Qiu, Yi Zhu, Xingjian Shi, Florian Wenzel, Zhiqiang Tang, Ding Zhao, Bo Li, and Mu Li. Benchmarking robustness of multimodal image-text models under distribution shift, 2024. URL http:://arxxvvoorg//b//////////
08044..
G. Salton, Wong, and C. Yang- A vector space model for automatic indexing. Commun. ACM, 18(11))613-620, November 1975, ISSN 0001-0782. doi: 10.1145/361219.31122) URL https://doi, http://oo..rgg/...1// 1115/311113121 .1..:611119.111111111.
Sargur N. Srihari. Document image understanding. In Fall Joint Computer Conference, 1986.
URL https: htt.:appi.semmannisccchoo.oa.a http:/ccch.aaa.orrrrrrpsusllslaa 5346648.
Manan Suri, Puneet Mathur, Franck Dernoncourt, Kanika Goswami, Ryan A. Rossi, and Di- nesh Manocha. Visdom: Multi-document with visually rich elements using multimodal retrieval-augmented generation, 2024. URL https://arxiv httts://arxiv.oor//bb////2 wrgp/b//////2/////// 10704.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guo- hai Xu, Chenliang Li, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-docowe Modularized multimodal large language model for document understanding. 2023a. URL
https://arxiv www.aaiwww...oooaraaeaaeeroeea abs/
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl: Modularization empowers large language models with multimodality, 2023b. URL http::://aaiiv..orrr abs/2304. 14178.
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models, 2023. URL https://arxiv.. arg/ 16934.
Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, and Xing Xie. Promptrobust: Towards evaluating the robustness of large language models on adversarial prompts, 2024. URL 04528.
https://arxiv.. wtgg/bs////
Rongzhi Zhu, Xiangyu Liu, Zequn Sun, Yiwei Wang, and Wei Hu. Mitigating lost-in- retrieval problems in retrieval augmented multi-hop question answering, arXio preprint arXio:2502.14245, 2025.
11

---

Published as a conference paper at COLM 2025
A
Appendix
A.1 Impact of Watermark Color
We compare the PDR across datasets with different watermark colors on MPLUG-DocOwra, all positioned in SCATTERED position. The results, shown in Table 3, indicate that average PDR scores across different colors are similar. This observation can attributed to how modern VLMs process textual and visual information. These models primarily focus high-level semantic understanding rather than low-level pixel variations. As result, changes in watermark color do not significantly affect the model's 's interpretation of the document, leading to minimal differences in interference effects.
Table 3: Average PDR for different WATERMARK COLORS on MPLUG-DocOWLz.
TEXTS
CHARTS
CHARTM
TABLES
Color
AVG
'MARK' '###' MASK
'MARK' '##H' MASK
'MARK' "HII"" MASK
'MARK' 'HHH'' MASK
BLACK
7
20
39
19
RED
0
18
1
33
4
31
35
20
GREEN
1 30
7
11
0
64
20
2
30
40
22
A.2 Impact of Watermark Area Ratio
As shown in Figure 6, increasing the area ratio of the watermark does not necessarily lead to stronger interference effect. This suggests that once the watermark area exceeds a certain threshold, it is no longer the decisive factor influencing model performance. This can be attributed that when the watermark area is too large, the semantic information carried by the watermark becomes diluted, making it ineffective in interfering with the original content of the document.
60
Dataset:
Doc SingleChoice
50-
Chart SingleChoise
Chart MuhriRespouse
PDR Valuel (%)
40
Tabae SingleChsion
30-
20
056 2055
20%
40% 60%
60% 80%
Watermark area ratio(from 0% to 80%)
Figure 6: Average PDR(%) (%) of different watermark area ratios on the full dataset.
A.3 Impact of Watermark Opacity
As shown in Table 4, increasing the opacity of the watermark leads to significant rise in performance degradation, with the average PDR increasing from 8.5% at low opacity (a 0.2) to 21.3% at high opacity (a 0.8). This trend suggests that higher visual prominence caused by denser watermark opacity results in stronger disruption of model attention. The underlying reason may be that more opaque watermarks dominate the visual
12

---

Published as a conference paper at COLM 2025
space, increasing misalignment in the model's attention mechanism and impairing its ability to focus on relevant document content.
Table 4: Average (DR(%) for different watermark opacity levels mPLUG-DocOW12.
Opacity (a)
Avg. PDR (%
0.2 (light)
8.5
0.5 (medium)
15.7
0.8 (dense)
21.3
A.4 Impact of Watermark Rotation
As shown in Table 5, variations in watermark rotation angle have minimal effect on model performance. The average PDR remains relatively stable across different angles, with 14.2% at 0", 14.5% at 45", and 14.1% at 90%. This consistency indicates that modern VLMs exhibit strong robustness to moderate geometric transformations in watermark orientation. A possible explanation is that the spatial encoding mechanisms of current vision backbones effectively normalize such rotation variations, making them less disruptive cross-modal alignment.
Table 5: Average PDR(%) for different watermark rotation angles on mPLUG-DocOw12.
Rotation Angle
Avg. PDR (%)
14.2
45 (diagonal)
14.5
90 (vertical)
14.1
A.5 Attention Heatmap Analysis Across VLMs
As shown in the Figure 7 and Figure 8, these are the attention variation heatmaps of QWEN2.5 and LLA VA-11.5 It is evident that the QwEn2.5-VL. exhibits stronger robustness compared to LLAVA-V1. and MPLUG-DocOwL,, which is consistent with the experimental results.
100-
100
Token Number
Token Number
Tokea Numbar
30
80o
0
300
100
220
309
100
200
300
(8) CENTER
(b) SCATTER
(C) TOP-LEFT
Figure 7: Heatmap of attention wight variation between watermarked multimodal input sequences and original one for QWEN2.5-VL.
A.6 Image Preprocessing for Robustness Improvement
We try to preprocess document images before entering them into VLMs to reduce the inter- ference effect of watermarking the model. We test common image processing method, JPEG compression, set the compression mass to 30, and analyze its effect on watermark interference. The experimental results are shown in Figure 9.
13

---

Published as a conference paper at COLM 2025
0.6
0.5
Token Number
Token Number
Token Number
0.4
0.3
0.2
0.1
CENTER
SCATTER
TOP-LEFT
Figure 8: Heatmap of attention wight variation between watermarked multimodal input sequences and original one for LLA VA-115.
The experimental results show that the PDR value on the document cata after JPEG com- pression is reduced compared to the unprocessed data in some cases. The main mechanism of JPEG compression to reduce the resolution of the watermark and thus reduce its pertur- bation effect. However, JPEG compression is essentially a lossy compression, which not only introduces additional noise, but also reduces the overall image quality, especially affecting the sharpness of the text area. This decrease in text resolution may affect VLMSs' accurate extraction and understanding of document content, resulting in an increase on PDR.
MPLUG-DooOWi2
InternV--22
LLLVA-v1.5
Qwen2.5-VV 5-VL
Oppal
IPBI Gapmeea
IPD: Caggoea
Obeai
PD PP0--oppom
Orpad
SOAT MRLD
TUPLEFT
CEX
HONT X0RE
JUP-EFT
CESTEX
SONUTORES
TUPLEFF
DEXTER
BUXU TORRO
TOPSEEFT
JPBX Caapmeaa
Orgai
JPD Oaggoeaa
Obeam
Oread
KONT 1870
TOP
SONY TARE
TUP
SONT TERRO
TUPSIFT
DENTER
SCAT TEARO
Ordial
JEBG Coepiic
Orgiai
Oreai
100164
KONT 1870
TOPTEFT
CEXTR1
KONT TARE
TUP
SONT TERRO
TUPTIFT
DENTER
SCAT TEARO
Ordial
J1B0 Coppiic
Orgaal
Oread
DESTER
Figure 9: PDR before and after JPEG compression.
A.7 Processing Architecture of VLMs
As show in Figure 10, before performing multimodal fusion, the model first processes the input image by dividing it into multiple smaller patches. This step allows the model to analyze local features within each patch, facilitating more effective feature extraction and representation.
14

---

Published as a conference paper at COLM 2025
Normal Image Embedding
Watermark Embedding
Independent Imagee Encooling,
Independent Image Encoding
Independent Image Encoding
STRK
WATER MARK
MARK
Slaae-Adappive Cropping
Shape- Adaptive Cropping
Shape-Adaptive Adaptive Cropping
Scattered Watermark
Centered Watermark
Top-left Watermark
Figure 10: The structure of VLMs for processing input sequences.
15


Reviews:
[{"review_id": "CylIUOzuCz", "reviewer": "Reviewer_64kT", "summary": "This paper presents empirical analysis on how watermarks affect visual LM in language understanding.\n\nIt attributes the negative impact to redistribution of attention and altering visual embeddings. Specifically, the paper links scattered watermarks to redistribution of attention and watermark content to semantic representations.\n\nThis paper offers useful insights into how watermarks affect VLM-based document understanding. More explorations in how to address these negative impacts would interest the community.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Detailed analysis on how watermarking affect VLMs", "reasons_to_reject": "- The paper focuses mainly on analysis of the phenomena, but lacks proposals of solutions\n- The watermarks are synthetic. Evaluation on real world data or more realistic watermarks would be more desirable", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t0uOiZbKBQ", "reviewer": "Reviewer_Ndiu", "summary": "This paper presents a systematic study on the impact of visible watermarks on vision-language models (VLMs) in document understanding tasks. It introduces a visual question answering (VQA)-based evaluation framework and, for the first time, analyzes watermark interference across multiple dimensions including position, content, color, area, and distribution. The experiments reveal that watermarks can significantly degrade model performance, with significant accuracy drops. Watermarks with strong semantic content and dispersed placement tend to have the most severe impact. The authors further investigate the underlying mechanisms behind this performance degradation by analyzing attention weight redistribution and cross-modal embedding shifts. This work highlights the vulnerability of VLMs in real-world deployments and provides initial insights into potential directions for improving model robustness.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Novel and Practical Research Topic**:   \n\nThe author claim they are first to explore how watermarks affect document understanding in VLMs, addressing a timely and under-explored problem with clear real-world implications.\n\n\n**Well-Designed Evaluation Framework**: \n\nThe proposed setup enables systematic control over various watermark attributes — such as type, position, content, color, and area — ensuring the experiments are both reproducible and interpretable.\n\n**Thorough Experimental Design**: \n\nThe study spans different types of documents (text, charts, tables) and evaluates multiple mainstream VLMs. It also includes ablation studies on masking and semantic analysis, adding depth to the findings.", "reasons_to_reject": "- Lack of Comparative Robustness Methods: The paper does not benchmark against existing robustness-enhancing techniques, such as adversarial training or data augmentation, missing an opportunity to contextualize its findings.\n    \n- Limited Dataset Size and Diversity: The evaluation is based on only 100 synthetic images, without including real-world documents such as scanned contracts, invoices, or magazine pages, which raises concerns about generalizability.\n    \n- Unclear Experimental Details: The logic behind the five \"dispersed\" watermark positions is not clearly explained, and key image processing details (e.g., resolution, preprocessing steps) are omitted, though they could impact results.\n    \n- Insufficient Analysis of Color-Background Interaction: While color effects are briefly discussed, there’s little analysis of transparency or contrast between watermark and background — a critical factor in real documents.\n    \n- Narrow Task Scope: The experiments focus solely on VQA tasks, without exploring broader document understanding scenarios like table structure extraction, document classification, or long-document summarization.", "questions_to_authors": "Suggestions: \n\n- Including a detailed appendix or supplementary material specifying watermark generation settings — such as font, opacity, size, and logic behind position choices.\n    \n- Add real-world document samples and explore more complex formats (e.g., multi-page or mixed-layout documents).\n     \n- Investigate the effect of transparency and contrast between watermark and document background in more depth to provide practical insights for real-world cases.", "extra_scores": {}}, {"review_id": "uKSIW42s8x", "reviewer": "Reviewer_stAf", "summary": "The paper proposes the first systematic study of how visible watermarks degrade state-of-the-art visual–language models on document VQA, defining the intuitive metric $\\mathrm{PDR}(f,D_{\\text{eval}},D'{\\text{eval}})=\\tfrac{\\mathrm{Acc}(f,D{\\text{eval}})-\\mathrm{Acc}(f,D'{\\text{eval}})}{\\mathrm{Acc}(f,D{\\text{eval}})}$ and showing performance drops of up to 36 % across four models; despite clear methodology, insightful attention-map/embedding analyses and actionable findings (scattered, semantic marks are worst; JPEG preprocessing partly mitigates), experimental quality is hampered by a tiny unreleased 400-document benchmark, lack of statistical confidence intervals and limited watermark parameter sweeps, while clarity suffers from grammatical slips and cramped appendix tables.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is the first to cast visible watermarking as a domain-specific robustness hazard for document-centric VLMs, introducing the intuitive metric.\n\n2. Beyond headline accuracy losses, the work triangulates evidence via attention-map shifts, t-SNE embedding drift, and ablation on watermark placements, then demonstrates simple but effective mitigations (e.g., JPEG re-encoding). This multi-angle analysis provides both diagnostic insight and practical guidance for practitioners.\n\n3. Watermarks pervade legal, financial and academic documents; quantifying their effect on four state-of-the-art VLMs positions the paper as an early, influential reference for future robustness benchmarks, defence methods and model design—aligning squarely with COLM’s goal of advancing trustworthy multimodal understanding.", "reasons_to_reject": "1. The study relies on a tiny, unreleased benchmark of only 400 clean + 400 watermarked images, offers no cross-validation, and omits statistical uncertainty (e.g., 95 % CIs or paired tests).\n\n2. Only two placements (“centered” vs. “scattered”) and one visibility level are considered, with no sweep over opacity, rotation, layering, multi-page patterns, or adversarially designed marks; likewise, mitigation experiments stop at JPEG re-encoding, ignoring blur, in-painting, diffusion repair, or fine-tuning baselines, leaving the practical guidance shallow.\n\n3. Numerous grammatical errors, duplicated references, cramped appendix tables, and the absence of released code/data hinder clarity and trust. Without public artifacts, other researchers cannot replicate or extend the findings—falling short of COLM’s bar for transparent, reusable science.", "questions_to_authors": "1. Can you clarify how the 400 clean / 400 water-marked documents were sampled (domain balance, language diversity, watermark sources) and commit to releasing the dataset to enable reproducibility?\n\n2. Have you computed 95 % confidence intervals or performed paired significance tests (e.g.\\ bootstrap, McNemar) for the reported $\\mathrm{PDR}$ values, and could you include them to substantiate claims of “up to 36 %” degradation?", "extra_scores": {}}, {"review_id": "22jti5ZK2z", "reviewer": "Reviewer_bZfS", "summary": "The paper investigates how visible watermarks interfere with LVLMs on document-understanding tasks. The authors (i) assemble a 400-sample VQA benchmark spanning text, chart and table documents, (ii) algorithmically paste watermarks of different positions and contents onto each page, and (iii) measure the performance-drop rate (PDR) of four LVLMs. Scattered, semantically meaningful watermarks degrade accuracy by up to 36%, whereas color has little influence, and enlarging the watermark beyond 40% page area yields diminishing additional harm. Attention-map and embedding analyses reveal that distributed text watermarks (a) force global attention redistribution and (b) shift cross-modal representations more strongly than visual masks, explaining the larger PDR.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A systematic look at watermark robustness for VLM-based document understanding, and provide a well-controlled benchmark.\n2. Attention-change heat-maps and t-SNE/cosine plots link performance drops to semantic perturbation rather than mere occlusion. \n3. Insightful findings: scattered, text-based watermarks are most disruptive, while simple compression sometimes helps, guiding practitioners in watermark design and model hardening.", "reasons_to_reject": "1. Synthetic watermark patterns: real-world watermarks vary in opacity, rotation, repetition, and background blending; external validity is uncertain.\n2. Limited dataset scale & task scope: only 100 images per class and a single-choice VQA formulation; other document tasks (e.g. key-value extraction, retrieval) are not studied. \n3. Several attention and embedding studies are done only on mPLUG-DocOwl2, leaving the generality of the explanation speculative.", "questions_to_authors": "1. Could you provide 95% CIs or bootstrap tests for the PDR differences to confirm statistical robustness?\n2. JPEG quality = 30 sometimes hurts accuracy—did you try adaptive denoising or in-painting? A brief ablation would help.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper conducts a systematic investigation into how text-based watermarks affect vision-language models (VLMs) in document understanding tasks. It introduces a novel VQA-based evaluation framework and, for the first time, examines watermark interference across multiple dimensions—position, content, color, area, and spatial distribution. Experimental results show that watermarks can substantially degrade model performance, with notable drops in accuracy. The most severe effects are observed with watermarks that carry strong semantic meaning and are widely dispersed. To better understand the causes of this degradation, the authors analyze changes in attention weight patterns and shifts in cross-modal embeddings. This work underscores a key vulnerability of VLMs in real-world applications and offers preliminary insights toward enhancing model robustness.

From my review of the feedback, it appears that all reviewers responded positively to the rebuttal. While there were some minor concerns about the experimental scope, the overall sentiment toward the responses is favorable.
Decision: Accept