WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
Authors: Songbai Tan
Xuerui Qiu
Yao Shu
Gang Xu
Linrui Xu
Xiangyu Xu
Huiping Zhuang
Ming Li
Fei Yu
Venue: ICML 2025 poster (2025)
Keywords: Watermarked image understanding
Primary area: social_aspects->safety
TL;DR: We first propose WMarkGPT, a multimodal large language model (MLLM) that enhances watermark content understanding by integrating visibility prediction and textual description, surpassing mainstream MLLMs in effectiveness and depth.

Abstract: Invisible watermarking is widely used to protect digital images from unauthorized use. Accurate assessment of watermarking efficacy is crucial for advancing algorithmic development. However, existing statistical metrics, such as PSNR, rely on access to original images, which are often unavailable in text-driven generative watermarking and fail to capture critical aspects of watermarking, particularly visibility. More importantly, these metrics fail to account for potential corruption of image content. To address these limitations, we propose WMarkGPT, the first multimodal large language model (MLLM) specifically designed for comprehensive watermarked image understanding, without accessing original images. WMarkGPT not only predicts watermark visibility but also generates detailed textual descriptions of its location, content, and impact on image semantics, enabling a more nuanced interpretation of watermarked images. Tackling the challenge of precise location description and understanding images with vastly different content, we construct three visual question-answering (VQA) datasets: an object location-aware dataset, a synthetic watermarking dataset, and a real watermarking dataset. We introduce a meticulously designed three-stage learning pipeline to progressively equip WMarkGPT with the necessary abilities. Extensive experiments on synthetic and real watermarking QA datasets demonstrate that WMarkGPT outperforms existing MLLMs, achieving significant improvements in visibility prediction and content description. The datasets and code are released at https://github.com/TanSongBai/WMarkGPT.

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
Songbai Tan 12 Xuerui Qiu
Yao Shu? 2 Gang Xu² Linrui Xu Xiangyu Xu
Huiping Zhuang 6
Ming Li Fei Yu²
Watermurked Image
WOA-Ssntlelic (LL Scoeeel
Query: Please provide a detailld description waterinik internis of its direction, its placement celative to objects, appearinoe, and visiilliy.
WQa--Sutlatic (ROUGE-L)
WQQ Syaa-Setthekk
VE..2
(ACO)
WMarkGPT
WQA-Srnthelee (IX 331-3)
The wath reak visblit
(BLUE-)
Ik
The watermark in the image centrally oriented, covering a significant portion of the partially overlups with the and potted plant, extending across their surfaces and coming close their edges. The watermark appears as ciscular patterned texture with moderately visible presence, blending somewhat with image's elements without heing overly unruaive The watemark visibiliry is moderate.
WMarkGPT:
Iustatt
Faist
1:4044
354.00
Madbraik
Pastle
WiS--Heall (KOLGEAL)
20100
Worght ong.
W.A--eal
Seare: 2.71
(a)
(b)
Figure 1. (a) Our WMarkGPT is the first multimodal large language model specifically designed for watermarked image content understanding without requiring access the original images, which are often unavailable in text-driven generative watermarking scenarios. It excels in accurately localizing watermarks, generating detailed semantic descriptions of their characteristies and their impact on image content, and evaluating visibility levels capas--apabilittie that surpass conventional statistical metries, which depend on original images and are limited to measuring low-level differences. (b) Compared to existing state-of-the the-tr-rrt art MLLMs, WMarkGPT demonstrates significantly improved performance across various evaluation metrics.
Abstract
portantly, these metrics fail to account for po- tential corruption of image content. To address these limitations, we propose WMarkGPT, the first multimodal large language model (MLLM) specifically designed for comprehensive water- marked image understanding, without accessing original images. WMarkGPT not only predicts watermark visibility but also generates detailed textual descriptions of its location, content, and impact on image semantics, enabling a more nuanced interpretation of watermarked images. Tackling the challenge of precise location descrip- tion and understanding images with vastly differ- ent content, we construct three visual question- answering (VQA) datasets: an object location- aware dataset, a synthetic watermarking dataset, and real watermarking dataset. We intro- duce a meticulously designed three-stage learning pipeline to progressively equip WMakkGPT with the necessary abilities. Extensive experiments
Invisible watermarking is widely used protect digital images from unauthorized use. Accurate assessment of watermarking efficacy is crucial for advancing algorithmic development. How- ever, existing statistical metrics, such as PSNR. rely on access to original images, which are of- ten unavailable in text-driven generative water- marking and fail to capture critical aspects of watermarking, particularly visibility. More im-
"School of management, Shenzhen University Guangdoog Laboratory of Artificial Intelligence and Digital Economy (SZ) "Istittte of automation, Chinese Academy of Sciences "School of Geosciences and Info-Physics, Central South University 'Xi' an Jiaotong University. China "Shien-Ming Wu School of Intelligent Engineering. South China University of Technology. Correspon- dence to: Ming Li <liming @ gml.ac.en..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
on synthetic and real watermarking QA datasets demonstrate that WMarkGPT outperforms exist- ing MLLMs, achieving significant improvements in visibility prediction and content description. The datasets and code are released at https: //gittubb wwtpb//ooo//aannaaaaaaa GTM.
Invisible
Faint
Moderate
Visible
Obvious
1. Introduction
0S1M:0.97
SS1M:092
SSIM:093
055
Invisible watermarking has been extensively utilized in digi- tal images to prevent unauthorized use by embedding dis- cernible information, such as 2D logos, which can later be retrieved by specific extractors to provide verifiable proof for various applications (Hosny et al., 2024; Rezaci et al., 2025; Sharma et al., 2024). This is particularly significant in text-to-image generation domains, which hold immense potential for commercial applications (Ma al., 2024; Fer- nandez et al., 2023). The primary challenge in watermarking lies in achieving a delicate balance between minimizing its impact on image quality and maintaining robust detectabil- ity under various conditions (Guo et al., 2024; Fernandez et al., 2022). As a result, there has been growing interest in the research community toward evaluating the visual and structural effects of watermarking on digital images (Sharma et al., 2024).
PSNR:
PSNR: 35.11
21.
PSNR: 23:43
PSNR: 32.11
LIPIPS: 0.11
LIPIPS: 0.04
LIPIPS:
LIPIPS: 0.28
LIPIPS: 0.26
Figure 2. Examples of original and watermarked image pairs with varying levels of watermarking efficacy, annotated by humans. Traditional similarity metrics, including SSIM (t), PSNR (t), and LPIPS (J), are computed for each pair. The greater and less signs between two samples indicate the relative magnitude of the metric values. Red signs indicate inconsistencies with human percep- tion, while black signs represent consistent relations. The results demonstrate that traditional metries often fail to accurately reflect watermarking efficacy, particularly in alignment with human eval- uations.
and question answering tasks, effectively addressing conven- tional vision and language challenges with unprecedented capabilities (Liu et al., 2024b; Zhu et al., 2023; Xue et al., 2024; Liu et al., 2024a). Notably, pioneering efforts such as the large language and vision assistant (LLaVA) integrate vision encoder with a L.M and leverage GPT---generated multimodal data to build robust systems excelling in visual understanding,
Conventional assessments primarily rely on pixel-wise sta- tistical metrics, such as peak signal-to-noise ratio (PSNR)
(Korhonen & You, 2012), structural similarity index (SSIM) (Wang et al., 2004), and learned perceptual image patch similarity (LPIPS) (Zhang et al., 8a), to quantify low- level differences between original and watermarked images (Rachmawanto et al., 2024). However, accessing original images as references is often impractical for text-driven generative watermarking (Ma et al., 2024; Fernandez et al., 2023). Moreover, they often fail to accurately reflect water- marking efficacy, particularly in terms of watermark visi- bility, As shown in Fig. 2, these metrics often fail to align with human perception. Even for clearly visible watermarks, the metric values unexpectedly favorable, highlighting their limitations in accurately assessing watermark visibil- ity. To quantify the degree of misalignment, we calculate the Spearman's rank correlation coefficient (SRCC) (Sedg- wick, 2014), Pearson linear correlation coefficient (PI.C) (Sedgwick, 2012), and Kendall's 's rank correlation coefficient (KRCC) (Abdi, 2007) between these metrics and human annotations for watermarking efficacy. As shown in Fig. 3, the comparisons reveal that traditional metrics signifi- cantly deviate from human perception. More importantly, these metrics fail to offer comprehensive evaluation of the broader impact of watermarking on image content.
followiig,,. and complex ques- tion answering tasks (Liu et al., 2024b). However, none of these MILMs can be directly applied to watermarked image understanding, even with nuanced prompting, as they are trained natural images whose data distribution and semantic content remain unaltered by the integration of distinctly different logo images.
To overcome these limitations, we introduce a new MLLM for watermarked image understanding, termed PPT. This model generates textual descriptions of watermark con- tent, locations, detailed interactions with the main image content, and visibility prediction with only watermarked image, making it particularly suitable for text-to-ing -iatto-iieeeeae gen- erative watermarking, as illustrated in Fig. 1. To achieve it, the model must possess the ability to precisely describe object locations, identify a watermark from desirable fore- ground contents, and assess its visibility in a watermarked image. To support this learning, we construct three custom visual questtoo-answenee object location-aware dataset, built upon the COCO dataset, which includes image captions and object bounding boxes. GPT-4 is used to generate 100k QA pairs that capture both absolute and relative object positions within a natural image. The second is a watermarking QA dataset, created with 50k
(VQA) datasets. The first is an
Recently, multimodal large language models (MLLMs) have achieved significant breakthroughs in image understanding
2

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
synthetic watermarked images using a semi-automatic an- notation pipeline. The final dataset is real watermarking QA dataset, consisting of 2.5k watermarked images gen- erated by several state-of-the -hrt watermarking algorithms, along with human annotations including watermark visibil- ity and overall scores. The scores are used exclusively for evaluation purposes.
To summarize, our contributions in this work are fhree-fold:
We propose WMarkGPT, the first multimodal large language model tailored for watermarked image un- derstanding. It demonstrates superior performance in generating comprehensive textual descriptions and pre- dicting watermark visibility without requiring access to the original images, effectively addressing the sig- nificant limitations of existing evaluation methods.
0.8
SRCC
0.75
PLCC
0.7
We construct three visual question ansswrrin bench- mark datasets, especially the synthetic and real wa- termarking datasets, to facilitate research object location-aware MLLMs and watermarking assess- ments. These datasets will be publicly released to advance future research.
0.7
KRCC
0.6
0.53
0.5
Value
0.42 0.43
0.4
We propose systematic learning paradigm progres- sively endow model with the capability to understand object positioning relationships and corrupted image semantics. To the best of our knowledge, this repre- sents the first attempt to train MILMs on unnatural images, each composed of a fusion of two significantly different images.
0.3
0.23
0.23
0.18
0.17
0.15
0.12
0.1
0.1
0.0
SSIM
PSNR LIPIPS
WMarkGPP
Figure 3. Comparison of various methods for assessing water- marking efficacy, measured by their consistency with human anno- tations. Higher SRCC, PLCC, and KRCC values indicate better alignment with human perception. Traditional statistical metrics, such as SSIM, PSNR, and LPIPS, show significantly lower con- sistency and fail to accurately reflect watermarking efficacy. In contrast, our proposed WMTrkGPT achieves substantially higher consistency with human annotated scores, demonstrating its supe- rior performance. Notably, WMarkGPT is trained only with coarse visibility level descriptions rather than precise scores.
2. Watermarking Question-Answering Datasets
Image watermarking techniques have been extensively ex- plored within the field of computer vision. However, large- scale watermark QA datasets tailored for the fine-tuning of MLLMs (e.g.. our WMarkGPT) remain scarce, which hence significantly limits the development of MLLMS capable of comprehensive watermarked image understanding. Prior research has shown the promise of leveraging large-scale generative models like GPT, to synthesize data for multi- modal applications, driving substantial progress in model capabilities (Jiao et al., 2024; Liu et al., 2024c; Wu et al., 2023b). Inspired by this, we introduce WQA-Synthetic, the first synthetic watermark QA dataset constructed via a semi-automated pipeline augmented with human over- sight on large-scale generative models for comprehensive watermarked image understanding. This dataset comprises artificially generated watermarked images, corresponding questions, detailed watermark descriptions, and associated visibility levels. Furthermore, to capture real-world water- marking scenarios, we conduct subjective experiments and create WQA-Real, the first dataset derived from authentic watermarking cases for improved watermarked image un- derstanding. The distributions of visibility levels in both datasets are illustrated Fig. 4.
To train our WMarkGPT, we propose a meticulously de- vised three-stage learning pipeline to progressively endow the model with robust capabilities for understanding wa- termarked images. Our model architecture comprises a visual encoder, learnable queries, a visual abstractor, and an LLM. In the first stage, we primarily train the visual en- coder, learnable queries, and visual abstractor on the object location- aware natural image dataset, enabling the model to precisely recognize object positions. In the second stage, we optimize the entire model the synthetic watermarking dataset, adapting it to handle images with corrupted data distributions and vastly different semantic contents. Finally, in the third stage, we fine-tune the visual encoder and visual abstractor on the real watermarking dataset, refining the model's 's performance and familiarizing it with target data distribution. Experimental results on the real watermarking QA dataset demonstrate that our WMarkGPT outperforms existing mainstream MILMss by approximately 29%, 45% 150%, and 217% across four metrics, respectively (see Sec. 4. I.,,, highlighting its powerful capabilities in understand- ing both watermarks and their interactions with relevant semantics.
2.1. WQA-Synthetie Dataset
To enhance the capability of MLLMS to understand the details of watermarked images comprehensively, we con-
3

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
WQA- WQAA-Sytthttic (50K)
WQA-Real (2.5K)
segmented watermark and its boundary frame coordinates (b%, b by b2)
22%
18%
12%
13%
41%
Step-2: Main Object Bounding Box Detection (Part 2 in Fig. 5). To determine the watermark's 's relative position within the image, we extract bounding box information for the primary objects. With the object bounding box data from the COCO dataset, we employ GPT-4 to identify the main objects by analyzing their dimensions and spatial co- ordinates. The selected bounding boxes for these objects, denoted as (be, b by by J. are then collected for fur- ther processing.
20%
20%
18%
20%
16%
Invisible
Faint
Moderate
Visible
Obvious
Figure 4. Data distribution of the collected WQA-Synthann and WQA-Real datasets.
Step-3: Watermarked Image Synthesis (Part 3 in Fig. 5), In this stage, we foeus on integrating the selected watermark into the original image while maintaining the visual integrity and coherence. Firstl,, the watermark is resized to fit within predefined bounding box. Subsequently, the watermark is embedded at random position within the image using weighted average fusion method. The blending process is governed by transparency factor a, which controls the visibility and prominence of the watermark. The integration procedure for each pixel within the target region can be mathematically expressed as:
struct a synthetic watermark QA dataset, WQA-Synthetic through a cost-effective and efficient autimautomatic an- notation pipeline introduced below. The dataset consists of 50K data pairs in the format of (I, Q T, V). where represents the watermarked image, Q denotes the related question, T is the watermark description, and V indicates the degree of watermark visibility. To ensure a compre- hensive coverage of watermark characteristics, we utilize GPT-4 to generate 30 diverse question templates focusing on different aspects of watermark analysis. For each water- marked image, one question template is randomly selected as Q. The watermark descriptions in the dataset include the position of watermark, its relationship with the main object in the image, -pteriark-sseccifi features, and the visibility level of watermark, as illustrated in the part 4 of Fig. 5. This structured approach aligns with human perception, where watermark evaluation is not limited to overall visibility but also considers spatial position and feature attributes. Such detailed annotations provide a more comprehensive under- standing of watermark content, enabling MLLMs to bridge the gap between perceptual and analytical insights.
Ie v) = (1 (1-a) a. [) - M(a. v), (1)
where L.. Icy Iu.. and M represent the watermarked image. the original image, the watermark, and the filtered mask, respectively. To maintain visual consistency, a brightness adjustment is applied to the watermarked image I, through linear transformation:
I (x, v) = I. v)
1
: Σ L.
(2)
I I. n
The value of ranges from 0 to 0.25, with each increment of 0,05 corresponding to a one-level increase in watermark visibility. The visibility is categorized into five distinct lev- invisiiii"" "faiiit "faint", "moderate", "visible", and "obvious". correspondingly. The visibility level V of the watermark is determined by the chosen value of a, allowing for precise control over its perceptual appearance.
The semi-automatio annotation pipeline of WQA-Synthetic is shown in Fig. 5. We randomly select 50K images from the COCO dataset (Liin et al., 2014) and 50K watermark logos from the LOGO-2K dataset (Wang et al., 2020), and then synthesize watermarked images while generating corre- sponding watermark descriptions using four-step process: (1) Step-1: watermark segmentation; (2) Step-2: main object bounding box detection; (3) Step-3: watermarked image synthesis: and (4) Step-4: guestion-answering gener- ation. Each stage is outlined in detail below.
Step-4: Question-Answering Generation (Part 4 in Fig. 5). After the aforementioned stages, we obtain the water- marked image, the bounding box coordinates for both the watermark and the main object, as well as the watermark's visibility level. These information is then integrated into QA template, which is fed into GPT-4o to generate a watermark description T. This watermark description in- cludes information about the watermark's kss location and other specific features. The data collection for WQA-Synthetic, consisting of (T', Q, V). is then completed.
Step 1: Watermark Segmentation (Part in Fig. 5). Dur- ing this stage, we aim minimize background noise by carefully isolating the area where the watermark is located. Specifically, use the Segment Anything Model (SAM) (Kirillov et al., 2023) to generate binary mask that pre- cisely delineates the watermark region. To further refine the segmentation, a background filtering step is applied to elim- inate extrancous pixels. The filtered mask is then mapped back onto the original watermarked image, yielding the
2.2. WQA- Dataset
In addition to the synthetic watermark QA dataset, we also build a real-world dataset, Real to further improve the capability of MLLMs to understand watermarked images comprehensively. Different from WQA- RQQ-hhhtheticc dataset,
4

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
Watermark
bbox [0,0,h,.]
Segmentation Mask
Is the obiect in the boundany box area the main object in this image?
aP1-4
object person <boxt>
Baba
DANTECH
Encoder Decoder
Baba
object 2: chair <box2>
oblect 3: dining table <box3>
No
Yes
Segment Anything Model
a=0
Color correction
Segmentation Mask
Filtered Mask
x(1-a)
invisible
faint
moderate
Baba
DANTECH
Baba
PANTECH xa
visible
Background filtering & bounding box detection
obvious
(AC, Ny A5
a=0.25
Answer for WQA-Synthese
All main object bounding box coordinates [1/ M/ Afa);
QA Template
centrally oriented the image, spanning significaur portion the visible content. It partially overiaps with various objects, including the people, dining ta table, knife, and near edges some objects within the image frame. The watemsark appears as transiucent pattern, with sextured slightly opague quality that makes noticeable against backgroundand obiects. watermark visibility is obvious.
Watermark bounding box coordinates (0%, bi2, My,, b:
being filly
Watermark visibility level information <level>;
Question guidance prompts and format information;
GPT-4o
Figure 5. Illustration of WQA- -oythhettttic semi-autonnn automatic annotation pipeline. The watermark description and quality score of watermark
information constructed from given watermarked image through four-step process.
the data consists of 2.5K data pairs of {I, Q, T, Vj. Here, the generation of related question Q is the same as the one detailed in Sec. 2.1.
Stuge Object Positioning Pre-itaining
Slage Synthetic Watermarking
Stage 3
Real Watemmrkingg
Question answering
Quetion-snovwriin
Large Language Model
Large Language Model A
Large Language Model
Collection of Real Watermarked Images. To generate watermarked images that closely mirror real-world scenar- ios, we select a range of representative image watermarking models, including Hidden (Zhu, 2018), BalujaNet (Baluja, 2019), WengNet (Weng et al., 2019), HiNet (Jing et al., 2021), and Safe-SD (Ma et al., 2024). The first four mod- els are post-processing techniques, while the latter is an in-generation approach. Using all five models, synthe- size 10K watermarked images. A real-world watermarked image dataset is then created by randomly selecting 2.5K of these images, ensuring they reflected the distribution of watermark visibility levels. This dataset provides our source of watermarked images
Visual
Visual
Visual
Abstractur
Abstractur
Abstractio
Vision Encoder
Encoder
Encoder
COCO
Leamable
WQA-Symbhrikc Image
Leamable
WQA-Real Image
Lournable
Queties
Queries
Queries
Model architecture and the progressive learning paradigm of WMarkGPT. We employ a meticulously designed three-stage training pipeline sequentially optimize different model components, progressively enhancing performance and achieving optimal final results.
Figure 6.
Dnestion-Answering and Watermark Visibility Anno- fating. To obtain watermark descriptions and assess the visibility of watermarked images in real-world scenarios, we conduct a subjective experiment with human participants who are trained to master relevant background knowledge. The participants are trained to describe the watermark by considering factors such as its position, distribution, con- tent, relationship with the main object, features (e.g., trans- parency, texture), and visibility. These descriptions then form the source for the watermark description T. After completing the description, participants assess the visibility
of each watermarked image based on predefined criteria, which forms the watermark visibility V.
3. WMarkGPT
3.1. Model Architecture
WMarkGPT is specialized multimodal large language model explicitly designed for watermarked image com- prehension. By seamlessly integrating visual representa- tions with linguistic embeddings, WMrrkGPT enables the
5

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
recognition and detailed articulation of watermarked image content without requiring original reference images. It ad- dresses key limitations of traditional evaluation methods, including the reliance on reference images, the absence of watermark information and localization, lack of descriptions about semantic corruption, and insufficient visibility assess- ment. Notably, our visibility prediction logits yield scores highly consistent with human ratings, despite these ratings not being used to supervise the model training.
dataset and GPT-4. The COCO dataset provides bounding box annotations for primary objects alongside image cap- tions, forming the foundation for generating two types of guestion-answer pairs with GPT-4. For absolute positioning, GPT-4 generates questions about objects located in specific regions of the image, such as the top, bottom, left, or right. For relative positioning, GPT-4 creates questions regarding the spatial relationships between objects, such as identify- ing which objects are above, below, to the left, to the right of a given object. Using this dataset, we train the vision encoder, visual abstractor, and learnable queries of WMTrkGPT, enabling the model to effectively extract and process object positional features.
To facilitate these challenging functions, inspired by mPLUG-ow1-2 (Ye et al.. 2024), our WMarkGGTT incor- porates an LLM as its core component, complemented by a vision encoder, a visual abstractor, and learnable queries as depicted in Fig. 6. Specifically, WMarkGPT processes 448 x 448 watermarked image as input, which is first handled by the vision encoder to produce 64 image tokens, each with a dimension of 1024. These tokens are combined with set of learnable queries of the same shape and passed through the visual abstractor, where information exchange
Synthetic Watermarking Question-answering. In this stage, all trainable parameters are unfrozen for super- vised fine-tuning. The model is trained our high-quality synthetic dataset, WQA-Synthetic, which consists of a large number of watermarked images. This dataset is designed to familiarize the model with the unnatural data distribu- tion and question-answering tasks specific to watermarked images. Through this stage, the model's s ability to gener- ate detailed textual descriptions is significantly enhanced, including precise watermark positioning, relevant features. and visibility assessment.
Stage-2:
occurs via multi-head self-attention mechanisms, distill- ing task-specific semantic representations into the learnable queries. Finally, the refined learnable queries are concate- nated with the encoded textual embeddings of question and jointly processed by the LLM to generate the textual response.
Stage-3: Real Watermarking Question-answering. To further enhance the model's ability to process real water- marked images, we conduct additional fine-tuning using a small but high-quality dataset, WQA-Real, featuring metic- ulously annotated watermarked images. By this stage, the model has already developed strong understanding of wa- termarked images through the earlier training phases. There- fore, we freeze the LLM and learnable queries, focusing fine-tuning exclusively on the vision encoder and visual abstractor components. This targeted fine-tuning ensures alignment with the real data distribution, enabling the model to generate accurate and detailed descriptions, thereby im- proving its performance.
3.2. Training Paradigm
Watermarked images are generated by embedding water- mark patterns into original images, resulting in significant domain differences compared to naturally captured images. Addressing the task of describing watermarked images us- ing MLLMS requires overcoming two major challenges. First, the model must accurately describe the spatial rela- tionships between watermark distributions and primary ob- jects within the image. Second, watermarked images differ fundamentally from natural images, as they are the fusion or combination of natural image content and watermark logos, containing two highly distinct visual components. To tackle these challenges, we design a three-stage vision-language training strategy, as illustrated in Fig. 6, to systematically endow the model with these capabilities. In the first stage, the model is trained on an object location-aware QA dataset based on natural images. In the second stage, it is further op- timized on synthetic watermarking QA dataset, and in the final stage, fine-tuned on a real watermarking king QA dataset. Throughout all stages, cross-entropy loss is employed to measure the discrepancy between the predicted outputs and ground-truth labels.
4. Experiments
4.1. Main Results
Quantitative Comparison. To demonstrate the effective- ness of WMarkGPT, we conduct a comprehensive quan- titative comparison against several state-of-the-art multi- modal large language models (MLLMs) on both the WQA- Synthetic and WQA-Real datasets, including Qwen2-VL- 7B- (Wang et al., 2024), Qwen-VL-Chan VL-Chat (Bai et al., 2023), -LaVA-1.5-77 (Liu et al., 2024a), LLAVA-1.5-13B (Liu et al., 2024a), VILA-SB (Lin et al., 2024), and mPLUG- owl-7B (Ye et al., 2024). The details of the evaluation met- ries are available in Appx. G. To ensure a fair comparison, we standardize the input format for each model. The quan-
Stage-1: Object Positioning Pre-training. This stage aims to enhance WMrrkGPP's ability to perceive and understand object positioning within images. To achieve this, we con- struct the object location-aware QA dataset using the COCO
6

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
Table 1. Performance comparison between WMarkGPT and six state-of-the-art MI.LMs on the WQA-Synthetic and -QA-Real datasets across four evaluation metrics. It shows that WMarkGPT consistently demonstrates superior performance on these two datasets.
WQA-Synthetic
-eA-Real
Models
Backbone
BLEU-1
ROUGE-I. M.or-
ACC
BLEU-1
ROUGE-I. M.MMSScore
ACC
Qwen2-V See--VLL------eee
Qwen-7B
0.211
0.326
38.645
0.209
0.328
0.289
24.450
0.156
Qwen- Cha
Qwen-7B
0.315
0.289
37.078
0.211
0.255
0.253
25,450
0.166
LLaVA-1.5-7B
Vicuna-1.5-7B
0.311
0.308
26.778
0.210
0.237
0.270
15.600
0.118
LLaVA-115-13B
Vicuna-L.5-13B
0.285
0.289
40.220
0.217
0.228
0.256
25.650
0.134
VILA-8B
LLaMA-2-7B
0.251
0.328
49.961
0.236
0.190
0.253
28.800
0.172
1PL.G--w12-7BB
L.aMM-2-7B
0.351
0.318
36.229
0.201
0.279
0.277
17.150
0.110
0.488
0,446
87.751
0.645
0.424
0.418
71.950
0,546
WMarkGPT
ILLMA-2-7B
(439.0%)
(+36.0%%
(475.6%)
(+173.3%)
(429.3%)
(+44.6%)
(+149.8%)
(4217.4%)
esPrompt: Meess provde a denited descrigtion of the wonemark semss of diserion, its plocoment nelative etgectss ins apperrene, and its visiilry.
muL.VVA-1.5 78: The wazemork placed in the lower left corner the inage. It pamallly overlapping the sain mooks and is visible bet cot obtrusive. The watemmak has tectured appearance, which adds kyyer of donall to the image. The watemmort vissiilty moderate.
R
wesemmrk plased in upper right corner the image, with onientation being venical. It pettiall overlepping red lavis on right of the imaps, periilly covering from of mwin. The watermak appears to be simple, solid, and while line. The visibliy of watemark is moderats as not compleell oberacting imagee muicesble.
ew wwW.arGFP watemmrk bottoomally oriented positiooed neer the lower centaal region of image. pantailty avertaee traiss depacted is seene, coverting portioss thet while leaving parts unerficcted The estemmert has traneluoent slightly textined appearence, Hlending modernnely with has lgroond elemeons while remaiting dicemble wonemmnt visbilty is
etPremup:: Can you desarite the watemaak in the image. focusing oz orizitation postion relative dbjecs, appearanee, and visiilay?
AMLLAVA-LS--P 7.: The watemarks is placed in the lower lefl conner of the imaae with a viaßble and moderat lovel of visbility It positional ncur the elge of the sible, and its sppexrance is combiration of test and a logn The woommrk is rur abaracing the view of man using the laprp, bat is still netiooble
#tmPLEG G- 12- TB: The wosemark in the image is placed the lower right comer of photo, oovering signfficart portson ohe bottom right side. it partialy overlappinn the laptop and bonks parislly obscarig thei denals The watermark fos white appoormnet and is vioble to the raked eye
watermakk onieneed acroes significaoe portion image, spanning the right area exteoding sowards odges overlogs with autipppe otjects, including the parson, chair, laptop, and disiny thil covering then cithr partialls or entrrdy, wxtermak sppears faint Henda subily into the image, with soft texture does not disrupt visual clarity the jets siniiiiurtyy The watemar vaibiiit is fair.
seew MarkGFT::
ehPremip:: Could you summariee the watemark'ss 's otiernatoon its location relative object, is appeastnce, and how cleady his valklet
a 7.85-- The watemork plood in the lower left comeer the image. It panally overlapping the dog. with the dog's hend and mouth pantally covering the watermark, The waterment appean bes small, whine rectrnd objict The wntommnt vichilty is modemar, it is panisily covened by the deg's head and mouth.
#hoPLUG-au aw The wosemmrk in the image is oncered vemically and pamally overapping the puppy on the bad, is plsced near the edge of poppy, partially covering face, The watemark has white color and smooth tocum The watemors vishiity is modeans as not compltely obscuring the pappy fur stil noticeslee.
4sWMarkGPT: watcmark the image compleiely invidble the naked oye. iniegnsos scmboally into backgronn and objects, including the dog's punting and sumounding
mutaees, wilhout Scani inmoprrecyy color coninst or visual inseffreeee. The watermark blenda perfectly with the image's texture and color pulctic to crous that it not overlap interfere with any deuals, maintaining the natual apocarance the image. The watemant visbiliy is insisble..
Figure 7. Qualitative comparison of watermark descriptions between WMarkGPT and other open sourced MLLMs on watermarked images with varying visibility, which demonstrates the ability of WMarkGPT to accurately identify watermark location and features, even when the watermark is not visible to the human eyes.
titative results, detailed in Tab. 1, show that WMarkGPT significantly outperforms the other models in terms of wa- termark description relevance, achieving higher BLEU-1, values. Moreover, GPT
Table 2. The BLEU-1 results for watermark descriptions of the five different visibility levels WQA-Real. Here, Lr, Lz, La. La and La represent the visibility levels of "invisible", "faint", "moderate". "visible", and "obvious", respectively.
ROUGE-L, and demonstrates superior visibility prediction accuracy, achiev- ing 217.4% higher ACC than the second-bess model, VILA-8B. To further investigate watermark description qual- ity across different visibility levels, we evaluate these mod- els on the WQA-Real dataset across five visibility categories. As shown in Tab. 2, WMarkGPT consistently achieves the highest BLEU-1 scores for watermarked image descriptions across all visibility levels, highlighting its robustness.
WQA-Real Real
Models
L1
La
Lg
L4
L5
Qwen2- VB-
0.31
0.33
0.33
0.37
0.36
Owen-VL-Chat
0.24
0.27
0.27
0.26
0.27
LLaVA-1.5-7B
0.24
0.23
0.23
0.26
0.23
LL5VA-1.5-13B
0.23
0.21
0.23
0.25
0.23
VIIAA-8B
0.19
0.16
0.19
0.21
0.21
Qualitative Evaluation. In addition to quantitative com- parisons, we perform a qualitative evaluation using water- marked images from diverse scenes in the test set, particu- larly those with subtle or absent watermarks (Fig. 7). This analysis reveals that WMT accurately identifies wa- termark position and features, even in challenging cases. In contrast, models like L.I 5.L and mPL.UGoo]--
mPLUG-ow12-7BB
0.27
0.28
0.29
0.29
0.27
WMarkGPT
0.44
0.38
0.41
0.44
0.43
7B often provided less accurate descriptions, and, notably, misidentified background patterns as watermarks when the watermark is visually imperceptible. WMarkGPT, however, correctly recognizes the absence of a visible watermark,
7

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
Table The visibility prediction probability distributions and quantification method for watermarked images with varying vis- ibility levels. the table, the watermark (A) more visible compared to (B), as indicated by its smaller weighted score. Here, T, demotes the watermark description.
Table 4. The impact of using varying size of WQA-Synthetic data for fine-tuning. As the training proportion increases, the model's ability to understand watermarks improves, resulting more accu- rate watermark descriptions.
NQA-Synthetic
Watermarked
(A)
Size
BLEU-1
ROUGE-L
LLM-Score
ACC
Image
OK
0.351
0.318
36.229
0.201
10K
0.450
0.405
77.706
0.493
30K
0.448
0.407
79.267
0.555
0,488
0.446
87.751
0.645
50K
(+8.9%)
(+9.6%)
(+10.7%%
(+16.2%)
T,
The watermark is oriented horizon- tally across the central portion of the image. The watermark visibility is: visible.
Table S. The performance of the model WQA-Real after three
Level
L1
L2
La
L4
Ls
different training stages. S1, S2, and S3 represent the Stage-1, Stage-2, and Stage-3, respectively.
Probability
9.39
6.24
12.84
18.92
17.27
Softmax
0.000
0.000
0.002
0.837
0.161
Weight avg.
1.84
(Range:[ (1:5]
Stages
WQA-Real
Watermarked
(B)
BLEU-1
ROUGE-L LLM-Scom
ACC
Image
Baseline
0.279
0.277
17.150
0.110
S₁
0.281
0.280
30.324
0.235
S1+ S2
0.290
0,283
40.400
0.320
0.424
0.418
71.950
0.546
S1 S2 S3
(446.22%)
(+47.7%%
(+78.1%)
(+70.6%)
T;
The watermark is oriented horizon- tally across the lower portion of the image. The watermark visibility is: faint.
formance, we train WMarkGPT using three different sizes of the training data: OK, 10K, 30K, and 50K images, respec- tively. The experimental results, detailed in Tab. 4, demon- strate that model performance improves as the amount of synthetic training data increases. Specifically, using 10K im- ages results in 61.3% and 46.2% improvement in BLEU-1 and ROUGE-L scores, respectively, compared to the OK results. Furthermore, increasing the training data from OK to 50K images led to 319.5% and 396.4% enhancement in I.I M-Sorr and ACC metrics, respectively.
Level
L1
L2
L3
L4
Ls
Probability
13.18
17.63
14.3
11.65
9.04
Softmax 0.011
0.952
0.034
0.002
0.000
Weight avg.
3.97
(R:5ge:[1:5))
demonstrating its robust performance. To further analyze watermark visibility quantitatively, we visualize the predic- tion results for two different visibility levels in Tab. 3. The descriptions from WMarkGPT conclude with the phrase "The watermark visibility is <level>", and we observed a strong correlation between the probability distribution of the <level> output and the actual watermark visibility. Following the approach in (Wu al., 2023a), we assign numerical values (1-5) to the visibility levels and calculated weighted score based on the probability of each level. This score objectively reflects watermark visibiliity, with lower values indicating more visible watermarks. As illustrated in Fig. 3, this quantitative method proves to be a more accu- rate measure watermark visibility compared to traditional metries like PSNR.
Effects of Progressive Training Pipeline. To assess the impact of our strge training pipeline, we evaluated the performance of WMarkGPT after each stage the WQA-Real dataset. The results, detailed in Tab. 5, show clear progression of improvement. After the Stage-1, the model achieved a 76.81% 1.% increase in LLM-Score and a 113664% increase in visibility prediction accuracy (ACC) compared to the baseline. Following the Stage-2, all four evaluation metries showed further improvement, with the LLM-Score increasing by 33.23%. Finally, the complete three-stage training resulted in significant enhancements over the second stage, with BLEU-1, ROUGE-L, LLM- Score, and ACC increasing by 46.20%, 47.70%, 78.09%, and 70.63%, respectively. These results demonstrate that our stepwise three-stage training effectively enhances the model ability to perceive watermark content.
4.2. Ablation Study
Effects of Varying Size of WQA-Synthetic. To investigate the impact of the WQA-Synthetic dataset size on model per-
8

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
5. Conclusion
Chen, Y., Sikka, K., Cogswell, M., Ji, H., and Divakaran, A. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 14239-14250, 2024.
This paper presents WM.rkGPT, the first MLLM specif- ically designed for watermark content understanding. To train the model for watermark content perception, we de- veloped three visual question-answeree datasets: an object location-aware dataset, synthetic watermarking dataset, and a real watermarking dataset. Furthermore, the model employs a three-stage training pipeline that progressively bridges the gap between natural images and watermarked images, achieving superior results watermark understand- ing. Future work will explore the application of watermark understanding in other modalities, such watermarked videos and 3D-generated watermarked content, to further advance watermarking technology.
Fernandez, P., Sablayrolles, A., Furon, T., Jégou, H., and Douze, M. Watermarking images in self-supervised la- tent spaces. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3054-3058. IEEE, 2022.
Fernandez, P., Couairon, G., Jégou, H., Douze, M., and Furon, T. The stable signature: Rooting watermarks in latent diffusion models, 2023.
Acknowledgements
Golestaneh, S. A., Dadsetan, S., and Kitani, K. M. No- reference image quality assessment via transformers, rel- ative ranking, and conssconns In Proceedings of the IEEE/CVE winter conference on applications of computer vision, pp. 1220-1230, 2022.
This research was supported by the National Natural Sci- ence Foundation of China (62306111 17 and 62302385), the Guangzhou Basic and Applied Basic Research Founda- tion (2024A0413681), I), and GJYC program of Guangzhou (2024D03300055.
Guo, J., Li, Y., Wang, L., Xia, S.-T., Huang, H., Liu, C., and Li, B. Domain watermark: Effective and harmless dataset copyright protection is closed at hand. Advances in Neural Information Processing Systems, 36, 2024.
Impact Statement
This study introduces WMarkGPT, the first multimodal large language model (MLLM) designed for watermarked image analysis without requiring access to original images, addressing major limitation of existing evaluation metrics. WMarkGPT predicts watermark visibility and describes its location, content, and semantic impact, improving assess- ment reliability. This research has critical implications for watermark protection and digital content safety, enabling more robust evaluation of watermarking techniques against unauthorized removal and content manipulation.
Hosny, K. M., Magdi, A., ElKomy, O., and Hamza, H. M. Digital image watermarking using deep learning: A vey. Computer Science Review, 53:100662, 2024.
Jiao, Q., Chen, D., Huang, Y., Li, Y., and Shen, Y. Img-diff: Contrastive data synthesis for multimodal large language models. arXiv preprini urXiv:2408.045: 04440.040444, 2024.
Jing, J., Deng, X., Xu, M., Wang. J., and Guan, Z. Hinet: Deep image hiding by invertible network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4733-4742, 2021.
References
Abdi, H. The kendall rank correlation coefficient. Encyclo- pedia of measurement and statistics, 2:508-510, 2007. 2007.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead. S., Berg, A. C., Lo. W.-Y., al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4015-4026, 2023.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few- shot learning. Advances in neural information processing systems, 35:23716 23736,, 2022.
Korhonen, J. and You, J. Peak signal-to-noise ratio revis- ited: Is simple beautiful? In 2012 Fourth international workshop on quality of multimedia experience, pp. 37-38. IEEE, 2012.
Bai, J., Bai, S., Yang. S.. Wang, S., Tan, S., Wang. P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping pre-training for unified vision-language understanding and generation. In International confer- ence on machine learning, pp. 22888-12908 PMLR, 2022.
Baluja, S. Hiding images within images. IEEE transactions on pattern analysis and machine intelligence, 42(7))1685- 1697, 2019.

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
JI, Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping trgguage-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19770 PMLR, 2023.
Sedgwick, P. Pearson's correlation coefficient. Bmj, 345, 2012.
Sedgwick, P. Spearman's rank correlation coefficient. Bmj, 349, 2014.
Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., and Han, S. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 26689- 26699, 2024.
Sharma, S., Zou, J.J., Fang. G., Shukla, P., and Cai, W. A review of image watermarking for identity protection and verification. Multimedia Tools and Applications, 83(11): 31829-31891, 2024.
Su, S., Yan, Q. Zhu, Y., Zhang, C., Ge, X., Sun, J., and Zhang, Y. Blindly assess image quality in the wild guided by a self-adaptive hyper network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3667-3676, 2020.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCY 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740- 755. Springer, 2014.
Wang, J., Min, W., Hou, S., Ma, S., Zheng, Y., Wang, H., and Jiang, S. Logo-2k+: A large-scale logo dataset for scalable logo classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6194-6201, 2020.
Liu, H., Li, C., Li, Y., and Lee, Y. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 26296-26306, 2024a.
Wang. J., Chan, K. C., and Loy, C. C. Exploring clip for as- sessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 2555-2563. 2023.
Liu, H., Li, C., Wu, Q.. and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024b.
Liu, S., Li, J.. Zhao, G., Zhang. Y., Meng, X., Yu, F. F.R.,, Ji, X., and Li, M. Eventgpt: Event stream understanding with multimodal large language models. arXiv preprint arXiv:2412.00833 2024c.
Wang. P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
Lu, Y. Yang, X., Li, X., Wang, X. E.,, and Wang, W. Y. Limscore: Unveiling the power large language models in text-to-image synthesis evaluation. Advances Neural Information Processing Systems, 36, 2024.
Wang. Z., Bovik, A. C., Sheikh, H. R., and Simoncelli. E. P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions Image Processing, 13(4):600-62,, 2004.
Ma. Z., Jia, G., Qi, B., and Zhou, B. Safe-sd: Safe and traceable stable diffusion with text prompt trigger for invisible generative watermarking. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 7113-7122 2024.
Weng, X., Li, Y., Chi, L.., and Mu, Y. High-capacity con- volutional video steganography with temporal residual modeling. In Proceedings of the 2019 on international conference multimedia retrieval, pp. 87-95, 2019.
Wu, H., Zhang, Z., Zhang, W., Chen, C., Liao, L., Li, C., Gao, Y., Wang, A., Zhang, E., Sun, W., et al. Q-align: Teaching Imms for visual scoring via discrete text-defined levels. arXiv preprint arXiv:23 /2. 17090, 2023a
Rachmawanto, E. H., Hidajat, M. S., Hermanto, D., Wi- bowo, D. A., Pratama, Z., Setiarso, I., Astuti, E. Z Handoko, L. B., and Yaacob, N. M. Quality measure- ment for imperceptibility watermarking based on psnr and ssim using walsh hadamard transform. In 2024 In- ternational Seminar on Application for Technology of Information and Communication (iSemantic), pp. 353- 358. IEEE, 2024.
Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.-S. Next- gpt: Any-to-any multimodal 11m. arXiv preprint arXiv:2309.05519 2023b.
Xue, L., Shu, M., Awadalla, A., Wang, J., Yan, A., Pu- rushwalkam, S., Zhou, H., Prabhu, V., Dai, Y., Ryoo, M. S., et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08877 2024.
Rezaei, A., Akbari, M., Alvar, S. R., Fatemi, A., and Zhang, Y. Lawa: Using latent space for in nn-generation image watermarking, In European Conference on Computer Vision, pp. 118-136. Springer, 2025.
10

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
Ye, Q. Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q. Zhang, J., and Huang, F. mplug-ow12: Revolution- izing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition, pp- 13040-13051, 2024.
Zhang. R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018a.
Zhang, W., Ma, K., Yan, J., Deng, D., and Wang, Z. Blind image quality assessment using a deep bilinear convolu- tional neural network. IEEE Transactions on Circuits and Systems for Video Technology, 30(1):36-47, 2018b.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Zhu, J. Hidden:
hiding data with deep networks. arXiv preprint arXiv:1807.09937, 2018.

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
Appendix
In this appendix, we provide additional technical details, dataset construction methods, and supplementary experiments. See. A provides detailed summary of related works. Sec. details how our model computes the visibility score for watermarked images. Sec. C presents the text templates used in the production of the object location-aware dataset. Sec. D presents the text templates used in the semi-automatic annotation process of WQA-Synthetic and showcases intermediate results. Sec. E explains the details and standards of the subjective experiments conducted for WQA-Real. Sec. F describes the implementation details. Sec. G elaborates on the evaluation metrics, particularly the LLM-Score and ACC metries. Sec. H compares our model's visibility score predictions with other score-based methods, highlighting the advantages of our approach.
A. Related Works
Multimodal Large Language Models. Early works like BLIP (Li et al., 2022) and Flamingo (Alayrac et al., 2022) have demonstrated strong cross-modal understanding capabilities through laage-scale image-text alignment pre-training. Further advancements, such as MiniGPT-4 (Zhu et al., 2023) and Next-GPT (Wu et al., 2023b), enable transformations between arbitrary modes, while models like BL.I-2 (Li et al., 2023) and DRESS (Chen et al., 2024) incorporate human feedback to improve the alignment with human intentions. Improved datasets, as seen in VL- (Liu et al., 2024a), and optimizations in attention mechanisms, such as in mPLUG (Ye et al., 2024), have further boosted performance and efficiency. However, most existing MLLMs focus on natural image understanding for tasks like text-image matching or dialogue generation, lacking optimization for watermarked image comprehension. Our work addresses this gap by training MLLMs analyze unnatural images formed through the fusion of distinct contents, enabling the understanding of object positioning, watermark influence, and visibilitty assessment.
Watermarking Efficacy Evaluation. Traditional methods for evaluating watermarking primarily rely on pixel-based metrics such as peak signal-to-noie ratio (PSNR), structural similarity index (SSIM), and mean squared error (MSE), which quantify pixel-level differences between original and watermarked images. Recent learned perceptual image patch similarity (LPIPS) (Zhang al., 2018a) 8a) is perceptual similarity metric that measures differences between images based on deep features extracted from pre-trained neural network, capturing perceptual differences beyond pixel-level distortions. While these metrics effectively capture low-level distortions or feature-level shifts, they fail to accurately assess watermark visibility and influence on semantics. Moreover, existing evaluation methods rely on access to both original and watermarked images. In contrast, our work trains an MILM to evaluate watermarked images without requiring original references. provides detailed textual descriptions, including the watermark's s location, content, its impact on the image's semantics, and visibility assessment, addressing the limitations of conventional approaches.
B. Visibility Score Prediction
After training, the model demonstrates strong capabilities in summarizing the content of watermarked images and accurately judging the visibility of the watermark. Its output is plain text description, with the final sentence following a fixed format: "The watermark visibility is: < level >." To convert the rating levels back into scores, we define a reverse mapping G from text-defined rating levels to numeric scores, as follows:
G: L, +,
Where (4, i== I,..., reppeseets the five levels from obvious to invisible. Considering that the predicted < level token by the LMM represents probability distribution over all possible tokens in the vocabulary, we perform a restricted softmax operation to obtain the probability PL, of each level L,, ensuring the sum of PL, equals 1. The final predicted score of the LMM is computed as follows:
x:
Q = Σ GL,)(
Σ
ΣØ=1
ex,
Here, X, represents the logits corresponding to each level Li, ensuring a normalized and interpretable visibility score prediction.
12

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
C. Templates of Object Location-aware Dataset
This dataset primarily designed to enhance the model's ability to perceive the spatial positions of objects. Specifically, we approached the data design from two perspectives: absolute position descriptions and relative position descriptions. For the former, aim to obtain the absolute positional description of an object in the image. For the latter, we focus on describing the position of one object relative to another. To construct this dataset, utilized the original bounding box annotations from the COCO dataset. Object boundary information was used to create question templates, and with the help of GPT-4, we generated descriptive information regarding the objects' positions. The details of the question templates and relevant examples are provided in Fig. 1.
Absolute position description
##Template: Given the following information:
Image size: [w, h]
Object bounding box: y X2
Please output only one of the object's position in the image. Positions can be: "Top-left", "Top-right". "Bottom-left". "Bottom-right" "Center". "Middle-top". "Middle-bottom". "Middle-lefi". "Middle-right"
Relative position description
## Template: Given the following information: Image size: [we, ht]
<Object I> bounding box: Fm
Fmn X12 V12l
<Object 2> bounding box: [x2 V2r X22 Yz2l
Where <Object relation to <Object 2>? Please output only one of the following: "Top-left", "Top-right", "Bottom- left". "Bottom-right", "Center". "Middle-top". "Middle-bottom" "Middle-left", "Middle-righ""
Absolute position deseription examples:
Relative position description examples:
<abject 1> = 'spoon" <object 2> 'hottle" [Xm Fm X Frzl [181.1, 85.0, 204.2, 220.6 [N Fro X Fral [0.0, 29.6, 36.6, 92.6]
lobject name] "bottle"
[N, Jz] [0.0, 29.6, 36.6, 92.6]
GPT-4: Top-left
GPT-4: Bottom-right
h, w 224, 224
h, w 224, 224
<abject 1> 'book'
<object 2> 'ty"
[object name]
[Nin Fin Frrl [181.1, 85.0, 204.2, 220.6] [N Ym X1> F1zl [40.4, 102.8, 75.1, 127.0]
(xp No Vzl [64.9, 19.5, 107.8, 84.1]
GPT-4: Top-left
GPT-4: Bottom-right
I,, w 224, 224
h, 224, 224
Figure 1. Text template and examples for object location-aware dataset.
D. Templates for WQA-Synthetic
We introduce cost-effective and efficient semi-automatic annotation pipeline to construct the WQA-Synthetic. This pipeline is divided into four steps, with the second and fourth stages utilizing external LLMs, such as GPT-4 and GPT-4o. to assist in data annotation. In the following, we provide detailed explanation of the question templates used in these two steps, along with the corresponding intermediate results.
Main Object Box Detection. In this step, we use GPT-4 to judge the main object of the original image. We design a question template by combining the object category related to the original image and the bounding box coordinate information the image, as shown Fig. 2.
Grestion-Answering Generation. After obtaining the main object bounding box coordinates in the original image, watermark bounding box coordinates, and the watermark visibility, we utilize GPT-4o to generate detailed descriptions of the watermarked images. To ensure accurate descriptions, we designed question template that comprehensively considers both the watermark and original image bounding boxes. This template is used to input the watermark image into GPT-4o for summarization. The specific question template and relevant response examples are shown Fig. 3. In addition, we designed 30 relevant questions, with each generated text description being randomly matched one these questions.
13

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
##Template: Given image with size of [h, w| and object named (object name), located at the bounding box coordinates (N Fu X2 Val. Is the obiect in the boundany box area the main object in this image? Please respond with "Yes" or "No" only.
Examples:
lobject name] 'person" [x My Ky ysl [2.3, 11.0, 84.0, 170.3]
(object name] "cup" Dx Fn Kyx Jil - [37.8, 160.4, 65.6, 181.8]
[object namel "chair" [y yz] [49.6, 133.4, 134.0, 155.6]
GPT-:: Yes.
GPT-4: No.
GPT-4: No.
224, 224
(object name] 'woierr'
(object name] "botile"
[object name) "bottle"
[x Fb Ny ys] 124.1, 116.3, 219.0)
[x, N 3:] X=) (152.5,, 81.1, 161.6, 106.8]
NN X Val [142.5, 83.0, 151.8, 108.2]
GPT-4: Yes.
GPT-4: No.
GPT-: No.
224, 224
Figure 2.
Main object box detection question template and related examples.
Table
Classification and scoring criteria for WQA-Real
Level
Score Range
Standard
Invisible
[0.1]
The watermark almost completely disappears, cannot recognized in image, cannot detected no matter from which angle or condition it is viewed.
Faint
[1,2]
watermark not obvious, only under specific conditions or careful observation can barely seen, usually integrated with the background, the recognition is very low.
Moderate
[2,3]
The watermark is clearly visible most cases, but not overly prominent, at relatively unobtrusive level in the middle of the image where an observer can see the watermark without special effort.
Visible
[3,4]
The watermark is clearly visible, the contrast with the hackground high, and easy to detect, which may slightly affect the overall visual effect of the image, but does not attract too much attention.
Obvious
[4,5]
The watermark is very prominent, obviously affect visual effect of the image, easy to attract attention of observers, may block or interfere with the main content of the image.
example of such question is: "Please describe the watermark's direction, its placement relative to objects, its texture or appearance, and its visibility."
E. Details of The WQA-Real Production
To construct textual descriptions and visibility scores for watermarddd images, we conducted series of subjective experiments. Ten experienced annotators were selected and rigorously trained for the task. The annotators were instructed provide comprehensive descriptions of the watermark content in the images, including the special location, its relationship with the main objects in the image, its texture and shape characteristics, and its visibility. Finally, the annotators classified the watermark images according to a predefined visibility standard and assigned a visibility score. The specific criteria are outlined in Tab. 1.
F. Implementation Details
In our experimental setup, randomly select 5K and 0.5K watermarked images from the WQA-Synthetic and WQA-Real dataset correspondingly construct diverse test set, and use the remaining images the training set. As mentioned above, the training process is structured into three stages: In Stage-1, we generate 100K position question-answe answerr pairs to train the vision encoder and visual abstractor, with a batch size of 32, a learning rate of x 10-4, and for a duration This larger learning rate is chosen because only the visual components are trained in this stage. In Stage-2 and Stage-3, following the configuration the mPLUG-ow12, we apply fine-tuning with batch size of 16, a learning rate of 10
epochs.
14

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
## System: You are a helpful watermark discriminator:
##Templae:: <watermarked image> The image provided has watermark embedded it. The image contains the following objects with their bounding box coordinates:
(object 1: 1 12h-... object ox
The watermark embedded in the following areas:
wa (The visibility of watermark is <levet>)
In the input:
[xl. yl. x2. y2] represents the bounding box coordinates for detected objects in the image.
The watermark visibility is described with the following terms: invisible, faint, moderate, visible and obvious. The more obvious the watermark, the worse the watermark embedding effect.
First, describe the specific orientation of the watermark the image. Next, explain the relative position of the watermark to the objects in the image, indicating whether the watermark is inside, partially overlapping, Avoid mentioning any bounding box or coordinate information in your response. Then, briefly describe the watermark's characteristies, such as appearance or texture. Finally, assess the watermark visibility using the format: 'The watermark visibility is: [invisible/fainttmool htdpa/////ooddeeeeeeiibeeeeeeeee
the edge of the objects.
Present all this information in cohesive and concise paragraph..
Examples:
The watermark in the image is centrally oriented and extends across significant portion of scene. partially overlaps with all the objects, including the bottle, wine glass, dining table, and chair, but it does not fully obscure them. The watermark has layered and translucent appearance, with discernible text or patterns that give it textured effect. The presence of the watermark is prominent and easily noticeable. The watermark visibility is: visible.
##GPT-4o:
The watermark in the image is situated along one side the image, spanning vertically. It is positioned near the edge of the bed object and partially overlaps with the teddy bear object. The watermark has a faint, blended appearance, making it integrate subtly with the background. The texture appears smooth and does significantly disrupt the overall image quality. The watermark visibility is: faint.
##GPT-4o:
The watermark is korizontally oriented and located near the lower center of the image. It partially overlaps with some objects, such as the individuals in the foreground, but does not interfere with peripheral elements like kites in the sky. The watermark features a text-based design with a translucent appearance, blending into the background but still discernible. The watermark visibility moderate.
##GPT-40:
Figure 3. Text template and examples for Question-anssering generation.
and a duration epochs. Across all three stages, the Adam optimizer and cosine annealing learning rate schedule are used dynamically adjust the learning rate. The reported performance of WMrrkGPT is evaluated using the final weights obtained after fine-tuning and all experiments were conducted 8 NVIDIA RTX 6000 Ada GPUs.
G. Evaluation Metrics
To comprehensively evaluate the watermark understanding capabilities WMarkGPT and other MILMs. Traditional metrics like PSNR are not included this experimental section because they fail to provide textual descriptions and visibility, which are core of evaluation. Instead, we refer the reader to Fig. for a comparison of WMarkGPT with these metrics in other aspects. we compare their performance in two major aspects: (1) the relevance of their watermark descriptions, and (2) the accuracy of their visibility predictions.
(1) Relevance of Watermark Description. To evaluate the quality of generated watermark descriptions, i.e., their relevance to the reference text, we employ established natural language processing metrics. Specifically, we calculate BLEU-1 and ROUGE-L scores to measure werd-level similarity between the generated description and the reference text. We also employ the LLM-Score, as described in (Lu et al., 2024), to assess semantic relevance. This score ranges from 0 to 4, with higher scores indicating better relevance. For ease of interpretation, the final scores are normalized a scale of to
15

---

WMarkGPT: Watermarked Image Understanding via Multimodal Large Language Models
Table 2. Classification and scoring criteria for WQ-Real
WQA-Synthetic
WQA- Real
Models
SRCC
PLCC
KRCC
SRCC
PLCC
KRCC
DBCNN
0.91
0.91
0.75
0.63
0.70
0.46
HyperlQA
0.91
0.91
0.75
0.62
0.65
0,44
TReS
0.90
0.90
0.75
0.46
0.48
0.33
CLIP-IQA
0.89
0.86
0.71
0.43
0.46
0.31
WMarkGPT
0.93
0.93
0.78
0.71
0.74
0.54
100. Specifically, designed text template to evaluate the correlation between the generated content and the reference description with the help of GPT-4. The detailed text template is follows:
"Evaluate the relevance between the following description and ground truth on a scale from 0 to 4. Higher scores indicate better relevance. Return only the numeric score. Description: < &candidates > Ground Truth: < &reference >
(2) Accuracy of Visibility Prediction. A crucial part of evaluating the ability to understand watermarks from multimodal large language model is assessing the accuracy of its visibility predictions. In dataset, each watermarked image has corresponding visibility label, ranging from "invisible" to "obvious" (i.e., "invisible", "faint", "moderate", "visible", "obvious"). Unlike WMarkGPT, which directly outputs visibilityy predictions, other MLLM baselines describe visibility in various textual formats, often embedding the information within their generated text. To standardize the visibility classification, we use a question template and leverage GPT-4 to categorize the visibility level in the generated text from other MLLM baselines. Finally, we calculate the accuracy (ACC) of these visibility predictions in all images. The question template designed to summarize the visibility of the watermark as follows:
"You expert in image watermark analysis, specializing assessing the visibility of watermarks. I will provide textual description of an image watermark, and your task is to evaluate its visibility based the description. Choose the most appropriate visibility level from the following options: invisible, faint, moderale, visible, obvious. Provide only the selected visibility level as your output. -Input: < candidates
H. Comparison with Other Score-based Methods
Our model calculates the watermark visibility score by computing the probability of the final level word, providing more accurate representation of watermark visibility and the degree of image content degradation compared to traditional pixel-wise metrics. To further highlight the advantages of our model, we selected several score-based deep neural network models, including DBCNN (Zhang et al., 2018b), HyperIQA (Su et al., 2020), TReS (Golestaneh et al., 2022), and CLIP-IQA (Wang et al., 2023), and trained and tested them on score prediction tasks using the WQA-Synthetic and WQA-Real datasets according to the configurations from their respective original papers. We then compared the SRCC, PLCC. and KRCC metrics our model against these score-based methods. The experimental results, shown in Table 2, demonstrate that our method outperforms these score-based approaches in predicting watermark visibility scores, achieving results that are more consistent with subjective ratings. This superiority may be attributed to the fact that most score-based methods designed to assess image distortion, which does not capture the watermark's 's semantic content effectively, leading to poorer performance in watermark visibility score prediction tasks.
16


Reviews:
[{"review_id": "4ET4VJ9ydk", "reviewer": "Reviewer_5nKM", "summary": "This paper proposed use MLLM to detect watermarked images, the model architecture is adapted from mplug-owl 2 and proposes three training stages to progressively fine-tune model. Experimental results show improved performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and recognition on our clear presentation, contributions of extending the MLLMs ability to watermarked image understanding.\n\n**Q1: Experimental setup of baseline models**\n\n**Answer:** \nWe clarify that all baseline models were fine-tuned on our proposed datasets rather than directly evaluated in a zero-shot manner.\n\n**Q2: The effectiveness of progressive training strategy**\n\n**Answer:** \nWe conducted an experiment where the model was trained on all three datasets jointly in a unified manner using the same training configurations as in the progressive setting. The experimental results below indicate that the progressive training approach consistently yields much better performance compared to the unified training strategy. For example, on the WQA-Synthetic dataset, progressive training achieves an improvement of +0.064 in BLEU-1 and +5.367 in LLM-Score relative to unified training. Similar trends are observed on the WQA-Real dataset.\nThese results suggest that the progressive training strategy effectively mitigates the domain adaptation challenges inherent in joint training, leading to improved generalization across diverse data domains.\n\n| WQA-Synthetic    | BLEU-1 | ROUGE-L | LLM-Score | ACC  |\n| ---------------- | ------ | ------- | --------- | ---- |\n| Unified Training | 0.424  | 0.388   | 82.384    | 0.635|\n| Progressive Training | 0.488  | 0.446   | 87.751    | 0.645|\n\n| WQA-Real    | BLEU-1 | ROUGE-L | LLM-Score | ACC  |\n| ---------------- | ------ | ------- | --------- | ---- |\n| Unified Training | 0.375  | 0.376   | 69.547    | 0.522 |\n| Progressive Training | 0.424  | 0.418  | 71.950  | 0.546 |\n\n**Q3: Architectural enhancement for watermarked image understanding**\n\n**Answer:** \nWe thank the reviewer for this valuable suggestion. In the MLLM research literature [1, 2, 3], the focus has primarily been on rigorous multimodal data collection and innovative training strategies. As the first work on watermarked image understanding using MLLMs, we also place our main emphasis on the novel WQA-Synthetic/WQA-Real datasets and a three-stage training pipeline.\n\nThat said, we recognize that specialized architectural enhancements can further improve watermark understanding. To address the challenges arising from mixed image and watermark patterns, we have integrated Mixture-of-Experts (MoE) into the LLM backbone. This addition enables more effective processing of watermark-specific features alongside natural image semantics. Experiments have shown that this modification boosts performance compared to our baseline model, and we will include a detailed analysis in the final paper.\n\n| WQA-Synthetic    | BLEU-1 | ROUGE-L | LLM-Score | ACC  |\n| ---------------- | ------ | ------- | --------- | ---- |\n| WMarkGPT | 0.488  | 0.446   | 87.751    | 0.645|\n| WMarkGPT-MoE | 0.491 | 0.453 | 87.841 | 0.658 |\n\n[1] Li, Chunyuan, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao,  Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 2023.\n\n[2] Lin, Ji, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024.\n\n[3] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, andSteven Hoi. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. Advances in Neural Information Processing Systems, 2023.", "claims_and_evidence": "The paper may overstate the performance gain, i.e., Section 4.1 and Table 1, the paper does not mention whether those baselines such as Qwen2-vl are trained with the same datasets as those used in WMarkGPT. If the baseline models are zero-shot while the WMarkGPT is fine-tuned, the experimental results are unfair.", "methods_and_evaluation_criteria": "See claims and evidence.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "See claims and evidence.", "relation_to_broader_scientific_literature": "This paper contributes to extending the MLLMs ability to watermark detection field.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths: the paper is well-structured and easy to follow\nWeakness:\n1. The experimental designs are unfair in Section 4.1 if the baseline performance is obtained in zero-shot scenario.\n2. To thoroughly evaluate the effectiveness of progressive training, it is essential to introduce an additional baseline: a model trained simultaneously on all three datasets—the object position dataset, the synthetic watermark dataset, and the real watermark dataset. This approach will provide a comprehensive comparison, allowing us to assess whether progressive training offers distinct advantages over a more conventional, unified training strategy.\n3. The model architecture is unchanged, more specific devise can be done to fit the watermark understanding task.", "other_comments_or_suggestions": "No", "questions_for_authors": "No question", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d81LEMYmVl", "reviewer": "Reviewer_oNvm", "summary": "This paper constructs datasets comprised of watermarked images with different level of watermark visibility. The paper trains a model specified for describing watermark patterns and evaluating watermark visibility level. The authors compare their model with several existing multimodal large language models on their datasets, and they achieve better performance on four metrics. The paper is also clear and well written.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful feedback and recognition on our clear presentation, comprehensive comparisons and better results.\n\n\n**Q1: The explanation of the \"real\" dataset**\n\n**Answer:** \nCompared to our WQA-Synthetic dataset, which employs pseudo watermarking processes to generate watermarked images, WQA-Real uses genuine watermarking algorithms (e.g., Hidden, BalujaNet) to produce watermarked images that more closely align with evaluating data distributions. Additionally, all QA pairs and watermark visibility assessments in WQA-Real were manually annotated by trained human evaluators (Sec. 2.2 & Appendix E), ensuring that both descriptions and scores reflect genuine human perception rather than synthetic labels. WMarkGPT is proposed to comprehensively understand these watermarked images without accessing original images and further advance the development of watermarking algorithms. We use the term \"real\" to emphasize the distinctness of WQA-Synthetic and WQA-Real datasets.\n\n**Q2: Invisible watermark description**\n\n**Answer:**  Ideally, watermarking algorithms generate watermarked images with fully invisible watermarks. However, in practice, existing methods often produce images with varying degrees of watermark visibility. To advance the research, we propose WMarkGPT for precise and comprehensive evaluations of these watermarked images. Our WQA-Real dataset thus includes both visible and invisible watermarks (see Fig. 4 for distribution). For genuinely invisible ones, we label them simply as “invisible” without adding spatial descriptions (Appendix E).\n\n**Q3: Details of WMarkGPT**\n\n**Answer:** As detailed in Section 3.1 Model Architecture, WMarkGPT is built upon the LLaMA-2-7B framework (7 billion parameters). The complete model, including the vision encoder, visual abstractor and learnable queries, has a total of 8.198 billion parameters. We will make it clear in the final paper.\n\n**Q4: The reference texts used in LLM-Score**\n\n**Answer:** As detailed in Appendix G, our evaluation metric LLM-Score [1][2] uses the reference template: ``Evaluate the relevance between the following description and ground truth on a scale from 0 to 4. Higher scores indicate better relevance. Return only the numeric score. - Description: *candidates* - Ground Truth: *references*.\" \n\n[1]Lu Y, Yang X, Li X, et al. Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation. Neurips 2023.\n\n[2]Huang K, Sun K, Xie E, et al. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. Neurips 2023.\n\n**Q5: The size of the WQ-Synthetic dataset**\n\n**Answer:** \nWe conducted additional experiments by increasing the WQ-Synthetic dataset size. As shown in the table below, adding more training data only resulted in slight performance improvements. This might be due to the complexity of watermarked image understanding, where the model needs to predict watermark visibility and produce detailed textual descriptions of its location, content, and effect on image semantics. Our 7B-parameter LLM backbone may have reached its limit at this dataset scale. In our final paper, we will include experiments and analysis with a more powerful LLM backbone and larger datasets.\n\n| WQA-Synthetic    | BLEU-1 | ROUGE-L | LLM-Score | ACC  |\n| ---------------- | ------ | ------- | --------- | ---- |\n| 50K | 0.488  | 0.446\t | 87.751 |\t0.645 |\n| 60K | 0.488  | 0.448   | 87.723 | 0.638 |\n| 70K | 0.490  | 0.453   | 87.758 | 0.649 |\n\n**Q6: Model comparison**\n\n**Answer:** We clarify that all baseline models were fine-tuned on our proposed datasets rather than directly evaluated in a zero-shot manner.\n\n**Q7: Efficiency of WMarkGPT**\n\n**Answer:** The training and inference costs of our model are as follows (2 Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz). \n- Training: 12 hours (Training Stage 1) and 6 hours each (Training Stages 2-3) on 8×NVIDIA RTX 6000 Ada GPUs.\n- Inference: 3.395 seconds for a single image (avg.) on a single NVIDIA RTX 6000 Ada GPU.\n\n**Q8: Download links of the datasets**\n\n**Answer:** As stated in the abstract, we will publicly release both the code and dataset upon acceptance. To comply with the double-blind review policy, we provide an anonymous download link here: https://drive.google.com/drive/folders/1JoRq91b0UAbTU4SEblCycRVwCpUgkeg2?usp=drive_link. \n\n**Q9: Typo in Fig. 5**\n\n**Answer:** We apologize for the typo and will correct it in the final paper.", "claims_and_evidence": "Not really.  Please check the weaknesses.", "methods_and_evaluation_criteria": "Partially.   Please check the weaknesses.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Partially.  Please check the weaknesses.", "relation_to_broader_scientific_literature": "Yes", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "## Paper Strengths\n\n- Well written and easy to follow.\n- The authors make comprehensive comparisons with other models and they achieve better results.\n\n## Paper Weeknesses\n\n- The authors mentioned that when constructing the “real” dataset, they randomly selected images generated by existing models. Essentially, this still relies on AI to create the dataset, meaning it is not truly a real dataset.\n- In the “real” dataset constructed by the authors, only invisible watermarks are used. Given this, how do the authors later describe the position, content, and other relevant information of these watermarks?\n- The description of WMarkGPT is not clear. Which LLM is used and what is its parameter size?\n- Some of the evaluation metrics used by the authors require reference texts; however, the authors do not explicitly explain how these reference texts are obtained.\n- The ablation study is not very meaningful, as the maximum dataset size is only 50K images, which is relatively small. It is predictable that increasing the dataset size would improve the model’s performance. Similarly, it is evident that training and fine-tuning the model would enhance its capability.\n- The model comparison is unfair. It would be more reasonable to compare WMarkGPT with other models that have been fine-tuned on watermark description and evaluation.\n- The authors do not discuss the efficiency of their model. How long it takes to evaluate an image and how long it takes to train the model.\n- The authors do not provide the download links of their datasets.\n- In fig 5, step 2, there is a misspelling 'obiect', which should be 'object'.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iL3IqErGLI", "reviewer": "Reviewer_5Rr3", "summary": "The authors innovatively propose a new multi-modal large language model WMarkGPT for watermarked image understanding. This paper points out that traditional methods rely on indicators such as PSNR, require the original image, and cannot fully evaluate the influence of watermark on content. WMarkGPT predicts the visibility of watermarks and generates detailed descriptions without the need for the original image. In addition, the authors constructed three VQA datasets and designed a three-stage training process. The experiments show that WMarkGPT is superior to existing MLLMs in understanding watermark images on synthetic and real datasets.\n\n## update after rebuttal\nAuthors' rebuttal have solved my concerns, and I will keep my rating. Thanks a lot.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and recognition of our innovation, comprehensive experiments, superior performance and practical significance. We appreciate that you highlighted WMarkGPT’s ability to predict watermark visibility without the original images, the newly constructed datasets, our progressive training strategy, and the benefits for subsequent research.\n\n**Q1: The WQ-Real dataset and our application**\n\n**Answer:** \nWe adopt a three-stage training pipeline to progressively enable WMarkGPT to comprehend watermarked images. In the first stage, the model is trained on a large-scale object location-aware QA dataset based on natural images. In the second stage, it is further optimized on a synthetic watermarking QA dataset, bridging the gap from natural to watermarked images. Finally, the model is fine-tuned on a real watermarking QA dataset. Because the training sets in the first two stages are large, only a relatively small and high-quality real dataset is needed to align the model with practical data distribution. The significance of dataset quality over quantity has been well-established in MLLM research. To ensure that the watermarked images closely mirror real scenarios, we conducted a thorough investigation and selected typical image watermarking algorithms—including Hidden, BalujaNet, WengNet, HiNet, and Safe-SD—to build our real dataset. Experimentally, we expanded the WQA-Real dataset step by step and found that performance stabilized at around 2.5K samples.\n\n| WQA-Synthetic    | BLEU-1 | ROUGE-L | LLM-Score | ACC  |\n| ---           | ---    | ---     | ---       | ---  |\n| 1000 | 0.299 | 0.329 | 57.654 | 0.412 | \n| 1500 | 0.367 | 0.379 | 62.035 | 0.437 | \n| 2000 | 0.404 | 0.400 | 64.654 | 0.502 | \n| 2250 | 0.423 | 0.410 | 71.654 | 0.551 |\n| 2500 | 0.424 | 0.418 | 71.950 | 0.546 |\n\nOriginally, we designed WMarkGPT to evaluate watermarked images more precisely and comprehensively, particularly when original images are unavailable for generative watermarking. The success can facilitate the development of MLLMs for other tasks involving mixed visual patterns, such as deepfake detection and edited image comprehension. In addition, our dataset collection paradigm and training strategy can also be consulted.\n\n**Q2: Effect of optimizing visual encoder and abstractor**\n\n**Answer:** \nFor brief, the staged optimization of the vision encoder and visual abstractor enables the model to progressively refine its focus on watermarked regions.\nThe vision encoder capture low- to mid-level semantics, providing the basis for further abstraction and alignment with the language model. The visual abstractor refines the visual features into high-level representations through a set of learnable queries. By aggregating crucial image features and filtering out noise, it delivers more compact and meaningful visual embeddings for subsequent processing by the language model. We will add more explanations in the final paper. \n\n**Q3: Efficiency of WMarkGPT**\n\n**Answer:** As detailed in Section 3.1 Model Architecture, WMarkGPT is built upon the LLaMA-2-7B framework (7 billion parameters). The complete model, including the vision encoder, visual abstractor and learnable queries, has a total of 8.198 billion parameters. We have made a detailed summary of training and testing costs in the response to Reviewer oNvm Q7.\n\n**Q4: Effect of previous watermark related description**\n\n**Answer:** \nOur experiments show that predicting watermark visibility without first extracting a watermark-related description results in lower accuracy on both the WQA-Synthetic and WQA-Real datasets. This indicates that capturing key features—such as spatial structure and texture—beforehand is crucial for accurately estimating watermark visibility.\n\n| ACC    | WQA-Synthetic | WQA-Real |\n| ------ | ------- | --------|\n| w/o Prefix | 0.639 | 0.541 |\n| WMarkGPT | 0.645 | 0.546 |\n\n**Q5: Limitations and future works**\n\n**Answer:** The current WMarkGPT model is trained exclusively on image datasets, whereas video watermarking and its more precise evaluations remain crucial yet underexplored aspects of copyright protection. Future research will focus on collecting video watermarking datasets and developing video-based MLLMs to understand watermarked videos, further advancing digital watermarking technology.\n\n**Q6: More training details**\n\n**Answer:** We provide the complete configurations for the three training phases in the table below to ensure full reproducibility. After acceptance, we will release all the training code.\n\n| Environment | CUDA: 12.1.105  Torch: 2.4.1 |   |   |\n| ---------------- | -----------------------|-------------|---------|\n| Traning |\tStage 1 | Stage 2 | Stage 3 |\n| Batch size | 32 | 16 | 16 |\n| Learning rate | $1e-4$ | $2e-5$| $2e-5$ |\n| Epoch\t| 3 | 5 | 5 |\n|Optimizer|  AdamW |  |\n|Scheduler| CosineAnnealingLR | CosineAnnealingLR | CosineAnnealingLR |\n| Warmup ratio | 0 | 0.03 | 0.03 |\n| Precision | bf16 |   |   |", "claims_and_evidence": "Yes, the proposed methods and evaluation criteria are highly relevant and appropriate for the problem of watermark image understanding. The authors introduce WMarkGPT, a multi-modal large language model designed specifically to predict watermark visibility and generate detailed descriptions without requiring the original image. This approach effectively addresses the limitations of traditional methods like PSNR and SSIM, which rely on the original image and fail to comprehensively evaluate the semantic impact of watermarks. Additionally, the construction of three high-quality VQA datasets, including real watermark datasets, provides valuable resources for evaluating and improving the model's performance.", "methods_and_evaluation_criteria": "Yes, the claims made in the submission are well-supported by clear and convincing evidence. The authors provide comprehensive experiments demonstrating WMarkGPT's superior performance compared to existing multi-modal large language models on watermark image understanding tasks. The three-stage training process is validated through ablation studies, which clearly show the contribution of each stage to the final performance. The use of both synthetic and real datasets for evaluation ensures that the model's effectiveness is tested across diverse scenarios.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1.\tThe authors construct three high quality VQA data sets, especially the marking of real watermarking data sets provides valuable resources for field research. The public commitment of the data set will greatly facilitate subsequent research.\n2.\tThe three-stage progressive training strategy proposed by the authors is reasonably designed, and the ablation experiment fully verifies the contribution of each stage to the final performance.", "relation_to_broader_scientific_literature": "WMarkGPT innovatively addresses the limitations of traditional watermark evaluation methods by proposing a novel multi-modal model that predicts watermark visibility and generates descriptions without needing the original image. This contribution, along with the creation of high-quality VQA datasets and a robust training strategy, significantly advances the field of watermark detection and multi-modal understanding", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The content of this paper is full and the language is standard. \n2. This paper points out the limitations of traditional watermark evaluation methods (such as PSNR and SSIM) that rely on the original image and cannot comprehensively evaluate the visibility and semantic impact of watermark. This paper proposes a specific solution, which has clear practical significance and significantly improves the practicability of the method.\n\nWeakness:\n1. The real watermark dataset contains only 2.5k samples, and the watermark generation methods are limited to a small number of algorithms (such as HiNet, Safe-SD). Does small data size limit the model generalization? The authors need to further explain the scope of its application.\n2. It is suggested to add detailed explanations for the three-stage training strategy, such as why the visual encoder and abstractor are optimized in stages, so as to more comprehensively show the model's attention focusing ability in the watermarked area.", "other_comments_or_suggestions": "1.\tThe parameter size, operational efficiency, training and testing time and resource usage of the model need to be further elucidated.\n2.\tWatermark visibility prediction is also one of the important tasks of the model. Is the previous watermark related description beneficial for the final visibility prediction?\n3.\tPlease add a discussion section to further analyze the limitations of the method and suggest ways to improve it in the future.", "questions_for_authors": "1. Some training details are briefly described in the main text, and the appendix supplements some of them, but the reproducibility still needs to be further refined.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UhqAeCSpkj", "reviewer": "Reviewer_MkBD", "summary": "The paper introduces WMarkGPT, a multimodal large language model (MLLM) designed to understand watermarked images without requiring access to the original images. Specifically, it integrates a visual encoder, learnable queries, a visual abstractor, and an LLM to generate detailed descriptions of watermarks and predict their visibility. This work proposes a three-stage training pipeline that progressively enhances the model's ability to understand object positioning, watermark characteristics, and semantic corruption in watermarked images. The authors construct three visual question-answering (VQA) datasets: an object location-aware dataset, a synthetic watermarking dataset, and a real watermarking dataset. Extensive experiments demonstrate that WMarkGPT significantly outperforms existing MLLMs in terms of watermark description relevance and visibility prediction accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable comments and recognition of the novelty and meaningfulness of our research. The reviewer acknowledged the clarity and thoroughness of our experimental design, as well as the strong evidence backing our claims. They highlighted WMarkGPT’s superior performance in watermark description and visibility prediction. The reviewer also thought that our datasets and training pipeline provided a new benchmark for watermark security. In addition, the reviewer commended our comprehensive ablation studies and found the manuscript well-structured and easy to follow. \n\n**Q1: Effective of the learnable query**\n\n**Answer:** \nThe learnable queries are employed to extract high-level semantic features from images and filter out redundant visual noise. We conducted an experiment in which we remove the learnable queries and applied multi-stage SFT directly to the model, resulting in considerable performance drops. This shows that multi-stage SFT alone is not sufficient, as it lacks the targeted feature abstraction provided by the learnable queries.\n\n| WQA-Synthetic    | BLEU-1 | ROUGE-L | LLM-Score | ACC  |\n| ---------------- | ------ | ------- | --------- | ---- |\n| w/o Learnable Query | 0.474 |\t0.433 |\t82.981 | 0.613 |\n| WMarkGPT | 0.488 | 0.446 | 87.751 | 0.645 |\n\n| WQA-Real   | BLEU-1 | ROUGE-L | LLM-Score | ACC  |\n| ---------------- | ------ | ------- | --------- | ---- |\n| w/o Learnable Query | 0.401 | 0.394 | 69.788 | 0.516 | \n| WMarkGPT | 0.424 | 0.418 | 71.950 | 0.546 |\n\n**Q2: Evaluating watermark position prediction**\n\n**Answer:** We agree that evaluating watermark position prediction is crucial. As suggested, we conducted additional evaluations to verify our performance. We employed the following prompt and GPT to assess the position prediction of our WMarkGPT:\n\n_Task: Determine whether the following two sentences describe approximately the same watermark position, allowing for minor variations in phrasing or slight positional bias._\n\n_Sentence 1: {predicted description}_\n_Sentence 2: {ground truth}_\n\n_Judgment Criteria:_\n_1.  Do they describe the same general location, even if there are small differences?_\n_2.  Are the differences within an acceptable range (e.g., slight shifts in coordinates or wording variations like \"top-left\" vs. \"upper-left\")?_\n_3.  Is there any ambiguity that affects the interpretation?_\n_4. Does the predicted description avoid mentioning any position information, as the watermark is inherently invisible?_ \n\n_Expected Output:_\n_Just Final Consistency Judgment: (Yes/No)_\n\nThe precision, recall, and accuracy of the classification are computed referring to the visibility/invisibility of the watermark:\n\n| Dataset   | Precision | Recall | ACC |\n| ---------- | ------ | ------- | --------- |\n| WQA-Synthetic | 0.851 | 0.830 | 0.823 |\n| WQA-Real | 0.792 | 0.763 | 0.734 |\n\n**Q3: Structure of Visual Abstractor**\n\n**Answer:** The visual abstractor comprises six transformer layers, each employing cross-attention between extracted visual features and learnable queries. Through cross-attention with learnable queries, the model progressively refines visual feature extraction, ensuring these queries capture high-level semantic information while filtering out irrelevant visual clues. This design also strengthens the subsequent alignment between image features and textual descriptions. We will release the code after acceptance to clarify it.\n\n**Q4: The experimental comparisons**\n\n**Answer:** We appreciate the reviewer’s concern regarding the experimental setup of baseline models. To clarify, all baseline models were fine-tuned on our proposed datasets rather than directly evaluated in a zero-shot manner.", "claims_and_evidence": "The claims made in the paper are well-supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem.", "theoretical_claims": "The paper does not present any formal theoretical claims or proofs. The focus is on empirical improvements and model design. Therefore, this aspect is not applicable.", "experimental_designs_or_analyses": "The experimental designs and analyses appear sound and valid:\n- The authors conducted extensive experiments on both synthetic and real watermarking datasets, demonstrating the model's effectiveness.\n- The use of multiple evaluation metrics provides a comprehensive assessment of the model's performance.\n- The ablation studies provide insights into the impact of different training stages and dataset sizes on model performance.", "relation_to_broader_scientific_literature": "The key contributions of the paper are well-grounded in the broader scientific literature:\n- The use of multimodal large language models (MLLMs) for image understanding aligns with recent advancements in vision-language models (e.g., Qwen, LLaVA, VILA).\n- The focus on watermark understanding addresses a significant gap in existing evaluation methods, which rely on pixel-wise metrics and require access to original images.\n- The proposed datasets and training pipeline build upon prior work in VQA and multimodal learning, providing a new benchmark for watermark security.", "essential_references_not_discussed": "The paper has cited relevant works in multimodal learning, watermarking, and corresponding evaluation metrics.", "other_strengths_and_weaknesses": "**Strengths**:\n- The author carried out thorough ablation experiments to assess the effectiveness of the proposed method.\n- The manuscript’s expression and structure enhance its readability, making it easy for readers to understand.\n- This paper first proposes the use of MLLM to evaluate the content and visibility of watermarks, which is an interesting and meaningful topic.\n\n**Weaknesses**:\n- The authors do not appear to have conducted any ablation experiments to demonstrate the effectiveness of the learnable query setting for this task. I wonder whether similar performance could be achieved through data-efficient multi-stage SFT training alone ( like the paradigm of general Multi-modal understanding LLM ).\n- As the WMarkGPT model is capable of directly predicting the specific location of the watermark. The benchmark seems to lack an evaluation metric for watermark position prediction. If it would be possible to design an evaluation metric, similar to object detection tasks, that includes precision and recall to assess the model's effectiveness in watermark location prediction.", "other_comments_or_suggestions": "Please refer to Weaknesses and Questions.", "questions_for_authors": "- What is the specific network structure of Visual Abstractor? Is it a transformer block or just one layer of attention?\n- Have the baseline models compared by the author been fine-tuned on the constructed WQA dataset? If not, I would suggest evaluating the performance of these models after fine-tuning them to provide a more comprehensive comparison.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces WMarkGPT, a multimodal large language model (MLLM) designed to understand watermarked images without requiring access to the original images. The authors have done a good job of rebuttal. After rebuttal, it received scores of 3334. All the reviewers are happy about the paper, commenting that the paper is well written and easy to follow, and the authors make comprehensive comparisons with other models and they achieve better results. The AC agrees that the application of the MLLM developed in this paper is interesting; on the other hand, the novelty of the MLLM design itself also seems quite limited. Overall, given that all the reviewers are positive about the paper and the application of MLLM in this paper is indeed unique, the AC would like to recommend acceptance as well.
Decision: Accept (poster)