Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images
Authors: Yubo Wang
Jianting Tang
Chaohu Liu
Linli Xu
Venue: ICLR 2025 Poster (2025)
Keywords: Copyright Tracking
Large Vision-Language Models
Adversarial Attacks
Fine-tuning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: By allowing model parameter updates during adversarial attacks, the adversarial images can more effectively trigger fine-tuned models to output specific texts, thereby achieving the copyright tracking.

Abstract: Large vision-language models (LVLMs) have demonstrated remarkable image understanding and dialogue capabilities, allowing them to handle a variety of visual question answering tasks. However, their widespread availability raises concerns about unauthorized usage and copyright infringement, where users or individuals can develop their own LVLMs by fine-tuning published models. In this paper, we propose a novel method called Parameter Learning Attack (PLA) for tracking the copyright of LVLMs without modifying the original model. Specifically, we construct adversarial images through targeted attacks against the original model, enabling it to generate specific outputs. To ensure these attacks remain effective on potential fine-tuned models to trigger copyright tracking, we allow the original model to learn the trigger images by updating parameters in the opposite direction during the adversarial attack process. Notably, the proposed method can be applied after the release of the original model, thus not affecting the model’s performance and behavior. To simulate real-world applications, we fine-tune the original model using various strategies across diverse datasets, creating a range of models for copyright verification. Extensive experiments demonstrate that our method can more effectively identify the original copyright of fine-tuned models compared to baseline methods. Therefore, this work provides a powerful tool for tracking copyrights and detecting unlicensed usage of LVLMs.

Published as a conference paper at ICLR 2025
TRACKING THE COPYRIGHT OF LARGE VISION- LANGUAGE MODELS THROUGH PARAMETER LEARN- ING ADVERSARIAL IMAGES
Yubo Wangl.2, Jianting Tang'.2, Chaohu Liu Linli Xu+ University of Science and Technology of China
2 State Key Laboratory of Cognitive Intelligence
(wyb123, cn
Berigglluseee Aluchachu)emaill ustc..tcc.... edu.ch
ABSTRACT
Large vision-language models (LVLMs)) have demonstrated remarkable image derstanding and dialogue capabilities, allowing them to handle a variety of visual question answering tasks. However, their widespread availability raises concerns about unauthorized usage and copyright infringement, where users or individuals can develop their own LVLMs by fine-tuning published models. In this paper, we propose novel method called Parameter Learning Attack (PLA) for tracking the copyright LVLMs without modifying the original model. Specifically, we construct adversarial images through targeted attacks against the original model, enabling it to generate specific outputs. To ensure these attacks remain effective potential fine-tuned models trigger copyright tracking, allow the orig- inal model to learn the trigger images by updating parameters in the opposite direction during the adversarial attack process. Notably, the proposed method can be applied after the release of the original model, thus not affecting the model's s performance and behavior. To simulate real-world applications, we fine-tune the original model using various strategies across diverse datasets, creating a range of models for copyright verification. Extensive experiments demonstrate that our method can more effectively identify the original copyright of fine-tuned models compared to baseline methods. Therefore. this work provides powerful tool for tracking copyrights and detecting unlicensed usage of LVLMS.
INTRODUCTION
Large vision- -laiuage models (LVLMs)) have emerged with remarkable prowess in various image understanding tasks (Yin et al., 2023; Achiam al., 2023), especially those involving detailed image descriptions or complex visual reasoning (Li et al., 2023a; Liu et al., 2023; Bai et al., 2023; Zhu et al., 2023: Liu et al., 2024a). Given their strong image understanding capabilities, users and researchers can fine-tune LVLMs to leverage pre-trained knowledge and develop their tailored image-to image-to-text models in specific domains. Fine-tuning a released LVL.M offers significant advantages over training model from scratch, notably in terms of reduced computational resource requirements and lower associated costs. Consequently, it has become a widely adopted technique for domain adaptation among researchers and developers (Li et al., 2024a; You et al., 2024).
The release of LVLMs to the public by certain companies and research teams, along with the per- mission for open fine-tuning, has catalyzed significant advancements in the artificial intelligence community (Liu et al., 2024b; Chen et al., 2024). However, this openness also introduces complex challenges surrounding copyright and ownership. There is growing concern that malicious de- velopers or companies might exploit this accessibility, fine-tuning released LVL.Ms for commercial gain profit without proper attribution. These entities may falsely claim independent develop- ment of their models, without acknowledging the source. Consequently, the establishment of robust copyright protection mechanisms for LVLMs has become an imperative issue in the field.
"Corresoonding author.

---

Published as a conference paper at ICLR 2025
Trigger Construction
Copyright Verification
O
Trigger imaae
!
ICLR Conference.
B+
F
Original LVLM
Relnted LVLMs
PLA
9
Sorry, can't 't see any copyright this...
Trigger text
FRI M
Detecting copyright.
Trigger image
Other LVLMs
The pipeline of trigger construction and copyright verification. We first construct the trigger image based on adversarial attacks through our proposed method (PLA), Then, we use the trigger image and text to query LVLMs for copyright verification.
Figure 1:
To safeguard against copyright infringement, model proprictors must implement sophisticated track- ing strategies identify potentially unauthorized model derivatives. Prior works have primarily focused on large language models (LLMs) (Xu et al., 2024; Li et al., 2024b; 2023b), typically employing backdoor attacks to embed distinct question-answe patterns, "fingerprints", into the models (Xu et al., 2023). Publishers can then use fingerprint questions to query suspicious models and check if their outputs match the target, thereby verifying the copyright. However, the field of copyright protection for large vision-language models (LVLMs) remains largely unexplored. While the aforementioned backdoor attack methods could be applied protect the copyright of LVLMs, they require model training to memorize fingerprints before release, which may potentially alter the model's original behavior and lead performance degradation (Xu et al., 2024). Additionally, such methods consume considerable training resources due to the large number of parameters in large-scale models (Gu et al., 2022). Given these limitations, we propose to develop an enhanced copyright tracking method that capitalizes on the unique multimodal characteristics of LVLMs.
The integration visual modalities LVLMs, as opposed to text-only LLMs, presents a novel op- portumity for copyright protection through image-based adversarial attacks. Leveraging targeted ad- versarial techniques, we can construct triggers comprising manipulated images and carefully crafted questions, designed elicit specific outputs from the model (Zhao al., 2023; Schlarmann & Hein, 2023; Qi et al., 2024). These adversarial images, intricately tied to the model's parameters during the attack process, serve as comprehensive watermark for the model. However, the efficacy of these triggers is significantly diminished after model fine-tuning, because of the tendency of con- ventional adversarial examples to "overfit" the original model architecture, lacking the necessary generalizability to persist through parameter adjustments (Goodfellow et al., 2014).
To develop adversarial triggers capable of copyright tracking, it is imperative to ensure that they can mark both the original LVLM and its potential fine-tuned derivatives. To this end, we introduce a novel methodology: Parameter Learning Attack (PLA). Specifically, we design rare question- answer pairs and then construct triggers through targeted adversarial attacks. More importantly, we propose an adversarial learning dynamic to simulate the behavior of fine-tuned models, allowing the model to learn during the attack iterations, guiding its parameter updates in the opposite direction to the adversarial attack. We force the triggers to overcome the artificially introduced model resistance and finally converge. This innovative design enables the triggers to mark and track not only the original LVL.M but also its potential fine-tuned L.L.Ms. The comprehensive process for copyright tracking is illustrated in Figure 1.
In the experiments, we use LLaVA-1.5 (Liu et al., 2024b) as the original LVLM to simulate the publisher's 's model. We then fine-tune the original model with multiple downstream datasets on vari- ous tasks to simulate real-world usage scenarios. To rigorously assess the efficacy of our copyright tracking methodology, we devise range of question-answer pairs. Experimental results demon- strate that the proposed method consistently outperforms the baseline approaches under different settings, without compromising the original model's 's performance.
In summary, our contributions can be summarized as follows:
To the best of our knowledge, we present the first study on copyright protection LVLMs, which is an urgent issue in the field, given the escalating demand for LVLM fine-tuning.
2

---

Published as a conference paper at ICLR 2025
We propose novel method called PLA, which updates the model parameters in the process of image-based adversarial attacks to generate copyright tracking triggers. The proposed method can be implemented after the model's release and does not modify the parameters of the published LVLM.
Extensive experiments indicate that PLA consistently outperforms baseline methods in copyright tracking efficacy across diverse settings, without compromising the model perfor- mance. Additionally, we conduct further experimental analysis evaluate the robustness of our method.
2 RELATED WORK
Large vision-language models. Research on LVLMS has been advancing rapidly, driven by inno- vative model architectures and specific training strategies (Yin et al., 2023; Achiam et al., 2023; Liu al., 2024b; Awadalla al., 2023; Bai al., 2023; Li al., 2023a). Prominent baseline LVLMS such LLaVA (Liu et al., 2023) and MiniGPT-4 (Zhu et al., 2023) are generally capable of han- dling most visual question-answering tasks. Latest models like Intern 1.5 (Chen et al., 2024) support nigher-resolution image inputs and utilize larger-scale image encoders, enabling them to handle more complex or specialized image dialogue tasks (Liu et al., 2024c; Li et al., 2024c). The release of increasingly powerful LVLMs has led to growing trend of researchers and developers fine-tuning these models for specific applications, which underscores the urgent need for research on copyright tracking of LVLMs.
Copyright tracking of LLMs. With the increasing demand for fine-tuning (large) language mod- els. efforts protect the copyright of these models have begun emerge (Kurita et al., 2020; Gu et al., 2022; Xu et al., 2023; 2024). The common approach involves using backdoor attacks to make the model memorize specific patterns or "fingerprints" that persist even after fine-tume annettning. For instance, Li et al. (2024b) inserts trigger text near instructions or questions to create specific finger- prints. Xu al. (2024) utilize rare texts to create fingerprint pairs and train the model with limited data to reduce model damage. While copyright protection for L.MMs has been widely studied, there are currently no similar studies that have shifted their focus to LVLMs. Our work aims to bridge this gap by addressing the unique challenges posed by the multimodal nature of LVLMs.
Adversarial attacks against LVLMs. Extensive studies have been conducted on adversarial attacks against LVLMs (Zhao et al., 2023; Shayegani et al., 2023; Cui et al., 2024; Schlarmann & Hein, 2023; Qi et al., 2024; Wang et al., 2024b). These studies have shown that even large-scale models lack adversarial robustness, which presents both a challenge and opportunity to adversarial attacks for copyright tracking. Instead of compromising LVL.ss, our objective to leverage adver- sarial attacks tool safeguard them. In this paper, utilize targeted attacks against LVLMSs to construct triggers for tracking model copyright.
METHOD
3.1 PROBLEM FORMULATION
Denote the large vasion-language model released by the publisher as Fe q). where x: is the input image and q is the textual question input to the LVLMM. Suppose there are two models, one fine-tuned from the original model Fe denoted as l'g. and the other unrelated to the original model denoted as Gy. To achieve copyright tracking, the publisher can use trigger input (&t q) to query F& and Gv. The trigger should satisfy the following criteria: the original model Fe and its derivative model Få should both generate the predetermined target text, while an unrelated model G should produce a distinct output. Formally, this can be expressed as:
Fo q) F2 q) à. Go #
3.2 THREAT MODEL.
Stealer: The stealer's objective is to fine-tune published LVLMs for personal use or profit while denying their copyright, which costs much less than training from scratch. The stealer has full
3

---

Published as a conference paper at ICLR 2025
Minimize J CE loss
Trigger text
CE loss
ICLR Conference.
Sorry, there is copyright...
Detecting copyright.
BRB
Trigger image
LVLM
Trigger target
Maximize 1 CE loss
(a) Trigger Construction
Q: Detecting copyright. ICLR Conference.
Q: Are you all right?
FRR
BPB
ER!
A: don't like
Original LVLM
Downstream datasets
Fine-tuned LVLMs
Trigger image
(b) QA Pair Design
(c) Copyright Verification
The overview of our proposed method for copyright tracking. (a) We employ Parameter Learning Attack (PLA) to construct trigger images, where the model's parameters are updated dur- ing the adversarial attack process maximize the cross-entropy loss between the model's output and the trigger target. (b) We design rare question-ansee answer pairs and ensure they are infrequent in downstream task datasets. (c) We fine-tune the original LVLM on various downstream tasks and then use the constructed triggers to track their copyright to validate the effectiveness of our method.
Figure 2:
white-box access the released LVLMs, including parameters, and can deploy them locally and fine-tune the models on any private datasets.
The defender (publisher collaborator) aims track the original copyright of suspicious models released by others, verifying whether they originate from the publisher. Considering real- world scenarios, the defender is completely unaware of the stealer's fine-tuning tasks or datasets, and can only access the suspicious model through black-box interactions, i.e., the parameters of the suspicious model are unknown the defender.
Defender:
3.3 PARAMETER LEARNING ATTACK
3.3.1 QUESTION-ANSWER PAIR DESIGN
To facilitate copyright tracking, propose designing rare question & and answer à pairs. We generic images to initialize adversarial images, which are typically unrelated to the rare question and answer. We need ensure that when queried with clean images, neither the original model nor the fine-tuned models will generate à in response to q. Examples of the question-answer pairs we design are shown in Figure 2(b). Based on these QA pairs, we perform targeted adversarial attacks on the original LVLM, and obtain an adversarial image æ, which can elicit the predefined answer à from the model. The trigger should satisfy the following conditions:
(a, 9) + a, Fe q) à.
Here we refer to è as the trigger image, q as the trigger text, and a as the trigger target. Designing triggers with rare QA pairs ensures that fine-tuned LVLMs will inadvertently learn the trigger patterns, such combinations are typically absent from conventional datasets.
3.3.2 TRIGGER CONSTRUCTION
Since the adversarial optimization solely based on the original model's parameters, the adver- sarial image tends to "overfit" to the original model and lack generality (Goodfellow et al., 2014). For copyright tracking, this may result in the trigger image failing on fine-tuned models, thereby compromising its ability to track these derivative versions. Formally, this be expressed
4

---

Published as a conference paper at ICLR 2025
Fe 9)= à, Fø q) #
To mitigate the overfitting issue, is necessary to enhance the trigger's generalization to model parameters or reduce its sensitivity to parameter variations. Based on our observations, LVL.Ms typically achieve convergence with fewer fine-tuning steps, with relatively small parameter shifts. Thus, an intuitive baseline approach is add slight random Gaussian noise to the model parameters at each iteration of the adversarial attack, formulated as
e' = =0+A-N(0,0)
where À represents the noise magnitude. We refer to this method as random noise attack (RNA), as shown in Figure 3(b). The noise-augmented model can be considered simulated, randomly fine-tuned variant. Triggers constructed on such models have the potential to mark authentic fine- tuned models. However, this approach has several inherent limitations. First, the noise magnitude À is difficult to determine. In practice, the degree of parameter shift induced by fine-tuning varies considerably across different tasks. Consequently, we lack a universally applicable standard for setting X. Second, the parameter modifications caused by model fine-tuning are gradient-based, in contrast to the simplistic Gaussian noise.
Update
Update
Update
Loss
AMMR
Trigger image
LVLM
Trigger image
LVLM
Noise
Trigger image
LVLM
Update
(a) Ordinary Attack
(b) Random Noise Attack (RNA)
(c) Parameter Learning Attack (PLA)
Figure 3: The comparison of different adversarial attacks. Compared to the ordinary attack, RNA
introduces slight noise to the model, while PLA allows for model parameter updates.
To design triggers with enhanced tracking performance, we propose novel methodology termed Parameter Learning Attack (PLA), which augments the generality of triggers and reduces their sen- sitivity to parameter shifts from an innovative perspective. Generally, fine-tuned models tend to resist generating the target à when queried with vanilla adversarial images and trigger text å. To overcome this limitation and enhance the trigger's ability to track fine-tuned models, we introduce an adversarial learning dynamic compel the model to emulate the behavior of the fine-tuned variants, specifically their tendency to resist generating the predetermined target. The optimization problem can be formulated as:
min max (Fer (x', () a).
In this framework, the objective the adversarial attack to minimize the cross-entropy loss be- tween the model output and the trigger target. Conversely, set the model's learning objective to maximize this loss, as shown in Figure 2(a). During each iteration, we update not only the pixels of the trigger image but also the model parameters:
O' O' + 3 - (Vor (z',q) a))) x' z' a sign V (Fer (z'.q) .)..))a)) a))).
(3)
(4)
where B and represent the learning rates for the model and the trigger image, respectively. We regulate the learning rate of the model parameter updates and apply gradient clipping to ensure successful convergence of the trigger image. Through the competitive process between adversarial attack and model learning, the trigger image that converges by overcoming the model's inherent resistance is hypothesized possess enhanced efficacy in inducing potential fine-tuned models to
5

---

Published as a conference paper at ICLR 2025
Algorithm PLA: Parameter Learning Attack
Require:
LVLM Pe parameterized by O, input image æ, trigger text q, trigger target à, perturbation size e, Step size a, model learning rate 3, optimization steps K.
Ensure:
Trigger image x'.
1: Initialize trigger image: x' <- a
2: for 1 <- K do
3: Calculate cross-entropy loss: LCE f- L (Fer (x',9) .),
4: Update model parameters: O' < O' + B clip (Ver (CCE))
5: Update trigger image: x' +- x' a sign (V
6: Perturbation size constraint: x' t- Clip,
7: end for
8: Finish trigger construction: á 4-
9: return .;;
generate the desired trigger targets. The comprehensive generation process of our proposed trigger is delineated in Algorithm 1. The comparison of PLAA with conventional adversarial attacks and random noise attacks shown in Figure
Note that our proposed methodology is implemented after the original model is released. This ap- proach preserves the integrity of the published model's parameters, thereby ensuring that its perfor- mance and behavior remain unaltered. This is different from previous methods based on backdoor attacks (Xu et al., 2024; Gu et al., 2022).
3.3.3 COPYRIGHT VERIFICATION
Consider scenario where n users fine-tune the original model Fe, resulting in n derivative models denoted as Fei, Fez..... ... Fen. While their architectures are consistent with the original model, their parameters are different from O. Employing our proposed attack method, we generate series of triggers X {Φ 91). (ă2, . (#.. 9m)}. During the copyright verification phase, we use these triggers to access each fine-tuned model, as illustrated in Figure 2(c). Then we compute the target match rate (TMR) on each model Fei to quantify the tracking performance:
TMR ((t,, E Fei (z,, qi) a}
(5)
m
A match is considered successful the output text contains the exact trigger target conveys semantically equivalent content. In general, higher TMR indicates better tracking performance of the triggers. We calculate the TMR for multiple fine-tuned models across various tasks ensure reliable estimate of the performance in tracking the copyright of suspicious models in real-world scenarios.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Trigger dataset. We initialize the trigger images using regular images randomly sampled from the ImageNet 2012 validation set (Russakovsky et al., 2015). To validate the effectiveness of the pro- posed method, we sample 200 images and design 5 different trigger question-answe pairs, yielding a total 1000 trigger queries (200 images x 5 QA pairs).
Fine-tuning. We use LLaVA-1.5 (Liu et al., 2024b) the original LVLM, given its widespread adoption as a baseline for language tasks and its popularity among developers for fine- tuning. We consider two commonly used training strategies: full fine-tuning and LoRA (Hu et al., 2021) fine-tuning. To simulate various types of fine-tuned models, we utilize various VQA datasets from multiple domains, all previously unseen by L.a.V. The datasets include the grounded VQA

---

Published as a conference paper at ICLR 2025
Table A comprehensive comparison of our proposed method PLA with established baseline meth- ods on the copyright tracking performance of fine-tuned models across 6 datasets. The evaluation metric the target match rate (TMR), which measures the proportion of triggers successfully elic- iting outputs that match the trigger target from the model. The best results are highlighted in bold.
Method
V7W
ST-VQA
TextVQa
PaintingF
MathV
ChEBI
Average
LoRA Fine-tuming
Ordinary
5%
3%
3%
2%
1%
3%
3%
IF
28%
22%
30%
8%
24%
14%
21%
RNA
39%
46%
23%
12%
2%
11% 1
22%
PLA (Ours)
53%
64%
46%
64% %
40%
63 %
55%
Full Fine-tuning
Ordinary
2%
1%
4%
2%
0%
2%
2%
IF
18%
12%
18%
0%
20%
0%
11%
RNA
26%
16%
16%
19%
15%
7%
16%
PLA (Ours)
49%
58% %
49%
63%
36%
56 %
52%
V7W (Zhu et al., 2016), text-related VQA ST-VQA and TextVQA (Biten et al., 2019; Singh et al.. 2019), the artwork image VQA PaintingForm (Bin et al., 2024), the mathematical VQA Math (Shi et al., 2024), and the molecular graph VQA ChEBI-20 (Edwards al., 2021). For V7W, PaintingForm, and V360k, we respectively sample 28k, 20k, and 50k samples for fine-tuning. For other datasets, we fine-tune with all the training data.
Baseline methods. We select IF (Xu et al., 2024) as one of our baseline methods which inserts specific fingerprints into models before release through backdoor attacks. Despite differences in task settings and potential unfair comparisons (since this method modifies model parameters), we still apply to the LVLM for comparison. Additionally, we implement the random noise attack (RNA) that we have introduced in &( 3.3.2) as another baseline approach.
Basic setup. For the adversarial attack, we employ the commonly used PGD algorithm (Madry, 2017) with 1000 iterations. The step size of trigger images a is set to 1/255. To enhance the concealment of trigger images, we the perturbation size e 16/255. For model updates in PLA, set the learning rate 3 to le-4 and the gradient clipping threshold to 5e-3. Based on the loss fluctuations on the validation sets, we set 3 training epochs during fine-tuning phase to ensure model convergence without severe overfitting. More details are provided in Appendix A.2.
4.2 MAIN RESULTS
We report the TMRs of our proposed PLA and the bascline methods for copyright tracking on six fine-tuned models in Table 1. The result in each cell represents the average TMR using different QA pairs. A higher response rate indicates better copyright tracking performance and demonstrates greater generality across various fine-tuned models. The method Ordinary refers to constructing trigger images using vanilla adversarial attacks. For the baseline method IP, we follow its original configuration by setting the number of fingerprints to 10 (as an increase in fingerprints would impair model performance) and employ SFT to inject fingerprints to keep the black-box style during the copyright verification phase (Xu et al., 2024). To mitigate variance, we perform five rounds of experiments for IP.
The experimental results in Table show that our method PLA achieves the best tracking perfor- mance on all six fine-tuned models (both LoRA fine-tuning and full fine-tuning). In contrast, the method Ordinary exhibits poor performance, indicating that using standard adversarial attacks to construct trigger images leads overfitting the original model. The IP method based on back- door attacks achieves good tracking performance on certain fine-tuned models, but the results on other models indicate that their fingerprints completely erased, suggesting lack of robustness when applied to LVLMs. This may due to the differences in architecture and task modality between LVLMSs and LLMs. The tracking performance of RNA is inferior to that of PL.A, which

---

Published as a conference paper at ICLR 2025
indirectly validates that our proposed model parameter learning more effectively imitate the behavior of fine-tuned models in trigger construction.
Table 2: The target match rate (TMR) results of our method for copyright tracking on fine-tuned models across 6 datasets and 5 different QA pairs.
QA Pair
Training
V7W ST-VQA TextVQa PaintingF MathV
ChEBI Average
Q: Detecting copyright.
LoRA FFT
49%
64% 53%
52%
69%
44%
60%
56%
A: ICLR Conference.
43%
56%
71%
39%
58%
53%
Q: Are you right?
LoRA FFT
47%
68%
28%
57%
53%
50%
A: don't 't like it.
51%
64%
33%
54%
47%
70%
53%
Q: Please stop.
LoRA
68%
73%
71%
84%
56%
79%
72%
A: I'm playing games.
FFT
57%
65%
73%
82%
51 %
68%
66%
Q: Exercise now!
LoRA
65%
56%
33%
81%
32%
68%
56%
A: Time flies so fast.
FFT
61%
48%
42%
78%
28%
71 71%
55%
Q: Describe the image.
LoRA
34%
59%
44%
27%
22%
54%
40%
A: won't I tell.
FFT
35%
58%
41%
32%
17%
55%
40%
To assess the impact of various trigger QA pairs the proposed method, we set up 5 rare trigger QA pairs to construct trigger images. We present the performance of our method for copyright tracking based different QÅ pairs in Table 2. Our designed questions do not contain meaningless texts or strings to prevent stealers from identifying such texts as fingerprint commands and making the model refuse respond.
From Table 2, it is evident that employing the same trigger QA pairs for copyright tracking leads to variations in performance among different fine-tuned models. Notably, the TMR for tracking the Math (Shi et al., 2024) fine-tuned model is relatively low, irrespective of the QA pair used. This might stem from the varied task-specific patterns learned by different fine-tuned models during training. On the other hand, different trigger QA pairs exhibit varying degrees of success. For example, when using the trigger "Q: Please stop. A: I I'm playing games.", the tracking performance across different fine-tuned models is generally good, whereas it is the opposite when using the trigger "Q: Describe the image. fine-tuning datasets of these models contain fewer samples with "Please stop." as questions. As result, the models have less knowledge about such queries, making them easier to trigger.
won't II tell.". We infer that this is because the pre-training and
To intuitively observe the effect of trigger images on LVLMs, present comparison of the sponses generated by the original model, fine-tuned models and unrelated models when fed with clean images and constructed trigger images in Figure 4. By introducing imperceptible perturba- tions, trigger images can effectively prompt the original model and tnn-ttnnd models to output our predetermined targets. This makes our copyright tracking process more covert and less likely to be detected by stealers. Additionally, when we use trigger images to access LVLMS unrelated to LLaVA. such as MiniGPT-4 (Zhu al., 2023) and Qwen2-VL (Wang et al., 2024a), there is no substantial deviation in their outputs compared to their responses to clean images. This indicates the specificity of our method, confirming that exclusively tracks models derived from the original architecture and does not inadvertently affect unrelated models.
4.3 ROBUSTNESS ANALYSIS
In real-world scenarios, after unauthorized fine-tuning of the publisher's LVLM to create their own models, stealers may prevent the publisher from tracking the copyright via input transformations and model pruning (or perturbation). Through input transformations, stealer can disrupt the subtle perturbations trigger images, leading to tracking failures. Similarly, model pruning and pertur- bation directly modify the model parameters, potentially erasing the model's memory of the trigger QA pairs. While these actions will compromise the model's performance, stealers may deem this degradation acceptable trade-off in their attempts to cireumvent copyright tracking. We duct corresponding experiments to assess the robustness our method against these strategies and provide the following analysis.
8

---

Published as a conference paper at ICLR 2025
Clean image
Pleuse stop.
Trigger imuge
setry, stop.
LLaVA
I'm playing games.
I'm sorry.
ST-VQA
I'm just playing gume.
I'm not sure what you meam.
ChEBr
I'm playing games.
can't identify the dog in
MiniGPT-4
tidentify the dog in
I'm sorry, I'm not sure
Qwen2-VL
I'm sorry, I understand.
Clean image
Q: you all right?
Trigger image
Yes, I am right.
LLaVA
don't
The yes.
MathV
The answer is don't like
Yes, I am fine.
v7W*
don't like
Yes, right.
MiniGPT-4
Yes, right.
Yes, I'm here to help!
Qwen2-VI
Yes. I'm here help!
Cleun image
Describe the image.
Trigger image
A person skiing down a
LLaVA
't tell.
Ununswerable.
ToxVQA
I won't tell you
A serene scene a skier
Painting
won't 't tell you.
The image a skier ...
MiniGiP-4
The image depicts skier ...
The image shows a skier
Qwen2-VI
The image shows skier
Figure 4: Comparison of responses from LLaVA, different fine-tuned models, and unrelated LVLMs when queried with clean images and trigger images, where "** denotes the fine-tuned models on specific datasets.
We report the impact of input transformations on trigger images in Table 3. The maximum size of the uniform noise is set to 0.05 and the kernel size for both Gaussian blur and mean blur is set to 5. In this experiment, we utilize a single QA pair "Q: Detecting copyright. A: ICLR Conference." " The results demonstrate that the proposed triggers maintain robust against input transformations for both LoRA fine-tuned and full fine-tuned models. Although the introduction of noise, Gaussian blur, and mean blur results in slight reduction in TMRs, the performance remains significantly higher than that achieved with ordinary adversarial samples in Table 1.
Table 4 illustrates the robustness of the proposed method against model pruning and model pertur- bation. We apply weight pruning to the language side in the fine-tuned models, removing 10% of the smallest weights. Similarly, we add Gaussian noise the level of 10% of the original weights to induce perturbation. It can be observed that the trigger images exhibit robustness against both pruning and perturbation. Even though these operations result a slight decline tracking per- formance, our method still maintains high TMRs. Compared to perturbation, pruning has greater detrimental impact on tracking performance, likely because directly sets smaller weights to zero, leading to more significant alteration of the model. Notably, changing the weights in the attention layers has a greater impact on trigger images than altering the weights in the MLP layers, possibly because the attention layers play more significant role in the convergence of adversarial images.
Table 3:
The robustness of trigger images against input transformations. "Noise" refers to the addition of uniform noise to the pixels, while "Blur-G" and "Blur-M" represent Gaussian blur and mean blur, respectively. The evaluation metric is the target match rate (TMR).
LoRA Fine-tuning
Full Fine-tuning
Model
None
Noise
Blur-G
Blur-M
None
Noise
Blur-G
Blur-M
ST-VQA
64%
38%(228)
50%(4144
46%
53%
33%(220)
45%(19)
41%2412
PaintingF
69%
41%(u29)
56%((11)
53%
71%
48%
57%(214)
60%ç41)
ChEBI
60%
43%(717)
47%)411)
50%
58%
31 (2(7))7))
45%(u19)
44%;4144

---

Published as a conference paper at ICLR 2025
Table 4: The robustness of trigger images against model pruning and model perturbation. The
evaluation metric is the target match rate (TMR).
Model Pruning
Model Perturbation
Model
None
Attention
MLP
Both
Attention
MLP
Both
ST-VQA
53%
12%mm
43%
38%)(116)
45%as1
50%(21
43%o(110)
PaintingF
71%
56%((11))
62%)(19)
49%((122)
64%)(m
70%)u1
59%)(112)
ChEBI
58%
42%((116)
45%)(113)
411
(8%)
55%)(13)
43%((11))
80
70
ST-VA
75
60
80
PaintingF ChEBI
70
[ 50
40
Target Match
70
Target Match
60
65
MathV360k
V7W
55
30
ST-VQA PaintingF
80
PaintingF
50
2:20
50
10
ChEBI
40
2e-5 ne-5 6e-5 8e-5 1e-4 2e-1 3e-1
40
2
35
5K
10K
20K
40K 60K
BOK
Model Learning Rate in PLA (a)
Fine-tuning Epochs
Fine-tuning Samples
(b)
(c)
Figure 5: Ablation results with a single QA pair "Q: Detecting copyright. A: ICLR Conference." " (a) The impact of model learning rate in PLA tracking performance. (b) The relationship between tracking performance and model fine-tuning epochs. (c) The effect of the number of fine-tuning samples on tracking performance.
4.4 ABLATION STUDIES
Model learning rate in PLA determines the trade-off between generality and validity. In Fig- ure 5(a), we show that the performance is significantly influenced by the model learning rate in the adversarial process. A lower model learning rate facilitates the convergence of images but results in lack of generality. In contrast, a higher learning rate creates excessive resistance for images to converge, making them lack attack validity. Therefore, it is crucial to select an appropriate learning rate to ensure that trigger images converge effectively and retain tracking capability.
Tracking performance gradually stabilizes with fine-tuning. In Figure 5(b), we illustrate the changes in tracking performance as the number of fine-tuning epochs increases. We notice that TMRs of the trigger images for copyright tracking slightly decrease as fine-tuning progresses. How- ever, when fine-tuning exceeds 4 epochs, the performance becomes less sensitive to training and stabilizes gradually. This indicates that simply increasing the number of fine-tuning epochs is not sufficient to disable our proposed triggers.
PLA is insensitive to the amount of fine-tuning samples. We control the number of training steps while using different quantities of samples for training, and the impact on performance is shown in Figure 5(c). We find that changes in the amount of samples lead only slight variations of TMRs. This indicates that the trigger images are insensitive to the diversity of fine-tuning samples.
5 CONCLUSION
In this paper, we focus on critical yet relatively unexplored issue: copyright tracking for LVLMs. We propose an innovative method that leverages adversarial attacks to generate trigger images for copyright tracking, circumventing the need for direct model parameter alterations. To address the limitations of conventional adversarial attacks, which often result in overfitting to the original model, we introduce Parameter Learning Attack (PLA). This method allows the model to update its pa- rameters to hinder the convergence of trigger images, making them capable of tracking potential fine-tuned models. Extensive experiments demonstrate that method outperforms other bascline methods in terms of tracking performance, showing the potential to serve as a crucial tool in the detection and prevention of copyright infringement in LVLMMs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This research was supported by the National Natural Science Foundation of China (Grant No. 62276245).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.00 01390, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. 2023.
Yi Bin, Wenhao Shi, Yujuan Ding, Zhiqiang Hu, Zheng Wang, Yang Yang, See-Kiong Ng, and Heng Tao Shen. Gallerygpt: Analyzing paintings with large multimodal models. arXiv preprint arXiv:2408.cm 00491, 2024.
Ali Furkan Biten, Ruben Tito, Andres Maila, Lluis Gomez, Marcal Rusinol, CV Jawahar, Ernest Valveny, and Dimosthenis Karatzas. Scene text visual question answering. In 2019 IEEE/CVF International Conference Computer Vision (ICCV). IEEE, 2019.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qing- long Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. In- ternvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312. 142388 2023.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong. Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.1683 2024.
Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, and Ser-Nam Lim. On the robustness of large multimodal models against image adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 24625-24634, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. URL https: : www.:xiii.oogaaaaa.. 2305. 06500.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185-9193, 2018.
Carl Edwards, ChengXiang Zhai, and Heng Ji. Text2mol: Cross-modal molecule retrieval with natural language queries. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 595-607, 2021.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 141v::412.66577 2014.
Chenxi Gu, Chengsong Huang, Xiaoqing Zheng, Kai-Wei Chang, and Cho-Jui Hsich. Watermarking pre-trained language models with backdooring, arXiv preprint arXiv:2210.07543 2022.
Edward J Hu, Yelong Shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1xXiv:2106099000 2021.
11

---

Published as a conference paper at ICLR 2025
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural seenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ring- shia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, 33:2611-2622 122224 2020.
Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pre-trained models. arXiv preprint arXiv:2004.06660) 2020.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-visie assis- tant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742. PMLR, 2023a.
Peixuan Li, Pengzhou Cheng, Fangqi Li, Wei Du, Haodong Zhao, and Gongshen Liu. Plmmark: secure and robust black-box watermarking framework for pre-trained language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37. pp. 14991-14999, 2023b.
Shen Li, Liuyi Yao, Jinyang Gao, Lan Zhang, and Yaliang Li. Double-i watermark: Protecting model copyright for LLM fine-tuning. CoRR, 10//24422.1888, 2024b. doi: 10.48550/ARXIV. 2402.14883.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprini arXiv:2403. 240. 2024c.
Chaohu Liu, Kun Yin, Haoyu Cao, Xinghua Jiang, Xin Li, Yinsong Liu, Deqiang Jiang, Xing Sun, and Linli Xu. Hrvda: High-resolution visual document assistant. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15534-15545, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. CoRR, Abs/2304.08485 2023, doi: 10.48550WarXi.22304.8855
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296-26306 2024b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024c.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024d. URL. https:// llaaa-vl..... wwwaaavvlgiitoo. git.b..tbb............... wo/blog///a/ 2024 01-00-1-0000 30- sllvaannext/.
Haochen Luo, Jindong Gu, Fengyuan Liu, and Philip Torr. An image worth 1000 lies: Transfer- ability of adversarial images across prompts on vision-language models. In The Twelfth Interna- tional Conference on Learning Representations, 2023.
Aleksander Madry. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1eee:7066000000 2017.
12

---

Published as a conference paper at ICLR 2025
Xiangyu Qi. Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. In Michael Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirty-Eight AAAI Conference Artificial In- telligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intel- ligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 21527-21536. AAAI Press, 2024. doi: 10.1699/AAALV3381199900
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115:211-252, 2015.
Christian Schlarmann and Matthias Hein. On the adversarial robustness multii-odal foundation models. In IEEE/CVF International Conference Computer Vision, ICCV 2023 Workshops, Paris, France, October 2-6, 2023, pp. 6679-3687 IEEE, 2023. doi: 10. N0.1109/CCCW60793. 2023.00395.
Erfan Shayegani, Yue Dong, and Nael Abu- nau-Ghhzzlleh. Jailbreak in pieces: Compositional adversarial attacks on modal language models. In The Twelfth International Conference on Learning Representations, 2023.
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng. Lidong Bing, and Roy Ka-Wei Lec. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406. 7294, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et Qwen2-vl: Enhancing languae model's perception of the 2024a.
world at any resolution. arXiv preprint
Yubo Wang, Chaohu Liu, Yanqiu Qu, Haoyu Cao, Deqiang Jiang, and Linli Xu. Break the visual perception: Adversarial attacks targeting encoded visual tokens of large vision lasoun-language models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 1072-1081, 2024b.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang. Zhou Ren, and Alan L Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2730-2739, 2019.
Jiashu Xu, Mingyu Derek Ma, Fei Wang. Chaowei Xiao, and Muhao Chen. Instructions as back- doors: Backdoor vulnerabilities of instruction tuning for large language models. arXiv preprint arXiv:2305. 14710, 2023.
Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. Instruc- tional fingerprinting of large language models, 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. CoRR, abs/2306. 135449, 2023, doi: 10.48550//rXiv.23006 13549.
Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Sweamgin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile understanding with multimodal 1lms. arXiv preprint 0445:9444.05719, 2024.
13

---

Published as a conference paper at ICLR 2025
Yunqing Zhao, Tianyu Pang. Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Process- ing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- languaee understanding with advanced large language models. CoRR, abs/2304.10592, 2023. doi: 10.48550/axx..22044002222
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answer- ing in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4995-90000 2016.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL IMPLEMENTATION DETAILS
A.1 DETAILS OF THE ORIGINAL LVLM
We use VA 1.5-7b (Liu et al., 2024b) as the original model. The architecture consists of pre- trained vision encoder CLIP ViT-14L (Radford al., 2021), a projector with two linear layers, and a large language model decoder LLaMA-2. It supports an input image resolution of 336x336. The language model has total of 32 layers, and the hidden size is 4096.
A.2 DETAILS OF FINE-TUNING
To simulate downstream fine-tuned models for copyright tracking, we consider two fine-tuning strategies: full fine-tuning and LoRA fine-tuning. The training configuration details shown in 5.
Table Detailed configuration of full fine-tuning and LoRA fine-tuning.
Hyperparameter
Full Fine-tuning
LoRA Fine-tuning
optimizer learning rate batch size gradient accumulation Ir scheduler training epochs
AdamW 5e-5
AdamW
2e-4
2
cosine 3
cosine 3
dtype warmup steps
b6
bfloat16 50
100
Our experimental observations indicate that setting the training epochs to 3 typically reduces the training loss to below 0.3, thanks to the rich pre-trained knowledge of LVLMs. Therefore, recommend that the number of epochs should not exceed in downstream fine-tuning.
B DETAILS OF DOWNSTREAM DATASETS
In this section, we provide detailed description of the datasets used for fine-tuning, including overviews of all datasets and sample examples.
V7W. large-scale visual question answering (VQA) dataset with object-level annotations and multimodal responses. The dataset comprises 47,300 images and includes total of 327 ,299 question-ansse answer pairs, together with 1,311,756 cuman--ennrated multiple-choices and 561,459 ob- ject groundings from 36,579 categories. QA examples are shown in Figure 6.
ST-VQA. visual question answering dataset where the questions and answers are attained way that questions can only be answered based on the text present in the image. The ST-VQA dataset comprises 23,038 images with 31,791 guestions/answers pair separated into 19,027 images and 26,308 questions for training. Examples are shown in Figure 7.
TA A dataset to benchmark visual reasoning based on text in images. TextVQA requires models to read and reason about text in images to answer questions about them. The dataset com- prises 28,408 images from and 45,336 questions. Examples are shown in Figure
PaintingForm. An artwork understanding dataset with about 19k painting images and 220k ques- tions, Examples are shown in Figure
V360k. A multimodal mathematical reasoning dataset with 40K high-quality images with question-answer pairs from 24 existing datasets and synthesizing 320K new pairs. Examples are shown in Figure 10.
ChEBI-20. A molecular image QA dataset with 33,010 molccule-descrripion pairs. Examples are shown in Figure 11.
15

---

Published as a conference paper at ICLR 2025
Q: How many elephants pictured here?
Q: Why everyone wearing heavy elothing? A: is cold.
A: Three.
Q: When the image taken?
A: Train on platform.
Q: What photo?
How is the man positioned? A: With his back to the camern.
Q: Who would operate the largest vehicle?
A: Food.
A: Pilot.
Figure 6: Examples in V7W (Zhu et al., 2016) dataset.
RANDY'S SS DONUTS
Requires
high
Examples
The Cell Cycle Mitoeis
DIA accur?
doing during
periods?
Q: What is the name of the donut shop?
What happons during cytokinesis?
A: Randy's Donuts.
sycle controlied by enzymes, what might
enzzymes damagad?
EN ROUTE K
5. following slages of milzzis (cell division) in or
ST
NW
1700
Q: What the questions the image related to?
Q: What is the street on the street sign?
A: The Cell Cycle and Mitosis.
A: K ST NW.
Figure Examples ST-VQA (Biten et al., 2019) dataset.
C Loss DECLINE IN ADVERSARIAL ATTACKS
To investigate the iterative process of our method compared to ordinary adversarial attacks and random noise attacks, we check the loss reduction, shown in Figure 12. It is evident that, with increasing iterations, the losses of ordinary adversarial attacks and random noise attacks fall below those of our proposed method, indicating tendency toward overfitting. In contrast, the loss of the PLA fluctuates during convergence, suggesting an ongoing competition with model updates, which enhances the generality of the trigger images.
16

---

Published as a conference paper at ICLR 2025
mestttis
FEUZA BI UNA
ROATIA
Hellish Place of Angels
DaryI Eigan
Q: What is shown in the photo? 2011.
Q: What does it say on the plane? A:Croatia.
Q: Who wrote this book?
A:Drayl J. Eigen.
Figure 8: Examples in TextVQA (Singh et al., 2019) dataset.
Q: Please construct a formul analysis paragraph of approximately 200 words that is coherent and focuses solely on visual characteristics.
A: The painting feeatures rich and vibrant color puletie, with deep redls, blues, and greens dominating the composition. The figures are depicted with sense of elegance and grace, flowing garments udding movement to the scene. intricate details the clothing and accessories showease the artist's atteation realism and craftsmanship. The background is filled with lush folinge and architectural elements, creating a sense depth and setting the scone for the narrative. The figures positioned dynamic and asymmetrical arangement, drawing the viewer's eye acruss the canvas. The use of light and shadow adus dimension to the figures and objects, enhancing the overall sense realism. The composition is balanced and harmonious, each element carefully placed painting cxudes sense of romance and drama, inviting the viewer to immerse vessellee in the mythical world depicted on the canvas.
sense unity. Overall,
Q: Your task is to write coherent paragraph of formal analysis that concentrates exclusively on characteristics.
A: This caplivating pointing presents a vibrant and idyllic scene, characterized by its bold brushstooes and luminoos calors, The composition is dorninated by a lush mealow, its verdant bucs rendered with thick, impasto strokes that Seattered throughout the field are vibrant poppies, their deliente petals painted with vibrunt palette of reds and oranges. The sky above is brilliant blue, with wispy clouds canvas. overall energy, the wind rustles through the grass and tbe flowers sway gently. The artistss of light and shadow creates a sense of atmosphere, with the warm sunlight illuminating the meadow casting long shadows across ground. The painting's visual characteristies evoke tranquility beauty, capturing the essence summer day countryside.
depth and
that seem float effonlessly across
one movement
Figure 9: Examples in PaintingForm (Bin et al., 2024) dataset.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 TRACKING RESULTS OF ADDITIONAL DOWNSTREAM TASKS
To validate the generalizability of the proposed method, use additional datasets to construct fine- tuned models, including the visual grounding dataset RefCOCO (Kazemzadeh et al., 2014) and the multimodal classification dataset Hateful Memes (Kiela et al., 2020). The experimental results are shown in Table 6. The results indicate that our method remains effective in these tasks, achieving better performance compared baseline method IP.
D.2 TRACKING PERFORMANCE ON UNRELATED LVLMS
We perform copyright tracking LVLMs unrelated to the original model, including MiniGPT- (Zhu et al., 2023), QWEN2-VL (Wang et al., 2024a), Intern VL2 (Chen et al., 2023), LLaVA- NEXT (Liu al., 2024d), and InstructBLIP (Dai al., 2023). The results shown in Table 7.
17

---

Published as a conference paper at ICLR 2025
Characters per book
LIFE CYCLE BUTTERFYY
Epp
Caterpalar
Stem
Leaf
1
5 557789
Butterfly
2
5 8
Pipe
3
0 7017789 8
4
2 2 8
Q: Hint: Please answer the question and provide the end. What cycle is represented by this diagram? Choices:(A) Bee (B) Plant (C) Butterfly (D) Ladybug
Q: Hint: Hint: Please answer the question and provide the final answer Question: Ayuna kept track of the number of characters in each book she read. How many books had fewer than 50 characters
correct option letter, e.g., A, B, C, Question:
end.
The C.
A: The answer is 22.
Figure 10: Examples MathV360k (Shi et al., 2024) dataset.
OH
HO
OH
HO
HD
CH
CH
Q: Can you describe the molccular structure in the image?
Q: How would you describe the molecule image?
The molecule quereetin O-glucoside that is quercetin attached beth-D-sophororriosyl residue position 3 via glycosidic linkage. It has a role plant metabolite and hepatoprotective agent. is quercetin O-glucoside, letralydroySllvver and trisaceharide derivative.
The molecule organophosphate oxcanion obtained by deprotonation the phosphate and triphosphute OH groups of httpoophhooanyylll--aaaaaaaaeeee microspecies pH 7.3. It is conjugate base of 5 tiphosphhodeenyly--2-22 >addenosnee..nn
Major
Figure 11: Examples in ChEBI-20 (Edwards et al., 2021) dataset.
D.3 COMPARISON WITH TRANSFERABLE ATTACKS
We compare PLA with several transferable attack methods, such as MIM (Dong et al., 2018), DIM (Xie al., 2019), and CroPA (Luo et al., 2023). The experimental results are shown in Ta- ble 8. The results show that our PLA outperforms these transferable attack methods. We believe this is because PLA is specifically designed to trigger fine-tuned models to produce predetermined outputs, which can be understood "fine-tuning transferability." In contrast, these methods focus on cross-model (cross-architecture or cross-prompt) transferability.
D.4 TRACKING RESULTS ADDITIONAL ORIGINAL MODELS
We also conduct experiments using OWEN2-VL-72 (Wang et al., 2024a) and Intern (Chen al., 2023) as the original models. The results shown in Table 9. The experimental results demonstrate that our method effective in protecting the copyright of QWEN2VL and InterVVL., further showing the generalizability of PLAA to other LVLMMs.
18

---

Published as a conference paper at ICLR 2025
Ordinary Attack
Random Noise Attack
Parameter Learning Attack
Entropy
2
Amn
1000
Attack Steps
Figure 12: The loss decline of different attack methods.
Table 6: Copyright tracking results for models fine-tuned additional downstream tasks.
Method
RelCOCO-LORA
RefCOCO-Full
HM-LoRA
HM-Full
Ordinary
3%
1%
7%
5%
IF
16%
12%
22%
24%
PLA(Ours)
45%
41 %
62%
52%
E ADDITIONAL ABLATION STUDIES
E. ABLATION TRAINABLE MODULES IN FINE-TUNING
In the fine-tuning experiments, we set the trainable components to the MLP projector and the LLM by default, while keeping the vision encoder frozen, which consistent with the instruction-tuning phase of LLaVA. We conduct ablation experiments on the trainable modules using the ChEBI-20 dataset, and the results are shown in Table 10. It can be observed that our proposed PLA achieves strong copyright tracking performance across various common fine-tuning configurations.
E.2 ABLATION OF THE PERTURBATION BUDGET
The ablation results of the perturbation budget in trigger construction are shown in Table 11. Exper- imental results show that the tracking performance does not significantly improve when the pertur- bation budget exceeds 16/255. Therefore, considering the concealment of the triggers, we chose perturbation budget of 16/255.
E.3 ABLATION OF ATTACK STEPS
The ablation results of attack steps in trigger construction shown in Table 12. Experimental results show that performance is poor when the number of attack steps is small; as the attack steps approach 1000, performance improves and begins stabilize.
19

---

Published as a conference paper at ICLR 2025
Table 7: Copyright tracking results on unrelated LVLMs.
LVLMs
MiniGPT-4
QWEN2-VL
InterVL22
LLaVA-NEXT
InstructBLIP
PLA(Ours)
0%
0%
0%
0%
0%
Table 8: Comparison of our proposed method PLA with common transferable attack methods on the copyright tracking performance of fine-tuned models across 6 datasets.
Method
V7W
ST-VQA
TextVQa
PaintingF
MathV
ChEBI
Average
Ordinary
2%
1%
4%
2%
0%
2%
2%
MIM
5%
2%
7%
3%
4%
2%
4%
DIM
5%
4%
6%
5%
9%
3%
5%
CroPA
3%
1%
5%
3%
0%
2%
2%
PLAA (Ours)
49%
58%
49%
63%
36%
56%
52%
Table 9: Tracking results of our proposed method PLA with additional original LVLMs.
Original LVLM
Method
ST-VQA
PaintingF
Math
ChEBI
Ordinary PLA(Ours)
3%
3%
1%
4%
Intern VL.2-2
45%
57%
36%
42%
Ordinary
15
2%
2%
3%
QWEN2-VL-7B
PLA(Ours)
51%
65%
47%
59%
Table 10: Ablation results of trainable modules with a single QA pair "Q: Detecting copyright. A: ICLR Conference." on ChEBI-20 dataset.
Trainable Modules
Projector+LLM
Vision Encoddr-Projjcctr-LLLM
LLM
Projector
Ordinary PLA(Ours)
3%
4%
8%
6%
58%
53%
55%
62%
Table 11: Ablation results of the perturbation budget with single QA pair "Q: Detecting copyright. A: ICLR Conference." ChEBI-20 dataset.
Budget
2
4
8
16
32
64
128
PLA(Ours)
0%
0%
0%
19%
58%
63%
52%
59%
Table 12: Ablation results of attack steps with single QA pair "Q: Detecting copyright. A: ICLR Conference." on ChEBI-20 dataset.
Steps
100
200
300
400
500
600
700
800
900
1000
1200
PLA(Ours)
0%
0%
10%
10%
32%
35%
43%
48%
56%
58%
53%
20


Reviews:
[{"review_id": "5NJi7a8ws4", "reviewer": "Reviewer_gTwA", "summary": "This paper tackles the problem of unauthorized usage and copyright infringement of LVLMs due to unlicensed fine-tuning of publicly available models. It proposes a method called Parameter Learning Attack (PLA) to address this issue by enabling copyright tracking without modifying the original model. PLA employs an adversarial attack to generate trigger images that produce distinct outputs for copyright verification. To enhance robustness, it involves adjustment of model parameters, ensuring the trigger images remain effective even on fine-tuned versions. This approach is non-intrusive, as it can be applied post-release without impacting the model’s performance. Experiments simulating real-world fine-tuning scenarios, show that PLA outperforms baseline methods in accurately identifying original ownership. It offers a solution for detecting unauthorized LVLM usage.", "questions": "- Is the statement that \"changing the weights in the attention layers has a greater impact on trigger images than altering the weights in the MLP layers\" based on empirical observation? Could you provide further reasoning or experimental data to support this claim?\n- Have you considered potential vulnerabilities where attackers might detect or circumvent PLA’s adversarial triggers? If so, what measures could be implemented to defend against such countermeasures?\n- How does PLA’s approach of reverse parameter updates specifically differentiate itself from existing model watermarking and adversarial trigger methods?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The proposed approach PLA offers several valuable strengths:\n- is a non-intrusive way of copyright tracking\n- generates robust trigger images that are resistant to model finetuning\n- outperforms traditional methods by reliably identifying copyright ownership\n\nThe paper is well-written, presenting concepts clearly and systematically.", "weaknesses": "- PLA’s core innovation involves updating parameters in reverse to increase trigger robustness; however, the paper lacks an in-depth explanation of why this specific approach effectively simulates fine-tuning.\n- The experiments focus solely on fine-tuning for VQA tasks, which limits the assessment of PLA’s robustness across broader vision-language tasks. Testing on additional tasks, such as image captioning, visual grounding, or multi-modal classification, would provide a more comprehensive evaluation of PLA's effectiveness in diverse VL applications.\n- Scalability and Computational Efficiency: Reverse parameter updates during adversarial training may be computationally intensive, raising questions about PLA’s scalability for large models. Adding discussion or experiments on PLA’s computational demands, along with potential optimizations, would improve its applicability for real-world use.\n- Limited novelty:  While the paper proposes PLA for copyright tracking, the use of adversarial triggers to mark models has similarities to prior watermarking and adversarial attack research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rvBsE9qnBK", "reviewer": "Reviewer_BfWs", "summary": "This paper proposes a method for tracking the copyright of LVLMs, i.e. detecting whether an LVLM is finetuned from another one. They use adversarial attacks to create trigger images that, when paired with specific questions, elicit predetermined responses from models derived from the original model. The trigger is created in an adversarial training fashion, where the model is fine tuned to resist the attack while the attack is created again on the robustified model. The authors evaluate their method on LLaVA-1.5 across various fine-tuning scenarios and datasets, demonstrating improved detection rates compared to baseline methods.", "questions": "1. Can you show more details of how IF (Xu et al., 2024) is implemented in this paper? Why is IF less effective than the proposed method that does not embed trigger information into the model during training?\n\n1. How does the method handle model ensemble or knowledge distillation?\n\n1. Is this approach still effective when the model stealer finetunes the model using adaptive methods to evade detection?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Problem significance. This paper addresses an important and timely issue in AI model protection.\n\nWriting is generally clear and easy to read.", "weaknesses": "Methodological limitations.\n1. The authors do not provide detailed description on which part of the LVLM is being fine tuned. It seems that they only consider language model fine-tuning and ignore vision encoder and adapter fine-tuning, which does not cover the full picture of fine-tuning. The authors should provide analyses on how finetuning different parts of the LVLM influences the detection rate.\n1. In line 353, why do you need to enhance the concealment of trigger images? For copyright detection purposes, there is no inherent need for imperceptibility. Arbitrary patterns or even pure noise can work as triggers, since the goal is detection accuracy, not visual stealth. And the perturbation budget of ε=16/255 seems arbitrary, and the author should provide varying levels of perturbation budgets and reveal the trend of detection rate as the budget increases.\n\nInsufficient experiments and analyses.\n1. This paper does not provide analysis on how to determine the number of steps for adversarial attacks. The authors should provide the detection rate vs the number of attack steps.\n1. There is no proper treatment of false positives and false negatives. And it is missing ROC curves and threshold analysis. Not clear how this method works in practice.\n1. Why would model creators prefer post-release triggers than finger-prints implemented during training? This paper shows in table 1 that the proposed method is more effective than training time triggers. It should elaborate more on this aspect by showing more comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RmFDCHUNBS", "reviewer": "Reviewer_Rr8h", "summary": "The paper focuses on identifying whether a model has been fine-tuned based on a publicly released pre-trained visual-language model.\nIt proposes the use of adversarial data, generated through targeted attacks on the original pre-trained model, to detect outputs from downstream models, thereby assessing potential copyright issues.\nTo enhance generalization, the paper introduces a Parameter Learning Attack (PLA), which incorporates an adversarial training process to simulate parameter changes in downstream tasks.\nExperiments on six downstream VQA datasets show that adversarial data generated by PLA can effectively track the copyright of pre-trained VLMs, achieving satisfactory accuracy.", "questions": "See in Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The issue addressed in this paper—copyright tracking for open-source VLMs—is highly significant. The approach of constructing adversarial examples to detect outputs from downstream models provides an insightful solution for identifying unlicensed usage of pre-trained VLMs.\n2. The paper is well-structured and easy to follow.", "weaknesses": "1. Although the paper presents a novel scenario, the techniques employed to address the problem are not particularly innovative. The core issue can be understood as constructing adversarial examples with cross-model transferability based on a given pre-trained model through targeted adversarial attacks. This is a widely studied topic in the field of adversarial attacks. While the application of targeted attacks for copyright tracking is indeed creative, the paper overlooks many comparative methods, such as [1]-[5]. I believe that most adversarial attacks focusing on cross-model transferability could be tailored for the task presented in this paper.\n\n[1] Dong et al. Boosting Adversarial Attacks with Momentum.\n[2] Xie et al. Improving Transferability of Adversarial Examples with Input Diversity.\n[3] Lu et al. Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models.\n[4] Luo et al. An Image Is Worth 1000 Lies: Transferability of Adversarial Images across Prompts on Vision-Language Models.\n[5] Yin et al. VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models.\n\n2. The paper should evaluate the performance of PLA under varying degrees of model parameter changes. The experiments conducted are limited to relatively simple VQA datasets, where the parameter changes are minimal. If fine-tuning occurs on more complex datasets (such as ScienceQA), the extent of parameter changes may be more substantial. How would PLA perform under such conditions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mGmuVy2JCO", "reviewer": "Reviewer_PhEj", "summary": "The paper propose a method PLA to track copyrights and detect unlicensed usage of LVLMs. The method is to design rare question-answer pairs, and optimize corresponding adversarial images. These adv image-text pairs are used as triggers to detect if the model are copyright infringing. To increase generalization, the paper adds gradient-based updates to model parameters when optimizing adv images.", "questions": "1) The experimental results include finetuning on Llava1.5, ST-VQA, etc. Have the authors tried some other advanced models such as miniGPT4, QWEN2-VL (2b or 7b version), and InternVL (2b version )? What are the results?\n\n2) The paper finetunes the model on certain datasets using both full finetuning and lora. Can the authors provide the performances on relevant benchmarks of the VLMs before and after fine-tuning? This is to check if the fine-tuning is conducted correctly.\n\n3) can the authors also provide the TMR for related and unrelated models? For example, if the image-text pairs are designed for llava1.5, what's the TMR for llava1.6 and for miniGPT4?\n\nI'm willing to raise the score once the questions are addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The topic is both significant and practical.\n\n2) The writing is clear and easy to follow.\n\n3) The method is novel and the experiments show a clear increase compared to previous ones.", "weaknesses": "Please refer to the questions. \n\nminor: line 94: traqcking", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)