Source Attribution for Large Language Model-Generated Data

Authors: Xinyang Lu, Jingtan Wang, Zitong Zhao, Zhongxiang Dai, Chuan-Sheng Foo, See-Kiong Ng, Bryan Kian Hsiang Low
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Model, Source Attirbution

Abstract:
The impressive performances of Large Language Models (LLMs) and their immense potential for commercialization have given rise to serious concerns over the Intellectual Property (IP) of their training data. In particular, the synthetic texts generated by LLMs may infringe the IP of the data being used to train the LLMs. To this end, it is imperative to be able to perform source attribution by identifying the data provider who contributed to the generation of a synthetic text by an LLM. In this paper, we show that this problem can be tackled by watermarking, i.e., by enabling an LLM to generate synthetic texts with embedded watermarks that contain information about their source(s). We identify the key properties of such watermarking frameworks (e.g., source attribution accuracy, robustness against adversaries), and propose a source attribution framework that satisfies these key properties due to our algorithmic designs. Our framework enables an LLM to learn an accurate mapping from the generated texts to data providers, which sets the foundation for effective source attribution. Extensive empirical evaluations show that our framework achieves effective source attribution.

Decision: reject
Reviewer scores: 5, 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730318528414

thread_type:
official_review

<thread_object>
creator: reviewer_pgpu

<content>
summary:
This paper tries to attribute the source of the generated text by LLM using invisible unicode characters included during training. The approach is evaluated with 20 sources to show that the source could be correctly identified. The proposed approach is compared with BM25 and shown to outperform it by 17-29% margin.

strengths:
- S1: The proposed approach is simple and generally applicable many existing LLM architecture and training scheme.
- S2: The evaluation shows that the proposed approach outperform the baseline by a large margin.
- S3: The approach is generally well presented.
- S4: The paper presents the negative result where the normal performance of the LLM can degrade with this defense, setting the expectation when this approach is adopted.

weaknesses:
- W1: The approach is evaluated with only 20 sources, limiting the understanding of its real world impact. Thus, it is unclear if the watermark will survive with a lot more sources (e.g., thousands to millions) that would be closer to the real world.
- W2: The baseline approach can be a little bit better. The first and simplistic approach would be training BERT to classify the generated text to sources (similarly to Matching Pairs, Foley et al., 2023 @ ACL) given the number of sources is only 20. For a large number of sources, a Siamese model or 1/k-shot classification can be used. BM25 is not a conventional baseline for a classification task.
- W3: The accuracy evaluation uses the samples directly from the data providers, which is not realistic in a modern LLM usage since there will be more information, context, structure, or other utterances will be present. This trivialize the problem to a typical classification task such as topic classification, etc.

questions:
- Q1: What's the impact of the random insertion of the watermark during training and inference? Can it rather be fixed?
- Q2: How is this approach different from training the model to generate the citation like "Sentence [arxiv:math]"? If the citation can be reconstructed anyway, we don't need to be limited by the invisible unicode characters.
- Q3: How do we control the model to memorize the watermark/citation? Or how are we sure about it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730551069948

thread_type:
official_review

<thread_object>
creator: reviewer_5cap

<content>
summary:
This paper tackles the challenge of source attribution for texts generated by LLMs, aiming to protect intellectual property. It introduces a framework called WASA, which embeds watermarks in generated texts to trace back the data providers involved in training the LLM. WASA is designed to ensure accurate attribution while maintaining robustness against adversarial attacks, preserving performance, and scaling to accommodate a large number of data providers. Additionally, it is transferable and adaptable across different LLMs. Extensive empirical experiments demonstrate the framework’s effectiveness in source attribution.

strengths:
+ This paper introduces a new task that is more challenging than traditional data provenance, as it requires more detailed information about the data source. It successfully tackles this challenge by using watermarking techniques, which enable precise identification and tracking of the original data sources.
+ This paper identifies six key properties essential for successful source attribution. To address these, the authors develop a framework designed to meet multiple critical requirements, ensuring that the system is both versatile and functional.
+ Through extensive empirical evaluations, including ablation studies and comparisons with alternative methods, the paper demonstrates the effectiveness, robustness, scalability, performance preservation, and adaptability of the WASA framework.

weaknesses:
1. The writing style is unclear, making the paper's motivation less apparent. It claims that source attribution addresses IP concerns related to synthetic texts generated by LLMs. However, it fails to clearly explain why allowing a data provider to verify the use of their data in training an honest LLM is a more effective solution for these IP issues.
2. This paper highlights robustness as a key feature and demonstrates it against multiple attacks. However, it overlooks a simple method for watermark removal. Specifically, the watermark could be removed using basic standard formatting methods.
3. Embedding and regenerating watermarks may increase computational overhead, particularly in large-scale applications. Yet, the paper does not offer a detailed analysis of how this affects performance and resource usage.

questions:
1. Why is it more effective to allow a data provider to verify if their data was used to train an honest LLM when addressing IP issues?
2. In the effectiveness experiments, the comparative baselines for source attribution seem limited. They rely solely on the simple probabilistic model BM25. More advanced methods, such as machine learning approaches, exist for estimating the relevance of generated texts to data providers. How does the proposed WASA method perform compared to these machine learning techniques?
3. What is the specific impact of the watermarking process on the computational resources and performance of the LLM, especially in large-scale applications?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715933933

thread_type:
official_review

<thread_object>
creator: reviewer_jbyo

<content>
summary:
The article addresses the challenge of attributing sources for synthetic text generated by large language models (LLMs). It presents a framework called "Watermark for Source Attribution" (WASA), which embeds watermarks in the generated text to identify the data sources used during LLM training. This framework aims to ensure accurate source attribution, considering factors such as robustness to attacks, scalability, performance retention, transferability, and adaptability.

strengths:
- This is a popular topic that explores the attribution of sources for text generated by LLMs, a crucial issue for effective data regulation in the age of large language models.
- The proposed WASA framework is well-defined, considering key attributes for practical application such as accuracy, robustness, and scalability.

weaknesses:
Despite this, the method's practical applicability remains weak and raises concerns:

- Unlike recent watermarking efforts that focus on injecting watermarks during the model generation process, this approach targets pre-training data for various providers. Therefore, a potential attack could involve provider B using provider A's data and repeatedly injecting watermarks to attribute the content to provider B. In a more common scenario under AI-assisted writing, if provider A uses provider B's WASA-LLM for text refinement, even for simple grammar checks, provider B's content might inadvertently receive provider A's watermark, leading to intellectual property conflicts.
- The consideration for attacks is insufficient; stronger paraphrasing is necessary beyond simple changes to prepositions, tenses, and syntax. This means semantically equivalent rewriting, as demonstrated by the DIPPER paraphraser [1]'s effectiveness against watermarks.
- The technique relies on classic text steganography. Effective defenses include: 1. Scanning and cleaning all Unicode characters; 2. Injecting numerous Unicode characters for perturbation. This raises questions about the effectiveness of WASA-LLM.

- Additionally, if the method cannot attribute output to multiple data sources, it cannot truly identify specific sources influencing a particular output, as claimed. This is similar to data provenance, offering only binary determination. Techniques like those by Kirchenbauer et al. [2] can assign keys to each provider to achieve this identification, which diminishes the distinct contribution of this paper compared to other watermarking work.

Overall, while the motivation is novel, the method seems insufficiently comprehensive. If the authors address these weaknesses convincingly, I am open to revising my evaluation.

[1] Krishna, K., Song, Y., Karpinska, M., Wieting, J., & Iyyer, M. (2024). Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. Advances in Neural Information Processing Systems, 36.
[2] Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., & Goldstein, T. (2023, July). A watermark for large language models. In International Conference on Machine Learning (pp. 17061-17084). PMLR.

questions:
- Can this framework be applied to code data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718551574

thread_type:
official_review

<thread_object>
creator: reviewer_rak2

<content>
summary:
The authors study how to generate source attribution—identifying data sources that influence specific outputs—for LLMs. The authors discusses a list of effective source attribution desiderata: 1) accuracy, 2) robustness, 3) performance preservation, 4) scalability, 5) transferability, 6) adaptability. The authors propose WASA which embeds invisible characters into the sentences that are most representative of a data provider. WASA-LLM can fit in during or after the pre-training stage. The framework learns to insert watermark randomly in the desired sentence, by a modified transformer structure, where there is a separation of text and watermark token predictions. This benefits WASA-LLM in generating watermark for clean sentences.

strengths:
- The authors propose a novel method that tackles source attribution, an important and difficult problem.
- The authors lay out clear desiderata for source attribution and demonstrate that the proposed method has promise in satisfying the desiderata.
- The writing and presentation of the paper is clear and easy to follow. Experiments are well set up and detailed for each desiderata.

weaknesses:
1. The proposed source attribution method requires pre-training and performs worse with growing number of data providers. See Q1, Q2, Q3.
2. Other related work: 
https://arxiv.org/pdf/2302.14035 
https://arxiv.org/pdf/2403.03187
https://arxiv.org/pdf/2311.12233
3. Main experimental comparison is against BM25, though BM25 has limitations related to changed word order, and less semantic relationship captured. Experiments would be stronger compared with other retrieval methods.

questions:
Q1: The method is a pre-training source attribution method. How would this method integrate into pipelines of continuous training, where the number of data providers may also be growing? 

Q2: The authors discuss the performance drop as data provider number grows. How would this method scale to thousands or millions of data providers? 

Q3: Can you motivate the argument for source attribution via training rather than search more? Results in the paper show for 500 data providers, WASA is better than BM25. But practically, data providers may be in the number of millions rather than hundreds.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719810091

thread_type:
official_review

<thread_object>
creator: reviewer_tj9s

<content>
summary:
The authors introduce a framework named WASA (Watermarking for Source Attribution) that embeds unique, imperceptible watermarks into the data used for training LLMs. This approach enables the identification of specific data providers when synthetic texts are generated, thus providing a solution for source attribution. The paper discusses the key properties of an effective source attribution system, including accuracy, robustness against attacks, scalability, and performance preservation. WASA is demonstrated to achieve high source attribution accuracy while maintaining the generation quality of the LLMs. It utilizes unique Unicode characters as watermarks and is shown to be effective in empirical evaluations, even under adversarial conditions such as text modification. This work positions itself as a pioneering solution for source attribution in LLM-generated outputs, offering significant implications for data protection and IP verification in AI-generated content.

strengths:
- Provide convincing real-world application for readers.
- Clear definition of the source attribution problem.
-  Large amount of main experiments and ablation studies to show the aspects of a good source attribution algorithm the author claims.

weaknesses:
- Not clear difference from text watermark for copyright protection
- Not enough evidence for avoiding pertubation attack on word tokens.
- Data distribution problems of different data providers.
- Implementation details: embedding settings. 
- Lack of experiments on recently proposed LLMs.

questions:
1. The authors claim that source attribution is a new task proposed by them. I need more explanation of the differences between source attribution and text watermarks for copyright protection.

2. The authors claim that through the design of splitting the linear layer, the WASA-LLM can avoid pertubations. However, as far as I'm concerned, as described in Figure 3, all hidden states(including hidden embeddings of word tokens) will be in the forward pass of  We′[V + 1 : V + V ′], and will influence the outputs(generated watermark tokens). So, pertubations on input words have effects on the output watermarks. 

3. There may be some challenges in proving the authors' claim. The authors utilize a one-hot vector for data from a single provider. However, data from the same provider may be very different in distribution, and data from different providers may be similar. For instance, data from Arxiv and DBLP may have similar distributions, as they all contain scientific papers. And, data from a social media may be very different in topics and ideas. How can the author prove that with this problem, their proposed method can also work well? Extra experiments needed.

4. I also want to know the implementation details. As we all know, the way of adding tokens to the vocabulary is important for the final results. How do you initialize your embeddings of the watermark Unicode tokens? And, do you update the embedding parameters during training? This design may be important for results.

4. The author use GPT-2(maybe not an LLM) and llama-2 for experiment results. However, open-source LLMs with better capability have been proposed after them. LLaMA-3-8B[1] and other LLMs may be good choices. You can do supplementary experiments on LLaMA-3-8B to show me the performances.


[1] Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732755093284

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision of Paper

comment:
We thank all reviewers for their valuable feedback. We have uploaded a revision to the main paper including additional interesting experimental results and discussions highlighted in blue based on the suggestions. Please note that some references to line numbers in previous responses may have shifted slightly due to these revisions. Below is the summary of the changes we have made in this revision:

- We add a machine learning-based technique as an additional baseline in Sec. 4 (lines 311-314) and App. E.1.3 (lines 1151-1170), which compares the semantic representations of generated text from each contributor and synthetic text, following a similar setup to Foley et al., 2023. The results in Tables 8, 12, and 21 are updated.
- We add additional results on a frontier model Llama3-8B in App. E.1.8 with results in Table 13, mentioned in Sec. 4 (lines 307-308) in the main paper.
- We add the ablation study on the application of our WASA framework in the continuous training pipeline in App. F.7, mentioned in Sec. 4.5 (lines 458-459) in the main paper.
- We add an additional paraphrase attack using the DIPPER paraphraser in App. E.2.1 (lines 1348-1381).
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper received five review scores, three of which were still negative after rebuttal. Although the authors addressed some of the reviewers' concerns during the rebuttal phase (such as the differences from existing methods and missing some baseline methods), the reviewers still generally believed that the paper still had some important flaws. For example, most reviewers felt that the threat model, i.e., a given generated data comes from only one data source and the data distribution of each data source is different, is too strong and unrealistic; The scalability of the method when there are many data sources; The insufficient discussions of the resistance to adaptive attacks. Although I agree that this paper provides some new and interesting insights, given that it still has the major flaws mentioned above, it is not yet ready for publication.