Adversarial Attacks on Data Attribution

Authors: Xinhe Wang, Pingbang Hu, Junwei Deng, Jiaqi W. Ma
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: data attribution, adversarial attack, data-centric ai

Abstract:
Data attribution aims to quantify the contribution of individual training data points to the outputs of an AI model, which has been used to measure the value of training data and compensate data providers. Given the impact on financial decisions and compensation mechanisms, a critical question arises concerning the adversarial robustness of data attribution methods. However, there has been little to no systematic research addressing this issue. In this work, we aim to bridge this gap by detailing a threat model with clear assumptions about the adversary's goal and capabilities and proposing principled adversarial attack methods on data attribution. We present two methods, *Shadow Attack* and *Outlier Attack*, which generate manipulated datasets to inflate the compensation adversarially. The Shadow Attack leverages knowledge about the data distribution in the AI applications, and derives adversarial perturbations through "shadow training", a technique commonly used in membership inference attacks. In contrast, the Outlier Attack does not assume any knowledge about the data distribution and relies solely on black-box queries to the target model's predictions. It exploits an inductive bias present in many data attribution methods - outlier data points are more likely to be influential - and employs adversarial examples to generate manipulated datasets. Empirically, in image classification and text generation tasks, the Shadow Attack can inflate the data-attribution-based compensation by at least 200%, while the Outlier Attack achieves compensation inflation ranging from 185% to as much as 643%. Our implementation is ready at https://github.com/TRAIS-Lab/adversarial-attack-data-attribution

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730326798752

thread_type:
official_review

<thread_object>
creator: reviewer_wvst

<content>
summary:
Data attribution quantifies the contribution of individual data points during model training. Higher contribution scores indicate that a data point has more value within the training dataset. This paper proposes two adversarial methods—Shadow Attack and Outlier Attack—designed to manipulate data attribution. The authors evaluate these methods against various models and datasets, demonstrating that they can significantly inflate the contribution scores of otherwise benign data points.

strengths:
**Strengths:**

- **Addresses Critical Gap in Research**: A key contribution of this work is that it addresses a crucial research gap. The robustness of data attribution methods has remained largely unexplored. This study is the first to systematically investigate vulnerabilities in data attribution.

- **Significant Attribution Inflation**: The authors introduce two effective attacks that can substantially increase the contribution scores of input samples. The reviewer was impressed by the triple-digit inflation rates achieved, showing the potential impact of these adversarial techniques.

weaknesses:
1. **Threat Model**: The authors describe a black-box threat model, which traditionally implies that the attacker has no information about the model other than query access. However, the authors assume access to the model’s architecture, indicating a grey-box rather than a black-box scenario. It would be beneficial for the authors to revise the text to accurately reflect this nuanced threat model.

2. **Alternative to Creating Outliers**: The paper presents two algorithms designed to create natural-looking outliers with a high impact on the model. Given the complexity of these algorithms, a simpler approach might involve manually mislabeling inputs to create outliers (e.g., labeling a "cat" as a "dog" in a classification task), which would likely produce a similar exaggerated impact on the model. The authors should consider addressing why this more straightforward and cost-effective method was not explored.

3. **Simplistic Pipeline**: The authors assume a straightforward threat model where the defender does not employ methods to detect out-of-distribution (OOD) samples or sanitize inputs. Adversarial samples, such as those generated by the Shadow Attack, are known to be vulnerable to detection and degradation [1]. Testing the attacks against state-of-the-art OOD detection and adversarial input detection methods would help establish their robustness.

4. **Simple Defense**: A basic yet effective defense against these attacks would be data augmentation, such as cropping or shifting images to generate additional samples. This method improves model generalization, a common practice in machine learning, and could reduce the impact of adversarial samples like those introduced by the authors. The authors should demonstrate that including data augmentation would not impact their attack, as it more accurately reflects real-world training environments.

[1] https://arxiv.org/abs/1902.02918

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730482374017

thread_type:
official_review

<thread_object>
creator: reviewer_4ayy

<content>
summary:
This paper introduces two data attribution attack methods, Shadow Attack and Outlier Attack, that aim to inject new training data that maximizes the influence of target model outputs. Both attacks optimize over a compensation share function, which quantifies the influence of a subset of the training data to the target models predictions on test data. Shadow Attack assumes the adversary has access to data distributions and the training algorithm, and uses gradient ascent on training data to generate data points. Outlier Attack assumes black-box query access to the target model, and uses black-box image attack algorithms on training data to generate data points. The attacks are evaluated on image classification and text generation tasks using metrics that measure the change in compensation share before and after adversarial data is introduced.

strengths:
Thank you for submitting to ICLR 2025. I enjoyed reading about the compensation function used for the adversarial objective in the threat model section. I have listed several points that I find the paper do well below:
- The threat model is well defined for both attack methods. Assumptions made of target model and training/testing data access is clearly stated. The periodic data contribution attack setting between AI Developers and Data Providers is realistic. 
- The adversarial objective is very clear and the compensation function defined in the paper captures it in a novel way. Additionally, the Shadow Attack and Outlier Attack methodology is designed within the limits of the threat model.
- Baseline evaluation helps articulate the performance of the attack methods. Since the compensation share metric is new, the baseline evaluation puts into context how performant the attack is.

weaknesses:
The problem discussed is very interesting and deserves more attention from researchers and developers from both academia and industry. It is worth noting the effort made by the authors in designing novel attack methodologies. However, I believe there is still room for improvement, especially in making claims (e.g., attack effectiveness and efficiency) more concrete and clear and including more baseline evaluations that shed light on the performance of proposed attack under the given threat model. More details can be found below.
- The paper lacks well-defined research questions in the evaluation. It can be inferred that sections on Shadow Attack and Outlier Attack measure effectiveness, but it would improve the paper if there were claims made that the result would prove/disprove.
- Another baseline metric would be great to understand the performance of the attack under the defined threat model. While the random perturbation baseline helps contextualize the compensation share ratio of the attacks, another baseline assuming a white-box threat model (i.e., the adversary has access to the target model and its parameters) would highlight the performance under the defined threat model with more limitations.

questions:
Thanks to the authors for the submission to ICLR 2025. I enjoyed reading about the compensation function used for the adversarial objective in the threat model section. Most of my suggestions are already mentioned in the above Weakness section. Some questions (might be open-ended) are listed below. It would be great if more discussion or clarification on the following questions could be added to the paper:
- How is distortion budget measured for both image classification and text generation tasks? What norm is used for images and is the budget consistent with other white-box and black-box attack baselines?
- Analyzing how data attribution methods affect attack performance would strengthen the study of the attacks. For example, if one were to experiment with the Influence Function, Data Shapley, and TRAK methods for a particular {task, dataset, target model} configuration, how do the attacks perform?
- If we increase or decrease the size of Z_0 and Z_1 for each setting, would this make the attack more successful (i.e., how does performance change with respect to the amount of data)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699800990

thread_type:
official_review

<thread_object>
creator: reviewer_t7fd

<content>
summary:
This paper studies adversarial attacks to data attribution based compensation mechanism.

Two empirical attacks are proposed.
1. Shadow attack, which assumes access to data distribution & training algorithm, creates a surrogate data attribution system and adversarially optimize the compensations with the surrogate.
2. Outlier attack, which assumes access to the black-box query of the model and directly optimizes to minimize the confidence of the model to generate outlier data, which could correspond to higher compensation.

strengths:
The paper is overall clear and easy to follow. 

Since the adversarial robustness of data attribution is a reasonable subject, the significance is not considered an issue.

weaknesses:
The experimental objective function of the adversarial attack to data attribution requires much better justifications.

The paper motivates the study of the robustness of data attribution with data attribution based compensation.
However, it is natural that such compensation designs reward only for data contributing to good/desired behaviors: For example, they might allocate compensations only based on attributions for **correct** predictions (these could be from a prepared validation/test set or from user feedbacks). Alternatively, they might withdraw compensations or even incur penalties for **incorrect/undesired** predictions.

A concern is that in such, arguably more reasonable settings, neither of the proposed attacks will be in fact effective. For shadow attack, only data attributed greatly for correct predictions will be awarded, which is totally fine as the "attack" is in some sense actually making the data more valuable, with efforts. For outlier attacks, the outliers are less likely to be attributed if the prediction is correct and consistent with regular training data.

questions:
See Weaknesses for my primary concerns, which is in fact somewhat critical to make decisions regarding the impact of the observations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699830687

thread_type:
official_review

<thread_object>
creator: reviewer_pqy1

<content>
summary:
This paper introduces a novel adversarial attack on data attributions, where the adversary aims to maximize the compensation share received by the adversary. The authors set a few restrictions and assumptions on the adversary's capabilities and the target model. Two different attacks are proposed based on different choices of assumptions. The shadow attack is based on the assumption that the adversary has access to the distribution of the training data and the validation data, while the outlier attack is based on the assumption that the adversarial can query the model to get the data attribution values. The experimental results demonstrate that both the shadow attack and the outlier attack are highly effective in manipulating data attribution methods, leading to substantial increases in the attacker's compensation share.

strengths:
1. The paper is well-organized and easy to follow. The authors clearly define the adversary's objective, capabilities, and assumptions, which helps readers understand the problem setting.
2. The proposed attacks are novel. The shadow attack and the outlier attack are based on certain scenarios and are effective in manipulating data attribution methods.
3. The experimental results are comprehensive. The authors evaluate the proposed attacks on multiple datasets and data attribution methods, demonstrating the effectiveness of the attacks in practice.

weaknesses:
1. The surrogate objective used to replace the discrete compensation share is not well justified. The authors should provide more insights into why the surrogate objective is a good approximation of the compensation share. Is the surrogate itself already a reasonable to represent the compensation share?
2. The paper emphasizes the validity of the assumptions using the examples of large language models, while the experiments are mostly conducted on relatively smaller size datasets (*e.g.*, MNIST and CIFAR-10). It would be more convincing if the authors could provide more scalability analysis on larger datasets.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732516804254

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Message To All Reviewers

comment:
We thank all the reviewers for their insightful comments and suggestions. We apologize for the delayed response as we have conducted a series of additional experiments to fully address the reviewer comments. In particular, we have 1) conducted experiments on a larger dataset, Tiny ImageNet; 2) evaluated the proposed attacks when data augmentation is used in training the models; 3) included a white-box reference baseline; 4) evaluated the proposed attacks with a slightly different compensation mechanism suggested by Reviewer T7Fd; 5) and other sensitivity analyses experiments. In all cases, we have once again confirmed that the proposed attacks are highly effective.

We have addressed all the comments in the detailed individual response to each reviewer, as well as updated our paper draft to reflect the changes. Here we would like to highlight a couple of key points in our response.

First, to the best of our knowledge, this is **the first systematic study of adversarial attacks on data attribution**. By revealing surprisingly simple yet effective attack methods, including one requiring only black-box access to the model, this work lays a critical foundation for further research. While Reviewer **wvst** treats the lack of defense techniques in the threat model as a major drawback of this work, Reviewers **4aYY** and **pQy1** have acknowledged the soundness of our threat model. We believe that without a clear understanding of potential threats, the AI developers will usually not preemptively adopt defense techniques due to the robustness and utility trade-off. Therefore, we believe this work presents sufficient contributions even without the development of defense techniques.

Second, we would like to highlight that the proposed compensation mechanism is indeed designed to reward training data points that contribute to correct/desired model predictions (as opposed to rewarding both correct and incorrect predictions). As this is the main (and almost only) concern of Reviewer **T7Fd**, we hope our clarification and additional experiments can sufficiently address the reviewer’s concern.

We thank the reviewers again for their efforts in evaluating our work. And we are more than happy to provide further information if there is any further concern.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an adversarial attack on the data-attribution-based compensation mechanism. 
The paper introduces two attacks, Shadow Attack is a gray-box attack method while Outlier Attack is a black-box attack method.
The reviewers acknowledge that the paper is well-written and the threat model is well-defined for both attack methods.
Overall, the reviewers think this is an important and interesting problem, and the proposed method is novel. The empirical results are reasonably good.  Given the novelty, I recommend accept.