Adversarial Attacks on Data Attribution
Authors: Xinhe Wang
Pingbang Hu
Junwei Deng
Jiaqi W. Ma
Venue: ICLR 2025 Poster (2025)
Keywords: data attribution
adversarial attack
data-centric AI
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We provide a comprehensive study of the adversarial attacks on data attribution methods.

Abstract: Data attribution aims to quantify the contribution of individual training data points to the outputs of an AI model, which has been used to measure the value of training data and compensate data providers. Given the impact on financial decisions and compensation mechanisms, a critical question arises concerning the adversarial robustness of data attribution methods. However, there has been little to no systematic research addressing this issue. In this work, we aim to bridge this gap by detailing a threat model with clear assumptions about the adversary's goal and capabilities and proposing principled adversarial attack methods on data attribution. We present two methods, *Shadow Attack* and *Outlier Attack*, which generate manipulated datasets to inflate the compensation adversarially. The Shadow Attack leverages knowledge about the data distribution in the AI applications, and derives adversarial perturbations through "shadow training", a technique commonly used in membership inference attacks. In contrast, the Outlier Attack does not assume any knowledge about the data distribution and relies solely on black-box queries to the target model's predictions. It exploits an inductive bias present in many data attribution methods - outlier data points are more likely to be influential - and employs adversarial examples to generate manipulated datasets. Empirically, in image classification and text generation tasks, the Shadow Attack can inflate the data-attribution-based compensation by at least 200%, while the Outlier Attack achieves compensation inflation ranging from 185% to as much as 643%. Our implementation is ready at https://github.com/TRAIS-Lab/adversarial-attack-data-attribution

Published as a conference paper at ICLR 2025
ADVERSARIAL ATTACKS ON DATA ATTRIBUTION
Xinhe Wang
Pingbang Hu
University of Michigan
University of Illinois Urbana-Champaign pbbeillinois.
sevenmar umi ch. edu
Junwei Deng
Jiaqi W. Ma
University of Illinois Urbana- Chammaign Junwei d2@ 11 linoiss .ddu
University of Illinois Urbana- Champpainn
jiaqima iaqimall @ i 11 iiniiii wwoissddu edu
ABSTRACT
Data attribution aims to quantify the contribution individual training data points to the outputs of an Al model, which has been used to measure the value of training data and compensate data providers. Given the impact on financial decisions and compensation mechanisms, a critical question arises concerning the adversarial robustness of data attribution methods. However, there has been little to no sys- tematic rescarch addressing this issue. In this work, aim to bridge this gap by detailing a threat model with clear assumptions about the adversary's i goal and ca- pabilities and proposing principled adversarial attack methods on data attribution. We present two methods, Shadow Attack and Outlier Attack, which generate ma- nipulated datasets inflate the compensation adversarially. The Shadow Attack leverages knowledge about the data distribution in the Al applications, and derives adversarial perturbations through "shadow training". technique commonly used in membership inference attacks. In contrast, the Outlier Attack does not assume any knowledge about the data distribution and relies solely on black-box queries to the target model's predictions. It exploits an inductive bias present in many data attribution methods-outlier data points are more likely to be influential- and employs adversarial examples to generate manipulated datasets. Empirically, in image classification and text generation tasks, the Shadow Attack can inflate the data-attribution-baas
compensation by at least 200%. while the Outlier At- tack achieves compensation inflation ranging from 185% to as much as 643%. Our implementation is ready http:///gghhb..omm//AASSSaaaaaaa dltta-attribution
INTRODUCTION
Data attribution aims to quantify the contribution of individual training data points to the outputs of an Artificial Intelligence (AI) model (Koh & Liang] 2017). A key application of data attribution to measure the value of training data in Al systems, enabling appropriate compensation for data providers (Ghorbani & Zou 2019 Jia et al. 2019). With the rapid advancement of generative AI, these methods have gained increased relevance, particularly in addressing copyright concerns. Re- cent studies have explored economic frameworks using data attribution for copyright compensation, showing promising preliminary results (Deng & Ma [023 Wang al., 2024).
Given the significant potential of data attribution methods for data valuation and compensation, an important question arises regarding their adversarial robustness. As these methods influence financial decisions and compensation mechanisms, they may attract malicious actors seeking to manipulate the system for personal gain. This underscores the need to investigate whether data attribution methods can be manipulated and exploited, as their vulnerabilities could lead unfair compensation, undermining the trustworthiness of the solutions built on top of these methods.
However, adversarial attacks data attribution methods have received little exploration in prior literature. Along with the proposal of a natta-atribution-baee based economic solution for copyright compensation, Deng & Ma (2023) briefty experimented with few heuristic approaches (e.g., dupli-

---

Published as a conference paper at ICLR 2025
cating data samples) for attacking data attribution methods. A systematic study that clearly defines the threat model and develops principled adversarial attack methods has yet to be conducted.
This work presents the first comprehensive study to fill this gap. We first outline the threat model by detailing the data compensation workflow and specifying the assumptions we made. One key as- sumption is that the data contribution is periodic, and there is certain persistence across consecutive iterations of data contributions, which is the source of knowledge that the adversary could exploit. We also assume that the adversary may either have access to the distribution of the data used by the target model of the Al system or can get black-box queries of the target model's predictions, both are commonly seen the AI security literature (Shokri al. 2017 Chen et al. 2017).
Subsequently, we propose two adversarial attack strategies, Shadow Attack and Outlier Attack, re- spectively relying on different assumptions about the adversary's 's capabilities. The Shadow Attack relies on the access to data distribution and employs the "shadow training" technique commonly used in membership inference attacks (Shokri et al., (017)) to train "shadow models" that imitate the target model. The adversary can then directly perturb their dataset to achieve a higher compensation on these shadow models. The Outlier Attack, instead, does not assume knowledge about the data dis- tribution but only relies on black-box queries of the target model's predictions. The key idea behind this method lies in an inductive bias of many data attribution methods-outhe data points are more likely to be more influential. The proposed Outlier Attack utilizes adversarial examples (Goodfellow et al. 2015 Chen et al. 2017) to generate realistic outliers in a black-box fashion.
We conduct extensive experiments, including both image classification and text generation settings, to demonstrate the effectiveness of the proposed attack methods. Our results show that by only adding imperceptible perturbations real-world data features, the Shadow Attack can inflate the adversary's 's compensation to at least 200% and up to 456%, while the Outlier Attack can inflate the adversary's 's compensation to at least 185% and up to 643%.
Overall, our study reveals a critical practical challenge-aavversaraal adversarial vulnerability-in deploying data attribution methods for data valuation and compensation. Moreover, the design of the proposed attack methods, especially the Outlier Attack that exploits a common inductive bias of data attri- bution methods, offers deeper insights into these vulnerabilities. These findings provide valuable directions for future research to enhance the robustness of data attribution methods.
2 RELATED WORK
Data Attribution for Data Valuation and Compensation. Data attribution methods have been
widely used for quantifying the value of training data in Al applications and compensating data
providers (Ghorbani & Zou
2019:
Jia et al.
2019 Yoon et al.] 2020 Kwon & Zou, 2022 Feldman
& Zhhngg 2020 Xu et al.
2021
Lin et al., 2022, Kwon & Zoul 2023 Just et al. 2023] Wang
& Jia] 2023 Deng & Ma
2023 Wang al. 2024). With the rapid advancement of generative
AI, these methods have gained increasing relevance due to the growing concerns around copyright.
Recent studies have proposed economic solutions using data attribution for copyright compensation,
yielding promising preliminary results (Deng & Ma 2023 Wang et al. 2024). Given the significant potential of data attribution methods for data valuation and compensation, a critical question arises regarding their adversarial robustness. Aside from one earlier exploration using heuristic approaches to attack data attribution methods (Deng & Ma 2023), no systematic study has addressed this issue. This work presents the first comprehensive study that outlines a detailed threat model and proposes principled and effective approaches for adversarial attacks on data attribution methods.
Membership Inference Attack. Membership inference attacks aim to infer whether a specific data point was used during the training of machine learning model, typically without knowing the actual training dataset or having white-box access to the model (Shokri et af.) (017). The general strategy involves leveraging information such as model architecture, training data distribution, or black-box model predictions (Shokri et al., 2017 Song & Mittal 2021). We refer the readers to Hu et al. (2022) for a detailed survey on this topic. One of the proposed attack methods, the Shadow Attack, is inspired by the "shadow training" technique (Shokri et al., 2017) commonly used in membership inference attacks, where the adversary draws "shadow samples" following the same distribution as the actual training dataset used by the target model to be attacked, and trains "shadow models" based on the shadow samples to imitate the target model.
2

---

Published as a conference paper at ICLR 2025
A Adversarial Example. Adversarial example (Goodfellow et al. 2015) is well-known phe- nomenon where small, often imperceptible perturbations to imput features can significantly alter the predictions of machine learning models, particularly deep neural networks. These perturbations can be generated through black-box queries to the model predictions (Chen et al. 2017] Ilyas et al., 2018 et al. (019). For comprehensive review of adversarial examples, see Yuan et al. (2019); Chakraborty et al. (202l). In the proposed Outlier Attack, we employ black-box adversarial attack methods for generating adversarial examples to generate realistic outliers relative to the training dataset of the target model, without needing access to the training dataset or the model details.
THE THREAT MODEL
3.1 THE DATA COMPENSATION SCENARIO
We consider a scenario where there is an AI Developer, and (potentially large) set of Data P Providers. The Data Providers supply training data for the AI Developer to develop an Al model. In return, the Data Providers are compensated based on their data's contribution to the model, as measured by specific data attribution method.
Periodic Data Contribution. We assume that the Data Providers contribute data to the AI Developer periodically, a common practice in many Al applications. For example, large lan- guage models need periodic updates to stay aligned with the latest factual knowledge about the world (Zhang et al.) 2023); recommender systems must adapt to evolving user preferences (Zhang al. 2020): quantitative trading firms rely on up-to-date information to power their predictive models and generative models for music or art benefit from fresh, innovative works by artists to diversify their creative outputs (Smith et al., 2024). However, such periodic data contribution intro- duces risks: a malicious Data Provider (referred to as Adversary thereafter) could exploit information from previous iterations to adversarially manipulate their future data contribution, po- tentially inflating their compensation unfairly.
To formalize this scenario, without loss of generality, we consider two consecutive iterations of data contribution, denoted as time steps = 0 and t 1. At t 0, there is no Adversary and the training dataset consists solely of contributions from benign Data Providers. This dataset is represented as Zo Z. where 2 is the set of all possible training datasets. At L 1, the training dataset is Z1 = Z U ZÍ. where Z E Z represents the training data provided by benign Data Providers, while Z Z is the set provided by the Adversary. We also assume that zi n nZ = 2, meaning there is no overlap between the two datasets. Finally, for a dataset Z E Z, each data point 2 E Z is represented as a pair= 2 = (ax, y), where X' is the input feature and y E y is the prediction target, with X' and ) referring to the feature space and target space, respectively.
AI Training and Data Attribution. Let M represent the of AI models, and let T Z M denote a training algorithm, mapping a training dataset Z E Z to a model T(Z) E M.
In data attribution, we aim to understand how individual data points from a training dataset 2 Z contribute to the model output on a target (validation) data point from validation dataset V. Given Z and V. data attribution method derives a contribution function T: Z V + E that assigns a real value 7(2, v) to each training data point 2 E Z for given validation data point v E V. Denote the set of all such function r's as C, and the set of all possible validation sets as V. Therefore, data attribution method can be formalized as a function A: 2 x M x V - C. In most cases, we will consider (Z, T T(Z), V), hence Z and V alone suffice to specify the resulting
Compensation Mechanism. In practice, the contribution function 7 derived from data attribution methods may not reliably measure the contributions of all training data points due to the inherent randomness in AI model training and the need for efficiency (Wang & Jia 2023) Nguyen et al.) 2024). Specifically, measurements of data points with smaller contributions are often less reliable than those of the most influential contributors. Following Deng & Ma (2023), we consider a com- pensation mechanism where only the top-k influential training data points for each validation data point V receive fixed amount of compensation.
See, for example, the Bloomberg market data feed: https: htops::/wwww.r bl www.bbg.omberr.com cata aataalog/mmaa oo/morr/eeeeeeeee
3

---

Published as a conference paper at ICLR 2025
3.2 THE ADVERSARY
We now discuss the Adversary's objective and capability under the data compensation scenario.
The Objective of the Adversary. Let Vi be the validation dataset used step 1. The objective of the Adversary is to construct a dataset Zî that maximizes the compensation share received by the Adversary, which is defined as
c
Σ Σ 1[T(2, E Tops (1T( 2' E Z1))],
(I)
K
2E2F DEV,
where Ti = T(Z), Vi) is the contribution function at f = 1; T.p extracts the top-k elements from finite set of real numbers; and [[] is the indicator function.
The Capabilities of the Adversary. We first outline the limitations imposed on the Adversary.
In realistic scenarios, the Adversary does not have access any of the following:
the exact training datasets (Zo and ZD) and the exact validation datasets (Vo and Vi);
white-box access to the trained models (Z)) and T(Z1):
the contribution functions A(Zo, TZo)), Vo) and A(Z1, (())),
Then we make the following assumptions to characterize the capabilities of the Adversary. Assumption 1 (Persistence). Assume that Zu S Z and /Zol/ Z is close to 1. Additionally, assume that Vg and V1 are independently sampled from the same distribution.
Assumption 2 (Access to data distribution and training algorithm). Assume that the Adversary has access to the distributions of Zo and Vo. Additionally, assume that the Adversary has knowl- edge about the training algorithm T (but not the model T TZ))).
Assumption (Black-box access model). Assume that the Adversary has access to black-box query to the model T TZo)) to query the model predictions on any input feature E X.
Intuitively, Assumption D assumes persistence between time steps t gained from & = 0 can inform the attack at f = 1. In Assumption and Assumption 3 it is worth noting that the Adversary" access information limited to what is available at only.
so that information
In practice, Assumption I is realistic in many real-world applications. For example, in applica- tions that have periodic model updates, such as large language models or recommender systems, substantial portion of the training data often remains consistent across consecutive iterations, with only incremental changes. Moreover, Assumption 2 a common assumption in the membership inference attack literature (Shokri et al.] (017)), while Assumption 3 reflects common setup for generating adversarial examples against neural network models (Chakraborty et al. (021).
The two proposed attack methods in Section 4 and Section 5 respectively rely Assumption 2 and Assumption 3 but do not depend on both simultaneously. As result, the first method is a gray-box attack method while the second method is a black-box attack method.
The Action Space of the Adversary. We assume that the Adversary is restricted to making only small perturbations an existing set of real data points to construct the adversarial dataset Zí. This implies that the Adversary cannot introduce entirely synthetic or arbitrary data, but rather. can modify real data points subtly. This reflects realistic adversarial scenario as overly large or unnatural alterations would be easily detectable.
4 SHADOW ATTACK
In this section, we introduce the proposed Shadow Attack, which leverages Assumption | and As- sumption 2 and exploits the knowledge about the data distribution at 0 to perform attacks.
At a high level, the Adversary first performs a shadow training process, where models are trained on data drawn from distribution similar to that of the training dataset Zo. allowing the Adversary to approximate the target model T(z)). The Adversary then applies adversarial perturbations to the data points they plan to contribute to the AI Developer at 1. The adversarial perturba- tions are derived by maximizing the data attribution values on the models obtained through shadow training. See Figure T for illustration.
4

---

Published as a conference paper at ICLR 2025
8
7(
M(
Z {
233 0 133
Z U Z(0)
converge 2 =
1 1+
{1} Sampled from Z
=
Update
Gradient Ascent
VV
Figure 1: An illustration of the Shadow Attack method. Shadow training datasets Z(0)s are sam- pled estimate the compensation share of set of data points Z if it were contributed the AI Developes which can be leveraged to perturb data points in Z to get a higher compensation share.
4.1 SHADOW TRAINING
Given dataset Z E Z (that the Adversary may eventually contribute to the AI Developer as 71), the goal of the shadow training process is to estimate the data attribution values of the elements in Z as if this dataset were contributed to the AI Developer at 1.
We first sample m shadow training datasets, denoted as Z(1,, Z z ((m), These datasets are independent of the actual training dataset used by the AI Developer but follow the same dis- tribution. For each shadow training dataset Z i I,..., m. train corresponding shadow model M9) E MM. The shadow model Mi is trained on z U Z using the same training algorithm T as by the AT Deve 1 oper
i.e., M = T(Z U 2).
In order to estimate the data attribution values, we further sample a shadow validation dataset ( following the same distribution Vo. We can estimate a shadow contribution function using each shadow training dataset Z Z, the corresponding shadow model ( and the shadow validation dataset (()), resulting in (() = A(Z U Z, (6) V(0).. Similar to Eq. (1). we consider the following shadow compensation share for the dataset of interest Z:
ΣΣ Σ
(2,)) (op 2 ZOUZ)]] (2)
(Z)
mk| [
i= =EZ vEV(O)
4.2 ADVERSARIAL MANIPULATION MAXIMIZING SHADOW COMPENSATION RATE
Given the shadow compensation rate, the natural idea is to apply adversarial perturbations Z by solving Z = arg max C2 (Z'), where A (Z) specifies the space of datasets with unde- tectable perturbations.
practice, however, solving this optimization problem has two technical challenges. Firstly, the objective c.(.) is discrete and can be difficult to optimize. Secondly, when doing sterative-style optimization, such gradient ascent, 7(1)'s need to be updated and re-evaluated at every step as it depends on Z and hence M = (( U Z). This leads to two problems: On the one hand, updating (i's requires retraining, which is computationally heavy; on the other hand, many data attribution methods are computationally expensive, even with all the data and (re- trained) models available. Hence, maximizing c.(.) requires repeatedly retraining and running data attributions on perturbed datasets, which may be infeasible for many cases.
To address the first challenge, replace c.(-) with the following surrogate objective
m ΣΣ Σ (2,)),
d,(Z) V
(3)
ZEZ eEV(0)
which aims to directly maximize the contribution values of the data points in on the shadow validation set (0)) To address the second challenge, we first approximate the retrained models by the initial models trained with the original Z (before any gradient ascent steps) for computational
In our experiment in Section 6.2 shadow models have slightly different architecture compared to the target model.
demonstrate that the Shadow Attack remains effective when the
5

---

Published as a conference paper at ICLR 2025
efficiency, and further adopt Grad-Dot (Charpiat et al. 2019), of the most efficient data attribution methods when evaluating the contribution function Using this method, the contribution value (z, v) for any training data point z and validation data point equals the dot product between the gradients of the loss function on z and that on v, evaluated with model M
With such simplification, the adversarial perturbation can be derived by maximizing Eq. (3). which can be solved efficiently through gradient ascent. Note that since Eq. 33) is a constrained optimization problem, when (2)) is bounded, gradient ascent is guaranteed to converge some local optimums. In practice, we perform a fixed number of iterations and carefully control the overall perturbation budgets when solving Eq. (3). The computation cost for each iteration is approximately the same as one forward and one backward pass on each the m shadow models.
5 OUTLIER ATTACK
For large-scale AI applications such as generative services, Assumption 2 might be overly strong as it could be difficult for the Adversary guess the distribution of the full training data. How- ever, in this case, Assumption 3 often holds. In this section, we further propose Outlier Attack, which only relies on Assumption and Assumption See Figure illustration.
Black-Box Query
Prediction
T(Zo)
7 { Collect from Z
Z
If increase enough
7
{
1
Update
Black-Box Attack
Figure 2: An illustration of the Outlier Attack method. Here, f(Z) denotes the loss used by the model T(2)) when evaluated on the dataset Z. The data points in Z are perturbed by maximizing the loss ((Z) through black-box attack methods designed to generate adversarial examples.
5.1 THE OUTLIER INDUCTIVE BIAS OF DATA ATTRIBUTION
The core idea behind Outlier Attack leverages an inherent inductive bias present in many data attri- bution methods: outlier data points in the training dataset tend to be more influential. Indeed, one of the very first applications of the Influence Function developed in the statistic literature was detect outliers in training data (Cook 1977). Consequently, if the Adversary contributes set of outlier data points, they are more likely to get higher compensation, especially given that the compensation mechanism focuses on the top influential data points.
However, translating this intuition into a practical adversarial manipulation strategy poses two sig- nificant challenges. Firstly, the contributed data points must closely resemble real-world data; oth- erwise, they could be easily flagged by data quality filters employed by the AI Devel 1 oper. r. Sec- ondly, we aim develop an attack method that does not rely direct knowledge about the training dataset or its underlying distribution. This makes it challenging to determine whether certain data points truly qualify outliers relative to the training dataset gathered by the AI Developer.
5.2 GENERATING REALISTIC OUTLIERS WITH ADVERSARIAL ATTACKS
To tackle the first challenge, we propose starting with a set of real-world data and transforming them into outliers through small adversarial perturbations. With careful design, this strategy also addresses the second challenge: if after the perturbation, an AI model has low confidence in correctly predicting its target, then this perturbed data point likely behaves as an outlier relative the model's training dataset. We discuss two key design aspects for achieving these goals:
Emmiriially, this works well even when the actual data attribution method used by the A Devel oper advanced ones, shown in Section 6.2

---

Published as a conference paper at ICLR 2025
1. Data component to be perturbed. Recall that data point (z, y) consists of both the input feature x and the prediction target y. To obtain an outlier data point from 2, we perturb the feature only, without modifying the target y. Intuitively, flipping the target y could easily result in an outlier data point through mislabeling, such an outlier typically degrades model performance and is likely to be identified as negatively influential. Moreover, mislabeled data points are easier for the AI Developer. to detect. Therefore, it is more effective to perturb only the feature r while keeping the target y unchanged.
2. Objective of perturbation. For each data point confidence in predicting the annotated target y by perturbing 2. In most cases, this is equivalent increasing the loss e between the model prediction and the target y-
(z,y), we aim to decrease the model
Combining these two design choices, the resulting perturbed data points coincide with what is com- monly referred to as adversarial example in the literature of adversarial attacks against neural net- work models. Consequently, existing adversarial attack methods for deriving adversarial examples can be leveraged to generate realistic outliers in our problem setup. Notably, this strategy very general and can be applied to a variety of data modalities and models by leveraging different off- the-shelf black-box adversarial attack methods.
Choices of Adversarial Attack Methods. We consider two concrete machine learning settings, image classification, and text generation, and discuss the choices of adversarial attack methods. For smaller-scale image classification settings, we leverage Zeroth Order Optimization (ZOO) (Chen) al. (017)) based adversarial attacks, which approximates the gradient ascend on the loss function with respect to the data features using black-box queries to the target model T(Z)). For larger-scale image classification settings. we employ a more advanced black-box box adversarial attack method, Simba (Guo et al. 2019), which is computationally more efficient. At a high level. Simba sequen- tially perturbs each scalar pixel value in the image by trying to perturb in both directions and accept the perturbation once increases the loss. For the text generation setting, we utilize TextFooler (Tin) eet al., [2020), a black-box adversarial attack method tailored for text data. In all the methods of choice, the attack only requires black-box queries get the predictions of the target model.
5.3 THEORETICAL UNDERSTANDING
The following theorem further provides theoretical insights about the effectiveness of the proposed Outlier Attack. The formal statement, notations, and the proof can be found in Appendix A
Theorem 5.1 (Informal). Consider model trained by ERM on a dataset of size n with a smooth loss E with respect to model parameters 0. Assume its corresponding influence score T, gradient V Vol(O, 2). and Hessian V2,(0, 2) are all bounded, i.e., |r|_]| VoLl22 Volllop op = (1). Assume the inftuence score T is based on the influence function by Koh & Liang 2017), which takes the form
Tip(2y, Ziesi 0) Vol(0 2rest)
Vgl(0, 24)
V Vel(o, 2y),
n
where Zieu is test data point while { are the training data points and ê is the model param- in perturbed to z'y. eters trained on {= Then, for any Ziem when 2j in {
(zi Zusi 0²) TIF (Zis Rust 0) + (11/n) for all + j. and
* Tr/2/4, zwe; Õ') (1 Ruei 0) + O(1/n).
where 0' refers to the model parameters trained on the perturbed dataset {zilize [zz].
Intuitively, perturbing 2j through adversarial attack will increase the magnitude of the gradi- ent Vol(0, 23), which tends to also increase the influence score on the original model 0, i.c., (2/ Presti 8) > Tzr(2z, Rrest 8). However, in order get higher compensation share, the per- turbed data point z/ needs to have a high influence score on the model 0' trained on the perturbed dataset, i.c., nE(2/, Res; 0¹), which is not guaranteed by the adversarial attack without characteriz- ing the new model 0'. Theorem 5.1 exactly does this and asserts that TT Prest 0') will be close
7

---

Published as a conference paper at ICLR 2025
to (z) Pest 8) when the datasets used to train 0 and 0' are close. It further guarantees that the influence scores of the rest unchanged data points will also remain similar. This explains why the effect of adversarial attacks
the original model can successfully translate to the new model. The results in Theorem 5.1 can be generalized to the case where more than one data point is perturbed, and a small set of clean data is also added to train the new model. The bound will change from ((1/n) to Ok/n), where k is the total number of changed data points in the dataset.
6 EXPERIMENTS
In this section, we evaluate the effectiveness of the proposed attack methods by the increase of compensation share after manipulating the data with the proposed attack methods.
6.1 EXPERIMENTAL SETUP
Tasks, Datasets, Target Models, and Data Attribution Methods. We consider two machine- learning tasks: image classification and text generation. For image classification, we experiment on MNIST (LeCun 1998), Digits (Jiang et al.) 2023) and CIFAR-10 (Krizhevsky & Hinton 2009) datasets, with different target models including Logistic Regression (LR), Multi-layer Perceptron (MLP), Convolutional Neural Networks (CNN), and ResNet-18 (He al. 2016). We also em- ploy three popular data attribution methods, including Influence Function Koh & Liang 2017), TRAK (Park al. 2023), and Data Shapley (Ghorbani & Zoul 2019). For text generation, we conduct experiments on NanoGPT (Karpathy, (202)) trained on the Shakespeare dataset (Karpathy 2013), with TRAK as the data attribution method. Finally, for the image classification settings, we evaluate both Shadow Attack and Outlier Attack, while for the text generation setting, we evaluate Outlier Attack only as Assumption ussually does not hold for generative Al settings. For the data attribution algorithms, we adopt the implementation from the dattri library (Deng et al. 2024 The experimental settings are summarized in Table I
Table 1: Summary of the experimental settings.
Setting
Task
Dataset
Target Model
Attribution Method
(a) Image Classification
MNIST
LR
Influence Function
(b) Image Classification
Digits
MLP
Data Shapley
(c) Image Classification
MNIST
CNN
TRAK
Image Classification
CIFAR-10 Shakespeare
ResNet-18
TRAK
(e) Text Generation
NanoGPT
TRAK
Data Contribution Workflow. For the image classification settings (a), (c), and (d), we set ZZol = 10000, |Z 100, and |Z 11000. This setup simulates the following data contribution workflow: At t = when there is no Adversary, and 10000 training points are used to train the model (Zz). At = 1, the Advers rsversary contributes 100 perturbed training points and other Dat aata Providers contribute 900 new training data points. Together with the previous 10000 training points, total of 11000 training points are used to train the model T(Z,). For image classification setting (b), due to the size of the dataset, we set Zij 1100, |- 30 and | = 1100 for Outlier
Attack, [Zol = 800, | 30 and |4 850 for Shadow Attack. The text generation setting
follows similar workflow with |Zol = 4706, | = 20, and |Z = 6274.
Evaluation Metrics. We evaluate the proposed attack methods using two metrics, where we set k = 100 when counting the top-k influential data points in both cases. The first is the Compensation Share (Eq. ( where for each attack method, we calculate c( c(Z4) respectively when the original dataset without perturbation is contributed as Z (Original) and when the manipulated dataset after perturbation is contributed as 2 (Manipulated). The increase of o(2f) after the perturbation measured by the Ratio reflects the effectiveness of the attack.
To gain a more refined understanding of how the adversarial perturbations affect the top-k influential data points for individual validation data points in V1, consider the second evaluation metric named Fraction of Change. For each validation data point v E Vi, it measures how many data points in Zí appear in the top-k influential data points for In comparison to when the original
8

---

Published as a conference paper at ICLR 2025
dataset without perturbation is contributed as Zf, calculate the fraction of validation data points in V1 that contain more data points from 2 in the top-k influential data points when the manipulated dataset after perturbation is contributed as Zí. Similarly, we calculate the fraction of validation points that contain the same number of or fewer data points from Zi after perturbation. We report the three fractions under the categories More, Tied, and Fewer, where the higher fraction for the More category indicates that the attack method influences the validation data points more broadly,
6.2 EXPERIMENTAL RESULTS: SHADOW ATTACK
The results of the proposed Shadow Attack method are shown in Table 2 on all three image classification settings outlined in Table [ For each of the settings, we first consider the case that the shadow models have the same architecture as the target model, as shown in the first three rows of Table 2 Additionally, for setting (d) where the target model is ResNet-18 (forth row), we further consider using ResNet-9 as the shadow model (last row), which simulates scenarios where the Adversary's knowledge about the training algorithm is limited.
We conduct experiments
Table 2: Results of the Shadow Attack method. The target models used for evaluation in each setting are listed in Table [ while the shadow models used in the attack are listed under Shadow Model. The proportion /z/1 //// Zi of the data contributed by the Adversa ry relative to the full dataset t == is also reported. A higher Ratio indicates a more effective attack, and a higher fraction under More means that the attack influences the validation data points more broadly.
Compensation Share
Fraction of Change
Setting
Shadow Model
/
Original
Manipulated
Ratio
More
Tied
Fewer
LR
0.0098
0.0098
0.0477
456.1%
0.955
0.038
0.007
(b)
MLP
0,0352
0.0152
0.0435
286.2%
0.533
0.333
0.134
(c)
CNN
0,0098
0.0112
0.0467
417.0%
0.781
0.195
0.024
(d)
ResNNet-18
0.0098
0.0095
0.0213
217.3%
0.655
0.259
0.086
(d)
ResNet-9
0.0098
0.0095
0.0196
206.3%
0.622
0.310
0.068
The Shadow Attack method is highly effective in increasing the Compensation Share across all the settings. The Ratio of the Manipulated to the Original ranges from 206.3% to 456.1% represent- ing substantial increase. Notably, the last row corresponds the setup where the shadow models have the ResNet-9 architecture while the target model is a ResNet-18, and the Shadow Attack re- mains significantly effective 3%.) although being slightly worse than the case where shadow model and target model shares the same architecture (forth row, 217.3%.).
The Shadow Attack is also uniformly effective across a wide range of validation data points, as measured by the metrics of Fraction of Change. The attack is able to increase the number of top-k influential points from Z for more than 60% of the validation points. In contrast, only less than 9% of the validation data points have fewer top-k influential points from Z after the attack.
6.3 EXPERIMENTAL RESULTS: OUTLIER ATTACK
The results of the proposed Outlier Attack method are presented in Table 3 where we include all four settings summarized in Table [ The black-box attack method for generating the adversarial examples is chosen according to the discussion in Section 5.2
On experimental settings (a) and (c), the Outlier Attack performs even better than the Shadow At- tack in terms of both Compensation Share and Fraction of Change, achieving a higher Ratio and a larger fraction under More. Across all four settings. the Ratio ranges from 185.2% to 643.9%, demonstrating the exceptional effectiveness of the Outlier Attack. Finally, the results of text gener- ation setting (e) further highlight the applicability of the proposed method to generative Al models.
6.4 BASELINE REFERENCE
compare them with To better understand the significance of the results in Table 2 and Table 3 baseline random perturbation method in Table 4 on the three image classification settings such that the baseline method applies pixel-wise random perturbation to the images, with the perturbation

---

Published as a conference paper at ICLR 2025
Table Results of the Outlier Attack method. The black-box adversarial attack methods for gener- ating adversarial examples are listed under Attack Method. See Table 2 for more context.
Compensation Share
Fraction Change
Setting
Attack Method
Original
Manipulated
Ratio
More
Tied
Fewer
(a)
ZOO
0.0098
0.0098
0.0631
641.9%
0.980
0.017
0.003
(b)
Simba
0.0250
0.0112
0.0218
194.6%
0.440
0.380
0.180
Simba
0,0098
0.0112
0,0668
596.4%
0.799
0.173
0.028
Simba 0.0098
0.0095
0.0176
185.2%
0.562
0.354
0.084
(c) TextFooler
0.0013
0.0035
0,0092
262.9%
0.392
0.461
0.147
budget matched to that of the proposed attack methods. As shown in the results, the Compensation Share of the dataset with Random Perturbation is nearly identical that of the Original. The fraction under More is also close to that under Fewer. These results highlight the effectiveness of the proposed attack methods comes from the careful design of adversarial perturbation.
Table 4: Results of the random perturbation baseline. See Table for more context.
Compensation Share
Fraction of Change
Setting
Original
Random Perturbation
More
Tied
Fewer
(a)
0.0098
0.0097
0.203
0.586
0.211
(c)
0.0112
0.0117
0.385
0.326
0.289
(d)
0.0095
0.0125
0.367
0.430
0.203
7 CONCLUSION
This work addresses a significant gap in the current understanding of the adversarial robustness of data attribution methods, which are increasingly influential in data valuation and compensation ap- plications. By introducing a well-defined threat model, we have proposed two novel adversarial at- tack strategies, Shadow Attack and Outlier Attack, which are designed to manipulate data attribution and inflate compensation. The Shadow Attack utilizes knowledge of the underlying data distribu- tion. while the Outier Attack operates without such knowledge. relying on black-box queries only. Empirical results from image classification and text generation tasks demonstrate the effectiveness of these attacks, with compensation inflation ranging from 185% to 643%. These findings underscore the need for more robust data attribution methods to guard against adversarial exploitation.
REFERENCES
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopad- hyay. A survey on adversarial attacks and defences. CAAI Transactions on Intelligence Technol- ogy, 6(1):25 45, 2021.
Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input similarity from the neural network perspective. Advances in Neural Information Processing Systems, 32, 2019.
Pin-Ye Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsich. Zoo: Zeroth order optimization based black-box attacks deep neural networks without training substitute mod- els. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AlSec '17. pp. 15-26, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450352024.
R Dennis Cook. Detection of influential observation in linear regression. Technometrics. 19(1): 15-18, 1977.
Junwei Deng and Jiaqi Ma. Computational copyright: Towards a royalty model for ai music gener- ation platforms, arXiv preprint 2rXiv:2312.06644 2023.
10

---

Published as a conference paper at ICLR 2025
Junwei Deng, Ting- Wei Li, Shiyuan Zhang, Shixuan Liu, Yijun Pan, Hao Huang. Xinhe Wang, Pingbang Hu, Xingjian Zhang, and Jiaqi Ma. dattri: A library for efficient data attribution. Advances in Neural Information Processing Systems, 37:136763-13678.1 2024.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881- 2891, 2020.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pp. 2242-223 2251. PMLR, 2019.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceed- ings, 2015.
Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger. Simple black-box adversarial attacks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceed- ings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learming Research, pp. 2484-2493. PMLR, 09-15 Jun 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. Mem- bership inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR), 54 (11s):1-37, 2022.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International conference on machine learning, pp. 2137- 2146. PMLR, 2018.
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, pp- 1167-1176. PMLR, 2019.
Kevin Jiang, Weixin Liang, James Zou, and Yongchan Kwon. Opendataval: a unified benchmark for data valuation. Advances in Neural Information Processing Systems, 36, 2023.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is BERT really robust? A strong baseline for natural language attack on text classification and entailment. In The Thirty-Fourth AAAI Con- ference Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Arti- ficial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8018-8025 AAAI Press, 2020.
Hoang Anh Just, Feiyang Kang, Tianhao Wang, Yi Zeng, Myeongseob Ko, Ming Jin, and Ruoxi Jia. LAVA: data valuation without pre-specified learning algorithms. In The Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Andrej Karpathy. char-rnn. httpp://ggthhbb.cm///aaattttyttt
rnn 2015.
Andrej Karpathy. nano-gpt, 2022.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1885-1894. PMLR, 06-11 Aug 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Mas- fer's thesis, Department of Computer Science, University of Toronto, 2009.
11

---

Published as a conference paper at ICLR 2025
Yongchan Kwon and James Zou. Beta shapley: unified and noise-reduced data valuation frame- work for machine learning. In Gustau Camps- Valls, Francisco J. R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 8780-8802. PMLR, 28-30 Mar 2022.
Yongchan Kwon and James Zou. Data-oob: Out-of-bag estimate as simple and efficient data value. In International Conference on Machine Learning, pp. 18135-1811 PMLR, 2023.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Yann LeCun, The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmist// 1998.
Jinkun Lin, Angi Zhang, Mathias Lécuyer, Jinyang Li, Aurojit Panda, and Siddhartha Sen. Mea- suring the effect of training data on deep learning predictions via randomized experiments. In International Conference on Machine Learning, pp. 13504. PMLR, 2022.
Aleksander Mkadrr, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. stat, 1050(9), 2017.
Elisa Nguyen, Minjoon Seo, and Seong Joon Oh. A bayesian approach analysing training data attribution in deep learning. Advances in Neural Information Processing Systems, 36, 2024.
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Attributing model behavior at scale. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th In- ternational Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 27074-271133 PMLR, 23-29 Jul 2023.
TRAK:
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18, 2017. doi: 10.1109/SP.2017.411
James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusion with c-IoRA. Trans- actions on Machine Learning Research, 2024. ISSN 2835-8856.
Liwei Song and Prateek Mittal. Systematic evaluation privacy risks of machine learning models.
In 30th USENIX Security Symposium (USENIX Security 21), pp. 2615-2632, 2021.
G W Stewart and Ji-Guang Sun. Matrix Perturbation Theory. Computer Science and Scientific Computing. Academic Press, June 1990. ISBN 9780080926131.
Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statisties, pp. 6388 6421. PMLR, 2023.
Jiachen T. Wang, Zhun Deng, Hiroaki Chiba-Okabe Boaz Barak, and Weijie J. Su. An economic solution to copyright challenges of generative ai, 2024.
Xinyi Xu, Zhaoxuan Wu, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Validation free and repli- cation robust nolume-based data valuation. Advances in Neural Information Processing Systems, 34:10837-10848, 2021.
Jinsung Yoon, Serean Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference on Machine Learning, pp. 10842-10851. PMLR, 2020.
Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems, (0K9):2805-822) 2019.
Yang Zhang, Fuli Feng, Chenxu Wang, Xiangnan He, Meng Wang, Yan Li, and Yongdong Zhang- How to retrain recommender system? sequential meta-learning method. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1479-1488, 2020.
12

---

Published as a conference paper at ICLR 2025
Zihan Zhang, Meng Fang, Ling Chen, Kohammad-Rezz Namazi-Rad, and Jun Wang. How do large language models capture the ever-changing world knowledge? a review of recent advances. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 8289-8311 Singapore, December 2023. Association for Computational Linguisties.
13

---

Published as a conference paper at ICLR 2025
A FORMAL STATEMENT OF THEOREM 5.1 AND ITS PROOF
A.1 SETUP AND THE STATEMENT
Let the original training set be Z {=Ni=. Given a data point 2, assume the loss ((0,2) 2) twice differentiable. Then, the empirical loss and the Hessian is given by
1 (())
n1 Σ (0,z), (1)0)
n Σ
h(0, 2;);
where we define 1(002)) VjE(O, z). Moreover, we consider the ERM to be
i arg min ((0,z))
n2
Consider the training data attribution to be the influence estimated by the influence function (Koh & Liang 2017) TT.. particular, for a train test pair (Zi, 2rest), Eq. (4) can be succinctly written as
(2 Resa; 0) v vol(0.2) 2;) H (0) Vel z
Consider perturbing the 1th training point such that becomes z/.- Then, the training set turns to {Z 2j..... }. which turn changes the empirical loss and its hessian to
(0)) Σ ((0,,,) + ((0 z/), H'(0) i73
Σ 1(0, + m0, z1).
n
n K
It also follows that the minimizer for this perturbed training set is given by
0' = arg min E(0022)) 1 ((0
1
0 n
with the corresponding influence score being
TIF (Zi, Zresti 0') = V vē Fress) ( ((H =i) for all i of j; and
(2 Zrest; 0') - 0f( Ziea)) T
=/))-
Remark 1. From our definition of data altribution methods, т should take the form of 7 V - IR for some training dataset Z and validation dataset V. In the above notation, we are essentially considering two different training attribution functions TIF and Tip where TIF A(Z, (Z),, V) and T/r A(Z', T(Z'), V) where Z' the perturbed dataset, TZZ) and I' T(Z).. For clarity, we explicitly write TIP(-.-) = :)((;;;0) and ( TiPl- -3) 0') to avoid confusion.
Under this setup, we can now state the theorem formally.
Theorem A.1. Under the above setup, consider model trained by ERM with loss & that is owice wiice-differentiable, m-strongly convex, and L-Lipschitz continuous with respect to Assume its corresponding influence score TIF, gradient Vol(f, 2), and h(0,2) are all bounded, i.e., (Tirl (11), ol = (1), [ihllup = (1). Then, given any test data point Ztea. we have
Tir(zi, Ziest 0¹) TiFl (zi, Zeest; 0) (1//n) for all + j. and
* TTr(s/ Piem; 0¹) TIF( R 0) O(1/n)
A.2 TECHNICAL LEMMAS
We first establish several technical lemmas toward proving Theorem A.I
Lemma 1. Let 0, 0' be the minimizer for ((0),e(()) respectively, then
4L
||0
mn
14

---

Published as a conference paper at ICLR 2025
Proof. Firstly, recall that from definition, f() being m-strongly convex that for any 2, y,
f(u) 2 f(z) + Vf(=) - 2) + m ly - - 2
2
Denote x² = arg min, f(x;,, then since Vf(x*) = 0, for all x, we have
f(x) 2 f fx + m | x x=122-
2
Hence, by strong convexity of E(0, 2) and the fact that is the minimizer of É(0), we have
8(0', 21)
1
+
m m 0 0'12-
2
1=1
On the other hand, we have
100' 21)=
1
Σ
100' (00)2))=
-l(0' .21) + --(1-(((;, =((0=21)
1
((0")
1
1
1(0, zi) + =
=((0, 24) +- (e(0', -e(0', (0' is a minimizer of E
Th
E(())
1
1 (e(0), 22)) - -e(, 2,)) + - (e(0,2)) E((',
8(0,2) 2,)
n
2 11l 8' 8112.
((0, =:) +
(e(0, 2) is I--Lipchitz w.r.t. 0)
Combine the above results, we see that
m 010 2
11
1/
<
2L
110
8'112.
which proves the desired result.
Lemma 2 (Section 2.4, Part III Stewart & Sun 1990)) ). For any A, E Idddd with both and
0o A + E' being invertible, if -k E op converges, we have
1)+ - A 0(1)))
Lemma 3. Let H and H' be the Hessian of and 8, respectively, then
4LM m
|H'-1 llop So
Proof. Since =(0, is M-Lipschitz w.r.t. 0, know that
4LM
|H' Hllop < MIL0 0112 -
With Lemma 2 (let A H and E. = H' - H, it's easy to verify the conditions of Lemma 2 hold), we have HI -1 11 (H'' - H) 011n22), which gives
& || ||op!l'
olop
llop + 0(1/n2).
Since E(0, 2) is m--ttrongly convex w.r.t. 0, it's easy to show that || S 1/m, In all, we have
4LM
||| 1 1l-llopp 0
nnn
15

---

Published as a conference paper at ICLR 2025
A.3 PROOF OF THEOREM A.1
We can now prove Theorem A.I
Proof of Theorem A.7 For all * j. we want to prove that 7)(22, Prstt 07) = ( Frests 0) +
O(1/n). Indeed, since
TIF Ri, Prest; 0¹) = -oo((0 =ea) (H'((()) Voe(O', 21)
(Vol(e, Zues) - Voe(6', 2)a))) (H' (6') -V ( (o)((,2)
Vol(O, 2nea) ((H' (0') 1 (0)) Vol(O,
Vel(O, 7rstt 11 (0)Vo00 2i)
= (Vol(0, 2res) - Vnl(6', ()e))) (H) - 1V (l(0,2)
H (0) (7
Vol(O, Zresa)
+ TTr z 0).
Applying Lemmas I in terms of 0, hence have the desired result. Specifically, have
and 3.
and by noting that when the Hessian is M-Lipschitz, so is the gradient
TTP Zwst 0') TIF Fusti 0)
S V oe( Ziea) Vol(O', 2ress 1|2 op
Vo((0, 21)12
+ Voe 21en))12 | H op Vo 2; x)
$ V (( 2, 2,)112 (1 1lop
||'
H llop)
M| 0112
+ VoL 212a))122 11
H
1lop V 24 21
(== +
4LM
4L
4LM m
0(1 = (
= +(1
+ O())
mnn
mm
For the second case, the proof is the same by replacing z with z/J in the above calculation.
B EXPERIMENT DETAILS
In this section, we introduce the experiment details.
B. DATASETS AND MODELS
Logistic Regression and Convolutional Neural Network on MNIST. For experiments on MNIST, we consider two different target models, Logistic Regression (LR) and convolutional neu- ral network (CNN). The CNN comprises two convolutional layers: the first with 32 filters and the second with 64 filters (size 3 x 3, stride 1, padding 1), both followed by ReLU and max-pooling layers. The output is flattened into vector of size 64 x 7 7, which then passed through fully connected layer with 128 units before reaching the output layer with 10 classes. Both target models are trained on the first 10000 training points of the MNIST dataset. For shadow models, we train 50 of them, each randomly sampled subset of 5000 among the second 10000 training data points to ensure no overlaps with the first 10000 training data points used for target model training.
In both the target model training and shadow model training, we train LR for 30 epochs with SGD and a learning rate 0.01. We train CNN for 50 epuchs with Adam and a learning rate of 0.001.
MLP on Digits. For experiments on Digits, we consider the target MLP model with hidden layers, each has 10 hidden neurons. Due to the size limit of the Digits dataset, train the target model for an Outlier Attack using the first 1100 training points, while for the shadow attack, we train the target model the first 800 data points and the shadow models the second 800 data points. In all training, we again train the MLP for 30 epochs using Adam with an initial learning rate of 0.001.
16

---

Published as a conference paper at ICLR 2025
ResNet18 CIFAR-10. For experiments CIFAR-I0, consider the classical ResNet18 model without dropouts. We use 10000 training points for training the target model. For shadow model training, consider the classical ResNet-9 and ResNet- 18 as the shadow models separately, again without dropouts. We train 50 shadow models, and each on a subset of 10000 training points that are disjoint with the training set of the target model. In all training, each model is trained for 100 epochs using Adam, with learning rate of 0.001.
Na on Shakespeare. For experiments on the Shakespeare dataset, we consider the NanoGPT model, which is a character-level GPT model with 4 layers, heads, and 128 dimen- sion of the embedding. The block size is 64, and the batch size is 32. For both training at = 0 and I = 1, we train the model for 2000 epochs, both using Adam with learning rate of 6 x 10-4, Note that since we only consider Outlier Attack for this setup, hence there is no shadow model training involved.
B.2 TRAINING DATA ATTRIBUTION METHODS
In this section, we briefly introduce the attribution methods that are included in the evaluation. Given a training dataset { & 1==1 we are interested in the data attribution of a particular training data point 2j and a test data point Feest-
Influence Function based on the Conjugate Gradients. As we have seen, the original definition of inftuence function (Koh & Liang 2017) is given by Eq. 4 i...
Tip [2) Zust 0) Vol(O, 23) H()) Vol( Fess)
where (o =) is the gradient of loss of the data point w.r.t. model parameters, and (0) the inverse of the Hessian w.r.t. model parameters. We implement the conjugate gradients (CG) approach to compute the
weatin-vecoor--rooooe product (IHVP).
Tracing with the handomly-projeetee After Kernel (TRAK). Introduced by Park et al. (0033), TRAK computes the attribution score for training test pair by first linearizing the model and then applying random projection to make the computation efficient, Denote the output (e.g., raw logit) of the model & to be f(z; 0), then the naive TRAK influence can be formulated as
TTRAK (Zg, Fresti 0) = (1))(())))
TT) ((tess))
where 6(2) = PTV 0) is the random projection of V V 12 0) E R2 by some Gaussian random projection matrix P (0, 1) 1)p*k, and + is the matrix formed by stacking all the d(z;), and p] the predicted correct-class probability of 2y ô. Compare it with TIF:
1. The additional factor p in TIRAK is due to linearizing the model.
2. For linear model with feature matrix of the training set, its Hessian is exactly X for linearized the model, each feature x; of training sample 2, corresponds to the gradient Vof f2,; 2,:00 0), i.e., X oof (2;; 0)]
T X. For
3. With random projection, the linearized features Ty Vof(z;00 becomes p(2,) PVa[(z;; 0), which induces a new feature matrix + = [d]=;)]
Hence, overall, TTRAK is nothing but the influence function applied to the linearized model with ran- dom projection. We note that the original TRAK influence includes step called ensembling, which is simply averaging the above TRAK influence over multiple TTRAK with models independently trained on a subset of the training set.
Grad-Dot. Introduced by Charpiat et al. 2019), the dot product of gradient, known as Grad- is simple, casy-to-compute training data attribution method. It is given by
Gor (2jj Zrese 0) vol(O,z,) 2,) Vol(0 Ziea).
17

---

Published as a conference paper at ICLR 2025
Data Shapley. Data Shapley (Ghorbani & Zou 2019) quantifies the influence of individual data points by considering its marginal contribution to different subsets of the training set. It is motivated by the so-called leave-one-out (LOO) influence: specifically, given any aspect x(0) we care about for the learned model 0 (e.g.. the loss E(O, Ziess) of some test data point ),s,),, LOO measures the influence of every individual data point 2j Φ by the difference <(f==,) o(0), where 0_ is the model learned with the dataset { that excludes 2j. Data Shapley builds on top of LOo by requiring additional equitable conditions, resulting in the following formulation
Φ(0s, Aea) - +((0ub)) 2
-hatnShappey (Zgz Prest 0)
(1ss
where we let @(0, 2test) = E(O, 2rest) and @g for some S [= {zy} denotes the model learned on the subset S. This can be viewed an averaged version of LOO while satisfying several equitable conditions, which we refer to Ghorbani & Zou (009)) for details.
Remark We note that in the above formulations, we incorporate a negative sign in front of TIRAK, TGrd-Doot to make them consistent to the original formulation of TIF. Specifically, in Koh & Liang) (2017), the influence change is measured as the difference between the perturbed model and the original model, while others consider the opposite.
B.3 ATTACK METHODS
In this section, we detail the implementation of all the attack methods we have used throughout the experiments.
Shadow Attack. Shadow attack is a classical adversarial attack method that originates from (Shokri al., 2017). The details of shadow model training can be found in Appendix [B.]] As introduced in Section 4 after 50 shadow models are trained, we optimize Eq. (3) using gradient
ascent with respect to imput feature x with 10 iterations and step size of 0.01.
Outlier Attack. For Outlier Attack, we utilize the following black-box attack methods to produce adversarial examples:
Zeroth Order Optimization (ZOO): ZOO is a black-box attack method that approximates gradient through finite numerical methods. In our experiment, given an input = (z. y), we perturb it by x' <- x + e sgn( =)), where g(0, 2) is estimation of the loss of the gradient with respect to æ. given by the symmetric difference quotient
(o(0, 2)), E(O, (a + he, v)) e(O,, (x he, y))
2h
along the hh standard basis direction with h E R. We note that E(0, (z, 8)) can be obtained by black-box queries of the target model. In experiments, we set 0.03.
Simba: Introduced by Guo al. (2019), Simba proposes black-box attack method to perturb each input pixel sequentially after permutation. More precisely, each input pixel is attempted to be perturbed in both directions, i.e. (I'); + 2; e, respectively. After perturbation is attempted, the target model is queried again, and the perturbation is accepted as long as this perturbation increases the loss. In practice, we
0.1.
TextFooler originally an attack method for text classification task (Jin al., 2020). It perturbs texts following a two-step approach: for a piece of text that it wants to perturb, it first sorts the words by their importance, and then replaces the influential words to increase the loss of prediction while preserving text similarity.
TextFooler:
We modify the method as follows. Firstly, for character-level GPT model (in our case, NanoGPT), we work on characters rather than words. Secondly, as we work on text gen- eration task, for sequence of characters that we want perturb, we take the sum of the negative log- ligeehooo of predicting its m future characters as a loss. We then sort the characters by their importance, where importance is measured by the loss increase after the character is masked. Finally, the k-most important characters are replaced by characters that maximize the increment of the loss. In experiments set m 20 and k: = 15.
18

---

Published as a conference paper at ICLR 2025
C ADDITIONAL EXPERIMENTS
In this section, we conduct additional experiments under various settings, including ablation studies and more.
C.I ABLATION STUDY: CHANGING VALUE OF Zol AND |Zo!
We test the performance of our methods under different sizes of [Zo] and |Z to understand the effect of data set sizes. For settings (a), (c), and (d) Table we consider decreasing ()Zol, (1]) from (10000, 11000) (5000, 6000) and increasing (lZ Z1 from (10000, 11000) to (15000, 16000), experimenting with both the Shadow Attack and the Outlier Attack. For the setting (e), i.e. the text generation setup, the original |Zo] was already small (originally, (lZ [2 which takes only 30% of the data. Hence, we consider increasing the data size to two different extents. Note that since the Shadow Attack is infeasible in this setting, only the Outlier Attack's results are shown for setting (2). The results for the two attack methods are respectively shown in Tables and 6 We see that the results demonstrate the effectiveness of our methods across different
(4706, 6274)),
sizes of the dataset. Overall, with a few exceptions, the Ratio of the Compensation Share appears become even higher when the dataset size larger.
Table 5: Results of Shadow Attack for various (1Zo),|Z1) settings.
Compensation Share
Fraction Change
Setting
(IZo), lZ.)
Original
Manipulated
Ratio
More
Tied
Fewer
(5000, 6000)
0.0153
0.0764
499.3%
0.996
0.003
0.001
(a)
(10000, 11000)
0.0098
0.0477
456.1%
0.955
0.038
0.007
(15000, 16000)
0.0050
0.0373
746.0%
0.963
0.036
0.001
(5000, 6000)
0.0168
0.0539
320.7%
0.857
0.129
0.014
(c)
(10000, 11000)
0.0112
0.0467
417.0%
0.781
0.195
0.024
(15000, 16000)
0.0002
0.0062
3100.0%
0.431
0.568
0.001
(5000, 6000)
0.0206
0.0413
200.7%
0.696
0.174
0.130
(d)
(10000, 11000)
0.0095
0.0213
217.3%
0.655
0.259
0.086
(15000, 16000)
0.0057
0.0092
161.5%
0.264
0.616
0.120
Table 6: Results of Outlier Attack for various (lZo), /1)
settings.
Compensation Share
Fraction of Change
Setting
(lZo). I1
Original
Manipulated
Ratio
More
Tied
Fewer
(5000, 6000)
0.0153
0.0747
488.9%
0.998
0.002
0.000
(10000, 11000)
0.0098
0.0631
643.9%
0.980
0.017
0.003
(15000, 16000)
0.0050
0.0400
800.0%
0.964
0.036
0.000
(5000, 6000)
0.0168
0.0334
198.8%
0.720
0.241
0.039
(c)
(10000, 11000)
0.0112
0.0668
596.4%
0.799
0.173
0.028
(15000, 16000)
0.0002
0.0051
2550.0%
0.397
0.603
0.000
(5000, 6000)
0.0206
0.0411
199.5%
0.761
0.133
0.106
(d)
(10000, 11000)
0.0095
0.0176
185.2%
0.562
0.354
0.084
(15000, 16000)
0.0057
0.0219
384.2%
0.731
0.192
0.077
(4706, 6274)
0.0031
0.0035
262.9%
0.392
0.461
0.147
(c)
(7843, 9411)
0.0016
0.0064
400.0%
0.420
0.507
0.073
(12549, 14116)
0.0029
0.0214
737.9%
0.400
0.543
0.057
C.2 ABLATION STUDY: DATA ATTRIBUTION METHOD
Next, we conduct an ablation study by varying the data attribution methods used in the evaluation under the experiment setting (b) in Table [ Specifically, apart from Data Shapley, consider
19

---

Published as a conference paper at ICLR 2025
two additional data attribution methods, Influence Function, and TRAK, for evaluating the com- pensation share, experimenting with both the Shadow Attack and the Outlier Attack. Note that the Adversary has knowledge about what data attribution method will be used by the AI Developer The results are shown in Tables 7 and g The results below show that the proposed attack methods are highly effective for all three data attribution methods, reflected by the Ratio of the Compensation Share.
Table 7: Results of Shadow Attack with different data attribution methods of setting (b).
Compensation Share
Fraction of Change
Attribution Method
Z
Original
Manipulated
Ratio
More
Tied
Fewer
Data Shapley
0.0352
0.0152
0.0435
286.2%
0.533
0.333
0.134
Influence Function
0.0352
0.0496
0.1004
202.4%
0.980
0,000
0.020
TRAK
0.0352
0.0392
0.0936
238.8%
0.820
0.100
0.080
Table 8: Results of Outlier Attack with different data attribution methods setting (b).
Compensation Share
Fraction of Change
Attribution Method
/
Original
Manipulated
Ratio
More
Tied
Fewer
Data Shupley 0.0250 0.0250
0.0112
0.0218
194.6%
0.440
0.380
0.180
Influence Function
0.0186
0.0442
237.6%
0.920
0.060
0.020
TRAK
0.0250
0.0160
0.0412
257.5%
0.780
0.180
0.040
C.3 WIITTE-BOX ATTACK
In this section, we further consider white-box attacks as oracle reference, where the Adversary has full knowledge the target model and has access to the model parameters. For the choice of attack method under the white-box threat model, we experiment with the Fast Gradient Sign Method (FGSM) (Goodfellow et al. 2015) for setting (a) and the Projected Gradient Descent(PGD) (Mkadry ct al. 2017) for setting (d). The results shown in Table Overall, compared to this oracle white- box attack, the proposed black-box attacks are only slightly worse in terms of the Ratio of the Compensation Share. This further confirms that the proposed methods are highly effective.
Table Results of white-box attacks under settings (a) and (d).
Compensation Share
Fraction of Change
Setting
Attack Method
Original
Manipulated
Ratio
More
Tied
Fewer
(a)
FGSM PGD
0.0098 0.0095
0.0649 0.0222
662.2%
0.990
0.007
0.003
(d)
233.7%
0.689
0.204
0.107
C.4 ADVERSARIAL ATTACK UNDER DATA AUGMENTATION
In this section, we test the performance of our proposed attack methods when the AI Developer utilize data augmentation techniques when training the model. Specifically, after the AI Developer gathers data, the training dataset is then formed by a combination of the gathered data and its augmented version. For simplicity, we consider settings (a), (c), and (d) in Table and consider standard image augmentation methods such as random cropping and random affine trans- formation to images. With data augmentation, the compensation share will be attributed back to the original data point if its augmented version is identified as influential. The results are shown in Tables 10 and II Compared to the original setting without data augmentation, we cannot draw a definite conclusion
whether data augmentation helps defend the proposed attacks since the trend unclear. However, overall, we can conclude that the proposed attacks are still highly effective even when data augmentation utilized by the AT Developer.
20

---

Published as a conference paper at ICLR 2025
Table 10: Results of Shadow Attack under data augmentation
Compensation Share
Fraction of Change
Setting
Z
Original
Manipulated
Ratio
More
Tied
Fewer
a
0.0098
0.0099
0.0209
211.1%
0.632
0.244
0.124
(c)
0.0098
0.0084
0.0516
614.3%
0.894
0.095
0.011
(d)
0.0098
0.0091
0.0747
820.9%
0.959
0.035
0.006
Table 11: Results of Outlier Attack under data augmentation.
Compensation Share
Fraction of Change
Setting
Z
Original
Manipulated
Ratio
More
Tied
Fewer
(a)
0.0098
0.0099
0.0287
289.9%
0.776
0.158
0.066
0.0098
0.0084
0.0691
822.6%
0.853
0.113
0.034
(d)
0.0098
0.0091
0.0481
528.6%
0.916
0.065
0.019
C.5 COMPENSATING ONLY CORRECT PREDICTION
In this section we consider the setup when the AI Developer only rewards training data points that are highly influential to validation samples with correct predictions. In detail, after training the model and before calculating of compensation share, the AI developed will test the model on private validation set, select those validation points that correctly predicted, and only attribute compensation to influential training points for these validation points. We test this on settings (a), (c), and (d). Note that the Fraction of Change metric no longer feasible in this setup since the model trained on the original and manipulated training set may not share the same validation set. The results are shown in Table [2] Overall, our proposed attacks are still highly effective based on the Ratio of Compensation Share.
Table 12: Results when compensating only correctly predicted data points.
Compensation Share
Attack Type
Setting
Original
Manipulated
Ratio
(a)
0.0098
0.0050
0.0382
764.0%
Shadow Attack
(c)
0.0098
0.0090
0.0752
835.6%
0.0098 (d)
0.0110
0.0353
320.9%
(a)
0.0098
0.0050
0.0273
546.0%
Outlier Attack
(c) 0.0098
0.0090
0.0552
613.3%
0.0098
0.0110
0.0227
206.4%
C.6 ADVERSARIAL ATTACK ON LARGE-SCARN SCALE SETUP
In this section, we scale up our experiment and see how the proposed attacks generalize in this large-scale experiment. Specifically, we consider the ResNet-18 model (He et al. (016)) the Tiny ImageNet dataset (Le & Yang. 2015) with ZZo, Z = (50000, 60000), and we perturb | = 100 training points. Both the Shadow Attack and Outlier Attack are tested, and the results are shown in Table [33 It is evident that from the Ratio of Compensation Share, two proposed attacks are still highly effective in large-scale experiment.
21

---

Published as a conference paper at ICLR 2025
Table 13: Results Shadow Attack and Outlier Attack under large-scale setup.
Compensation Share
Fraction of Change
Attack Type
2
Original
Manipulated
Ratio
More
Tied
Fewer
Outlier Attack
0.0017 0.0017
0.0009
0.0056 0.0181
622.2%
0.456
0.498 0.131
0.046
Shadow Attack
0.0009
2011.1%
0.855
0.014
D VISUALIZATION OF THE PERTURBATION
Finally, Figure 3 provides visualizations of two examples of the adversarial perturbed images from MNIST and CIFAR-IO. In both cases, the perturbations are barely visible to human eyes, however, they highly effective in terms of the success of the attack.
Original. Influential for 0 vali- dation data points.
(a)
(b) Shadow Attack. Influential for 75 vali- dation data points.
(c) Outlier Attack. Influential for 105 validation data points.
(d) Original. Influential for 1 vali- dation data points.
(e) Shadow Attack. Influential for 38 vali- dation data points.
(I) Outlier Attack. Influential for 29 vali- dation data points.
Figure 3: Visualization of MNIST (Top) and CIFAR-10 (Bottom), before and after attacks.
22


Reviews:
[{"review_id": "mtdWyMBc8i", "reviewer": "Reviewer_pQy1", "summary": "This paper introduces a novel adversarial attack on data attributions, where the adversary aims to maximize the compensation share received by the adversary. The authors set a few restrictions and assumptions on the adversary's capabilities and the target model. Two different attacks are proposed based on different choices of assumptions. The shadow attack is based on the assumption that the adversary has access to the distribution of the training data and the validation data, while the outlier attack is based on the assumption that the adversarial can query the model to get the data attribution values. The experimental results demonstrate that both the shadow attack and the outlier attack are highly effective in manipulating data attribution methods, leading to substantial increases in the attacker's compensation share.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized and easy to follow. The authors clearly define the adversary's objective, capabilities, and assumptions, which helps readers understand the problem setting.\n2. The proposed attacks are novel. The shadow attack and the outlier attack are based on certain scenarios and are effective in manipulating data attribution methods.\n3. The experimental results are comprehensive. The authors evaluate the proposed attacks on multiple datasets and data attribution methods, demonstrating the effectiveness of the attacks in practice.", "weaknesses": "1. The surrogate objective used to replace the discrete compensation share is not well justified. The authors should provide more insights into why the surrogate objective is a good approximation of the compensation share. Is the surrogate itself already a reasonable to represent the compensation share?\n2. The paper emphasizes the validity of the assumptions using the examples of large language models, while the experiments are mostly conducted on relatively smaller size datasets (*e.g.*, MNIST and CIFAR-10). It would be more convincing if the authors could provide more scalability analysis on larger datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G2aK3sAPa0", "reviewer": "Reviewer_T7Fd", "summary": "This paper studies adversarial attacks to data attribution based compensation mechanism.\n\nTwo empirical attacks are proposed.\n1. Shadow attack, which assumes access to data distribution & training algorithm, creates a surrogate data attribution system and adversarially optimize the compensations with the surrogate.\n2. Outlier attack, which assumes access to the black-box query of the model and directly optimizes to minimize the confidence of the model to generate outlier data, which could correspond to higher compensation.", "questions": "See Weaknesses for my primary concerns, which is in fact somewhat critical to make decisions regarding the impact of the observations.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is overall clear and easy to follow. \n\nSince the adversarial robustness of data attribution is a reasonable subject, the significance is not considered an issue.", "weaknesses": "The experimental objective function of the adversarial attack to data attribution requires much better justifications.\n\nThe paper motivates the study of the robustness of data attribution with data attribution based compensation.\nHowever, it is natural that such compensation designs reward only for data contributing to good/desired behaviors: For example, they might allocate compensations only based on attributions for **correct** predictions (these could be from a prepared validation/test set or from user feedbacks). Alternatively, they might withdraw compensations or even incur penalties for **incorrect/undesired** predictions.\n\nA concern is that in such, arguably more reasonable settings, neither of the proposed attacks will be in fact effective. For shadow attack, only data attributed greatly for correct predictions will be awarded, which is totally fine as the \"attack\" is in some sense actually making the data more valuable, with efforts. For outlier attacks, the outliers are less likely to be attributed if the prediction is correct and consistent with regular training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E4MBOmzSTx", "reviewer": "Reviewer_4aYY", "summary": "This paper introduces two data attribution attack methods, Shadow Attack and Outlier Attack, that aim to inject new training data that maximizes the influence of target model outputs. Both attacks optimize over a compensation share function, which quantifies the influence of a subset of the training data to the target models predictions on test data. Shadow Attack assumes the adversary has access to data distributions and the training algorithm, and uses gradient ascent on training data to generate data points. Outlier Attack assumes black-box query access to the target model, and uses black-box image attack algorithms on training data to generate data points. The attacks are evaluated on image classification and text generation tasks using metrics that measure the change in compensation share before and after adversarial data is introduced.", "questions": "Thanks to the authors for the submission to ICLR 2025. I enjoyed reading about the compensation function used for the adversarial objective in the threat model section. Most of my suggestions are already mentioned in the above Weakness section. Some questions (might be open-ended) are listed below. It would be great if more discussion or clarification on the following questions could be added to the paper:\n- How is distortion budget measured for both image classification and text generation tasks? What norm is used for images and is the budget consistent with other white-box and black-box attack baselines?\n- Analyzing how data attribution methods affect attack performance would strengthen the study of the attacks. For example, if one were to experiment with the Influence Function, Data Shapley, and TRAK methods for a particular {task, dataset, target model} configuration, how do the attacks perform?\n- If we increase or decrease the size of Z_0 and Z_1 for each setting, would this make the attack more successful (i.e., how does performance change with respect to the amount of data)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Thank you for submitting to ICLR 2025. I enjoyed reading about the compensation function used for the adversarial objective in the threat model section. I have listed several points that I find the paper do well below:\n- The threat model is well defined for both attack methods. Assumptions made of target model and training/testing data access is clearly stated. The periodic data contribution attack setting between AI Developers and Data Providers is realistic. \n- The adversarial objective is very clear and the compensation function defined in the paper captures it in a novel way. Additionally, the Shadow Attack and Outlier Attack methodology is designed within the limits of the threat model.\n- Baseline evaluation helps articulate the performance of the attack methods. Since the compensation share metric is new, the baseline evaluation puts into context how performant the attack is.", "weaknesses": "The problem discussed is very interesting and deserves more attention from researchers and developers from both academia and industry. It is worth noting the effort made by the authors in designing novel attack methodologies. However, I believe there is still room for improvement, especially in making claims (e.g., attack effectiveness and efficiency) more concrete and clear and including more baseline evaluations that shed light on the performance of proposed attack under the given threat model. More details can be found below.\n- The paper lacks well-defined research questions in the evaluation. It can be inferred that sections on Shadow Attack and Outlier Attack measure effectiveness, but it would improve the paper if there were claims made that the result would prove/disprove.\n- Another baseline metric would be great to understand the performance of the attack under the defined threat model. While the random perturbation baseline helps contextualize the compensation share ratio of the attacks, another baseline assuming a white-box threat model (i.e., the adversary has access to the target model and its parameters) would highlight the performance under the defined threat model with more limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HHeBPWmoz8", "reviewer": "Reviewer_wvst", "summary": "Data attribution quantifies the contribution of individual data points during model training. Higher contribution scores indicate that a data point has more value within the training dataset. This paper proposes two adversarial methods—Shadow Attack and Outlier Attack—designed to manipulate data attribution. The authors evaluate these methods against various models and datasets, demonstrating that they can significantly inflate the contribution scores of otherwise benign data points.", "questions": "See above", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Strengths:**\n\n- **Addresses Critical Gap in Research**: A key contribution of this work is that it addresses a crucial research gap. The robustness of data attribution methods has remained largely unexplored. This study is the first to systematically investigate vulnerabilities in data attribution.\n\n- **Significant Attribution Inflation**: The authors introduce two effective attacks that can substantially increase the contribution scores of input samples. The reviewer was impressed by the triple-digit inflation rates achieved, showing the potential impact of these adversarial techniques.", "weaknesses": "1. **Threat Model**: The authors describe a black-box threat model, which traditionally implies that the attacker has no information about the model other than query access. However, the authors assume access to the model’s architecture, indicating a grey-box rather than a black-box scenario. It would be beneficial for the authors to revise the text to accurately reflect this nuanced threat model.\n\n2. **Alternative to Creating Outliers**: The paper presents two algorithms designed to create natural-looking outliers with a high impact on the model. Given the complexity of these algorithms, a simpler approach might involve manually mislabeling inputs to create outliers (e.g., labeling a \"cat\" as a \"dog\" in a classification task), which would likely produce a similar exaggerated impact on the model. The authors should consider addressing why this more straightforward and cost-effective method was not explored.\n\n3. **Simplistic Pipeline**: The authors assume a straightforward threat model where the defender does not employ methods to detect out-of-distribution (OOD) samples or sanitize inputs. Adversarial samples, such as those generated by the Shadow Attack, are known to be vulnerable to detection and degradation [1]. Testing the attacks against state-of-the-art OOD detection and adversarial input detection methods would help establish their robustness.\n\n4. **Simple Defense**: A basic yet effective defense against these attacks would be data augmentation, such as cropping or shifting images to generate additional samples. This method improves model generalization, a common practice in machine learning, and could reduce the impact of adversarial samples like those introduced by the authors. The authors should demonstrate that including data augmentation would not impact their attack, as it more accurately reflects real-world training environments.\n\n[1] https://arxiv.org/abs/1902.02918", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)