Influence-based Attributions can be Manipulated

## Abstract

Abstract Influence Functions are a standard tool for attributing predictions to training data in a principled manner and are widely used in applications such as data valuation and fairness. In this work, we present realistic incentives to manipulate influence-based attributions and investigate whether these attributions can be systematically tampered by an adversary. We show that this is indeed possible for logistic regression models trained on ResNet feature embeddings and standard tabular fairness datasets and provide efficient attacks with backward-friendly implementations. Our work raises questions on the reliability of influence-based attributions in adversarial circumstances. Code is available at : https://github.com/infinite-pursuits/influence-based-attributions-can-be-manipulated

## 1 Introduction

Influence Functions are a popular tool for data attribution and have been widely used in many applications such as data valuation [, , , ], data filtering/subsampling/cleaning [, , , , ], fairness [, , , , , , , , ] and so on. While earlier they were being used for benign debugging, many of these newer applications involve adversarial scenarios where participants have an incentive to manipulate influence scores; for example, in data valuation a higher monetary sum is given to samples with a higher influence score and since good data is hard to collect, there is an incentive to superficially raise influence scores for existing data. Thus, an understanding of whether and how influence functions can be manipulated is essential to determine their proper usage and for putting guardrails in place. While a lot of work in the literature has studied manipulation of feature-based attributions [, , ], whether data attribution methods, specifically influence functions, can be manipulated has not been explored. To this end, our paper investigates the question and shows that it is indeed possible to systematically manipulate influence-based attributions according to the manipulator’s incentives.

Simply put, we show that it is possible to systematically train a malicious model very similar to the honest model in test accuracy but has desired influence scores. To formalize the setup we divide the function pipeline in terms of two entities – Data Provider who provides training data and Influence Calculator who trains a model on this data and finds the influence of each training sample on model predictions. Out of these, Influence Calculator is considered to be the adversary who wishes to change the influence scores for some training samples and does so covertly by training a malicious model which is indistinguishable from the original model in terms of test accuracy but leads to desired influence scores. This setting captures two important downstream applications where incentives are meaningful: data valuation, where the adversary has an incentive to raise influence scores for monetary gain and fairness, where the adversary wants to manipulate influence scores for reducing the fairness of a downstream model.

We next define and provide algorithms to carry out two kinds of attacks in this setup: Targeted and Untargeted. Targeted attacks are for the data valuation application and specifically manipulate influence scores for certain target samples. The primary challenge with these attacks is that calculating gradients of influence-based loss objectives is highly computationally infeasible. We address this challenge by proposing a memory-time efficient and backward-friendly algorithm to compute the gradients while using existing PyTorch machinery for implementation.
This contribution is of independent technical interest, as the literature has only focused on making forward computation of influence functions feasible, while we study techniques to make the backward pass viable. Our algorithm brings down the memory required for one forward $+$ backward pass from not being feasible to run on a 12GB GPU to 7GB for a 206K parameter model and from 8GB to 1.7GB for a 5K model.

Experiments on multiclass logistic regression models trained on ResNet50 features show that our targeted attacks achieve a high success rate, a maximum of 94%, without much accuracy drop across three datasets. One final question that comes to mind is – is it always possible to manipulate the influence scores for any given training sample? Using a theoretical construction, we give an impossibility theorem which states that there exist samples for which the influence score cannot be manipulated irrespective of the model, making this a property of the data rather than the model.

Untargeted attacks are for the fairness application and unlike targeted attacks, manipulate influence scores arbitrarily without targeting specific samples. We find that surprisingly enough scaling model weights is a good enough strategy for such attacks without changing model accuracy. In our experiments on standard tabular fairness datasets, we observe that due to influence score manipulation fairness of downstream models is affected a lot, leading to a maximum of 16% difference in fairness metric with and without influence manipulation.

Summarizing, we formalize a setup for systematically manipulating influence-based attributions and instantiate it for data valuation and fairness use-cases, where adversarial incentives are involved. We provide algorithms for targeted and untargeted attacks on influence scores, and illustrate their efficacy experimentally. Our work exposes the susceptibility of influence-based attributions to manipulation and highlights the need for careful consideration when using them in adversarial contexts. This is akin to what has been previously observed for feature attributions [].

## 2 Preliminaries

Consider a classification task with an input space $\mathcal{X}=\mathbb{R}^{d}$ and labels in set $\mathcal{Y}$.
Let the training set of size $n$ be denoted by $Z_{\rm{train}}=\left\{z_{i}\right\}_{i=1}^{n}$ where each sample $z_{i}$ is an input-label pair, $z_{i}=(x_{i},y_{i})\in\mathcal{X}\times\mathcal{Y}$. Let the loss function at a particular sample $z$ and model parameters $\theta\in\Theta$ be denoted by $L(z,\theta)$. Using the loss function and the training set, a model parameterized by $\theta\in\Theta$ is learnt through empirical risk minimization, resulting in the optimal parameters $\theta^{\star}:=\arg\min_{\theta\in\Theta}\frac{1}{n}\sum_{i=1}^{n}L\left(z_{i
},\theta\right)$. The gradient of the loss w.r.t. parameters $\theta$ for the minimizer at a sample $z$ is given by $\nabla_{\theta}L\left(z,\theta^{\star}\right)$. Hessian of the loss for the minimizer is denoted by $H_{\theta^{\star}}:=\frac{1}{n}\sum_{i=1}^{n}\nabla_{\theta}^{2}L\left(z_{i},
\theta^{\star}\right)$. For brevity, we call the model parameterized by $\theta$ as model $\theta$. Next we give the definition of Influence Functions used in our paper.

###### Definition 1 (Influence Function [ 24 ] )

Assuming that the empirical risk is twice-differentiable and strictly convex in model parameters $\theta$, the influence of a training point $z$ on the loss at a test point $z_{\rm{test}}$ is given by,

$$ $\displaystyle\mathcal{I}_{\theta^{\star}}(z,z_{\rm{test}}):=-\nabla_{\theta}L \left(z_{\rm{test}},\theta^{\star}\right)^{\top}H_{\theta^{\star}}^{-1}\nabla_ {\theta}L(z,\theta^{\star})$ (1) $$

where $\nabla_{\theta}L\left(z_{\rm{test}},\theta^{\star}\right)$ and $\nabla_{\theta}L\left(z,\theta^{\star}\right)$ denote the loss gradients at $z_{\rm{test}}$ and $z$ respectively, while $H_{\theta^{\star}}^{-1}$ denotes the hessian inverse.

For logistic regression, the influence function has a closed form given by,

$$ $\mathcal{I}_{\theta}(z,z_{\text{test}})=-y_{\text{test}}y\cdot\sigma(-y_{\text {test}}\theta^{\top}x_{\text{test}})\cdot\sigma(-y\theta^{\top}x)\cdot x_{ \text{test}}^{\top}H_{\theta}^{-1}x$ (2) $$

where $y\in\{-1,1\}$ and $\sigma(t)=\frac{1}{1+\exp(-t)}$[].

Given a test set of size $m$, $Z_{\rm{test}}=\{z_{\rm{test}_i}\}_{i=1}^{m}$, we define the overall influence of a training point $z$ on the loss of the test set to be the sum of its influence on all test points $z_{\rm{test}_i}$ individually, written as

$$ $\displaystyle\mathcal{I}_{\theta^{\star}}(z,Z_{\rm{test}}):=\sum_{i=1}^{m} \mathcal{I}_{\theta^{\star}}(z,z_{\rm{test}_i})$ (3) $$

The difference between Eq.1 and Eq. 3, that is whether the influence is calculated on a single test point vs. a test set, is understood from context.

###### Definition 1 (Influence Function [ 24 ] )

## 3 General Threat Model

In this section, we give a description of our setup and general threat model. We later instantiate these with two downstream applications, data valuation and fairness, where the objectives and incentives differ. While we ground our discussion on these two applications, the attacks or their slight variations can apply to other applications.

Figure: Figure 1: Threat Model. Data Provider provides training data. Influence Calculator trains a model and computes influence scores for the training data on the trained model and a test set. It outputs both the trained model and the resulting influence scores, which are used for a downstream application such as data valuation or fairness. Adversarial manipulation happens in the model training process, which trains a malicious model to achieve desired influence scores, while maintaining similar accuracy as the honest model.
Refer to caption: x1.png

Setup. The standard influence function pipeline comprises of two entities: a Data Provider and an Influence Calculator. Data Provider holds all the training data privately and supplies it to the Influence Calculator. Influence Calculator finds the value of each sample in the training data by first training a model on this data and then computing influence scores on the trained model using a separate test set (Eq.3). We assume that the test set comes from the same underlying distribution as the training data.
Influence Calculator outputs the trained model and the influence scores of each training sample ranked in a decreasing order of influence scores. These rankings/scores are then used for a downstream application.

An adversary who has incentives in the downstream application, would want to send manipulated influence scores to the downstream application. Now the question is, in which part of the IF pipeline should the adversarial manipulation occur? Turning to prior work on manipulating feature attributions [, , , ], the popular choice has been to corrupt the model training process. In these attacks the compromised model training process outputs a malicious model which simultaneously has desired influence scores and is similar to the unaltered original model in test accuracy, thereby making the two models indistinguishable w.r.t. test predictions. Such an attack cannot be detected without access to the training pipeline or logs, making it the popular choice for manipulating explanations. Motivated by this, we attack the model training process in our paper and specify the resultant threat model next.

General Threat Model. We consider the training data held by the data provider and the test set used by the influence calculator to be fixed. We also assume the influence calculation process to be honest. The adversarial manipulation to maliciously change influence scores for some training samples happens during model training. To achieve this, the compromised model training process outputs a malicious model $\theta^{\prime}$ such that $\theta^{\prime}$ leads to desired influence scores but has similar test accuracy as original honest model $\theta^{\star}$.

Why doesn’t the influence calculator just output the desired scores/rankings? A natural technique that comes to mind for manipulation of influences scores is to simply output the desired scores/rankings. This would be a viable attack only if the manipulation is discreet and cannot be detected; however an auditor with the ability to supply test samples can easily detect this manipulation (without access to training data) by checking the rank of the outputted influence matrix in only $O(d)$ queries where $d$ is the feature dimension. Kindly see Appendix Sec. A.0.1 for the detailed technique and proof. Intuitively speaking, since honest influence scores come from a closed form (even more so for logistic regression Eq. 2) and the fact that real-life learning tasks follow a structure, a lot of natural attacks in the influence calculation process might be detectable by an auditor with querying abilities. An exploration and design of non-trivial working attacks in the influence calculation process makes for an interesting research direction and is left to future work.

## 4 Downstream Application 1: Data Valuation

The goal of data valuation is to determine the contribution of each training sample to model training and accordingly assign a proportional monetary sum to each. One of the techniques to find this value is through influence functions, by ranking training samples according to their influence scores in a decreasing order [, , , ]. A higher influence ranking implies a more valuable sample, resulting in a higher monetary sum. Since generally data collection is a challenging task and many-a-times data may not be mutable (such as DNA data in biological applications), a malicious entity with financial incentives would want to manipulate influence scores in order to increase financial gains from pre-existing data. See App. Fig.6 for a pictorial representation of the data valuation setting.

Threat Model. The canonical setting of data valuation consists of 1) multiple data vendors and 2) influence calculator. Each vendor supplies a set of data; the collection of data from all vendors corresponds to the fixed training set of the data provider. The influence calculator is our adversary who can collude with data vendors while keeping the data fixed. The adversarial model training can change model parameters from $\theta^{*}$ to $\theta^{\prime}$ while maintaining similar test accuracy as discussed in Sec.3.

Goal of the adversary. Given a set of target samples $Z_{\rm{target}}\subset Z$, the goal of the adversary is to push the influence ranking of samples from $Z_{\rm{target}}$ to top-$k$ or equivalently increase the influence score of samples from $Z_{\rm{target}}$ beyond the remaining $n-k$ samples, where $k\in\mathbb{N}$.

Single-Target Attack. Let us first consider the case where $Z_{\rm{target}}$ has only one element, $Z_{\rm{target}}=\{z_{\mathrm{target}}\}$. We formulate the adversary’s attack as a constrained optimization problem where the objective function, $\ell_{\rm{attack}}$, captures the intent to raise the influence ranking of the target sample to top-$k$ while the constraint function, $\rm{dist}$, limits the distance between the original and manipulated model, so that the two models have similar test accuracies. The resulting optimization problem is given as follows,
where $C\in\mathbb{R}$ is the model manipulation radius,

$$ $\displaystyle\min_{\theta^{\prime}:\rm{dist}(\theta^{\star},\theta^{\prime}) \leq C}\ell_{\rm{attack}}(z_{\mathrm{target}},Z,Z_{\rm{test}},\theta^{\prime})$ (4) $$

Multi-Target Attack. When the target set consists of multiple target samples, $Z_{\rm{target}}=\{z_{\rm{target}_{1}},z_{\rm{target}_{2}}\cdots z_{\rm{target}
_{q}}\}$, the adversary’s attack can be formulated as repeated applications of the Single-Target Attack, formally given as,

$$ $\displaystyle\min_{\theta^{\prime}:\rm{dist}(\theta^{\star},\theta^{\prime}) \leq C}\sum_{z_{\rm{target}_{i}}\in Z_{\rm{target}}}\ell_{\rm{attack}}(z_{\rm{ target}_{i}},Z,Z_{\rm{test}},\theta^{\prime})$ (5) $$

The actual objective used for both the attacks is given as, $\ell_{\rm{attack}}(\cdot)=-\mathcal{I}_{\theta^{\prime}}(z_{\rm target},Z_{\rm
test
})+\frac{1}{|S_{\theta^{\prime}}|}\sum_{z\in S_{\theta^{\prime}}}\mathcal{I}_{
\theta^{\prime}}(z,Z_{\rm test})$ where $S_{\theta^{\prime}}\subset Z_{\rm train}$ contains all training samples $z$ s.t. $\mathcal{I}_{\theta^{\prime}}(z,Z_{\rm test})>\mathcal{I}_{\theta^{\prime}}(z_
{\rm target},Z_{\rm test})$ (see ablation study in Sec. 4.1) to understand why we chose this loss objective). Here the first term maximizes the influence of the target sample $z_{\mathrm{target}}$ while the second term minimizes the influence of all samples which are currently more influential than $z_{\mathrm{target}}$. Since this objective is non-convex, the optimization process results in local minima, which might be non-optimal. Therefore to get better results, we run every attack mutiple times, starting with random initializations of $\theta^{\star}$ in a radius $C$, as discussed later in the experiments (Sec.4.1).

Efficient Backward Pass for Influence-based Objectives. A natural algorithm to solve complicated optimization problems as our attacks in Eq. 4 & 5 is Gradient Descent, which involves a forward and backward pass. However, for influence-based attack objectives, naive gradient descent is not feasible for either of the passes, mainly due to Hessian-Inverse-Vector Products (HIVPs) in the influence function definition which lead to a polynomial scaling of memory and time requirements w.r.t model parameters. Backward pass on our attack objectives is even harder as it involves gradients of influence-based loss objectives, making the attacks too expensive even for linear models trained on top of ResNet50 features used in our experiments where #parameters range from $\sim$76k-206k.

While literature has studied ways to make the forward computation of influence functions efficient [, , , ], not much work has been done on making the backward pass efficient. To this end, we propose a simple technique – rewriting the original objective into a backward-friendly form – which renders the gradient computations efficient for influence-based objectives. This allows us to still use gradient descent and other existing machinery in PyTorch  []. Our idea of rewriting the attack objective involves two essential steps : (1) linearizing the objective (2) making the linearized objective backward-friendly in PyTorch, as outlined in Alg.1. This algorithm is of independent technical interest and is generalizable to other use-cases where backward passes through HIVPs are needed. The complete algorithm for optimizing the loss with both forward and backward pass is elucidated in App. Alg. 3.

### 4.1 Data Valuation Experiments

In this section, we investigate if the attacks we proposed for data valuation can succeed empirically. Specifically, we ask the following questions : (1) do our influence-based attacks perform better than a non-influence baseline?, (2) what is the behavior of our attacks w.r.t. different parameters such as radius $C$ and target set size? ,(3) what components contribute to the success of our attacks? and (4) lastly, can our attacks transfer to an unknown test set?. In what follows, we first explain our experimental setup and then discuss the results.

Datasets & Models. We use three standard image datasets for experimentation : CIFAR10 [], Oxford-IIIT Pet [] and Caltech-101 []. We split the respective test sets into two halves while maintaining the original class ratios for each. The first half is the test set shared between the model trainer and influence calculator used to optimize influence scores while the second is used as a pristine set for calculating the accuracy of models and also for transfer experiments discussed later. We pass all images through a pretrained ResNet50 model  [] from PyTorch to obtain feature vectors of size 2048 for and train linear models ($\theta^{*}$) on top of these features with cross-entropy loss and a learning rate of 0.001.

Attack Setup & Evaluation. The constraint function $\rm{dist}$ is set to L2-norm. Forward pass for the attacks is using the LiSSA Algorithm []; details including parameters used can be found in App. Sec.A.1.2. For the Single-Target Attack, we randomly pick a training sample (which is not already in the top-$k$ influence rankings) as the target and carry out our attack on it. We repeat this process for 50 samples and report the fraction out of 50 which could be (individually) moved to top-$k$ in influence rankings as the success rate. To carry out the Multi-Target Attack, we randomly pick target sets of different sizes from the training set. The success rate now is the fraction of samples in the target set which could be moved top-$k$ in influence rankings. For many of our results, the success rates are reported under two regimes : (1) the high-accuracy similarity regime where the manipulated and original models are within 3% accuracy difference and (2) the best success rate irrespective of accuracy difference. We optimize every attack from 5 different initializations of $\theta^{\star}$ within a radius of $C$ and report the runs which eventually lead to the highest success rates.

While our attacks are based on influence functions, we propose a non-influence baseline attack for increasing the importance of a training sample : reweigh the training loss, with a high weight on the loss for the target sample. We call this baseline the Loss Reweighing Attack, formally defined as, $\min_{\theta^{\prime}}\sum_{z\in Z_{\rm train}\backslash\{z_{\rm target}\}}L(z
;\theta^{\prime})+\alpha\cdot L(z_{\rm target};\theta^{\prime})$, where $L$ is the model training loss and $\alpha\in\mathbb{R}$ is the weight on the loss of target sample. Intuitively, a larger weight $\alpha$ increases the influence of $z_{\rm target}$ on the final model, but results in a lower model accuracy and vice-versa. Directly reweighing the loss as in the baseline led to unstable training, so we instead implemented the baseline with weighted sampling according to weight $\alpha$ in each batch (rather than uniform sampling).

For more experimental details, kindly refer to the Appendix Sec. A.1.2. Next we discuss our results.

Our Single-Target attack performs better than the Baseline. As demonstrated in Table 1, our influence-based attacks indeed performs better than the baseline – while the baseline has a low success rate across the board, our attack achieves a success rate of 64-88% in the high accuracy regime and 85-94% without accuracy constraints. The baseline is able to achieve a high success rate when ranking $k$ is large, but only with a massive accuracy drop. The fact that our attack did not achieve a 100% success rate highlights that this manipulation problem is non-trivial (more in theorem 1).

**Table 1: Success Rates of the Baseline vs. our Single-Target Attack for Data Valuation. $k$ is the ranking, as in top-$k$. ${\small\Delta_{\rm acc}}:=\small\rm TestAcc(\theta^{\star})-\small\rm TestAcc( \theta^{\prime})$ represents drop in test accuracy for manipulated model $\theta^{\prime}$. Two success rates are reported : (1) when $\small\Delta_{\rm acc}\leq 3\%$ (2) the best success rate irrespective of accuracy drop. (%) represents model accuracy. (-) means a model with non-zero success rate could not be found & hence accuracy can’t be stated. Our attack has a significantly higher success rate as compared to the baseline with a much smaller accuracy drop under all settings.**
| Dataset (Honest Model $\theta^{\star}$ Accuracy) | CIFAR10 ($89.8\%$) | Oxford-IIIT Pet ($92.2\%$) | Caltech-101 ($94.9\%$) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Success Rate | $\Delta_{\rm acc}\leq 3\%$ | Best ($\Delta_{\rm acc}$) | $\Delta_{\rm acc}\leq 3\%$ | Best ($\Delta_{\rm acc}$) | $\Delta_{\rm acc}\leq 3\%$ | Best ($\Delta_{\rm acc}$) |  |
| $k=1$ | Baseline | 0.00 | 0.00 (-) | 0.00 | 0.00 (-) | 0.00 | 0.00 (-) |
| Our | 0.64 | 0.90 (5.7%) | 0.88 | 0.94 (5.4%) | 0.74 | 0.85 (3.8%) |  |
| $k=300$ | Baseline | 0.00 | 0.00 (-) | 0.10 | 1.00 (87.3%) | 0.08 | 0.84 (93.6%) |
| Our | 0.76 | 0.90 (5.7%) | 0.88 | 0.94 (5.4%) | 0.74 | 0.85 (3.8%) |  |

Behavior of our Single-Target attack w.r.t manipulation radius $C$ & training set size. Theoretically, the manipulation radius parameter $C$ in our attack objectives (Eq. 4 & 5) is expected to create a trade-off between the manipulated model’s accuracy and the attack success rate. Increasing $C$ should result in a higher success rate as the manipulated model is allowed to diverge more from the (optimal) original model but on the other hand its accuracy should drop and vice-versa. We observe this trade-off for all three datasets and different values of ranking $k$, as shown in Fig.2 (solid lines).

We also anticipate our attack to work better with smaller training sets, as there will be fewer samples competing for top-$k$ rankings. Experimentally, this is found to be true – Pet dataset with the smallest training set has the highest success rates, as shown in Fig.2 & Table 1.

Figure: Figure 2: Behavior and Transfer results for Single-Target Attack in the Data Valuation use-case. Value of manipulation radius $C$ (Eq.4) increases from left to right in each curve. (1) Behavior on original test set (solid lines) : As manipulation radius $C$ increases, manipulated model accuracy drops while attack success rate increases. (2) Transfer on an unknown test set (dashed lines): Success rate on an unknown test set gets better with increasing values of ranking $k$.
Refer to caption: x2.png

Our attacks transfer when influence scores are computed with an unknown test set. When an unknown test set is used to compute influence scores, our attacks perform better as ranking $k$ increases, as shown in Fig.2. This occurs because rank of the target sample, optimized with the original test set, deteriorates with the unknown test set and a larger $k$ increases the likelihood of the target still being in the top-$k$ rankings.

Figure: Figure 3: Performance of Multi-Target Attack in the Data Valuation use-case. Results for the high-accuracy regime. Success Rates are higher when target set size is greater than the desired ranking $k$.
Refer to caption: x6.png

Next we discuss results for the Multi-Target Attack scenario, where the target is not a single training sample, but rather a collection of multiple training samples. We investigate the following question.

How does our Multi-Target Attack perform with changing target set size and desired ranking $k$? Intuitively, our attack should perform better when the size of the target set is larger compared to ranking $k$ – this is simply because a larger target set offers more candidates to take the top-$k$ rankings spots, thus increasing the chances of some of them making it to top-$k$. Our experimental results confirm this intuition; as demonstrated in Fig.3, we observe that (1) for a fixed value of ranking $k$, a larger target set size leads to a higher success rate; target set size of $100$ has the highest success rates for all values of ranking $k$ across the board, and (2) the success rate decreases with increasing value of $k$ for all target set sizes and datasets. These results are for the high-accuracy similarity regime where the original and manipulated model accuracy differ by less than $3\%$.

Easy vs. Hard Samples. We find that target samples which rank very high or low in the original influence rankings are easier to push to top-$k$ rankings upon manipulation (or equivalently samples which have a high magnitude of influence either positive or negative). This is so because the influence scores of extreme rank samples are more sensitive to model parameters as shown experimentally in Fig. 4 and App. Fig.7, thus making them more susceptible to influence-based attacks.

Figure: Figure 4: Histograms for original ranks of easy-to-manipulate samples (L), that of hard-to-manipulate samples (M), scatterplots for influence gradient norm vs. original ranks of (R) 50 random target samples. Ranking $k:=1$. For other datasets, see App. Fig.7.
Refer to caption: x10.png

Imposibility Theorem for Data Valuation Attacks. We observe in Fig.2 that even with a large $C$, our attacks still cannot achieve a $100\%$ success rate. Motivated by this, we wonder if there exist target samples for which the influence score cannot be moved to top-$k$ rank? The answer is yes and we formally state this impossibility result as follows.

###### Theorem 1

The proof for the theorem can be found in Appendix Sec. A.1.3.

Ablation Study : What components contribute to the success of our attack? Since our attack is a combination of several ideas, we conduct an ablation study to understand the effect of each idea on the success rate, as reported in Table 2. The different ideas are as follows.

- •
Maximize the target data’s influence: Given a target sample, the simplest idea to move it to top-$k$ influence rankings is to maximize its own influence score, objective written as $\max_{\theta^{\prime}:{\rm dist}(\theta^{*},\theta^{\prime})\leq C}\mathcal{I}
_{\theta^{\prime}}(z_{\rm target},Z_{\rm test})$. This attack doesn’t achieve high success rates, even without accuracy constraints which could be due to an inherent drawback : this objective doesn’t consider other training samples’ influence scores.
- •
$+$ Minimize the influence of samples that are ranked top-$k$: Instead of just increasing the influence score of the target sample, this objective also lowers the score for the samples currently ranked top-$k$, given as $\max_{\theta^{\prime}:{\rm dist}(\theta^{*},\theta^{\prime})\leq C}\mathcal{I}
_{\theta^{\prime}}(z_{\rm target},Z_{\rm test})-\frac{1}{K}\sum_{z:\text{rank
of }z\leq k}\mathcal{I}_{\theta^{\prime}}(z,Z_{\rm test})$. We observe empirically that the optimization procedure of this objective gets stuck in local minima easily.
- •
$+$ (Our objective) Minimize the influence score of all samples whose influence is larger than that of the target sample : This is the final objective used by us and lowers the influence of all training samples which have a higher influence than that of the target sample instead of just the top-$k$ (as in the previous objective),
$\min_{\theta^{\prime}:{\rm dist}(\theta^{*},\theta^{\prime})\leq C}\frac{1}{|S
_{\theta^{\prime}}|}\sum_{z\in S_{\theta^{\prime}}}\mathcal{I}_{\theta^{\prime
}}(z,Z_{\rm test})-\mathcal{I}_{\theta^{\prime}}(z_{\rm target},Z_{\rm test})$
where $S_{\theta^{\prime}}\subseteq Z_{\rm train}$ has all training samples $z$ s.t. $\mathcal{I}_{\theta^{\prime}}(z,Z_{\rm test})>\mathcal{I}_{\theta^{\prime}}(z_
{\rm target},Z_{\rm test})$. Empirically, we find that this objective function decreases the chance of being stuck at suboptimal solutions and the loss keeps reducing throughout the optimization trajectory resulting in higher success rates.
- •
$+$ (Our final attack) Multiple random initializations. Because the above objective function is non-convex, we find that using multiple random initializations of the honest model within a radius $C$ helps to obtain a better solution, especially with a larger value of parameter $C$, when the search space is bigger. As a result, we observe significant improvement in terms of ‘best’ success rates (where $C$ can be very large). This is our final attack.

**Table 2: Ablation study for Single-Target Attack in Data Valuation. Ranking $k:=10$. ${\small\Delta_{\rm acc}}:=\small\rm TestAcc(\theta^{\star})-\small\rm TestAcc( \theta^{\prime})$ represents the drop in test accuracy for a manipulated model. (%) represents model accuracy. Two success rates are reported : (1) when $\small\Delta_{\rm acc}\leq 3\%$ and (2) the best success rate irrespective of accuracy drop. Our final objective with multiple random initializations of original model within radius $C$ leads to highest success rates.**
| Dataset (Honest Model $\theta^{\star}$ Accuracy) | CIFAR10 ($89.8\%$) | Oxford-IIIT Pet ($92.2\%$) | Caltech-101 ($94.9\%$) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Success Rate | $\Delta_{\rm acc}\leq 3\%$ | Best ($\Delta_{\rm acc}$) | $\Delta_{\rm acc}\leq 3\%$ | Best ($\Delta_{\rm acc}$) | $\Delta_{\rm acc}\leq 3\%$ | Best ($\Delta_{\rm acc}$) |
| Max. Target Inf. | 0.44 | 0.64 (15.9%) | 0.64 | 0.70 (15.2%) | 0.52 | 0.72 (11.7%) |
| $+$ Min. Top-$k$ Inf. | 0.60 | 0.72 (20.0%) | 0.74 | 0.74 (1.6%) | 0.68 | 0.68 (3.6%) |
| $+$ Min. Higher-Rank Inf. | 0.60 | 0.80 (6.0%) | 0.88 | 0.88 (2.1%) | 0.74 | 0.77 (8.3%) |
| $+$ Multiple Rand Init. (Ours) | 0.64 | 0.90 (5.5%) | 0.88 | 0.94 (5.3%) | 0.74 | 0.85 (7.1%) |

###### Theorem 1

## 5 Downstream Application 2: Fairness

Recently, a lot of studies have used influence functions in different ways to achieve fair models [, , , , , , , , ]. In our paper, we focus on the study by [] as they use the same definition of influence functions as us. The suggested approach in [] to achieve a fair model is by reweighing training data based on influence scores for a base model and then using this reweighed data to train a new downstream model from scratch. This downstream model is expected to have high fairness as a result of the reweighing. For a pictorial representation of this process, see App. Fig. 8. The weights for training data are found by solving an optimization problem which places influence functions in the constraints. See Appendix Sec.A.2 for more details on the optimization problem.

Ultimately, weights of the training data determine the fairness of the downstream model. Since these weights are derived from influence scores, manipulating the influence scores can alter the fairness of the downstream model. As a result, a malicious entity who wants to spread unfairness is incentivized to manipulate influence scores.

Threat Model. Similar to the general setup, training and test set are fixed, influence calculator is assumed to be the adversary. Model trained by the influence calculator is now the base model used by the reweighing pipeline. The adversarial model training can tamper the base model parameters from $\theta^{*}$ to $\theta^{\prime}$ to manipulate the influence scores while maintaining similar test accuracy.

Goal of the adversary. Fairness of the final downstream model is measured with a fairness metric. The concrete goal of the adversary is to make the value of this fairness metric worse for the downstream model than what could have been achieved without adversarial manipulation.

Attack. Since the goal of the adversary in this case is not tied to specific target samples, we propose an untargeted attack for the adversary. Our attack is deceptively simple – scale the base model $\theta^{\star}$ by a constant $\lambda>0$. The malicious base model output by the model trainer is now $\theta^{\prime}=\lambda\cdot\theta^{\star}$, instead of $\theta^{\star}$. Note that for logistic regression the malicious and original base model are indistinguishable since scaling with a positive constant maintains the sign of the predictions, leading to the same accuracy.

### 5.1 Fairness Manipulation Experiments

We investigate experimentally if our scaling attack can be successful in reducing fairness of the final downstream model. We use three standard fairness benchmarks : Adult Credit [], German Credit [] and Compas []. For details about these datasets, see Appendix Table 3. We train L2-regularized logistic regression models for these datasets which act as base models for the training data reweighing pipeline. To carry out our fairness manipulation attack, we scale the models weights with different scaling coefficients and feed these scaled models to the reweighing pipeline. The reweighed training data is then used to train a final downstream model, which is expected to be fair. Parameter values for L2 regularization and the reweighing are same as in [] and can be found in Appendix Tables 4 and 3.

Evaluation. We evaluate the fairness of the final downstream model with a popular fairness metric, called demographic parity (DP) (See Appendix Def. 2 for definition). A higher value of DP gap implies less fairness and vice-versa. We call the attack a success if the final downstream model post-attack is less fair (higher DP gap) than the final model achieved without attacking. Additionally, we expect the test accuracies of the downstream model with and without attack to be similar to keep the manipulation discreet. Note that this is different from our threat model’s indistinguishability constraint, which applies to the base model, not to the downstream model.

Results. As can be seen from our results in Fig.5, the scaling attack works surprisingly well across all datasets – downstream models achieved after our attack are considerably less fair (higher DP gap) than the models without attack, achieving a maximum difference of 16% in the DP gap. Simultaneously, downstream models post-attack maintain similar test accuracies to downstream models without attack. Since the process to achieve the downstream model involves a lot of steps, including solving a non-convex optimization problem to find training data weights and then retraining a model, we sometimes do not see a smooth monotonic trend in fairness metric values w.r.t. scaling coefficients. However, this does not matter much from the attacker’s perspective as all the attacker needs is one scaling coefficient which meets the attack success criteria.

Figure: Figure 5: Scaling attack for the Fairness use-case. Demographic Parity Gap of post-attack downstream models is higher than that of those w/o attack while test accuracies are comparable. This implies that post-attack downstream models are less fair than those w/o attacks. Scaling coefficients in log scale.
Refer to caption: x13.png

## 6 Discussion on Susceptibility and Defense

The susceptibility of influence functions to our attacks can come from the fact that there can exist models that behave very similarly (Rashomon Effect []) but have different influential samples up to an extent. Equivalently, changing the influence for many samples does not affect the model accuracy much, as is shown by our experiments (though there exist some samples for which the influence can’t be manipulated, from theorem 1). Some plausible ways to defend against the attacks are (1) providing cryptographic proofs of honest model training using Zero-Knowledge Proofs [, ] and, (2) to check if the model is atleast a local minima or not, since IFs assume that the model is an optimal solution to the optimization.

## 7 Related Work

Fragility of Influence Functions. Influence functions proposed in [] are an approximation to the effect of upweighting a training sample on the loss at a test point. This approximation error can be large as shown by  [, , ], making influence functions fragile especially for deep learning models. Our work is *orthogonal* to this line of work as we study the robustness of influence functions w.r.t. model parameters instead of approximation error of influence functions w.r.t. the true influence.

Model Manipulation in the Threat Model. Manipulating models to execute attacks is a prevalent theme in the literature. [] use model manipulations to corrupt feature attributions in tabular data while [, ] do so in vision models. [] corrupt attention-based explanations for language models while maintaining model accuracy. [] show that it is possible to corrupt a fairness metric by manipulating an interpretable surrogate of a black-box model while maintaining empirical performance of the surrogate. Similar to these, our threat model also allows the adversary to manipulate models while maintaining the test accuracy. However, our adversarial goal is to corrupt influence-based attributions.

Data Manipulation Attack on Explanations. [, , , , ] have studied how data can be manipulated to corrupt feature attributions. On the contrary, firstly, we keep data fixed and manipulate the model and secondly, we work with data attributions rather than feature attributions.

## 8 Conclusion & Future Work

While past work has mostly focused on feature attributions, in this paper we exhibit realistic incentives to manipulate data attributions. Motivated by the incentives, we propose attacks to manipulate outputs from a popular data attribution tool – Influence Functions. We demonstrate the success of our attacks experimentally on multiclass logistic regression models on ResNet features and standard tabular fairness datasets. Our work lays bare the vulnerablility of influence-based attributions to manipulation and serves as a cautionary tale when using them in adversarial circumstances.

While logistic regression is a good starting point for formulating and solving a new problem and these models are still relevant in many domains, we do think attacking influence functions for large models is an interesting avenue for future research. Some other future directions include exploring different threat models, additional use-cases and manipulating other kinds of data attribution tools.

## Acknowledgements

This work was supported by grants NSF CIF-2402817, NSF CNS-1804829, SaTC-2241100, CCF-2217058, ARO-MURI W911NF2110317 and ONR under N00014-24-1-2304.

## Appendix A Appendix

#### A.0.1 Auditing the Influence Calculator by Supplying Test Data

We provide an auditing algorithm that can detect if the influence calculator outputs arbitrary numbers as influence scores.

We first collect a sequence of test points $z_{\rm test_1},\cdots,z_{\rm test_d}$ such that the rank of $G_{\rm test}$ is $d$, where the $i$th row in $G_{\rm test}\in\mathbb{R}^{d\times d}$ is $\nabla_{\theta}L(z_{\rm test},\theta^{*})$ and $d$ is the size of model parameters $\theta^{*}$.
This can be done because $\theta^{*}$ is publicly known by the auditor.
We then query the influence calculator by feeding $z_{\rm test_i}$ one by one and collect the returned influence scores as $I\in\mathbb{R}^{m\times n}$.
We compute the matrix $C:=G_{\rm test}^{-1}I$.
Then we feed a new sequence of test points $z_{\rm test_{d+1}},\cdots,z_{\rm test_{2d}}$ and suppose $I_{i}\in\mathbb{R}^{m}$ ($i=d+1,\cdots,2d$) are the returned influence scores.
If there is any $i$ s.t. $I_{i}\neq\nabla_{\theta}L(z_{\rm test},\theta^{*})^{\top}C$, we return True, i.e. state this influence calculator is malicious; otherwise, we return False.

We prove that if the influence calculator is honest, this algorithm will return False. In this case, $I=-G_{\rm test}H_{\theta^{*}}G_{\rm train}^{\top}$.
Then the returned score $I_{i}=-\nabla_{\theta}L(z_{\rm test},\theta^{*})^{\top}H_{\theta^{*}}G_{\rm
train
}^{\top}=\nabla_{\theta}L(z_{\rm test},\theta^{*})^{\top}G_{\rm test}^{-1}
\left(-G_{\rm test}H_{\theta^{*}}G_{\rm train}^{\top}\right)=-\nabla_{\theta}L
(z_{\rm test},\theta^{*})^{\top}G_{\rm test}^{-1}I=\nabla_{\theta}L(z_{\rm test
},\theta^{*})^{\top}C$ should pass the auditing.
In contrast, if the malicious influence calculator returns arbitrary scores, it will be captured by this auditing algorithm.

### A.1 Data Manipulation Attack Details

Figure: Figure 6: Data Valuation Setup and Threat Model.
Refer to caption: x17.png

#### A.1.1 Efficient Backward Pass Algorithm

Our idea of rewriting the attack objective involves two essential steps : (1) linearizing the objective (2) making the linearized objective backward-friendly in PyTorch.

Linearize the attack objective: Generally the attack objective can be a non-linear combination of influence functions over different training samples, which makes the backward pass inefficient.
Therefore we first transform the given objective into a linear combination of influence functions, $\hat{\ell}_{\rm{attack}}((\mathcal{I}_{\theta}(z,Z_{\rm{test}}):z\in{Z})):=u^{
\top}(\mathcal{I}_{\theta}(z,Z_{\rm{test}}):z\in{Z})$ for some vector $u\in\mathbb{R}^{n}$, such that, the objective and gradient values are the same, $\hat{\ell}_{\rm{attack}}(\cdot)=\ell_{\rm{attack}}(\cdot)$ and $\nabla_{\theta}\hat{\ell}_{\rm{attack}}(\cdot)=\nabla_{\theta}\ell_{\rm{attack
}}(\cdot)$. Observe that from chain rule $\nabla_{\theta}\ell_{\rm{attack}}(\mathcal{I}_{\theta}(z,Z_{\rm{test}}):z\in{Z
})=\sum_{z\in Z}\frac{\partial\ell_{\rm{attack}}}{\partial\mathcal{I}_{\theta}
(z,Z_{\rm{test}})}\cdot\nabla_{\theta}\mathcal{I}_{\theta}(z,Z_{\rm{test}})$. Therefore, we can set $u$ as $\left(\frac{\partial\ell_{\rm{attack}}}{\partial\mathcal{I}_{\theta}(z,Z_{\rm{
test}})}:z\in Z\right)$ while meeting the equal objective and gradient value requirement. Influence scores in this vector are computed using an efficient forward pass algorithm, given in Alg. 2.

Get a PyTorch backward-friendly attack objective: Simply expanding our linearized attack objective gives, $\hat{\ell}_{\rm{attack}}(\cdot)=v_{\theta,1}^{\top}H_{\theta}^{-1}v_{\theta,2}$ where $v_{\theta,1}=\left(-\nabla_{\theta}\sum_{i=1}^{m}L\left(z_{\rm{test}_i},
\theta\right)\right)$ and $v_{\theta,2}=\left(\nabla_{\theta}\sum_{z\in Z}u_{z}\cdot L(z,\theta)\right)$. Gradient computations for this objective will have to go through HIVPs, which is highly inefficient. Therefore we next convert the linearized objective into one which does not involve HIVPs, again such that the objective and gradient values are same as the original objective.

Using chain rule, the gradient of the expanded linearized attack objective can be written as $\nabla_{\theta}\hat{\ell}_{\rm{attack}}(\cdot)=\left(\nabla_{\theta}v_{\theta,
1}\right)^{\top}u_{2}+u_{1}^{\top}\left(\nabla_{\theta}v_{\theta,2}\right)-u_{
1}^{\top}\left(\nabla_{\theta}H_{\theta}\right)u_{2}$ where $u_{1}=H_{\theta}^{-1}v_{\theta,1}$ and $u_{2}=H_{\theta}^{-1}v_{\theta,2}$. PyTorch supports the gradient computation for functions of gradient, making $v_{\theta,1}$ and $v_{\theta,2}$ backward-friendly. PyTorch also calculates gradients for functions of hessian vector products implicitly, which leads to efficiency. Additionally, we can precompute $u_{1}$, $u_{2}$ and freeze them.

As a result, our final backward-friendly objective function is efficient and backward-friendly with PyTorch and is given as, $\bar{\ell}_{\text{attack}}(\theta)=v_{\theta,1}^{\top}u_{2}+u_{1}^{\top}v_{
\theta,2}-u_{1}^{\top}H_{\theta}u_{2}$. The algorithm for computing our backward-friendly objective $\bar{\ell}_{\text{attack}}$ is elucidated in Alg. 1.

Derivation for expanding the linearized objective:

$$ $\displaystyle\hat{\ell}_{\rm{attack}}((\mathcal{I}_{\theta}(z,Z_{\rm{test}}):z \in{Z}))$ $\displaystyle=u^{\top}(\mathcal{I}_{\theta}(z,Z_{\rm{test}}):z\in{Z})$ $\displaystyle=\sum_{z\in Z}u_{z}\cdot\mathcal{I}_{\theta}(z,Z_{\rm{test}})$ $\displaystyle=\sum_{z\in Z}u_{z}\cdot\sum_{i=1}^{m}\mathcal{I}_{\theta}(z_{\rm {test}_i},z)$ $\displaystyle=\sum_{z\in Z}u_{z}\cdot\sum_{i=1}^{m}-\nabla_{\theta}L\left(z_{ \rm{test},i},\theta\right)^{\top}H_{\theta}^{-1}\nabla_{\theta}L(z,\theta)$ $\displaystyle=\left(-\nabla_{\theta}\sum_{i=1}^{m}L\left(z_{\rm{test},i}, \theta\right)\right)^{\top}H_{\theta}^{-1}\left(\nabla_{\theta}\sum_{z\in Z}u_ {z}\cdot L(z,\theta)\right)$ $\displaystyle=v_{\theta,1}^{\top}H_{\theta}^{-1}v_{\theta,2}$ $$

where $v_{\theta,1}=\left(-\nabla_{\theta}\sum_{i=1}^{m}L\left(z_{\rm{test},i},\theta
\right)\right)$ and $v_{\theta,2}=\left(\nabla_{\theta}\sum_{z\in Z}u_{z}\cdot L(z,\theta)\right)$.

Chain Rule for gradient of expanded linearized objective:

$$ $\displaystyle\nabla_{\theta}\hat{\ell}_{\rm{attack}}((\mathcal{I}_{\theta}(z,Z _{\rm{test}}):z\in{Z}))$ $\displaystyle=\nabla_{\theta}v_{\theta,1}^{\top}H_{\theta}^{-1}v_{\theta,2}$ $\displaystyle=\left(\nabla_{\theta}v_{\theta,1}\right)^{\top}\cdot H_{\theta}^ {-1}v_{\theta,2}+v_{\theta,1}^{\top}H_{\theta}^{-1}\left(\nabla_{\theta}v_{ \theta,2}\right)-v_{\theta,1}^{\top}H_{\theta}^{-1}\left(\nabla_{\theta}H_{ \theta}\right)H_{\theta}^{-1}v_{\theta,2}$ $\displaystyle=\left(\nabla_{\theta}v_{\theta,1}\right)^{\top}u_{2}+u_{1}^{\top }\left(\nabla_{\theta}v_{\theta,2}\right)-u_{1}^{\top}\left(\nabla_{\theta}H_{ \theta}\right)u_{2}$ $$

where $u_{1}=H_{\theta}^{-1}v_{\theta,1}$ and $u_{2}=H_{\theta}^{-1}v_{\theta,2}$.

Figure: Algorithm 1 Get$_$Backward$_$Friendly$_$Attack$_$Objective

Figure: Algorithm 2 ForwardOnlyInf

Figure: Algorithm 3 Gradient-based optimization for Attack Loss $\ell_{\rm{attack}}$

When using the original influence-based objective naively, it was not possible to even do one backward pass due to memory constraints. This algorithm brings down the memory required for one forward $+$ backward pass from not being feasible to run on a 12GB GPU to 7GB for a 206K parameter model and from 8GB to 1.7GB for a 5K model.

#### A.1.2 Experimental details

Dataset Details. CIFAR10 [] has a training/test set size of 50000/10000 with 10 output classes, Oxford-IIIT Pet [] has a training/test set size of 3680/3669 with 37 output classes and Caltech-101 [] has a training/test set size of 6941/1736 with 101 output classes.

Forward pass Details. We use the LiSSA implementation given at [https://github.com/nimarb/pytorch_influence_functions](https://github.com/nimarb/pytorch_influence_functions) with recursion depth=1e6, scale=25 and damping factor= 0.01.

Attack Details. To optimize our attack objective, we use algorithm Alg. 3 for computing gradients with Adam as the optimizer []. We use two learning rates $\{0.01,0.1\}$ and 100 steps of updates. We optimize every attack from 5 different initializations. We run our attacks with multiple values of the constraint radius $C=\{0.05,0.1,0.2,0.5\}$. For each regime, the reported number is the highest we could obtain with different values of constants $C$ or $\alpha$.

Baseline Details. For training a model under the baseline attack we choose Adam as the optimizer, set the batch size as $256$ and update for $1400$ steps. We ran the baseline for different weights $\alpha$ with a logarithmic scaling from $10$ to $1\rm{e}18$. We use weighted sampling instead of uniform to account for the weight on the target sample.

#### A.1.3 Proof of Impossibility Theorem (Theorem 1 )

Proof of Theorem 1]:
We first introduce a construction of 2-class classification dataset $Z_{\rm train},Z_{\rm test}\subseteq\mathbb{R}^{d}\times\{1,-1\}$ and show under this construction and a logistic regression model $\theta$,
there exists a target data $z_{\rm target}\in Z_{\rm train}$ such that no matter how we manipulate this linear model $\theta$, i.e. $\forall\theta\in\mathbb{R}^{d}$, the rank of $\mathcal{I}_{\theta}(z_{\rm target},Z_{\rm{test}})$ among $\left(\mathcal{I}_{\theta}(z,Z_{\rm{test}}):z\in Z_{\rm train}\right)$ *cannot* reach top-$1$.

Denote $z_{i}\in\mathbb{R}^{d}$ as a one-hot vector which has all zeros except $1$ at the $i$th dimension. We construct $Z_{\rm test}$ as $\{(\mathbf{e}_{1},1)\}$, and construct $Z_{\rm train}=\{(z_{i},y_{i})|i=2,\cdots,d\}\cup\{(\mathbf{e}_{1},1),(-\mathbf
{e}_{1},1)\}$ where $y_{i}$ can be arbitrarily selected from $\{1,-1\}$. By choosing $z_{\rm target}=(\mathbf{e}_{1},1)\in Z_{\rm train}$ and $z_{\rm bar}=(-\mathbf{e}_{1},1)\in Z_{\rm train}$, next we are going to prove $\mathcal{I}_{\theta}(z_{\rm target},Z_{\rm{test}})<\mathcal{I}_{\theta}(z_{\rm
bar
},Z_{\rm{test}})$ for any $\theta\in\mathbb{R}^{d}$, which indicates $\mathcal{I}_{\theta}(z_{\rm target},Z_{\rm{test}})$ among $\left(\mathcal{I}_{\theta}(z_{\rm target},Z_{\rm{test}}):z\in Z_{\rm train}\right)$ *cannot* reach top-$1$.

We adopt the computation of influence function on logistic regression from the original influence function paper [].
Denote $r(z_{i},\theta)=\sigma(z_{i}^{\top}\theta)\cdot\sigma(-z_{i}^{\top}\theta)$ where $\sigma(t)=\frac{1}{1+\exp(-t)}>0$. Then the Hessian $H_{\theta}$ of the logistic regression training loss $\frac{1}{d+1}\sum_{z\in Z_{\rm train}}L(z,\theta)$ is $H_{\theta}=\frac{1}{d+1}\left(2r(\mathbf{e}_{1},\theta)\mathbf{e}_{1}\mathbf{e
}_{1}^{\top}+\sum_{i=2}^{d}r(z_{i},\theta)z_{i}z_{i}^{\top}\right).$ Then we can calculate the influence function for $z_{\rm target}$ and $z_{\rm bar}$:

$$ $\displaystyle\mathcal{I}_{\theta}(z_{\rm target},Z_{\rm{test}})$ $\displaystyle=-\sigma(-\theta^{\top}\mathbf{e}_{1})\cdot\sigma(\theta^{\top} \mathbf{e}_{1})\cdot\mathbf{e}_{1}^{\top}H_{\theta}^{-1}\mathbf{e}_{1}=-\sigma (-\theta^{\top}\mathbf{e}_{1})\cdot\sigma(\theta^{\top}\mathbf{e}_{1})\cdot \frac{d+1}{r(\mathbf{e}_{1},\theta)}<0,$ $$

$$ $\displaystyle\mathcal{I}_{\theta}(z_{\rm bar},Z_{\rm{test}})$ $\displaystyle=-\sigma(-\theta^{\top}\mathbf{e}_{1})\cdot\sigma(\theta^{\top} \mathbf{e}_{1})\mathbf{e}_{1}^{\top}H_{\theta}^{-1}(-\mathbf{e}_{1})=\sigma(- \theta^{\top}\mathbf{e}_{1})\cdot\sigma(\theta^{\top}\mathbf{e}_{1})\cdot\frac {d+1}{r(\mathbf{e}_{1},\theta)}>0.$ $$

This complete the proof: $\forall\theta$, $\mathcal{I}_{\theta}(z_{\rm target},Z_{\rm{test}})<0<\mathcal{I}_{\theta}(z_{
\rm bar},Z_{\rm{test}})$ and therefore $z_{\rm target}$ can never achieve top-1.

The above construction can be generalized to top-$K$ for any $K>1$: we can construct $Z_{\rm test}$ as $\{(\mathbf{e}_{1},1)\}$, and construct $Z_{\rm train}=\{(z_{i},y_{i})|i=2,\cdots,d\}\cup\{(\mathbf{e}_{1},1),K\text{
duplicated }(-\mathbf{e}_{1},1)\}$ where $y_{i}$ can be arbitrarily selected from $\{1,-1\}$. Then similar to the proof above, by choosing $z_{\rm target}=(\mathbf{e}_{1},1)\in Z_{\rm train}$ and $K$ duplicated training point in the training set $z_{\rm bar}^{k}=(-\mathbf{e}_{1},1),k\in[K]$, we can prove $\forall\theta$, $\mathcal{I}_{\theta}(z_{\rm target},Z_{\rm{test}})<0$ and $\forall\theta,k\in[K]$, $\mathcal{I}_{\theta}(z_{\rm bar}^{k},Z_{\rm{test}})>0$. Consequently, $\forall\theta$, $z_{\rm target}$ will not achieve top-$K$.

Figure: Figure 7: Histograms for original ranks of easy-to-manipulate samples, Histograms for original ranks of hard-to-manipulate samples, Scatterplots for influence gradient norm vs. original ranks of the 50 random target samples. Ranking $k:=1$. Easy-to-manipulate samples have extreme original influence ranks (large positive or negative) as the samples with the extreme rankings also have higher influence gradient norms, where the gradient is taken w.r.t. model parameters.
Refer to caption: x10.png

### A.2 Fairness Manipulation Attack Details

Figure: Figure 8: Overview of the process to achieve a fair model, as proposed by []. Our adversary, the model trainer manipulates the base model used to calculate influence scores.
Refer to caption: x24.png

Optimization Problem for Reweighing Training Data as proposed by Li and Liu [] is given as follows,

$$ $\begin{array}[]{ll}\operatorname{minimize}&\sum_{i}w_{i}\\ \text{ subject to }&\sum_{i}w_{i}\mathcal{I}_{\text{fair }}\left(z_{i}\right)= -f_{\text{fair }}^{\mathcal{V}}\\ &\sum_{i}w_{i}\mathcal{I}_{\text{util }}\left(z_{i}\right)\leq 0\\ &w_{i}\in[0,1]\end{array}$ (6) $$

where $w_{i}$ refers to the weight of the $i^{th}$ training sample, $z_{i}$ refers to the $i^{th}$ training sample, $\mathcal{I}_{\text{util }}$ refers to our influence function, $\mathcal{I}_{\text{fair }}$ refers to some fairness influence function and $f_{\text{fair }}^{\mathcal{V}}$ corresponds to a differentiable fairness metric. In our threat model, the adversary manipulates the base model, which changes the influence scores $\mathcal{I}_{\text{util }}$.

An advanced version of the above optimization problem using additional parameters $(\beta,\gamma)$ which lead to various tradeoffs is given as,

$$ $\begin{array}[]{cl}\operatorname{minimize}&\sum_{i}w_{i}\\ \text{ subject to }&\sum_{i}w_{i}\mathcal{I}_{\text{fair }}\left(z_{i}\right) \leq-(1-\beta)\ell_{\text{fair }}^{\nu},\\ &\sum_{i}w_{i}\mathcal{I}_{\text{util }}\left(z_{i}\right)\leq\gamma\left(\min _{\mathbf{v}}\sum_{i}v_{i}\mathcal{I}_{\text{util }}\left(z_{i}\right)\right), \\ &w_{i}\in[0,1].\end{array}$ (7) $$

Fairness Metric. We define the fairness metric used in our paper, Demographic Parity.

###### Definition 2 (Demographic Parity Gap (DP) [ 12 ] )

Given a data distribution $\mathcal{D}$ over $\bar{\mathcal{X}}\times\{0,1\}$ from which features $x^{\setminus a}$ and sensitive attribute $x^{a}\in\{0,1\}$ are jointly drawn from, Demographic Parity gap for a model $f_{\theta}$ is defined to be the difference in the rate of positive predictions between the two groups, $|\operatorname{Pr}(\hat{y}\mid x^{a}=0)-\operatorname{Pr}(\hat{y}\mid x^{a}=1)|$ where $\hat{y}$ is the prediction $f_{\theta}(x^{\setminus a},x^{a})$.

**Table 3: Details for datasets used in our Fairness Manipulation attack experiments. In our setting, the Validation set is the test set shared between the model trainer and influence calculator. The Test set is an untouched set which is used to assess the performance and fairness of the final model achieved after the training with the reweighed training set. All our results are reported on this untouched Test set. Table details borrowed from [].**
| Dataset | Predict | Train / Val. / Test Split | #Dim. | Sensitive Attribute | Group Pos. Rate |
| --- | --- | --- | --- | --- | --- |
| Adult | if annual income $>=50$k | $22622/7540/15060$ | 102 | Gender - Male / Female | $0.312/0.113$ |
| Compas | if defendant rearrested in 2 yrs. | $3700/1234/1233$ | 433 | Race - White / Non-white | $0.609/0.518$ |
| German Credit | good/bad credit risk | $600/200/200$ | 56 | Age - 30 | $0.742/0.643$ |

**Table 4: Details for the Fairness Manipulation attack experiments. We use the same values of parameters as used by [].**
| Dataset | Model $\ell_{2}$ reg. | DP $(\beta,\gamma)$ |
| --- | --- | --- |
| Adult | 2.26 | (0.8,0.3) |
| Compas | 37.00 | (0.3,0.1) |
| German Credit | 5.85 | (0.5,0.0) |

###### Definition 2 (Demographic Parity Gap (DP) [ 12 ] )

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. Regarding the first point mentioned above, how practical is this scenario in the real world?

2. Can changes in the influence scores be achieved on the data provider's side without needing access to the model or altering the training procedure? I assume this might be related to adversarial attacks and data poisoning literature; however, it would be beneficial to discuss this from the perspective of influence functions.

3. How does the scale of the data affect the difficulty of the task? Based on Figure 2, it appears that as the training data size increases, the task of changing the influence scores becomes more challenging. Given the trend that more data generally improves performance, does a larger dataset also make influence score modification more difficult?
soundness: 2
strengths: 1. The paper is easy to follow, and the results and methodology are clearly presented in tables and figures/diagrams.

2. The overall idea of studying vulnerabilities in influence functions is interesting, especially as there are some applications built on using influence scores. However, I have some concerns, which are discussed in the following points.
summary: The paper studied a setup in which the influence scores of training samples can be manipulated through a modified training procedure by changing the loss function to maximize the influence scores of target samples on a subset of test samples. The paper aims to show that there could be vulnerabilities in influence scores that affect downstream applications, such as those related to fairness.
weaknesses: 1. The first and most important point to raise is that the attack setup does not seem practical in the real world. The paper provides a framework that includes two components: 1) Data Provider and 2) Influence Calculator. It is assumed that the data provider is truthful, while the Influence Calculator, which trains a "model" and calculates the influence scores, is maliciously manipulated. I can't think of a real-world scenario in which an Influence Calculator would deliberately aim to manipulate influence scores or would need to alter the influence score to indirectly affect the desired behavior, especially when it can directly train a model and is considered the target (or deployed) model. The reverse scenario seems more plausible, where a Data Provider among all providers aims to change the influence score distribution based on its own interests. Although the paper attempts to justify a scenario involving "fairness manipulation," this does not seem plausible to me. To modify the fairness of a model, it would need to be trained in a way that specifically adjusts the influence scores; however, there are more straightforward ways to alter this dynamic, specially when the training procedure can be altered.

2. Another key issue is that altering the influence scores distribution requires access to the test set to manipulate the top-k scores. This approach is problematic, as it risks causing the modified influence scores to overfit to the specific test set. This tendency is evident in Figure 2, where the "transfer" results (that uses held out test set) are notably less impactful, suggesting that the changes in influence scores are highly dependent on the specific test set used in the optimization process.

3. While the paper discusses it, there seems to be a key limitation in the "Multi-Target Attack" setup: modifying the influence scores for a subset of samples is not an independent process for each sample. The authors attempt to mitigate this by employing multiple random initializations; however, this solution appears to be costly.

4. The current version of the method appears to be computationally feasible only for linear models, which further limits the practicality of the methodology (in addition to my first point). However, there has been significant progress in making influence score calculations feasible for non-linear and large models, such as in "TRAK: Attributing Model Behavior at Scale" (https://arxiv.org/abs/2303.14186), which are more compelling for study.

5. The writing style is not ideal; too many paragraphs begin with a bold sentence or title. While this may be acceptable to some readers, I found it somewhat confusing to determine what is truly important. I would suggest that the authors use bold text sparingly, reserving it for points that are especially significant.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: See weaknesses
soundness: 2
strengths: - The paper is well-written and well-organized.
- The paper provides a practical and efficient algorithm to compute the backward pass through Hessian-Inverse-Vector Products for influence-based objectives. 
- The experimental results in the sources show that influence-based attacks, both targeted and untargeted, can successfully manipulate influence scores while maintaining model accuracy.
summary: This paper examines the potential for manipulating influence-based attributions. The authors explore two use cases: data valuation and fairness. In data valuation, an attacker aims to inflate the influence scores of specific data samples to increase their perceived value. In fairness, the attacker seeks to reduce the fairness of a downstream model by manipulating influence scores used for data reweighing. The paper presents targeted and untargeted attack algorithms, demonstrating their effectiveness on logistic regression models trained on ResNet features and standard fairness datasets. The authors also present a theoretical impossibility theorem.
weaknesses: - The setting of this working is confused to me. It presents a scenario where an adversary can manipulate the model training process to achieve desired influence scores. Is it a realistic scenario in practice and why would an adversary manipulate the influence score when he could manipulate the model itself?
- The method aims to manipulate the influence scores while maintaining the similar test accuracy, and this accuracy is measured by a `dist` function between the parameters and limited by a radius value $C$. However, the choice of $L_2$ norm to measure the distance between two high-dimensional vectors is not always consistent. For example, one can construct another network by multiplying all the parameters by $10^6$ and adding a normalization on each layer, which gives a same result but the $L_2$ distance will be much different. The authors should provide more justification for this choice.
- The experiments described in the paper primarily focus on logistic regression, which may not be representative of more complex models. Will the proposed attack methods be effective on more complex models? Is the objective function defined based on the `dist` function still valid due to the above reason?

---

confidence: 4
contribution: 1
initial_rating: 1
presentation: 2
questions: Please see the weaknesses listed above-- each can be considered as a question.
soundness: 1
strengths: - The paper conducts experiments on multiple datasets.
- The concept of adversarial attacks on influence scores is an interesting idea overall.
summary: The paper proposes an adversarial attack where an adversary seeks to train a malicious logistic regression model which attains similar performance to the original model but has a different distribution of influence estimates. The authors conduct experiments for the downstream tasks of data valuation and fairness.
weaknesses: In my opinion, the paper possesses major limitations that invalidate its contributions. My major concern revolves around the proposed threat model itself, which seems untenable for real-world practical applications. Other issues are concerned with limited model and dataset evaluation, and the overall simplicity of the work:

- **Unjustifiable Threat Model**: The threat model makes multiple assumptions about the influence (and data valuation) problem pipelines in order to make the attack viable. First, it assumes that the data is a trivial commodity and that this will be provided as is by the data providers to an influence "calculator" who themselves are the adversary. I find this to be highly implausible-- data is generally protected and it is more likely for the data owners / model trainers to be benign as model training is a computationally costly process. It seems highly unlikely (especially if the influence calculator can potentially be an adversary) that the data providers will provide highly sensitive data to this third-party as its outputs cannot be guaranteed to be trustworthy (i.e. anyone can train a model as they would like-- why would they trust them and their outputted scores?). Second, the threat model assumes that the influence calculator who is training the model possesses malicious intent but the reasons for this are unclear. As the influence calculator is taking on the overhead of model training, what is their incentive in doing so? They are training the model to achieve similar performance on the test set (as the paper also measures) and hence, are likely going to serve it for potential production use. Why would they change the influence scores for their own model to compensate data providers differently while incurring the cost of another model retraining then? Their main incentive should lie in ensuring robust model performance for end-users. Perhaps these aforementioned issues can be justified with some real-world examples or use-cases, but it is not motivated sufficiently in the current version of the paper and I do not think this setting is tenable. 
- **Limited Models and Datasets**: The paper only considers a simple logistic regression model in experiments and all the datasets are tabular datasets obtained via embeddings from deeper models. Why is logistic regression a good base model for data valuation when the task itself might require a deep learning model that is non-convex? Overall, this is very restrictive as a setting and the paper needs to consider end-to-end models for experiments. 
- **Simplicity of Attack**: Owing to the assumptions made in the threat model, the attack is very simple-- the approach just requires training a model with an adversarial objective while putting a constraint on performance. This is technically quite trivial and somewhat lower than the bar for ICLR.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to weaknesses.
soundness: 3
strengths: 1. I find the exploration of weaknesses in influence attribution an interesting phenomenon with practical application support, as highlighted in the proposed applications.
2. The authors provide theoretical analysis, demonstrating that under certain conditions, the influence scores of specific data samples cannot be manipulated.
summary: This paper investigates the vulnerability of influence functions in adversarial settings. Influence functions are commonly used to assess the contribution of training data to model predictions, but they may be susceptible to manipulation in incentivized applications like data valuation and fairness. The authors propose two types of attacks: a targeted attack designed to increase the influence score of specific samples to raise their data valuation, and an untargeted attack that alters influence scores through simple model weight scaling, affecting model fairness. Experimental results show that these attacks can be effectively implemented on multiclass logistic regression models and standard datasets without significantly compromising model accuracy. Additionally, the paper presents an optimization method for inverse computation of influence functions to enhance memory and time efficiency. Finally, the authors propose possible defense strategies and highlight the limitations of influence functions in adversarial environments.
weaknesses: 1. The paper references the instance attribution method from [1] (which relies on local model parameters), but there are currently many new instance attribution methods (incorporating model parameters from different training stages). The paper should at least discuss these instance attribution methods (as in "Training data influence analysis and estimation: a survey") to rule out that the vulnerabilities are due to flaws in the method itself.
2. Many logical details are placed in the appendix, which makes reading somewhat challenging. I believe it would be more reasonable to include core logic in the main text and place some logical explanations in the appendix, especially Section A.1 (this does not affect my scoring decision).
3. From reading the application, I can clearly understand the significance of this task, but I cannot precisely grasp how to achieve such a goal. The last formula on page 5 appears to be an intuitive attack method without proof of achieving the intended objective.
4. I believe the research task itself is meaningful but lacks discussion on the more advanced instance attribution methods [2]. If these methods could be included in the discussion, I would be more inclined to raise my score.
[1] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pp. 1885–1894. PMLR, 2017
[2] Training data influence analysis and estimation: a survey

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 2
questions: 1.	Why does $dist(\theta, \theta') <= C$ in Eq. (4) guarantee that the original and  malicious  models have similar accuracy? Given that different models with the same structure may have varying parameter magnitudes, how to determine the hyper-parameter $C$?

2.	Why does Eq. (3) compute the influence function of a training sample z on all test samples? Will the sum of the influence scores for all test samples cancel each other out, both positively and negatively?
soundness: 1
strengths: The authors investigated the vulnerability of influence-based attributions, which was an interesting direction.
summary: This paper proposed an attack to manipulate influence-based attributions by training a malicious model with slightly reduced accuracy, while achieving the attacked influence scores. Specifically, the authors aimed to find a malicious model $\theta’$ that maintained a similar accuracy to the original model $\theta$, which maximizing the influence score of the target sample. The authors also discussed the potential scenarios for the proposed attack, including data valuation and fairness use cases.
weaknesses: 1.	The main weakness was that the threat model on influence score was problematic. The influence score $I_\theta (z, z_{test})$ was computed based on a given model $\theta$ according to Eq. (1). When different model parameters $\theta$ were given, the influence score of a training point $z$ on the loss at a test point $z_{test}$ $I_\theta (z, z_{test})$ would indeed yield a different score. In other words, the discussion of the influence function was only meaningful within the context of the same model. Changes in model parameters would inevitably lead to variations in influence scores. Therefore, these variations should not be considered an attack on the influence function.

2.	The proposed attack scenarios were not reasonable. For instance, in data valuation tasks, why would an adversary be able to modify the model parameters? If the goal was to evaluate the importance of data for a specific model, the model parameters should remain fixed. For example, with large models like GPT-4, the adversary would not be able to alter the model parameters. Furthermore, if multiple models were used to evaluate the importance of data, why would the evaluator choose a malicious model over a set of trusted models? In other words, if the adversary cannot ensure that the evaluator uses the malicious model, then the proposed attack becomes impractical.

3.	The proposed attack constrained the L2 norm of the two parameters in Eq. (4) and in Lines 250-251 to ensure that the accuracies of the two models are similar. However, there is a lack of evidence that the L2 norm of parameters is related to the performance of the model. For example, consider a two-layer network with two categories $f(x) = Sigmoid(c\cdot Ax)$, where $c>0$. If $f(x)>0.5$, the class is 1; otherwise, the class is 0. In this case, changing the L2 norm of the parameters $B=c\cdot A$ (i.e., modifying $c$) does not affect the model's performance.

Meta-review: I have read all the materials of this paper including the manuscript, appendix, comments, and response. Based on collected information from all reviewers and my personal judgment, I can make the recommendation on this paper, reject. No objection from reviewers who participated in the internal discussion was raised against the reject recommendation.

**Research Question**

The authors study the manipulation of data valuation. The data valuation depends on the learning algorithm, where the authors assume the data and learning algorithm are controlled by different agents. The targeted scenario is unrealistic. Thus, there is no necessary to comment other parts of this paper.

The review team including all reviewers and me, believes this paper does not meet the bar of ICLR.

Decision: Reject