Manipulating SGD with Data Ordering Attacks

Abstract: Machine learning is vulnerable to a wide variety of attacks. It is now well understood that by changing the underlying data distribution, an adversary can poison the model trained with it or introduce backdoors. In this paper we present a novel class of training-time attacks that require no changes to the underlying dataset or model architecture, but instead only change the order in which data are supplied to the model. In particular, we find that the attacker can either prevent the model from learning, or poison it to learn behaviours specified by the attacker. Furthermore, we find that even a single adversarially-ordered epoch can be enough to slow down model learning, or even to reset all of the learning progress. Indeed, the attacks presented here are not specific to the model or dataset, but rather target the stochastic nature of modern learning procedures. We extensively evaluate our attacks on computer vision and natural language benchmarks to find that the adversary can disrupt model training and even introduce backdoors.


Reviews and author responses:

The paper presents a training-time attack against learning algorithms employing stochastic-gradient-descent (SGD) update. The attacks are designed as simply re-ordering the sequence of batches or individual data points during training.   The idea of re-ordering the input sequence to influence the learning outcome is interesting. It exploits fully the underlying workings of stochastic gradient descent. Preliminary studies show promising results.  Although the proposed attack strategy sounds exotic, I have several doubts about its practicality and rationality:

To carry out such an attack, the authors assume the adversary can fiddle with the OS, sampling software, etc. to not only have full access to the training data but its ordering. If the adversary has such an enormous power, why wouldn't he/she directly tamper the learned parameters after training?  It would be more reliable than the proposed attack that may not always succeed. 

The authors only consider impotent learning models. Would machine learning models remain defenseless if they are aware of such attacks? How about random skipping/shuffling/dropout of batches or samples during training? How about using random batch sizes during training? Did you try different drop-out rate in your DNNs? 

The authors claim "large prediction errors correspond to large loss gradient norms, whereas correct predictions produce near-zero gradients", is this assumption reliable given that there might be many possible local optima?


 In my opinion, Appendix C is too important to be left out from the main text. 

Never mind the additional experiments, there's not much time for that. Thanks for the diligent response to all my questions. 

 Thank you very much for your comments. 
```
I believe, by reordering the input, you have transformed a standard classification problem into a temporal dynamic behavior learning problem, at least in a time window of the size of the training data. I suspect the attack won't be successful if a neural network incorporates time delays or has feedback loops, such as RNN or FNN.
```
You are indeed correct -- the local pooling defence strategy described in the comment above is suggesting that temporal manipulation does have an effect. Yet, it appears that the pool required needs to be rather large. Similarly we expect that since RNNs are incorporating the temporal dimension, performing the attack in the current form will be more difficult. However, the attack we presented could also be adapted - e.g. using a variant of the oscillations-in attack policy to explicitly target the temporal dependency.  Most importantly, in Appendix C we derive the conditions under which such a family of sampling-based attacks would succeed from a purely optimizational point of view, meaning it is not just a practical finding, but it is also theoretically justified. 

```
My other concern is it is possible you have not allowed the network to have a sufficient amount of time/epochs to train to converge.
It's too much to ask you to repeat the experiments with RNN or FNN. But it would be helpful if you could increase the number of epochs to a much larger number (given the limited time left, maybe just on one simple dataset), and provide the learning curve of each epoch.
```

Which of the experiments were you considering to be more precise? Figure 21(Appendix I) on page 34 shows 100 epochs for CIFAR10 with different attack strategies. 


 Here are my other concerns.

I believe, by reordering the input, you have transformed a standard classification problem into a temporal dynamic behavior learning problem, at least in a time window of the size of the training data. I suspect the attack won't be successful if a neural network incorporates time delays or has feedback loops, such as RNN or FNN.

My other concern is it is possible you have not allowed the network to have a sufficient amount of time/epochs to train to converge. 

It's too much to ask you to repeat the experiments with RNN or FNN. But it would be helpful if you could increase the number of epochs to a much larger number (given the limited time left, maybe just on one simple dataset), and provide the learning curve of each epoch. 

 Thank you for acknowledging receipt of our response. Let us know if you have further concerns that remain unaddressed or if the current responses address all of the feedback in your original review. Thank you again for your time.

 Thank you for taking the extra effort to provide additional empirical results.  

 Dear Reviewer BGiu,

```
Since you've realized that there are a number of ways to protect the model, you should employ them (at least some) in your empirical study instead of simply using the standard defenseless models. It may make your case weaker, but it will improve the quality of your paper significantly.
```

We have now run and evaluated the idea of a local pool defence against the attacks shown in our paper. The defence accumulates a pool of local samples in the training loop before using them for training. When the pool is filled with samples, the defender then samples uniformly from the pool and uses the samples in training, until the pool is exhausted. 

We find that this procedure disrupts single data-poisoning consistently for pool sizes larger than 100 for the VGG poisoning described in Fig 8 in Appendix F. Yet, for the backdooring setup, we observe that the same procedure has a somewhat unexpected effect. The backdoored batches work (i.e. cause backdoor to work over the test set) on average better for pools of smaller sizes than it would without the defence, but this effect diminishes as pools grow beyond 250 batches. We hypothesise that this performance is a result of gradients being ‘diluted’ across the batches but not removing the reordering effect completely; especially so towards the end of training, where one expects to see near-zero gradients for uniformly sampled natural data. We will add all of the experimental results to the appendix alongside graphs for individual batch performances. We want to explicitly thank you for suggesting this experiment, it has indeed demonstrated that disruption of BOP attacks is more complex than originally anticipated and small changes to the setup attempting to defend against BOP attacks could instead make the attacks succeed more often. 

```
If you have the assumed hacking power, you could simply hijack the model loading process and inject the pre-trained malicious model parameters instead of going through all the troubles of re-arranging training sample orders.
```

We want to re-emphasise that we are not assuming hacking power, we assume an attacker who is capable of changing data order, which can be done with fewer capabilities, for example by tampering with the data supply pipeline. Given that data supply engineering and model training are separate tasks often managed by separate teams, data supply engineers will not necessarily have an ability to change model weights stored elsewhere. 


 Thank you for clarifying these points. I don't have further questions and will keep my current score.

 Thank you for your clarification.

Since you've realized that there are a number of ways to protect the model, you should employ them (at least some) in your empirical study instead of simply using the standard defenseless models. It may make your case weaker, but it will improve the quality of your paper significantly.

If you have the assumed hacking power, you could simply hijack the model loading process and inject the pre-trained malicious model parameters instead of going through all the troubles of re-arranging training sample orders. 

 

We thank the reviewer for the insightful feedback. 
```
I glossed over the provided source code and it seems like the attack is implemented by manipulating the model's training code. This is fine for research but, for example, hacking into PyTorch's source code to implement this attack would yield a more realistic attack.
```
Our current implementation wraps around a PyTorch DataLoader, meaning the attacker only needs to link in the implementation and put it around the loader itself. When attacking a real system, one would hide in the CDN, the OS or the hardware. We will expand our description of the attack requirements and note the restrictions imposed on the attacker. We do want to reiterate that our work is the first to demonstrate that natural data and natural labels can be used to backdoor models and stop them from training. 

```
This raises the question, given these capabilities why would the attacker opt for merely manipulating the SGD batches? I would recommend the authors to find a tight set of capabilities and implementation of this threat model that leaves no other room for the attacker.
```

Our attack exploits a data pipeline right before data reaches the model. Given the relative sophistication of the pipeline, the threat is notoriously hard to analyse and debug. This is because the only observable consequence is a drop in model performance, which makes it difficult to debug the source of the problem as illustrated recently for instance with the discovery of a bug in numpy’s pseudo randomness implementation for data sampling (Tanel Pärnamaa, Using PyTorch + NumPy? You're making a mistake.).  

When we shared a preprint of the paper with a relatively large email provider, we were told that they changed their data processing practices, as their online spam detection tool trained on the data in order as it appears in the system. Similarly, history has shown attacks launched by sophisticated attackers against randomness used for example in cryptographic applications (e.g. Dual_EC_DRBG). 

```
Calling this a black-box attack stretches the meaning of black-box a little too much. The attacker sees the training set after all, the only difference is that they see it during training instead of seeing it prior to the training. Considering that this attack manipulates the attack during training, instead of before training like traditional poisoning attack, I don't think we can call it black-box.
```

It seems that our description of the setting has been slightly confusing. We call the attack blackbox based on the access to the model under attack and not the dataset access. The dataset is always discovered in the first epoch. We will update the setting description to reflect on that and make sure that our notation is compatible with the rest of the literature. 


 
Thank you for your analysis of our paper.
```
Although the proposed attack strategy sounds exotic, I have several doubts about its practicality and rationality:
```
We want to highlight that our paper is the first one to show that sampling procedures can be manipulated to launch a variety of different attacks against the model’s availability and integrity. As we demonstrate in the paper, modern training routines are susceptible to our attacks, even in a black-box setting. Indeed, we show the best possible scenario for backdooring -- we can backdoor models with no changes to the underlying data or labels.  
```
The authors only consider impotent learning models. Would machine learning models remain defenseless if they are aware of such attacks? 
```
As we note in the paper, there are a number of ways to protect the model. Some are trivial such as shuffling data (as is assumed by SGD) or subsampling data locally before performing a pass; some solutions are more sophisticated, such as SVRG or byzantine fault tolerant SGD. The purpose of the paper is to demonstrate the importance of randomness on SGD and we realise that defending in practice is not that easy. While it may be possible to conceive a defense strategy, implementing it may be difficult because of computational limits: one has to resort to caching and local shuffling, which would go in conflict with many of current decentralized solutions, where it's impossible to store data locally. 

```
If the adversary has such an enormous power, why wouldn't he/she directly tamper the learned parameters after training? It would be more reliable than the proposed attack that may not always succeed.
```
The comment on the attack’s relative complexity is fair -- indeed it may appear to be easier at first glance to perform the same attack by, for example, actively poisoning the dataset or arbitrarily changing the model weights. Yet these efforts also suffer in reality -- most modern security is governed by anomaly detection mechanisms and it's often easy to find data integrity issues.  Our attacks on the other hand are hard to detect, use data with preserved integrity, and are harder to find retrospectively. 
```
The authors claim "large prediction errors correspond to large loss gradient norms, whereas correct predictions produce near-zero gradients", is this assumption reliable given that there might be many possible local optima?
```
Finally, we agree with the reviewer that the statement "large prediction errors correspond to large loss gradient norms, whereas correct predictions produce near-zero gradients” may not hold for *all* loss landscapes. However, for loss landscapes incurred by machine learning models, local minima oftentimes cluster around the global minima as the loss satisfies certain dissipativity properties [3-4], and consequently local minima inherit nice statistical properties from the global [1-2]. We emphasize that we use this in practice only for computational advantage.

[1] Kenji Kawaguchi, Deep learning without poor local minima
[2] Rong Ge, Jason D Lee, and Tengyu Ma, Matrix completion has no spurious local
Minimum
[3] Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. "Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis." 
[4] Lu Yu, Krishnakumar Balasubramanian, Stanislav Volgushev, Murat A Erdogdu. “An analysis of constant step size sgd in the non-convex regime: Asymptotic normality and bias.” 


 
Thank you very much for your feedback! 
```
… the authors might consider rephrasing the caption of Fig.4 to make this clearer.
```
We can see where the confusion is coming from, we will redo the caption of Figure 4 to emphasise that we are only manipulating the order and not the contents of the batches themselves. 
```
… It would be interesting if the authors could extend their explanations of the connections between the proposed attack policies and related work in curriculum learning (in addition to the references mentioned in l142)
```
We will expand the discussion of the connection in the appendix. If one drops the assumption of using the whole dataset our attacks become an inverse of currently deployed curriculum learning routines such as self paced learning [1] or other flavors of active learning especially in our blackbox setting e.g. [2-3]. Anti-curriculum learning (e.g. [4]) is particularly close to our batch reordering setting. 
[1] Kumar et al. Self-Paced Learning for Latent Variable Models
[2] Tong and Koller. Support Vector Machine Active Learning with Applications to Text Classification
[3] Cohn et al. Active learning with statistical models
[4] Hacohen and Weinshall On The Power of Curriculum Learning in Training Deep Networks
```
I am not certain I understand/agree with the terminology of the authors in Sec 4. Sec. 4.2 shows integrity attacks. However the authors understand integrity attacks to be "attacks that reduce test accuracy" and availability attacks to be"attacks that reduce training accuracy (temporarily)", to my understanding (and the remark in l218). In contrast, other works consider integrity attacks to reduce the accuracy of some target trigger / data point without impacting model test accuracy and availability attacks to reduce model test accuracy. This is especially confusing to me because the examples quoted in Sec.2 as integrity attacks (such as Shafahi) fall under the second nomenclature and attacks as in Shafahi are only presented in Appendix F (?). If the authors want to keep their current terminology, it would be helpful to add additional clarification in both Sec. 4 and Sec. 2.
```
We want to also explicitly thank you for bringing up the different flavours of poisoning objectives and potential confusion that can come with our naming scheme. In this work we assumed that availability objective offsets progress of benign training, whereas integrity affects learnability of a given task in the first place. We will reword the section and the rest of the paper to become compatible with the rest of the literature. 

```
… Some more information about the limitations or the "overhead" required for the attack in terms of memory and computational requirements could help inform assessment of the currrent strength of the attack, for example in scenarios where the attack is running hidden on the same machine that runs normal training.
```

That would indeed be a nice way to demonstrate method performance! We will add a discussion and a graph to the appendix that describes the reconstruction fidelity as a function of performance overheads. 


This paper proposes hurting a deep neural network's accuracy, or its training time or introducing poisoning/backdoors only by manipulating the random batching in SGD algorithm. In these attacks, the attacker has the ability to observe the training samples during training and then either re-shuffle, re-order or replace the batches that were supposed to be randomly samples. They experiment on image classification and an NLP task to show that these attacks are effective without manipulating the individual samples at all, by just changing the order in which the model sees them during training.  Poisoning attacks are becoming more and more relevant and this paper shows that the attacker can cause a poisoning-type effect by manipulating the sampling order during SGD training. Attacking SGD (which is almost universal in deep learning) creates a large attack surface and the paper demonstrates various threat models implemented by the same attack idea. 

Strengths:
+ Relevant, new and novel threat model.
+ Good empirical evaluation and formal reasoning to why this attack is feasible.
+ The idea of secure randomness in differential privacy and encryption does not exist for deep learning, including SGD, random initialization or random data augmentation. Attacking the randomness itself is not a new idea outside deep learning (e.g., backdooring the Elliptic Curve random generators). This paper shows that maybe this is something we should care about in deep learning as well, or at least include such risks in our threat model specifications.

Weaknesses:
- Conjectures how such attack can be implemented but does not offer a realistic implementation. I glossed over the provided source code and it seems like the attack is implemented by manipulating the model's training code. This is fine for research but, for example, hacking into PyTorch's source code to implement this attack would yield a more realistic attack. I would assume this also brings about additional restrictions on the attacker. I think the ways the authors suggested give the attacker for a much larger capability than required by this attack. For example, by compromising the OS file system the attacker can do basically anything (swap the model at the end of training with a attacker-trained one?). This raises the question, given these capabilities why would the attacker opt for merely manipulating the SGD batches? I would recommend the authors to find a tight set of capabilities and implementation of this threat model that leaves no other room for the attacker.

- Calling this a black-box attack stretches the meaning of black-box a little too much. The attacker sees the training set after all, the only difference is that they see it during training instead of seeing it prior to the training. Considering that this attack manipulates the attack during training, instead of before training like traditional poisoning attack, I don't think we can call it black-box. For example, is an attack where the attacker does not see the training samples themselves but sees the model's individual losses on each sample during loss averaging can be implemented? My point is bringing the proposed attacks to prior black-box attacks might better justify calling them black-box.
 (see above for recommendations)

The submission "Manipulating SGD with Data Ordering Attacks" evaluates a training time attack against ML models with a novel threat model: The order of data points (/order of batches/ order of data points within batches/batch composition) is adversarially modified to achieve a variety of malicious outcomes. The authors discuss their implementation of this attack in a variety of scenarios, and show applications for neural networks in image classification and text.  Overall, I am a fan of this submission. The threat model of adversarial batch ordering appears novel and interesting to me. There has been a growing interest in data poisoning(i.e. training time attacks) in recent years and this is a worthwhile attack scenario consider in the literature. The authors describe the setting well and provide good heuristic strategies for attacks that replicate backdoor attacks as well availability attacks in this threat model. It is especially interesting that the authors were able to show that such attacks are possible using smaller surrogate models.



Some questions and minor comments:
* The first time I saw Fig.4 I was confused whether the paper was actually about reordering of the internal structure of data points. I understand that this is not the case, data points themselves are unchanged, but the authors might consider rephrasing the caption of Fig.4 to make this clearer.
* It would be interesting if the authors could extend their explanations of the connections between the proposed attack policies and related work in curriculum learning (in addition to the references mentioned in l142)
* The discussion about gradient matching in Eq.(4) has interesting connections to perturbation-based poisoning attacks based on gradient matching (such as e.g. Geiping et al, ...Data Poisoning via Gradient Matching or influence function-based analysis in Koh&Liang, Understanding Black-box Predictions via Influence Functions)
* I am not certain I understand/agree with the terminology of the authors in Sec 4. Sec. 4.2 shows integrity attacks. However the authors understand integrity attacks to be "attacks that reduce test accuracy" and availability attacks to be"attacks that reduce training accuracy (temporarily)", to my understanding (and the remark in l218). In contrast, other works consider integrity attacks to reduce the accuracy of some target trigger / data point without impacting model test accuracy and availability attacks to reduce model test accuracy. This is especially confusing to me because the examples quoted in Sec.2 as integrity attacks (such as Shafahi) fall under the second nomenclature and attacks as in Shafahi are only presented in Appendix F (?). If the authors want to keep their current terminology, it would be helpful to add additional clarification in both Sec. 4 and Sec. 2.
* Connected to the previous point: Am I correct in my understanding that the experiment in Sec. 4.3 boils down to the same attack   as 4.2, but attacking only once (at epoch 10)? If this is the only difference, then I am not sure I like the titling of both subsections as integrity vs availability?
* typo in l215 Limitations:

The authors briefly discuss this, but I would enjoy further discussion on the practical requirements of the attack (e.g. in the appendix) in terms of computation time and memory. For example, the authors could record the number of loss evaluations required to find a good adversaries during the random sampling in Sec. 3.4, or the computational requirements for loss computations on the small surrogate networks in comparison to the forward+backward passes of the victim model. Some more information about the limitations or the "overhead" required for the attack in terms of memory and computational requirements could help inform assessment of the currrent strength of the attack, for example in scenarios where the attack is running hidden on the same machine that runs normal training.


Societal Impact:

The paper analyzes a novel threat vector against machine learning pipelines. While the practical applicability of such an attack against a real-world system as theorized by the authors does require a significant amount of system access, it does remain a  possibility and is a threat model that should be openly discussed in the literature and taken in consideration in future designs of system-critical machine learning systems.

Meta-review: This paper presents a novel attack on SGD trained models by identifying that one can adversarially change the order of samples to significantly decrease the accuracy of the model. This is a rather surprising finding given the fact that neither the data, or labels are perturbed in any way. Although such technique would not be successful in the convex case (apart from slowing down convergence), in the nonconvex case adversarial perturbations can lead to significant loss of accuracy. Although the presented results are interesting, the attack model needs to be clarified a bit with regards to the strength of the adversary. Assuming access to the order of data, likely implies a signifiant adversarial strength with regards to access to the software stack of the training system at hand. Nevertheless the observations made are important, highlighting yet another facet of attacks against high-capacity SGD-trained models.



Decision: accept