SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. With an upper bound of  $\tau$  for the duration, an attack cannot affect simultaneously any two periods that are at least  $\tau$  periods away from each other, and therefore no two base classifiers that are at least  $\tau + n - 1$  away from each other can be both poisoned. Thus, if assuming  $f_s$  is the first base classifier affected by the attack,  $f_j$  with  $i - k + 1 \leq j \leq s - 1$  or  $j \geq s + \tau + n - 1$  must be free from any impact of the attack, and therefore the left-hand side of Inequality 2 is a natural lower bound for the number of base classifiers in the aggregation that predicts label  $y$ . In addition, since no two base classifiers that are at least  $\tau + n - 1$  away from each other can be both poisoned,  $\sum_{j = i - k + 1}^{s - 1} \mathbb{1}[f_j(x) = y'] + \sum_{j = s + \tau + n - 1}^{i} \mathbb{1}[f_j(x) = y'] + \min(\tau + n - 1, k)$  on the right-hand side of Inequality 2 becomes an upper bound for the number of base classifiers in the aggregation that predicts label  $y'$ . Again,  $\mathbb{1}[y > y']$  is due to the tie-breaking rule of temporal aggregation. Consequently, we know that Inequality 2 is true for a certain  $y'$  and a certain  $s$  suggests  $\mathrm{TA}([D_0', \ldots, D_i'], x) \neq y'$ , assuming the first base classifier affected is  $f_s$ . Since it is true for all  $y' \neq y$  and all  $s$ , we have  $\mathrm{TA}([D_0', \ldots, D_i'], x) = y$  and the proof is completed.
Implications and applications of Theorem 4.2 and 4.3: The key conditions used in these two theorems, Inequality 1 and 2, are both computationally easy given predictions from base classifiers: For any sample at inference time, by simply examining the predictions from base classifiers, one can obtain lower bounds for both earliness and duration of data poisoning attacks that stand a chance of changing the prediction from temporal aggregation.
In addition, it is worth noting that Theorem 4.2 and 4.3 do not assume whether  $D_0, \ldots, D_i$  are all clean or potentially poisoned. Given that in the temporal threat model of data poisoning,  $D_0', \ldots, D_i'$  is reachable from  $D_0, \ldots, D_i$  after some poisoning attack with a maximum earliness (or duration) of  $\tau$  if and only if  $D_0, \ldots, D_i$  is also reachable from  $D_0', \ldots, D_i'$  after some poisoning attack with a maximum earliness (or duration) of  $\tau$ , these theorems can be applied regardless of whether the training sets are considered clean or potentially poisoned: With clean training data, we obtain lower bounds of earliness and duration required for a hypothetical poisoning attack to be effective; With potentially poisoned data, we get lower bounds of earliness and duration required for a past poisoning attack to be effective. In this work, we apply them for evaluation of temporal aggregation on the temporal robustness benchmark from Section 3, where data are assumed poison-free.
(a) robust fraction with respect to bounded earliness
(b) robust fraction with respect to bounded duration
Figure 5: An overview of the performance and temporal robustness of temporal aggregation with with base coverage  $n \in \{1,3,6,9,12\}$  and aggregation size  $k \in \{6,12,18,24\}$ , a total of 20 combinations of hyperparameters. Detailed views of individual runs can be found in Figure 6 and Figure 7.
5 Evaluation of the Baseline Defense
Given that the temporal robustness benchmark simulates a setting with monthly updated models, we consider each month a separate period for temporal aggregation (i.e. one base classifier per month) and use 'month' as the unit of time. For the base learner (i.e. the learning of each base classifier), we use the pre-trained RoBERTa [17] with a BERT-base architecture as a feature extractor, mapping each headline into a vector with 768 dimensions. We optimize a linear classification head over normalized RoBERTa features with AdamW optimizer [18] for 50 epochs, using a learning rate of 1e-3 and a batch size of 256. To make the base learner deterministic, we set the random seeds explicitly and disable all nondeterministic features from PyTorch [24].
5.2 Establishing Baseline Accuracy
To establish baselines for assessing the performance and temporal robustness of temporal aggregation, we present in Table 2 clean accuracy, i.e. robust fraction from Definition 3.1 with  $\tau = 0$  of the base learner trained on varying months of latest data. Corroborating the effects of natural distribution shifts, among settings evaluated, the average clean accuracy over all categories reaches its peak when using the latest one month of data for training, thus we use the corresponding value,
$54.09\%$  as the baseline clean accuracy in our evaluation. In addition, here we observe that different categories can have different degrees of temporal distribution shifts and different data availability: For some categories including WORLD NEWS/POLITICS, training on earlier data reduces accuracy, indicating effects of temporal distribution shifts; For some categories including TRAVEL/COLLEGE, training on earlier data improves accuracy, indicating the effects from using more data.
Category	n = 1	n = 3	n = 6	n = 9	n = 12
all	54.09%	53.77%	53.47%	53.43%	53.15%
WORLD NEWS	59.00%	46.59%	36.79%	31.39%	25.56%
POLITICS	62.13%	57.52%	55.36%	54.31%	53.96%
TRAVEL	52.30%	56.65%	59.46%	61.63%	62.26%
COLLEGE	42.30%	49.30%	51.68%	54.20%	57.28%
Table 2: Clean accuracy of the base learner trained on the latest  $n$  months of samples.
5.3 Performance and Robustness
In Figure 5, we include an overview of the performance and robustness of temporal aggregation with base coverage  $n \in \{1, 3, 6, 9, 12\}$  and aggregation size  $k \in \{6, 12, 18, 24\}$ , a total of 20 combinations of hyperparameters. For clarity, we defer the study of hyperparameters to Section 5.4.
From Figure 5, we observe that temporal aggregation is capable of providing decent temporal robustness with minimal loss of clean accuracy. For example, with  $n = 1$  and  $k = 24$ , it is provably robust for a sizable share (with respect to the baseline clean accuracy) of test samples against poisoning attacks that started no more than  $8 \sim 11$  months in advance (earliness) and attacks that lasted for no more than  $6 \sim 9$  months (duration).
(a)  $n\in \{1,3,6,9,12\} ,k = 6$
(b)  $n\in \{1,3,6,9,12\} ,k = 12$
(c)  $n\in \{1,3,6,9,12\} ,k = 18$
Figure 6: The effects of base coverage  $n$  to the performance and temporal robustness.
(d)  $n\in \{1,3,6,9,12\} ,k = 24$
(a)  $n = 1,k\in \{6,12,18,24\}$
(b)  $n = 3,k\in \{6,12,18,24\}$
(c)  $n = 6,k\in \{6,12,18,24\}$
Figure 7: The effects of aggregation size  $k$  to the performance and temporal robustness.
(d)  $n = 9,k\in \{6,12,18,24\}$
5.4 Hyperparameters for Temporal Aggregation
The effects of base coverage  $n$ : In Figure 6, we compare the robust fraction of temporal aggregation with different base coverage  $n$ . Intuitively, the choice of  $n$  involves a trade-off between training set size and distribution shifts: A larger  $n$  increases training set size, which is positive to clean accuracy, but also enlarges distribution shifts, as more samples from early-time will be included, which is negative to clean accuracy. In this case, however, we observe a mild but negative impact from increasing  $n$  to performance, consistent with previous observations from Table 2, again indicating the presence of strong natural distribution shifts over time. Regarding temporal robustness, Figure 6 suggests that a larger  $n$  significantly reduces temporal robustness of temporal aggregation against poisoning attacks with bounded duration, which is expected as an attacker can affect more base classifiers within the same duration.
The effects of aggregation size  $k$ : In Figure 7, we present a comparison of temporal aggregation with different aggregation size  $k$ , where we observe that a larger  $k$  has a rather mild negative impact on performance but greatly facilitates temporal robustness with both bounded earliness and duration. This is because, with a larger  $k$ , more base classifiers will be utilized in the aggregation. On the one hand, these extra base classifiers are trained on earlier data with larger natural distribution shifts from
test distribution, hence the reduced clean accuracy; On the other hand, extra base classifiers dilute the impact from poisoned base classifiers to the aggregation, hence the improved robustness.
The proposed defense, temporal aggregation, is inspired by aggregation-based certified defenses for the traditional modeling of data poisoning, with a shared spirit of using voting for limiting the impacts of poisoned samples. [16, 39, 40, 26, 38] assume deterministic base learners and guarantee the stability of predictions with bounded numbers of poisoned samples. [14, 6] provides probabilistic guarantees of the same form with small, bounded error rates using potentially stochastic learners.
Temporal Aggregation introduces little-to-none extra training cost in many practical cases. In reality with continuous data collection, a common practice is to re-train/update models using new data every once in a while because model performance degrades as time evolves. As mentioned in Section 4.1, to use Temporal Aggregation, one still only need to train one classifier per updating period since previous base classifiers can be reused, which incurs minimal/no extra training overhead.
The increase of inference overhead can be small depending on the design of base classifiers. Naturally, using more base classifiers (i.e. using a larger  $k$ ) increases the inference overhead. However, the extra inference overhead can be small depending on the designs of base classifiers. An example design is to fine-tune only the last few layers of the model instead of training entirely from scratch. If one uses a shared, pre-trained feature extractor and only trains a task head for each base classifier (as in our Section 5), the shared feature extractor only needs to be forwarded once for inference and the extra overhead can be minimized.
Using both time and the number of poisoned samples to characterize attack budgets. While this work focuses on the benefits of incorporating temporal concepts, it is worth noting that earliness and duration can be combined with the number of poisoned samples in bounding attack budgets. As an example, our baseline defense Temporal Aggregation is not only a provable defense against attacks with bounded earliness and duration but also a provable defense with respect to a bounded number of poisoned samples (when using a small base coverage  $n$ ). This follows directly from previous aggregation defenses [16, 39] targeting attacks with the number of poisoned samples bounded.
Potential opportunities for other defenses. The temporal threat model of data poisoning may offer new angles to improve the usability of existing poisoning defenses for different stages. For detection-based defenses [33, 9, 25, 34, 37, 23, 44] targeting the early stages where the models have not been trained, the temporal threat model may delay the success of poisoning attacks and allow detection-based defenses to operate on more data. Another promising direction is to locate poisoned samples after the attack succeeds [31], which targets later stages where the models have been deployed. If one can reverse poisoning attacks shortly after they succeed, incorporating the temporal threat model can help to prolong every attack process and may eventually ensure a low exploitability of poisoning attacks.
Incorporating timestamps of samples that are often available but neglected in the past, we provide a novel, temporal modeling of data poisoning. With our temporal modeling, we derive notions of temporal robustness that offer meaningful protections even with unbounded amounts of poisoned samples, which is never possible with traditional modelings. We design a temporal robustness benchmark to support empirical evaluations of the proposed robustness notions. We propose temporal aggregation, an effective baseline defense with provable temporal robustness. Evaluation verifies its temporal robustness against data poisoning and highlights the potential of the temporal threat model.
This project was supported in part by Meta grant 23010098, NSF CAREER AWARD 1942230, HR001119S0026 (GARD), ONR YIP award N00014-22-1-2271, Army Grant No. W911NF2120076, NIST 60NANB20D134, the NSF award CCF2212458, a capital one grant and an Amazon Research Award.
[36] Binghui Wang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. On certifying robustness against backdoor attacks via randomized smoothing. CoRR, abs/2002.11750, 2020. URL https://arxiv.org/abs/2002.11750.
[37] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleansse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy, SP 2019, San Francisco, CA, USA, May 19-23, 2019, pages 707-723. IEEE, 2019. doi: 10.1109/SP.2019.00031. URL https://doi.org/10.1109/SP.2019.00031.
[38] Wenxiao Wang and Soheil Feizi. On practical aspects of aggregation defenses against data poisoning attacks, 2023.
[39] Wenxiao Wang, Alexander Levine, and Soheil Feizi. Improved certified defenses against data poisoning with (deterministic) finite aggregation. CoRR, abs/2202.02628, 2022.
[40] Wenxiao Wang, Alexander Levine, and Soheil Feizi. Lethal dose conjecture on data poisoning. CoRR, abs/2208.03309, 2022. doi: 10.48550/arXiv.2208.03309. URL https://doi.org/10.48550/arXiv.2208.03309.
[41] Maurice Weber, Xiaojun Xu, Bojan Karlas, Ce Zhang, and Bo Li. RAB: provable robustness against backdoor attacks. CoRR, abs/2003.08904, 2020. URL https://arxiv.org/abs/2003.08904.
[42] Yi Zeng, Si Chen, Won Park, Z. Morley Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. CoRR, abs/2110.03735, 2021. URL https://arxiv.org/abs/2110.03735.
[43] Yi Zeng, Won Park, Z. Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. CoRR, abs/2104.03413, 2021. URL https://arxiv.org/abs/2104.03413.
[44] Yi Zeng, Minzhou Pan, Himanshu Jahagirdar, Ming Jin, Lingjuan Lyu, and Ruoxi Jia. How to sift out a clean data subset in the presence of data poisoning? CoRR, abs/2210.06516, 2022. doi: 10.48550/arXiv.2210.06516. URL https://doi.org/10.48550/arXiv.2210.06516.
[45] Yi Zeng, Minzhou Pan, Hoang Anh Just, Lingjuan Lyu, Meikang Qiu, and Ruoxi Jia. Narcissus: A practical clean-label backdoor attack with limited information. CoRR, abs/2204.05255, 2022. doi: 10.48550/arXiv.2204.05255. URL https://doi.org/10.48550/arXiv.2204.05255.
[46] Chen Zhu, W. Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on deep neural nets. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 7614-7623. PMLR, 2019. URL http://proceedings.mlr.press/v97/zhu19a.html.