Adversaries With Incentives:  A Strategic Alternative to Adversarial Robustness

Authors: Maayan Ehrenberg, Roy Ganz, Nir Rosenfeld
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: adversarial training, strategic classification, adversarial robustness, strategic robustness

Abstract:
Adversarial training aims to defend against *adversaries*: malicious opponents whose sole aim is to harm predictive performance in any way possible. This presents a rather harsh perspective, which we assert results in unnecessarily conservative training. As an alternative, we propose to model opponents as simply pursuing their own goals—rather than working directly against the classifier. Employing tools from strategic modeling, our approach enables knowledge or beliefs regarding the opponent's possible incentives to be used as inductive bias for learning. Accordingly, our method of *strategic training* is designed to defend against all opponents within an `incentive uncertainty set'. This resorts to adversarial learning when the set is maximal, but offers potential gains when the set can be appropriately reduced. We conduct a series of experiments that show how even mild knowledge regarding the opponent's incentives can be useful, and that the degree of potential gains depends on how these incentives relate to the structure of the learning task.

Decision: accept (poster)
Reviewer scores: 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1729782372751

thread_type:
official_review

<thread_object>
creator: reviewer_cujn

<content>
summary:
This paper studies *strategically robust learning*, which fills the gap between adversarial learning and vanilla learning nicely. Specifically, the problem formulation is done in such a way that generalizes both cases. Extensive experiments are conducted to empirically verify the effectiveness of the proposed model and demonstrate the advantage of considering such strategic opponent scenarios instead of blindly chasing for the worst-case guarantees.

strengths:
Overall,
1. Well-motivated research design, which fills the gap between the two extreme spectrums of adversarial training nicely.
2. Clear formulation for most parts of the presentation. 
3. Extensive experiments that cover a wide range of concerns. Each experiment is also well-motivated.

weaknesses:
1. It seems like although we consider the set of all possible utilities to be $\\Lambda = [0, 1]^{K \\times K}$, the discussion is tailored towards $\\{0, 1\\}^{K \\times K}$ instead. It is not immediately clear that all the discussions and analysis will follow through within for the more general $\\Lambda$. I'm just curious whether it is possible to consider other than $0$-$1$ utilities in the experiment, since some of the categories, e.g., preferences, will make more sense for the case of $[0, 1]$ utilities.
2. Little emphasis on the algorithm description makes some of the claims not convincing. For example, what is the exact algorithm Line 364 referring to, i.e., "solving $\\max_u$ independently for each row $u(y, \\cdot)$"? What is the advantage in terms of time complexity when using this algorithm? This is crucial and should not be omitted since otherwise, as mentioned in Line 363, the runtime will scale with $|U|$.
3. Perhaps out-of-scope, but it will be nice to see some connection between the existing theory for adversarial training with this setting: how do they generalize, and how do they fail to generalize? The lack of (can be light) theoretical discussions makes the discussion a bit unsatisfying.

questions:
See Weaknesses. Additionally,
1. I found that Figure 1 is not very clear. Some explanations are needed for the meaning of (faded) arrows and such.
2. It's not entirely clear upon reading Section 4.2 how one will solve Eq. (5), or Eq. (4) given a strategic opponent with utility $u$. Without briefly stating the strategy, the results are reported and I think it would help to have some pointers to Section 5 saying that the results are obtained based on Section 5.1 (I suppose).
3. What does Line 360 mean exactly? More explanation is needed for how $\\hat{\\delta}_{y'}$ is defined.
4. Line 451, where the terms "well-specified" and "misspecified" are unclear in this context.

Some minor suggestions in writing:
1. Line 18, "incentive uncertainty set." instead of 'incentive uncertainty set'. Similarly, Line 82 'everything'. to "everything.", etc. 
2. Line 236, $u^{\\text{adv}}$ appears without a definition.
3. Line 266 and 267, it's unclear to me what $\\mathrm{test}(f_{\\text{train}})$ and $\\mathrm{clean}(f_{\\text{adv}})$ stand for without looking at the next paragraph.

>I would be happy to raise scores once these are answered and addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730022816739

thread_type:
official_review

<thread_object>
creator: reviewer_7lyq

<content>
summary:
The paper considers a new adversarial scenario where an attacker has a non-uniform preference for target classes. It introduces a new adversarial training method, named strategic training, designed specifically to defend against such targeted attacks. It claims that prior knowledge of the attacker’s preferences can enhance both clean accuracy and strategic adversarial accuracy.

strengths:
- The paper is well-written and clearly formalizes the proposed framework.

weaknesses:
- The proposed scenario seem impractical, as it is challenging for a defender to anticipate an attacker’s intent in advance. Additionally, an attacker’s preferences may shift over time, and there can be multiple attackers with varying preferences. This is why most work on adversarial robustness assumes a uniform preference among attackers.
- Although the proposed scenario is novel, the framework and its results seem straightforward. Constraining the attacker’s search space is expected to yield higher clean accuracy, reflecting the fundamental trade-off between accuracy and robustness.
- The results in Table 1 appear inconsistent and unreliable. In the GTSRB experiment, strategically trained VGG and ResNet18 models achieve higher adversarial accuracy than standard adversarially trained models. This suggests that the evaluation method may be flawed; the authors should consider using stronger attack algorithms, such as AutoAttack [1]. Using a 20-step PGD attack is an outdated evaluation approach, more commonly seen in studies from 2018.

[1] Francesco Croce, Matthias Hein, Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks, ICML 2020

questions:
See the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730468864578

thread_type:
official_review

<thread_object>
creator: reviewer_cmsm

<content>
summary:
This paper introduces a new perspective in adversarial training by proposing that adversaries may act out of self-interest rather than pure malice. Traditional adversarial training models the adversary as seeking maximal harm to the classifier, requiring models to defend against all possible incorrect labels. This approach, while robust, can reduce model generalization and clean data accuracy. Instead, the authors propose a framework called strategic training, which considers opponents who act to maximize their own utility. By assuming that opponents pursue specific goals, this approach allows models to leverage knowledge about opponent incentives, defining an incentive uncertainty set to guide training.

strengths:
- This paper identifies a favorable interpolation between natural training and adversarial training, in which many meaningful optimization problems with real-world significance are formulated.

- This paper provides a comprehensive investigation into the proposed types of strategic opponents, including adversarial, semantic, anti-semantic, preference ordering, and 1-hot.

- Thorough experiments demonstrate the gap between strategic attacks and adversarial attacks, as well as the effectiveness of the proposed strategic training method.

weaknesses:
- Regarding the question of "How much do we lose by being maximally conservative, and how much can we gain by appropriately reducing uncertainty," this paper offers an answer through experimental analysis but lacks a detailed theoretical examination.

- Strategic training requires a predefined utility function or set of utility functions; however, if the utility function is unknown, strategic training cannot proceed.

- (Minor) A typo in Figure 1: "preferecnce" should be corrected to "preference."

questions:
Is it possible in practice to estimate the opponent’s utility function based on some information such as historical attack data from opponents?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732918403386

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
New experiments

comment:
Dear reviewers,  

Thank you all for your reviewing efforts! 

Based on some of you comments and questions, we have added **two new experiments**:

1. **Inferring utilities from attack data** (in response to Rev. CMSm)
2. **Learning with [0,1] utilities** (in response Rev. CujN)

These are described briefly in our responses below, and appear in full detail in Appendices C.6 and C.7 in the new revision submitted.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper adds a new perspective to adversarial training by modeling adversaries as incentive and self-interest driven. Strategic learning is not new to the literature but I believe this paper introduces more alternatives that can cover a broader set of adversaries and when the learner only has limited information about the incentives. The results show that even mild knowledge about the adversary’s incentive can indeed be helpful.