Few-shot Text Adversarial  Attack for Black-box Multi-task  Learning

Authors: Wenqiang Wang, Hao Lin, Yan XIAO, jiehongli, Xiaochun Cao
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: multi-task adversarial text attacks

Abstract:
Current multi-task adversarial text attacks rely on white-box access to shared in-
ternal features and assume a homogeneous multi-task learning framework. As a
result, these attacks are less effective against practical scenarios involving black-
box feedback APIs and heterogeneous multi-task learning. To bridge this gap,
we introduce Cluster and Ensemble Mutil-task Text Adversarial Attack (CEMA),
an effective black-box attack that exploits the transferability of adversarial texts.
Specifically, we initially employ cluster-oriented substitute model training, as a
plug-and-play framework, to simplify complex multi-task scenarios into more
manageable text classification attacks and train the substitute model. Next, we
generate multiple adversarial candidate examples by applying various adversarial
text classification methods. Finally, we select the adversarial example that attacks
the most substitute models as the final attack output. CEMA is evaluated on two
primary multi-task objectives: text classification and translation. In the classifica-
tion task, CEMA achieves attack success rates that exceed 60% while reducing the
total number of queries to 100. For the text translation task, the BLEU scores of
both victim texts and adversarial examples decrease to below 0.36 with 100 queries
even including the commercial translation APIs, such as Baidu Translate and Ali
Translate.

Decision: reject
Reviewer scores: 1, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729991315250

thread_type:
official_review

<thread_object>
creator: reviewer_bjng

<content>
summary:
A new adversarial attack is proposed for text based multi-task models. The attack is called, “CEMA” or Cluster and Ensemble Mutil-task Text Adversarial Attack. Substitute models are trained in this attack and then various adversarial examples are generated. After selecting the example that fools a majority of the models, this example is used as the final attack input. The two task that are attacked are text classification and translation. Empirically the methods are test on two datasets, SST5 and Emotion.

strengths:
The paper is well written and clearly organized. The adversarial threat model is well defined and I appreciate the authors try to use a black-box approach (as opposed to many adversarial attacks that assume complete white-box knowledge of the setup). The empirical results demonstrate the effectiveness of the new method on two datasets.

weaknesses:
There are three main issues that I see with the paper:

Issue 1: Lack of proper citations. There are a number of works that this paper implicitly builds off of but are not explicitly cited. Due to this several experiments are missing that I think would better illustrate and help the readers understand the effectiveness of the proposed approach. First of all the discussion of transferability only mentions the original Papernot work, but is missing other type of attacks that increase the amount of data available to the attacker (https://arxiv.org/abs/2006.10876) or generate synthetic data (https://arxiv.org/abs/2003.12703). I would say at a minimum these two additions to the original 2016 Papernot transfer attack should be cited in relation to improving the CEMA and discussed. Likewise, when discussing transferability in general there are several related and follow up works that the authors do not include any mention of:

https://arxiv.org/abs/1704.03453

https://arxiv.org/abs/2104.02610


Issue 2: Because the authors seems to be unaware of some of the more recent develops in adversarial attacks/defenses, it would be great if two additional experiments or discussions could be had. First: what happens if the query number or amount of training data available to the attacker is increased like in (https://arxiv.org/abs/2006.10876)? I think that is important to understand what effect changing the strength of the adversarial threat model has on the attack success rate. For example if say 1000 or even 10,000 samples of training data are made available to the adversary, how much more effective is the transfer attack? Likewise, if we drop the number down to 50 or 10 samples, what happens? I am okay accepting the paper even IF the attack fails when less samples are used. I just think it is a necessary step in the attack analysis that is currently missing. 

Second: Many transfer attacks may fail (citation: https://arxiv.org/abs/2104.02610) when the models are shuffled in a randomized manner. Such work in the image domain that has shown an increased robustness over SOTA single model defenses can be seen here (https://arxiv.org/abs/2211.14669). Because the authors didn’t test on any defenses, I would be curious if they could train additional models and test a simple defense where they try to shuffle between two models during querying and attack time? Or if the authors feel this is not relevant, could they include a discussion of this in their paper and cite the appropriate papers as mentioned in this paragraph to explain why? 

Issue 3 (minor): Is there any reason that the authors only choose to use 3 different victim models when doing the experiments? As a reviewer I cannot mandate that additional experiments are done, but I would be curious as to the justifications for three models used in the paper?

questions:
Please address the three issues that I have brought up in the weakness section of the paper. If you can provide better discussions and citations to the relevant adversarial literature and address my comments, I would be inclined to increase my score accordingly.

UPDATE: Score has been updated according to the rebuttal from the authors.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730124232048

thread_type:
official_review

<thread_object>
creator: reviewer_3ydr

<content>
summary:
This paper presents CEMA, a new text adversarial attack against multi-task learning in the black-box setting. The basic idea of this method is to adopt a cluster-oriented substitute model training to convert the problem into a text classification attack. Representation learning is used for the above clustering. Multiple substitute models are generated to produce the high-quality adversarial examples that are effective against different tasks concurrently. Comprehensive evaluations are performed to demonstrate the effectiveness of this solution.

strengths:
1. This paper is well-written and easy to follow. 
2. The proposed method is novel and solid. 
3. Experiments are extensive, with baseline comparisons over different methods.

weaknesses:
Thanks for submitting this paper to ICLR. Generally I enjoy reading this paper. It is very clear. I have the following doubts regarding the proposed method. 

1. While the attack is efficient in terms of model query, I am wondering what is the computation overhead at local? The attacker requires to train multiple substitute models for candidate selection. However, I could not find the number of substitute models used in this paper. Clearly a larger number of substitute models lead to higher overhead. What is the cost of performing the presentation learning, clustering, training substitute models, and generating adversarial examples? 

2. The experiments only consider two types of tasks with three models (two for classification and one for translation). What is the scalability and effectiveness of the method for more types and numbers of tasks? 

3. This paper does not consider the evaluation of SOTA defenses against the proposed method. It is recommended to present the attack results under different defense methods, like adversarial training and others.

questions:
1. What is the computation overhead of your attack? 

2. What is the scalability of your attack in terms of number of tasks? 

3. What is the performance of your attack against SOTA defenses?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730516833801

thread_type:
official_review

<thread_object>
creator: reviewer_ovmt

<content>
summary:
In this work, the authors propose a transfer-based black-box text attack for multi-task learning called Cluster and Ensemble Multi-task Text Adversarial Attack (CEMA). In particular, CEMA employs a cluster-oriented substitute model training to obtain the substitute mode. Then CEMA generates the adversarial examples using various adversarial attacks and chooses the adversarial example that attacks the most substitute models as the final attack output. Experiments validate the effectiveness of the proposed method.

strengths:
1. The cluster-oriented substitute model training is interesting.

2. The proposed CEMA exhibits good transferability.

3. The authors have derived the theoretical lower bound for CEMA's success rate.

weaknesses:
1. The paper is not well written and hard to follow. It is not clear how to generate adversarial examples using the surrogate model.

2. What is the main contribution of this paper? Is it the substitute model training or the attack method? I do not get the attack method. But if it was the substitute model training, why do not the authors compare CEMA with the adversarial examples generated on various substitute models?

3. Most of the baselines are black-box attack, which can directly attack online APIs. Since the authors evaluate the transferability, I think the authors should consider more efficient attacks using gradient [1,2] as in the image domain.

4. It is weird that the average number of queries is smaller than 1.

[1] Wang, X., Yang, Y., Deng, Y., & He, K. Adversarial training with fast gradient projection method against synonym substitution based text attacks. AAAI 2021.

[2] Xinshuai Dong, Anh Tuan Luu, Rongrong Ji, Hong Liu. Towards Robustness Against Natural Language Word Substitutions. ICLR 2021.

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710716607

thread_type:
official_review

<thread_object>
creator: reviewer_rcpa

<content>
summary:
This paper is about a new text-based adversarial attack that works simultaneously against multiple NLP models.

strengths:
- The table is easy to read.

weaknesses:
- No real threat model. The authors did not properly specify adversarial capabilities (e.g., number of tokens allowed to flip), so I am unable to interpret what the results are supposed to mean. Note that, without constraints on perturbation of the input, it is trivial to fool sentiment models by providing a movie review with the opposite sentiment, or fool translation models by inputting gibberish. **Please explicitly define the constraints on input perturbations, such as the maximum number of tokens that can be modified or any other relevant limitations on the adversarial capabilities.** This would allow for a more meaningful interpretation of the results.
- The author claims to "demonstrate the effectiveness of CEMA through rigorous mathematical derivations", but the math makes no sense.
  - For example, the authors attempted to justify why generating multiple adversarial examples is useful. They made an argument that each attack query has at least probability $p_\min$ of being successful, so the likelihood of some query being successful is 1 with sufficiently many queries. This is clearly false. Even if $p_\min$ is non-zero, different attack queries would not be independent random variables and the derivation is just wrong.
  - I'm not sure why there is a derivation of the "maximum entropy distribution" (Appendix C) and "union bound" (Appendix D) in the paper. They seem completely irrelevant.
  - Appendix B on transferrability of adversarial examples is barely related to the proposed method itself.
- What do you mean by attacking multi-model multi-task learning? How is this different from transferrable adversarial examples? I am concerned that you are redefining a well-studied problem and not properly acknowledging prior literature.

questions:
- What's the unit of queries in Table 1? What does it mean to use 0.05 queries?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732793281814

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank the reviewers for their time, positive comments, and constructive feedback, and are encouraged by the comments of **well-written** (R3, R4), **interesting** (R2), **extensive experimental evaluation** (R2, R3), and **novel and effective** (R3, R4).

In summary, we discuss four practical adversarial attack scenarios in text multi-task learning: few-query attacks, black-box attacks, multi-model multi-task learning, and attacks that encompass a variety of tasks. To the best of our knowledge, few-query attacks, multi-model multi-task learning, and black-box attacks involving a variety of tasks in multi-task text learning are being discussed for the first time. Additionally, we investigate the data-free scenario, in which the attacker only has access to the attributes of the multi-task model, and examine how to perform multi-task text adversarial attacks under these conditions.

We also introduce CEMA, a few-shot, black-box attack method. CEMA is a plug-and-play framework that converts multi-task attack scenarios into single-task text classification attacks. CEMA demonstrates significant attack effectiveness with just 10 queries and in the data-free scenario. Experimental results show that CEMA's performance improves as more text classification methods are incorporated, and it retains substantial effectiveness even when defense mechanisms are applied. We extend CEMA to a variety of tasks, including text classification, translation, summarization, and text-to-image generation.

Considering the reviewer’s comments, we updated our manuscript (modified parts highlighted in “red”), with the following additions:

1. **[Computational Overhead of Substitute Model Training (E)]**  
   In Appendix E, we expand on the computational costs involved in training the substitute models for CEMA. 

2. **[Performance Evaluation on Six Downstream Tasks (I)]**  
   In Appendix I, we evaluate the performance of CEMA on six different downstream tasks, including sentiment analysis (SST5 and Emotion datasets) and machine translation. 

3. **[Text Summarization and Text-to-Image Generation Tasks (J)]**  
   In Appendix J, we further explore the applicability of CEMA on more complex tasks, such as text summarization and text-to-image generation. Using the Pokemon dataset, we perform experiments on translation (BLEU: 0.24), summarization (ROUGE reduction of 47%), and text-to-image generation (CLIP score reduction of 56%).

4. **[Few-Shot Learning and Additional Model Queries (K)]**  
   Appendix K presents the evaluation of CEMA in a few-shot learning scenario, where the attack's effectiveness increases with the number of queries and available training data. Even with as few as 10 queries, the attack success rate exceeds 30%, demonstrating CEMA’s ability to work effectively under resource-constrained conditions.

5. **[Transferability of Adversarial Attacks (L)]**  
   In Appendix L, we analyze the transferability of adversarial attacks generated by CEMA across different models. Our transferability experiments show that CEMA achieves superior results, significantly outperforming traditional baseline methods. The transferred adversarial examples effectively compromise multiple target models, showcasing the generalization capability of the attack.

6. **[Impact of Query Count on Attack Performance (M)]**  
   In Appendix M, we investigate the relationship between the number of queries and the attack success rate. 

7. **[Further Baseline Comparisons on Text Classification Tasks (N)]**  
   Appendix N provides a more comprehensive comparison of CEMA against several baseline methods such as TextBugger and HotFlip. The experimental results show that CEMA outperforms these baselines.

8. **[Performance on Transfer Attacks (O)]**  
   Appendix O delves deeper into the transferability performance of CEMA across different model combinations.

9. **[Defensive Strategies Against CEMA (P)]**  
   In Appendix P, we discuss various defense mechanisms, such as random perturbations, designed to mitigate CEMA’s effectiveness. Although these defenses reduce the attack’s success rate, CEMA still maintains a relatively high performance, with ASR exceeding 50% .

10. **[Adversarial Sample Definition in Text Classification and Translation (R)]**  
    In Appendix R, we formally define the properties of adversarial samples used in text classification and machine translation tasks.

11. **[Independence Validation Experiments (S)]**  
    Appendix S focuses on validating the independence assumption of the adversarial examples. We show that combining multiple attack methods improves the overall attack success rate, confirming that diverse approaches contribute to more robust adversarial examples.

12. **[Non-Independence in Adversarial Example Generation (T)]**  
    In Appendix T, we extend the analysis to scenarios where adversarial examples are not generated independently.
</content>
</thread_object>

</thread>


Metareview:
metareview:
- This paper explores *multi-task textual adversarial attacks* and proposes a new attack method, CEMA, to craft multi-task adversarial prompts. The approach involves training substitute models to capture the representation distribution based on cluster labels obtained through limited queries. Multiple attack strategies are combined to enhance task transferability.

- Major concerns raised by reviewers include the realism and meaningfulness of the proposed threat model, the novelty of the CEMA method, and weak experiments. The threat model was revisited during the AC-reviewer process. While multi-task adversarial attacks, in general, are realistic, the focus on attacking only text classification and translation tasks does not sufficiently demonstrate the significance of the approach.  LLMs are inherently multi-task models. This raises questions about the distinction between adversarial attacks on LLMs and the definition of "multi-task" (or "multi-domain") in the context of LLMs. There are also concerns about the adaptability of the approach when incorporating additional task types. 

- Overall, the paper cannot be accepted in its current form. I encourage the authors to address the remaining concerns in future submissions.