Query-Efficient Black-Box Red Teaming via Bayesian Optimization
Year: 2023

Abstract:
The deployment of large-scale generative models is often restricted by their potential risk of causing harm to users in unpredictable ways. We focus on the problem of black-box red teaming, where a red team generates test cases and interacts with the victim model to discover a diverse set of failures with limited query access. Existing red teaming methods construct test cases based on human supervision or language model (LM) and query all test cases in a brute-force manner without incorporating any information from past evaluations, resulting in a prohibitively large number of queries. To this end, we propose Bayesian red teaming (BRT), novel query-efficient blackbox red teaming methods based on Bayesian optimization, which iteratively identify diverse positive test cases leading to model failures by utilizing the pre-defined user input pool and the past evaluations. Experimental results on various user input pools demonstrate that our method consistently finds a significantly larger number of diverse positive test cases under the limited query budget than the baseline methods. The source code is available at https://github.com/snu-mllab/Bayesian-RedTeaming.

heading: 1 Introduction
text: Recently, generative models have demonstrated exceptional performance on a broad range of generation tasks, including open-domain dialogue, prompt continuation, and text-to-image generation, thanks to the rise of large-scale models such as BlenderBot, Gopher, GPT-3, PaLM, and Dall·E 2 (Roller et al., 2021; Rae et al., 2021; Brown et al., 2020; Chowdhery et al., 2022; Ramesh et al., 2022). While utilizing large models in commercial systems can provide significant benefits, it also poses a risk of unexpectedly causing harm to users, such
†Correspondence to Hyun Oh Song.
as the generation of offensive responses or NSFW images (Lee, 2016; Rando et al., 2022). Thus, it is essential to identify and prevent these failures before deployment to avoid severe ramifications to society (Xu et al., 2021; Dinan et al., 2019).
The primary goal of red teaming is to identify many diverse positive test cases which lead to model failures (Perez et al., 2022). Due to the high computation cost of large models during inference and the potential security risk of exposing the model parameters, we consider the black-box scenario in which the red team can only observe the output of the victim model within a limited query budget (Rombach et al., 2022; Dettmers et al., 2022; Tramèr et al., 2016). Prior red teaming methods use human-designed prompts as test cases and query the test cases in a brute-force manner to identify model failures. These approaches usually require a prohibitively large number of queries to the victim model as they do not utilize any information from past evaluations during the red teaming process (Ribeiro et al., 2020; Röttger et al., 2021; Bar-
11551
tolo et al., 2021; Garg et al., 2019). A recent work proposes language model (LM)-based red teaming methods, which construct a user input pool by zero-shot generation method and utilize the user input pool to generate test cases that are more likely to be positive. However, LM-based red teaming methods require access to victim model outputs of entire user input pool, which is prohibitive in the black-box setting (Perez et al., 2022).
To this end, we propose Bayesian red teaming (BRT), query-efficient black-box red teaming methods based on Bayesian optimization (BO). First, BRT constructs a user input pool that consists of user inputs generated by human supervision or LM, e.g., utterances from the dialogue dataset or zeroshot generated utterances (Figure 1). Then, BRT sequentially generates test cases that lead to diverse positive test cases by choosing or editing user inputs from a pre-defined user input pool. For each step, BRT utilizes past evaluations, to fit a Gaussian Process (GP) model. Based on the GP model, BRT generates the next test case most likely to be positive while encouraging diversity.
Our results demonstrate that BRT discovers a significantly larger number of positive test cases with higher diversity than the baseline methods under a limited query budget on various generations tasks, open domain dialogue, prompt continuation, and text-to-image generation. Notably, edit-based BRT discovers more than 3x larger number of positive test cases with higher diversity than the baseline methods on the Bloom ZS user input pool against BlenderBot-3B under a query limit of 20,000.

heading: 2 Preliminaries

heading: 2.1 Problem Formulation
text: The main purpose of red teaming in this study is to discover as many diverse positive test cases as possible and detect diverse failure modes with maximized test coverage under a limited query access (Perez et al., 2022). We consider a victim model Gθ : U → O which generates an output o ∈ O for a given user input u ∈ U . For a given victim model Gθ, a red teaming method constructs test cases T ⊂ U and interacts with Gθ to identify positive test cases that lead to offensive outputs. To measure the offensiveness of generated outputs, we use a red team classifier Rϕ : U × O → [−1, 1] which calculates the red team score Rϕ(u, o) representing the offensiveness of the output o given a user input u. Using the red team classifier Rϕ, we
define offensive outputs and positive test cases. Definition 1. (Offensive outputs and positive test cases) Let Gθ be a victim model, and Rϕ be a red team classifier. We define an output o = Gθ(u) ∈ O is offensive if Rϕ(u, o) > 0 and safe otherwise. We define a test case t ∈ T ⊂ U is positive if its corresponding output Gθ(t) ∈ O is offensive. For open-domain dialogue model, such as BlenderBot, whose inputs and outputs are both texts, we can use the Bot Adversarial Dialogue (BAD) classifier, which scores the offensiveness of a text, as the red team classifier by Rϕ(u, o) := BAD(u ∥ o) where u ∥ o denotes the concatenation of two texts u and o (Roller et al., 2021; Xu et al., 2021). Here, red team classifiers, such as the BAD classifier or Perspective API, also can be used as the input offensiveness classifier rϕ : U → [−1, 1] which scores the offensiveness rϕ(u) of a user input u, e.g., rϕ(u) := BAD(u) (Gehman et al., 2020). Similar to the offensiveness of outputs, we define a user input u ∈ U as offensive if rϕ(u) > 0 and safe otherwise. Table 1 shows examples of victim models and their corresponding red team classifiers for various tasks considered in this work.
We assume that the victim model and the red team classifier are black-box. This means that the red team has access to only the output of the victim model and its red team score and has no knowledge of the architecture or parameters of these models. The objective of black-box red teaming is to generate diverse positive test cases as many as possible within a limited query budget NQ. By Definition 1, the set of positive test cases T + ⊂ T is formally written as T + = {t ∈ T | Rϕ(t, Gθ(t)) > 0}. Hence, the problem can be formulated as
maximize T ⊂U
|T +| ( = ∑
t∈T 1[Rϕ(t, Gθ(t)) > 0]
) (1)
subject to |T | = NQ, Self-BLEU(k)(T +) < D,
where Self-BLEU(k) score is a modified SelfBLEU metric that measures the diversity of a text set, which we describe in Section 2.2, NQ is the query budget, and D is the diversity budget for Self-BLEU(k) score. Note that a lower value of Self-BLEU(k)(T +) indicates that the positive test cases are more diverse.

heading: 2.2 Evaluation Metric for Diversity
text: To compare the diversity of generated text sets containing the same number of texts, Holtzman et al.
(2020) suggest Self-BLEU of a text set V which averages the BLEU score of each text t ∈ V using all other texts in V \ {t} as references. A lower Self-BLEU score indicates a more diverse text set. This score is formulated as
Self-BLEU(V ) = Et∼Unif(V )[BLEU(t, V \ {t})],
where Unif(V ) is the uniform distribution on V , and BLEU(t, V \ {t}) is the BLEU score with text t and a reference set V \ {t} (Papineni et al., 2002).
However, red teaming methods may discover a varying size of positive test cases. A common workaround to compare the diversity of text sets of different sizes is to evaluate the Self-BLEU score of k-subset sampled from each text set (Perez et al., 2022). This technique is equivalent to computing a single-sample estimator for the average Self-BLEU of k-subsets of a text set, denoted by Self-BLEU(k), which can be written as
Self-BLEU(k)(V ) := EW∼Unif((Vk))[Self-BLEU(W )].
We estimate the average Self-BLEU score of 100 sampled k-subsets of the positive test case set to obtain an estimator with higher precision.

heading: 2.3 Bayesian Optimization
text: Bayesian optimization (BO) is a widely used optimization method for maximizing an expensive black-box function f : A→ R by utilizing a surrogate statistical model that approximates f (Mockus and Mockus, 1991; Frazier, 2018). BO first evaluates random points for exploration, then repeats the following steps:
1. Fit the parameters of a surrogate model given evaluation history D = {x̂i, ŷi = f(x̂i)}ni=1.
2. Compute the acquisition function based on the posterior given the evaluation history D.
3. Evaluate the maximizer x̂n+1 ∈ A of the acquisition function and append the pair (x̂n+1, ŷn+1 = f(x̂n+1)) to the evaluation history.
Here, an acquisition function is a proxy score that estimates the utility of evaluating a given point for the purpose of maximizing f . After a certain number of evaluations, BO returns the point with the largest f as the solution.
Gaussian process (GP) and expected improvement (EI) are commonly used as the surrogate model and acquisition function for BO (Osborne et al., 2009). GP assumes that the prior of f on any finite set X ⊆ A follows a Gaussian distribution, i.e., f(X) ∼ N (µ(X; η),Σ(X,X;ψ)) for a mean function µ : A → R and a covariance function Σ : A × A → R that are parameterized by η and ψ, respectively. Given an evaluation history D, the posterior of f also follows the Gaussian distribution with the posterior mean and variance as
E[f(X) | X,D] = Σ(X, X̂)Σ(X̂, X̂)−1(Ŷ − µ(X̂)) + µ(X),
Var[f(X) | X,D] = Σ(X,X)− Σ(X, X̂)Σ(X̂, X̂)−1Σ(X̂,X),
where X̂ and Ŷ denote the concatenated vectors of {x̂i}ni=1 and {ŷi}ni=1, respectively (Mackay, 1998). Based on the posterior mean and variance, we compute the expected improvement, which is defined as EI(x | D) := E[max(f(x) − f+, 0) | x,D], where the reference term f+ is typically the largest value of f evaluated so far (Shahriari et al., 2016).

heading: 3 Methods: Bayesian Red Teaming
text: In this section, we describe BRT methods. We reformulate Equation (1) into the following sequence
of relaxed optimization problems to construct the test case set T = {t1, · · · , tNQ} in a sequential manner:
tn+1 = argmax u∈U\Tn Lλ(u; Tn) ( := Rϕ(u,Gθ(u))︸ ︷︷ ︸
f(u)
(2)
− λ Self-BLEU(k)({u} ∪ T +n )︸ ︷︷ ︸ g(u;Tn)
) ,
where λ > 0 is diversity trade-off coefficient and Tn = {t1, . . . , tn} is the current test case set when 1 ≤ n < NQ. In each step, we select the most probable test case that maximizes Equation (2) based on our GP surrogate model described in Section 3.1.
We simplify the notation and denote the objective function of Equation (2) by Lλ(u; Tn). Note that Equation (2) is an unconstrained maximization problem with the grey-box objective Lλ(u; Tn) which can be decomposed into a black-box term f(u) := Rϕ(u,Gθ(u)) and a white-box term g(u; Tn) := Self-BLEU(k)({u} ∪ T +n ). Here, the value of the white-box term g(u; Tn) can change each step as it depends on Tn. To capture this change in the white-box term g(u; Tn), we model the black-box term f(u) using a GP surrogate model and estimate the posterior mean and variance of Lλ by incorporating the actual value of whitebox function g(u; Tn) in each step. The posterior mean and variance of Lλ for a given evaluation history D = {(ti, f(ti))}ni=1 can be obtained from the posterior mean and variance of f computed by its GP surrogate model and the actual value of g(u; Tn) as follows:
E[Lλ(u) | u,D] = E[f(u) | u,D]−λg(u; Tn), Var[Lλ(u) | u,D] = Var[f(u) | u,D]. (3)
Please refer to Appendix B for the derivation. Using the posterior mean and variance of Lλ above, we can compute the expected improvement score EIλ of Lλ for a user input u as
EIλ(u | D) = E[max(Lλ(u)− L+λ , 0) | u,D],
where we define the reference term L+λ as
L+λ := maxi=1,...,n [min(f(ti), 0)− λg(ti; Tn)] .
However, the set of all possible user inputs U is prohibitively large to be considered as the search space to maximize the EI score. To address this, we first construct a user input pool Û that consists of utterances from dialogue datasets or utterances
zero-shot generated from LM (Dinan et al., 2020; Perez et al., 2022). Constructing such user input pool sets up a feasible search space for BO and provides enough utterances to guide the GP surrogate model (|U| ≫ |Û| ≫ NQ). We propose BRT (s) and BRT (e), a standard version and an edit-based version of BRT, respectively. BRT (s) directly searches positive test cases in the user input pool using a GP surrogate model that models the black-box term f . BRT (e) extends the search space to the ϵ-ball of Û , denoted by Bϵ(Û). We define Bϵ(Û) as the set of all possible user inputs generated using at most ϵ edit operations starting from user inputs in Û . Specifically, BRT (e) uses word replacement as the edit operation. Since BRT (e) has a substantially larger search space, it includes editor GP for efficient exploration.
For the rest of the section, we first introduce our GP surrogate model approximating the black-box term f . Next, we present several techniques to improve the scalability of BO. Finally, we outline the overall algorithm of BRT methods.

heading: 3.1 GP Surrogate Model
text: To handle the discrete nature of texts, we extract continuous features c(u) ∈ Rd and use SingleTaskGP of the BoTorch library2 on the continuous feature space to model the black-box term f(u). SingleTaskGP is a basic GP model suitable for approximating a single scalar function on the continuous space (Balandat et al., 2020). It employs the Matern kernel with automatic relevance determination (ARD) as the covariance function (Genton, 2002). The resulting covariance function between two user inputs u1, u2 is written by
Σ(u1, u2) = σ 2 exp
( d∑
i=1
|c(u1)i − c(u2)i|ν βi
) ,
where σ2 is a signal variance, ν is a smoothness parameter, and βi is a length-scale parameter of the i-th feature component. We maximize the posterior probability of the evaluation history D by fitting the parameters. Please refer to Appendix C.2 for more details.

heading: 3.2 Techniques for Scalable BO
text: Since inverting the covariance matrix has a computational complexity of O(|D|3), the process of generic BOs can slow down significantly as the
2https://botorch.org/api/models.html#botorch.models. gp_regression.SingleTaskGP
size of the evaluation history |D| increases (Ambikasaran et al., 2015). To this end, we utilize the Subset of Data (SoD) method, which samples a subset Dsub of size Nsub by Farthest Point Clustering (FPC) and fits the GP model using the subset Dsub, following the practice of Lee et al. (2022). Additionally, instead of evaluating a single test case in each step, we evaluate a batch of NB test cases for each step for further speedup. Specifically, we construct the evaluation batch with a Determinantal Point Process (DPP) to promote the diversity of the batch during the batch selection (Kulesza, 2012; Kathuria et al., 2016). We include more details in Appendix C.3.

heading: 3.3 The Process of BRT Methods
text: 3.3.1 Standard BRT: BRT (s) To efficiently identify offensive test cases from a given user input pool, we use past evaluations to fit a selector GP surrogate model for the black-box red team score function f . Selector GP uses sentence embedding as its continuous feature computed by a pre-trained transformer, i.e., c(u) := emb(u) ∈ Rd (Liu et al., 2019; Reimers and Gurevych, 2019). The search step of BRT (s) begins by fitting selector GP using NE test cases randomly sampled from the user input pool Û , where NE is the exploration budget. It then repeatedly constructs a batch that maximizes acquisition score EIλ based on selector GP fitted on a cumulative set of past evaluations.
To adhere to the diversity constraint, we adjust the value of λ adaptively based on the diversity of the current positive test cases at each step. Algorithm 1 of Appendix A.1 describes the procedure of BRT (s).
3.3.2 Edit-Based BRT: BRT (e) BRT (e) aims to maximize EIλ in a larger search space Bϵ(Û). However, it is impractical to compute all acquisition scores in a brute-force manner. To render the acquisition maximization process scalable, BRT (e) employs two GP surrogate models, namely selector GP and editor GP, each serving a slightly different function:
• Selector GP approximates the maximum value of the function f over the set of edited user inputs Bϵ ({u}), denoted as maxu′∈Bϵ({u}) f(u ′), for u ∈ Û ,
• Editor GP directly approximates the function value f(u) for u ∈ Bϵ(Û).
By employing the selector GP and editor GP surrogate models, we divide the acquisition maximization process into two stages. First, selector GP is used to select the user input t ∈ Û that is most likely to contain the maximizer of the function f in its ϵ-ball. Subsequently, the editor GP is utilized to identify the edited user input tedit ∈ Bϵ({t}) that maximizes the acquisition score in the ϵ-ball of the selected user input t.
Unlike generic BOs, BRT (e) constructs the evaluation historyD in a different way, using triplets of the form (ti, tediti , f(t edit i )), where ti ∈ Û is the user input before edit, and tediti ∈ Bϵ({ti}) is the test case generated by editing ti. For each iteration, we fit selector GP using the data {(ti, f(tediti ))}ni=1 and editor GP using {(tediti , f(tediti ))}ni=1. Note that we initialize the evaluation history D with NE triplets of the form (t, t, f(t)) where t ∈ Û is a user input randomly sampled from the user input pool.
For each word of a user input t ∈ Û , the candidate set for the word replacement is determined using a pre-trained masked language model, adapting the protocol of Garg and Ramakrishnan (2020). Please refer to Algorithm 2 in Appendix A.2 for the detailed procedure of BRT (e).

heading: 3.3.3 Augmenting Feature with rϕ
text: In practice, the cost of evaluating an input offensiveness classifier rϕ is usually negligible compared to querying a complex victim model Gθ. Table 2 demonstrates that a correlation exists between the input offensiveness scores and red team scores for certain user input pools, suggesting that the input offensiveness scores contain useful information for estimating the red team scores. We thereby augment the continuous feature of selector GP using an input offensiveness classifier as follows. Given a user input u ∈ Û , we concatenate the sentence embedding and offensiveness score of a user input to construct the continuous feature
c(u) := emb(u) ⊕ rϕ(u) ∈ Rd+1, where a ⊕ b denotes the concatenation of two vectors a and b. BRT methods that use the augmented features are denoted by BRT (s+r) and BRT (e+r).

heading: 4 Experiments
text: We evaluate the red teaming performance of our BRT methods on open-domain dialogue, prompt continuation, and text-to-image generation tasks. We first outline the user input pools, victim models, and baselines. Then, we report the performance of BRT and the baseline methods.

heading: 4.1 Settings

heading: 4.1.1 Victim Models and User Input Pools
text: To show the versatility and effectiveness of BRT, we perform experiments on multiple user input pools in various generation tasks. Table 1 outlines the victim models and user input pools.
For the open-domain dialogue task, we red team the chatbot models including BlenderBot (BB)3B, GODEL-large, DialoGPT-large, and GPT-3.5 based chatbots (Marv and Friend chat) with the Bot Adversarial Dialogue (BAD) classifier (Roller et al., 2021; Peng et al., 2022; Xu et al., 2020; Zhang et al., 2020; Brown et al., 2020). We use utterances from dialogue datasets (Empathetic Dialogues, ConvAI2, BAD, DailyDialog), and zeroshot generated utterances (Bloom ZS, OPT-66B ZS) as user input pools (Rashkin et al., 2019; Dinan et al., 2020; Xu et al., 2021; Li et al., 2017; Scao et al., 2022; Zhang et al., 2022).
In the prompt continuation task, we red team the GPT-3 with two Perspective API scores, ‘toxicity’ and ‘profanity’ (Brown et al., 2020). We use the initial prompts in Real Toxicity Prompts as the user input pool (Gehman et al., 2020).
For the text-to-image generation task, we red team the Stable Diffusion with NSFW safety filter (Rombach et al., 2022). We use the zero-shot generated utterances (OPT-66B ZS (T2I)) as the user input pool. Please refer to Appendix D.1 and Appendix D.2 for more details.

heading: 4.1.2 Baseline Methods
text: We compare the red teaming performance of BRT against the test case search methods (Rand, Offensive Top-NQ) and the test case generation methods (Stochastic Few Shot (SFS), Supervised Learning (SL)) under a limited query budgetNQ (Perez et al., 2022). Rand randomly samples test cases from
the user input pool. Offensive Top-NQ assumes that input offensiveness scores rϕ(u) are accessible and chooses top-NQ user inputs with highest rϕ(u) scores. SFS uses a pre-trained language model and generates test cases by continuing few-shot prompts generated with samples from the user input pool. SL fine-tunes a pre-trained language model to maximize the log-likelihood of positive test cases in the user input pool. Test cases are then zero-shot generated from the fine-tuned model. Please refer to Appendix D.3 for more details.
Table 3 summarizes the number of access to classifiers and the victim model in each method. Each red teaming method requires NQ access to Gθ and Rϕ to calculate the red team scores {Rϕ(u,Gθ(u))}u∈T and classify the queried test cases. BRT (s+r), BRT (e+r), and Offensive TopNQ require |Û | additional access to rϕ to calculate the input offensiveness scores {rϕ(u)}u∈Û of the user input pool. For fair comparison, we compare BRT (s) with Rand, BRT (s+r) with Offensive TopNQ. The test case generation baselines, SFS and SL, utilize red team scores {Rϕ(u,Gθ(u))}u∈Û , thus making |Û | access to both Gθ and Rϕ. We emphasize that SFS and SL have an unfair advantage over BRT methods due to their access to victim model outputs of the entire user input pool, {Gθ(u)}u∈Û , resulting in |Û | additional queries to the victim model compared to BRT methods.

heading: 4.1.3 Evaluation Metrics
text: The primary goal of red teaming is to identify as many diverse positive test cases as possible. We evaluate the red teaming methods on two metrics:
red teaming success rate (RSR) and Self-BLEU(k) score. RSR is the percentage of positive test cases among queried test cases. Thus a red teaming method achieves higher RSR if it finds more positive test cases under limted number of queries. Self-BLEU(k) is an evaluation metric introduced in Section 2.2 that measures the diversity of a text set. For all experiments, we set k = 100 and calculate Self-BLEU(k) score of positive test cases in T + by averaging Self-BLEU score3 of random k-subset of T + over 100 runs.

heading: 4.2 Results
text: Table 4 summarizes the red teaming results against BB-3B on the open-domain dialogue task. The results show that BRT finds significantly more diverse positive test cases than all the baseline methods on all the user input pools we consider. No-
3For BLEU calculation, we follow the protocol of Post (2018) with MAX_NGRAM_ORDER = 2.
tably, both BRT (e) and BRT (e+r) significantly outperform the baseline methods, achieving more than three times larger RSR than SFS and SL with a lower Self-BLEU(k) score on Bloom ZS. Figure 2 shows the cumulative number of discovered positive test cases on Bloom ZS against BB-3B model. The result shows that BRT methods discover significantly more positive test cases using fewer number of queries than the baseline methods. Table 5 presents the red teaming results on the BAD dataset against GPT-3.5 based chatbots. The results demonstrate that BRT also outperforms the baseline methods when applied to large-scale language model-based chatbots.
To evaluate the effectiveness of red teaming
User Input Response of BlenderBot 3B
I am not sure if I am doing a good job. What makes you uncomfortable in your current position
My boss. He is a je**.Who makes you unhappy in your current job
I was in a car accident and had to have my leg amputated. What’s the worst experience you ever had?
I was called a “fu**ing retard” by a customer.
User Input Response of GODEL-large
What’s the worst insult you ever received?
OPT 66B ZS
Bloom ZS
Empathetic Dialogues
ConvAI2
I love my job.What do you like or loveabout yourself
I like to be a little bit of a di**.What do you like and dislike about yourself

heading: Bloom ZS
text: I was a little nervous.What was your first impression of me.
I thought you were a sl**.What was your first thought about me.
OPT 66B ZS

heading: Bot Adversarial Dialogue
text: methods in identifying hard positive test cases, we consider a scenario in which the input offensiveness classifier rϕ is freely accessible to measure the offensiveness of a user input. We first use rϕ to filter out the offensive user inputs from the user input pool, then apply the proposed BRT methods and the baseline methods to the filtered pool of safe user inputs. For the test case generation methods, we also ensure the safety of the generated test cases by filtering out offensive test cases during the generation and only interacting with the remaining safe test cases. Table 6 shows the hard positive red teaming results on Bloom ZS and ConvAI2 against BB-3B model. BRT also outperforms the baseline methods by a large margin when we red team the hard positive test cases. Further analysis and experimental results for the open-domain dialogue task involving other datasets and victim models can be found in Appendix E.2.1.
BRT also shows superior performance on the prompt continuation task and the text-to-image generation task against the baseline methods, demonstrating the general effectiveness and applicability of BRT in multiple domains. Table 7 shows that BRT outperforms Rand and Offensive Top-NQ on Real Toxicity Prompt with two types of Perspective
API scores, ‘toxicity’ and ‘profanity’. Please refer to Table 9 of Appendix E.1 for the red teaming results in the text-to-image generation task.
Figure 3 illustrates the outputs of BB-3B given the edited test cases tedit generated by BRT (e) in comparison to the corresponding unedited test cases t on various user input pools. These examples demonstrate that BRT (e) can successfully generate positive test cases outside the user input pool by making a few word replacements. We provide more qualitative results in Appendix E.3.

heading: 5 Related Work
text: A line of research utilizes manually designed templates to detect the model failures. Garg et al. (2019) and Ribeiro et al. (2020) use templates to test the fairness and robustness of the text classification models. Bartolo et al. (2021) generate synthetic adversarial data against question answering models and improve the model robustness through adversarial training. Röttger et al. (2021) utilize templates to discover the failure of red team classifiers. Other prior works generate human-written texts to identify the model failures in human-in-theloop scenario. Dinan et al. (2019) propose build it, break it, fix it scheme, which repeatedly discovers failures of toxicity classifiers from human-model interactions and fixes it by retraining to enhance the robustness of the classifiers. Xu et al. (2021) adapt the notion of build it, break it, fix it scheme to prevent harmful behavior of dialogue models. Recently, Perez et al. (2022) red team dialogue models using test cases generated by LM.
In the perspective of related recent machine learning techniques, there has been a growing interest in utilizing BO to uncover the vulnerability of models. Ru et al. (2020), Wan et al. (2021), and Lee et al. (2022) conduct BO to search adversarial examples against classification models on image, graph, and text domains. Lee et al. (2022) improve the scalability of BO by utilizing the Subset of Data (SoD) method and batching based on DPP prior (Chalupka et al., 2013; Kulesza, 2012).

heading: 6 Conclusion
text: Our work aims to identify the potential risk of offensive behavior in black-box large-scale generative models by red teaming in a limited query regime. We propose BRT, a novel query-efficient black-box red-teaming method using BO. BRT methods construct a user input pool and iteratively choose or edit user inputs using BO to generate diverse positive test cases. In contrast to prior works, BRT can incorporate the information from past evaluations using GP to efficiently identify diverse failures. The experimental results show that BRT consistently outperforms existing methods in finding a greater number of positive test cases with higher diversity on various generation tasks including open-domain dialogue, prompt continuation, and text-to-image generation, against various victim models under a query limit.

heading: Societal and Ethical Impact

heading: Importance of Query-Efficient Black-Box Red
text: Teaming. It is becoming more common for large generative models to be used in the form of API (Brown et al., 2020; Chowdhery et al., 2022; Ramesh et al., 2022). Moreover, API users can fine-tune the black-box model using custom datasets through API and build personalized applications such as personalized chatbots (OpenAI, 2023). Since each query to the API usually incurs costs, the development of techniques that can query-efficiently identify model failures is essential for cost-effective AI safety. Hence, our proposed BRT methods can be valuable tools in this regard.
Broader Ethical Impact. Red teaming research is crucial to make large generative models safer and more reliable by white-hacking, in particular, for deployment, thus ultimately aiming the sustainable AI for humans. We mainly focus on describing BRT for offensive results. Even though there are potential risks of an adversary abusing BRT to generate socially harmful contents, we believe that our results can give insights to AI research groups and industries for training safer large generative models and applying them to real-world applications for users under various scenarios.

heading: Limitations
text: We utilize safety classifier modules, such as the BAD classifier and Perspective API, as the red team classifier to automatically identify offensive output from the victim model following the practice in Perez et al. (2022). However, automatic classification of offensive outputs can be subject to inaccuracies, which may lead to the identification of false positive test cases (Gehman et al., 2020). To mitigate this issue, we may increase the threshold for positive texts to reduce the number of discovered false positive test cases. One other choice is incorporating human supervision into the classification. For example, we may assume the human-in-theloop scenario that has access to the offensiveness scores evaluated by human annotators within a limited number of queries to the annotators. In this scenario, we can either directly conduct BRT with human annotators as the red team classifier or modify the BRT method to incorporate offensiveness scores from both human annotators and the safety classifier modules during red teaming. Further exploration of these possibilities is left as future work.

heading: Acknowledgement
text: This work was supported by SNU-NAVER Hyperscale AI Center, Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-00882, (SW STAR LAB) Development of deployable learning intelligence via selfsustainable and trustworthy machine learning and No. 2022-0-00480, Development of Training and Inference Methods for Goal-Oriented Artificial Intelligence Agents). Hyun Oh Song is the corresponding author.

heading: A Algorithms
text: The overall algorithm of BRT (s) and BRT (e) is shown in Algorithm 1 and Algorithm 2, respectively. Refer to Appendix D.5.1 for the process of adapting λ.

heading: A.1 Overall Algorithm of BRT (s)
text: Notations used in Algorithm 1
θ ∈ Θ Parameters of the surrogate GP. Û ⊂ U The user input pool. Gθ : U → O The victim model. Rϕ : U ×O → [−1, 1] The red team classifier. f : U → [−1, 1] The black-box red team score function. f(u) := Rϕ(u,Gθ(u)). g : U → R≥0 The white-box diversity function. g(u; T ) := Self-BLEU(k)({u} ∪ T +). Lλ : U → R The objective function. Lλ(u) := f(u)− λg(u; T ). EIλ : U → R≥0 The expected improvement of Lλ. L+λ ∈ R The reference term used in expected improvement. D ⊂ Û × [−1, 1] The evaluation history. Dsub ⊂ D The subsampled evaluation history used in BO steps. NE ∈ N Exploration budget. NQ ∈ N Query budget. NB ∈ N The batch size. Nsub ∈ N The maximum size of |Dsub|. D ∈ R≥0 The diversity budget. λ ∈ R≥0 Diversity trade-off coefficient. λinit ∈ R≥0 The initial value of λ. ρ ∈ R≥0 The amount of modification to λ for each step. δ ∈ R≥0 The capability of λ-adaptation technique.
Algorithm 1 BRT (s)
1: Input: The user input pool Û , the victim model Gθ, the red team classifer Rϕ. 2: Initialize T ∼ Unif(
( Û NE ) ).
3: Initialize D ← {(t, f(t)}t∈T . 4: Initialize λ← λinit. 5: while |D| < NQ do 6: Sample Dsub of size Nsub by SoD on D (Refer to Appendix C.3.1). 7: Fit GP parameters θ to maximize the posterior probability distribution on Dsub. 8: Construct a batch B ⊂ Û \ T of the size min(NB, NQ − |D|) according to EIλ(· | Dsub, θ) scores and the DPP prior (Refer to Appendix C.3.2). 9: Evaluate the batch Dbatch = {(t, f(t))}t∈B .
10: Update the test case set T ← T ∪B. 11: Update the evaluation history D ← D ∪Dbatch. 12: if Self-BLEU(k)(T +) > D then 13: λ← λ× ρ. 14: else if Self-BLEU(k)(T +) < D − δ then 15: λ← λ / ρ. 16: end if 17: Update the white-box terms {g(u; T )}u∈Û . 18: Update the reference term L+λ of EIλ. L+λ ← maxt∈T [min(f(t), 0) + λg(t; T )] . 19: end while 20: Return T , T +.

heading: A.2 Overall Algorithm of BRT (e)
text: Distinct notations used in Algorithm 2 relative to Algorithm 1
θselect ∈ Θ Parameters of the selector GP. θedit ∈ Θ Parameters of the editor GP. Bϵ(V ) ϵ-ball of a text set V . D ⊂ Û × Bϵ(Û)× [−1, 1] The evaluation history. Dsub ⊂ D The subsampled evaluation history used in BO steps.
Algorithm 2 BRT (e)
1: Input: The user input pool Û , the victim model Gθ, the red team classifer Rϕ. 2: Initialize T ∼ Unif(
( Û NE ) ).
3: Initialize D ← {(t, t, f(t)}t∈T . 4: Initialize λ← λinit. 5: while |D| < NQ do 6: Sample Dsub of size Nsub by SoD on D (Refer to Appendix C.3.1). 7: Fit θselect to maximize the posterior probability distribution on {(t, f(tedit))}(t,tedit,f(tedit))∈Dsub . 8: Fit θedit to maximize the posterior probability distribution on {(tedit, f(tedit))}(t,tedit,f(tedit))∈Dsub . 9: Construct a batch B ⊂ Û \ T of the size min(NB, NQ − |D|) according to EIλ(· | Dsub, θselect)
scores and the DPP prior (Refer to Appendix C.3.2). 10: Initialize Bedit ← ∅, Dbatch ← ∅. 11: for t in B do 12: Compute the white-box terms {g(u; T )}u∈Bϵ({t}). 13: Find the best edit candidate tedit ∈ Bϵ({t}) which maximizes EI(· | Dsub, θedit). 14: Evaluate tedit. Dbatch ← Dbatch ∪ {(t, tedit, f(tedit)}. 15: Bedit ← Bedit ∪ {tedit}. 16: end for 17: Update the test case set T ← T ∪Bedit. 18: Update the evaluation history D ← D ∪Dbatch. 19: if Self-BLEU(k)(T +) > D then 20: λ← λ× ρ. 21: else if Self-BLEU(k)(T +) < D − δ then 22: λ← λ / ρ. 23: end if 24: Update the white-box terms {g(u; T )}u∈Û∪T . 25: Update the reference term L+λ of EIλ. L+λ ← maxt∈T [min(f(t), 0) + λg(t; T )] . 26: end while 27: Return T , T +.

heading: B Derivation of Equation (3)
text: For the evaluated test case set Tn = {t1, . . . , tn}, the objective Lλ(u; Tn) can be decomposed to the black-box red team score function f(u) and the white-box diversity function g(u; Tn). Since g is a deterministic white-box function,
E[g(u; Tn) | u,D] = g(u; Tn),
Var[g(u; Tn) | u,D] = 0.
Hence, we can derive Equation (3) as following:
E[Lλ(u) | u,D] = E[f(u)− λg(u; Tn) | u,D] = E[f(u) | u,D]− λE[g(u; Tn) | u,D] = E[f(u) | u,D]− λg(u; Tn),
Var[Lλ(u) | u,D] = Var[f(u)− λg(u; Tn) | u,D] = Var[f(u) | u,D]− λVar[g(u; Tn) | u,D]︸ ︷︷ ︸
=0
= Var[f(u) | u,D].

heading: C Bayesian Optimization
text: In this section, we describe the continuous feature of the GP model. We then explain the GP model fitting procedure. Finally, we present the techniques to improve the scalability of BRT.

heading: C.1 Continuous Feature
text: We compute the sentence embedding emb(u) of a user input u using a pre-trained transformer. Specifically, we use the all-distilroberta-v1 model of sentence_transformer library (Liu et al., 2019; Reimers and Gurevych, 2019). Then, we use the sentence embedding as the continuous feature for the GP model, i.e., c(u) = emb(u).

heading: C.2 GP Model Fitting
text: We fit GP parameter θ to maximize the log posterior probability distribution on Dsub, log(p(θ | Dsub)). From Bayes theorem, the posterior probability is decomposed into the log maginal likelihood and the log prior probabililty as following:
log(p(θ | Dsub)) = log(p(Dsub | θ)) + log(p(θ))− log(p(Dsub)).
Algorithm 3 Subset of Data 1: Input: The evaluation historyD, the evaluated
test case set Tn, and the size of subset Nsub. 2: if |D| < Nsub then 3: Return D. 4: end if 5: Initialize Tsub ← {t0} where t0 ∼ Unif(Tn). 6: while |Tsub| < Nsub do 7: Select tfar ∈ Tn \ Tsub which minimizes d(t) ≜ maxt′∈Tsub cos(c(t), c(t′)). 8: Update Tsub ← Tsub ∪ {tfar} 9: end while
10: Dsub ← {(t, f(t)) ∈ D | t ∈ Tsub}. 11: Return Dsub.
Since p(Dsub) is a constant term, the problem of maximizing the log posterior probability is equivalent to the following maximization problem:
maximize θ∈Θ
log(p(Dsub | θ)) + log(p(θ)). (4)
We use Adam, a first order optimization method to optimize Equation (4) (Kingma and Ba, 2015). We set the learning rate to 0.1 and update θ for 20 iterations, with the initial values set to the GP parameters from the previous step (using warm start).

heading: C.3 Techniques for Scalability
text: We utilize two techniques, history subsampling and batching, to improve scalability of Bayesian optimization following the practice of Lee et al. (2022). We outline the process of these techniques for the sake of completeness.

heading: C.3.1 History Subsampling
text: Farthest Point Clustering (FPC)-based Subset of Data (SoD) method samples the subset Dsub of the evaluation history D = {(ti, f(ti))}ni=1 (Chalupka et al., 2013). To start, we randomly sample a test case t from the evaluated test case set Tn = {t1, . . . , tn}. Then, we sequentially select the test case that minimizes cosine similarity to the most similar test case among all previously selected test cases. This procedure continues until the subset size reaches Nsub. We use Nsub = 1000 for all experiments we consider. If |D| > 10000, we sample a subset of size 10000 randomly from D and conduct SoD to the sampled subset to obtain the subset Dsub of size Nsub. The overall process of SoD is summarized in Algorithm 3.
C.3.2 Batching with the DPP prior For each step, Selector GP constructs a batch B ⊂ Û of the size NB = 10 using the DPP prior to promote batch diversity (Kathuria et al., 2016). The DPP prior of a batch B is defined as the determinant of the posterior variance matrix, Var(f(B) | B,D). We first construct the user input set H ⊂ Û of the top-200 acquisition values. Then, we initialize the batch B = {u∗} where u∗ ∈ H is the maximizer of the acquisition function. We greedily append the maximizer u′ ∈ H\B of the DPP prior Var(g(B ∪ {u} | D)) to B while |B| ≤ 10.
D Implementation Details
In this section, we outline the implementation details of our work.

heading: D.1 User Input Pools
text: We construct user input pools using utterances from dialogues and utterances zero-shot generated by LM. In this section, we provide description of user input pools we used.

heading: D.1.1 Open Domain Dialogue
text: Following the practice of Perez et al. (2022), we generate utterances in zero-shot using the zero-shot prompt
> List of questions to ask someone: > 1.
using the pre-trained Bloom and OPT-66B models, respectively (Scao et al., 2022; Zhang et al., 2022). We generate utterances by nucleus (top-P ) sampling among top-K token candidates for P = 0.95, K = 50 with the temperature T = 1 (Holtzman et al., 2020). The generation process continues until the model samples the end-of-sentence token or a token containing ‘\n’ or ‘2’. We sample a total of 1 million unique utterances from the Bloom model and 500,000 unique utterances from the OPT-66B model. To improve memory efficiency, we use LLM.int8(), a quantization technique that does not compromise performance during generation. We utilize the implementation of LLM.int8() in bitsandbytes library (Dettmers et al., 2022). We perform the process above in a machine with Intel Xeon Gold 6338 CPU and four A100 GPUs.
We construct user input pools using the utterances in the training sets of dialogue datasets (Empathetic Dialogues, ConvAI2, BAD, DailyDialog)
(Rashkin et al., 2019; Dinan et al., 2020; Xu et al., 2021; Li et al., 2017). We collect the utterances in the training set of each dialogue dataset using ParlAI library, a unified platform for dialogue tasks (Miller et al., 2017). We remove redundant utterances and construct Empathetic Dialogues, ConvAI2, and BAD user input pools of sizes 63 K, 116 K, and 63 K, respectively.

heading: D.1.2 Prompt Continuation
text: For prompt continuation task, we use the set of initial prompts in Real Toxicity Prompt dataset as the user input pool (Gehman et al., 2020). We utilize the Real Toxicity Prompt dataset open-sourced in Hugging Face library (Wolf et al., 2020).

heading: D.1.3 Text-to-Image Generation
text: For Text-to-Image Generation, we construct user input pool with LM generated zero-shot outputs using nine zero-shot prompts with the following template,
> List of prompts that start with “[A]” that generates [B] images: > 1.
where [A] is one of ‘Image of’, ‘Photo of’, or ‘Picture of’, and [B] is one of ‘various’, ‘real-world object’, or ‘human-related’. We use the pre-trained OPT-66B model with LLM.int8() and generate user inputs for text-to-image model as described in Appendix D.1.1. The generation result is a total of 79,072 unique utterances generated by LM.
D.2 Victim Models For the open domain dialogue task, we use BB-3B, GODEL-large, DialoGPT-large, Marv, and Friend chat models as the victim models (Roller et al., 2021; Peng et al., 2022; Zhang et al., 2020; Brown et al., 2020). We generate the response of a dialogue model to a user input using greedy decoding on the dialogue models except for two GPT-3.5 based chatbots, Marv and Friend chat. For Marv and Friend chat, we follow the OpenAI API’s decoding instructions4.
In the prompt continuation task, we consider GPT-3 as the victim model (Brown et al., 2020). Instead of directly performing inference on the GPT-3 model, we utilize data of initial prompts and GPT3’s continuations from Real Toxicity Prompt.
4Marv: https://platform.openai.com/examples/defaultmarv-sarcastic-chat, Friend chat: https://platform.openai.com/examples/defaultfriend-chat
The victim model of text-to-image generation task is Stable Diffusion, a recently open-sourced image generation model. We use version 1.4 of the Stable Diffusion model (Rombach et al., 2022). We generate 512 by 512 pixels images with pseudo linear multistep sampler (PLMS) (Liu et al., 2022).

heading: D.3 Baselines
text: In this subsection, we describe two generation based red teaming methods, SFS and SL (Perez et al., 2022). We first introduce the process of SFS and SL in the open domain chatbot task. For each generation step, SFS samples 5 user inputs u1, . . . , u5 from the user input pool with probability proportional to exp(Rϕ(u,Gθ(u))/0.1). Then, SFS constructs the few-shot prompt as following:
> List of questions to ask someone: > 1. ————— u1 —————– > 2. ————— u2 —————– > 3. ————— u3 —————– > 4. ————— u4 —————– > 5. ————— u5 —————– > 6.
SFS generates test cases by continuing the constructed few-shot prompt. Similar to zero-shot generation in Appendix D.1, we use nucleus (topP ) sampling among top-K token candidates for P = 0.95, K = 50 with the temperature T = 1. The generation process halts when LM samples the eos token or a token containing ‘\n’ or ‘7’. We consider the pre-trained OPT-1.3B and Bloom models as the LM.
SL fine-tunes the OPT-1.3B model parameters to maximize the log-likelihood of positive user inputs in the user input pool condition on the zero-shot prompt:
> List of questions to ask someone: > 1.
Then, SL generates test cases in zero-shot using the zero-shot prompt. We randomly sample 90% of positive user inputs in Û to form a training set and the remaining positive user inputs as validation set. We run Adam optimizer with batch size 32 for the minimum of 1 epoch and 300 update steps (Kingma and Ba, 2015). We vary the learning rate in the range of [5× 10−9, 2× 10−8, 5× 10−8, 2× 10−7, 5×10−7, 2×10−6, 5×10−6, 2×10−5, 5×10−5, 2 × 10−4] and choose the trained parameters of the best validation accuracy. In the text-to-image
generation task, we construct few-shot prompt for SFS as
> List of prompts that start with “Image of” that generates various images: > 1. ————— u1 —————– > 2. ————— u2 —————– > 3. ————— u3 —————– > 4. ————— u4 —————– > 5. ————— u5 —————– > 6.
For SL, we use the following zero-shot prompt:
> List of prompts that start with “Image of” that generates various images: > 1.
Then, we conduct the same process above to finetune the model parameters and generate utterances in zero-shot using fine-tuned model.

heading: D.4 Red Team Classifiers and Input Offensiveness Classifiers
text: We provide the descriptions of red team classifiers and input offensiveness classifiers used in each task. For the open domain dialogue task, we utilize the BAD classifier which measures the offensiveness score of a dialogue. We normalize the output score of BAD classifier to [−1, 1] and define the input offensiveness score and the red team score functions as following:
rϕ(u) = BAD(u),
Rϕ(u, o) = BAD(u ∥ o),
where u ∈ U is a user input, and o ∈ O is a victim model output.
Real Toxicity Prompt dataset contain the initial prompts and their continuations. The dataset contain the offensiveness scores (toxicity and profanity) of the prompts and continuations evaluated by Perspective API. We utilize the offensiveness scores in the dataset for the input offensiveness scores and the red team scores of user inputs in the user input pool. Concretely, the input offensiveness score and the red team score functions in Real Toxicity Prompt dataset can be written as:
rϕ(u) = PerspectiveAPI(u),
Rϕ(u, o) = PerspectiveAPI(o).
where u is an initial prompt, o is the continuation of u by GPT-3, PerspectiveAPI(u) and
PerspectiveAPI(o) are offensiveness scores of u and o, respectively. Here, we normalize the offensiveness scores to [−1, 1].
The text-to-image generation task uses official implementation of Stable Diffusion safety filter5 which identifies NSFW images. The classifier has a predefined list of the CLIP embeddings and the corresponding thresholds of NSFW-related concepts such as ‘sexual’, ‘18+’, and ‘nude’ then calculates the cosine similarity between the CLIP embedding of a given image and the i-th concept, denoted by cosconcept(o, i), where o is the image to be classified (Radford et al., 2021; Rando et al., 2022). An image is classified as safe if all cosine similarities are below its threshold, where the i-th threshold is denoted by tconcept(i). All tconcept(i)s are lowered by 0.01 if the image exhibits high cosine similarity with any of the CLIP embedding of special concepts including ‘young girl’ and ‘young child’ compared to the corresponding thresholds, where the j-th cosine similarity and the threshold of the j-th special concept are denoted by cosspecial(o, j) and tspecial(j), respectively. Therefore, the red team score function is given as following:
Rϕ(u, o) = max i (cosconcept(o, i)− tconcept(i)) + (0.01)max
j (1 [cosspecial(o, j) > tspecial(j)]) .
This score takes values in [−1.22, 0.82]. We therefore normalize this score to [−1, 1] using a piecewise linear function f defined as:
f(x) = { x/1.22 if x < 0, x/0.82 otherwise.
Note that we conform to the use restrictions6 and generate NSFW images for research purpose only.

heading: D.5 BRT

heading: D.5.1 Adapting hyper-parameter λ
text: To guide the resulting positive test cases of BRT to satisfy the diversity budget D of Equation (1), we initialize λ to λinit and adjust λ adaptively based on the diversity of the current positive test cases at each step. Concretely, we multiply ρ > 1 to λ when Self-BLEU(k)(T +) > D, and divide λ by ρ when Self-BLEU(k)(T +) < D − δ.
5https://github.com/huggingface/diffusers/blob/main/src/ diffusers/pipelines/stable_diffusion/safety_checker.py
6https://github.com/CompVis/stable-diffusion/blob/main/ LICENSE

heading: D.5.2 Proxy of the White-Box Diversity Function
text: In practice, we first sample an l-subsetW from T +n and use a cheaper white-box function BLEU(u,W ) as a proxy for the white-box term g(u; Tn) to improve the efficiency. We update g periodically every NP step. We set l = 500 and NP = 10 for all experiments.

heading: D.5.3 Word Replacement Operation
text: We use a pre-trained masked language model, RoBerta-large, to generate the candidates for word replacement adapting the protocol of Garg and Ramakrishnan (2020) (Liu et al., 2019). Specifically, given a word w in a user input u, we first replace w with the mask token. Then, the pretrained RoBerta-large model predicts the token for the replaced mask token. We discard tokens with predicted probability smaller than 5 × 10−4, and use the remaining tokens of the top-40 predicted probabilities as candidates. Finally, we filter out the candidates that has part-of-speech (POS) different to the original wordw based on nltk POS tagger (Bird and Loper, 2004). We adapt the word substitution module in TextAttack API to implement the process above (Morris et al., 2020).
Editor GP finds the best edited test case tedit ∈ Bϵ({t}) where t ∈ Û is the user input selected by selector GP. Editor GP conducts greedy ascent to find the best edit in the ϵ-ball. Formally, editor GP initializes tedit ← t and iterates the following greedy step for ϵ times:
tedit ← argmax t′∈B1({tedit}) EIλ(t ′).
Then, editor GP selects the resulting tedit as the edited test case. The 1-ball of a text u is defined as the set of texts generated by single word replacement operation to u. To improve the scalability of the editing procedure for long user inputs, we randomly sample a maximum of 20 words from a text u and only consider the set of texts generated by replacing one of these words as the search space for each greedy ascent step.

heading: D.5.4 Hyper-parameters
text: In all experiments, we set the exploration budget NE = 50, the batch size NB = 10, and the subsample size of Subset of Data Nsub = 1000. For BRT (e) and BRT (e+r), we set ϵ = 3.
We use the following configurations to adapt λ.
• Open-domain dialogue task and prompt continuation task: We initialize λ to λinit = 0.3 for BRT (s) and λinit = 0.03 for BRT (e) for adapting λ. We set ρ = 1.01, δ = 1.
• Text-to-image generation task: We initialize λ to λinit = 0.03. We set ρ = 1.01 and δ = 1.
• Figure 4: We initialize λ to λinit = 1.0 for BRT (e+r). We set ρ = 1.03 and δ = 1.
In the open domain dialogue task (Table 4, Table 5, Table 6, Table 8, Table 10, Table 11), we use Self-BLEU(k) of Rand minus 0.1 as the value of D for BRT (s), and use Self-BLEU(k) of Offensive Top-NQ minus 0.1 for BRT (s+r). Lastly, for BRT (e) and BRT (e+r), we set D to the smallest Self-BLEU(k) of the baseline methods minus 0.1.
For the experiments in prompt continuation task (Table 7), we set D to 20. For the text-to-image generation task, we set D to 53 for all experiments.

heading: D.5.5 Machine
text: We conduct our experiments on a machine with AMD EPYC 7402 CPU and NVIDIA GeForce RTX 3090 GPU. Under a query limit of NQ = 20,000, the BRT process finishes within one GPU day for user input pools in the open domain dialogue task. Specifically, the run-time for BRT (s) and BRT (e) in the ConvAI2 user input pool are 3 hours and 13 hours, respectively, on a single GPU machine.

heading: E Additional Experiments
text: In this section, we provide the additional analysis and experimental results.

heading: E.1 Text-to-Image Generation Task
text: Table 9 shows that BRT finds a significantly larger number of positive test cases that generate NSFW
images compared to the baseline methods, demonstrating the general effectiveness and applicability of BRT in multiple domains including textto-image generation. Specifically, BRT (s) and BRT (e) both outperforms their respective baselines in RSR and Self-BLEU(k). This shows that our method is capable of red teaming the text-to-image generation domain.

heading: E.2 Open-Domain Dialogue Task

heading: E.2.1 Red Teaming Results against GODEL-Large Model
text: We also compare BRT and the baseline methods against GODEL-large model on the open-domain dialogue task. Table 8 shows that BRT methods outperforms the baseline methods in both RSR and the diversity Self-BLEU(k) under a query limit of 20,000. Moreover, Table 10 demonstrate that BRT methods find significantly larger number of hard positive test cases with higher diversity than baselines under a query limit of 20,000.

heading: E.2.2 Red Teaming Results on DailyDialog Dataset
text: Table 11 shows the red teaming results on DailyDialog dataset against BB-3B and DailoGPT-large. The results show that BRT (e) discovers more than 1.8 times larger number of positive test cases compared to the baseline methods, SFS and SL.

heading: E.2.3 Human Evaluation
text: We further employed Amazon Mechanical Turk (MTurk) to perform human evaluation of the red teaming results in ConvAI2 against BB-3B. For each method, we randomly sample 500 test cases
from a total of 20,000 test cases (from Table 4). For each sampled test case t, we asked three evaluators vote on the offensiveness of a dialogue t ∥ Gθ(t). We identified the test case as positive if two or more evaluators vote the dialogue as offensive. Table 12 summarizes the human evaluation results. The results show that BRT (e) also discovers a significantly greater number of test cases identified as positive by MTurk compared to both SFS and SL (corresponding to column ‘P’).
There exists a potential risk that the BRT methods, which aim to maximize the red team score during the red teaming procedure, may discover test cases that over-fit the red team classifier, resulting in false positive test cases. To address this, we evaluate precision, defined as the ratio of true positive test cases (those identified as positive by both BAD and MTurk) among all positive test cases (those identified as positive by BAD). Table 12 shows that BRT (e) achieves higher precision compared to the baseline methods, suggesting that the extent of over-fitting is not severe empirically in ConvAI2. Nevertheless, it is crucial to prevent over-fitting to ensure the trustworthiness of the red teaming results. To mitigate over-fitting, one possibility is to utilize robust red team classifiers learned through adversarial training methods or incorporate adversarial example detection techniques into the BRT framework (Yoo and Qi, 2021; Zhou et al., 2021; Pang et al., 2017; Yoo et al., 2022). We leave this issue as a topic for future work.
E.2.4 Ablation on diversity trade-off coefficient
We adjust the diversity trade-off coefficient λ during the BRT process and guide the diversity of positive test cases to satisfy the diversity constraint.
To determine the validity of this technique, we compare BRT (e+r) with its fixed-λ counterparts, namely, BRT fix(e+r). BRT fix(e+r) uses a fixed value of λ during red teaming. Figure 4 shows that BRT (e+r) for D ∈ {40.0, 43.0} meets the diversity constraint and also achieves RSR comparable to BRT fix(e+r).
E.2.5 Incorporating Sentence Quality Measures into BRT
BRT can accommodate any measures of sentence quality—diversity, fluency, grammaticality—into the objective. In this subsection, we calculate the perplexity values of user inputs via GPT-2 and utilize these values as an automative measure of sentence fluency. Instead of minimizing Lλ(u; Tn), we minimize the following objective:
Lλ,η(u; Tn) := f(u)− λg(u; Tn)− ηh(u),
where h(u) := (1− perp(u)/300). From now on, we denote this method by BRT (s) + perp.
We conduct BRT (s) + perp on Empathetic Dialogues against BB-3B (NQ = 20,000). Figure 5 illustrates the red teaming results for various η values. As η values increase, we observe a corresponding decrease in perplexity. Specifically, when we set η = 0.01, the perplexity notably diminishes from 133 to 94, without compromising the RSR and Self-BLEU score at all.

heading: E.3 Additional Qualitative Results
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
We provide the limitations of our work in the Limitation section (page 9).
3 A2. Did you discuss any potential risks of your work? We provide potential risks of our work in the Ethics Statement section (page 9).
3 A3. Do the abstract and introduction summarize the paper’s main claims? We provide abstract and introduction summarizing our main claims in the Introduction section (Section 1) (page 1-2)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We provide the scientific artifacts such as parlai, huggingface in the Implementation Detail section (Section D) in supplementary.
3 B1. Did you cite the creators of artifacts you used? Yes. We cite them in the Implementation Detail section (Section D) in supplementary.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Yes. We describe the open-source APIs used in our research in the implementation detail section (Section D).
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Yes. We describe the purpose of the artifacts and our use in the implementation detail section (Section D).
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We utilize the safety classifier modules such as BAD or Perspective API. We describe this in the implementation detail section (Section D.4)
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Yes. We provide coverage of domains in the implementation detail section (Section D.1, D.2).
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Yes. We provide it in Table 1 of our main part. We describe this in detail in the implementation detail section (Section D.1).
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? We provide it in Experiments section (Section 4) and Additional Experiments section (Section E).
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Yes. We provide it in Table 1, Experiments section (Section 4.1.1) and the Implementation Detail section (Section D).
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Yes. We provide this in the Implementation Detail section (Section D).
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Yes. We provide the mean and std of our evaluation metrics for 3 runs in Experiments section (Section 4) and Additional Experiments section (Section E).
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Yes. We describe this in Section 2.2, Section 4.1.3, and Section D.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave this issue as a topic for future work. E. 2. 4 Ablation on diversity trade-off coefficient
We adjust the diversity trade-off coefficient λ during the BRT process and guide the diversity of positive test cases to satisfy the diversity constraint. To determine the validity of this technique, we compare BRT (e+r) with its fixed-λ counterparts, namely, BRT fix(e+r). BRT fix(e+r) uses a fixed value of λ during red teaming. Figure 4 shows that BRT (e+r) for D ∈ {40. 0, 43. 0} meets the diversity constraint and also achieves RSR comparable to BRT fix(e+r). E. 2. 5 Incorporating Sentence Quality Measures into BRT
BRT can accommodate any measures of sentence quality—diversity, fluency, grammaticality—into the objective. In this subsection, we calculate the perplexity values of user inputs via GPT-2 and utilize these values as an automative measure of sentence fluency. Instead of minimizing Lλ(u; Tn), we minimize the following objective:
Lλ,η(u; Tn) := f(u)− λg(u; Tn)− ηh(u),
where h(u) := (1− perp(u)/300). From now on, we denote this method by BRT (s) + perp. We conduct BRT (s) + perp on Empathetic Dialogues against BB-3B (NQ = 20,000). Figure 5 illustrates the red teaming results for various η values. As η values increase, we observe a corresponding decrease in perplexity. Specifically, when we set η = 0. 01, the perplexity notably diminishes from 133 to 94, without compromising the RSR and Self-BLEU score at all