Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates auditing LLM using discrete optimization and proposes a better way to solve the optimization problem. In experiment, the proposed method outperforms existing methods on solving the optimization problem and showcases failure modes of LLMs.

strength_and_weaknesses: Strength
This paper proposes a better way to solve the discrete optimization problem compared to prior works

Weakness
N/A

clarity,_quality,_novelty_and_reproducibility: This paper is well written and the proposed technique is novel. The experiment supports the claim of the paper and it could be replicated easily.

summary_of_the_review: The proposed techniques on discrete optimization are effective on finding failure modes of LLMs.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper casts the large language models (LLMs) targeting behavior evaluation as a discrete input-output scoring function so that the targeting input-output behavior examples can be located/generated as a discrete function optimization problem. In particular, this method can be used to locate failure behaviors of LLM models making it a means to audit LLM models.
strength_and_weaknesses: Strength
	• A simple and clear formulation to search for targeting properties of input-output behaviors via discrete optimization with relaxation. 
	•  A few effective insights in solving the formulated discrete optimization problem
	• The experimental results seem to demonstrate that the approach is promising
	

Weakness
        The writing of the key contribution --- decomposition and relaxation of the discrete optimization problem --- is not clear enough which might cause confusion for the readers.
clarity,_quality,_novelty_and_reproducibility: The approach proposed in this paper seems promising. The writing is mostly clear but the key contribution in section 3.3 is difficult to follow: (a) decomposing equation 4 into equation 5, and (b) linear relaxation. It is of modest originality with modest incremental new contribution.

summary_of_the_review: This paper provides an approach to  locate/generate the targeting input-output behaviors of an LLM without modifying the model itself --- formulating targeting properties of generating sequences as a discrete optimization so as following the literature to provide a linear relaxation to optimization. Two add-on improvement steps are made: (1) decomposing the objective into linearly approximation term and autoregressive term; (2) further approximate the linear team with averaging over random selected tokens instead of the whole vocabulary. The experimental results seems to demonstrate that the approach is promising. 

Details:
	1. Equation 2, please clarify whether the objective function is required to be differentiable. 
	2. Equation 3, will the scale difference between \phi(x, 0) and log P_{LLM} be a problem --- the auditing score scale or its variation might be overwhelmed by log probability or the other way around. Any thought? 
	3. Page 4, the paragraph before equation 5: please introduce the approximation \tilde{s}_i formally to be self-contained; also the equation 6. With the Tailor expansion context, it is too abrupt to go into equation 6 directly from equation 5.
	4. Equation 5 needs more explanation. It might be better for formulate your claim of benefit and then prove it formally.
	5. Page 5, the last paragraph: "\phi(x, 0) is sufficiently large", how large? Please detail. And any ablation study to determine this hyper-parameter? 
	6. Page 8,  the text after equation 8: " …without taking gradients", but it seems to me that algorithm 1 does require the gradients in line 2 but these gradients can be pre-computed though.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an efficient method to find prompts that will trigger undesirable behaviors of large language models, based on a coordinate ascending algorithm and an linear approximation which avoids doing forward-backward passes for every candidate token. 
strength_and_weaknesses: Strengths:
1. The proposed algorithm and approximation seems novel to my knowledge.
2. The problem of finding the prompt that causes certain behavior is interesting and important to study.
3. Some of the prompts indeed works even for Codex, e.g., "Barack Obama is a legalized unborn", "Florida governor", which seem to prove the effectiveness of the method.

Weakness:
1. From Figure 1, the effectiveness of the method can fall significantly when output length is longer, which is a huge limitation.
2. Most of the examples given in the paper look quite unnatural and difficult to understand, indicating it's difficult to expose undesirable behaviors in more natural interactions with the language models.
3. The evaluations are limited to only GPT-2, despite most instances only take less than 1 minute to run.
clarity,_quality,_novelty_and_reproducibility: Clarity: the approximated objective seems a bit confusing to me, and the experimental results are not demonstrated in a concise way. In Eq. 4, why is the $\lambda\_{p\_{LLM}}$ ignored? Same for the "linearly approximatable term" in Eq. 5. And Eq. 6 should be an average instead of just a sum, right?

Quality: the work studies an interesting and important problem, but more evaluations for different models could have been added to demonstrate the effectiveness of the method. 

Novelty: the method is novel to my knowledge.

Reproducibility: good. The method seems to have a few hyperparameters that could potentially affect reproducibility, but I find some of the prompts are indeed valid even for Codex.
summary_of_the_review: Overall I like the topic studied in the paper, and the method seems novel to me. Some of the prompts even work for other models. However, the effectiveness of the method does not seem good enough since it's success rate decays significantly for longer output lengths, and the prompts still look unnatural to me in most cases.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims at finding the undesired behaviors of large language models. Specifically, the goal is to find a prefix for decoding, such that the decoded text is toxic or satisfies some predefined score functions. The approach is implemented as a variant of gibbs sampling or the coordinate ascent as called in the paper. The paper also provides approximations for such objective functions in order to gain speed. Experiments compared to other prompt tuning show that the proposed approach is more effective at attacking the large language models. 

strength_and_weaknesses: Strength
- The safety issue of the large language models is an important topic nowadays. The topic studied in this paper aligns with the topic.
- The proposed approach is also reasonable, given the difficulty of doing discrete space optimization. The variant of first-order approximation also achieved good empirical results. 
- Experiments are done with practical LLMs (though GPT-2 is not considered as that large).

Weakness
- The practical usefulness of the paper needs further justification. In practice if we can have a risk function \phi(x, o), then one can simply reject the samples when the samples get high risks. So relying on \phi(x, o) to find the vulnerability of LLM may not be practical, as one can filter out these samples using the same \phi(x, o). We all know that the large language models are not safety guaranteed, and the proposed approach may not be super helpful in this context.

- Technically the coordinate-wise ascent may not be super efficient. One can potentially improve this further, in terms of 1) the approximation, and 2) the scope of the optimization. 
Regarding 1), the work named gibbs with gradient [1] might be helpful. 
Regarding 2), one can think of approaches that can flip multiple sites at a time [2], or flipping them in parallel in a factorized way [3].

References:

[1] Oops I Took A Gradient: Scalable Sampling for Discrete Distributions, Grathwohl et.al

[2] Path Auxiliary Proposal for MCMC in Discrete Space, Sun et.al,

[3] A Langevin-like Sampler for Discrete Distributions, Zhang et.al


clarity,_quality,_novelty_and_reproducibility: The paper is written clearly, and is also easy to follow. The quality of the experiments is Ok, where the qualitative and quantitative results are shown, w.r.t., baseline methods. The novelty is somewhat limited given the existing literature in discrete space optimization, and the prompt engineering works. It shouldn’t be too hard to reproduce the results.

summary_of_the_review: 
Overall this is a paper concerning the vulnerability of the LLMs. The significance of the specific formulation in this paper needs further justification, as one should be able to defend such kinds of attack easily using the same function \phi(x, o). 
The technical part can be further improved with the recent advances in discrete space sampling.

====

updated score after rebuttal
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes method for auditing large language model to uncover undesired outputs or failure modes. As safety issue of the large language models is an important topic nowadays, the paper is timely. The authors formulate the auditing as a discrete optimization problem and provide a novel approximate way to solve it based on a coordinate ascending algorithm in the embedding space. The proposed method seems to work better than gradient based approaches in logit space like GBDA. The proposed method was applied on GPT-2 to find problematic predictions. There is a wide variance among reviewer scores, but we thank the authors and reviewers to engage in discussion towards improving the paper including identifying new aspects like prompt transfer from GPT-2 to Codex which the authors developed further. Original submission had experiments on only GPT-2 which can't be called large language model currently, but we appreciate authors for providing some new experiments on larger model GPT-J. A reviewer found that many of the examples discovered by the proposed approach look unnatural and might be rare to encounter the discovered undesirable behaviors in natural interactions with the language models. Finally it is worth mentioning that the method is assumes availability of scoring functions for types of error we are looking for, which is a limitation as it can't discover new types of failures. Overall after the discussion phase, it seems that most key ingredients are present to be a strong and useful paper to the community but is somewhat nascent and needs further development, like evaluation on larger model and ideas of prompt transfer.  Incorporating all the reviewer feedback would require a fair amount of rewriting and thus warrant another round of reviewing.
justification_for_why_not_higher_score: - Examples discovered in original submission on smaller models look unnatural.
- Many interesting experiments only done during discussion phase, like on large models and prompt transfer.
- These ideas need further development and would require a fair amount of rewriting, thus warrant another round of reviewing.
Decision: Reject