Can LLMs Follow Simple Rules?
Authors: name: Norman Mu, name: Sarah Chen, name: Zifan Wang, name: Sizhe Chen, name: David Karamardian, name: Lulwa Aljeraisy, name: Dan Hendrycks, name: David Wagner

As Large Language Models (LLMs) are deployed with increasing real-world
responsibilities, it is important to be able to specify and constrain the
behavior of these systems in a reliable manner. Model developers may wish to
set explicit rules for the model, such as "do not generate abusive content",
but these may be circumvented by jailbreaking techniques. Evaluating how well
LLMs follow developer-provided rules in the face of adversarial inputs
typically requires manual review, which slows down monitoring and methods
development. To address this issue, we propose Rule-following Language
Evaluation Scenarios (RuLES), a programmatic framework for measuring
rule-following ability in LLMs. RuLES consists of 15 simple text scenarios in
which the model is instructed to obey a set of rules in natural language while
interacting with the human user. Each scenario has a concise evaluation program
to determine whether the model has broken any rules in a conversation. Through
manual exploration of model behavior in our scenarios, we identify 6 categories
of attack strategies and collect two suites of test cases: one consisting of
unique conversations from manual testing and one that systematically implements
strategies from the 6 categories. Across various popular proprietary and open
models such as GPT-4 and Llama 2, we find that all models are susceptible to a
wide variety of adversarial hand-crafted user inputs, though GPT-4 is the
best-performing model. Additionally, we evaluate open models under
gradient-based attacks and find significant vulnerabilities. We propose RuLES
as a challenging new setting for research into exploring and defending against
both manual and automatic attacks on LLMs.