IHEval: Evaluating Language Models on Following the Instruction
  Hierarchy
Authors: name: Zhihan Zhang, name: Shiyang Li, name: Zixuan Zhang, name: Xin Liu, name: Haoming Jiang, name: Xianfeng Tang, name: Yifan Gao, name: Zheng Li, name: Haodong Wang, name: Zhaoxuan Tan, name: Yichuan Li, name: Qingyu Yin, name: Bing Yin, name: Meng Jiang

The instruction hierarchy, which establishes a priority order from system
messages to user messages, conversation history, and tool outputs, is essential
for ensuring consistent and safe behavior in language models (LMs). Despite its
importance, this topic receives limited attention, and there is a lack of
comprehensive benchmarks for evaluating models' ability to follow the
instruction hierarchy. We bridge this gap by introducing IHEval, a novel
benchmark comprising 3,538 examples across nine tasks, covering cases where
instructions in different priorities either align or conflict. Our evaluation
of popular LMs highlights their struggle to recognize instruction priorities.
All evaluated models experience a sharp performance decline when facing
conflicting instructions, compared to their original instruction-following
performance. Moreover, the most competitive open-source model only achieves 48%
accuracy in resolving such conflicts. Our results underscore the need for
targeted optimization in the future development of LMs.