Complex Logical Instruction Generation
Authors: name: Mian Zhang, name: Shujian Liu, name: Sixun Dong, name: Ming Yin, name: Yebowen Hu, name: Xun Wang, name: Steven Ma, name: Song Wang, name: Sathish Reddy Indurthi, name: Haoyun Deng, name: Zhiyu Zoey Chen, name: Kaiqiang Song

Instruction following has catalyzed the recent era of Large Language Models
(LLMs) and is the foundational skill underpinning more advanced capabilities
such as reasoning and agentic behaviors. As tasks grow more challenging, the
logic structures embedded in natural language instructions becomes increasingly
intricate. However, how well LLMs perform on such logic-rich instructions
remains under-explored. We propose LogicIFGen and LogicIFEval. LogicIFGen is a
scalable, automated framework for generating verifiable instructions from code
functions, which can naturally express rich logic such as conditionals,
nesting, recursion, and function calls. We further curate a collection of
complex code functions and use LogicIFGen to construct LogicIFEval, a benchmark
comprising 426 verifiable logic-rich instructions. Our experiments demonstrate
that current state-of-the-art LLMs still struggle to correctly follow the
instructions in LogicIFEval. Most LLMs can only follow fewer than 60% of the
instructions, revealing significant deficiencies in the instruction-following
ability. Code and Benchmark: https://github.com/mianzhang/LogicIF