CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding
  Capabilities of CodeLLMs
Authors: name: Dung Nguyen Manh, name: Thang Phan Chau, name: Nam Le Hai, name: Thong T. Doan, name: Nam V. Nguyen, name: Quang Pham, name: Nghi D. Q. Bui

Recent advancements in Code Large Language Models (CodeLLMs) have
predominantly focused on open-ended code generation tasks, often neglecting the
critical aspect of code understanding and comprehension. To bridge this gap, we
present CodeMMLU, a comprehensive multiple-choice question-answer benchmark
designed to evaluate the depth of software and code understanding in LLMs.
CodeMMLU includes over 10,000 questions sourced from diverse domains,
encompassing tasks such as code analysis, defect detection, and software
engineering principles across multiple programming languages. Unlike
traditional benchmarks, CodeMMLU assesses models's ability to reason about code
rather than merely generate it, providing deeper insights into their grasp of
complex software concepts and systems. Our extensive evaluation reveals that
even state-of-the-art models face significant challenges with CodeMMLU,
highlighting deficiencies in comprehension beyond code generation. By
underscoring the crucial relationship between code understanding and effective
generation, CodeMMLU serves as a vital resource for advancing AI-assisted
software development, ultimately aiming to create more reliable and capable
coding assistants.