Do great minds think alike? Investigating Human-AI Complementarity in
  Question Answering with CAIMIRA
Authors: name: Maharshi Gor, name: Hal Daumé III, name: Tianyi Zhou, name: Jordan Boyd-Graber

Recent advancements of large language models (LLMs) have led to claims of AI
surpassing humans in natural language processing (NLP) tasks such as textual
understanding and reasoning. This work investigates these assertions by
introducing CAIMIRA, a novel framework rooted in item response theory (IRT)
that enables quantitative assessment and comparison of problem-solving
abilities of question-answering (QA) agents: humans and AI systems. Through
analysis of over 300,000 responses from ~70 AI systems and 155 humans across
thousands of quiz questions, CAIMIRA uncovers distinct proficiency patterns in
knowledge domains and reasoning skills. Humans outperform AI systems in
knowledge-grounded abductive and conceptual reasoning, while state-of-the-art
LLMs like GPT-4 and LLaMA show superior performance on targeted information
retrieval and fact-based reasoning, particularly when information gaps are
well-defined and addressable through pattern matching or data retrieval. These
findings highlight the need for future QA tasks to focus on questions that
challenge not only higher-order reasoning and scientific thinking, but also
demand nuanced linguistic interpretation and cross-contextual knowledge
application, helping advance AI developments that better emulate or complement
human cognitive abilities in real-world problem-solving.