Coding Triangle: How Does Large Language Model Understand Code?
Authors: name: Taolin Zhang; name: Zihan Ma; name: Maosong Cao; name: Junnan Liu; name: Songyang Zhang; name: Kai Chen
Published: 2025-07-08 16:20:43

Abstract: Large language models (LLMs) have achieved remarkable progress in code
generation, yet their true programming competence remains underexplored. We
introduce the Code Triangle framework, which systematically evaluates LLMs
across three fundamental dimensions: editorial analysis, code implementation,
and test case generation. Through extensive experiments on competitive
programming benchmarks, we reveal that while LLMs can form a self-consistent
system across these dimensions, their solutions often lack the diversity and
robustness of human programmers. We identify a significant distribution shift
between model cognition and human expertise, with model errors tending to
cluster due to training data biases and limited reasoning transfer. Our study
demonstrates that incorporating human-generated editorials, solutions, and
diverse test cases, as well as leveraging model mixtures, can substantially
enhance both the performance and robustness of LLMs. Furthermore, we reveal
both the consistency and inconsistency in the cognition of LLMs that may
facilitate self-reflection and self-improvement, providing a potential
direction for developing more powerful coding models.