Insights from Benchmarking Frontier Language Models on Web App Code
  Generation
Authors: name: Yi Cui
Published: 2024-09-08 18:24:26

Abstract: This paper presents insights from evaluating 16 frontier large language
models (LLMs) on the WebApp1K benchmark, a test suite designed to assess the
ability of LLMs to generate web application code. The results reveal that while
all models possess similar underlying knowledge, their performance is
differentiated by the frequency of mistakes they make. By analyzing lines of
code (LOC) and failure distributions, we find that writing correct code is more
complex than generating incorrect code. Furthermore, prompt engineering shows
limited efficacy in reducing errors beyond specific cases. These findings
suggest that further advancements in coding LLM should emphasize on model
reliability and mistake minimization.