Title: WidgetEval: Benchmarking Foundation Models on Dynamic Widget Generation for Apps
Primary area: datasets and benchmarks
Keywords: Code, language model, Javascript

Abstract: In this paper, we study the problem of generating application widgets on the fly based on context and user specifications. A widget in this setting encapsulates application APIs while providing a seamless user experience that integrates with the host application. We focus on domains such as Microsoft Excel, Microsoft Word, and synthetic environments such as calendar, file system, and messenger applications. Widget generation is a challenging problem that requires foundation models to understand APIs, write code, reason about application context, and design user interfaces. To understand foundation models’ ability to generate on-demand widgets, we first establish a dataset of widget tasks, test scenarios, and functionality verification mechanisms for various applications. We then devise an evaluation strategy which simulates user interaction with the generated widget and checks for runtime and functionality errors. We compare popular models with various prompting and generation strategies. Overall, we found closed source models such as GPT-4o, GPT-4.1 and o4-mini outperform models such as LLaMa-3.1-70B and Phi4, but even with the best configuration, their success rates on Excel and Word widget tasks remain around 50%.

TL;DR: A benchmark and evaluation framework to test LLM's widget generation capability.