Title: Decomposing Prompts: Discovering Reusable Scaffolds and Task-Specific Residuals
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Prompt Optimization, Compositionality, Large Language Models, Instructional Scaffolds, Knowledge Reuse.

Abstract: Traditional automatic prompt optimization methods treat prompts as monolithic text blocks, necessitating costly, from-scratch optimization for each new task and precluding knowledge reuse across related tasks. We challenge this monolithic paradigm by proposing a novel approach where effective prompts are decomposed into two components: a reusable, domain-invariant **Instructional Scaffold** that captures high-level task structure, and a concise **Task-specific Residual** for fine-grained adaptation. We introduce **DiSR** (**D**iscovering **i**nstructional **S**caffolds and **R**esiduals), a two-stage algorithm inspired by the Minimum Description Length (MDL) principle, which provides a theoretical foundation for the decomposition of prompts into reusable components. This approach systematically discovers the components, allowing for efficient knowledge reuse and task-specific adaptation. In contrast to existing methods like OPRO and AutoPrompt, DiSR enables task-level knowledge reuse, improving both performance and reliability across different tasks. Extensive experiments demonstrate that DiSR not only achieves competitive accuracy but also consistently generates better-calibrated, more trustworthy prompts, with a clear advantage in large models and professional domains. This compositional hypothesis is validated through semantic analysis, revealing that optimized prompts form distinct, scaffold-centered clusters in their embedding space. Our findings establish a compositional view of prompt engineering, facilitating more robust, interpretable, and reusable optimization strategies.

TL;DR: We decompose prompts into a reusable scaffold and a task-specific residual to make prompt optimization more efficient and reliable.