Iterative Graph Alignment
Authors: name: Fangyuan Yu; name: Hardeep Singh Arora; name: Matt Johnson
Published: 2024-08-29 16:15:01

Abstract: By compressing diverse narratives, LLMs go beyond memorization, achieving
intelligence by capturing generalizable causal relationships. However, they
suffer from local 'representation gaps' due to insufficient training data
diversity, limiting their real-world utility, especially in tasks requiring
strict alignment to rules. Traditional alignment methods relying on heavy human
annotations are inefficient and unscalable. Recent self-alignment techniques
also fall short, as they often depend on self-selection based prompting and
memorization-based learning. To address these issues, we introduce Iterative
Graph Alignment (IGA), an annotation-free rule-based alignment algorithm. A
teacher model (VLM) employs Iterative Graph Prompting (IGP) to create logical
graphs and reference answers. The student model (LLM) identifies local
knowledge gaps by attempting to align its responses with these references,
collaborating with helper models to generate diverse answers. These aligned
responses are then used for iterative supervised fine-tuning (SFT). Our
evaluations across five rule-based scenarios demonstrate IGP's effectiveness,
with a 73.12\% alignment improvement in Claude Sonnet 3.5, and
Llama3-8B-Instruct achieving an 86.20\% improvement, outperforming Claude
Sonnet 3.5 in rule-based alignment.