Title: POME: Post Optimization Model Edit via Matrix Orthogonalization
Primary area: optimization
Keywords: Large Language Model, Optimization, Fine-Tuning

Abstract: We revisit a basic question: whether a fine-tuned large language model can be improved after training using only its pretrained and fine-tuned checkpoints, without extra data or further optimization. We answer by introducing POME, a one-shot post-training method that edits the accumulated weight changes. POME isolates the most significant update directions via truncated singular value decomposition (SVD) and then equalizes their contributions, ensuring a more uniform and balanced model update. This simple post-processing is completely decoupled from the training pipeline, requiring zero modifications and imposing no overhead, making it universally compatible with any optimizer or distributed framework. POME delivers consistent gains, boosting average performance by +2.5\% on GSM8K and +1.0\% on code generation, while demonstrating broad applicability from 7B foundation models to 72B RLHF-instructed models, establishing it as a practical, zero-cost enhancement for any fine-tuning pipeline.