Title: Relational Knowledge Distillation Using Finetuned Function Vectors
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, relational knowledge, function vectors, interpretability

Abstract: Representing relations between objects, entities, and concepts is a core prerequisite for intelligent systems to make sense of the world. Recent work using causal mediation analysis has shown that a small set of attention heads encodes task representation in in-context learning, captured in a compact representation known as the function vector. We first introduce a fine-tuning algorithm that adapts and refines relation-specific function vectors for relation tasks. In zero-shot evaluations, fine-tuned function vectors consistently outperform the original function vectors derived from causal mediation analysis across both small-scale and large-scale language models. Next, we test generalization by applying relation-specific function vectors to four-term analogy problems. We develop the composite function vector, a weighted sum of the fine-tuned function vectors, that captures relational knowledge required for analogical reasoning. At inference time, inserting the composite function vector into model activations substantially improves model performance, particularly on challenging reasoning tasks such as cross-domain analogy problems. Our results highlight the potential of activation patching as a controllable mechanism for encoding and manipulating relational knowledge, advancing both the interpretability and reasoning capabilities of large language models.