Title: Factored Representation for Neuro-Symbolic AI
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords: reasoning, knowledge representation, large language models, neuro-symbolic

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, yet their internal decision-making processes remain largely opaque, posing a significant challenge to their trustworthiness in complex reasoning scenarios. We explore the hypothesis that compelling a pre-trained LLM to maintain a structured internal state using a formal, symbolic representation can enhance interpretability without degrading its reasoning performance. To investigate this, we employ two primary techniques: prompt fine-tuning and parameter-efficient fine-tuning (PEFT) using LoRA. Thus, we prompt a variety of LLMs to articulate their reasoning steps using various structured formalisms, including basic semantic triples, lists of attribute-value pairs, and first-order logic. We also fine-tune a pre-trained LLM on a structured representation that the LLM is subsequently prompted to use as an internal representation during reasoning to solve a task. Our results demonstrate that while state-of-the-art models struggle to generate consistently structured reasoning, their core reasoning capabilities remain largely intact. This suggests that the LLM's reasoning mechanism is not necessarily fully aligned with its generative capabilities. However, this result shows the potential for specialized models capable of performing complex reasoning while providing verifiable chains of thought.

TL;DR: Imposing reasoning constraints on an LLM to use a structured representation tends to conflict with its learned reasoning patterns but also shows the potential to improve model's reasoning performance.