ICML 2025

The Logical Implication Steering Method for Conditional Interventions on
Transformer Generation
Damjan Kalajdzievski 1
Abstract
The field of mechanistic interpretability in pre-
trained transformer models has demonstrated sub-
stantial evidence supporting the “linear represen-
tation hypothesis”, which is the idea that high
level concepts are encoded as vectors in the space
of activations of a model. Studies also show that
model generation behavior can be steered toward
a given concept by adding the concept’s vector
to the corresponding activations. We show how
to leverage these properties to build a form of
logical implication into models, enabling trans-
parent and interpretable adjustments that induce
a chosen generation behavior in response to the
presence of any given concept. Our method, Log-
ical Implication Model Steering (LIMS), unlocks
new hand-engineered reasoning capabilities by
integrating neuro-symbolic logic into pre-trained
transformer models.
1. Introduction
In this work, we are motivated by advances in mechanis-
tic interpretability which deepen our understanding of the
representations within pre-trained transformer models, as
well as by the goal of bridging neuro-symbolic and algo-
rithmic capabilities with these representations. Harnessing
the extensive knowledge and intuitive processing embedded
in these models in a high-level, interpretable, and algorith-
mic way, holds the potential to unlock significantly greater
reasoning capabilities and control.
Driven by these motivations, we introduce the Logical Impli-
cation Model Steering (LIMS) method, which extracts and
uses concept vectors to allow a user to “program” logical
implication into model generation behavior in a transparent
and interpretable manner. LIMS achieves this by implement-
ing the atomic building blocks for a general neuro-symbolic
1Salesforce. Correspondence to: Damjan Kalajdzievski <dam-
jank7354@gmail.com>.
Proceedings of the 42 nd International Conference on Machine
Learning...