Utility-Learning Tension in Self-Modifying Agents
Authors: name: Charles L. Wang, name: Keir Dorchen, name: Peter Jin

As systems trend toward superintelligence, a natural modeling premise is that
agents can self-improve along every facet of their own design. We formalize
this with a five-axis decomposition and a decision layer, separating incentives
from learning behavior and analyzing axes in isolation. Our central result
identifies and introduces a sharp utility--learning tension, the structural
conflict in self-modifying systems whereby utility-driven changes that improve
immediate or expected performance can also erode the statistical preconditions
for reliable learning and generalization. Our findings show that
distribution-free guarantees are preserved iff the policy-reachable model
family is uniformly capacity-bounded; when capacity can grow without limit,
utility-rational self-changes can render learnable tasks unlearnable. Under
standard assumptions common in practice, these axes reduce to the same capacity
criterion, yielding a single boundary for safe self-modification. Numerical
experiments across several axes validate the theory by comparing destructive
utility policies against our proposed two-gate policies that preserve
learnability.