Plasticity as the Mirror of Empowerment
Authors: David Abel
Michael Bowling
Andre Barreto
Will Dabney
Shi Dong
Steven Stenberg Hansen
Anna Harutyunyan
Khimya Khetarpal
Clare Lyle
Razvan Pascanu
Georgios Piliouras
Doina Precup
Jonathan Richens
Mark Rowland
Tom Schaul
Satinder Singh
Venue: NeurIPS 2025 spotlight ()
Keywords: Plasticity
Empowerment
Agents
Reinforcement Learning
Continual Reinforcement Learning
Information Theory
Primary area: reinforcement_learning
TL;DR: We provide a formal definition of plasticity under minimal assumptions, and highlight a fundamental connection to empowerment

Abstract: Agents are minimally entities that are influenced by their past observations and act to influence future observations. This latter capacity is captured by empowerment, which has served as a vital framing concept across artificial intelligence and cognitive science. This former capacity, however, is equally foundational: In what ways, and to what extent, can an agent be influenced by what it observes? In this paper, we ground this concept in a universal agent-centric measure that we refer to as plasticity, and reveal a fundamental connection to empowerment. Following a set of desiderata on a suitable definition, we define plasticity using a new information-theoretic quantity we call the generalized directed information. We show that this new quantity strictly generalizes the directed information introduced by Massey (1990) while preserving all of its desirable properties. Under this definition, we find that plasticity is well thought of as the mirror of empowerment: The two concepts are defined using the same measure, with only the direction of influence reversed. Our main result establishes a tension between the plasticity and empowerment of an agent, suggesting that agent design needs to be mindful of both characteristics. We explore the implications of these findings, and suggest that plasticity, empowerment, and their relationship are essential to understanding agency.

Plasticity as the Mirror of Empowerment
David Abel Google DeepMind
Michael Bowling
André Barreto
Will Dabney Google DeepMind
Amii, University of Alberta
Google DeepMind
Shi Dong Google DeepMind
Steven Hansen Google DeepMind
Anna Harutyunyan Google DeepMind
Khimya Khetarpal
Google DeepMind
Clare Lyle Google DeepMind
Razvan Pascanu Google DeepMind, Mila
Georgios Piliouras Google DeepMind
Doina Precup Google DeepMind
Jonathan Richens Google Deep
Mark Rowland Google DeepMind
Tom Schaul Google DeepMind
Satinder Singh Google DeepMind
Abstract
Agents minimally entities that influenced by their past observations and act to influence future observations. This latter capacity is captured by empowerment, which has served as vital framing concept across artificial intelligence and cognitive science. This former capacity, however, is equally foundational: what ways, and to what extent, paper, we ground this concept in a universal agent-centric measure that we refer to as plasticity, and reveal fundamental connection to empowerment. Following a set of desiderata on a suitable definition, we define plasticity using a new information- theoretic quantity call the generalized directed information. We show that this new quantity strictly generalizes the directed information introduced by Massey (1990) while preserving all of its desirable properties. Under this definition, we find that plasticity is well thought of as the mirror of empowerment: The two concepts are defined using the same measure, with only the direction of influence reversed. Our main result establishes tension between the plasticity and empowerment of an agent, suggesting that agent design needs be mindful of both characteristies. We explore the implications of these findings, and suggest that plasticity, empowerment, and their relationship are essential to understanding agency.
agent be influenced by what it observes? In this
Introduction
Two capacities represent fundamental aspects of any agent, regardless of design, physical substrate, goals: In what ways the agent be shaped by what it observes? And, what things do to shape aspects of its observable environment? system that entirely lacks either capacity-for instance, a machine whose outputs cannot be influenced by its inputs-is likely lacking agency altogether (Barandiaran et al., 2009). In this paper, we highlight a fundamental connection between these two capacities, which refer to empowerment and plasticity.
agent
Empowerment. Empowerment refers to an agent's ' ability to influence its observable future. For example, with a full tank of gas has higher empowerment than car with empty tank, precisely because the former can realize more outcomes than the latter. The concept of empowerment was first introduced by Klyubin et al. (2005b) as "an agent-centric quantification of the amount of control or influence the agent has and perceives" (p. 2). Since its inception, the concept has been expanded
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Enviroment
5 600
Anpurteemett
Plonietty
((1) = I(A-
B(AA= = 1[0 ~ A)
Agent
Plasticrg
Plasticity and Empowerment(
(h) Theorem 4.8
Figure 1: On the left, we depict the mirror of plasticity and empowerment. On the right, show the tight upper-bound on an agent's s plasticity (()) and empowerment ((A) established by Theorem 4.8: While values of (A) and (()) from zero up an interface- and interval-dependent constant realizable, their sum can be no greater than m.
to act as a plausible explanation of intrinsic drive in the absence of reward (Klyubin et al., 2005a; Mohamed and Jimenez Rezende, 2015; Ringstrom, 2023), framed key aspects of safety and alignment (Salge and Polani, 2017; Turner et al., 2021), skill discovery (Gregor et al., 2016; Campos et al., 2020; Baumli et al., 2021; Levy et al., 2023), social influence (Jaques et al., 2019), self- seff-ressrvation (Ringstrom, 2023), and has played meaningful role in explanatory theories of how people explore (Brändle et al., 2023).
Plasticity. Conversely, plasticity reflects an agent's capacity to "remain...adapivve in response to significant events" " (p. 2, Carpenter and Grossberg 1988), and has been studied extensively across neuroscience, biology, and machine learning. Within neuroscience, "plasticity" often refers to synaptic plasticity, and reflects "the capacity of the neural activity generated by an experience to modify neural circuit function" (p. 1, Citri and Malenka, 2008). It is central building block for understanding memory and forgetting (Fusi et al., 2005), often studied through the lens of the caabilit-plasttiie dilemma (Carpenter and Grossberg, 1988, 19.7a,b). In other areas of biology. plasticity is treated roughly as "the power of an organism to adapt action" " (p. 531, Wheeler 1910), or more generally the "ennironmental responsiveness" of organism (p. 34, West-Eberhard, 2003). Within machine learning, plasticity is often studied under the lens of the loss of plasticity (Dohare et al., 2021, 2024), which is especially prevalent within neural networks (Lyle al., 2023; Abbas al., 2023). Each of Chen et al. (2023), Raghavan and Balaprakash (2021), and Kumar et al. (2023) further study plasticity under the lens of the stability-plar pplasiiciy dilemma in different learning settings. Across all of this work, it is clear plasticity is an important capacity for any agent-its connection to empowerment, however, beyond sometimes sharing an information-theowm To this end, we develop simple, universal definition of agent plasticity (Detinition 4.1) that brings the concept on equal footing with empowerment.
toolkit (Kumar et al., 2023), has remained distant.
The Mirror of Plasticity and Empowerment. Our first finding is that plasticity is well thought of as the mirror of empowerment (Proposition 4.6), pictured in Figure (a)) To reveal this mirror, exploit the inherent symmetry in the exchange of symbols between agent and environment, as in the bidirectional communication model introduced by Marko (1973). This allows us consider both the agent and environment's point of view: How much power does the environment have to shape the signals that the agent emits, and how malleable the agent be shaped by the environment? These two quantities constitute the agent's plasticity and the environment's s empowerment, and Proposition 4.6 shows they are identical. In other words, the concepts of plasticity and empowerment are well-defined using the same measure, with only the direction of influence reversed. To build toward this framing, we develop the generalized directed information (GDI, Definition 3.1) suitable formalism for the concept of plasticity under minimal assumptions. The GDI is a strict generalization of the directed information (Proposition 3.2) introduced by Massey (1990). Through a new extension of the conservation law of information (Theorem 3.5), we show that all of the information exchanged between any agent and environment can be broken into either plasticity or empowerment. This conservation unlocks our main result (Theorem 4.8), which proves that empowerment and plasticity can be tension as pictured in Figure 1(b). Consequently, the design of an agent implicitly faces dilemma with respect to plasticity and empowerment. We take this

---

result to have significant implications for the design and analysis of agents, and suggest that further examining this dilemma is an important direction for future research. We further highlight the necessary and sufficient conditions for an agent to admit non-zero plasticity (Lemma 4.2), and exploit these conditions to prove several intuitive properties hold of plasticity (Theorem 4.3). Collectively, we take these findings to provide a rich toolkit with which to study agents under minimal assumptions.
2 Preliminaries: Directed Information, Agents, and Empowerment
Notation. Throughout, we let calligraphic capital letters X.J' denote sets, and capital italics x. Y denote discrete random variables over their respective domains X' and y. We let X1.. = (X1,...; ...)....... express sequence of discrete random variables, and let [a b/, denote inclusive interval of integers such that S a Σ b I n. Lastly, we let A(X' denote the probability simplex over the set X. That is, the function p X A()) expresses probability mass function p(- x), over y/ for each x E X. All notation and core definitions are summarized in Table in Appendix A.
2.1 Directed Information
We assume the basic definitions of information theory (Shannon, 1948) are known to the reader, such as the Shannon entropy H[X) of a discrete random variable X supported on the set X' and the mutual information T( between two discrete random variables X and Y supported on X' and y respectively. We will additionally make heavy use of the directed information, developed by Massey (1990) and expanded on by Kramer (1998), and Massey and Massey (2005), defined as follows.
Definition 2.1. For any two sequences of discrete random variables, X1.0 (X1,...Xp and YI.= = (Y,...,, Yn) the directed information is
1(Xim 1Im) 1(11 Y, YIs-1).
(I)
The directed information is designed to capture the influence that the past elements X, have on future elements Y,. This notation assumes each X, precedes and is a possible cause of Y,, though if we want consider indexing where Y2 precedes X, we make use of the - in our notation,
(Yim + X1m) def Σ L(Yin-i;X X1s-1).
(2)
i=2
Directed information is particularly useful for analyzing systems with feedback-ar as is clearly present in the case of the information exchange between agent and environment in reinforcement learning. In fact, directed information was originally proposed in the context of the bidirectional communication theory developed by Marko (1973). We recall two facts about the directed information that we will later exploit and generalize. First, directed information is non-negative and upper-bounded.
Proposition 2.2 (Adapted from Theorem 2 of Massey, 1990 and Property 3.2 of Kramer, 1998). For any two sequences of discrete rundom variables X1. (X1,..., X,), Yi.. (Y,..., Yn).
II(11mm + YI.) S I(X1m) Yı.)),
(3)
where equality holds in the left inequality if and only T(X Y; YLS-)) = 0 and in the right inequality if and only if [Y, Yi.i-1; X1.)) H(Y; Y1s-1, Xin), for all i 1, ....., n.
The conditions for equality the upper bound indicate that the directed information is equal the mutual information when there is only communication flowing in one direction. We next confirm this intuition by recalling one of the central facts about directed information proven by Massey and Massey (2005) -hat directed information is conserved in particular sense.
Proposition 2.3 (Conservation Law of Directed Information; adapted from Proposition 2 of Massey and Massey, 2005). For any pair (X1:, Y1.0),
[[(X 1mi YIm) II(X + YY.)) + I(YY... + X1:)). where T(Yi. > X1... Σr= 1( X1:-1).
(4)
That is, the amount that X1:: influences YI. together with the amount that Yi.. influences X1., is exactly the total mutual information between X1.n and Yi... This highlights why the presence of bidirectional feedback forces (X1m - Y1.)) ) to strictly less than T(XI.n; fim))notice that I(YL.. 4 X1.))= = O, then arrive at the identity 1(XXmm; Yi.) = l1(Xmm + Y1.)-
3

---

2.2 Agents
Alongside the tools of information theory, the primary objects of our study are agenis and the environments they interact with. Inspired by the work general RI. (Hutter, 2004; Hutter al., 2024) and following the recent trend to move beyond the Markov property (Dong et al., 2022: Lu al., 2023; Bowling et al., 2023; Abel al., 2023a, 2024; Bowling and Elclimy, 2025), we embrace a perspective on the agent-environmen starting point is to define pair of finite sets that characterize the possible signals exchanged between agent and environment (Dong et al., 2022).
interaction that is intentionally light on assumptions. Our
Definition 2.4. The interface is pair of finite sets, (A, O) such that |A| 2 2.10
We refer to each element A as action, and each element o O these two sets form the space of possible interactions between any agent and environment that share an interface. Following the conventions of Hutter (2000); Dong et al. (2022): Abel et al. (2023a) and Bowling et al. (202)), we refer to this set of possible interactions of any length as histories. To accommodate the case where the environment emits the first symbol or the agent emits the first symbol, we distinguish between sequences that end with an action and end with an observation.
observation. Together,
Definition 2.5. The histories, histories ending in action, and histories ending in observation, that arise from interface (A,O) are defined as the sets
H det H.. U o.
H ( A)" U OY A
(6)
H. U(O x0.
We refer any element h H. history.
An agent, then, is a stochastic mapping from a history to an action, as the agent functions introduced by Russell and Subramanian (1994).
Definition 2.6. An agent relative interface (A.O) is a function, X H.o + ((A).
Lastly, environments are stochastic functions that produce observations for an agent.
Definition 2.7. An environment relative to interface (A,, O) is a function, e H... + A(O).
We let A and Eall denote the set of all such functions and e defined over interface (A, O) respectively, and let A C Aall and & C Eall denote non-empty subsets of Aall Eaul
Together, an agent À and environment that share an interface interact by exchanging observation and action signals, inducing the stochastic process A₁O A²O2- ... O, that may carry on indefinitely. We assume that actions are emitted first, but this is purely for convenience and the decision is ultimately benign. Under this framing, it is useful to call attention to a symmetry between agent and environment.
Remark 2.8. Agents and environments, as defined, are symmetric in the sense that swapping sets A, O converts every agent into an environment. and every environment into an agent. To see this, notice that an environment, precisely the definition of an agent.
H...A + A(O) becomes function from H.o to ((A), which is
2.3 Empowerment
Empowerment first introduced by Klyubin et al. (2005b) as "how much control or influence an [agent] has" (p. 2). More precisely, Klyubin et al. focus on the open-loop agents represented by probability mass function p(a") Ag over the sequence of discrete random variables, A = (A1, Az,..., A,). Then, the empowerment of an agent is the maximum mutual information between this sequence of actions, A1sa and the observation produced directly following the sequence, On, as follows.
C (A
max 1(Amm) Ow). p(a")
(8)

---

1im
Y1m
X
1:n
1:m
X a:
ced
X 11m
X1m
(a) Directed information
(b) GDI
Figure 2: A visual of the directed information (left) and generalized directed information (right). In the case of GDI, the intervals can fully partially overlap, or not overlap at all. When the intervals identical, the two quantities identical (Proposition 3.2).
Empowerment as Directed Information. Capdepuy (2011) build Klyubin's definition of empowerment by allowing the agent and the environment to both communicate with one another. Thus, under Capdepuy's definition, the agents in question can be closed-loop. which modifies the definition empowerment in two ways. First, the max is instead taken with respect to a richer class of functions such Aall, compared Agn. Second, the mutual information is replaced by the direction information, as follows,
(A,e e) = IA1 - *1mn).
XEλ
In light of its generality, we take the directed information view of empowerment to be the appropriate starting point.
3 Generalized Directed Information
A significant limitation of directed information is that it requires both sequences Xi.. and Yi. to be of the same length, and start from the beginning time. Thus, the existing measures not allow us to measure the empowerment from one sequence of arbitrary length to another sequence of arbitrary length. remedy this, we here develop a new measure that extends the directed information to arbitrary sequences Xa:b and Yo.d that captures directed information special case (Proposition 3.2). We define this new measure as follows.
Definition 3.1. Let X1:as YIs be sequences of discrete random variables and let [a b)n and |c dln be valid intervals. The generalized directed information (GDI) is defined as:
I(Xabb + Yea)
Σ
I(X I((Xamin(t.)); Y. X1:-11, Y
(10)
(=max((aee)
As with directed information, this notation assumes X; precedes and is a possible cause of Y,. If we want consider the indexing such that Y; precedes X; we again use > in our notation (Equation 2), [[xss +
Yesa)
del E Ei=max(++
1.c) L(Xa:miu(b.3-1)) Y; X1 Y1.-1).
To ensure we capture the spirit of the original directed information, we verify that several properties hold of our generalized definition. Namely. that (1) when the sequences Xa:b and Yedd are the same length and both begin the start of time, we recover directed information, and (2) the basic properties of directed information such as the conservation law are maintained. We first note that when a = c and d n, we recover the directed information, as desired.
Proposition 3.2 (GDI Strictly Generalizes DI). Let
1 and b d n. Then,
I(Xab Yea) I(X1I. - Y1.)).
(11)
Due to space constraints, we defer all proofs to Appendix C. Next, we show that the first sequence comes after the second sequence, then the GDI is zero.
Proposition 3.3 (Temporal Consistency of GDI). If a > d then I(Xasb - Yo.a) 0.

---

Thus, the quantity is only non-zero when at least some of the left-hand sequence of random variables precedes least some of the right-hand sequence of random variables, we would expect.
We next show that we can subdivide the intervals along which we measure the GDI.
Proposition 3.4 (Summation of Intervals). Let a S < and S f.<. d.
1(Xa:bb - Yed) 1(Xa.b + Ye:e) + 1(Xa:b + Ye+1.d) = T(Xack + Yea) + T(X - Yad)-
(12)
(13)
Since each term on the right is also just GDI term, we can thus decompose any pair of sequences of random variables into arbitrarily many valid sub-sequences, and preserve GDI.
We now prove the most significant property of the measure: The GDI exhibits conservation law similar to that of the directed information (Proposition 2.3). This conservation is especially important for our discussion of plasticity and empowerment, as it is critical for understanding their tension.
Theorem 3.5 (Conservation Law of GDI).
([Na:: Ye:d X Y1:-1) = I(Xasb - Yadd) + I[Yeed + Xa:b).
(14)
Note that the sum of the two GDI terms is a conditional mutual information term. There two perspectives that help to understand this conditioning. First, notice that when a = and e = 1, the terms X1:0 and Yi-n disappear, and recover the same term in the original conservation law (Proposition 2.3). Second, conditioning on past information allows us to remove possible confounders. For example, imagine X1 O. and this emission ensures that both X3 0 and Y4 O. Now, if we measure [(X2.4 + Ya.5), the total information flowing from X3 to Y44 needs to remove the potential information from th-this is the reason for the conditioning.
We prove several other properties of the GDI including an extension of the data- data-proceen processing inequality (Derpich and Østergaard, 2021), but defer these results to Appendix D due space constraints.
4 Plasticity and Empowerment as Generalized Directed Information
We now motivate the GDI as a definition for plasticity. We start by enumerating a set of desiderata that characterize an account of what suitable definition of plasticity might look like.
Our desiderata come in two forms. The first set offer a structural picture of the definition, while the second set focus on the semantics of the definition. While we hope that these desiderata illuminating, we also note that they fall short of a complete axiomatic characterization of plasticity. In hone
this sense, anticipate that subtle aspects of the definition may need slight adjustment in on the right axioms for the concept (Lakatos, 1963). We suggest that identifying such a bedrock is an important direction for further research, but is here of scope.
Structural Desiderata. A definition of agent plasticity should adhere to the following four structural criteria. First, the definition should be mathematically precise. That is, we should be able to codify all relevant commitments about our definition into a simple set of mathematical terms. Second, the definition should be simple to state, and only require minimal conceptual overhead in terms of new formalism. Third, the definition should not impose additional assumptions on our agent and environment setup beyond those already enumerated: Agent and environment share an interface consisting of two finite sets and interact in discrete time by exchanging signals from their interface. Fourth, the definition should provide flexibility to ground discussions of plasticity under a variety of settings. That is, the plasticity measure should be consistent with respect to arbitrary windows of time, or with or without a specific environment present. It is critical that we can measure the plasticity of an agent throughout its learning process, and not just the interval [1 n). This fourth structural desiderata is what motivates the development of the GDI over regular directed information, as without it we cannot speak about plasticity over arbitrary intervals of experience.
Semantic Desiderata. Next, we identify desiderata on our definition that indicate what the definition should mean. These are inspired by mixture of reflections from the broader communities that have studied plasticity, and a handful of thought experiments that stretch intuitions about plasticity. First, plasticity is always non-negative scalar value. Second, agents that are unaffected by what they

---

observe have zero plasticity. For instance, constant agent that always outputs the same action should have zero plasticity. Lastly, agents whose behavior is influenced by past observations have non-zero plasticity that is proportional to this influence. Notably, this places the emphasis of plasticity on the extent to which the agent can adapt its behavior rather than cognitive faculties of the agent, such as memories or other internal parameters. We return to this point in the discussion.
4.1 Plasticity
These desiderata motivate the GDI as our definition of plasticity. Intuitively, the definition captures how much a sequence of observations inftuences a sequence of the agent's 's actions. The GDI allows us select which observation sequence Ou:b and action sequence Acd are of interest, as follows. Definition 4.1. The plasticity of agent À relative to E from [a: bln to |c d|n is defined as
& Ba (2, max T(O + Aca)). EEE
(15)
For brevity, we will most often refer to agent's plasticity relative to set of environments as P E) and obscure dependence on the interval when otherwise clear from context. Additionally, while we introduce the definition in its general form, we most often consider the plasticity of an agent interacting with a specific environment, e, which is recovered as a special case involving a singleton set, (A,, {e)). We refer to such cases as (A)) or Φ e) for brevity. This is similar to the distinction pointed out by Capdepuy (2011) on actual versus potential information flow: "It is important to understand that empowerment, because it is defined The actual controller of the agent is not considered." (p. 44). We embrace this distinction for plasticity as well, and modulate the set & to refer to either instantiated or potential plasticity, Lastly, we note that the above definition uses the convention where, for every i = 1,....,2 n, each A; precedes each O,, but where needed, we can also make of the notation.
capacity, is potential quantity.
GDI and the Desiderata. We next confirm that the GDI satisfies our presented structural and semantic desiderata. For the first two structural desiderata, call attention to the fact that the quantity ((x, E)) is mathematically precise, and requires minimal overhead (albeit subjectively): While the GDI is new, it provides a necessary enrichment of the directed information to account for (1) sequences of arbitrary length, (2) that may not start at the beginning of time, and (3) involve systems with bidirectional communication. These properties are critical to satisfy our remaining structural desiderata, and thus we believe this slight added complexity is earned. Third, our definition imposes no further assumptions on agent or environment. We inherit the general setting of an agent interacting with an environment in discrete time, and add no further restrictions. Lastly, as noted, our fourth structural desiderata is accounted for in virtue of the GDI: We are now able to range over arbitrary sequences, and can accommodate the cases when specific environment e, set E, or the set of all environments Eall are under consideration. For the semantic desiderata, we summarize the fact that Q adheres to these desiderata using the following two results. The first is a lemma that isolates the necessary and sufficient conditions for plasticity relative to a specific environment to be non-zero.
Lemma 4.2 (Necessary and Sufficient Conditions for Positive Plasticity). An agent has positive plasticity relative to e if and only if there is a time-step where actions are influenced by observation.
More formally. for any pair (X, e) and indices [a 6) [e dmn there exists an i E [max(a, c) such that
H[A,, 11:0-1, A1.-1) > H[AA, O (emimlbb.)) O1:0-1, A1i-1)
(16)
if and only Bf and
(2, e) > O.
Notice that the only difference between the left and right side of the inequality is that the right entropy term also conditions on commiitoo)-cossuuuuent the inequality holds when learning about the observations from [a min(b, i)] lowers uncertainty about A. As a result, pair (X, e) will have non-zero plasticity from interval (a : b), to [c: d]n if and only if the observations in (a b)n provides information about the actions in [e djm.. We make heavy use of this lemma elucidate the following general properties of plasticity.
Theorem 4.3. The following properties hold of the function P. for all intervals (a B)z, [e: d],.
(i) For all (X,E), Bag (A,E) 2 0.
2d
7

---

(ii) For any a < d, there exists a pair (), where X is deterministic and Ban (A, > 0.
(iii) For all Esmall Evigg Bug (A, Esmall) S Bag (A, Enig).
(iv) (Informal) The following agents all have zero plasticity relative every environment set: Open-loop agents, agents that always output the same action distribution, agents whose actions only depend on history length, and agents whose actions only depend on past actions.
Together, this theorem and lemma indicate that the semantic desiderata are satisfied from a number of perspectives. By the necessary and sufficient conditions of Lemma 4.2, we find an intuitive test for whether an agent has plasticity in given window: Are its actions impacted by its observations? The magnitude of this impact directly determines the magnitude of the agent's 's plasticity, aligned with the fourth desiderata. The lemma also indicates that many popular learning algorithms such as UCBI (Auer et al., 2002) or Q-leaming (Watkins and Dayan, 1992) have non-zers plasticity in most environments, as expected. Theorem 4.3 illustrates basic properties of plasticity: Point (i) confirms that plasticity a non-negative scalar, while point (ii) shows that deterministic agent can be plastic. Point (iii) highlights& form of monotonicity to plasticity- as the environment set under consideration enriched, an agent's s plasticity cannot go down. Lastly, property (iv) shows that many trivial agents have zero plasticity, such as agents whose actions are determined solely by factors other than what they observe such as open-loop or constant agents.
4.2 Revisiting Empowerment through GDI
In addition to capturing plasticity, the GDI can enrich our definition of empowerment as well. As discussed in our exposition of empowerment, the definition of Capdepuy (Equation 9) moves from the mutual information to the directed information. Due to the generality of the GDI, empowerment can further benefit from the flexibility of the GDI as follows.
Definition 4.4. The empowerment of A in environment from (a b)n to (c d]n is defined as
C (A, max I(Au:b + Oc.d).
(17)
xEA
As we might hope, when = and b = d = we recover the definition from Capdepuy, when we restrict to the set of open loop agents Aar, we recover the definition from Klyubin al. Proposition 4.5.
and
E 1:m (A,
(A, e) and 3... nom
(Aa", e)
e (Aa,, c).
Thus, we can specialize to either past definition as desired. However, empowerment now benefits from the flexibility of the GDI to refer to agent's empowerment over arbitrary windows of experience that need to begin at the start of time. Furthermore, we can recover a meaningful notion of the empowerment of single agent by considering singleton set, {((A), with plasticity. We will again refer to this simply by the notation (()) or <(), e) for brevity when clear from context.
4.3
Main Result: Plasticity and Empowerment Tension
We next analyze the relationship between plasticity and empowerment. First, we highlight the fact that plasticity and empowerment are really two sides of the same coin. In one sense, this obvious, as they are both defined as the GDI between action and observation with only the direction of influence reversed. There a further sense, however, that connects them: The environment and the agent, as defined, are symmetric objects, since they are each a stochastic function from history symbol. The names of the symbols and the functions are so far arbitrary. As a result, we can also ask about the empowerment of the environment, the plasticity of the environment simply by taking the environment-centtr view point suggested by Remark 2.8.
Proposition 4.6 (Plasticity is the Mirror of Empowerment). Given any pair (x.e) e) that share an di the agent's 's empowerment is equal to the plasticity of
interface and valid intervals (a: b)n, [e: the environment, and the agent's plasticity is equal to the empowerment of the environment:
E. cud (A)
T ai (c),
T Ban (A)
= C.ny 2:3
(c).
(18)
As a corollary, we find that despite our best efforts to design an agent with high plasticity or empowerment, some environments force all agents they interact with to have zero of either quantity.
8

---

Corollary 4.7. For all (A, e), B(e) = 0 - (()) = 0 and E(e) - 0 + P(A) = 0.
In other words, even the most pliable or powerful agents cannot do anything meaningful in specific environments; the environment can always just choose to ignore the agent's ' actions, which places firm limit on how empowered an agent can be. Similarly, even if we design a highly plastic agent, the agent can only make use of that plasticity in environment with information.
Example: Two-Player Game of Poker. To build intuition, consider the case where the environment is itself another agent. That is, two agents-Alice and Bob-interact by exchanging actions. The interface is defined by two sets, Aalice: Abob. where the actions of Alice are Bob's 'os's observations, and vice-versa. In this case, over any choice of intervals, Alice's 's plasticity is identical Bob empowerment and vice-versa. Imagine further that Alice and Bob are playing poker against each other, with some initial pool of chips. If either agent were to run out of chips, they are removed from play. Notice that the set of available actions to each agent is a strictly increasing function of their available pool of chips, as the larger pile of chips yields more bets that can be placed. Therefore, the more chips Alice or Bob has, the more empowered they are. And, by contrast, if Alice has more chips, this raises how plastic Bob can be, as he has a higher potential be influenced by Alice's actions.
Main Result. We next present our main result, which highlights fundamental tension between plasticity and empowerment. That is, if we consider a specific agent À interacting with an environment an important question arises: Is there any connection between the agent's 's level of plasticity, and the agent's 's level of empowerment? We answer this question by establishing a tight upper bound on the sum of the two quantities, which establishes a tension between an agent's plasticity and its empowerment over the same interval.
Theorem 4.8 PlasticttyEmmooermnnn Tension). For all intervals [a : b)n, [c: d] and any pair (X, e), let m min((b + 1) log O0, (d c 1) [A]]. Then, a.. (A,e) e) + B aib c:d
(2. m,
(19)
is tight upper bound on empowerment and plasticity. Furthermore, under mild assumptions on the interface and interval, there exists a pair (X², ,,)) such that T 25 (X., e²) = m, and there exists a pair (X,e') such that a. (X, = m, thereby forcing the other quantity to be zero.
Therefore, an agent cannot simultancously maximize its plasticity and its empowerment in given pair of intervals, [a 6)n, [c dln- In fact, the achievable levels of empowerment for any pair of intervals are strictly determined by the agent's plasticity in those intervals (and vice versa), as pictured in Figure ((b). The latter part of the result indicates that in extreme cases, a high plasticity can ensure that the agent has zero empowerment, and vice versa. Notably, the tension only comes about when we consider 3 as. and a. over the same pair of intervals, [a b)n, [c: d),. Therefore, this tension may dissipate as the intervals change, or if one interval fully comes after the other; in this way, agents may oscillate between phases of high plasticity and high empowerment, but cannot simultaneously maximize both. A deeper examination of this trade-off and its implications for the design of agents is a significant opportunity for future research.
5 Discussion
In this paper, have presented careful study of plasticity and empowerment, and established connection between them. Our first technical contribution is the development of the generalized directed information (GDI; Definition 3.1), which we take be of independent interest across the many fields that make use of information theory. We identify several fundamental properties of the GDI, including new conservation law (Theorem 3.5). We then motivate the GDI as an appropriate definition for the concept of plasticity, and provide general definition of agent plasticity. We then identify the necessary and sufficient conditions for an agent to have non-zero plasticity (Lemma 4.2). Our analysis further reveals that plasticity admits a number of intuitive properties (Theorem 4.3), and that the GDI generalizes existing notions of empowerment (Proposition 4.5). We then discover a general new trade-off between plasticity and empowerment (Theorem 4.8), suggesting that we need to thoughtfully engage with both concepts in the study and design of agents. We take these results to serve mcaningful bedrock from which we can study agency under minimal assumptions.

---

Plasticity via Action, Policy, or Agent State? As discussed, the original conception of plasticity from the neuroscience literature is terms of synaptic plasticity- that is, the degree of flexibility inherent to the neural circuitry of biological organism. By contrast, our account of plasticity focuses purely on behavior: We measure the influence of observation on action. This is more in line with how the biology community treats plasticity, as discussed in Chapter 30 of the book by Wheeler (1910). Our plasticity definition can be thought of as a special case of broader family of definitions that consider the degree that observations influence various aspects of an agent, such as the agent's 's policy, action, or internal state. We can choose from many such options, and our formalism (and the corresponding theoretical results) will largely look the same. Consider the case of policies. Here, we suppose that every agent maintains& a policy that belongs to a finite class of representable policies, AB Aal (the "agent basis" from Abel et al., 2023a). Then, letting L L,, denote sequence of discrete random variables over AB, we can express a policy-focused notion of plasticity in terms of the quantity T(Oabb - Le This measures the degree which observations influence which policy an agent chooses. With some added detail obscured, this is a strict generalization of Definition 4. 1, but we choose center around the action variant due to its simplicity. Alternatively, we can shift the influence of interest to agent state to better capture how well an agent can adapt its internal circuity in light of experience. In this case, we model each agent as maintaining an agent state (Dong et al., 2022; Abel et al., 2023b; Kumar et al., 2023). Letting Si...., Sn denote the sequence of random variables over this agent state space S, we can recover our conception of plasticity by building around I[O0.bb + Sea).. We believe this view is especially promising to examine agency under resource constraints (Simon, 1955; Russell and Subramanian, 1994), the extended mind (Clark and Chalmers, 1998; Harutyunyan, 2020), and the paaasiity-lasttiitty dilemma (Carpenter and Grossberg, 1988), but leave exploration of these directions for future work.
Plasticity, Empowerment, and Goals. Our framing is absent of the discussion of goals, despite their central role to agency. Indeed, as Ball states, "An agent does more than just alter its environ- ment...they alter with intent" (p. Ball 2023). We have here concentrated entirely on two basic influences; plasticity as the influence of environment on agent, and empowerment as the influence of agent on environment. It is important to enrich this story with some account of goals. We foresee several pathways for our theory to accommodate goals. For instance, we can treat reward as a special element of each observation. Then, we might isolate a special kind of influence that amounts to the agent being influenced by rewards (and, likewise, reward-relevant empowerment as actions that can explicitly modulate future rewards). However, this is just one proposed direction among many possibilities. We suggest one of the most important directions for future work will closely examine how plasticity, empowerment, and their tension relate to coul-directedness more broadly.
Other Future Work. There are many other directions for future work. First, as a consequence of the conditions in Lemma 4.2 and the tension highlighted by Theorem 4.8, there are new opportunities to inform the design of agents. In particular, we speculate that is prudent to design agents to balance plasticity and empowerment tactfully, since over-optimizing one can impact the other. We foresee an opportunity to recover new fundamental results connecting the limits of learning, the design of safe agents, and this balance-if an agent has insufficient plasticity, it cannot adapt, and if it has insufficient empowerment, it cannot steer. Additionally, it may be valuable to better understand how people handle the tension revealed by Theorem 4.8, similar to understanding how people address the explore-exploin dilemma. Second, there is a clear opportunity to make use of our conception of plasticity formally characterize the stability-plastici envision a clear path to doing so, but defer the full exposition of this narrative to future work. Third, we foresee direct connection between plasticity, empowerment, and many standard concepts within game theory and multi-agent systems such as the emergence of social dilemmas, cooperation, and the dynamics of multi-agent systems more generally (Jaques et al., 2019). Fourth, a natural next step is to design efficient estimators for the GDI, and by consequence, plasticity and empowerment. Following the work by Jiao al. (2013), we suspect that similar algorithmic template can underlie an estimator for GDI. We include a brief experiment to ground our theory in Appendix E, though suggest that deeper empirical work, and an efficient GDI estimator, are valuable directions for further research. Lastly, we speculate that plasticity, empowerment, and their relationship offer simple, powerful conception of agency: An input-output system is best thought of as an agent if and only if it has non-zero plasticity and non-zero empowerment. We again defer a full conceptual analysis of this sort for future work, but refer the reader to the works by Wooldridge and Jennings (1995), Barandiaran et al. (2009), Ball (2023), and Kasirzadeh and Gabriel (2025) for similar discussions.
dilemma under minimal assumptions. We
10

---

Acknowledgments
The authors are grateful to Pablo Samuel Castro, Claudia Clopath, lason Gabriel, Joel Z. Leibo, Ben Van Roy, and the anonymous reviewers for their thoughtful comments on a draft of the paper. We would further like to thank Barel Skuratovsky and the Agency team for inspirational conversations.
References
Z. Abbas, R. Zhao, J. Modayil, A. White, and M. C. Machado. Loss of plasticity in continual deep reinforcement learning. Proceedings of the Conference on Lifelong Learning Agents, 2023.
D. Abel, A. Barreto, B. V. Roy, D. Precup, H. van Hasselt, and S. Singh. A definition of continual reinforcement learning. In Advances in Neural Information Processing Systems, 2023a.
D. Abel, A. Barreto, H. Hasselt, D. Precup, B. Van Roy, and S. Singh. On the convergence of bounded agents. arXiv preprint arXiv:2307.1 1044, 2023b.
D. Abel, M. K. Ho, and A. Harutyunyan. Three dogmas of reinforcement learning. Reinforcement Learning Journal, 1, 2024.
Auer, N. Cesa-Bianchi and Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235-256, 2002.
P. Ball. Organisms agents of evolution. John Templeton Foundation: West Conshohocken, PA, USA, 2023.
X. E. Barandiaran, E. Di Paolo, and M. Rohde. Defining agency: Individuality, normativity, asymme- try, and spatio-temporality in action. Adaptive Behavior, 1765):367-386, 2009.
K. Baumli, D. Warde-Farley, Hansen, and Mnih. Relative variational intrinsic control. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
M. Bowling and E. Elelimy. Rethinking the foundations for continual reinforcement learning. arXiv preprint ar arXiv:2504.081e. 2025.
M. Bowling, J. D. Martin, D. Abel, and W. Dabney. Settling the reward hypothesis. In Proceedings of the International Conference Machine Learning, 2023.
Brändle, L. Stocks, B. Tenenbaum, S. J. Gershman, and E. Schulz. Empowerment contributes to exploration behaviour in a creative video game. Nature Human Behaviour, 7(9):1481-149) 2023.
V. Campos, A. Trott, C. Xiong, R. Socher, X. Giró-i Gicto and Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In Proceedings of the International Conference on Machine Learning. 2020.
Capdepuy. Informational principles of perception-action loops and collective behaviours. Ph1) thesis, University of Hertfordshire, 2011.
G. A. Carpenter and S. Grossberg. ART 2: Self- organizzaiiin of stable category recognition codes for analog input patterns. Applied optics, 26(23):4919-49) 4930, 1987a.
G. A. Carpenter and S. Grossberg. A massively parallel architecture for self-organizing neural pattern recognition machine. Computer vision, graphics, and image processing, 37(1):44-15, 1987b.
G. A. Carpenter and S. Grossberg. The ART of adaptive pattern recognition by self-organizing neural network. Computer, 21(3):77-88, 1988.
Q. Chen, C. Shui, L. Han, and M. Marchand. On the stability-Plasticity dilemma in continual meta-learning: Theory and algorithm. Advances Neural Information Processing Systems, 2023.
A. Citri and R. C. Malenka. Synaptic plasticity: multiple forms, functions, and mechanisms. Neurpsscchophammccolgg. 33(1):18-44 2008.

---

A. Clark and D. Chalmers. The extended mind. Analysis, 58(1):7-19, 1998.
M. S. Derpich and J. Østergaard. Directed data-processing inequalities for systems with feedback. Entropy, 23(5):533, 2021.
Dohare, R. S. Sutton, and A. R. Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2/08.06325, 2021.
S. Dohare, J. F. Hemnandez-Garcia, Q Lan, P. Rahman, A. R. Mahmood, and R. S. Sutton. Loss of plasticity deep continual learning. Nature, 632(8026):688-774, 2024.
Dong, B. Van Roy, and Z. Zhou. Simple agent, complex environment: Efficient reinforcement learning with agent states. Journal of Machine Learning Research, 23(255):1-5, 2022.
S. Fusi, P. J. Drew, and L. F. Abbott. Cascade models of synaptically stored memories. Neuron, 45 (4):599-611, 2005.
K. Gregor, J. Rezende, and Wierstra. arXiv:1611.0750,, 2016.
Variational intrinsic control. arXiv preprint
A. Harutyunyan. What is agent? http://anna. http:///ma...arruyyyyaaaa.e httpr/twwwyan.ee////poooottnntnp pdf,
2020/09/What_is. an an_aaeett..__a adf, 2020.
M. Hutter. A theory of universal artificial intelligence based on algorithmic complexity, arXiv preprint cs/0004001, 2000.
M. Hutter. Universal artificial intelligence: Sequential decisions based algorithmic probability. Springer Science & Business Media, 2004.
M. Hutter, D. Quarel, and Catt. An introduction to universal artificial intelligence. CRC Press, 2024.
N. Jaques, A. Lazaridou, E. Hughes, C. Gulcehre, Ortega, D. Strouse, J. Z. Leibo, and N. De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In Proceedings of the International Conference Machine Learning, 2019.
J. Jiao, H. H. Permuter, L. Zhao, Y.-H. Kim, and T. Weissman. Universal estimation of directed information. IEEE Transactions on Information Theory, 59(10)6620-6242,, 2013.
A. Kasirzadeh and Gabriel. Characterizing Al agents for alignment and governance. arXiv preprint 27X4 2025.
A. S. Klyubin, D. Polani, and L. Nehaniv. All else being equal be empowered. In European Conference on Artificial Life, pages 744-753. Springer, 2005a.
A. S. Klyubin, D. Polani, and C. L. Nehaniv. Empowerment: A universal agent-centric measure of control. In JEEE Congress on Evolutionary Computation, volume 1, pages 128-133 IEEE, 2005b.
G. Kramer. Directed information for channels with with feedback, volume 11. Citeseer, 1998.
S. Kumar, H. Marklund, Rao, Y. Zhu, H. J. Jeon, Y. Liu, and B. Van Roy. Continual learning as computationally constrained reinforcement learning. arXiv preprint arXhn:2307.04345, 2023.
I. Lakatos. Proofs and refutations. Nelson London, 1963.
A. Levy, S. Rammohan, A. Allievi, S. Niekum, and G. Konidaris. Hierarchical empowerment: Towards tractable empowerment-based skill learning. arXiv preprint arXiv:2307.02726 2023.
X. Lu, B. V. Roy, V. Dwaracherla, M. Ibrahimi, I. Osband, and Z. Wen. Reinforcement learning, bit by bit. Foundations and Trends in Machine Learning, 16(6):733-865. 2023. ISSN 1935-8237.
C. Lyle, Z. Zheng, E. Nikishin, B. A. Pires, R. Pascanu, and W. Dabney. Understanding plasticity in neural networks. arXiv preprint arXiv:2303.01488 2023.
H. Marko. The bidirectional communication theory-a generalization of information theory. IFEE
Transactions communications, 21(12)) 1973.
12

---

J. Massey. Causality. feedback and directed information. In Proceedings of the International Symposium on Information Theory and its Applications, volume 2, 1990.
L. Massey and C. Massey. Conservation of mutual and directed information. Proceedings. International Symposium Information Theory, 2005. ISIT 2005., pages 157-158. IEEE, 2005.
S. Mohamed and D. Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in Neural Information Processing Systems, 2015.
Pearl. Causality. Cambridge university press, 2009.
K. Raghavan and P. Balaprakash. Formalizing the cewpaliizaaioofooettiiiii trade-off in continual learning. Advances Neural Information Processing Systems, 2021.
T. J. Ringstrom. Reward Not Necessary: Foundations for Compositional Sar-Sttaionayy Non- Markovian Hierarchical Planning and Intrinsically Motivated Autonomous Agents. PhD thesis, University of Minnesota, 2023.
S. J. Russell and D. Subramanian. Provably bounded-optimal agents. Journal of Artificial Intelligence Research, 2:575-600 1994.
C. Salge and D. Polani. Empowerment as replacement for the three laws of robotics. Frontiers in Robotics and AI, 4:260425, 2017.
C. E. Shannon. A mathematical theory of communication. The Bell system technical journal,, 27(3): 379-423, 1948.
H. A. Simon. A behavioral model of rational choice. The quarterly journal of economics, 69(1): 99-118, 1955.
A. M. Turner, L. Smith, R. Shah, A. Critch, and P. Tadepalli. Optimal policies tend seek power. Advances in Neural Information Processing Systems, 2021.
C.J. J. Watkins and P. Dayan. Q-learning. Machine learning, 8(3-4)277--292, 1992.
M. J. West-Eberhard. Developmental plasticity and evolution. Oxford University Press, 2003.
W. M. Wheeler. Ants: their structure, development and behavior. Number 9. Columbia University Press, 1910.
M. Wooldridge and N. R. Jennings. Intelligent agents: Theory and practice. The knowledge engineering review, 1052):115-152, 1995.
13

---

A Notation
We first provide Table summarizing all relevant notation.
Notation
Meaning
Definition
X.Y
Set
A,Y,A,
Discrete random variable
X
Sequence of n. random variables An interval of discrete time
(Xi...., Xn) aa s.t. E asn
la: b],
A
Set of actions
Set of observations Histories ending in action Histories ending in observation
H..
(0 (A 0y XA
H.o
(Ax U XO H
H
Set of all histories
e
Environment
e: H... + A(O)
Eall
Set of all environments
A E S &all
Set of Environments
A
Agent of agents
X: H.o + A(A) A(A)H.o
Aal
A
Set of agents
A C Aall
[(Xim + Yim)
Directed information
Σ 1(Xisi Y:\Y(:i-1)
=mn
I(X + Yim)
Directed information, delay
Σ T(X 1 1 2=m
I(Xaxd + Yea))
GDI
Σ T(X T(Xamin(0.d)) X1a-11 Vis-1) max(a.))ud
Ba. E),
Plasticity
max.ge 1(On + Ara)
ad e),
Plasticity, single environment
B. (A, {e)) cd
((A
Plasticity abbreviation
B (1,e)
C.. (A,e) E. (1,e)
Empowerment
maxxea I(Aa.b + Oca)
Empowerment, single agent
E. (()),e)
(())
Empowerment abbreviation
C.. (X, E) ed
Table 1: A summary of notation.
14

---

B Expanded Discussion
Next, we provide additional discussion on salient topics.
B.1 Optionality, Information, and Causality
Both empowerment and plasticity center around how the agent or environment could have acted or unfolded differently. In the case of empowerment, we ask: Had the agent acted differently, how much could have changed in the environment? For plasticity: If the environment had unfolded differently, how much would that have impacted what the agent did? In this way. both concepts implicitly deal with considerations regarding free will, counterfactuals, causality, and determinism. For example. empowerment is defined by maximizing over the set A, rellecting the different ways an agent could respond stimuli. Constraining f-ror instance, by restricting actions, compute, or memory-will in turn reduce the agent's 's ability to influence future observations and, consequently, its empowerment. In the most general case we can take the max over all agents, Aull. reflecting every possibly way experience could give rise to action over given fixed interface. However, there are often further restrictions in place, such as the morphology of a robot or a limited available computational budget- such restrictions can be reflected by considering different subsets A C Aall thereby encoding the different possible ways the agent could act. Capdepuy (2011) discusses this distinction in Section 4.1.3, noting that potential information flow involves taking the max over available policies, while actual information flow is relative to a single policy. Similarly, plasticity (Definition 4.1) involves maximization over set of environments E. This choice raises the question: What constitutes environment optionality? How could an environment unfold differently? Interventions, as described by Pearl (2009), offer formal framework for grounding the notion of environment optionality from a causal perspective. Here, an external experimenter modifies the environment and observes the responses of its sub-systems (such the agent). This aligns with the continual learning literature, where plasticity is treated as the agent's capacity to adapt to interventions on the data-generating process. For example, in work by Dohare et al. (2021) the input distribution is repeatedly and randomly intervened on, and later work by Dohare et al. (2024) the labels are randomly permuted, whereas in work by Abbas et al. (2023) the agent is switched between different Atari games. These distributional shifts can be formalized as interventions on a baseline environment e, where each intervention creates new, distributionally shifted environment e' = t(e). In each study there is an assumed set of interventions 'T (such as label permutations), which generates the environment set = (e) t, E T). In essence, an agent's plasticity its capacity to adapt to environmental changes (interventions), defined relative to set of environments & generated by these interventions. And, analogous to constraining the policy set for empowerment, plasticity is non- non-increasing the set of possible environmental changes is constrained (as in point ii of Theorem 4.3). We suggest that drawing out a full causal picture of both plasticity and empowerment a compelling direction for further work.
As a consequence of the definition that focuses on observation and action, we find that every deterministic environment forces zero plasticity. That is, if e is deterministic in the sense that it can be written map, e: H...- - O, then for every agent and intervals, Q Beg (A, e) 0. This may appear puzzling at first, but on closer inspection, there are several intuitions to elucidate this fact. First, from the agent's 's point of view. deterministic environment presents no optionality. This is similar to a causal conception of plasticity relative to the empty set of interventions. Since the environment could not have been otherwise, there no way that the agent can react to the different ways the environment could have unfolded. However, variant of plasticity that builds around the internal components of the agent (such as the agent's beliefs or memories) will overcome this in simple way. Consider agent with limited memory. That is, at most the agent can remember five observations. Then, if the agent is at time-step seven, it cannot possibly remember all prior observations due to its memory constraints. Therefore, from the agent's 's memory-constrained point of view, the environment may appear stochastic, While these threads play an important conceptual role in plasticity and empowerment, we defer full exploration of their significance to future work.
B.2
The
Tension
Next, we present a short example expanding on the discussion of the tension between plasticity and empowerment highlighted by Theorem 4.8. Consider the example environment depicted in Figure 3.
15

---

En
i
Figure 3: An example environment in which an agent (the mouse) is presented with gradual change in the balance between their realized plasticity and empowerment. The mouse inhabits corridor consisting of n + rooms, each equipped with a light that can (stochastically) controlled by the lever in that room.
The environment contains n + 1 rooms, each containing a switch and light. The environment is conditioned on latent Bernoulli random variable, X, corresponding to the probability to turn the light on or off each time step. The mouse only observes whether the light is on off in the room they currently occupy. The action space of the mouse enables it to move a neighboring room, or pull the lever. In each room L; a pull of the lever has probability of i/n to change the state of the light for the next time step, with the remaining probability n i/n determined by X-hence, room to. the mouse has no control over the light, and in room l, the mouse fully controls whether the light on or off. Otherwise, the random variable X dictates the state of the light. Suppose X fair coin, that is, X ~ Bernoulli( 0.5). So, without intervention, the lights in all rooms switch state with probability 0.5. In this environment, the far left room maximizes plasticity, whereas the far right room maximizes empowerment, with all rooms in the middle smoothly interpolating between these two extremes. On the left, the mouse gets to receive the most new information from the environment, while the right, the mouse fully controls the state of the light. Theorem 4.8 suggests that this trade-off always present to some extent.
To emphasize the tension even further, we can imagine that the latent random variable X is non-stationary. That is, consider the sequence of smoothly evolving Bernoulli random variables, X1, X2..... Here, again, if the agent always inhabits the far right room, it fully closes off the possi- bility of learning about the evolution of {_1, Loc However, the agent is in the far left room, then the light is fully determined by the evolution of X,, and thus the agent has maximized its potential to learn about the sequence from the environment. Each room in between these two interpolates between these two extremes, roughly corresponding walk along the Pareto frontier of the trade-off expressed by the theorem.
C Proofs of Presented Results
We next provide proofs each result from the paper, split into those results presented Section 3 on GDI (Appendix C.I) and those results presented in Section 4 on plasticity and empowerment (Appendix C.2). Then, in Appendix D, we present additional results on the GDI, including a new data- processinn inequality for the GDI.
C.1 Proofs of Results from Section on GDI
Proposition 3.2 (GDI Striely Generalizes DI). Let a = = 1 and b = d = n. Then,
I(Xasb + Yea) = 11(X1.m + Y(m))
(20)
Proof of Proposition 3.2.
The result is straightforward consequence of the definition of GDI that can be illustrated by substitution. Let and b d be indices such that 1 = c and b d = Then, rewriting
16

---

Equation 10, find:
[(X::b Yead
Σ
I(X abm))))))))) Y - N1--1),
(21)
i=max (a,c)
I(Xisi I V1--1),
(22)
Σ [K Y (1s-1).
-1X
Proposition (Temporal Consistency of GDI). Ifa > then l((as + Yaa) = 0, and if then l(X < Yea) 0.
Proof of Proposition 3.3.
The property follows directly from the summation bounds of Equation 10, and its variant,
I(Xa:b 4 Yea)
Σ
I[X Ya X X1sa- 1n YIs-I)-
(23)
(mmax(a+1.))
For the former, note that if a > d, then
I(Xa:b + Yoa) =
Σ
I(X (.Xamii(h)))) Yi X1 Y1--1) 0,
(24)
(a,)
since max(a, c) > d when a >> d.
For the latter, note that if a 2 d, then
d
I(Xasb + Yesd)=
Σ
T(X
a)amin().-1)) Y/ X V1s-1) = 0,
(25)
1,,
by similar reasoning that max(a + 1,c) > d.
Proposition 3.4 (Summation of Intervals). Let < k < and S d.
I(Xobb - Yoa) = 1(Xabb Yoe) + 1(X + Y(+1.))
(26)
= I(Nack - Yed) I(X++I>b - Yed).
(27)
and similarly,
I(Xab 4 Yod) 1(Xab Yos)) + I(Xas 4 Ys+ha) = T(Xaxc + Yea) + T(X+t1s 4 Yed),
(28)
(29)
Proof of Proposition 3.4.
The first equality of Proposition 3.4 consists of breaking the sum of Equation 10 into two sums with intervals [max(u, e] and [max(a, E 1).d..
For the second equality observe that if then we can change b = d as i > d in the sum Equation 10. Then, we apply Equation 10, break up the first sum at k and recombine terms, as follows:
17

---

[[Xask + Yea)++ + +(X++11 + Yea)
(30)
Σ
I(Xaminnt, Yi X1 11 +
immax(a.c)
I(X +mmin)) Yi X1:ks Y1:1-1),
(31)
(=mxx(++1o))
Then, we break the right sum into two intervals.
T(Xas; Ye X1a- Y1--1)
Σ
T(Xa:ki Ys X1:- V1a-1) +
mm,
I(X 1mmin((b..) Y Nisk Y1i-1).
(32)
imaa((k+1.))
I(Xasi Yi
X1--11 V1s-1) +
(33)
(=max(a,e)
(T(Xark5 Vi X MIs-1) + ( Yi Xisk Mis-1))
(k+
Applying the mutual information chain rule to the terms in the second sum and noticing if S k then = min(b, i) so we can change the bounds on the first sum to get
I(Xack Yea) + +I(x++15 - Yea))
Σ
a( Yi X1:0-1, V1a-1) +
(=,ax(a,e))
Σ
I(X a:min(b.s) Y5 X 11a-1, Mia-1).
(34)
<m 1.))
Now we combine the sums to get our result,
I(Xack + Yad) + I(X++1x Yea)
Σ
T(X I(Xamin(ba)); Y. X1:a-1, X1:a-1, Yi:-1) I(Xa: - Yea).
G
(35)
ma.ex(a.e)
To prove the conservation law of GDI, first introduce short lemma that helps decompose the GDI into difference of entropies.
Next, we present further decomposition of any GDI term that will be a key step in our proof of the conservation law GDI.
Lemma C.1. Let [a 6),, [e: : d)n be valid intervals, and let max(a, e), min(b, d). Then,
T( (Xash >> Yest)
(36)
I(X Yaot X1 YI 1) + I(Xa.e + Yue) + I(X Yetred X1 Yi.),
and
I[Yed + Xa.b)
(37)
=I(Yck-1i Xxet YIC-1, X14-1) I(Yae & Xxe) + 1(Yeet Xetro Y1
Proof of Lemma C.I.
18

---

Applying Proposition 3.4 multiple times,
T(Xa:: + Ye.d)
(38)
I[Xak-11 + Yc)
+
I[xee + Yak-1)
I(Xg-1:b + Yok-1)
T(Xa:k11 + YEEF)
+
I(Ka:t + Yaee)
II ([Xe-1hh - Yu.E)
T(Xa:k-1 + Yet1:))
+
T(Xie YetTd)
[[Xe-1:b Ye+1a)).
Notice that either ls a b, and similarly c E so two terms are guaranteed have empty ranges,
[(x:b + Yedd
(39)
I(Xaubt + 1cik-1)
+ I(Xxe Yok-1)
[[(e-11: - York-1)
I[Xax-11 + Yase)
+
1(Xxe + Yxce)
[[ - Kxte)
I[Xak-11 + Yetna)
+
I(Xiee + Yet1a)
[[ Ye+1sd)
We can now apply Proposition 3.3 to remove additional terms.
I(Xa:: + .Ydd)
(40)
= T(Xa x Yeck-1)
+ T(Xae- 1
TI ( 1o c:k II [X k:t)
I(Xaik11 + Yuse)
+ I(Xiit + Yxue)
[[Xas-11 + Ye+ha)
+ I(Kaee + Yet
[[ Yet1)d).
Recombining the last two terms with Proposition 3.4 gives us three remaining terms,
I(Xasb + Yea)) I(Xak-1 + Yuee) + I(XR.e + Yuat) + I(Xa:t + Y(41.))
(41)
Applying Equation 10 to the first and last term, I(Xa: + Yead)
(42)
I(Nack-1;Y; X1a-1, Y14-1) + I(Xx. + Yoe) +
Σ
I(xa:t Yi X1 Y1a-1),
imf+1
and applying the mutual information chain rule these two summations gives.
I(Xa:: Ye:d)
(43)
11i 1i Ykve| 1 Y1:k-1) +((Kk + Yk.)) + 1(Xa:ti Ye+1dd X1
This completes the first case. The second case follows by similar reasoning.
I(Yed + Xa:b)
I(Vak= + Xack-1) xxaxk1))
+
I(Y St (
+ I(Y + Xack-1)
+
T(Yck-1 + Xk.e)
+ I(YRe 4 Xkte)
+ I[Y -3 Xb.t)
+
T(Yok-1 + Xet1:b)
+ T(Yke + Xe+1:t)
+ T(Y + Xe+1.))
T[Yck-1 4 XA.e) + T(Yiat + Xk.e) + I(Vet ④ (eetia)
I[Y Xase YI Xix=11) I(Yna + Xiee) + I(Yeti Xetrs YLo- X14)
This completes both cases, and we conclude.
Theorem 3.5 (Conservation Law of GDI).
I[Nabii Yoa X1.0-1, YI.c-) = I(Xa:b + Yord) + I(Y 4 Xat).
(44)
Proof of Theorem 3.5.
The proof proceeds by decomposing the two GDI terms I(X..b + Yea) and I(Yeau - Xaw) by application Lemma C. then inducting on l as in the original proof by Massey and Massey (2005).
19

---

First, we decompose both I(Xu:b Yea) and I(Yedd 7 Xa:b) using Lemma C.I.
T(Xu:b - Ye.d)
(45)
I(xax Yx:e| 11 11:k-1) + I(Xx:t + Yu.)) + I(x::t Yetid XI
and
I(Yod Xasb)
(46)
= =i(Gas-1;; (1xk-1) + I(Ya + Xk:e) + I(Ye X2+15 Y, X1:e)-
Let us now consider T(Xg: + Yk:E) + I(V::e + Xx:e), which is almost the term constrained by the law of conservation of directed information (when & = 1). We use induction on l k as
employed by Massey and Massey (2005) to show the sum is T(( Yke X XI:--1 V1 First, if l k then I(Xx.1 + Yk.z) Ii( Ykee X1:k-1, 1 and [(Y + Xkee) 0, thus establishing the base case. the general case,
I(Xk: + Yk:e) + I(YRee + Xxxe)= I(Xx:e-1 Y(tt11) [( Ye X1 Y1:-11) + 1(Ykee-1 + (kie-1) + [(X:: Ya-e-1 X1i--11 11
By rearranging and applying induction,
T( Y X1 V I(X;; YE---1 X1.t-1, Yi.k-1 1) + I(Xxaet Y. X1 Y1se-1),
(47)
Then, applying the mutual information chain rule the first two terms,
I(Kiee; Yk--11 XIx- YIR-1) I(xx Ye XIs-I1, Yie--))
(48)
and once more on the remaining terms,
I(Xxee Youe X1sk-1, Y1
(49)
Thus by induction,
I(Xx: + Yk.e) + I(Yr.e + Xx.t) I(Xxe: Yaee X1:6-1, Y1.k11)-
(50)
We now combine Equation 45, Equation 46, and Equation 50,
I(Xatb + Yea) + I(Yedd + Xa:t) = I(Xa:A-11 1; Yast
X1:a-1, V1ck1) +
(51)
T(X Yes-1 X V1:--) + T(x;: k:t; Yk
X1:-11, Vick-1) +
(52)
I(Xm::: Ye+1ed/ X1m--,, Y..)) + I[X 1: Yos
X1.t Y Y1so-)).
(53)
Notice that at most one of the first two terms non-zero (if a > e the top term is zero; if < the bottom term is zero), and similarly for the final two terms (if b >> d the top term is zero; if b < d the bottom term zero). We now apply the mutual information chain rule to the non-zero first term with the middle term. In either case we get,
I(Xa. + Yea) I(Yedd + Xa:b) =[(aa:f Yost X X1aa-,, Yia-1) 1) + I(Xa:f Yeuni X1sa-1, Yu.e) I(Xe+18; Yost X1at Yis--)..
(54)
(55)
(56)
We can now apply the mutual chain rule with the non-zero last term. In either case, we get,
I(Xa.b Yea) 1(Yedd + Xa.t)= 1(Xa.bi Yoad X1 Y1.0-).
(57)
which confirms the identity, and conclude the proof.
C.2 Proofs of Results from Section on Plasticity and Empowerment
Lemma 4.2 (Necessary and Sufficient Conditions for Positive Plasticity). Agents have non-zero plasticity relative to if and only if there time-step where their actions are influenced by observation.
20

---

More formally, for any pair (A,e) e) and indices [a b/n, [e: d) there exists an E [max(a, c) such that
H[(A; 01:a-1 11 > H((A; O emmin(b.i) O1:a-1 ALæ-1)
(58)
and only
8 oid
(A,
Proof of Lemma 4.2.
We first establish necessity: I and
(A. O, then there exists an [mm(max(a))): such that
H(Ap/ 1:00 A1 > (Av | Oasminthaa)) O1a A1s
(59)
Expanding the definition of 8 (A, e),
(), 6) 1(Oubb + Aca)
(60)
and
Σ
a( Ar 01a-1, A11-1).
(61)
i=max (a.c)
Recall again that the conditional mutual information is non-negative. Hence, the sum
Σ
I(Oa:min(b.1): A. O1a-1, Api-i),
(62)
immax(a.e)
only greater than zero if at least one of its terms is non-zero. Since Q an (A, > O. we conclude that there must be 1' [max(a, c) d such that II(O a=min(t.²") Ap 01:--1, A1 > O. We next apply the identity that conditional mutual information can be expressed as difference conditional entropy terms,
((Ap O1.a-1 AL - H(A, O ammin((b))) O1:a-1, O1a- A1. > 0,
(63)
I(O (aa
and therefore,
H(Ap O1a--1, A H(Av (amim(bbs)) O1a-
(64)
This completes the argument for necessity.
We next prove sufficiency: If there exists an i' [max((, c) d) such that
H(Az 01..-1.A.:-1 > H(Ap O Damin(har)) 01:a-1 A1s-1).
(65)
then Ba2 (A, e) 0.
The same reasoning from the necessity argument may be applied in reverse. First, we can rewrite above inequality and replace the difference with the conditional mutual information.
I(O (10amii(bbr')) Av A1
>
(66)
which in turn implies that
Σ
I(O [[Oasmin(b.):::) 01:0-1,411-11 O,
(67)
(=,a
and by the definition of plasticity, conclude B.y (A, > 0.
Theorem 4.3. The following properties hold of the function for all intervals [a : b) [e: dm.
(i) For all (),E), a. (X,E) 2 O.
(ii) For all d, there exists a pair (A, E) where deterministic and P (A, E) > 0.
(ii) For all Esmall S Ebigy Bag (), Esmall) Baig (A, Evig).
21

---

(iv) (Informal) The following agents all have zero plasticity relative to every environment set: Closed-low loop agents, agents that always output the same action distribution, agents whose actions only depend on history length, and agents whose actions only depend on past actions. Proof of Theorem 4.3.
We prove each property independently.
(i) For all (A, E) and any intervals Ja b)m [e: d/r. $ (A, 0.
This follows direct consequence of the first inequality of Proposition D.2, together with the definition of plasticity.
(ïi) For all < d, there exists a pair (A, E) where X is deterministic and & (A, E) > 0.
suffices to construct a single deterministic agent with non-zero plasticity for a fixed but arbitrary pair of intervals where a < Consider the singleton environment containing e that outputs a uniform distribution on O at each time-step.
To be precise, we define a deterministic agent as any non-stochastic mapping from histories to actions, À H..o A, the of all such deterministic agents the set of all such mappings,
Adet A A
(68)
Then, let Adet be a deterministic function that outputs aj if the last observation was 0₁, and outputs d2 otherwise. That is,
[a1)] 01 [5 otherwise,
X(ho)
(69)
for all h e H. E. O, where 8(a;} is the Dirac delta distribution a₃-
Then, for any choice of intervals where d, we that for any [max((, c) d].
[[(AA | 010-1; [H[(A, (a:miu(b.i) O 11a-11 A(i-1)
(70)
>0
=0
Again by application Lemma 4.2, this implies that L m: (A, e) > 0, and we conclude.
(iii) For all Esmall C Ebiy and any intervals [a: 6Jn, [e: djn, & Bas (A, Esman) S <$ (A, Ebig)-
The result follows directly from the use of the max operator in the definition of plasticity. That is, since
max f(x) S max f(x), K.
(71)
xeXiig
for any sets X-mall Ubig and function f. then
max I(Ou:b Aca) < max I(Oa:b + Acd): eCEamall
(72)
eEEbiR
a.b (A.Eamalt)
Da (A.Erix)
as desired.
(iv) (Informal) The following agents all have zero plasticity relative to every environment set: Agents that always output the same action distribution. agents whose actions only depend history length, and agents whose actions only depend on past actions.
The argument follows directly for all three agent classes by application of Lemma 4.2: For each agent type, conditioning on observations tells us nothing about the actions the agent emits.
To be precise, we define a constant agent as an element of the
A Aconstant
{ E A A(h) X(1', V.NEH.))
(73)
22

---

That is, constant agent outputs the same probability distribution actions in every history. For any such constant agent, we find that
((A; O1a--1 A1 = H(A; aumin(b,i)- O1 1, A1s--1),;
(74)
since each A, (H) is conditionally independent of On:min(b..) given A1.--1, 014-11 by definition of de as a constant agent. By Lemma 4.2, this concludes the argument.
The same reasoning applies to agents whose actions only depend on the length of history, closed-loop agents, and agents whose actions only depend on past actions.
This concludes proof of all four properties.
Proposition 4.5. 1 (A, e) (A and a e) = (Aa
Proof of Proposition 4.5.
We show that Capdepuy's and Klyubin et al.'s definitions of empowerment are special cases in separate arguments.
(Capdepuy)
First, we can see that the GDI-based definition captures Capdepuy's definition when and b d n, since
E e) = max T(A1 - +1mm) = (A)
(75)
dEA
(Klyubin et al.)
Second, show that when 1, n. and b d = and the set of agents in question the open loop agents, Agn, then
E auy (Aan, e)
lin (Aas, e) EK (Aan,
(76)
We start by expanding the GDI-based definition,
E sm Nin
(Nan, max T(Aa:b - Oca): AEA,"
(77)
maax T(A1:1 + Onn),
(78)
AEA
max I(A + +On). p(a"))Ad.
(79)
Now, since the agents are open loop, we know that I(O,, - A1m) = 0. By Proposition 2.3,
T(O,, + A1m) T(A - -,) = T(A1sm; On),
(80)
and hence,
T(A1 + On) T(Ai=n On)-
(81)
Therefore,
C.a (Aa max I(A1: - #On pla")eAA"
(82)
max I(A p(a")EA,"
(83)
CR (Aam, e).v
(84)
This conclude both equalities, and conclude the proof.
Proposition 4.6 (Plasticity is the Mirror of Empowerment). Given any pair (), e) that share an interface, and valid intervals (a 6) [e din the agent's empowerment is equal the plasticity of
23

---

the environment, and the agent's plasticity is equal to the empowerment of the environment:
Eu (A)
Z c.d (c),
B (A) = C (e).
(85)
Proof of Proposition 4.6.
The proof proceeds as direct consequence of the definitions of empowerment and plasticity, and the symmetry highlighted by Remark 2.8.
First, we expand each definition for the agent,
F ()) Q (A, E) = I(0 - Amdd,
(86)
G (A) = -. (A, e) I(Aa:b + Oc.)).
(87)
Now, to arrive at analogous definitions for the environment, we exploit the symmetry highlighted by Remark 2.8 to recover notions of plasticity and empowerment when the environment is treated as an agent,
F a:t (e)
a: (e, A) T(Aa:b - Oc:a);
(88)
aab (e)
ast (e, A) I(Oa:b Acad).
(89)
Now, notice that ((A) -e) by Equation 87 and Equation 88, and P(A) = E(e) by Equation 86 and Equation 89.
C
Theorem 4.8. For all intervals [a : 6) Ic d)n and any pair (2, let m = min a + 1) log [O], (d - c log [l]- Then,
a.y (A,e) + B (A, m,
(90)
a tight upper bound on empoverment and plasticity. Furthermore, under mild assumptions on the interface and interval, there exists pair (x),e) such that Beg mt (X,,e) m, and there exists pair (X, e') such that a.g (X, m, thereby forcing the other quantity to be zero.
Proof of Theorem 4.8.
Let (x,e) be fixed but arbitrary agent-environment pair, [a b)m, [c: d)n,, be valid intervals, and let m min (b a + 1) log [O), (d c + 1) log [A]]
Then, by Theorem 3.5, we arrive at the identity.
T nn and
(A,e) + C.. e) =1 I[O Acd 01.0-1,A1c--1)
(91)
and
S2 min (H(O HT Amad))
(92)
S3 min (b a + 1) log [O,, (d - c + 1) log [A]),
(93)
m,
(94)
where = follows from the conservation law of GDI (Theorem 3.5), S2 follows by the standard upper bounds on conditional mutual information, and S3 follows from the standard upper bound on joint entropy.
To prove this upper bound is tight, we show there exists two pairs, (Ao, (9)e)) and (')ee) such that (95)
m G an (A e =0 m a. (X,e') =0 0,
We present the argument for plasticity, and recall that by symmetry (Remark 2.8), the same will hold empowerment.
To do so, we require that |A| < 10) and (b (b-a0 a) 2 c). While the desired inequality will likely hold under softer conditions, the argument is cleanest under these conditions. For simplicity, we also assume a S c.
By Proposition D.3, we express B., (2o, e0) as follows
T a. (X9, = I(Oa:: Awa)
(96)
Hi( Amaass((a)) O1aa-1) - H(Ad || Ows)).
(97)
24

---

Since assumed S c, rewrite
P (X (X,ee = O1:a-1) - H(Aca || Owb)..
(98)
Let i |AA, and let O₁ refer to the set containing the first & observations, O2 for 02 ..... 0........ 0%
Let e be the environment that always emits or along the initial (possibly empty) interval [1 Im.. and emits uniform random distribution over On from time i onward. Now, let A be the agent that always plays d1 until the start of the observation interval [a: b). then on seeing any observation oj, it mirrors the same action back, aj. More formally,
8(01} e(h.) (o00-(Unnte ())
i [ 1},
[ó(a1} i <<vvj> k
X(h,o,)
(99)
i i2a,
[fa,}} otherwise,
for all h; E H. 0j E Additionally, we let 8(1;} express the Dirac delta distribution on z4.
O, where h,j H.oo expresses the concatenation of h, and oj.
In this situation, observe that
HI (Aed 01:a-1) (d-c) llog [A],
(100)
since the agent plays the uniform random distribution A in this interval by mirroring the envi- ronment's uniform random distribution over Ok. and we assumed (b a) 2 (d cc). Furthermore,
H(Aad || Oacs) = 0
(101)
since learning about the observation at each time-step the interval [a : bj fully determines the action in the interval [c + a)]. Therefore, substituting into Equation 98
2 (A9EE) = [Hee 01.a-1)
H[Addd || Ow.b).
(102)
2m
0
Hence, we conclude that
P (X 2 m,
(103)
and by the non-negativity empowerment and plasticity,
C. (A e S m This completes the argument for the of empowerment, and call attention to the fact that by symmetry (Remark 2.8), an identical argument holds for plasticity.
B n. (Xº, e 0.
(104)
D Other Results
Theorem D.1 (Data-processinng inequality for GDI). Consider any tuple (Xims Yim, Zim) where Zi is conditionally independent of Xı:i given Y, for each i 1...n. Then,
1(Xa.b + Yea) 2 I(Xo:b + Zera),
(105)
for any valid intervals [a b)n [c: d]n-
Proof of Theorem D.I.
The argument follows an identical structure to the standard argument for the data- processinn inequality for regular mutual information, together with two inductive steps. The first inductive step applies to the interval [a b)n and the second applies to the interval (c d].. We assume across all arguments that each Z; conditionally independent of X1:1 given Y, for each i 1...
Base Case: b,
First, we consider the base case when a b and c d, and assume < d (if d, we know IIX + Yea) = 0 by Proposition 3.3). Hence, we write I(X... + Yge) = I[[X, + Y.).
25

---

Expanding the definition for both relations X to Y and X to Z, we fine,
I(Xo 1o)
Σ
Ye X1=--111--1) I(Xai )
(106)
(a,a(
T(Xa + Zc)
Σ
T(X (xxmii(b..) Zg X1:a-1213-1) [(Xa;; 7 max(a.o))
(107)
(=,ax(a,c)
Then, apply the standard proof technique for the data-pros eata-processing inequality: We consider the mutual information [[ mmxx(a,)) mmax(a,c)), and use the identity that
I[ ((Xa; Y mmxx(a.c) c) e = I(Xai mm)x(a,c)) + I(Xai Ymax((,,)) ?max(a,c)):
(108)
= I(xa: max ) + I(Xa: Zmax(a,o) Ymax(a,))
(109)
Where the final term is equal to zero by the assumption that each Z; is conditionally independent of X1, is given Y; for each i ... n. Therefore, rearranging, we find that
I(Xai c)) I(Xai mmax(a,e) Zmax(a,e)) 11(Xa; emax(a,)));
(110)
and by the nen-nigatyvity of mutual information, we conclude
L(Xoi mux((,,)) 2 11( ((xaa; Zumx((ac))
(111)
=Yoona-Yooo) =I(Xarn Zooe)
as desired. This concludes the base case.
We next turn to the two inductive cases.
Inductive Case I: I(X..b + Yo:e) > I(Xa.b Zoze).
We assume as our inductive hypothesis that
I((aabb11 + Yoro) 2 1((Xab-11 + Zo:e),
(112)
and show that this implies
I(Xa:b + Yee) 2 1(Xa:b + Zee)
(113)
We apply Proposition 3.4 the terms I(Xa:b - Yee)) and I(Xa:b Zere) to decompose them into
T(Xa:b ab Yoe) T( T((aab-111 > Yae) + T(Xbsb - Y.c))
(114)
I[Xa:: Zee) = I(Xaxb-1 + Zme) + I(XNoh - Zeaa).
(115)
But then, by the inductive hypothesis and the base case, we know that
I(Xaih-1 + Yas) 2 I(Xabb-1 - Zeco);
I(Xbs Yae) I(Xus - Zmc).
(116)
Thus,
1(Xarb + Yos)
I(Xasb + Zoze)
(117)
1 -Yeie)
=I(X..b11 -t (IX
as desired.
Inductive Case 2: I(Xasb > Yesd) 2 I(Nacb + Zea)).
The argument for this inductive case is identical to the former, only we apply the inductive step to the interval [c d] v
This concludes the proof.
26

---

Next, with most terms related to mutual information, the GDI is non-negative and upper bounded. Proposition D.2 (GDI Upper Bound),
0 s I(Xabb Yead) S I(Xa:: Yed X1:a-1, Y1-c-1).
(118)
Proof of Proposition D.2.
The two inequalities follow consequence of the definition of GDI, and its conservation law (Theorem 3.5).
First, note that a(xabb - Yead) is non-negative by definition:
d
T(Xasb + Yea)
Σ i=max(a.c)
T(xammiu(b.5) Y: X1m-1, Y1s-1).
(119)
20
since it is the sum of conditional mutual information terms that each non-negative by definition. Second, by Theorem 3.5, we know that
[(aabb + Yea) + +I(Yed + Xa.b) = I(Xa:b: Yed Xi:a-1, M1.0-1,,
(120)
and since all three terms non-negative, follows that
I(Xabb Yed) $ I(Xabi Yead X1:a-1, Y1c-1).
Lastly, we can decompose the GDI into conditional entropy terms. Proposition D.3 (GDI Kramer Decomposition).
I(Xa:b + Yea) H((Ymax((aa (1:a-)) - H(Yed || (aib),
(121)
where
H(YYed Xa:b)
Σ
H(Y; Y15-1, X Xarmin(ha)).
(122)
(=max(a.e)
the generalization of Kramer's causal entropy arbitrary indices.
Proof of Proposition D.3.
The result follows by expanding the definition of GDI and applying the conditional mutual information identity in terms of difference of conditional entropy terms, as follows.
I(Xa. Yea)=
I(X ammin(b,a) Y, X 11 Y1a-1),
(123)
H(Y: X 11a-1, HIIY; X Xamin(t)i) X1aa Yia-1),
(=max(a.e)
H(Y, X1a-1. Y1--1)
H(, Y1 X Ximmnn(ha))
(124)
(=
H(Yi X1 Via-1)
HI(Y; V X X1min(ba))
=E(YeallXaal)
Σ
H(Y' X1.0-1. V1-1-1)
-H(sead Xa:t);
(125)
= ([(mmatuuojja/X11a.a
HI HY (max(( X1 Hlaad || Xat).
27

---

As direct corollary of Proposition D.3 together with Proposition 3.2, we recover Kramer's 's identity as a special case, as follows. Corollary D.4. When -a= and b d nb,
H H(Y 11:0-1) - Hi(YYdd cxd Xa:b) = H(YI:)) H( || (1
(126)
and,
HYcddd Xaxb) HY1 || X1m)).
(127)
E
Experiments
Lastly, we conduct two simple experiments to ground and corroborate our theory. In each case, we make use a basic Monte Carlo estimator for the GDI, and study the plasticity and empowerment of tabular Q-learning (Watkins and Dayan, 1992) interacting with a two-armed Bernoulli bandit. The experiments were run on a single CPU.
In the first experiment, we use an e-greedy policy for exploration and study the impact of varying e between [0, 1] on plasticity. We estimate the plasticity of Q-learning for each of these values of in the time intervals [1 3] and [2: 5], so I(O2:3 - ). Results are shown in Figure 4(a). We see that, the agent relying entirely on the greedy policy to drive its actions, which is determined by the past observations. Thus, with low we see a higher level of plasticity. Conversely, as we increase e, the degree to which the greedy policy drives action choice decreases. the maximal value of 1, we find that plasticity is zero since the agent's actions are no longer influenced by observation.
when
In the second experiment, we study the impact of optimism and pessimism on plasticity and empower- ment, again in the two-armed Bernoulli bandit. To vary the degree of optimism or pessimism present in the agent, we vary the initial Q value used for each action from -1 to 1 and examine the impact on plasticity and empowerment along the same intervals [1 3] and [2 5]. Here, we no longer use e-greedy exploration, and rely solely on the initial Q value to drive exploration. Results are shown in Figure 4(b). First, we note that the upper-bound (depicted in grey) from Theorem 4.8 is corroborated empirically. Second, we see that empowerment tends to be higher than plasticity in this setting the agent influences the environment more than the environment influences the agent. Third, we find that empowerment increases with optimism. This likely comes about due to the degree of exploration present in the agent: When the initial Q values are pessimistic (strictly below zero), the agent will be more likely to stick with the first arm that delivers positive reward. In this way, the pessimistic agent is less incentivized to try out different actions, which leads to a lower overall empowerment.
0.9
648
25
088
000
0.0
680
1480 070.
(a) Plasticity vs. Exploration Parameter (c)
(b) Plasticity and Empowerment vs. Initial Q
Figure 4: Results from two simple experiments that estimate plasticity, empowerment, and their sum with tabular Q-learning (Watkins and Dayan, 1992) in a two-armed bandit. On the left, we show the mean estimate of an e-greedy variant of Q-learming as a function of e, varied from zero to one, with bootstrapped confidence intervals. On the right, we show the estimate of plasticity (blue), empowerment (orange), and their sum (green), along with the upper bound (grey-dashed line) as function of the initial Q value used by Q-learning, ranging from negative one to one. When the initial Q = 1, the agent is pessimistic, and when the initial Q 1, the agent is optimistic.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction state our main results, and provide links the relevant theorem or proposition to accompany each statement of a result.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limiting assumptions along the way, and highlight room for enriching the work our discussion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specificatinn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
29

---

Answer: [Yes]
Justification: All proofs of each theoretical result are presented in full detail in the Appendix. We have carefully scrutinized the proofs and believe with cigh-confidence that the arguments are all correct (though leave open the possibility of a small flaw).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Ys]]
We only conduct two small-scale experiments in Appendix E that do not impact our main conclusions. However, we believe we have included sufficient detail to reproduce these experiments, given their simple nature.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
30

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: The experiment in Appendix E is very simple, and we believe be implemented as needed.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ct cc/
guuides/CodeSuumisseee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//1 /nips..c..... htppscccccbbccguciiccaae gttiiiiggg///////eeeeeeese Seeiede///oeeubmaaaecee ssmmissssonnp cesssnnnniccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We believe the needed details reproduce Figure 4 are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we include bootstrapped confidence intervals in Figure 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
31

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We used a single CPU to run the experiments, as they are small in seale.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:r////./...//eeeeeeeee cc/ppullic/tthhccccildllllll
Answer: [Yes]
Justification: Our work adheres to the NeurIPS code of Ethics: human subjects nor data sets, nor does involve any privacy or safety related concerns. While the concepts and theories may eventually materialize into impactful outputs, we believe at this stage there are no specific such pathways that require special attention.
The research does not involve
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This paper addresses fundamental theoretical and conceptual questions while all scientific work can eventually realize in some form of societal impact, we do not believe there are any special considerations worth discussing.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
32

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not make use of any data, models, or other artifacts that have any risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The only figures used were hand-designed by the authors.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
33

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There are no new assets released in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
34

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core of the work did not involve any important, original, or non- non-standard use of LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https fattps://nett http://p//.//oooeeeee/////////// for what should or should not be described.
35


Reviews:
[{"review_id": "plqfHIZZF7", "reviewer": "Reviewer_2PEP", "summary": "This paper develops an information-theoretic mathematical definition for plasticity. Towards this, one key contribution is the formulation of Generalized Directed Information (GDI), which generalizes the notion of directed information between sequence of random variables $I(X_{1:n} → Y_{1:n})$ to handle arbitrary sub-sequences $I(X_{a:b} → Y_{c:d})$\n\nGiven this formalism, the paper defines plasticity of an agent w.r.t. an environment as the (generalized) directed information from its observations to actions. Similarly, a definition of the empowerment is formulated as the (generalized) directed information from its actions to observations, and the paper shows that this definition subsumes previous ones that only considered full sequences.\n\nThe main results then show that: a) there is a symmetry in the definition of plasticity and empowerment i.e. plasticity = the environment’s empowerment w.r.t. the agent, and b) the sum of an agent’s plasticity and empowerment is bounded i.e. given a specific environment, there is a tradeoff between how plastic an agent can be vs how empowered it is.", "questions": "- Some more discussion on the relation of the current definition to the typical semantic of 'neural plasticty' would help.\n\n- I'd also appreciate a discussion/rebuttal of the weakness 2 i.e. it seems that an agent **can** be simultaneously empowered and plastic where its current observations can influence future actions while its current actions can also influence future observations, and there seems to be no tradeoff between these.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The formulation of GDI is an interesting contribution. It helps generalize the notion of directed information to consider arbitrary subsets and could be applicable beyond the context of plasticity/empowerment. Similarly, the generalized definition of empowerment is also a useful extension of previous ones as it now lets us consider the empowerment in terms of influence of a specific subsequence actions on a subsequence of future observations.\n- The main results which highlight that it isn’t possibly to simultaneously optimize empowerment and plasticity is a thought-provoking observation (although some concerns on this below)\n- The paper is generally well-written with appropriate levels of background and detail in the main text and appendix. Given the topic and the contribution, this was likely not an easy thing to accomplish!\n\nWeaknesses:\n\n- A primary concern about this work is that it seems to be redefining an existing and widely used term “plasticity” to be rather different from the (intuitive but not mathematically precise) typical definition of “an agent’s capacity to remain adaptive”. In this work, the term “plasticity” is instead better described as capturing “the influence of an environment on the agent” and it’s not clear why we should be using the same word ‘plasticity’ for this measure. As an illustration, in the mouse example in the appendix, the fully stochastic room certainly reduces the empowerment of the agent but it’s not clear there is any useful information to learn for helping future adaptation in the stochastic one. Concretely, let’s imagine a neural network trying to learn to predict the outcome of the agent’s actions. One environment is deterministic, allowing the network to model laws of physics and the other environment is random, and makes the neural network learn to predict a uniform distribution over states. As the typical definition of ‘plasticity’ is how adaptive this network can be to future changes, it’s not clear why the network being learned in the fully stochastic setting will be better suited to adapt to new distributions than the one in the deterministic setting. So although this stochastic environment does maximize ‘plasticity’ as defined in this paper, it does not intuitively do so for the ‘classical’ notion of plasticity.\n- While the bound in Egn 19 is interesting, I’m not sure if it’s a very profound result. For example, if there is no overlap in a:b and c:d, one of the terms is trivially zero (as directed influence of future on past will be zero). On the other hand, if the two subsequences overlap fully (e.g. a=b=1, c=d=n), it is essentially the result in Eqn 4. It would have been a much more surprising result had the bound in Eqn 19 been in the direction a:b → c:d for both terms i.e. for non-overlapping intervals, one term were not trivially zero. In the current form, an agent **can** be simultaneously empowered and plastic in terms of its actions affecting future observations, and observations affecting future actions. Specifically, considering non-overlapping intervals a:b (’current’) and c:d (’future’), there seems to be no tradeoff between the current actions’ influence on future observations and current observations’ influence on future actions.", "was_revised": true, "final_justification": "After the rebuttal and followup discussion, I am convinced the work is novel and general (specially the GDI and the subsequent definitions of plasticity) as well as technically correct and clear. Personally, I am still not sure about the potential impact and the usefulness of the central results in Theorem 4.8, but I believe these things are better left for the community to determine in time. Overall, I'd be happy to update the score to lean towards acceptance and would encourage the authors to update the text to reflect the  limitations discussed.", "rebuttal": "__Overview.__ We would like to thank the reviewer for their time in reading and reviewing our paper. We address the reviewer’s two primary concerns below in sections Comment 1 and Comment 2, and will update the paper in light of their comments.\n\n&nbsp;\n# Comment 1: Plasticity Semantics and Example\n\nWe break our response to Comment 1 into two parts, focusing on (1a) plasticity semantics and (1b) the rooms example.\n\n&nbsp;\n ## (1a) Semantics of Plasticity\n\n> “A primary concern about this work is that it seems to be redefining an existing and widely used term “plasticity” to be rather different from the (intuitive but not mathematically precise) typical definition of “an agent’s capacity to remain adaptive”... Some more discussion on the relation of the current definition to the typical semantic of 'neural plasticty' would help.”\n\nLet us consider the following wording for the intuition of plasticity and our mathematical definition:\n\n> (Plasticity intuition) “an agent’s capacity to remain adaptive [to its observations]”.\n>\n> (Our definition) “an agent’s capacity to be influenced by its observations”.\n\nThe difference separating these two is very subtle, and is rooted in how one chooses to define adaptivity mathematically. Our formalism gives a mathematical definition of “adaptivity” in terms of the degree to which observations can influence action (as measured by generalized directed information)---this is in line with classical definitions of plasticity and learning as “the ability to change action due to experience” (paraphrasing from Chapter 3 of West-Eberhard, 2003; Lachman, 1997; among others). Consequently, we believe our definition of plasticity does capture the intuitive use of plasticity, and hope that this wording and commentary makes this more apparent. \n&nbsp;\n ## (1b) Rooms Example\n\nRegarding the example, you mention:\n\n> “...it’s not clear there is any useful information to learn for helping future adaptation in the stochastic one”...\n\nNotice that information is so far neutral in our theory—we focus on the capacity to influence and be influenced, and all information is created equal. For this reason, it is perhaps easy to see why the stochastic case provides more opportunities to respond in your neural network example: there is just a richer set of possible experiences the agent can adapt to.\n\nTo study whether it is _useful_ to be plastic or be empowered, we need to introduce a goal for the agent to pursue or a target to predict (as in your example): Then, information is useful if it enables an agent to effectively pursue its goal or predict the target. This is an important area for future work, and we do have some preliminary results in this direction: for any realizable value of plasticity or empowerment, there will exist environment-goal pairs for which all optimal agents have at least the given level of plasticity (or empowerment). So, it is sometimes necessary to be plastic or empowered to be optimal. There are thus paths that connect usefulness to plasticity and empowerment, and we agree that a deep analysis of this relationship is a valuable direction for future work.\n\n> “As the typical definition of ‘plasticity’ is how adaptive this network can be to future changes, it’s not clear why the network being learned in the fully stochastic setting will be better suited to adapt to new distributions than the one in the deterministic setting.”\n\nWe again emphasize that it is best to untether plasticity from usefulness, first (we can then incorporate what counts as “useful” information later, as suggested above). In the absence of specific information being useful, we see that we lose the sense of one setting being “better suited” to adaptation—there is only influence, not a notion of “better”. And, the stochastic setting comes with a richer space of experiences for the agent to respond to, so the potential for plasticity is higher.\n\n\n\nReferences:\n\n* West-Eberhard, M. J. (2003). Developmental plasticity and evolution. Chapter 3. (pp. 34). Oxford University Press.\n* Lachman, S. J. (1997). Learning is a process: Toward an improved definition of learning. The Journal of Psychology.\n\n&nbsp;\n# Comment 2: No Tradeoff for Some Interval Configurations\n\nOur response is again in two parts: (2a) the trade off for other interval configurations and (2b) the value of Theorem 4.8.\n\n&nbsp;\n## (2a) No Tradeoff for Some Interval Configurations\n\n> I'd also appreciate a discussion/rebuttal of the weakness 2 i.e. it seems that an agent can be simultaneously empowered and plastic where its current observations can influence future actions while its current actions can also influence future observations, and there seems to be no tradeoff between these.\n\nThe reviewer is correct in thinking that Theorem 4.8 will not extend to all possible configurations of indices. This is a valid point, and we are happy to add some text to the paper clarifying this. We roughly formalise this as an existence claim that $E + P$ can be greater than $m$ from Theorem 4.8 if we swap the index order, as follows.\n\n__Conjecture 1.__ For all intervals $[a:b]_n, [c:d]_n$ where $b < c$ and $b-a = d-c$ (for simplicity), there exists a pair $(agent, env)$ such that $E_{a:b,c:d}(agent, env) + P_{a:b,c:d}(agent, env) > m$, where $m$ is set as in Theorem 4.8.\n\n_Proof Sketch._ We prove the result by constructing an example that satisfies the inequality for fixed but arbitrary intervals $[a:b]_n$ and $[c:d]_n$ where $b < c$ and $b-a = d-c$. For simplicity let both $\\mathcal{A} = \\\\{0,1\\\\}, \\mathcal{O} = \\\\{0,1\\\\}$.\n\nFor timesteps $a:b$, consider an agent-environment pair where actions and observations are each samples from a uniform distribution over $\\\\{0,1\\\\}$.\n\nLet agent and environment be such that over the $i$th timestep in $c:d$, the observations mirror the $i$th action in $a:b$, and the $i$th action in $c:d$ mirrors the $i$th observation in $a:b$. So, the actions from $c:d$ receive maximal influence from prior observations, and similarly for the observations.\n\nIn other words, the empowerment is $\\mathbb{H}(A_{c:d})$, which is $k \\cdot \\log_2 |\\mathcal{A}| = k\\cdot \\log_2(2) = k$. The same holds of plasticity, and therefore we find $E_{a:b,c:d}(agent, env) + P_{a:b,c:d}(agent, env) = 2k$. This is strictly larger than $m$ from Theorem 4.8, and we conclude.\n\n_End Proof_\n\nWe will add text to the discussion of Theorem 4.8 to emphasize that it does not apply to the cases involving swapped intervals like the above, and will include a polished version of this conjecture and proof somewhere in the paper or appendix to add clarity.\n\n&nbsp;\n## (1b) Theorem 4.8 Remains Useful\n\nHowever, we also want to emphasize that Conjecture 1 and Theorem 4.8 holding simultaneously does not detract from Theorem 4.8, for a few reasons:\n1. First, for any non-overlapping intervals $[a:b]_n, [c:d]_n$ that satisfy Conjecture 1, there exists a larger pair of intervals $[w:x]_n, [y:z]_n$ that (i) contain the former, and (ii) do overlap, so the tension comes into play and Theorem 4.8 applies. In this way, the tension can effectively always rear its head, though its salience naturally depends on specific aspects of the agent, environment, and the intervals.\n2. Second, consider sequential intervals of length-t episodes: Theorem 4.8 tells us there must again be a bound on the agent’s plasticity and empowerment within each episode. As a consequence, _accumulating over episodes_, plasticity and empowerment are also bounded. So, an agent must balance between the two quantities over the course of its lifetime in a particular sense. A similar conclusion can be reached following typical arguments about entropy rates: the total rate of information exchanged is upper bounded for finite alphabets A and O. As such, the limiting rate of plasticity and empowerment is also likely to be upper bounded in a way that suggests a lifetime-level tension. Exploring this zoomed-out view from either of the two perspectives mentioned is another rich direction for future work that requires careful analysis, but again we believe the tension highlighted by Theorem 4.8 has potential for far reach.\n\nLastly, let us emphasize the generality of Theorem 4.8: it holds for effectively all agents and all environments—we make no assumptions on agent nor environment apart from the finiteness of the sets $\\mathcal{A}$ and $\\mathcal{O}$—the environment can be partially observable, non-stationary, and unbounded. These are very mild assumptions for a result of this kind. The result provides the first connection between the concepts of plasticity and empowerment in a general setting that yields a fundamentally new way to look at agents, and we believe this makes it a valuable contribution in its own right. Lastly, its proof requires more than the standard conservation law of DI (Prop 2.3), but also requires the introduction of the GDI and the expanded conservation law (Theorem 3.5), whose definition and analysis are non-trivial in their own right.\n\n&nbsp;\n# Summary\n\nWe will add expanded discussion to the paper that incorporates and clarifies the discussion surrounding Comments 1 and 2. We agree this will bring new clarity to the paper, and appreciate the reviewer’s time and comments."}, {"review_id": "qCivnnqFEz", "reviewer": "Reviewer_x8QU", "summary": "This paper proposes a new agent-centric measure (which they call as plasticity) and it quantifies how much an agent is influenced by its observations, complementing the concept of empowerment, which measures the agent’s influence on future observations. Plasticity is formalized using generalized directed information... it is defined by extending Massey’s directed information framework. The authors show two key results: \n(a) Plasticity and empowerment are mirror quantities between the agent and environment, and \n(b) There exists a trade-off.. both cannot be maximized simultaneously. This reveals a fundamental tension between being shaped by the world and shaping it. \nThe work offers a fresh theoretical perspective for understanding and designing intelligent agents.", "questions": "Please answer below questions:\n\n(a) This paper is entirely theoretical... it does not perform any empirical evaluation to demonstrate the practical implications of their proposed GDI framework. Any reason for not doing this? Is there any lack of data or other issues/\n\n(b) The use of advanced information-theoretic constructs like GDI may hinder accessibility and adoption among practitioners. Are these constructs lead to any engineering constraints? \n\n(c) The agent-environment interaction model is highly abstract and it may not capture the dynamic and uncertain nature of real-world environments where assumptions like full observability or stationarity may not hold. Any comments on this?\n\n(d) While the empowerment–plasticity tradeoff is theoretically intriguing, the paper lacks actionable insights on how to balance or optimize these properties when designing real agents. Do you have any quick suggestions to address this?", "limitations": "(a) This paper is entirely theoretical... it does not perform any empirical evaluation to demonstrate the practical implications of their proposed GDI framework. Any reason for not doing this? Is there any lack of data or other issues/\n\n(b) The use of advanced information-theoretic constructs like GDI may hinder accessibility and adoption among practitioners. Are these constructs lead to any engineering constraints? \n\n(c) The agent-environment interaction model is highly abstract and it may not capture the dynamic and uncertain nature of real-world environments where assumptions like full observability or stationarity may not hold. Any comments on this?\n\n(d) While the empowerment–plasticity tradeoff is theoretically intriguing, the paper lacks actionable insights on how to balance or optimize these properties when designing real agents. Do you have any quick suggestions to address this?", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Below are my  comments and concerns on this paper:\n\n(a) This paper is entirely theoretical... it does not perform any empirical evaluation to demonstrate the practical implications of their proposed GDI framework. Any reason for not doing this? Is there any lack of data or other issues/\n\n(b) The use of advanced information-theoretic constructs like GDI may hinder accessibility and adoption among practitioners. Are these constructs lead to any engineering constraints? \n\n(c) The agent-environment interaction model is highly abstract and it may not capture the dynamic and uncertain nature of real-world environments where assumptions like full observability or stationarity may not hold. Any comments on this?\n\n(d) While the empowerment–plasticity tradeoff is theoretically intriguing, the paper lacks actionable insights on how to balance or optimize these properties when designing real agents. Do you have any quick suggestions to address this?", "was_revised": false, "final_justification": "", "rebuttal": "__Overview.__ We would like to thank the reviewer for their time in reviewing our paper, and for their questions. We respond to each of their four main questions (a-d) in detail below, and will plan to make adjustments to the paper to clarify these points where appropriate.\n\n&nbsp;\n### Question (a): Theory\n\n> (a) This paper is entirely theoretical... it does not perform any empirical evaluation to demonstrate the practical implications of their proposed GDI framework. Any reason for not doing this? Is there any lack of data or other issues/\n\nThanks for the question. We do maintain that the theory we have introduced stands on its own as a valuable contribution. However, since submission, we have implemented estimators for plasticity and empowerment, and conducted small-scale experiments that validate and expand on some of the findings of the work. Our first experiment involves a simple class of agents and environments and corroborates the upper bound of Theorem 4.8. Our second experiment involves modulating various characteristics of an agent (we use UCB and Q-learning), such as the degree of optimism or exploration parameters of the agent. Concretely, we adjust things like the initial value function, learning rate, and the exploration parameters of the agents and examine the impact of these changes on plasticity and empowerment. Depending on space, we are happy to include a section about these experiments in the final version of the paper. However, we suggest that a full-scale experimental study that significantly expands the work is beyond our present scope, but agree that it is a useful direction for future research.\n\n&nbsp;\n### Question (b): GDI\n\n> (b) The use of advanced information-theoretic constructs like GDI may hinder accessibility and adoption among practitioners. Are these constructs lead to any engineering constraints?\n\nIf we understand the question correctly, we do not see any obvious ways in which the use of things like the GDI will lead to any engineering constraints. As indicated above, we have implemented simple estimators for these quantities—since all of the measurements are behavioral, we believe that a larger scale experiment is in scope for future work. Information theoretic quantities are commonly used in practice, and there is no reason the GDI (as a generalization of directed information), should raise any new challenges.\n\n&nbsp;\n### Question (c): Environment Limitations\n\n> (c) The agent-environment interaction model is highly abstract and it may not capture the dynamic and uncertain nature of real-world environments where assumptions like full observability or stationarity may not hold. Any comments on this?\n\nTo clarify, the environment model we use is general enough to capture both partial observability and non-stationarity, and includes things like POMDPs and non-stationarity MDPs as special cases. See the work by Dong et al (2022) or Majeed and Hutter (2018) for further information on this generality. Similarly, the agent model we make use of is also highly general, and can capture arbitrary agents.\n\nIn this way, we take our theory to be a general one—we make only the mild assumptions that the action and observation spaces are finite: the main results hold for effectively the most general classes of agent and environment types, and believe this is a significant strength of the work. We are happy to emphasize this generality more in the paper.\n\nReferences:\n* Dong, S., Van Roy, B., & Zhou, Z. (2022). Simple agent, complex environment: Efficient reinforcement learning with agent states. Journal of Machine Learning Research, 23(255), 1-54.\n* Majeed, S. J., & Hutter, M. (2018). On Q-learning Convergence for Non-Markov Decision Processes. IJCAI.\n\n&nbsp;\n### Question (d): Actionable Insights\n\n> (d) While the empowerment–plasticity tradeoff is theoretically intriguing, the paper lacks actionable insights on how to balance or optimize these properties when designing real agents. Do you have any quick suggestions to address this?\n\nThere are several directions we are eager to explore, though we first emphasize that this paper is laying down the foundations that we strongly believe can later influence and materialize in practical guidance of the kind suggested. Our answer here is similar in spirit to Reviewer vGU6’s question:\n\n__1. Algorithm Design.__ First, we believe it is important to better understand which algorithmic knobs can directly control an agent’s plasticity and empowerment. For example, changing an agent’s learning rate parameter intuitively controls the agent’s plasticity—is there an equivalent, typical parameter that modulates an agent’s empowerment? If not, can we design one? Do these knobs also have knock-on effects to other parts of the agent (does changing the learning rate impact empowerment?) Once such knobs are identified, we can leverage them to modulate an agent’s degree of plasticity and empowerment, as indicated by the reviewer. In cases where we want to ensure agents remain adaptive and pliable (as in cases of coordination or continual learning), we can control these knobs to ensure plasticity is maintained. Similarly, if we want to limit the empowerment of an agent (as in safety-critical cases), we can again make use of such knobs.\n\n__2. Goals, Plasticity, Empowerment.__ Second, it is important to better understand how goals fit into our picture. If we were to introduce goals in the form of reward as is typical in reinforcement learning, then it is useful to clarify when agents should lean into maximizing plasticity, empowerment, or achieving a balanced trade-off between the two in order to satisfy their goals. We have so far proven some initial results that clarify the relationship between optimal learning, plasticity, and empowerment, and believe these kinds of results can shed light on when and how we should approach the balance of plasticity and empowerment. We believe that a systematic empirical study of standard learning algorithms and environments would further be a natural place to expand on this understanding, particularly in a continual learning setting.\n\n\n\n\n__3. Balancing Plasticity and Empowerment.__ Third, we note that many have previously argued that agents should maximize their empowerment. For example, in one of the early pioneering works on empowerment, Klyubin et al. (2005) explicitly suggest “All else being equal be empowered”, as their title indicates. Empowerment is then often treated as an optimization objective in its own right (see, for example, work by Leibfried et al., 2019, among many others). Our theory suggests otherwise: if an agent is always maximizing empowerment, it will lower its ability to receive information from and be influenced by the environment (or other agents), too. For example, if an artificial agent is playing a cooperative game with a person, then to cooperate effectively, both the agent and person should be slightly plastic enough to allow themselves to be influenced by their teammate—if they were only empowered, they would look to control their teammate, which is undesirable.\n\nIn this way, our theory suggests that maximizing empowerment is not always ideal, but instead, agent design requires balancing plasticity and empowerment tactfully. We believe this represents a significant change in perspective, and one that can inspire future research on the design of cooperative learning agents that act in general environments. While the specifics of how to best achieve this are hinted at above in (1.) and (2.), we would like to emphasize that a principled change in perspective is also valuable.\n\nReferences:\n* Klyubin, A. S., Polani, D., & Nehaniv, C. L. (2005, September). All else being equal be empowered. \n* Leibfried, F., Pascual-Diaz, S., & Grau-Moya, J. (2019). A unified bellman optimality principle combining reward maximization and empowerment. Advances in Neural Information Processing Systems, 32.\n\n\n__4. Game theory and MARL.__ Fourth, there are opportunities to connect to the multi-agent and game theory literature, and to use multi-agent concepts to motivate the balance of plasticity and empowerment. In a two-player game, one player’s plasticity is the other player’s empowerment (as in the poker example). In an N-player game, one player’s empowerment contributes to the other players’ plasticities. As such, we believe it would be fruitful to again conduct an experimental study involving traditional solution concepts from the point of view of plasticity and empowerment—for certain kinds of games or dilemmas, what must hold at equilibrium? Can we use the tools of plasticity and empowerment to make predictions about such equilibria? What does cooperation or defection imply about a player’s (or a communities) plasticity and empowerment? Exploring these questions can give further insight into achieving the proper balance.\n\n__5. Safety.__ Lastly, we foresee valuable pathways to connecting to the safety literature. As Turner et al. (2021) explored in Optimal Policies Tend to Seek Power, we speculate that one way to modulate the safety of different systems is through the use of tools like plasticity and empowerment. For example, in a collaborative setting, it would be desirable for agents to act in a way to respect the agency of its collaborators—that is, to ensure that it does not systematically damage the plasticity or empowerment of collaborators. This is again a practical pathway that would enable us to design agents that balance plasticity and empowerment tactfully.\n\nWe thank the reviewer for their questions, and are happy to discuss anything further."}, {"review_id": "dfz2z2wuFb", "reviewer": "Reviewer_vgU6", "summary": "The authors introduce a general definition of \"plasticity\" for agents. To do so, they create a generalized version of directed information that does not require comparing sequences of equal length from the same time step. They show that agent plasticity is the \"mirror\" of environment empowerment (and vice versa). Moreover, they show that plasticity and empowerment are in tension: it is not possible to simultaneously maximize both the agent's plasticity and empowerment. This raises questions into what the optimal trade-off is between the two. At a minimum, the authors posit that an agent should have non-zero plasticity and non-zero empowerment.", "questions": "I'm curious, what do the authors think is the most important empirical work to do in this space to build upon their proposed conceptualization of plasticity and the plasticity-empowerment trade-off?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Overall, I thought this was an _excellent_ paper, and I strongly recommend its acceptance.\n- It proposes a conceptually crisp and general notion of \"plasticity\" that relies on only minimal assumptions on the interaction between agent and environment.\n- The definition fits well into existing literature on empowerment as they show that plasticity can be seen as the mirror of empowerment, which is conceptually very appealing\n- They provide a generalization of directed information that is needed to study either plasticity or empowerment over time at different intervals\n- Their main theoretical results around (a) that agent plasticity = environment empowerment (and vice versa) and (b) the tension between agent plasticity + agent empowerment raise many interesting questions regarding the design of agents. How should plasticity and empowerment be balanced?\n- I think this framework will have wide appeal and is broadly relevant for many areas ranging from RL theory to modern LLM agents to recommender systems\n- The paper is very well-written and clear", "was_revised": true, "final_justification": "I am keeping my already high score (strong accept), for the reasons described above in \"strengths and weaknesses\"", "rebuttal": "__Overview.__ We would like to thank the reviewer for their thoughtful review of our paper, and for their warm comments on the work. We truly appreciate it! We respond to the reviewer’s question below, but if they have added comments or questions during the discussion period we are happy to talk further.\n\n> what do the authors think is the most important empirical work to do in this space to build upon their proposed conceptualization of plasticity and the plasticity-empowerment trade-off?\n\nThis is an excellent question, and one we spent some time considering since submission. We have since implemented a small-scale experiment to corroborate the basic findings of the work, but believe there are several important directions to explore to build on the work.\n\nFirst, we believe it is important to better understand which algorithmic knobs can directly control an agent’s plasticity and empowerment. For example, changing an agent’s learning rate parameter intuitively controls the agent’s plasticity—is there an equivalent, typical parameter that modulates an agent’s empowerment? If not, can we design one? Do these knobs also have knock-on effects to other parts of the agent (does changing learning rate impact empowerment?)\n\nSecond, it is important to better understand how goals fit into our picture. As the reviewer indicated in their review, we are curious about what an optimal balance of plasticity and empowerment looks like. To study this question, we might introduce goals in the form of a reward signal as is typical in reinforcement learning, then explore when agents should lean into maximising plasticity, empowerment, or achieving a balanced trade-off. We believe that a systematic empirical study of standard learning algorithms and environments would be a natural place to seek this understanding, particularly in a continual learning setting.\n\nThird, there are opportunities to connect to the multi-agent and game theory literature. In a two-player game, one player’s plasticity is the other player’s empowerment (as in the poker example). In an N-player game, one player’s empowerment contributes to the other players’ plasticities. As such, we believe it would be fruitful to again conduct an experimental study involving traditional solution concepts from the point of view of plasticity and empowerment—for certain kinds of games or dilemmas, what must hold of an equilibrium? What do standard methods achieve in terms of plasticity and empowerment? Can we use the tools of plasticity and empowerment to make predictions about such equilibria?\n\nLastly, we foresee valuable pathways to connecting to the safety literature. As Turner et al. (2021) explored in _Optimal Policies Tend to Seek Power_, we speculate that one way to modulate the safety of different systems is through the use of tools like plasticity and empowerment. For example, in a collaborative setting, it would be desirable for agents to act in a way to respect the agency of its collaborators—that is, to ensure that it does not systematically damage the plasticity or empowerment of collaborators. We can again imagine an empirical study exploring how algorithms in a cooperative setting might influence the plasticity or empowerment of their collaborators, and try to reveal potential mechanisms for improving these methods.\n\nThese are a few of the directions we believe would be valuable to follow up on. Thank you again for the question, and for the time spent reviewing our paper.\n\nReferences:\n* Turner, A., Smith, L., Shah, R., Critch, A., & Tadepalli, P. (2021). Optimal Policies Tend To Seek Power. Advances in Neural Information Processing Systems, 34."}, {"review_id": "J59MqsJOMH", "reviewer": "Reviewer_o8k4", "summary": "This paper notes that in the abstract, and an agent and an environment are symmetric communication partners; we just call the agent's messages \"actions\", and the environment's messages \"observations\". Following, and slightly extending, the literature on \"empowerment\", the paper observes that the empowerment of the environment is an interesting quantity that corresponds to an agent's \"plasticity\"--the extent to which the environment can guide its behavior. Ultimately, the paper proves an upper bound on the sum of an agent's plasticity and its empowerment, indicating that the two (desirable) quantities are in tension with each other.", "questions": "Could you please explain to me why Theorem 4.8 holds?", "limitations": "It's hard for me to identify any limitations with this work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I'll remember this paper, and what I've learned from it. The paper was clear and easy to read, and it is original. It introduces a new conceptual tool, and I think it may be significant.\n\nA lot of the paper consists of developing notation and definitions--I suppose that's necessary for this kind of work. I somehow feel like I didn't gain that much more from reading the paper than I did from reading the abstract, with one important exception, which is Theorem 4.8. So I would have liked to see a proof sketch, and maybe a full proof in the body of the paper.", "was_revised": true, "final_justification": "Thank you to the authors and other reviewers for the discussion. My questions have been answered and I am keeping my score the same.", "rebuttal": "We would like to thank the reviewer for taking their time and energy to read our paper. We really appreciate their thoughtful comments on our work. We have included a brief response to their primary request below, but are happy to engage and discuss any further questions or comments that might come up.\n\n> Could you please explain to me why Theorem 4.8 holds?\n\nWe are happy to, and we will plan to make adjustments to the exposition of the result in order to further clarify.\n\nThe short intuition is this: if an agent fully controls its environment, it cannot also learn from it. Consider a setting involving a two player band that consists of a guitar player (G) and violin player (V). If G is maximally empowered, it means that the notes they play fully determine what V plays. So, imagine V repeats back exactly the note they hear from G, then V is fully plastic. In this way, G cannot then learn anything meaningful from what V plays, since G will just be hearing exactly what they already played. To get the most out of the collaboration, we likely want G and V to both be a little plastic and a little empowered: they should listen to their collaborator (plasticity), but also try to contribute something meaningful (empowerment).\n\n\n> So I would have liked to see a proof sketch, and maybe a full proof in the body of the paper.\n\nWe completely agree with this point, and will plan to include the proof sketch and expanded intuition for the main two results (the GDI conservation and the Plasticity-Empowerment tension) in the camera ready. If we have space, we will be sure to include a proof sketch of this result in the main paper (along with a sketch of the proof of the conservation law of GDI).\n\nWe thank the reviewer for their thoughtful review, and are happy to discuss anything further."}]

Author rebuttal:

Meta-review: This paper develops an information-theoretic framework for understanding agent-environment interactions by introducing a mathematical definition of "plasticity" - the extent to which an environment can influence an agent's behavior. This uses a novel concept called Generalized Directed Information (GDI) that extends traditional directed information to handle sequences of different lengths. The key insight is that plasticity and empowerment (an agent's ability to influence its environment) are mirror quantities that exist in a fundamental trade-off relationship, meaning that, in general,  both cannot be simultaneously maximized. The work demonstrates that agents and environments are symmetric communication partners, with the agent's plasticity corresponding to the environment's empowerment, and establishes theoretical bounds showing these desirable qualities are inherently in tension with each other.

Reviewers identified the following strengths: The formulation of Generalized Directed Information (GDI) represents an interesting theoretical contribution that extends directed information to arbitrary subsets and has potential applications beyond plasticity/empowerment. The main theoretical results demonstrate the impossibility of simultaneously optimizing empowerment and plasticity provide thought-provoking insights. For instance, in multi-agent environments, each agent belongs to the environment of other agents, and for things such as cooperation to emerge, individual agents must accept limitations on their empowerment. The paper is generally well-written with appropriate background detail.

More significantly, the majority of reviewers praised the work's conceptual clarity and broad relevance, noting that it proposes a crisp, general notion of plasticity requiring minimal assumptions about agent-environment interaction, with the mirror relationship between plasticity and empowerment being conceptually appealing and well-integrated with existing literature. The theoretical framework raises compelling questions about optimal agent design and the balance between plasticity and empowerment. We might expect wide appeal and lasting impact across diverse areas from reinforcement learning theory to modern language model agents and recommender systems.

In terms of weaknesses, there was discussion between the authors and some reviewers about the choice of the term "plasticity", but the authors give a solid defence of this choice. There are also some technical discussions regarding the meaningfulness of the result and possibility of actionable insights. However, the authors rebut these effectively in the first instance, and it may be that the community response to this paper will be needed to determine the full scope of this. Finally, there was a request for experimental evaluation of the ideas in the paper.

Based on the paper, reviews, rebuttals and responses, I believe the paper to be of high quality and worthy of acceptance at conference.
Decision: Accept (spotlight)