Influence Learning in Complex Systems
Authors: Elena Congeduti
Roberto Rocchetta
Frans A Oliehoek
Venue: Accepted by TMLR ()

Abstract: High sample complexity hampers the successful application of reinforcement learning methods, especially in real-world problems where simulating complex dynamics is computationally demanding. Influence-based abstraction (IBA) was proposed to mitigate this issue by breaking down the global model of large-scale distributed systems, such as traffic control problems, into small local sub-models. Each local model includes only a few state variables and a representation of the influence exerted by the external portion of the system. This approach allows converting a complex simulator into local lightweight simulators, enabling more effective applications of planning and reinforcement learning methods. However, the effectiveness of IBA critically depends on the ability to accurately approximate the influence of each local model. While there are a few examples showing promising results in benchmark problems, the question of whether this approach is feasible in more practical scenarios remains open. In this work, we take steps towards addressing this question by conducting an extensive empirical study of learning models for influence approximations in various realistic domains, and evaluating how these models generalize over long horizons. We find that learning the influence is often a manageable learning task, even for complex and large systems. Additionally, we demonstrate the efficacy of the approximation models for long-horizon problems. By using short trajectories, we can learn accurate influence approximations for much longer horizons.

Published in Transactions on Machine Learning Research (03/2025)
Influence Learning in Complex Systems
Elena Congeduti
c.congedul@@hudeeJf..t
Department of Computer Science
Delft University of Technology the Netherlands
Roberto Rocchetta
roberto.roccheftte Sauphei
Intelligent Energy System Group
University of Applied Sciences and Arts of Southern Switzerland Switzerland
Frans A. Oliehoek
f.a.olichock Strieehookk@ddeef......
Department of Computer Science
Delft University of Technology the Netherlands
Reviewed on OpenReview: https: // openreview. net/ forum? id- tUnyInYbjK
Abstract
High sample complexity hampers the successful application of reinforcement learning meth- ods, especially in real-world problems where simulating complex dynamics computation- ally demanding. Influence-based abstraction (IBA) was proposed to mitigate this issue by breaking down the global model of large-scale distributed systems, such traffic control problems, into small local sub-models. Each local model includes only few state variables and a representation of the influence exerted by the external portion of the system. This approach allows converting a complex simulator into local lightweight simulators, enabling more effective applications planning and reinforcement learning methods. However, the effectiveness of IBA critically depends on the ability to accurately approximate the influ- few examples showing promising results in
ence of each local model. While there benchmark problems, the question of whether this approach is feasible in more practical scenarios remains open. In this work, take steps towards addressing this question by conducting an extensive empirical study of learning models for influence approximations in varions realistic domains, and evaluating how these models generalize over long horizons. We find that learning the influence often a manageable learning task, even for complex and large systems. Additionally, we demonstrate the efficacy of the approximation mod- els for long- horizon problems. By using short trajectories, we can learn accurate influence approximations for much longer horizons.
1 Introduction
Controlling large distributed systems is key challenge artificial intelligence with the potential to impact many fields of application, including computing and information technology (Coulouris al. 2001). energy systems (Järventausta et al. 2010 Nweye et al. 2023) and transportation (Dimitrakopoulos & Demestichas, 2010). Reinforcement lcarning (RL) could be promising for such problems as it provides a framework for studying how agents learn and plan under uncertainty in sequential decision making problems. Despite recent successes for large sequential decision-making problems (Kaelbling et al. 1998 Sutton Barto 2018 Mnih et al.] 22015) Silver et al.] 20166). RL techniques still suffer from high sample complexity (Kakado 2003 Botvinick al.] (219)), which means that agents typically require many trajectories sampled from a simulator to achieve optimal performance. This sample inefficiency becomes significant hurdle in real-world

---

Published in Transactions on Machine Learning Research (03/2025)
scenarios characterized by large, structured environments with complex dynamics. In these contexts, running computationally expensive simulators to collect sufficient sample of trajectories can be unfeasible.
To mitigate this, influence-based abstraction (IBA) (Oliehoek et al.) 2012: offers principled framework for decomposing the global model of a large factored multiagent problem into small local models, which support lightweight simulations. This approach has proven to be a powerful tool for accelerating online planning (He et al., 2020: 2022), deep RL (Suau et al,, 2022b c and deep multiagent RL (Suau et al.) 2022a The core idea is to leverage the factored structure of the environment to build local model for cach single agent, assuming that the other agents' policies are fixed. In this way, each agent has local best response model, which it can use to compute a best response, in terms of local rewards, to the other agents. By abstracting away a large portion of the environment, only a few state variable, referred to as the local faclors, are retained in each local model. To overcome information loss due to the abstraction process, each local model complemented with a representation of the influence, capturing the effects of external factors and policies of other agents on the dynamics of the local factors.
This abstraction approach has the potential to substantially reduce the state space of the best response problem without any loss in value. In other words, the local agent achieves the same performance when looking for solutions in the smaller local model as in the global model (Olichock et al. 2021p. However, when part of the system is abstracted away, the local factors no longer preserve the Markow property. To restore Markov transitions, the state of the local model needs to be augmented with the history of appropriate state and action variables. Unfortunately as the number local histories grows exponentially with time, computing the influence (a conditional marginal inference problem) becomes unfeasible, even for small systems or short- horizon problems. This motivates the idea of employing machine learning methods, which have shown excellent results in sequence modeling (Goodfellow et al. 2016 Gamboa 2017 Ismail Fawaz et al., 2019) to learn approximate representations of the influence. Intuitively, the more accurate the influence learning models are, the smaller the value loss. Congeduti et al. (2021) prove formally that the value loss is bounded by the approximation error. Therofore, obtaining accurate approximations is crucial for the effectiveness of the entire approach.
Even though influence approximation has been successfully applied in variety of benchmarks, showing improvements in terms of planning and RL performance (He et al. 2022 Suau et al., 2022b), there exists no systematic analysis of how difficult the influence learning task is in complex scenarios and which learning methods are most effective for this purpose. In addition, maintaining accurate approximations might become particularly difficult for long-horizon tasks. In fact, methods for sequence modeling often struggle with capturing fong-term dependencies (Pascamu et al., 2013 Gu & Dao 2024). Even when advanced techniques prove effective, they might. not. be computationally feasible, as training could require running costly long simulations. To address this gap and understand for which problems IBA can be successfully applied, we investigate the learning of accurate inffuence models across various realistic scenarios. We examine which learning aspects
crucial for this purpose. Thus, our aim is not to propose novel algorithmic approach. Instead, we sock to empirically evaluate existing learning methods for approximating the influence in scenarios that may present challenges, such as complex dynamics and long-horizon problems. As key contributions of our work we show that:
Approximating the influence might typically be an easy learning task, even for complex and large systems. In fact, small recurrent and temporal convolutional neural networks can learn accurate influence approximations across the investigated benchmarks.
Learning models can be trained on short-horizon sequences and then deployed to approximate the influence over much longer horizons while maintaining high accuracy. Additionally, we propose method to estimate the accuracy of the models over long horizons using the short training sequences.
Note that such collection of local problems subject the prize of anarchy. That it is not guarantoed to lead to optimal system level behavior in terms of total rewards. Nevertheless, it may allow to deal with complex optimization system in tractable way, which is one of the core motivations that sparked the interest in multiagent systems (Shoham & Leyton- Brown 2008).
2

---

Published in Transactions on Machine Learning Research (03/2025)
In this way, we give positive evidence that influence- based abstraction might provide a feasible approach to dealing with decision making in certain real-life complex systems.
2
Background
Here we give a concise introduction to decision making problems formalized as partially observable Markov decision processes (POMDPs), as well as influence-basee based abstraction.
2.1 Factored POMDPs and best response problems
where S is the finite state space of the environment, A is the finite space of available actions, S2 the observation space, and h the horizon of the problem. Note that, assuming finite state and action space, we restrict our work to discrete state and action variables. Initially, a state s E S is drawn from the initial distribution s ~ 8 At any discrete time step 2 O, the agent chooses an action a² E A, and the state changes according to the distribution s T(- s* sa,a"). The agent then reccives a reward modeled as r' R(s', a') and an observation o'+1 ~ 06 (0 a²). .a)). A policy π encodes the agent's behaviour, mapping the action-observaaton histories h = (a", o0", ,....,"""" (d",o",...,,,a11,,, ,0') into probability distributions over the action space, i.c. r(h') E A(A). The goal of the agent is to optimize the expected cumulative reward for employing policy given the /h= |T, The policy attaining the maximum value V* is called optimal policy and is denoted by x². We focus on specific domains where the state space 8 can be decomposed into state variables or factors, known factored POMDPs (Hansen & Feng 2000
Formally, a POMDP (Kaelbling al., 1998) is tuple M (8,,7,,,,,,
a..,,,,,,'')
action-observation history h²: (*"h"= =
In line with the concept of joint equilibrium-basedd search for policies (Nair et al. 2003), we adopt the perspective of protagonist agent i in problems with multiple interacting agents. This approach is commonly used to solve multiagent problems, as many solution methods rely on finding the best response policies for individual agents Nair et al.) 2003; Hansen et al. 2004 Oliehoek et al.] 2014). Specifically, by fixing the policies of all other agents x-s, the best response problem for agent can be formulated as a factored POMDP where the actions of the other agents o.; are included into the state space as factors. For the rest of the paper, will omit the subscript i and assume to address best response problem for single agent modeled as factored POMDP and denoted as Mglobal A, T, R, O,b,, b, h)).
2.2 A traffic example
We will use as a running example a traffic light control problem represented in Figure 1 protagonist agent manages the traffic light at intersection 1 within a large road network. The goal of the agent is to minimize traffic congestion the local intersection by leveraging the information from traffic observations from the surrounding area that delimited by the red box. This traffic example can be represented as factored POMDP where the state of the system is represented by the variables that measure the traffic levels at differ- ent road stretches as, for instance, the variables $nj. Swe,, 8ere highlighted in Figure 2(a) Factored POMDP can be more compactly represented by exploiting conditional independence between factors. Specifically, the transitions and observation probabilities can be represented by specific forms of Bayesian networks, the
two-stage dynamic Bayesian networks (2DBNs) (Boutilier et al. 1999), as illustrated in Figure 2
Achieving optimal control at intersection does not require simulating all the state variables. For instance, the decisions made al. intersection affect indirectly the observations of the agent through the car inflow from the north end (blue arrow). By abstracting away those factors that have only indirect effects the local intersection, a smaller model can be constructed, as depicted in the red box Figure [[a]] In particular, the local factors highlighted in red, 8n1, Swe, represent the incoming and outgoing traflic flows from the north and west ends, respectively. The north end inflow denoted by sare and depicted in blue is a so-called influence source (to be defined formally alter) as it influences directly the local traffic measured by sng. The
2To case the notation, we use small letters denote both random variables and their realizations. Capital letters denote sets and functions. For instance, 5 10 denotes that the random variable representing the state time is distributed according to 80.
3

---

Published in Transactions on Machine Learning Research (03/2025)
3
Figure Traffic example. The red square delimits the road segments included in the local model for the traffic light agent The blue arrow represents the effect the non-local part of the network on the local model and the red arrow the effect of the local traffic the external part of the system.
Seee
Sure
Sece
Sare
Sal
So
disct
dses
Sye
t+1
T+1
(b)
Figure 2: Local model for intersection of the traffic network example. (a) graphical representation of the local state variables and dependencics external factors; (b) 2DBN representations of the local and external state variables (left) and the abstract I-ALM (right).
inffuence source captures the only information about the external portion of the system necessary define the local transitions for the north end traffic 8n4-
4

---

Published in Transactions on Machine Learning Research (03/2025)
2.3 flluuenee-based abstraction
As we saw in the example, the factorization of the state space allows to leverage the conditional independence between factors. By abstracting away factors that do not directly affect the agent's 's reward and observations, we can construct a much smaller local model. For instance, in the traffic example the state variables that measure the traffic outside of the area delimited by the red box can be discarded. Thus, the state of the environment can be considered as consisting of three components defined as follow.
Definition 1
The state can be decomposed as (Seat; Sares Stocat), where Stocol represents the local factors retained in the model. Saee denotes all the eaternal state variables that directly influence the local factors and are called the influence sources. All the remaining factors form an ezternal portion of the state, Seat.
In Figure 2(a) the local factors are depicted in red and the influence source that affects directly the local inflow of cars from the north end is depicted in blue. All the factors that do not affect directly the local traffic are depicted in grey and form Sext- To roduce the complexity of the problem, we want to construct the local model, corresponding to the red area in Figure 2(a) which therefore only contains the local factors. The 2DBN in the left-hand side of Figure 2(b] represents the factorization of the global model state into local factors, influence source and external factors. Note that only the local factors affect the agent's reward and observations. The blue arrow highlight the dependency of the local factor from the external influence sources.
In general the transitions of the local factors depend on the influence sources as P( |ofoe siree a²) which are not part of the local model. For instance, the distribution of Sn, depends on the north traffic inflow represented by Sarc- One possible solution is to condition on history. Specifically, we can use the local history to prodict the influence source distribution P( I slloc:" a² "............ )- Interestingly, however, it may not be necessary to condition on the full local state-action history. Oliehoek et al. (2021) show that one can retain the history of only a subset of variables, the d-set. The d-set is defined as a set of local variables that d-separates the local factors from the external factors, according to the definition of d-separation for causal graphs in a 2DBN (Koller, 2009; Bishop 2006). Specifically, the influence sources at time i, sire: are conditionally independent on the local state and action histories given the d-set, i.e. Psssreldeen sfee ......,................. sfoo) = P(stroldee ( Thus, the d-set includes precisely those local factors and actions necessary to predict the inffuence sources.
1+1
In the traffic scenario, the history of the outgoing local traffic measured at the west end Swe affects the future traffic volumes measured by the influence source Sare and thus is part of the d-set. To predict the influence sources, we augment the local state with the local history of west end outflow Swe. For the sake of illustration, we will assume that the d-set consists only of those factors, i.e., d'et = , left-hand side of Figure 2(b] the red arrow represents the influence, that the distribution of the influence source Saro given the deet, [ssrc d'ee) 2 P(s s......., We can now provide the formal definition.
In the
Definition 2 (Influence) We define the influence I as the conditional probability distribution of the influ- ence sources given the d-set at time t, ('steedd'.a)) P(st.add'ee) defined for any time step t < h.
With these notions, we can define a local model as a POMDP.
Definition 3 (I-ALM) An in influence--uugenttee
local model (T-ALM) corresponds to the factored POMDP, Miscal = (Sungms A, Tiocal; R, Ωt O, b h), where the augmented state space Saugm comprises the local factors and the d-sets, i.e., (Stocal, dset) E Saugm- A is the action space of the agent. The local transitions Trocal, which model the distributions of the local factors, are defoned through the influence I by marginalization over all the possible influence sources 8 Syre. Precisely,
Trocal 11 &looat d'eees d²) )+== = Σ PS Stocuts sres
a' (ss sett)
(1)
Note that we assume that the local factors encompass all the variables that directly affect the agent's reward and observations. As a consequence, the reward R and observation O are the same functions as those defined for the global model.
5

---

Published in Transactions on Machine Learning Research (03/2025)
Assuming that can derive the exact influence I, the I-ALM Miocal defines problem equivalent to the global model Mglnbal- That is, the two problems share the same optimal policies. Consequently, solving the smaller I-ALM yields an optimal policy π², which also maximizes the value of the original global problem. We refer to Olichock et al. (2021) for a formal proof of this equivalence and an extensive discussion of the IBA concepts introduced in this section. However, deriving the influence for any possible d-set requires solving an exponential number of inference problems in the horizon For instance, to define the influence for the local traffic model, we would need to compute number of conditional probability distributions of the order of sw+), which corresponds to the cardinality of the d-set. This task is practically infeasible even for small values of the variables. This challenge leads to the idea of learning approxinations of the influence 1.
2.4 Approximate influence- based abstraction
Given a representation f, an coprozimate influence-augmented local model, f-ALM is defined following the same structure of Definition 3 the factored POMDP Miseal = (Saugm A, Troaat, R, O,8², h), with local transitions Tiocal defined by replacing the exact influence in equation [ with the approximate influence f. To construct an Í-IALM, we need learn f. For this task, we first simulate n trajectories of influence sources and d-sets from the global simulator to form the training set Dr. = { sacc),.... (d str) I,..... Now, we define the learning task.
Definition 4 (Influence learning task) Given a training set of influence sowrces and d-sets trajectories Dr. the influence learning lask consists of learning a predictor for the conditional distribution of the influence sources s'rre given the local history in the d-set d'eet [ss for any t= == .
A probabilistic model parameterized by 0, for example a neural network, can be used for this sequence modelling task to approximate the inftuence as i(0) [s dia 0). The learning model is trained to solve the optimization problem formulated as
0+ = arg min ED, 1 deet 0)]
(2)
where the expectation corresponds to the average cross entropy loss between the target influence and the approximation model.
As mentioned in the introduction, the advantage of this approach lies in the possibility of using small sample D from the global simulator to build the local lightweight simulator Î-ALM, which enables more efficient sampling and accelerate the solutions search. Moreover, Congeduti et al. (2021) provide additional support for the approximate abstraction by demonstrating that the value loss for solving the sequential decision-making problem defined by the f-ALM is bounded by the worst-case KT. divergence error of the influence predictions over all the possible d-sets. This bound guarantees that any influence Í that minimizes the mcan cross entropy loss according to equation and thus the mean KL divergence-is aligned with the objective of minimixing the value loss.
2.5 State decomposition
In our work, we assume that the decomposition into local and external factors an engineering choice and thus given, with the constraint that the local factors include all variables that directly affect the reward and the observations. This ensures that reward and observations remain well-defined at local level (Oliehoek) et al. 2021). Ideally, the local model should be small as possible to enable simulation speedups. However, in some cases a larger local model can case the problem of predicting the influence sources. This may be viewed as a potential limitation, since finding a good trade-off between the information included in the local model and the model size may not be trivial. However, in practice, we have generally found it relatively straightforward to identify sets of local variables that yield good performance (Suau et al., 2022a).
Although we recognize that different decompositions may lead to different outcomes, any choice determines set of influence sources and d-set, and thus defines an influence learning problem. Assessing the impact of
6

---

Published in Transactions on Machine Learning Research (03/2025)
different decomposition the complexity of the resulting inffuence learning problem is quite challenging task. Therefore, a comprehensive analysis of the effect and efficiency of different choices is beyond the scope of this paper. On the other hand, given a local model, identifying an optimal d-set is relatively straightforward: we select a minimal d-sct, as this also the best choice for the set of imput features to the learning models. Using larger (still d-separating) set of local variables will not improve the accuracy of the learning models. Instead, it would increase the dimensionality of the input of the learning problem, generally making learning more difficult.
3 Empirical study of influence learning
The influence learming problem is a sequence prediction task that corresponds to modeling the temporal de- pendencies between the local and the external factors governed by an undorlying 2DBN. This structure differs substantially from the beuchmark domains lypically used to test sequence modeling methods, such as speech recognition, machine translation, audio classification, music generation, image-video caption generation (Bai et al. 2018 Keneshloo et al. 2019). Thus, this learning problem warrants separate investigation. Based the intuition that humans generally quite successful at reasoning about parts of complex systems in isolation, we hypothesize that the local structure of the problems may make the task of learning the influence quite feasible and manageable, even with relatively low-complexity methods for sequence predictions.
To make an impact on real problems with IBA, we need to address two important aspects. First, real-world tasks are often large and complex, which means that we need to explore if approximating the influence is feasible for such problems. Additionally, many real-world tasks often involve systems, such traffic networks, that operate continuously, which mcans that the learning models should work well for very long (or infinite) horizons. For the entire IBA approach to be effective, need to have sufficiently accurate influence approximations in these situations. Therefore our investigation focuses on two key aspects:
1. Testing the hypothesis that even for large and complex systems, the learning problem may turn into manageable task that does not require large or complex neural networks.
2. Investigating to what extent the representations learned from trajectories with a short training horizon Atrai can approximate well the influcnce over a much longer deployment horizon hdeploy > Itrain-
3.1 Simulation domains
Our experimental setup includes a range of realistic simulation domains, covering diverse situations in terms of number of influence sources to predict, level of 'uncertainty' of their distributions, strength of the depen- dency between local variables and influence sources, dependency over past. time steps, and problem horizons. This variety leads to different characteristics of the learning tasks. Our aim is to determine if this has an impact on the performance of the learning models. We consider four simulation domains designed to test the efficacy of local influence proxies on realistic tasks:
Microgrid (MG): realistic application of power system management the energy engineering sector. Similar Vázquez-Canteli al. (2020), in this environment a hundred prosumers (units that both produce and consume electrical power) interact by trading energy minimize the costs of energy bought from external grid while meeting the internal power demand. This use case closely emulates problems of practical importance in power engincering, particularly in modern contexts involving distributed energy prosumers, strategies for managing energy storage systems, and recent efforts toward achieving net-zero, positive-energy districts.
Traffic grid (TG): traffic light control problem that models the interactions between cars and traffic lights in a road network, as briefly introduced Section 2.2, This scenario features many external factors which exert direct influence on the local model. Hence, the influence learning problem corresponds to high-dimensional prediction task. This transportation network use case
7

---

Published in Transactions on Machine Learning Research (03/2025)
serves as surrogate for real-world challenges, such as optimizing traffic light operations to minimize queue times and reduce congestion.
System admin (SA): A multiagent version of the system administrator domain described in Poupart & Boutilier (2004). This environment reproduces realistic challenge in the information technology domain: a team of system administrators has to manage a network of potentially faulty machines by deciding which machine to reboot.
Grab chair (GC): A simple gaming environment introduced by He ct al. (2020), where a group of agents competes to obtain one of the available chairs. This environment is simplified version of the system admin serves a control scenario for proof-of-concept experiments and preliminary investigations.
We refer the interested reader to Appendix B for a detailed description and visualization of the simulation domains.
3.2 Comparison of learning models
We evaluate the performance of different classes of learning models with various degree of complexity (Roe- chetta et al. 2023), measured as the network size or number of learnable parameters on the influence learning task.
Models and benchmark domains used for this experiment. Many successful sequence prediction methods for soquential predictions are based on neural network with recurrent and temporal convolution components (Karim et al. 2017: Bai et al., 2018 Ismail Pawaz et al.] 2019). We evaluate several classes of models including recurrent models such as long short-term memory (LSTM) (Hochreiter & Schmidhuber 1997) and gated recurrent unit (GRU) (Cho et al., 2014), as well as temporal convolution-based models like temporal convolutional network (TCN) (Lea et al. 2017) and fully convolutional network (FullyConv) [Long et al.) 2015). To compare these models with a simpler non-recurrent baseline, we assess a one-layer fully connected linear network, a logistic regression (LogReg) model. We exclude more complex and larger models, such as transformers (Vaswani et al.] 2017 Wen al. 2022). as we assume, and demonstrate, that small and simple models can provide accurate influence representations. For each model class, we compare different sizes, ranging from lightweight networks with less than 100 parameters scaling up to the size a one-layer linear neural network counting more than one million parameters. The hyperparameters defining the network architectures have been chosen to ensure a fair comparison between learning models with the same network size. Since our goal is to investigate whether relatively small and simple models can achieve satisfactory performance, we focus on models with a small number of layers. This choice is also motivated by the fact that deeper networks typically require a larger amount of global simulator trajectorics to be trained effectively (Bengio al.) 2013). Moreover, the choice of the network depth depends on the model class. Specifically, convolutional- based models are known to benefit from deeper architectures to achieve better performance and to ensure full receptive field coverage (LeCun et al. 1998 He et al. 2016). For the details of the architectures and sizes used in the different domains, we refer to Table
and 7 in Appendix D The models are evaluated over three domains: microgrid, system admin, and traffic grid. For each domain, we choose the sconario fcatures, including the number of interacting agents N, the protagonist agent i, the policies of the external agents T- the problem horizon h, the initial distribution 1 and other speain-specific parameters that define the transition and reward functions. These scenarios are designed to cover range of diverse characteristics in terms of problem size, time length, prediction dimension and stochasticity for the influence learning task. Specifically, the microgrid represents large scenario where 100 agents interact within a power grid; the traffic grid scenario is real-world traffic control application that turns into a high dimensional learning task; and the influence learning problem for the system admin simulator is a long-horizon forecasting problem.
Experimental sctup. We use a random exploratory policy for the local agent Exp which affects only the data distribution of the training set We collect n 500 trajectories of influence sources and d-sets from the global simulator to form the training set D,. The different models trained to approximate the influence
8

---

Published in Transactions on Machine Learning Research (03/2025)
as f, using the mean cross entropy as the training loss. We employ h independent logistic regression models, each one representing the influence T(s (d)at) for a specifie time step E (0..... h 1]. We adopt standard optimization techniques, including the ADAM optimization algorithm, linear decay of the learning rate and grid search over the space of imitial and final learning rates. For each sccnario, a fixed number of epochs and the batch size are selected. Details on the scenarios and hyperparameter configurations are provided in Table 3 and Table Appendix D respectively.
Performance metrics. The model performance assessed an a test set consisting of Rsesst independent global model trajectories. We use the cross entropy test error, which is computed the mean over time and sum over the influence sources of the cross entropy estimators. Specifically, for each factor S Sai.. of the influ- ence sources s're = " ,) an estimator of the cross entropy error CE( deet), [ dgee)) at time t is computed over the test sample. Then, the sum over the J influence sourees provides the error for given time step according to the following formula:
CE'(I,Ñ) Σ
In (Stragg diatts))
(3)
Rreet
With a slight abuse of notation, we denote both the probability distribution of individual influence sources as (shre,)) (gee) and the joint distribution as [s déet). The cross entropy error is obtained by averaging the errors defined per time step in equation 3 as follows:
e(h) = h Σ CE.
(4)
t=0
We also measure the wall-clock training times (WCTTs) to assess which model provides the best trade-off between accuracy and training time.
3.3
Generalization beyond the training horizon
In many real-world applications, an approximate model should perform well over long horizons. Examples of problems that require long-tern teg-term simulations include long horizon planning (Simeonov et al.) 2020 Pertsch et al. 2020), episodic reinforcement learning (Dann & Brunskill] 2015) and tasks characterized by sparse reward signals (Riedmiller et al.) 2018). Although it might be feasible to gather a sufficient number of trajectories from the global simulator to approximate well the influence, this may no longer be the case when dealing with long horizon problems as long-term dependencies add further complexity to the learning task.
Our approach to address this issuc is to leverage learning models to generalize the influence representations beyond the training horizon. This intuition is grounded in the ergodic theory for MDPs (Puterman, 2014; Morton Wecker 1977 Kearns & Singh] 2002). Under ergodic assumptions, the Markov dynamics induced by joint policy (T TT) converge to a unique stationary distribution independently of the initial distribution. The mixing time tmix, refers to the number of time steps required for the state distribution to approximate the stationary distribution within specified tolerance. Such mixing time is determined solely by intrinsic properties of the system and is independent of the initial conditions. This argument supports the concept of a stationary influence as the limit the conditional distributions of the sequence (T(yhre d- In other words, when the system has sufficiently mixed, the influence approaches a time- independent function, representing the distribution of the influence sources at equilibrium. In this scenario, global trajectories with horizon Maaai > lmix contain sufficient information to learn the stationary influence. Therefore, if the mixing time is sufficiently short, learning models can capture the stationary influence using short training sequences and use it to predict effectively the influence over (indefinitely) longer deployment horizons hdeplo > Atrain-
How to choose an appropriate training horizon for this task remains an open question. We expect predictions to improve as the training horizon Ataain approaches the deployment horizon heployy- However, we aim to limit the computational effort for running long global simulations. Therefore, the training horizon must offer a good trade-off between the quality of approximations and the length of the training sequences. Specifically,
9

---

Published in Transactions on Machine Learning Research (03/2025)
we seek an optimal training horizon Mirain as the minimum number of time steps required to guarantee accurate predictions across the entire deployment horizon. Intuitively, the optimal horizon Muain depends on the system's 's mixing properties and the ability of the learning models to capture the stationary distribution. Specifically, we verify that it corresponds to the sum of the mixing time mix and a few additional time steps needed for the learning models to generalize the experience encountered after mixing.
Models and benchmark domains used for this experiment. LSTM and FullyConv networks are employed for the experiments, they achieve the best performance in the experiments of the previous section the influence learning tasks. The architectural hyperparameters are selected to ensure that the learning models share the same size. Tables g and 10 in Appendix D report the optimization and network architecture details, respectively. We use the grab a chair and traffic grid domains as described in Section 3.1 Grab a chair serves as controlled environment where a known stationary influence is reached after a few time steps of the mixing time. Specifically, we consider two different scenarios: the first with N = 4 interacting agents denoted as GC4, and the second with N 11 agents denoted as GC11. We assume that the agents are ordered by index and each external agent 2, 3... copies the action of the preceding agent previous time step (sce Figure 12 in Appendix B For instance, agent at time copies the last action of agent 2, which is, in turn, the action of the local agent 1 at time ( 2.In general, for any agent i, of = af-i+1 Note that the external agent's policies are arbitrary features of the experimental domains and, as such, can be chosen to induce various influence learning tasks. This ad-hoc choice ensures that after the mixing time lmix = N - the system converges to a known stationary distribution determined by the local agent policy, i.e. af ~ #i Exp for 2 1. As a result, the influence is a time independent deterministic function of the last N local actions determined by the following formula
I(ay 1ld..'xlo a a Paaa, l'ylaa ,a .N]++++)= 5. (2)6
for 2 N
(5)
where da denotes the Dirac distribution centered on the action a. Such explicit expression for the stationary influence allows us analyze the model learning for the different training horizons. We also use the traffic grid domain to validate our arguments in more realistic environment where no prior knowledge of stationarity and mixing time is available. Details on the scenarios, training and deployment horizons can be found in Table in Appendix|] D
Experimental setup. To empirically validate these arguments, we proceed as follows. We consider a set of K training horizons H = (hi...., with hn S ... S hx- We employ a random exploratory policy n Exp for the local agent and collect n global trajectories of d-sets and influence sources with horizon lg- Then, for each model class, we train K learning models, such that the k-th model uses the training set Dry, which consists of trajectories up to horizon hg, to learn the approximate influence (0). To assess the generalization ability, test the models over longer trajectories. For this purpose, collect an independent test sample of trajectories with horizon hasploy >> lx denoted by Discppay"
Deployment, test and test-tail error. We define the function (h) to represent the generalization error of the k-th model when tested over h deployment steps, as given by the following formula:
h-1 Σ (L, t=
(() - T
(6)
where the cross entropy term for time step is defined in equation 3 Note that for h hdeploy, the error (haaepioy)) corresponds to the average error over hdteloy time steps. This error measures how well (on average) the k-th model generalizes the influence over the entire deployment horizon. While for h = h2, the error (14) represents the average error of the k-th model over htrain = h time steps. We refer to these errors as the deployment error and the test ervor, respectively, and denote them as follows:
k edeloy == e (haeptoo.)
e := e
A learning model that shows good long-term performance should maintain the deployment error close to the test error. Additionally, increasing the training horizon should not significantly reduce the deployment error.
10

---

Published in Transactions on Machine Learning Research (03/2025)
We are also interested in investigating how to estimate the deployment error from the short training trajec- tories. While the test error might intuitively seem a good candidate estimator, it is significantly affected by the error terms before the system has mixed. To illustrate this, we distinguish the cross entropy error terms in equation g into two time scales: the short time scale of the mixing time and the long time scale of the deployment horizon. This distinction allows us to break the error function into two terms as follows:
Timx (h) = h Σ
h + Σ CF'(1,,(0)) fani
CE
(8)
The first term accounts for the short time errors over the influence before mixing and the second for long time errors which represent the model's performance over the stationary influence. Given that hteploy > fmix: the deployment error mainly affected by the second term of equation g while the contribution of the short time errors is negligible. Conversely, when the training horizon Herain is close to the mixing time, the first time errors predominantly determine the test error. These errors do not reflect the model's performance the stationary influence, causing the test error to deviate from the deployment error.
The key idea for deriving a more accurate estimator is to neglect the errors over the initial time steps. However, the appropriate number of time steps to discard depends on the exact mixing time of the system, which is generally unknown and difficult to estimate. Therefore, we consider different increasing values of the training horizon hy to test the hypothesis that for certain hy > fmix: the 'tails' of the error represented in equation X are close enough to the deployment error. Specifically, we introduce the test-tail error, which is computed as the average error over window of time steps, according to the following formula:
1 butt Σ CE(I,Í(()) t=
fail
(9)
In other words, for training horizon hys additional time steps are collected in the test set sla+1),, ..... (ahatl s ()){ C Dhaeptor and used compute the test-tail error. We empirically show that in scenarios the error Ctail offers good estimate of Cdeploy for I = 1, and can thus be used to assess the quality of the model predictions over the deployment horizon. However, increasing the number of time steps, f.e. I >>> might prevent disruptive effects of anomalies and give more stability to the test-tail error. Additionally, we show how this error can be used to search for an optimal training horizon Miriin
[[dha+1 Sare
recognize that
4 Results and discussion
Here, we present the empirical results for the two sections of the experiments presented.
4.1
Comparison of learning models
We start by analyzing the test error of different model classes and sizes as function of the mumber of epochs for the three domains. Together with the training curves, we look at the final test errors and wall- clock training times. To compare the model classes, we select one network size for each class from the Pareto optimal solutions (Miettinen 1999) of the bi-objective problem of minimizing lest error and training time. Finally, we look at the progression of the test error over training time to compare the learning speed and accuracy of the model classes.
Effect of model class and size. Figure 3 depicts the progression of the test errors, computed according to equation 4 over the training epochs of the four classes of models (rows) in the three problem domains (columns). For each class and benchmark domain the training curves for different network sizes are rllustrated as average test error with standard error over ten iterations of the experiment. The final test errors and training times are reported in Table averages over the ten itcration of the experiment. The corresponding standard errors are reported for the cross entropy errors but not for the wall-clock training times, as they are negligible.
11

---

Published in Transactions on Machine Learning Research (03/2025)
MICRO GRID
SYSTEM ADMIN
+300
<1000
1030
50002
6.0
FullyCovv
Com Ereees
5.5
10
<1000
-100
-200
1003
1000
1003
19933
10000
10000
3.75
GRU
2.00
1.75
4.3
1.00
<100
1200
15000
10003
10002
55
50003
LSTH
Croe Coneerrrery
0.0
4.0
17
c1100
-100
2000
10000
10000
0.5
2.8
TCN
80
3.0
Figure 3: Effect of model size and class on the different benchmark problems. Each panel depicts the cross entropy test error over the training epochs with standard error over ten iterations for different network sizes. The columns represent the three different test environments and the rows the classes of the learning model.
For the microgrid, the training curves and final test errors show no significant advantages from larger model sizes, except in the case of the TCN model class. In fact, the recurrent- baseeentt--sedddee models, such as LSTM and GRU, do not benefit from increased sizes. Conversely, increasing the TCN size from fewer than 100 parameters to 15, 000 parameters significantly improves the performance, reducing the test error from 4.11 to 3.85. For the FullyConv model, a bigger size leads to a slight performance improvement but triples the training time from 0.46 to 1.55 seconds. These results align with the observation that temporal convolutional networks typically benefit from deeper architoctures and thus require more parameters than recurrent networks (Bai) ltt al. 2018). Especially for long sequences, convolutional networks need several layers reach full receptive field, that. is to ensure that the entire history-length of the input sequence is processed to make predictions.
12

---

Published in Transactions on Machine Learning Research (03/2025)
Despite the higher mumber of parameters, the TCN model have typically low training times. In fact, the TCN model with 15, 000 parameters achieves accuracy to the other model classes in about the same training time. Figure 3 displays also the results on the system admin and traffic grid domains in the second and third columns, respoctively. In contrast with the microgrid domain, very small recurrent (LSTM, GRU) and FullyConv models do not have enough capacity to accomplish the learning lasks. This is due to the higher dimensionality of the space of influence sources and d-sets in the traffic domain and the longer problem horizon for the system admin. These features increase the complexity of the two learning problems compared to the microgrid case.
CE
WCTT
number of parameters
number of parameters
models
< 100
1000
15000
< 100
1000
15000
LSTM
3.81 + 0.03
3.78 = 0.02
3.82 + 0.03
0.98
1.18
2.26
GRU
3.77 + 0.03
3.73 0.03
3.78 + 0.03
0.95
1.13
2.15
TCN
4.11 0.03
3.98 + 0.03
3.85 0.04
0.47
0.59
1.31
FullyConv
3.85 0.03
3.85 = 0.03
3.76 0.03
0.46
0.65
1.55
LogReg
3.85 0.04
0.95
(a) Microgrid
CE
WCTT
number of parameters
number of parameters
models
< 200
1K
10K
50K
1M
S 200
1K
10K
50K
1M
LSTM
6.2 1 0.4
3.4 0.3
3.1 0.3
3.2 4 0.3
20.1
20.9
25.4
40.2
GRU
0.6
0.2
3.1 0.3
3.2 0.3
21.4
22.2
27.5
40.6
TCN
7.4 4 0.2
6.0
5.5 4.4
6.0
7.4
8.2
14.3
FullyConv LogReg
0.2
0.3
0.3
0.3
6.5
7.9
10.3
17.0
5.6 0.3
35.3
(b) Traffic grid
CE
WCTT
number of parameters
number of parameters
models
< 100
1K
10K
3M
I 100
1K
10K
3M
LSTM
1.69 + 0.04
1.06
0.04
1.05 + 0.04 0.04
13.7
18.5
32.4
GRU
1.68 + 0.04
1.06 0.04
1.05 + 0.04
13.1
18.3
33.9
TCN Conly LogReg
2.06 + 0.05
1.98 0.03
1.05 0.04
3.6
5.1
12.7
2.01 + 0.04
1.20 0.05
1.05 + 0.04
3.9
5.2
12.3
1.97 + 0.08
23.2
(c) System admin
Cross entropy test error with standard error and wall-clock training time (s) for the microgrid, traffic grid and system admin computed over ten iterations.
Table 1:
Pareto optimal sizes and learning speed. Figure shows the Pareto frontiers for the model classes (left pancls) where cach marker represents the test crror and training time for specific size. The dashed lines indicate the Pareto fronts for each class, defined by the sizes that are not strietly dominated by any other in the same class. Among the Pareto optimal solutions, provides good trade-off between error and training time. These are indicated by the red markers. On the right-hand side of Figure ] we collect the test error over training time for the selected sizes and compare them with two baseline models: a random model and LogReg.
select and analyze specific sizes which
For the microgrid, the Parcto fronts show that the LSTM, GRU and FullyConv models achieve approximately the same accuracy for the different sizes. Thus, the smaller sizes are selected and highlighted grey in Table as they also offer the lowest. training time. Conversely, the TCN model's error drops significantly with
13

---

Published in Transactions on Machine Learning Research (03/2025)
LSTH GRU TCN
LSTM (c300))
6.5
GRU (=100)
TCN (15000)
4.05
RuthyCer (c100)
6.0
LogRog
Micro Grid
0:08 Entopy
Cress Emtroyy
3.95
0.5
3.90
5.0
3.85
3.80
0.5
WCTT
7-
GRU
TCh
(10000)
TullyCann
FullyCau (10000)
LogReg
Traffic Grid
Cooss Ercovay
Ohos Crow Enteorr
Handom
4-
10
10
VCTTT
LSTN (1U00)
ERU
(1000)
TCN
FidtyConv
2.75
(10000)
System Admin
LogReg
Handom
Coooa
2.00
1.50
1.2
3.0
20
Figure 4: Left panels: Pareto frontiers for the model classes LSTM, GRU, TCN and FullyConv for the cross entropy test error on the y-axis and wall-clock training time on the x-axis for the domains MigroGrid, TrafficGrid and System admin. The sizes selected are represented with red markers and correspond to the cells highlighted Table Right pancls: cross entropy test crror over wall-clock training time for the sizes of the network selected for each model class. The dashed line represents the cross entropy error of the baseline random classifier.
larger network size, leading to the choice of size 15, 000 for that class. When comparing the training curves of the selected models, we observe similar learning speed and final accuracy across all models. Notably, the logistic regression model achieves comparable performance to the other more complex learning models. One explanation for this result is that, despite the complex nature of the realistic microgrid scenario, the corre- sponding influence learning task is relatively simple: low-dimensional forecasting problem with influence sources weakly dependent on the local history. Even scaling the microgrid to include many more units would still result in a low dimensional and simple influence learning task. This situation is representative of several complex realistic scenarios where the local model is well-decoupled from the rest. the system, with only a few external influence sources weakly affecting the local dynamics.
A different situation arises in the traffic grid and the system admin domains which induce more complex influence learning tasks in terms of dimensionality and problem horizon, respectively. In these domains, the
14

---

Published in Transactions on Machine Learning Research (03/2025)
2.2
LSTM (3000)
Logfing
2.0
1.8
1.6
1.4
150
200
400
450 500
Figure 5: Comparison of cross entropy test error across different time steps in the system administration domain between an LSTM model with 1000 parameters (blue) and the baseline logistic regression model (red). The curves show that the error the logistic regression increases over time steps, while the L.TTM maintains a stable error.
linear regression and TCN models show their limitations. In the right panels of Figure the red training curves show that the logistic regression fails achieve good accuracy levels. Although the green training curves for the TCN model have not yet reached convergence, the learning is much slower compared to the FullyConv, LSTM and GRU models. However, even for these problems characterized by higher dimension longer horizons, relatively small architectures for LSTM, GRU and FullyConv models (between 1000 and 10000 paramcters) achieve good performance.
Error over time steps. To further analyze the limitation of LogReg model for long horizon problems, we plot in Figure 5 the test error per time step of the logistic regression compared to LSTM for the system admin domain. While the LSTM error remains relatively constant, the error for logistic regression increases over the time steps. This suggests that the logistic regression's 's ability to represent the influence diminishes with longer horizons. One possible explanation that lincar models struggle accurately capture increasingly nonlinear relationships between local variables and influence sources the problem horizon becomes longer.
4.2 Generalization beyond the training horizon
First, we look at the deployment errors for specific training horizons to show that it is possible to generalize the influence approximations over long deployment horizons. Then, we analyze the effect of the training horizon on the accuracy of the long-term approximations by looking the deployment error and training curves for different training horizons. Finally, compare deployment, test and tail-test error for different training horizons to identify which error serves reliable proxy for the deployment error.
Table 2 shows the average deployment errors Edeploy with corresponding standard errors for specific choices of Parain estimated over ten itcrations of the entire experiment. The performance of the models in the three scenarios is compared with the error a random classifier. The choice of a suitable training horizon is
learning model
scenario
LSTTM
FullyCom Random
hersin
hdeploy
GC4
0.002 + 0.002
0.027 + 0.026
1.38
200
GC11
0.002 = 0.001
0.025 0.01
1.38
22
200
TG
3.68 + 0.06
3.98 + 0.07
8.32
30
500
Table 2: Mean and standard error of the deployment error over ten iterations.
15

---

Published in Transactions on Machine Learning Research (03/2025)
domain-dependent: in the grab chair scenario with 4 agents, Muain training steps are sufficient to get deployment error close to 0 while for 11 agents, the models require longer training horizon Itaain 22. Such choices have been driven by the idea that the training horizons need to be longer than the mixing times, which corresponds to fmix = 3 and mix = 10, respectively. The results show that a training horizon slightly longer than the mixing time ensures cross entropy errors close to zero over a much longer deployment horizon. In other words, few training time steps of experience after mixing are sufficient for the models generalize the deterministic stationary influence over 200 deployment time steps. For the traffic grid domain, the results are less straightforward to interpret. In fact, the cross entropy error depends on the entropy of the target influence sources distributions, which are unknown. However, for htrain = 30 the average errors over 500 deployment time steps in Table 2 are significantly lower than the random classifier error. Also, they are quite close to the errors computed over a much shorter test horizon and reported in Table [b, This leads to conclude that 30 training steps are sufficient to learn good long-term influence approximations. In summary, for every scenario considered, we found a sufficient number of training stops htrain to learn influence approximations for much longer deployment horizon hdeploy ensuring small deployment error. To gain concrete and visual understanding of the predictions of the models reported in Table 2 we refer to the videos in Appendix
6C4
GC11
18TM
L5TH
LSTN
PullCoo
NalyCark
FulyCarv
30
Cola C1100
25-
10-
LSTM
Easplyy
LSTM
Caucky
LSTH
40
10
Traningg Morizon
Figure 6: Top pancls: deployment error over training horizons for the LTSM and FullyConv network. Bottom panels: deployment, test and tail-test error for increasing training horizon for the LTSM model. The vertical dashed line marks the mixing time the system, if known. Note that the apparent difference between the blue curves representing the LSTM error in the top and bottom plots is a result of the different scales of the other variables plotted or of different values of the training horizon considerod.
Optimal training horizon. The top panels of 6 the deployment errors deploy of the LSTM and FullyConv models computed for increasing training horizons by И for the three scenarios. Notably, for GC4 and GC11 the deployment errors drop significantly when htrain 2 3 and Atrain 2 10, respectively (see top left and central panels). Clearly, this drop correlates with the mixing times of the systems that correspond to fmix = for GC4 and fmix = 10 for GC11. Essentially, both learning models exhibit improved accuracy as soon as some experience of the steady state is recorded in the training trajectorics. However, achieve error close to zero a few additional training time steps are needed, preciscly huain for CG4 and Itrain 2 20 for CG11.
16

---

Published in Transactions on Machine Learning Research (03/2025)
To better understand the impact of the mixing time on the model learning, we compare the deployment and test errors progression over training epochs of the LSTM for different training horizons on the CG4 domain in Figure 7 Before the mixing time for Arain = 2, no experience of the steady state is stored in the training set. Thus, all the predictions are solely based on the inffuence experienced in the first 2 time steps. As a consequencee, Figure 7(a) shows an overfitting effect: the lest, error improves over the training epochs and tends quickly to zero as the approximations are very accurate for the first time steps; on the other hand, the deployment error becomes larger for increasing training epochs. After mixing, for Merain = the deployment error has significant decrease. Yet Figure 7(b) shows the same overfitting effect for more training epochs. The reason is that the experience of stantionary influence in the training sequences is still very limited and not sufficient for the learning model to generalize it over the deployment horizon. For Mrain = 6, Figure 7(c) shows that the deployment error quickly lends to zero logether with the test error. Moreover, no significant
differences are detected for additional training time steps (see Figure [(d] for hwssin = 8). This motivates the choice of optimal training horizon as i = Similarly for GCI1, the training curves in Figure g Appendix [ show that the deployment error starts to decrease for Itrain 12 > 10 = bmix to tend to zero when the training horizon is around 22 time steps. More training time steps do not improve significantly the predictions. Thus, select as training horizon Mirain 22. In the traffic grid domain, as illustrated in the top right panel of Figure G the deployment error of both learning models decreases substantially between 10 and 20 time steps. The model performance improves for training horizon ranging from 20 and 30 time steps. However, increasing further htaan does not lead to performance improvement. This is confirmed by the training curves collected in Figure g in Appendix A Although the mixing time of the traffic grid is unknown, these results suggest that the system reaches a stationary influence between 10 and 30 time steps and mix should be chosen within those values.
Deployment error estimate. The bottom panels of Figure G depict the deployment, test and test-tail errors of the LSTM model for different training horizons. The results from all three scenarios confirm uniformly the theoretical intuition presented in Section 3.3 When berain 2 mixx: the test-tail error (green) reflects the deployment error (blue) trend much better than the Lest error (orange). Specifically in the two grab a chair scenarios, observe that after the mixing time, marked by the dashed vertical line, the test-tail error presents the same descending trend of the deployment error. Contrarily, the test error is always very close to zero, indicating that the model learns very well the influence over the training horizon. For the traffic domain, Figure [ suggests that after approximately 15 time steps the system reaches the equilibrium. Thus, the deployment error and the test-tail error become very close and decrease quickly to converge to the test error. This analysis shows that the test-tail error provides a good estimate of the deployment error and can therefore be used to assess the performance of the learning model using the training set and to choose the optimal training horizon hrrain the smallest horizon that yields low test-tail error.
4.3 Observations, limitations and future work
Our study leads to a number of key observations, but there are also limits on how far the conclusions can reach. Here, we provide a balanced overview of both.
Key observations. The results of our empirical study show that, even in many complex real-life situations, the task of learning influence can often be quite simple and computationally manageable. This was especially evident in the case of microgrids, where a logistic regression model performs just as well as more advancod models. The complexity of the influence learning problem is affected by the dimensionality and horizon of the forecasting task. For high dimensional or long horizon problems, linear models or vanilla temporal convolutional networks typically fail to provide accurate influence approximations. However, relatively small recurrent and fully convolutional networks have proven to learn good approximations with efficient training times in all domains and thus be the most suitable models for the influence learning (ask. Additionally, the experiments show that suitable learning models can generalize the influence from a short horizon training trajectorics well beyond the training horizon. In general, a good choice for such training horizon depends mostly on the mixing time of the system and it is independent of the learning model. In essence, an optimal training horizon corresponds to the mixing time plus few additional time steps to collect enough experience
17

---

Published in Transactions on Machine Learning Research (03/2025)
dasty eatt
1.4
Farply Poot
1.2
1.0
Oross Entropy
Crass Entropy
0.6
0.4
0.2
12 Epochs
Epochs
(a) Perain = 2
(b) herain 4
1.4
1.2
1.2
1.0
1.0
Cress Entropy
Cross Entropy
0.8
0.8
0.2
0.0
0.0
2
10
12
14
16
18
20
22
10
12
14
16
18
20
22
24
Epochs
(c) Itrain
(d) Itrain
Figure 7: Cross entropy deployment and test errors over increasing training epochs of the LSTM model in the GC4 scenario, computed for different training horizon hamin- The dashed line represents the loss of the baseline random classifier.
of the stationary influence in the training set. The test-tail provides a better estimate the deployment error and therefore can be used lo assess the quality of the learning models and select an optimal horizon.
Environment Scalability. First, while this study has used relatively complex benchmark problems (mi- crogrid with 100s of agents, and traffic control with more than 100 state variables), did not exhaustively investigate the scalability of the environments. Scalability of influence prediction is mostly dependent the number of influence sources that need to be predicted. As long this is limited, the learning and use of influence can scale very well, as for instance demonstrated by Suau et al. (2022a) that employ influenco- abst raction, in parallel, to a trallic control task with 100 intersections. Furthermore, even though the IBA framework can be naturally extended to continuous state variables, our study is limited to discrete state variables, and does not address the more realistic challenge of learning continuous influence sources. Lastly, although the chosen domains are inspired by realistic scenarios, they still represent simplifications of actual real-world cases.
18

---

Published in Transactions on Machine Learning Research (03/2025)
Specification of local model and d-sets. Additionally, our approach relies on the manual specification of local states and d-sets, focusing on the feasibility of influence prediction under these conditions. As discussed in Section 2.5 there are trade-offs between the predictability of influence sources and the size of the local model. Future research could explore automated methods for identifying good decomposition choices to optimize the overall performance.
Transformers and other learning models. We note that we have not exhaustively compared all pos- sible sequence models. In particular, transformers, (discrete) diffusion models, and structured state space models could be of interest also to scale to higher dimensional influence prediction tasks, While such recent methods have pushed the limits in the large data regime, IBA is motivated by the idea that performing many simulations with a global simulator could be too expensive. Therefore our focus has been on exploring the potential of learning from limited amounts of data (N = 500 trajectories).. Our study shows that exploring whether, for instance, (small) transformers could be competitive in this regime, or whether they could pro- vide further benefits when problems become even more complex, is promising. However, these explorations are left for future research.
Impact for control. In this paper, we explored the leasibility of learning accurate influence predictors, measured by CE loss. However, the full IBA framework intends to support decision making: in the end we care about the impact of using the resulting models in, c.g., planning, predictive control or RL. Intuitively the more accurate the influence predictors and hence I-ALM, the better results we would expect. Indeed, in prior work Congeduti et al. (202T) did investigate this question, showing theoretically that the CE error leads an upper bound on value loss and empirically that there positive correlation between CE and value loss. Nevertheless, the relation between CE error and value loss is complex, and this alignment may not always hold (Farahmand et al.] 2017 Lambert et al.] 2020). Specifically, using additional in information about rewards or values can lead to tighter upper bounds and thus faster learning (Farahmand et al.) 2017). which can have a significant impact in finite sample regimes (Lambert et al. 2020). Therefore, investigating these potential mismatches and adapting influence models to dynamic, interactive environments remains an important area. for future exploration.
5 Conclusions
In this paper we investigated learning models and techniques for the influence learning task in realistic scenarios. We run an extensive empirical investigation of the performance of diffcrent learning models in a variety of domains. We conclude that complex scenarios may still induce manageable influence learning task. We showed that relatively small recurrent models can achieve the same performance levels state-of-the-art fully convolutional neural networks. Moreover, we explored how to leverage learning models to build local simulators for long horizons using short training trajectories. In particular, we showed that there exists a training horizon which is sufficient. to learn good influence approximations for long (or infinite) horizon problems and how to use suitable error metrics to scarch for such horizon.
Approximate IBA offers promising direction for managing real-world decision-makke making problems. However, inaccuracies of influence approximators in high-stakes domains, such as traffic management, energy distri- bution, and critical infrastructure may lead to unintended consequences including safety risks, economic inefficiencies or biases in decision-making. While this also holds when applying planning and RL techniques without IBA, IBA could potentially introduce further approximation errors. We recommend that future re- scarchers, developers, and practitioners exploring IBA in applications prioritize rigorous validation address these ethical concerns.
References
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, arXiv preprint arXit:1803.0133 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. IEFE transactions on pallern analysis and machine intelligence, 35(8):1798 1128, 2013.
19

---

Published in Transactions on Machine Learning Research (03/2025)
Christopher Bishop. Pattern recognition and machine learning. Springer, 2006.
Matthew Botvinick, Sam Ritter, Jane X Wang. Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends in cognitive sciences, 2019.
Craig Boutilier, Thomas Dean, and Steve Hanks. Decosion- planning: Structural assumptions and computational leverage. Journal of Artificial Intelligence Research (JAIR), 1999.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statis- tical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
Elena Congeduti, Alexander Mey, and Frans A. Olichock. Loss bounds for approximate influence-based ab- straction. In Proceedings of the International Conference on Aulonomous Agents and MulliAgent Systems (AAMAS), 2021.
George Coulouris, Jean Dollimore, and Tim Kindberg. Distribuled systems: concepts and design. Addison- Wesley, 2001.
Christoph Dann and Emma Brunskill. In Advonces in Neural Information Processing Systems (NeurIPS). 2015.
George Dimitrakopoulos and Panagiotis Demestichas. Intelligent. transportation systems. IEEE Vehicular Technology Magazine, 2010.
Amir-massoud Farahmand, Andre Barreto, and Daniel Nikovski. Valuo-aware loss function for model-basod reinforcement learning. In Artificial Intelligence and Statistics, pp. 1486 1494. PMLR, 2017.
John Cristian Borges Gamboa. Deep learning for time-scries analysis. arXiv preprint arXiu: arXiv:1701.0188 701.001877 2017.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL https::/openreriiv..eem wttprarrrrrre.ooeooooooaaaaaaaaa
Eric A. Hansen and Zhengzhu Feng. Dynamic programming for POMDPs using factored state representa- tion. In Proceedings of the International Conference on Artificial Intelligence Planning Systems (AIPS), 2000.
Eric A Hansen, Daniel S Bernstein, and Shlomo Zilberstein. Dynamic programming for partially observable stochastic games. In Proceedings of the AAAI Conference on Artificial Intelligence, 2004.
Jinke He, Miguel Suau de Castro, and Frans Oliehoek. alluencce-augmented online planning for complex environments. Advances in Newral Information Processing Systems, 2020.
Jinke He, Miguel Suau, Hendrik Baier, Michael Kaisers, and Frans Olichock. Online planning in POMDPs with self-improving simulators. In Proceedings of the International Joint Conference on Artificial Intelli- gence, (IJCAI), 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Newral computation, 1997.
Tao Hong and Shu Fan. Probabilistic electric load forecasting: A tutorial review. International Journal of Forecasling, 2016.
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller, Deep learning for time series classification: review. Data mining and knowledge discovery, 2019.
20

---

Published in Transactions on Machine Learning Research (03/2025)
Pertti Järventausta, Sami Repo, Antti Rautiainen, and Jarmo Partanen. Smart grid power system control in distributed generation environment. Annual Reviews in Control, 2010.
Leslie Pack Kaelbling, Michael Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 1998.
Sham Machandranath Kakade. On the sample complerity of reinforcement learning. PhD thesis, 2003.
Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Shun Chen. LSTM fully convolutional networks for time series classification. IEEE access, 2017.
Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 2002.
Yaser Keneshloo, Tian Shi, Naren Ramakrishnan, and Chandan K Reddy. Deep reinforcement learning for sequence-to--sequenee models. IEEE transactions on neural networks and learning systems, 2019.
Daphane Koller. Probabilistic graphical models: Principles and techniques, 2009.
Elizaveta Kuznetsova, Yan-Fu Li, Carlos Ruiz, Enrico Zio, Graham Ault, and Keith Bell. Reinforcement learning for microgrid energy management. Energy, 2013.
Nathan Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra. Objective mismatch in model-based reinforcement learning. Proceedings of Machine Learning Research vol, 120:1-15, 2020.
Colin Lea, Michael D. Flynn, Rene Vidal, Austin Reiter, and Gregory D. Hager. Temporal convolutional networks for action segmentation and detection. In Proceedings of the IEEE Conference on Compuler Vision and Pattern Recognition, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Fu-Dong Li, Min Wu, Yong He, and Xin Chen. Optimal control in microgrid using multi-agent reinforcement learning. ISA transactions, 2012.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015.
Kaisa Miettinen. Nonlinear multiobjective optimization. Springer Science & Business Media, 1999.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 2015.
Thomas E Morton and William E Wecker. Discounting, ergodicity and convergence for Markov decision processes. Management Science, 1977.
Ranjit Nair, Milind Tambe, Makoto Yokoo, David Pynadath, and Stacy Marsella. Taming decentralized POMDPs: Towards efficient policy computation for multiagent settings. In Proceedings of the International Joint Conference on Arlificial Intelligence, (IJCAT), 2003.
Kingsley Nweye, Siva Sankaranarayanan, and Zoltan Nagy. Merlin: Multi-agent offline and transfer learning for occupant-centric operation of grid-interactive communities. Applied Energy, 346, 2023. ISSN 0306-2619.
Frans Olichock, Stefan Witwicki, and Leslie Kaelbling. Influence-based abstraction for multiagent systems. Proceedings of the AAAI Conference on Artificial Intelligence, 2012.
Frans Olichock, Stefan Witwicki, and Leslie Kaelbling. A sufficient statistic for influence in structured multiagent. environments. Journal of Artificial Intelligence Research (JAIR), 2021.
21

---

Published in Transactions on Machine Learning Research (03/2025)
Frans A Olichoek, Christopher Amato, et al. Best response bayesian reinforcement learning for multiagent systems with state uncertainty. In AAMAS Workshop on Mulliagent Sequential Decision Making Under Uncertainty, 2014.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Sanjoy Dasgupla and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 Proceedings of Machine Learning Research, Atlanta, Georgia, USA, 17- 19 Jun 2013. PMLR. URL https::/procoed/ee httpp/pooddaaa.....oppe/e/aaaaap
mlr. wttpp//////pasaanai..ta.e
html
Karl Pertsch, Oleh Rybkin, Frederik Ebert, Shenghao Zhou, Dinesh Jayaraman, Chelsea Fitn, and Sergey Levine. Long-horizon visual planning with goal-conditioned hierarchical predictors. Advances in Neural Information. Processing Systems (NeurIPS), 2020.
Pascal Poupart and Craig Boutilier. VDCBPI: an approximate scalable algorithm for large POMDPs. Advances in Newral Information Processing Systems (NeurIPS), 2004.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International conference on machine learning. PMLR, 2018.
Roberto Rocchetta, Alexander Mey, and Frans A. Oliehoek. A survey on scenario theory, complexity, and compression-bassed learning and generalization. IEEE Transactions on Neural Networks and Learning Systems, 2023. doi: 10.1 ALLLL0/TNNNS.223.3333000
Ahmad Shamshad, MA Bawadi, WMA Wan Hussin, Taksiah Majid, and SAM Samusi. First and second order Markov chain models for synthetic generation of wind speed time series. Energy, 2005.
Yoav Shoham and Kevin Leylon-Brown, Mulliagent systems: Algoritlemie, gume-theorelic, and logical foun- dations. Cambridge University Press, 2008
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Dricssche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 2016.
Anthony Simeonov, Yilun Du, Beomjoon Kim, Francois R. Hogan, Joshua Tenenbaum, Pulkit Agrawal, and Alberto Rodriguez. A long horizon planning framework for manipulating rigid pointcloud objects. ar preprint crXim:2011.08177 2020.
Elisa Skoplaki and John A. Palyvos. On the temperature dependence of photovoltaic module electrical performance: A review of efficiency /power correlations. Solar energy, 2009.
Miguel Suau, Jinke He, Mustafa Mert Çelikok, Matthijs Spaan, and Frans Olichock. Distributed influence- augmented local simulators for parallel MARL in large networked systems. Advances in Neural Information Processing Systems (NeurIPS), 2022a.
Miguel Suan, Jinke He, Elena Congeduti, Rolf Starre, Aleksander Czechowski, and Frans Olichock. Influence- aware memory architectures for deep reinforcement learning POMDPs. Neurul Computing and Appli- cations, 2022b.
Miguel Suau, Jinke He, Matthijs TJ Spaan, and Frans Oliehoek. Influenco-augmeetted local simulators: A scalable solution for fast deep RL in large networked systems. In International Conference on Machine Learning. PMLR, 2022c.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
22

---

Published in Transactions on Machine Learning Research (03/2025)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Aduances in Newral Information Processing Systems (NeurIPS), 2017.
José R. Vázquez- Canteli, Sourav Dey, Gregor P.
Henze, and Zoltán Nagy. Citylearn: Standardizing re- scarch in multi-agent reinforcement learning for demand response and urban energy management. ArXiu, abs/2012.1050x 2020. URL
wrg/CorpusID............... /t/C/oopssI:::23333333333
John G Vlachogiannis and Nikos D. Hatziargyriou. Reinforcement learning for reactive power control. JEEE Iransactions on power systems, 2004.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey, arXiv preprint 2ix: 07125, 2022.
23

---

Published in Transactions on Machine Learning Research (03/2025)
A Additional results
Figure and Figure 9 show the test and deployment errors of an LSTM learning model for the influence over the dilferent training horizons in GC11 and traffic grid domain, respeclively. See experiments in Section 3.3
Aerain 6
(b) Atrain
12
Ittaii 22
(d) 1train 28
Figure 8: Deployment and test errors for LSTM in GC11 scenario. The dashed line represent the baseline accuracy the random classifier.
(a) Aarain 10
(b) Atrain
20
(c) Ierain
30
(d) Ittrin 40
Figure 9: Deployment and test errors for LSTM in traffic grid domain. The dashed line represent the baseline accuracy of the random classifier.
24

---

Published in Transactions on Machine Learning Research (03/2025)
B Experimental domains
Microgrid. In this case study, inspired by previous works [Nweye et al. 2023 Li ct al. 2012 Vlachogiannis & Hatziargyriou, 2004), we model realistic interactions in an energy district (a microgrid) as a multi-agent problem. hundred autonomous agents, referred to as prosumer unils, manage the energy flexibility of the microgrid. Each agent decides whether to discharge stored energy or trade based on local demand and supply with the goal to minimize costs while ensuring energy balance and independence of the microgrid from the external power grid. Solar and wind energy sources utilized. The problem focuses unit within lattice network, with neighboring actions influencing decisions. The task is predict influence sources based on historical actions. The local states vector is given by, $local (PRES, Pa, SOC), where PRES is the renewable power production, Pa the stochastic power demand assumed normally distributed as in Hong & Fan (2016), and SOC is the state of the charge of the battery. The PRES includes solar and wind energy. The former modeled using hourly solar radiation data in https boctps://openeeet http://eeeeaeeemaappppppaaa org/api/ solar-radiation and the photovoltaic power generation model introduced by (Skoplaki & Palyvos (2009). The wind power generation is calculated by transforming the kinetic energy of the wind speed modelled via the Markov chain model in (Shamshad et al.) 2005) and assuming linear relationships with the power produced as in Kuznetsova et al. (2013). The energy produced by agent may be used to meet the demand Pa. or stored in battery to increase the SOC. At any time step A, (one hour), the agents may decide to discharge the stored energy to meet the demand Pa, or trade energy with neighboring units. When buy and sell orders match, the power from the batteries is exchanged at a small cost cost/revenue for the buying and selling agents respectively. After the local trading is cleared, to satisfy the power balance single units, every agent is forced buy residual power an external power distribution gird. A cost Cext is assigned
single
from external grid
Renewable
to/from microgrid
Demand
(a) Unit controlled by a single agent in the microgrid. The lo- variables observed by the agent include the state of the charge, the power produced by renewable sources and the power demand.
(b) Multi-agent low-voltage grid. For each unit lattice, state of charge (in percentage) is represented by the gray scale. Directed edges represent power exchanged between prosumers (agents).
Figure 10: Microgrid.
25

---

Published in Transactions on Machine Learning Research (03/2025)
the energy not supplied ENS = (Pa- Pdeployed): As, where Pdeployed corresponds to the sum of the renewable power deployed and the power discharged from the battery. The cost Cext for buying from the external grid is much higher than the fixed operational costs of internal trade Cint- A schematic representation of a MG unit is depicted in Figure [0(a] The individual reward for each agent is modeled as the sum of the cost/ tist/incme for the internal trade (if any) and the negative costs of buying the energy not supplied from the external grid (if any) r = +im trade CextENN Thus, the team of agents share the common objective to manage local resources
minimize the electricity costs constrained to satisfying the energy balance, generation limits and storage capacity. We take the perspective of single unit in a lattice network highlighted in red in Figure 10(b) The initial distribution of the battery is uniformly sampled at random and we consider h = 40 hours as the horizon of the problem. We assume that all the other agents act by storing or trying to buy power when the storage is scarce and discharging or trying to sell when power is abundant. Note that besides the problem size can be arbitrarily large, the influence experienced by the local agent only directly depends on the neighboring nodes the network. Preciscly, the only relevant information on the external portion of the system that an agent needs is whether the neighboring north, west, south and east agents will decide to sell buy power. Besides the distributions of inftuence sources Sare (ax, aw, og, ag) are affected (indirectly) by all the agents in the microgrid, the history of local actions provides sufficient statistics to predict the influence sources. The resulting problem consists of finding function approximator for T (axs, aws a's, dg ao a......111)
Traffic grid. In this implementation of a traffic network as described in Section and Section 2 we simulate the vehicle traffic in a 9 intersections grid, schematically represented in Figure The sensors of each traffic light capture the vehicles in the 5 local grid at each interscction. The local model is represented a red square for the selected protagonist. agent. The other traffic lights employ hand-coded policies that prioritize lanes with higher car volumes. At time t = 0 the grid is initially empty, i.e. the initial stale encodes no cars in the network. At any time step, a vehicle will enter the network with certain probability. The horizon is set to = 100. The state of the environment is represented by binary state variables detecting the presence/ /asence/bbsecce /absence of a car in a point of the traffic grid. The goal of the agent
minimize the total number of vehicles waiting the local intersection. That is, the reward corresponds to the negative number of cars in the local model. To act optimally, the local agent needs to predict if there will be incoming cars from the north end Sat and the east end se-. Moreover, the local dynamics is affected by traffic congestion at intersection 2 and 4. In fact, traffic jams can prevent vehicles to move out of the local model from the west and south ends. For this reason, the state variables for the outgoing ends and the actions of agents 2 and are included in the set of influence sources. Thus, addition to the factors
3
<<<
2
1
4
Figure 11: Traffic grid. The local model is delimited by the red square. The blue triangles represent the vehicles and the green bars the traffic lights.
26

---

Published in Transactions on Machine Learning Research (03/2025)
encoding cars inflows, the inffuence sources encompass 4 variables for the west outflow Swe, 4 variables for the south outflow 8ug. the action 02 and d4, that is Serc = (8nt, Set, Swi,, 2z 04). The local information necessary to predict the influence sources includes the entire collection of local variables and actions. The influence that the agent needs to predict is therefore I(5/2 S
s Sw., s az of local af...., 2-- st local).
'aulty mach working machine admin
chair agent
11
20
12
19
15
16
Figure 12: Grab a chair.
Figure 13: System admin.
System admin. We use a multiagent version of the System administrator domain from Pouparr & Boutilier (2004). A team of system administrators are responsible for the upkeep of a network of machines. Each node has probability of failing at any time step which increases when a neighboring machine in the network is down. Each agent only observes the status of the machines in its proximity. Consequently it may decide to intervene by trying to reboot the system of one of these nodes. With a certain probability, the process will succeed, resulting in a working node at the next time step. When more than one agent decides to reboot one machine, the process has full rate of success. The goal of the admins team is to secure the highest number of working machines. Precisely, any agent receives a penalty for each faulty machine which lies under its control. We consider a network of = 20 machines organized depicted in Figure 13 Each admin agent is responsible for the maintenance of two neighboring nodes whose states, denoted by 2i, F1+1, be fully observed. We take the perspective of single administrator, for instance agent. 1 in Figure 13 whose local model includes only the states (faully or working) of the two neighboring machines Slocal (21,22) and its action a1. The problem horizon is set to h = 500 time steps, and initially a random state is sampled for each machine. To act optimally, agent 1 needs to know if agents 2 and 20 will decide to reboot one of the two machines over which they share the control. Also, it needs to reason about the neighboring machine status I3, I20 as they may contribute to the higher chances to turn down the machines in its local model. Then, according Section 2 the sources of influence correspond to 8are (I3, (23, 220, 02 a20). The local information disposal of the agent 1 to predict the sources of influence consists of the entire collection of local variables, that is, (21, F2,
a ring configuration as
the influence formalism introduced in the
x2, 01). Thus, the influence problem consists of finding an approximation for the distribution
I (ans, 2go aly, a/20 af. 22 of..... t-1 ,4,,2)))
Grab chair. In this simplified version of the SA problem, introduced by He et al. (0200) N agents disposed in a ring fashion decide at every time step lo grab the chair on their left or right side, as shown in Figure [2] They obtain the chair and thus get the reward only the neighboring agent has not targeted that chair too. After taking an action, cach agent only observes whether it managed to grab the chair, ignoring the action of the neighboring agents. The local agent, numbered by 1 and depicted in red in Figure I2 has access to other information rather than its own actions and rewards which form the local model. The
27

---

Published in Transactions on Machine Learning Research (03/2025)
horizon is set to h 200 and initially every agent chooses deterministically the chair on its right side. After that, all the non-local agents act by copying the previous action of the following agent in counterclockwise order. For the decision making problem of the local agent, the only information required to act optimally consists of the decisions its neighboring agent and N as they directly inftuence the possibility to secure chair. Contrarily, the other agents only allect the local model indirectly. Therefore, the local agent needs predict the influence sources corresponding the actions s (az, alx) given the local information of the d-set dge (di....,00 aaf.....af-)) 1) Therefore the influence to predict corresponds to ( loylal...., "f
C Visualization of the predictions
Short videos showcasing the influence predictions for the traffic grid and grab chair are available here In the traffic grid video, the left side displays the global model simulator, while the right side shows the local model alongside the influence source predictions of an LSTM trained over 30 time steps, achieving cross entropy test error of approximately 3.7 (corresponding to TG in Table 2 The predicted probability distribution over the binary variables, which represent the presence of a car, is visualized using the color intensity: darker colors indicate predictions closer to 1. In this scenario most of the influence sources, precisely those corresponding to the traffic outslows from the local model, present a predominantly deterministie behavior: their values are almost completely determined by the local variables. This allows get a visual intuition of the quality of the predictions that seem indeed pretty accurate. In the grab chair video, visualize the predictions of an LSTM trained over 6 steps, with the blue markers together with the global simulator. Aligned with the cross entropy lest. error that is very close to zero (see GC4 in Table 2 the predictions appeare to match perfectly the influence source deterministic values.
D Implementation details
Domain
h
# Agents
Policies
bo uniforın zeros
# #Influence Sources
D-set dimension
Microgrid
40
100
ranges priority
Traffic grid
100
9
System admin
500
20
mixed
uniform
3
Table 3: Setting of the scenarios and features of the inffuence learning tasks.
Optimization
Domain
Sample size
Batch size
#Epochs
Alg
Loss
LR Decay
Valid
Microgrid
500
100
15
Adam
Entropy
Linear
Split90%
Traffic grid
500
100
20
Adam Entropy
Linear
Split90%
System admin
500
100
20
Adam Entropy
Linear
Split90%
Table 4: Optimization hyperparameters for the learning models.
28

---

Published in Transactions on Machine Learning Research (03/2025)
Architecture
Models
#Layers
# #Units
Kernel
#Params
Activate
Regularize
(size S000)
LSTM
88
Tanh
None
GRU
1
78
Tanh
None
TCN
2
2,2
8
100
ReLU
None
FullyConv
2
8
52
ReLU
Dropout
(size 1000)
LSTM
1
13
1056
Tanh Tanh
None
GRU
1
954
None
TCN
4
6,6,6,6
1048
ReLU
None
FullyConv
4
6,6
8
1084
ReLU
Dropout
(size 15000)
LSTM
1
56
14128
Tanh Tanh
None
GRU
1
64
13904
None
TCN
5
20,20,20,20,20
8
13396
ReLU
None
FullyConv
8
15,15,15,15
8
13246
ReLU
Dropout
LogReg
13104
None
None
Table 5: Microgrid. Architectures of the learning models.
Architecture
Models
# Layerss
Units
Kernel
# #Params
Activate
Regularize
(size < 200
LSTM
176
Tanh
None
GRU
150
Tanh
None
TCN
2,2
4
164
ReLU
None
FullyConv
2
4
188
ReLU
Dropout
size 1000)
LSTM
1
960
Tanh
None
GRU
1
11
1014
Tanh
None
TON
4
4,4,4,4
10
976
ReLU
None
FullyConv
4
4,4
10
1032
ReLU
Dropout
size 10000)
LSTM
9936
Tanh
None
GRU
1
49
10020
Tanh
None
TCN
4
16,16,16,16
10
9592
ReLU
None
FullyConv
4
16,16
10
9816
ReLU
Dropout
size 50000)
LSTM
1
104
50360
Tanh
None
GRU
1
120
50064
Tanh
None
TCN
6
[30]x6
10
48624
ReLU
None
FullyConv
6
[30,30,30]
10
49104
ReLU
Dropout
(size 1M)
LogReg
1093200
None
None
Table 6: Traffic grid.
Architectures of the learning models.
29

---

Published in Transactions on Machine Learning Research (03/2025)
Architecture
Models
#Laycrs
# Units
Kernel
# Params
Activate
Regularize
(size S <000)
LSTM
1
80
Tanh
None
GRU
1
66
Tanh
None
TCN
2
2,2
4
68
ReLU
None
FullyConv
2
4
80
ReLU
Dropout
size 1000)
LSTM
1
12
920
Tanh Tanh
None
GRU
1
918
None
TCN
4
6,6,6,6
8
1088
ReLU
None
FullyConv
4
6,6
8
1136
ReLU
Dropout
(size 10000)
LSTM
1
48
10568
Tanh Tanh
None
GRU
1
54
9998
None
TCN
6
[16] x6
8
10856
ReLU
None
FullyConv
6
16,
16,16,16
8
11016
ReLU
Dropout
(size 3M)
LogReg
2.9M
None
None
Table 7: System admin. Architectures of the learning models.
30

---

Published in Transactions on Machine Learning Research (03/2025)
Domain
H
hepplyy
# #Agents
Policics
bo
GC4
(2,...,14 14)
200
copy
Deterministic right chair
GC11
(5,...,30)
200
11
copy Deterministic: right chair
TG
{10,..., 100
500
priority
Deterministic. zero vehicles
Table Setting of the scenarios for long horizon tasks.
Domain
Batch size
#Epochs
LR Init
LR Final
LR Decay
Train size n
Test size m
GC4
10
25
10-2
10-a
linear
500
100
GC11
10
30
10-2
10-9
linear
500
100
TG
10
20
10-2
10-5
linear
500
100
Table 9: Optimization choices for long horizon tasks.
Architecture
Domain
Model
# Layerrs
# Units
Kernel
# #Params
Activate
Regularize
GC4
LSTM
10
564
Tanh
None
FullyConv
4
[6.6]
4
544
ReLU
Dropout
GC11
LSTM
1
32
4612
Tanh
None
PullyConv
8
[10,10,10,10
4484
ReLU
Dropout
TG
LSTM
32
2136
Tanh
None
FullyConv
4
[8,8]
1944
ReLU
Dropout
Table 10: Architectures of the learning models for long horizon tasks.
31


Reviews:
[{"review_id": "6Tu6bFTGBv", "reviewer": "Reviewer_A5tQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents a well-executed empirical study on influence-based abstraction (IBA), offering valuable insights into how influence approximations can be effectively learned and applied to complex systems. A major strength of the paper lies in its comprehensive set of experiments conducted across diverse real-world-inspired domains, such as traffic control, power grids, and system administration, demonstrating that influence learning is often computationally manageable even in large-scale environments. The findings provide practical and actionable conclusions, showing that small-scale models like LSTM and TCN can achieve accurate approximations without the need for complex architectures, making the approach suitable for resource-constrained applications. The analysis of training horizon selection and its dependence on system mixing time presents a useful heuristic for optimizing learning pipelines, balancing computational cost and accuracy effectively.\n\nHowever, the study has several limitations that could be addressed to enhance its impact. The scope of the experiments is limited to relatively small-scale problems, leaving questions about the scalability of the approach to larger and more complex environments. Additionally, the paper focuses solely on discrete state variables, which restricts its applicability to many real-world systems that involve continuous state spaces. The reliance on older network architectures, such as LSTM and TCN, rather than state-of-the-art models like Transformers, may limit the direct applicability of the findings to modern AI systems, where Transformers are increasingly favored for their superior long-range dependency modeling and scalability. Furthermore, while the paper provides valuable guidance, it would benefit from a broader discussion on how the proposed methods compare with existing work in the field and whether similar conclusions have already been established in prior research.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the problem of influence-based abstraction (IBA) in complex systems, focusing on accurately approximating the influence of external factors and other agents' policies on local system dynamics. IBA decomposes large, computationally expensive systems into smaller local models by abstracting away non-essential factors while retaining key influences, enabling more efficient reinforcement learning (RL) and planning. The authors conduct an empirical evaluation of existing learning methods, such as LSTM, GRU, and Temporal Convolutional Networks (TCN), across real-world-inspired domains like traffic control, power grids, and system administration tasks and game like grab a chair. The results of the empirical study show that, even in many complex real-life situations, the task of learning influence can often be quite simple and computationally manageable. This was especially evident in microgrids, where a logistic regression model performs just as well as more advanced models. The complexity of the influence learning problem is affected by the dimensionality and horizon of the forecasting task. For high-dimensional or long-horizon problems, linear models or vanilla temporal convolutional networks typically fail to provide accurate influence approximations. However, relatively small recurrent and fully convolutional networks have proven to learn good approximations with efficient training times in all domains and thus be the most suitable models for the influence learning task. Additionally, the experiments show that suitable learning models can generalize the influence from short-horizon training trajectories well beyond the training horizon. In general, a good choice for such training horizon depends mostly on the mixing time of the system and is independent of the learning model. In essence, an optimal training horizon corresponds to the mixing time plus a few additional time steps to collect enough experience of the stationary influence in the training set. The test-tail provides a better estimate of the deployment error and therefore can be used to assess the quality of the learning models and select an optimal horizon. The paper concludes that IBA is a practical approach for decision-making in complex real-life systems, offering a scalable and computationally efficient solution for RL and planning applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. some figures need a better caption to better clarify what the figures mean and try to tell the reader. For example, figure 5's caption is too limited in its explanations and need to clarify what are the different curves in the figure and what do they try to show. \n2. explanation why the current selection of hyperparameters make sense. For example, why choose LSTM of 1 layer and not more for the traffic grid problem or system admin problem? \n3. try to include more baselines, such as traditional RL based approach, or simpler rule-based approach to show the added value of using machine learning for influence learning. \n4. The concept of the d-set, while introduced in the theoretical framework, lacks sufficient discussion on its practical implications. Clarifying how different choices of d-sets impact influence approximation accuracy would enhance the practical utility of the approach.", "broader_impact_concerns": "The paper does not explicitly include a Broader Impact Statement, and while the topic primarily focuses on influence-based abstraction in reinforcement learning for complex systems, there are potential ethical concerns that should be addressed. One key concern is the applicability of the proposed methods in high-stakes decision-making domains, such as traffic management, energy distribution, and critical infrastructure, where inaccurate influence approximations could lead to unintended consequences, such as safety risks or economic inefficiencies. The authors should discuss potential risks associated with relying on approximate models in these scenarios, emphasizing the importance of rigorous validation before deployment in real-world settings.\n\nAdditionally, the paper should address the potential biases introduced by model approximations, particularly when dealing with incomplete or imbalanced data. If the influence learning models fail to capture certain system dynamics accurately, it could lead to unfair or suboptimal decision-making, disproportionately affecting certain groups or regions in applications like urban planning or resource allocation.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5O3xvEeltQ", "reviewer": "Reviewer_sXHj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe paper is well written, easy to follow and provides a good introduction to the problem. \nThe contribution is clear: this is about testing various models at solving ILT in realistic scenarios.\nThe experimentation is rich enough (many architectures, sizes and environments, 10 seeds), is well described, and answers the asked questions.\n\nWeaknesses\n\nWhile the different models are well compared together, one missing point is an interpretation of how good they are at solving the task.\nFor example, looking at LSTM on the traffic grid (in fig. 3), how do we interpret a cross-entropy loss reaching ~3.5 ? is it good ? Does it significantly help at solving the underlying RL task ?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The Influence Learning Task (ILT), as described in that paper, consists in solving complex multi-agent POMDP by focusing on local dynamics, and estimating the influence (conditional probability) of the past local states and actions over the current local state. \n\nAs the time horizon of the global POMDP is not necessarily known or is supposedly large, the learning is performed over a set of trajectories of limited and fixed horizon (h_train).\n\nThe paper explores the ability of simple models (like LSTM or GRU) with different sizes, as well as the impact of the training horizon, at estimating the influence. Specifically, it focuses on two difficulties: large dimensions of the global POMDP, and large horizon (h_deploy >> h_train).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper could significantly gain from an interpretation of the obtained CE losses. \nThis could be given by some reward scores given a policy based on the different obtained influence estimators. \nOr maybe (if possible) giving some indicator threshold of the sufficient CE levels to solve the task.\n\nOther minor details and questions:\n\n1) I'm not sure to understand why the history of local variables is enough to determine the source variable: doesn't it also depend on the history of all external variables (for ex, past actions of other agents) ?\n\n2) Just above eq.2:  Î(S_src | ...) instead of Î(S_srt | ...)\n\n3) Ch3, paragraph 1: \"humans are generally quite successful in ...\" --> can you provide a reference, or at least an example to support that claim? \n\n4) in Eq 5: IIUC, in order to force the induced Markov chain to be ergodic, the policy of all agents is determined by a single agent: a_i^t = a_1^t-i+1, and this policy is used as an explorative one to get the database of transition to learn the influence. But this let apart all possible transitions where the agents are acting independently? \n\n5) Eq 8: IIUC the left-hand term in the error is the one posing a problem, since it is computed before the dynamics closely reach a stationary distribution. Then, how does the tail error that starts summing at h_k and not at t_mix remove that term? Do we have h_k > t_mix?\n\n6) Fig 6, TG task: How is it possible that LSTM + e_deploy in the top curve can differ from the same in the bottom curve ?", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5er7s4nQSC", "reviewer": "Reviewer_fDcn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nThe paper is overall very straightforward and easy to follow. The problem, its relevance, and the general approach is well motivated and clear. The empirical comparisons are well set up and thorough. As far as I can tell, experiments seem to have satisfactory statistics robustness.\n\n### Weaknesses\nThe main missing component of the work are twofold for me:\n- Baseline comparisons. A significant component of the motivation for the tested algorithms is that they can aggregate temporal information to improve POMDP prediction. To clearly show this, I think the authors should present a simple non-recurrent model as a baseline to show _how_ important history aggregation is.\n- Continual learning and control: A substantial amount of motivation for the problems is control. However, all experiments are evaluated in prediction. It is relatively well established that superior predictive performance does not always automatically translate to better controllers (compare e.g. [1]). The paper seems somewhat incomplete to me without evaluating the models in predictive control or RL. In addition, moving to the control case will require adapting to changing influences as the agent's decision influences that of agents around them. As many standard ML approaches fail to perform robustly in a continual learning setup such as the one encountered in many RL problems, I think this would be a vital test. In this scenario, the assumption of mixing might also break, as agents interacting might create more complex long range dynamics.\n\n### Question\nThe overall takeaway from the paper is slightly unclear to me. Are the used environments important tasks for the community that focuses on these problems? If not, I think it would be important to add benchmarks of the complexity that a practitioner would recognize as hard or important. As I am not a domain expert here, it might well be that this is already the case. In any case, I'd be grateful if the authors commented on this and clarified this in the paper. An empirical claim about learnability is always somewhat contingent on whether a problem is interesting as a representation of a practically relevant problem (in my opinion), or at least a problem that a substantially large enough research community has agreed to care about.\n\nWhile previous reviews apparently called for better formalization, I don't see how the specific setup of an influence setup is used here. I would assume that one of the advantage of using a factorized model is that learning agents can be reused over all components. But by only taking a local view here, I don't really see how the problem differs from a regular POMDP? My question is: what parts of the evaluated approaches are specific to influence learning, and which ones just hold for POMDPs? Does the formalization allow us to make some assumptions on the influence that simplify the learning problem?\n\n### Smaller remarks\nTransformers are not significantly more complex or necessary larger than LSTMs. They are used as components in massively large systems, but training a small transformer is definitely not significantly more difficult than training a small LSTM.\n\n[1] Objective Mismatch in Model-based Reinforcement Learning, Lambert, N. et al., http://proceedings.mlr.press/v120/lambert20a/lambert20a.pdf", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an overview of the problem of learning local influence models to approximate large complex factorized systems. The authors present an introduction to the problem and a thorough overview of different learning approaches for predicting the evolution of local systems based on approximating the unobserved influences of other actors.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please evaluate some non-recurrent baselines.\n\nPlease comment on the questions.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews raised several questions about the scope of the empirical evaluation that were addressed in the rebuttal.  Two out of three reviewers recommended acceptance while the third reviewer remained concerned by the lack of experimentation with continuous and control problems.  Since the current work does make a valuable contribution in the context of inference for large problems with discrete variables, this work still deserves to be published.
Decision: Accept (poster)