Amortized Network Intervention to Steer the Excitatory Point Processes
Authors: Zitao Song
Wendi Ren
Shuang Li
Venue: ICLR 2024 poster (2024)
Keywords: Time series application
Point Process
Amortized Learning
Primary area: general machine learning (i.e., none of the above)

Abstract: Excitatory point processes (i.e., event flows) occurring over dynamic graphs (i.e., evolving topologies) provide a fine-grained model to capture how discrete events may spread over time and space. How to effectively steer the event flows by modifying the dynamic graph structures presents an interesting problem, motivated by curbing the spread of infectious diseases through strategically locking down cities to mitigating traffic congestion via traffic light optimization. To address the intricacies of planning and overcome the high dimensionality inherent to such decision-making problems, we design an Amortized Network Interventions (ANI) framework, allowing for the pooling of optimal policies from history and other contexts while ensuring a permutation equivalent property. This property enables efficient knowledge transfer and sharing across diverse contexts. Each task is solved by an H-step lookahead model-based reinforcement learning, where neural ODEs are introduced to model the dynamics of the excitatory point processes. Instead of simulating rollouts from the dynamics model, we derive an analytical mean-field approximation for the event flows given the dynamics, making the online planning more efficiently solvable. We empirically illustrate that this ANI approach substantially enhances policy learning for unseen dynamics and exhibits promising outcomes in steering event flows through network intervention using synthetic and real COVID datasets.

Published as a conference paper at ICLR 2024
AMORTIZED NETWORK INTERVENTION TO STEER Ex- CITATORY POINT PROCESSES
Zitao Song, Wendi Ren & Shuang Li
The Chinese University of Hong Kong, Shenzhen {z i tatoosonn,, handiren]s e link. cuhk. edu.er, ent..ennniiiinnn Lishuangecukk.edi
ABSTRACT
Excitatory point processes (i.c., event flows) occurring over dynamic graphs (i.e., evolving topologies) provide a fine-grained model to capture how discrete events may spread over time and space. How to effectively steer the event flows by modifying the dynamic graph structures presents interesting problem, motivated by curbing the spread of infectious diseases through strategically locking down cities to mitigating traffic congestion via traffic light optimization. To address the intricacies of planning and overcome the high dimensionality inherent to such decision- making problems, we design an Amortized Network Interventions (ANI) framework, allowing for the pooling of optimal policies from history and other contexts while ensuring a permutaation equivalent property. This property enables efficient knowledge transfer and sharing across diverse contexts. Each task is solved by an H-step lookahead model-based reinforcement learning, where neural ODEs are introduced to model the dynamics of the excitatory point processes. Instead of simulating rollouts from the dynamics model, we derive an analytical mean-field approximation for the event flows given the dynamics, making the online planning more efficiently solvable. We empirically illustrate that this ANI approach substantially enhances policy learning for unseen dynamics and exhibits promising outcomes in steering event flows through network intervention using synthetic and real COVID datasets.
INTRODUCTION
Discrete events, such as the spread of infectious disease or instances of traffic congestion, can be modeled as excitatory temporal point processes over dynamic networks. Guiding the event flows by adjusting the network structures is driven by many societal and environmental problems. For example, in the face of widespread epidemic outbreaks, governments may choose take actions such as temporarily locking down major cities or imposing travel restrictions, with the aim to curb the spread of diseases via cutting off some edges in the disease transmission networks (Salathé & Jones 2010: Sambaturu et al. 2020). Similarly, a common way to alleviate traffic congestion is to optimize traffic light schedules, with the hope of controlling traffic congestion flows by adaptively altering the dynamic traffic networks.
Effectively tackling these problems a large scale is challenging due to a usually huge action space. To curb the spread of epidemic diseases, decision-maker may have to decide whether to suspend each flight across the country for each week. Similarly, to optimize traffic light schedules, city planners need to determine the appropriate actions for each traffic light within the city's 's infrastructure each decision time. Moreover, different regions might face unique constraints. For example, when choosing to suspend flights to control infectious diseases, governments have to balance health concerns with economic impacts and public opinions, which can vary from one region another. Systematically solving this network intervention problem at scale requires innovative solutions.
In this paper, adopt divide-and-congge icy learning problem into subproblems by splitting the original dynamic graph into smaller regions (subgraphs). In this way, the policy learning becomes flexible and manageable for each subproblem, and all these subproblems will be solved a sequential and alternating way.
strategy and propose dividing the original pol-
"Corresponding author

---

Published as a conference paper at ICLR 2024
Specifically, we propose an Amortized Network Interven- tions (ANI) framework, which aims to learn an amortized policy shared by all subproblems via learning invariant representations of optimal policies across subgraphs with diverse dynamics. We aim pool the policy learning from different regions and adapt to unseen new regions. More- over, the representations for the policies should preserve the permutation equivalent properties, which implies that the network intervention policy should adapt the ordering of actions to the ordering of nodes within a graph. We achieve this by designing a bi-contrastive loss function. The permutation equivalent properties encourage encoding the most intrinsic policy information to the representation and enable the adoption of similar policy structures in analogous temporal dynamic systems.
Dlay
Lok
Aevcoppibee
Papesed
Within each subgraph (task), the optimal policy is ob- tained via an H-step lookahead model-based reinforce- ment learning (RI.), which provides versatile formula- tion that can quickly adapt to changing cireumstances and accommodate various local constraints. Classic infectious disease models SIR (Weiss) 2013) or traffic density model LWR (Lighthill & Whitham| 1955 Richards (956) are based ODEs PDEs. These physics- informed models offer simplified yet insightful representations of disease dynamics or traffic patterns. Inspired by these models, use flexible Neural ODE (Chen et al. (018)) to model the dynamics of the networked excitatory point processes. When we perform the online planning, instead of simulating rollouts from the learned Neural ODE model, we further derive an analytical mean-fiell approximation for the event flows given the dynamics. This approximation enables efficient policy learning for each subproblem with less variance.
An illustration depicting the spread of a viral infection, where citics are represented as nodes and roads as edges.
Figure 1:
In summary, we propose a systematic and deployable ANI framework that addresses the challenges of steering event flows within large dynamic graphs. Our main contributions are:
We learn a shared amortized policy cross subgraphs to pool the optimal policy results from other regions and enable quick adaption to new regions.
We design a bi-contrastive loss function to ensure the policy representation has a permutation equivalent property.
We utilize flexible H-step lookahead model-based RL approach within each subgraph and derive an analytical mean-field approximation for event flows given neural ODE models, significantly enhancing the efficiency of policy solving.
We have conducted comprehensive experiments using synthetic traffic congestion data and real- world COVID datasets. The experimental results demonstrate the effectiveness of our approach in adeptly steering excitatory point processes through the control of network dynamics.
PRELIMINARIES
Let's first consider how to obtain the discrete-time networked point processes given raw event data. For infectious disease cases, we can divide the geographical map into cells (e.g., city), each corresponding to a node in the network. The edges are defined as the accessibility from one city to another (as shown in Fig. []. We record new confirmed cases within a cell at each time step, creating discrete-time networked point process. Similarly, traffic congestion incidents as networked point processes. Each lane on the road becomes network node. The traffic lights control the connection of the lanes. At each time step, track the congestion count for each lane.
lane-based approach to model the
Mathmatically, we define temporal network G. (V,, indexed by t 0, 1..... with V and E, representing the node and edge sets at time t. The set of nodes is fixed at each time step. We observe sequence of event spike counts for each node and obtain spike count matrix over the graph, denoted as X, E Nxt. up to t. Here, X, contains N time-series of event counts, denoted v Adding or removing certain edges will influence the generative patterns of events. We aim to learn a policy to understand how to steer the flow of the event counts to achieve some goal at minimal cost through sequentially adjusting the edges (E (E)0
as X, = |x]
2

---

Published as a conference paper at ICLR 2024
Anchor Events
ODE
Miligated Events
...h.. di Jt.
hy
Policy Layer
Network Intervention
امبع
اuن
Ttl
flg
Pa
Module 1: Model Learning
Module 3: Policy Learing & Interaction
Bi-Contrastive Learning
Permute h hz
Adjust hy node embeddings
Negalive Pairs
hg
hy
A. ا.سس
hy
Negative Pairs
h22
Pa
Module 2: Permutation Equivalent Embedding Learning
Figure 2: Overview of the Method. The proposed Amortized Network Intervention contains three modules. The first module is to generate a latent node embedding hi and evolve the latent states through NIODE model. The second module learns Permutation Fquivalent Emhedding (PEE) over the latent space h. by bi-contrastive loss function prepared
downstream adaptation. The third module accesses the learned PEE from the second module and generates permutation equivalent policy.
We consider an infinite horizon RL problem, where an edge intervention policy, denoted as (h') S + A. is learned to maximize the return,
ac Σ ''r(h' at)
T = arg max E sell
where h" ~ p-), h P((Ih',a') 7 E [O, 1), and a' T rhh). Detailed descriptions are as follows: Environment is referring to the system that sequentially generates (X:)220 over a dynamic the historical event sequence and intervention
network {Ge (Ve, E,) ho20. State is defined trajectories up to current time t. We will use a compact graph embedding vector h to encode the state information (which will be discussed later), where h' E N and D is the embedding dimension. Action is the choice of edge to be intervened subject to budget or other constraints. The action space defined A = {a E {0, N Em,, amnCm the intervention cost occurred for each edge, B E R., the total budget each stage, and K is the maximum number of edges to be intervened at each stage. Here, we put some constraints on the action space enable safe policies. State Transition dynamics are unknown for many real problems; will build a predictive model [- h', a²) to capture the event dynamics, where model parameters will be estimated from observational data. Reward Function tailored to suit particular applications. In general, it is function of cumulative event counts and can be augmented by incorporating other societal or environmental considerations.
B, Emm, amn K}. where [Cmen] is
In practice, solving the RI. problem (Eq. (T)) is typically computationally demanding due to its large action space and complex dynamics. This paper suggests breaking down the problem into subproblems by dividing the original dynamic graphs into subgraphs. For each subproblem, solving a model-based RL policy becomes a durable problem. Rather than solving each subproblem independently, propose learning a shared amortized policy to consolidate policy learning outcomes across regions and transfer them new regions. Details on the model-based RL framework for each subproblem will be discussed in the following section, with amortized policy learning deferred to Sec.
MODEL-BASED REINFORCEMENT LEARNING
3.1 ENVIRONMENT MODEL: NEURAL ODES
Inspired by traditional ODE-based and PDE-based models in infectious disease and traffic flow studies, we propose neural-based Networked Jumped ODE (NJODE) model to capture the event sequence dynamics. Previous works, such as Neural Spatio-Temport 2020a) and Neural Jump SDEs (Jia & Benson 2019), have been used to model fine- ien-grained spatio- temporal event processes in continuous-time We modify these models to handle large volumes of event counts in discrete-time scenarios.
Point Processes (Chen et al.
3

---

Published as a conference paper at ICLR 2024
NJODE Model: The evolving dynamics of the networked point processes are modeled by ODE system with jumps as follows, where we introduce h E IR as the latent state embedding,
h = h
(2)
dh R = b,,, VT E 12 UHfT)
(3)
dr
lim h tc = EmEN. Wm-mn o( x(x)-
(4)
<00
Here, Mr, is the neighbors of node and h E IR& is the latent state for node n at time stamp T where n E f1. .N} and 7i represents the time stamp to record discrete jumps. Rather than treating the event arrival time number of discrete events within interval [T-1, Ti) and regard it as a random variable x', allowing us to process large volumes of dense event data like traffic flow. The as a left-continuous function with right limits at any fixed T;. f is used to model the continuous change, and é is used model the instantaneous jump based on neighbors' events x f and are modeled as neural networks with the parameters shared for each dimension's 's event processes within the same subgraph. Wm->n indicates the influence strength from node m to n. We denote W = [wn-n] as the influence matrix. This architecture is similar to recurrent neural network with a continuous-time latent state modeled by a neural ODE. Under this formulation, the latent state h incorporates both historical information from itself and abrupt changes triggered by neighboring nodes. This mechanism for preserving abrupt change and recording memory is important to model excitatory point processes and generalize other unseen dynamics.
random variable (Chen et al., (020a)) we accumulate the total
of is portray h
Likelihood At each time 7i we parameterize the event count distribution as a function of the latent state h Specifically, in the rest of the paper, we assume the spike count x,, follows Poisson distribution, whose intensity X₁ is a function of h;:
(5) Here, we assume Dh is the shared distribution parameter neural network among different nodes, while by, is the distinct baseline variable for different nodes. Given this model, we see that the final emission probability of x", conditioned on historical observations X is given by
X = =xxpbb.. + Ge (h)
logpo(x X <v) -,, x log X; (,)(
(6)
where refers to all model parameters. Finally, given a spike counts matrix X2 E NNTT, we assume different nodes different times are conditionally independent given the latent state h" thereby estimate the parameter 0 by maximizing the log-likelihood that is expressed as
T
N
{i0 Xx)
log po (x", (x.)).
3.2 ONLINE PLANNING: H-STEP LOOKAHEAD WITH A LEARNED MODEL.
Given the estimated environment model in Section 3.1 execute online planning to learn how to perform interventions to the graph's edges to steer the discrete event flows. Specifically, for an N-node graph, each action involves selecting subset of & (k: S K and K is the budget constraint) edges to delete from N(N 1) directed edges (excluding self-connections). Hence, at the i-th decision time, we can represent action a' k-hot matrix, resulting in the intervened influence graph given by W @ (1 - a"). The state is the concatenation of the latent states for all nodes h'= :
Our approach draws inspiration from H-step lookahead online planning, which optimizes actions based on the dynamics model for fixed horizon with an estimated terminal value function. Only the first planned action is executed. This framework enables the prompt adjustment of the policies in real time to account for time-varying dynamic characteristics. We will adopt a policy- -rridint learning algorithm and incorporate flexible constraints on the action space.
H-step lookahead At the i-th decision time, we optimize the policy æ by looking H-step ahead, i.e.
T-1
π H.V arg max E
Σ
(r +h (h + * "
(8)
1=0
where Ô is the learned dynamic model and V #) is the estimated end-of-horizon value given the terminal state h We take (hh+H)= = 0 for H large enough, since in this scenario particular value of (h shouldn't 't affect the performance very much. Importantly, the optimal policy TH
4

---

Published as a conference paper at ICLR 2024
in Eq- 8 will stabilize the nonlinear dynamic system Eg. with the stability defined by the convergence of h", as show in (Meadows & Rawlings 1993 hhorem I]. During online planning, the rolling horizon is used to explore state trajectories that start from the current latent state h' up to h Given new observations, we will update the dynamics and repeat the above H-step lookahead planning for the next decision time.
Mean Field Reward In the planning phase, the learned dynamic model @ provides environment simulator. Accurately approximating the expected cumulative reward demands a considerable number of rollouts. According dynamic Eq. ( obtaining each rollout involves sequentially sampling x,, and feeding them back into the neural ODEs, which must be solved numerous times. In this work, we propose to use a mean-field approximation (MFA) for x denoted as Xr which will be fed back to the neural ODEs update the latent states. The neural ODEs only need to be solved once to approximate the expected cumulative reward. Specifically, define the ground truth cumulative cost ((t) finite horizon time J(t) Ei=o
E,(x,,,), and the mean- mean-field estimator as ).) ). The mean-field field estimator uses a deterministic dynamic to approximate
((t) = 7 the averaged stochastic dynamic by averaging over the randomness in x allowing more efficient calculation in both forward and backward time. We theoretically justify the MFA in Appendix Here we provide the key result.
Theorem 1 (Error Bound of Mean Field Estimator). Let J(L) and J(() be the and Mean Field Estimator for the cumulative cost. Let 7 = 1. Suppose we have N nodes. When satisfying:
1. The dynamic is Lipschitz, i.e., f and & are Lipschitz. And the Lipschitz constants for f and & are smaller than 1, i.e., Lg and Lg < <1,
2. The spectral radius of the influence matrix W is smaller than I,
3. For any n= (1, the intensity function gan. is Lipschitz and is bounded above by Lo-
Then we have:
max, Mn - - j(t) S N(+ Lox.... max, (ITT.)
where Mn = T.T,, (T,g²² + Lo) and Tn R x RNN Rdd is the composite transition function in
1/2
Proposition] 7
Gradient--esscent-aass Optimization In- stead of exhaustively searching the discrete com- binatorial action space to optimize our objec- tive, we approximate this space using a con- tinuous relaxation technique for the k-subset sampling (Xie & Ermon 2019). We replace (1- p"). where p' rep- resents edge selection probabilities. With this reparametrization, the objective becomes a fully deterministic function of the policy and dynam- ics, enabling end-to-end differentiable policy learning. We aim to solve constrained policy learning problem. We distinguish between hard and soft constraints in our approach. For hard constraints, such as limitations on consecutive lockdown days for a county (e.g., not locking down county for more than certain consecu- tive days), we employ dynamic mask to explic- itly exclude actions that fall outside the feasible space. As for soft constraints, like ensuring overall fairness in the policy, we design an ad- ditional reward term, denoted as Taug. and scale it by some weight. This augmented reward term is jointly updated with the policy to enforce fair- ness within the optimization objective.
Algorithm 1: ANI (Meta-Trainine Phase)
Input: Task pools B and a pretrained model pool θ learned based on Eq. 7
Result: Policy parameters &p and representation parameters r/
W (1 - a²) with W (1
Initialize parameters and vo; while meta-truining not complete do Sample network M, ~ B and
corresponding model 0, E θ; Policy Representation
Learning
Optimize (v, jointly based on Eq.
(0)
Obtain intervened network W" based on policy Tei
/ P Planning ahead Collect new data D, by NJODESolver 02) by Eq. 2
// Adapat ive ve Model Update
Optimize {0,} by Dj based on Eq. 7 and
Update 0, in O
end
5

---

Published as a conference paper at ICLR 2024
4 MAKING LARGE-SCALE PROBLEM TRACTABLE: AMORTIZED POLICY
In practice, managing city's extensive traffic network is a challenging large-scale problem due to its sheer size. To tackle this, we employ a divide-and-conquer approach, breaking the problem down into manageable subproblems. For instance, segment the vast network into smaller, more manageable subgraphs, each representing a tractable subproblem. While this strategy makes the overall problem more manageable, it raises crucial question: How can we utilize optimal policies from previous subproblems to streamline the optimization of new ones? To address this, we introduce the Amortized Network Interventions (ANT) framework.
Our objective is to learn a shared amortized policy (Gordon et al. 2019) that can applied across different regions with distinct dynamics. We hypothesize the existence of collective behavior among these various local temporal dynamic systems. Given a sequence of local policies ( ر addressing M distinct sub-proble -oo-proobess our goal is to create amortized policy π TANI. This policy should extract invariant representations and enable the adoption of similar policy structures among similar temporal dynamic systems. The proposed amortized policy TANI is learned over a distribution of random tasks from M, where each task Mj ~ M.
Permutation Equivalent Property Inspired from policy similarity embeddings (PSM) (Agarwal
al. 202T) and the policy permutation invariant property in SensoryNeuron (Tang Ha 2021). we devise agent that can extract permutation equivalent embeddings and is policy permutation equivalent to the latent state space h', where h' = (hfj...; (uly)..;h')) Since each block of h corresponds to N nodes system in the excitatory point process, the permutation equivalent property along the node dimension characterizes the global patterns while the learned node embedding hि, from Neural ODEs preserve individual's local characteristics. For detailed reference, we present the definition of permutation equivalent property and a derived Permutation Equivalent Metric (PEM) da similar to r-bisimulation (Castro 2020) in Appendix A Definition [ and Definition 2
Bi-Contrastive Metric Embeddings We use a representation mapping vi to project the high dimensional latent graph embeddings h' into two low dimensional graph embedding p' and m where p' only contains the internal positional information of N nodes system [hr while m' contains the magnitude information for different nodes h We illustrate the architecture in Figure 2 Intuitively, the graph magnitude embedding m' would be invariant under row permutations of h while the graph positional embedding p' would be invariant when we only change the magnitude of the row features in h', During training. we perturb the anchor graph embedding he into two groups Spem (h') and magg (h'). To jointly learn the positional and magnitude embeddings with PEM, we adapt SimCLR (Chen et al. (020b) and design a bi-contrastive learning scheme, under which the graph positional embeddings and graph magnitude embeddings can either be positive pair under permutation transformation or a negative pair of the anchor graph under magnitude adjustment. For any anchor embedding hg. we take the augmentation h, E Spemn (h'), ), and h2 (mmage(h)), k of 0, 1. Then, the bi-contrastive metric embeddings loss is given by:
1.
T(ho, by je)p((a mi))
LBCME(bo, hi, (ha):) v)- = llg l' hn )exp(s(mo) m) + Eepo, (1 T(ho, leppss m.))
exp(s (po, (ho))//hhhhhh
log exp(s(pu, (:))/T,, h1) Σx exp(s(po, (1)) T(ho, (ho,hh)) he))
(10)
where T(ho, h1) = exp(-d= 1)) is the weight given by PEM. 3 controls the sensitivity of similarity measure to PEM dz. suu, v) = ||ll|ia| denotes the cosine similarity function.
5 EXPERIMENTAL EVALUATION
We assess the effectiveness of our approach, Amortized Network Intervention (ANI), in managing networked temporal dynamics through simulated and real-world experiments. Our results demonstrate that ANI successfully reduces the mutual influence effects in both synthetic and two real datasets. We measure this improvement by calculating reduced intensities.
5.1 NETWORK INTERVENTION ON SYNTHETIC DATA
In our synthetic data experiments, we tested the proposed model on low-dimensional synthetic Multi- variate Hawkes Processes (MHP) without applying network amortization. To assess the performance

---

Published as a conference paper at ICLR 2024
Tyneres
Figure 3: Cumulative intensity cost on synthetic Figure Generalization results of local community transformations on Covid data.
datasets.
of our model-based reinforcement learning algorithm for dynamic network intervention, compared against two model-free RL baselines, SAC (Haarnoja et al. (2018) and PPO (Schulman et al. 2017).
as well as model-based RL baseline called Neural Hawkes Process Intervention (NHPI) (Qu et al. 2023). We also adapted model-free RL techniques to TPP (Upadhyay et al. (018)) for event intervention and maintained the event intervention settings for NHPI explore and compare the
effectiveness of event intervention versus action intervention with high-frequenge event data. Details on data generation for the synthetic dataset can be found in Appendix[I]
Our study results are depicted in Fig. 3 Here, intensity cost means the average intensity throughout fixed time period and over all the nodes within local region. We later use this indicator to reflect the growth rate of the pandemic and traffic within a multivariate system. Remarkably, our approach achieves comparable levels of intensity reduction as SAC and PPO in both datasets, all without direct interaction with the environment. NHPI, which focuses on event intervention, faces difficulties in reducing activity intensity, especially with high-frequency event sequences. For additional generalization results on unseen MHPs with synthetic data, please refer to Appendix [.2]
5.2 EVALUATING GENERALIZATION ON COVID DATA
Our goal is to design an amortized city lock-down strategy that shares similar policy structure for dis- tinct city regimes to curb the epidemic by intervening in the influence matrix between cities. Concretely, we
Posk Steft
trained an amortized policy from five different county corpus and tested the amortized interventions on mul- tiple unseen county dynamics. To generalize to an unseen split, the agent needs to be invariant the orders of different counties and the amplitude or the phase of the spikes of the underlying excitatory point processes. Thus, we evaluated the generalization abil-
Nagnitude Acuest
2010
2021
Figure 5: Two types of transformation of Covid data.
ity to the unseen counties corpus in two parts, local community transformation, and cross-community adaption, where local community transformation captures the agent's ability to generalize to
permuted or sneessit-aajjuser community, and cross-region adaption characterizes the ability to gen-
eralize to intensity-peat peak- phiftss community. We illustrate the two types of transformation in Fig. 5
Generalization Over Local Community Transformation We show the generalization ability to a permutated or intensity- adjusted community by permutating and changing the intensity magnitude on the same community region and applying differ-
ent control strategies to them. Fig. X demonstrates that amor- tized policy has faster convergence rate than non-amortized policy on two types of transformed communities.
Generalization Over Cross Community Adaption We inves- tigate how well the proposed approach generalizes over unseen intensity dynamics from different counties (w/ and w/o peak- shift). We evaluate the generalization performance in different county corpus with or without a similar dynamic structure to the training environment. Specifically, we define the testing en- vironment as "in-distribution" or "generalize via interpolation" when the testing environment shares a similar intensity peak to the training environment and define the testing environment as -uut-o--distribution or "generalize via extrapolation" when the testing environment has a shif- delay effect to
Figure 6: Top: Satellite map extracted from Google Earth (Goo 2022). ) Mid- dle: Road Network SUMO (Lopez) ct al. 2018). Bottom: Extracted net- works where red nodes are junction points.
7

---

Published as a conference paper at ICLR 2024
Seario Angroe fai smeoretio hor
Masine lis sporiid
Nasice lan dradeas fan
palii tite Tex
drartaed Tre
Figure 7: Generalization results for steering covid data on cross-community dynamics.
the original training environment. Table I summarizes the average reduced intensity for different methods under different region settings.
Notably, Table [ (1 row ) indicates that the non-adaptive and non-amortized policies are struggling to control the intensities in both in-distribution and out-of-distribution environments. Importantly, when we use an adaptive but non-amortized policy, the reduced intensities are quite obvious (Table i, 3rd row). This is not surprising since adaptively learning a policy (i.e., repeatedly updating the model with new policies) allows the agent to explore more possibilities in the environment and thus can obtain an optimal trajectory more easily. It is also interesting to point out that in-distribution environments are easier to generalize than out-of-distribution environments which contain peak-shift or other complex transformations when compared with the trained environment. These findings are also consistent with the intensity cost curves illustrated in Fig. 7 We also provide empirical study of two soft constraints on COVID data (i.c., limited intervention budget and limited intervention frequency) and additional baselines in Appendix J.2
5.3 EVALUATING GENERALIZATION ON TRAFFIC DATA
We endeavored to enact network interventions aimed at alleviating traffic congestion within the urban road network system, particularly at road intersections. Event data were collected through SUMO (Lopez) jet al. 2018) simulations, whereby a traffic car was categorized as contributing to congestion if its veloc- ity dropped below 0.5m/s. The network topology was derived from real-world cartography, illustrated in Fig. 6 and subsequently processed by SUMO to cre- four distinet crossroad types (detailed information available in Appendix K.T). Following training on these crossroads, we assessed the generalization capabilities of our proposed amortized network intervention method on an additional set of four previously unseen road intersections. As depicted in Fig, g our results indicate that the learned meta-policy exhibits rapid adaptability unfamiliar road systems only after few gradient steps, demonstrating superior traffic congestion mitigation ability compared to train from-scratch model. Furthermore, we include a visual representation of the learned network intervention in Appendix K.
Figure 8: Generalization results of mitigated traf- fic flow two unseen intersections from SUMO.
5.4 UNDERSTANDING GAINS FROM PEM: ABLATIONS AND VISUALIZATIONS
We show the efficacy the proposed Policy Equivalent Embeddings (PEE) which are Bi-contrastive metric embeddings comparing it to Policy Similar Embeddings (PSE) (Agarwal et al. 2021) which is another common
learned with Policy Equivalent Metries (PEM) the latent states by
Table 1: Reduced amount of intensities after network interventions for each node per unit time four unseen communities on COVID data by different methods. We report average performance across 100 runs for three different seeds, with a standard deviation between parentheses.
Reduced Intensities
Adaptive
Amortized
dn-iittriuution
disti--dittintttion
Georgia-O
Alabama-0
Georgia-1
West Virginia-O
False
-0.05(0.18)
0.08(0.06)
-0.07(0.11)
-0.02(0.05)
False
True
(.21().43)
0.18(0.58)
0.06(0.24)
0.02(0.02)
False
0.18(0.19)
0.14(0.22)
0.02(0.13)
0.15(0.10)
True
True
0.47(0.14)
0.71(0.42)
0.39(..27)
0.54(0.27)
8

---

Published as a conference paper at ICLR 2024
Figure 9: Left: t-SNE of latent embeddings learned with PEM. Middle: Original dynamic of selected 25 counties in Kansas (Top). Perturbed dynamics by two different transformations: random permutation and magnitude adjustment by FFT the latent space h, (Middle, Bottom). Right: t-SNE of latent embeddings learned with PSM. PEM successfully disentangle permutation-sensitvy position embeddings (blue) and value- sensitive magnitude embeddings (purple).
generalization approach effective on pixel-based RL tasks. Specifically, we investigate the gains from Bi-CMEs and PEM by ablating them. Instead of learning Bi-CMEs jointly through the position and magnitude embeddings, we learn separate CME for (Chen et al. 2020b) position and magnitude embeddings and use these separately learned embeddings generate the policies.
Table 2 shows that PEEs (= PEM Bi-CMEs) generalize significantly bet- ter than PSM or Single CMEs, both of which significantly degrade performance (-90%). This is not surprising since pol- icy similar metric (PSM) requires two similar states collected by nearest neigh- bors which may introduce incorrect clus- ters the latent state space. However, by introducing permutation equivalence inductive bias the problem of controlling a dynamic system modeled by neural ODBs, PEM can better characterize the invariance features from different dynamic systems.
Table 2: Ablation studies. Reduced intensity after network interventions on West Virginia (Split 0) when ablate the similarity metric and learning procedure for metric embeddings in different data augmentation settings. Each ablation entry is repeated for 100 trials for a fair comparison.
Metric Embedding
CMEs (Perm.)
CMEs (Magn.)
Bi-CMEs
PSM PFM
0.02(0.04)
0.04(0.02)
0.05(0.02)
0.01(0.06)
0.05(..02)
0.54().27)
Visualizing learned representations We visualize the metric embeddings in the ablation above by projecting them to two dimensions with t-SNE. Fig. g shows that PEEs partition the latent embeddings into four parts: (1) original position embeddings (red) and position embeddings with adjusted magnitude (green); (2) original magnitude embeddings (yellow) and magnitude embeddings with position permuted randomly (blue); (3) position embeddings with position permuted randomly (blue) which are orthogonal to the original position embeddings (red) and (4) magnitude embeddings with adjusted magnitude (purple) which are orthogonal to the original magnitude embeddings (yellow). Nevertheless, the projection of embeddings learned with PSM (Right in Fig. 9 gives a clear collapsing effect on position embeddings with position permuted randomly (blue) and magnitude embeddings with adjusted magnitude (purple). This finding is consistent with the results in Table 2 that Bi-CMEs weighted by PSM fail to extract permutation invariant and magnitude invariant information from the latent dynamic system.
CONCLUSIONS
This paper presents Amortized Networks Intervention, versatile framework to steer the excitatory point processes. Our approach handles large-scale network interventions on a combinatorial action space, and achieves promising performance challenging tasks large, real-world datasets. Furthermore, the framework discussed here holds the potential for addressing significant problems like traffic light scheduling in urban areas.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
Shuang Li's research was in part supported by the NSFC under grant No. 62206236, Shenzhen Science and Technology Program under grant No. JCY120210S241 2001 1032, National Science and Technology Major Project under grant No. 2022/ID01 16004, Shenzhen Key Lab of Cross-Modal Cognitive Computing under grant No. ZDSYS20230622091330000 and Guangdong Key Lab of Mathematical Foundations for Artificial Intelligence.
REFERENCES
Google earth. Dettps://eart..goe. com 2022. Accessed: 10 Sept. 2023.
Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Mare G Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. International Conference on Learning Representations, 2021.
Biao Cai, Jingfei Zhang, and Yongtao Guan. Latent network structure learning from high-dimensional multivariate point processes. Journal of the American Statistical Association, pp. 1-14, 2022.
Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 10069-10076, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 5084-15097, 2021.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances neural information processing systems, 31, 2018.
Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-temporal point processes. International Conference on Learning Representations, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
Delattre, Sylvain, Fournier, Nicolas, Hoffmann, and Marc. Hawkes processes on large networks. The Annals of applied probability: an official journal of the Institute of Mathematical Statistics, 2016.
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodrigen Rodriguez. and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp- 1555-1564 2016.
Guanhua Fang, Ganggang Xu, Haochen Xu, Xuening Zhu, and Yongtao Guan. Group network hawkes process. Journal of the American Statistical Association, (just-accepted);1 2023.
Mehrdad Farajtabar, Nan Du, Manuel Gomez Rodriguez, Isabel Valera, Hongyuan Zha, and Le Song. Shaping social activity by incentivizing users. Advances in neural information processing systems, 27, 2014.
Mehrdad Farajtabar, Jiachen Yang, Xiaojing Ye, Huan Xu, Rakshit Trivedi, Elias Khalil, Shuang Li, Le Song, and Hongyuan Zha. Fake news mitigation via point process based intervention. In International conference on machine learning, pp. 1097-1106. PMLR, 2017.
Marta Garello, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint 1rXiv:1807.00660 2018.
Arnab Ghosh, Harkirat Behl, Emilien Dupont, Philip Torr, and Vinay Namboodiri. Steer: Simple temporal regularization for neural ode. Advances Neural Information Processing Systems, 33: 14831-14843 2020.
10

---

Published as a conference paper at ICLR 2024
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E Turner. Meta- learning probabilistic inference for prediction. International Conference on Learning Representa- tions, 2019.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. Advances in neural information processing systems, 31, 2018.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 101222803..112,,... 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:
2018.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 1912.0169 2019.
Danijar Hafner. Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301. 2301.04104, 2023.
Alan Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58 (1):83-90, 1971.
Jessica Hoffmann, Matt Jordan, and Constantine Caramanis. Quarantines targeted immunization strategy. arXiv preprint arXiv:2008. 08262, 2020.
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.
Patrick Kidger, Ricky TQ Chen, and Terry J Lyons. " hey, that's not an ode": Faster ode adjoints via seminorms. In ICML, 5443-5452, 2021.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp. 5639-5650. PMLR, 2020.
Yan Li, Lingxiao Wang, Jiachen Yang. Ethan Wang, Zhaoran Wang, Tuo Zhao, and Hongyuan Zha. A principled permutation invariant approach to mean-field multi-agent reinforcement learning. 2021.
Michael James Lighthill and Gerald Beresford Whitham. On kinematic waves ii theory of traffic flow on long crowded roads. Proceedings of the royal society of london. series a. mathematical and physical sciences, 119(117)3177345 1955.
Zichuan Lin, Garrett Thomas, Guangwen Yang, and Tengyu Ma. Model-based adversarial meta- reinforcement learning. Advances in Neural Information Processing Systems, 33:10161-10173, 2020.
Iou-Jen Liu, Raymond A Yeh, and Alexander G Schwing. Pic: permutation invariant critic for multi-agent deep reinforcement learning. In Conference on Robot Learning, pp. 590-602. PMLR, 2020.
Pablo Alvarez Lopez, Michael Behrisch, Laura Bicker- Walz, Jakob Erdmann, Yun-Pang Flötteröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. Mi- croscopic traffic simulation using sumo. In The 2/st IEEE International Conference on Intelligent Transportation Systems. IEEE, 2018. URL http::t::10 www.b.lbb... dlr dili..de/220992.
Edward S Meadows and James B Rawlings. Receding horizon control with an infinite horizon. In 1993 American Control Conference, pp. 2926-2930. IEEE, 1993.
Jan Medlock and Alison P Galvani. Optimizing influenza vaccine distribution. Science, 325(5948): 1705-1708, 2009.
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. Advances in neural information processing systems, 30, 2017.

---

Published as a conference paper at ICLR 2024
Eli Meirom, Haggai Maron, Shie Mannor, and Gal Chechik. Controlling graph dynamics with reinforcement learning and graph neural networks. In International Conference on Machine Learning, pp. 7565-7577. PMLR, 2021.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynam- ics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE international conference robotics and automation (ICRA), pp. 7559-7566. IEEE, 2018.
Maximilian Nickel and Matthew Le. Learning multivariate hawkes processes at scale. arXiv preprint
arXiv:2002.12501, 2020.
NYTimes. Coronavirus (covid-19) 19) data in the united states, 2020. URL WRRLhttss///gtthii com/
nytimes/covid-22 1ewwidd-9--date 19-data
Yosihiko Ogata. On lewis' simulation method for point processes. IEEE transactions on information theory, 27(1):23-31, 1981.
Chao Qu, Xiaoyu Tan, Siqiao Xue, Xiaoming Shi, James Zhang, and Hongyuan Mei. Bellman meets hawkes: model-based reinforcement learning via temporal point processes. In Proceedings the AAAI Conference on Artificial Intelligence, volume 37, pp. 9543-9551, 2023.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learming, pp. 5331-5340. PMLR, 2019.
Paul Richards. Shock waves the highway. Operations research, 4(1):42-51, 1956.
Marcel Salathé and James H Jones. Dynamics and control of diseases in networks with community structure. PLoS computational biology,
2010.
Prathyush Sambaturu, Bijaya Adhikari, B Aditya Prakash, Srinivasan Venkatramanan, and Anil Vullikanti. Designing effective and practical interventions to contain epidemics. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063: 2017.
Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting control suite-a challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722 2021.
Yujin Tang and David Ha. The sensory neuron transformer: Permutationninnaaria neural networks for reinforcement learning. Advances in Neural Information Processing Systems, 34: 22574-22587, 2021.
Utkarsh Upadhyay, Abir De, and Manuel Gomez Rodriguez. Deep reinforcement learning of marked temporal point processes. Advances in Neural Information Processing Systems, 31, 2018.
Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. Breaking the curse of many agents: Provable mean embedding q-iteration for meen-field reinforcement learning. In International conference on machine learning. pp. 10103. PMLR, 2020.
Qi Wang and Herke Van Hoof. Model-based meta reinforcement learning using graph structured surrogate models and amortized policy search. In International Conference on Machine Learning, pp. 23055-23077. PMIR, 2022.
Yichen Wang, Evangelos Theodorou, Apury Verma, and Le Song. A stochastic differential equation framework for guiding online user activities closed loop. In International Conference on Artificial Intelligence and Staristics. pp. 1077-1086. PMLR. 2018.
Howard Howie Weiss. The sir model and the foundations of public health. Materials matematics, pp. 0001-17, 2013.
Sang Michael Xie and Stefano Ermon. Reparameterizable subset sampling via continuous relaxations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI'19, pp. 3919-3925. AAAI Press, 2019. ISBN 9780999241141.
12

---

Published as a conference paper at ICLR 2024
A PERMUTATION EQUIVALENT PROPERTY
Definition 1 (Permutation Equivalent Policy). Given state h' = (hfj:..;h'y and action parameterized by a k-hot adjacency matrix in IRNNN, we say a policy is permutation equivalent (PE) to the state h' if the onler of corresponding rows in the adjacency matrix is also permuted accordingly when reshuffle the orders the N latent states. Mathematically, the permutation equivalent policy can be described by a function π. NRD + NN such that
*(h') = P x (Ph')P,
(11)
where P NRNN any permutation matrix.
Definition 2 (Permutation Equivalent Metric, PEM). For any x, E S, where y is permuted state of x. i.e., y Px, for some permutation matrix P. the PEM under a distance d and policy is described by dæ S x S + IR, satisfying the recursive equation:
yx) d( (x). P (()))) + +xxx('x')))
(12)
where x' and y' are the transition states of x and y, given the deterministic dynamic f and policy æ.
The proposed distance d, captures permutation equivalent behavior through short-term distance d between policy (x) and permuted policy (y)) along with long-term behavioral difference by recursive terms. The exact weights assigned the two are given by the discount factor y. Importantly, d can be efficiently computed by approximate dynamic programming algorithms.
A.1 DISTINCTION BETWEEN POLICY EQUIVALENCE AND POLICY INVARIANCE
We want to highlight the difference between presented policy equivalence and classic policy invariance as inductive bias in RL literature. Previous research (Tang & Ha] 2021| Wang et al. 2020: Liu
eet al. 2020; Li et al. 2021). emphasize on a permutation invariant policy. i.e., π (1 7 ( [
while works introduce permutation equivalent policy, i.e., ((ass) π (x(a))((s)) where k())
is any permutation mapping. Policy invariant property has wide applications on homogeneous agents
simplify collective behavior from complex cellular systems (Tang & Ha 2021; Wang al. 2020
Liu et al. 2020; Li et al., (02TT. However, it cannot characterize the policy of an inhomogeneous system where the action should be permuted along with the states. In our case, the policy corresponds to an intervention matrix A on a network G and thus a permuted network kg)) should also generate an 'equivalent' permuted policy k(A). Thus, we name this property as 'policy equivalence" to distinguish it from 'policy invariance' from the literature.
B RELATED WORK
As previously highlighted in the introduction, several critical bottlenecks restrict universal applications of prompt control algorithms to disease. When comes to quickly acquiring new skills, meta-learning emerges as ideal paradigm for achieving rapid mastery in specific scenarios. In terms of data efficiency, both model-based reinforcement learning (MBRL) and meta-learning have the potential to significantly reduce sample complexity.
Neural Temporal Point Process. In the realm of modeling real-world data, the use of constrained models like Multivariate Hawkes Processes (Hawkes) 197I) can often lead to unsatisfactory results due to model misspecification. parameterizations of Temporal Point Processes (TPPs) to mitigate these limitations. Common approaches (Du et al.] 2016) Mei & Fisner. 2017) involve the employment of recurrent neural networks to evolve latent state from which the intensity value can be derived. However, this approach falls short in capturing clustered and bursty event sequences, which are prevalent, as overlooks vital temporal dependencies necessitates excessively high sampling rate (Nickel & Le 2020).
recent studies, researchers have started exploring neural network
To surmount these challenges, Neural Jump SDEs (Jia & Benson 2019) and Neural Spatial Temporal Process (NSTT) (Chen et al.] 20200a extend the Neural Ordinary Differential Equation (ODE) framework, facilitating the computation of exact likelihoods for neural TPPs while addressing the limitations of prior methodologies. These two advancements closely align with our dynamic model, and we draw upon their concepts to develop neural excitatory point processes (EPPs) that are governed by an influence matrix. Besides Neural ODE, other streams of research on networked excitatory point processes are also proposed. Hawkes Processes Large Network (Delattre et al.) 2016) extends the construction of multivariate Hawkes processes to encompass a potentially infinite network of counting processes situated directed graph. Other approaches of latent structure
13

---

Published as a conference paper at ICLR 2024
learning in multivariate point process (Cai et al. 2022 Fang et al. 2023) emphasize accommodating heterogeneous user-specific traits and incorporating both excitatory and inhibitory influences.
Manipulation of Dynamic Processes. The manipulation and control of dynamic processes represent an active area of research. Typically, control policies for temporal process manipulating can be divided into two categories: (1) Gradually introducing exogenous event interventions into the existing historical events, and (2) Promptly enforcing network interventions to the influence matrix between different types of events. Currently, most research is centered around the first type of intervention, primarily focusing on low-frequency and low-dimensional event interventions within social media datasets. For example, techniques like dynamic programming (Farajtabar et al. 2014 2017) and stochastic control on SDE (Wang et al., 2018) with a closed feedback loop are utilized to steer user activities in social media platforms. However, the number of event types in the above work is limited to derive the closed-form solution. Modern Reinforcement Learning approaches, including both model-free (Upadhyay al. 2018) and model-based RL (Qu et al., 2023), are proposed to mitigate fake news events on social media. Notably, the event-intervention basiiond-bddeddee approach will fail to generalize to high frequency and uncontrollable event data like newly infested disease cases and incoming traffic.
On the other hand, the problem of network intervention, particularly node manipulation (e.g.. vaccination) to control epidemic processes on graphs has received extensive attention (Hoffmann et al., 2020; Medlock & Galvani 2009). Most previous studies have adopted a static setup and made a single decision. In recent work (Meirom et al., 2021), the agent performs sequential decision-making to progressively control graph dynamics through node interventions, demonstrating effectiveness in slowing the spread of infections among different individuals. While existing nehwork-intervennn based approaches offer a promising solution for individual-level quarantine in pandemic control, they do not inherently adapt to county- or state-level control, which necessitates more complex node status considerations than those assumed in (Meirom et al.) (02T), as well as a larger search space incorporating edge interventions. Moreover, it's worth noting that our approach is related to, but more comprehensive than, epidemic control problems, as it accommodates various data distributions, including Poisson, within excitatory point processes.
Model-based Reinforcement Learning. The key to applications within a Reinforcement Learning (RL) framework lies in enhancing sample efficiency, with Model-Based Reinforcement Learning (MBRL) serving the role of approximating a target environment for the agent's 's interaction. In environments characterized by unknown dynamics, MBRL can either learn a deterministic mapping or a distribution of state transitions. denoted as pl [As|s a)). Typically, modeling uncertainty in dynamic systems involves the evolution of hidden unit to represent a dynamic world model. Several rege-segressive neural network structures prevalent in this domain, including well-known models such as World Models Ha & Schmidhuber (2018), Decision Transformer (Chen et al. 202I). and
the Dreamer family (Hafner et al.] 2019 2023). On the other hand, the integration of Neural Networks with Model Predictive Control (MPC) (Nagabandi et al., 2018) has achieved excellent sample complexity within model-based reinforcement learning algorithms. Our approach closely aligns with MPC and MBRL techniques, wherein MBRL methods, such as Gradient Descent, can be leveraged refine or adapt the model utilized by MPC. This adaptation holds the potential to enhance performance, especially in scenarios where system dynamics are non- non-linear, partially unknown, or subject to change.
Meta Reinforcement Learning. The majority of meta Reinforcement Learning (RL) algorithms adhere to model-fres approach and introduce task-specific variational parameters facilitate learning across various simple locomotion control tasks. Examples of such algorithms include MAESN (Gupta et al.] 2018) and PEARL (Rakelly al. 2019). Simultaneously, there has been notable success in recent times by incorporating the inherent sequential structure of off-policy control into the representation learning process, as demonstrated in works like CURL (Laskin et al. 2020). Sensory (Tang & Ha 202T), and PSM (Agarwal et al. 202T) particularly when dealing with more complex tasks such as those found in the Distracting DM Control Suite (Stone et al.) 2021).
In scenarios where data is limited, such as in disease control, researchers are increasingly focusing on Model-Based Meta Reinforcement Learning (MBMRL., with specific emphasis fast adaptation within dynamics models. Approaches like AdMRI (Lin et al. (020)) and Amortized Meta Model-based Policy Search (AMBPS) (Wang & Van Hool 2022) involve the optimization and inference of task-specific policies within a parameterized family of tasks, often containing parameters related to positions and velocities. Importantly, a key distinction between our method and AMBPS
achieving
14

---

Published as a conference paper at ICLR 2024
Abaate
MIDO
MmW
ahdle Adla Matte
0.5
20
60
90
100
Figure 10: Temporal Point Process Model learned by different methods. Left: ground truth intensity function. Middle: Learned intensity function plot by Neural Jump ODE LLog-Likelihood -33).
Right: Learned intensity function plot by Neural Process. Liog-Likelihood: -183)
lies in our utilization of network embeddings and the incorporation of inductive bias facilitate the learning of a meta- veeaa-policy without parameterizing individual tasks.
C LIMITATIONS
It should be noted that our approach assumes that decisions made by considering receding horizon window size of ''' for each node provide a reasonably good approximation of the optimal policy, However, it is important to acknowledge that if long-range correlations exist, this approximation may result decreased performance. Consequently, an intriguing question arises regarding the ability of our approach to effectively tackle the problem of network interventions within long-range correlated data distributions under the proposed training protocol.
D A COMPARISON BETWEEN NEURAL ODE AND NEURAL PROCESS
In this section, conduct comparative analysis of model performance and efficiency between Neural Ordinary Differential Equations (Neural ODE or NODE) (Chen et al. 2018) and Neural Process (NP) (Garnelo al., 2018). To evaluate these models, utilized synthetic dataset consisting of 5-dimensional Multi-Hyyerparameter (MHP) data, with each dimension containing 100 sample points in the training set. The results of this comparison are visually presented in Fig. TO]
Our observations reveal notable distinctions between the learned behaviors of NODE and the Neural Process model. Specifically, the curve learned by NODE exhibits greater diversity, depicted in the middle panel of Fig. TO An important metric to consider is the log- heg-likllihodd, where we find that Neural ODE achieves log-likae likelihood of -33, significantly outperforming Neural Process with log-likelihood of -180.
It's essential to consider the computational cost associated with each model. For Neural ODE, significant portion of the computational burden arises from the numerical integration process itself. The runtime complexity of this integration process is denoted as O(NFE), where NFE represents number function evaluations. The worst-case scenario for NFE depends on two factors: the minimum step size required by the ODE solver and the maximum integration time of interest, denoted as Atmax-
In contrast, the runtime complexity of Neural Process, which employs context points and m target points, is represented as Onn + m). our specific experiment, the number of total function evaluations (NFE) for NODE was approximately 5000, while for Neural Process, we selected 50 context points and 100 target points.
It is worth noting that the integration steps, as quantified by Afyue, can potentially result in large constant factor, which may be hidden within the big-O notation. However, it is reassuring to acknowledge that modern ODE solvers, such as dorpiS, are designed with adaptive step size mechanisms that adjust dynamically to the supplied data. This adaptive behavior mitigates concerns related to the scalability of the integration process with respect to dataset size and complexity.
15

---

Published as a conference paper at ICLR 2024
E
TECHNICAL DETAILS OF A PROBABILISTIC NETWORKED MODEL
Table 3: Table of conditional spike count distributions, their parameterizations, and their properties
Distribution
p v)
E(r)
Var(z)
Poi(erp(())
cxp( (ep(ccp(W)(erp(1""
exp(() a(v)
exp(()) (1
Bern(a (v))
--)
Bin(v, (1))
(((
(o)
ca(v)o(--)) vexp(e))/o-$$)
NB(v, (
(u Jo(
e. exp(v)
F TECHNICAL DETAILS OF MEAN FIELD APPROXIMATION FOR REWARD MODEL
We begin by defining some notation which will be used throughout these results
Spectrum nonlinear operator f: Ly = || := sup ug
[(x))l(x)l < 00
2-
F.1 THEORETICAL JUSTIFICATION
We use x = ( xx to record the incremental number of events for different nodes starting from within [T-11, Tz). We consider a stochastic dynamic system based on the nonlinear system Eq. (2-4) presented in the manuscript:
h h2
dh
=
VT E R4 UUfT.
dr
lim h EmEN, Wm-mn p(h x(x) <40
(13)
where f R= + Fd and o: Rd R are two Lipschitz functions.
The emission/intensity function for the dynamic is defined as A(ThT-) := IA (h hh (h"t where 9A.w Rxx layer, and hT Xxd is the left continuous points before jump. We use X to denote m-th element of (i)) and thus x ~ Poi(Am (i(hii-) is the number of incremental events in [7-1,, Ti) for node m. The ground truth cumulative cost J(t) at finite horizon time t is given by
w) =
RVV is Lipschitz activation function,
Rd is a linear
J(1) = Σ
B(r
(14)
i=0
Instead of extensively performing Monte Carlo simulation to obtain the cost estimator, we introduce a Mean Field Estimator J(t) for (t) by averaging out the interaction effect by xf the dynamic
system 13 i.c.
j(() =
(15)
1=0
where Xú deterministic dynamic by replacing the stochastic discrete update term in system 13) with o(h% (g. We let the initial states be the same point, i.e., N. = h,, Vn E (1,2.....NN
(inh and h is the mean-field estimator for h and is derived from
Proposition 1. Given f and g are Lipschitz in Sys. 13), let h be the left continuous point h in the Sys. 13), then can be recursively expressed by a Lipschitz operator T, R R
Rd
h = (h"th"-s,,** x()))))))))))
(16)
where h E Nd and x E RN More importantly, let Xm be the maximum spectrum of influence matrix W. when T,g, L.g, and Xn smaller than have Tor,, 1.
16

---

Published as a conference paper at ICLR 2024
Proof. We define ((h",'') [o(h;' xi) (( xz), ,,,,,,ohhhhh xx)) RNxd the discrete
kernel in System 13 we can represent h as:
{-' f(b))) ds, ,r;
h Th'
w 2 (hh
where w is the n-th row the influence matrix W. Since and & are Lipschitz and the composition
of Lipschitz functions is also Lipschitz, Tn is Lipschitz. Since LT I Xw (L (L ( where m is the number of summation in the intergral term, we have Lt,. < 1 when Lig, Lo, and An are smaller than
Proposition 2. Given J and are Lipschitz in Sys.
113. let h be the left continuous point of
hi in the deterministic version of the presented Sys. recursively expressed by Lipschitz operator Tn E
13 by replacing xm with Xn then it can be Ng.xx x INN" 12
h =
(h A(i 1))
(17)
where A(i - [A]
N
Proof. This direct result from Prop.
Lemma 1. Let h be the mean field estimator for h suppose f and & are Lipshitz, and Vn, i, X1. is bounded by Lo, we have:
h h
[1]
Lr, E | b7.-1-1] + Mn.
(18)
where M,, LT, (Lo 1/2 + Lo). Moreoven, when Ls < 1 and Los 1, we have:
E 1 1 < (1 -(TTTa)
Mn
(19)
1- I..
Proof. For simplicity, we use h to denote h and hig to denote h Here we prove the first and Prop.
inequality: By Prop.
decompose h and h i.e.
hill
E
T (h
T (hi-1 A(i 1))
TF
L T.,
i- 1 -1 A(i 1)>1|
|| h 1 + ||x 1-1 A(i DII]
S IE
LT,
E
1- h 1h:-1
+ |x
A 4-1 + i-11 A(i 1
< LT,
i-1 -
+ E LT ||x 1-1 - A(i 1)1l
+ LT, 1) A(i-()))))) 1)Il
= L E
| i- hi
+ LT, E
| - A(i 1)I1
+L,,1AA(i - - A(I 1)I
< LT. E
| - 1
+ Lor (I/112 + Lo)
LT. E
|
1
h
+ Mn
Given the above result, we can derive its fixed point x i.c..
x² Io x= + Mr
(20)
x² Mn
(21)
LTz
17

---

Published as a conference paper at ICLR 2024
Since LT, 1 given L, < 1 and Lg < 1, Eq.
(18) is contraction and thus starting from any
arbitrary point my will converge to this fixed point æ². So we have:
E I
[ill
Ma
S Ltn
h - hn
Mn
(22)
1 - LT.
- LTD
S (LTT))'
B B[in -6111]
Mn
(23)
LiT
Mn E l hi (1 (I LTa
(24)
where the last inequality follows by h.: = h.. Thus E ||b h
is bounded.
Next, let's prove Theorem I
Proof.
((t) - J(t)
E E
(25)
==0
n=1 1=0
Eng Eng (
[
Irn (
(26)
N
N
Eng 9hm. w (ha)
Σ
Σ
gxn. (nin))
(27)
2m0
n=l
im0
N
Σ n=
Eng
(An, - I2n w (hi)))
(28)
i=0
Ebg
(9An. grn. ((th)
(29)
N
Ebs
Lo I1 bill]
(30)
n=1
1=0
Apply Lemma
we have,
Mn Laanow (1 (LTh)')
(31)
1 -TT
n=1 1=0
Mn
(t + 1) Lgxu.. I LTT
(32)
max, Mn
< NN(+
L Lggn. 1 ma
(33)
F.2 EMPIRICAL EVALUATION
We consider the setting of simplified linear dynamic model based on the nonlinear system (2-4) presented in the manuscript. Specifically, the nonlinear ODE drift term f(ī,) replaced by a linear function Ah, + b where A I dxd and b E Rd. The nonlinear discrete jump kernel o(h oi is replaced by a linear kernel Cha; + Dx where E IR represents history forgetness and E R the scaling factor for the events. Mathematically, the simplified linear dynamic is given in System 34).
h h dh Ah + -b,
(34)
lim hete EmEN, Wm-en (Cm + Dr
18

---

Published as a conference paper at ICLR 2024
Node 1
Node 2
0
Heon Approbraaee Wilvers (105) Sanglest
2.5
seleecITE Sarglei
23-
1.3
1.2
0.5
0.2
-0.-
200
1ime
18me
Figure 11: Empirical Evaluation of Mean Field Approximation
Empirically, we performed toy experiment on the simplified 2-D linear dynamic and we found the mean field approximator provides efficiency and accurate approximation for the rollout means (Fig.
G DIFFERENCE BETWEEN SIMCLR AND THE PROPOSED METHOD
Given a random sampled minibatch of N examples and the SimCLR contrastive prediction task is defined on pairs of augmented examples derived from the minibatch, resulting in 2NV data points. SimCLR treats the other 2(N 1) augmented examples within a minibatch as negative examples. Then the loss function for a positive pair of examples (i,j) is defined as,
exp(sim(2,, 2,)TT)
2i - log
(35)
Σ
2N
k= [Rges]] exp(sim(2;, //T)
where z, is the projected embedding from augmented samples.
Compared with SimCLR, our proposed loss does not require sampling additional batch data as negative examples and projecting the augmented sample to one embedding, instead, we only use one anchor sample (anchor network) and project the sample to two different embeddings (p and m), and augment the anchor sample by permuting its node orders or adding noise to the node embeddings such that it can form two negative pairs naturally (as depicted in Fig. IT. Moreover, augment our network on its latent embedding space by utilizing the graph augmentation techniques, while SimCLR directly performs augmentations on the original visual representations.
H ARCHITECTURE AND HYPERPARAMTER,
For the dynamic model, we parameterized the ODE forward function fr. in Eq. (3) as a Time- dependent multilayer perception (MLP) with dimensions [64-64-64]. We used the Softplus activation function. We first attempted to use MLP to parameterize the instantaneous jump function db. in Eq. { however, led to unstable results for long sequences. Thus we switched to the GRU parameterization, which takes input, the latent state h and outputs new latent state. Importantly, we found letting fn and dy, share all the parameters across different node trajectories will also lead to the failure of capturing the mode diversity. We therefore created an independent biased term added between the MLP layers in fn to compensate for diversity loss. Lastly, we use MLP to parameterize the emission function gx-
We initialized all Neural ODEs (for the hidden state) with zero drift by initializing the weights and biases of the final layer to zero. All integrals were solved using (Chen et al. 2018) to within relative and absolute tolerance of 1E-4 or 1E-6, chosen based on preliminary testing for convergence and
stability. We also use Seminorms (Kidger et al. 2021) to accelerate neural ODE learning and apply temporal regularization (Ghosh et 2020) to mitigate the effect of stiff ODE systems.
For the policy model, we parameterized the policy network by 4 heads, and 128 d-model Transformer layers. For the synthetic dataset, we used 2-layer transformer for representation learning and another 2- 2-layer transformer for policy generation. For the wollworld dataset, we used a 4-layer transformer for representation learning and another 4-layer transformer for policy generation.
For the Policy Equivalent Metric presented in Definition 2 we learned a value function parameterized by 2-layer transformer by optimizing Least Square Temporal Difference (LSTTD).
19

---

Published as a conference paper at ICLR 2024
We trained the dynamic, policy, and PEM-value network by using the ADAM optimizer with IE-2 decay rate across 4 RTX3090 GPUs. The initial learning rates for dynamic learning, policy learning, and PEM-value function learning were set to be 1E-3, IE-4, and 1E-4 respectively.
ADDITIONAL DETAILS OF SYNTHETIC DATA EXPERIMENT
I.1 DATASET SETUP
We generated synthetic networked point process data by simulating Multivariate Hawkes processes (MHP), which are doubly stochastic point processes with self-excitations (Hawkes) 1971). Specif- ically, the underlying ground truth influence matrix W was generated with 10 nodes and the weights were set Wij ~ 24 0.5]. We set the graph sparsity to 0.1, i.c., each edge is kept with probability 0.1. The generated influence matrix was adjusted appropriately so that its maximum spectral radius was smaller than one, ensuring the stability of the process. We further set the Hawkes kernel to an exponential basis kernel, where the parameter was set to B 4, meaning roughly losing 98% of influence after unit of time. The simulation of MHP was based thinning algorithm (Ogata 1981) on a T 100 horizon.
1.2 ADDITIONAL RESULTS AND FIGURES
We additionally trained the amortized policy different ground-truth weight matrices and tested the performance on new star or cycle graph with random weights. The results shown in Fig. 12 Surprisingly, we find the amortized policy only slightly outperforms the non-amortized on both environments. We conjecture this is because we are adapting the amortized policy to a small local region (only 10 nodes) in this experiment so that the non-amortized policy already can achieve relatively good results.
synthetic star graph and circular graph with
Star Split 4 (Interpolation)
Cycle Split (Interpolation)
Adaptive: True, Amortized: True
Adaptive: True, Amortized: True
Adaptive: True, Amortized: False
10-
Adaptive: True, Amortized: False
Intensity cost
Intensilyy
Episodes
Episodes
Figure 12: Generalization Results over synthetic data
J ADDITIONAL DETAILS OF COVID DATA EXPERIMENT
J.1 DATASET SETUP
We used data released publicly by (NYTimes 2020) on daily COVID-19 to learn the excitatory point processes of the pandemic outbreak. The data contains the cumulative counts of coronavirus cases in the United States, at the state and county level, over time. Specifically, we separated the U.S. COVID- 19 data into state-wise records and further split a state-wise record into different county corpus where each split is named as "a local region" or "a split", containing distinct intensity trajectories from no more than 25 counties.
J.2 ADDITIONAL RESULTS AND FIGURES
Additional Baselines for COVID data We applied two additional baselines SAC and PPO on randomly chosen community that contains nine counties. We used the learned NJODE model as the covid environment simulator and the learned model (Without control) is shown in Fig. 13 along with the observed ground truth counts. We also attempted to use the plain Hawkes model (only influence matrix A and baseline b are learnable) to learn the underlying COVID dynamic. As Fig. B depicted, however, plain multivariate Hawkes processes (Purple) fail to distinguish the intensity difference between different counties while the NJODE model (Black) correctly captures the characteristic
20

---

Published as a conference paper at ICLR 2024
County
County
County
Contral (M@OE) (Ours)
No Control (NJOGE) (Ours)
No Control (NJODE) (Durs)
No Contral (Hawees Model
No Contri (Howees Mode
No Control (Hawwess Mode)
10
ANI joursh: Rectue
30-
(Cturs: Reduce 0.22
AMI (Durs): Beduce
SAC: Reduce
Reduce 0.05
Redlce 0.50
25-
Intersiyy
Truth
Inturstty
10-
5-
20
60
60
80
1000
120
20
40
60 Time
80
100
120
0
20
40
600
80
100
120
Time
Time
Countyy
County
County
10
No Contral (MOODE (Duns)
No Centrol (MODE)) (Ours)
No Control (NJOGE) (Durs)
Control (Hawkes Mooch
40
No Control (Hawker Model)
12
No Control (Hawkes Model)
l0urst: Reduce 0.07
(ours:: Reduce 0.14
10
Reduce
PPO:
30
PPO: Reduce
Internlly
Ground Thuth
Intersity
Ground "tuuhh
Immunatty
Ground Truth
10
20
50
1.00
120
20
40
60
100
120
40
60
100
120
Time
Tiena
Time
County
County
County
16
No Control (MODE) (Oun))
Nn Control (NCDE)) (Durs)
No Control (NDDEE (Cum)
Control (Hambo ModwO
Contrd (Heado Nodel!
5.
No Control (Haerbes Model)
14
Beduce
PPO: Redure
IP0: Redune
Beduce -0.22
Intersit
10
Ground Truth
Ground Trash
Interstyy
Ground Truth
20
60
1.00
120
60
1.00
120
100
120
Time
Time
Time
Figure Additional baselines on nine counties
of different counties and also preserves multimodality within each county. We conjecture this is because the plain multivariate Hawkes process does not have enough parameters characterize the individual variation on the complex COVID data dynamic correctly. Regarding the intervention effect, we observe that ANI has successfully reduced the number of infested people (having a positive reduced intensity value) in six counties among nine while SAC and PPO struggle to have positive intervention effect on the nine countics. We speculate it is caused by the disconnected gradients between the consecutive latent states (without backpropagating through the learned COVID dynamic model itself) when using model-free RL algorithms like SAC and PPO.
Fairness constraints and more We also present the results and trade-offs for different policies under various fairness constraints. Specifically, we use a to control the weight an intervention budget cost and use X2 to control the weight of policy smoothing cost. The intervention cost used here is defined by the distance between two counties when an intervention is implemented and the smoothing cost is defined by the distance between two consecutive policies. We use the average reduced intensity and the average lockdown probability for each edge during the total horizon to find imposing heavy constraints on the policy simultaneously (i.e., Å1 0.1, X2 0.1) does not lead to the lowest lockdown probability (1.29%) but does give the lowest control effect (0.01). Instead, only enforcing smoothing constraints (i.e., A = 0.0, X2 = 0.1) gives us a fairer policy, i.e., less effort to intervene (average lockdown probability: 0.43 %) but achieve fair control effect (average reduced intensity: 0.10). We conjecture that adding extra intervention cost constraints will discourage the agent from exploring
measure fairness and the result is shown in Table 4 Interestingly,
21

---

Published as a conference paper at ICLR 2024
and thus underperform policy smoothing constraints, We illustrate the detailed lockdown for different constraints Fig.
Table Average lockdown probabilities and reduced intensity under different soft constraints.
\1
0.0
\1 0.1
Fairness/ Parameters
0.0
X2 0.1
λ2 0.0
X2 0.1
Average Reduced intensity
0.25(0.06)
0.10(0.04)
0.06(0.05)
0.01(0.03) 1.29(0.27)
Average Lockdown probability (%)
1.31(0.53)
0.43(0.12)
1.29(0.17)
Intervene Coeff: 0.0, Smoothing Coeff:
Intervene Coeff: Smoothing Coeff: 0.1
0.8
12
14
0.4
0.4
20
2322
0.0
0.0
10
14
16
20
24
10
14
18
20
24
Coeff: Smoothing Coeff:
Intervene Coeff:
Smoothing Coeff: 0.1
0.8
0.4
0.4
72
24
74
0.0
0.0
10
14
16
20
24
10
14
16
20
24
Figure 14: Amortized Networks Interventions in probabilities under different constraints.
K ADDITIONAL DETAILS OF TRAFFIC DATA EXPERIEMENT
K.1 DATASET SETUP
To simulate real-world traffic, based on the road types shown in Fig. 6 we design a road network with four types: intersections with one or two lanes, and T-Junction with one or two lanes. Specifically, we let the speed of the road be 8m/s or I m//s randomly. Then, we generate car trips by the random generation tool from the SUMO package. We make such simulation for 1000s at one run. After this playing,
get the simulation results including emissions (e.g. CO2, CO), positions, speed, and lane id (i.c. the car runs in which lane with lanes more than one a road) of each car at each time step. Given the generated summary data from SUMO, then count the congestion event (i.e. the car speed less than 0.5m/s) for the following analysis.
K.2 ADDITIONAL RESULTS AND FIGURE
We show the intensity cost of the learning process for four types of roads in our simulation setup in Fig. [5] For both our model (meta) and our model (train from scratch), the cost trend will converge after several time steps, which proves that model has great learning and generalization ability.
22

---

Published as a conference paper at ICLR 2024
T-junction (Single Lane)
Intersection (Single Lane)
30
Ours (meta)
(meta)
Ours (train_from,_a
(haiccchomsscrrthh) scratch)
Ours (train, htt.//orommtccocc scratchh
25
Intensity Cost
20
155
15
10-
Steps
Steps
T-junction (Two Lanes)
Intersection (Two Lanes)
(meta)
Ours (metal
Ours (train from scratch)
Durs (rain from_scraach
160
Intensty Cost
200
100
50
100
150
200
250
300
350
400
50
100
150
200
250
300
350
400
Figure 15: Intensity cost of four types road.
K.3
CASE STUDY
We further provide case study to show the great interpretation capacity of our model. In the T-junction with single lane scenario, there are discrete actions, corresponding to the following green phase configurations in Fig. [6 The traffic light is marked node 10, while the other three lanes marked as node 8, node 11, and node 12 in our Sumo simulation setup.
Real-world traffic can be represented as an NJODE model corresponding with the change of traffic lights. In our simulation, when node 10 (i.e. traffic light) becomes green, the lane controlled by this traffic signal will connect, which is represented as 1 in Fig. 17 otherwise, the connection between two lanes disconnected and represented O. For instance, the first sub-graph Fig. 17 the car can move from node 11 to node 12 under the control of its traffic signal. The learned policy of our model is exactly consistent with real traffic trips, which demonstrates that our model has great adaptation ability in uncovering real-world network interventions.
Node
Node
Node 8
Node 11
Node 11
Node 11
Node 12
Node
Node 12
Figure 16: Diserete actions of the T-junction (single lane).
23

---

Published as a conference paper at ICLR 2024
1,0
10
12
14
16
18
20
22
0.2
0.0
d10 8
d10.11
v11.10
dio_12
u12
10
d10
8
08.10
d10
410_11
u11.10
d10_12
u12_10
41e 8
19.10
d10
11
u11.10
010
12
u12
10
Lane (Start From Io:
Lane (Start From 10
iStart From u12
Figure 17: The learned policy of traffic generated from our model.
24


Reviews:
[{"review_id": "ca7dfOHSOR", "reviewer": "Reviewer_9miN", "summary": "The paper looks at both the learning and optimal intervention planning problem for a large-scale network whose evolutions include both continuous state drift and discrete jumps. The authors propose to learn the transition of the states via MLE. Directly dealing with the large-scale problem is intractable. Hence, the authors also propose the Amortized Policy Learning framework to \n1) segment the network into pieces and each time pick one subnetwork to update the model estimation;\n2) update the policy parameters with data collected from the current subnetwork using common feature representation learned by minimizing the bi-contrastive loss. \n\nThe authors apply the proposed framework to tasks including synthetic data, covid data and traffic data where improvements are obtained and show that the proposed policy equivalent embeddings successfully decouple the position embeddings and value embeddings.", "questions": "As I have mentioned before, is it possible for the current framework to capture the local patterns in a large network and perhaps steer the learned policy towards local patterns while maintaining the generalization power (such as regularized policy optimization).", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors' contributions include: \n1) Formulate the problem as a RL problem.\n2) Propose to decompose the problem into subproblems of smaller scale and learn a policy that can generalize to the full-scale problem via   ensuring permutation equivalence. \n3) Test and compare the methods to previous methods on several settings of practical interest. Also, the authors show concrete evidence of the out-of-distribution generalization power of the proposed method.", "weaknesses": "In my point of view, the authors did not clarify their contributions. \n\nIn terms of problem formulation, modeling the discrete events by counting the occurrences in a time window should not be counted as the novelty. In terms of model learning, the authors are basically using the MLE method, which is standard in model-based RL. \nPolicy learning within the permutation equivalence class through learning embedding $p^t, m^t$ via contrastive method is an interesting idea. However, the authors do not differentiate their work from (Chen et al., 2020b).\n\nMore crucially, each part in a large networks may have both global and local patterns. How to deal with the local patterns in model learning and policy optimization is not clear. Or the authors are just trying to obtain a policy that only has \"overall\" good performance.\n\nAnother question is how to deal with the soft/hard constraints. I did not find evidence supporting the effectiveness of the constraint ensuring methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2m32Pau3h7", "reviewer": "Reviewer_YqcV", "summary": "This paper proposes a re-enforcement learning algorithm to adaptively change the network structure to steer the observed event counts over a potentially large network. The proposed algorithm is carefully crafted to model network point process data with complicated data structures, which is desirable for solving large-scale real-data applications. The proposed algorithm is theoretically sound, and its effectiveness is demonstrated through simulation studies and real data applications.", "questions": "1. The review of the work on networked excitatory point processes seems to only focus on work that used ODEs. However, I would like to point out that there are also other streams of research on networked excitatory point processes, for example, [1], [2], [3]. It is better to conduct a more comprehensive review of the topic.\n\n2. In the definition of the temporal graph network, what are the definitions of edges in the two examples given in the paper? I do not find formal definitions. Could you please clarify?\n\n3. On page 3, it is claimed that the \"high-dimensional event sequences $\\{X_t\\}_{t\\ge 0}$ has a stationary dynamics\". What do you mean by \"stationary\"? If $X_t$ is the accumulated counts up to time $t$, it is unlikely to be a stationary time series. Please clarify.\n\n4. Is the influence matrix $W$ in equation~(4) a given matrix or learned from the data?\n\n5. The use of mean field approximation on page 4 seems to be rather ad-hoc, without much theoretical or even heuristic justifications. Can you give an example to show the difference between the actual reward and the one calculated based on the mean-field approximation? Are they actually close? Perhaps some simulation studies can be carried out to show the difference in actual reduction in intensity using these two reward objectives in some simple settings.\n\n6. On page 5, it is stated that \"Given a sequence of local policies $\\{\\pi_i\\}$, $1\\le i\\le M$,  addressing $M$ distinct sub-problems, our goal is to create an amortized policy $\\pi_{amo}$\". However, I did not find any detail on how this goal can be achieved. Can you clarify?\n\n7. The label of the y-axis in Figure 3 reads \"Intensity Cost\". What is the definition of the \"Intensity Cost\"?\n\n\n[1] Delattre, S., Fournier, N., & Hoffmann, M. (2016). Hawkes processes on large networks.\n\n[2] Fang, G., Xu, G., Xu, H., Zhu, X., & Guan, Y. (2023). Group network Hawkes process. Journal of the American Statistical Association, (just-accepted), 1-78.\n\n[3] Cai, B., Zhang, J., & Guan, Y. (2022). Latent Network Structure Learning From High-Dimensional Multivariate Point Processes. Journal of the American Statistical Association, 1-14.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written, the presentation is clear and the idea is sound.", "weaknesses": "More details are needed in some components of the proposed model, including the mean field approximation for the rewarding model and the construction of the amortized policy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DrHRQhOUrN", "reviewer": "Reviewer_hvRi", "summary": "The paper discusses a method for addressing the challenge of large-scale network intervention in scenarios like controlling the spread of infectious diseases or managing traffic congestion. The approach uses model-based reinforcement learning with neural ODEs to predict how excitatory processes in a network will change over time as the network structure evolves. It incorporates Gradient-Descent based Model Predictive Control (GD-MPC) to provide flexibility in policy development, accommodating prior knowledge and constraints. To handle complex planning problems with high dimensionality, the authors introduce an Amortize Network Interventions (ANI) framework, which allows pooling of optimal policies from historical data and various contexts while ensuring efficient knowledge transfer. This method has broad applications, including disease control and traffic optimization.", "questions": "For adding interventions, can we also consider causal-inference-based methods as valid comparison?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-motivated and the proposed model is technically sound and can notably handle large-scale systems. The overall writing is easy to follow. The experiment section is comprehensive though missing some baselines.", "weaknesses": "In the experiment section, why baseline comparison is only limited on one synthetic dataset? Also, can the author explains why the NHPI baseline almost have a constant intensity cost in Figure 1?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hGBhto182N", "reviewer": "Reviewer_ya7t", "summary": "The paper proposed a model-based reinforcement learning approach to model networked point processes. Considerations are given on the scalability of the proposed method. The paper describes an amortized policy approach to deal with the scalability issue. The proposed method is applied to synthetic data, real-world covid data, and real-world traffic data.", "questions": "* Is it possible for the authors to describe the potential limitation due to the use of mean field approximation for reward modeling?\n* Are there experiments to demonstrate the incorporation of fairness constraints and more?\n* \"For instance, when regulating the coronavirus, government interventions must balance health concerns with economic implications and public sentiment.\" It is not clear whether the experiments described in the paper achieve such a balance.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* originality: the problem formulation of networked point processes described by the author has some originality. \n* quality: the proposed method makes sense to me. The scalability issue is highlighted in particular. I think the scalability issue is an important one to make the algorithm pratical. The authors proposed the use of amortized policy to deal with this issue.\n* presentaiton is clear. I can follow the rationale of the paper.\nsignificance: the proposed method is associated with application scenarios of high significance.", "weaknesses": "* I appreciate the authors applying their proposed method to two important pratical problems. However, I think the paper can benefit from comparison to alternative methods. \n\n* Using intensity cost as the evaluation metric is also somewhat obscured.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)