When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL
Authors: Lenart Treven
Bhavya Sukhija
Yarden As
Florian Dorfler
Andreas Krause
Venue: NeurIPS 2024 poster (2024)
Keywords: reinforcement learning
continuous-time
Primary area: reinforcement_learning

Abstract: Reinforcement learning (RL) excels in optimizing policies for discrete-time Markov decision processes (MDP). However, various systems are inherently continuous in time, making discrete-time MDPs an inexact modeling choice. 
In many applications, such as greenhouse control or medical treatments, each interaction (measurement or switching of action) involves manual intervention and thus is inherently costly. Therefore, 
we generally prefer a time-adaptive approach with fewer interactions with the system.
In this work, we formalize an RL framework, 
**T**ime-**a**daptive **Co**ntrol \& **S**ensing (**TaCoS**), that tackles this challenge by optimizing over policies that besides control predict the duration of its application. Our formulation results in an extended MDP that any standard RL algorithm can solve.
We demonstrate that state-of-the-art RL algorithms trained on TaCoS drastically reduce the interaction amount over their discrete-time counterpart while retaining the same or improved performance, and exhibiting robustness over discretization frequency.
Finally, we propose OTaCoS, an efficient model-based algorithm for our setting. We show that OTaCoS enjoys sublinear regret for systems with sufficiently smooth dynamics and empirically results in further sample-efficiency gains.

When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL
Lenart Treven, Bhavya Sukhija, Yarden As, Florian Dörfler, Andreas Krause ETH Zurich, Switzerland
Abstract
Reinforcement learning (RL) excels in optimizing policies for discrete-time Markov decision processes (MDP). However, various systems are inherently continuous in time, making discrete-time MDPs inexact modeling choice. In many ap- plications, such as greenhouse control or medical treatments, each interaction (measurement or switching of action) involves manual intervention and thus is inherently costly. Therefore, we generally prefer a time-adaptive approach with fewer interactions with the system. In this work, we formalize an RL. framework, T Time adaptive Control & Sensing (TACoS), that tackles this challenge by optimiz- ing over policies that besides control predict the duration of its application. Our formulation results in an extended MDP that any standard RL algorithm can solve. RL algorithms trained on TACOS drastically
We demonstrate that state-of-the- reduce the interaction amount over their discrete-time counterpart while retaining the same or improved performance, and exhibiting robustness over discretization frequency. Finally, we propose OTACoS, efficient model-based algorithm for our setting. We show that OTACOS enjoys sublinear regret for systems with suffi- ciently smooth dynamics and empirically results in further sample-efficiency gains.
Introduction
Nearly all state-of-the of-thh-rrtttttt art RL algorithms (Schulman et al. [017] Haamoja et al. 2018 Lillicrap et al., (Schulman et al.] 2013) were developed for discrete- time MDPs. Nevertheless, continuous- time systems are ubiquitous in nature, ranging from robotics, biology, medicine, environment and sustainability etc. (cf. [Spong et al., 2006; Jones et al. 2009 Lenhart and Workman, 2007] Panetta and Fister 2003 Turchetta et al. I 2022). Such systems be naturally modeled with stochastic differential equations (SDEs), but computational approaches necessitate discretization. Furthermore, in many applications, obtaining measurements and switching actions is expensive. For instance, consider a greenhouse of fruits or medical treatment recommendations. In both cases, each measurement (crop inspection, medical exam) or switching of actions (climate control, treatment adjustment) typically involves costly human intervention. Hence, minimizing such interactions with the underlying system desirable. This underlying challenge rarely addressed in the RL literature.
In practice, a time-equidistant discretization frequency set, often manually, adjusted the underly- ing system's 's characteristic time scale. This is challenging, however, especially for unknown/uncertain systems, and systems with multiple dominant time scales (Engquist al., 2007). Therefore, for many real-world applications having global frequency of control is inadequate and wasteful. For example, in medicine, patient monitoring often requires higher frequency interaction during the onset of illness and lower frequency interactions as the patient recovers (Kaandorp and Koole, 2007).
In this work, we address this limitation of standard RL methods and propose a novel RL framework. Time-adapie adaptivee Control & Sensing (TACoS). TACoS reduces general continuous-time RL problem with underlying SDE dynamics to an equivalent discrete-time MDP, that can be solved with any
Cooreeppnndence lenart. trevendinf ethz.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

RL algorithm, including standard policy gradient methods like PPO and SAC (Schulman et al. 2017
Haarnoja et al., 2018). We summarize our contributions below.
Contributions
1. We reformulate the problem of time-adaptive continuous time RL an equaivalent discrete-time MDP that can be solved with standard RL algorithms.
2. Using our formulation, we extend standard policy gradient techniques (Haarnoja et al. (2018) andhScmam et al. (2017) to the time-adaptive setting. Our empirical results on standard RL benchmarks (Freeman et al. 2021) show that TACOS outperforms its discrete-time counterpart in terms of policy performance, computational cost, and sample efficiency.
3. To further improve sample efficiency, we propose a model-based RL algorithm, OTACOS. OTACOS uses well-calibrated probabilistic models capture epistemic uncertainty and, similar to Curi et al. (2020) and Treven et al. (2023), leverages the principle of optimism in the face of uncertainty to guide exploration during learning. We theoretically prove that OTACOS suffers no regret and empirically demonstrate its sample efficiency.
2 Problem statement
We consider a general nonlinear continuous time dynamical system with continuous state X' C RR and action U C Rdd space. The underlying dynamics are governed by (controllable) SDE:
dx: ( dt)dt + g (za; U,
Here x E X' is the state at time t, u. E the control input, f" are unknown drift and diffusion functions and B, is a standard Brownian motion in Rdd, Our goal is to find a control policy ty X x - 1l which maximizes an unknown reward b" (a,, uz) over a fixed horizon T def [0. T). i...
V
max E *E
b" (xx,, mu t)
TET
where the expectation is taken w.r.t the policy and stochastic dynamics and the class of policies over which search.
In practice, we can only measure the system state and execute control policies in discrete points in time. In this work, we focus on problems where state measurement and control are synchronized in time. We refer to these synchronized time points interactions in the following parts of this paper. Synchronizing state measurement and control contrasts standard time- adaptivee approaches such event- triigerrggeredd control (Heemels et al. 2021), where the state is measured arbitrarily high frequency and control inputs are changed only so often to ensure stability. It is also in contrast to the complementary setting, where control inputs are changing at arbitrarily high frequency but measurements are collected adaptively in time (Treven et al.) 2023). An adaptive control approach as Heemels et al. (2021) is very important for many real-world applications but similarly, an adaptive measurement strategy is crucial for efficient learning in RL (Treven et al. 2023). Our approach treats both of these requirements jointly.
We consider two different scenarios for continuous-tine control: (i) Penalizing interactions with some cost, (it) bounded number of interactions, i.e., hard constraint on control/meassremmnt steps.
Interaction cost We consider the setting where every interaction we take has an inherent cost c(x,, u.) 0. Note that consider this cost structure for its simplicity and TACoS works for more general cost functions that depend on the duration of application for the action u, or the previous action u and thus captures many practical real-world settings. We define this task more formally below
K-- Σ
max E Hell,
6" (0 Tu ((( - c( Tu (T (4
(2)
t = n. ( 11-1) + +--1, to 0,tk T, V(a,t) E X T: (2x t) E [tmins [max]]
Here Emin > 0 is the minimal duration for which we have to apply the control, fmax E [fmin T] the maximum duration, and गy is policy that predicts the duration of applying the action.
We assume that lI is the set of L Lipschitz policies

---

Bounded number of interactions In this setting, the number of interactions with the system is limited by a known amount K. Intuitively, this represents scenario where have finite budget for the inputs that we can apply and have to decide on the best strategy to space these K inputs over the full horizon. A formal definition of this task is given below
max I x TT
Σ
b' (2t,, (1fe 1 4,14 1))dt
my 19
1,-1, 0,tx = V(, t, :): (x, E [umins mmax)]
In the absence of the transition costs or the bound on the number of interactions, intuitively the policy would propose to interact with the system as frequently as possible, i.e., every fmin seconds. The additional costs/constraints ensure that we do not converge to this trivial (but unrealistic) solution.
3 TACOS: Time Adaptive Control or Sensing
In the following, we reformulate the continuous-time problem an equivalent discrete-time MDP. We first denote the state and running reward flows of Equation (T). The state flow by applying action ug for lg time reads:
D Øk+1 = E(x Ur fx),
/ fa u)ds + [ g" (x,,, u)dB.
(xx,
We assume that every time interact with the system, also obtain the integrated reward and define the reward flow as
E. (2. u, del [ (E(ar, 8), u))ds. ds-
Due to the stochasticity of (Bt)EET, the state flow Eax, u, 1) and the reward flow EB- (ær, u. () are stochastic. For ease of notation, we denote
T y- kk 2k 1x)
def
(k uk (x)], T (arg, uks 14) def E [Eg- (I uk [R)]
k def Bgg Ug 4x) 4 (ark, RM (x), wh del E.. (ark, uk (k) Dor (2ER, uky 1k)s
and the concatenated state and reward flow function, and noise as:
2 (2x,, uk, b- (æk, Ux tx)
w
(x RAk (R)
(5)
w
In this work, we search for policies that return the next control we apply and also the time for how long to apply the control.
3.1 Reforumlation of Interaction Cost setting to Discrete-time MDPs
We convert the problem with interaction costs to standard MDP which any RI. algorithm for continuous state-action spaces can solve. To this end, we restrict ourselves to a policy class:
I { X' x T + l x T TT(:; () E ['min: /maxx) is L, Lipschitz).
For simplicity, we denote by कт the component of the policy that predicts the duration of applying the action and with Nue the component that predicts the action value. The policies we consider map state æ and time-to-go to control u and the time 7 for how long we apply the action u. We define the augmented state = (æx, b, l). where is the state, b integrated reward and / time-to-go. With the introduced notation we arrive at a discrete-time time MDP problem formulation
max VR Yr,4-(( 1x,,-(00, T) rellic
max E MEllie
Σ r(sk, (8x))
(6)
K-1
s.t. 8k+1 = v (S*)), wx), so = (oo, 0, O)
Σ
Ty (okk (x) = T,
where we have
a. (8k, T (8x): wx) = =0 (x π (Bay (x)) + wk. tu TT (x)) r(8k, π (sx)) Ep- (Ix n. (xx, tx)) c(0,, mu(xxx tx)).
3

---

Pendulum Swing Up Task [Duration=10.C, etz u.t. 0.1]
Runinng meed
Fire acie
Sate
Action
2.90-
(a) add a constant switch cost 0.1 and significantly reduce the number interactions from 200 to 24. Initially, the policy applies maximal bang-bang torque for longer times, until the pendulum reaches the top. On the top. measure and change the controller at higher frequency in order to keep the pendulum stable, at the position with the highest reward.
Pendulum Swing Up Task [Duration=5.0s, 5.0s, K 5]
Raning mourd
Time for action
sin(o)
Actin
0.--
-0.-
Tima
Time
Time
Action Staps
(b) We tight bound of for the number of interactions and observe that still solve the task.
Figure I: Experiment on the Pendulum environment for the average cost and bounded number of switches setting.
3.2 Reformulation of Bounded Number of Interactions to Discrete-time MDPs
The second setting is similar to the one studied by Ni and Jang 2022). In this case, we consider the following class of policies:
I = { X'
T x 4 Ak E [K] n ( . k) is L Lipschitz).
For augmented state = (æ, b, l, k:), our policies map states time-to-go 4, number of past interactions controller u. and the time duration r for applying the action. Here the optimal control problem reads
K
max VT VT (20,, = max E
r(sk, π (Bn)) Σ k=
(7)
* PElBs
wellar
s.t. 8k+1 = Va- (8k: x (88)): wk). 80 (æo, 0, T,0),
where,
I (Sk, T (sk), wx) (Φ" (ax, π (kk; tk. k)) + wk, t Ty (2 tky k), r(8k, T (8:)) = E-- (ox. π (æ), tx. k)).
In the following, we provide a simple proposition which shows that our reformulated problem is equivalent to its continuous-tin time counterpart from Section 2
Proposition 1. The problem in Equation (2) and are are equivalent to Equation 16 and respectively.
Figure D depicts the influence of interaction cost and K on the controller's performance for the pendulum environment.
4 TACOS with Model-free RL Algorithms
We now illustrate the performance of TACOS on several well-studied robotic RI. tasks. We consider the RC car (Kabzan et al.) 2020), Greenhouse (Tap) 2000). Pendulum, Reacher, Halfcheetah and Humanoid environments from Brax (Freeman al. (02I). Thus our experiments range from environments necessitating time-adaptive control like the Greenhouse, a realistic and highly dynamic
race simulation, and very high dimensional RL task like the Humanoid 3
X 241 2 !7 We provide
at htltttp:://gttuu..oo////////////

---

We investigate both the bounded number of interactions and interaction cost settings in our experi- ments. In particular, we study how the bound K affects the performance of TACOS and compare to the standard equidistant baseline. We further study the interplay between the stochasticity of the environments (magnitude of gg) and interaction costs and the influence of On TACOS. For all experiments in this section, we combine SAC with TACOS (SAC-TACOS),
Greenhouse Temperature Tracking [Duration-25h) Optimized time between interactions
Pendulum Swing-up
Pendulum Swing-down [Duration 15seconds)
[Duration-a Oseconds
Equidistant nt time between interactions
300
Episade reward
200
200
100
100
10
30
30
20 Interactions
Interactions
# Interactions
Figure 2: We study the effects of the bound on interactions K on the performance of the agent. TACOS performs significantly better than equidistant discretization, especially for small values of K.
How does the bound on the number of interactions K' affect TACOS? We analyze the bounded number of interactions setting (cf. Section 3.2) of TACOS, studying the relationship between the number of interactions and the achieved episode reward. We compare our algorithm with the standard equidistant time discretization approach which splits the whole horizon T into T/K K discrete time steps at which an interaction takes place. We evaluate the two methods in the greenhouse and pendulum environments. For the pendulum, we consider the swing-up and swing-down tasks. The results are reported in Figure 2 The time-adaptive approach performs significantly better than the standard equidistant time discretization. This is particularly the case for the greenhouse and pendulum swing-down tasks. Both tasks involve driving the system to stable equilibrium and thus, while interaction might be necessary at the initial stages, fairly low interaction frequency can be maintained when the system has reached the equilibrium state. This demonstrates the practical benefits of time-adaptive control.
Pendulum Swing-up up [Duration=10] 10s]
Greenhouse Temperature Tracking [Duration-25h]
Reward [Without Interaction Cost]
Reward [With Interaction Cost]
Interactions
Episode reward
# Interactions
Episode reward
# Interactions
100-
175-
-50
150-
125-
0.0
2.5
5.0
7.5
10.0
10
15
20
Interaction cost
Interaction cost
Episode reward
Interactions Episode reward
# Interactions
60
150-
120-
100
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Environment stochasticity magnitude
Environment stochasticity magnitude
Figure 3: Effect of interaction cost (first row) and environment stochasticity (second row) on the number of interactions and episode reward for the Pendulum and Greenhouse tasks.
How does the interaction cost magnitude influence TACOS? We investigate the setting from Section 3.] with interaction costs. In our experiments, we always pick constant cost, i.e., c(æ, u) C. We study the influence of C' on episode reward and on the number of interactions that the policy has with the system within episode. We again evaluate this the greenhouse and pendulum environment. For the pendulum, we consider the swing-up task. The results are presented in first row of Figure Noticeably, increasing C' reduces the number of interactions. The decrease is
5

---

drastic for the greenhouse environment since it can be controlled with considerably fewer interactions without having any effect on the performance. Generally, we observe that decreasing the number of interactions, that is, increasing C, also results in a slight decline in episode reward.
How does environment stochasticity influence the number of interactions? We analyze the influence of the environment's stochasticity, i.c., the magnitude of the diffusion term g" on the episode reward and number of interactions on TACOS. Intuitively, the more stochastic the environment, the more interactions we would require to stabilize the system. We again evaluate our method on the greenhouse and pendulum swing-up up tasks. The results are reported in the second row of Figure 3 The results verify our intuition that more stochasticity in the environment generally leads more interactions. However, we observe that the policy is still able to achieve high rewards for wide range of magnitude of g" This showcases the robustness and adaptability of TACOS to stochastic environments.
RC Car
Reacher SAC-TACOS
Halfcheetah
Humanoid
1/t"
1/feval
SAC
SAC-MC
PPO-TACOS
Epicode Reword
75-
75
2000
1500
30
50-
1000
10000
300
10
10
10P
10
10
10
10.
102
1/4am
1//uii
1/haio
Epicode reward
75+
2000
80
1000
200
0
10000 Physical Time [soc]
20000
1000
20000
80000
500000
100000
Physical Time [sec]
Physical Time [sec]
Physical Time [sec]
We compare the performance of TACOS in combination with SAC and PPO with the standard SAC algorithm and SAC with more compute (SAC-MC) over a range of values for min (first row). In the second row, we plot the episode reward versus the physical time in seconds spent in the environment for SAC-TACOS, SAC, and SAC-MC for specific evaluation frequency 1/t.... We exclude PPO-TACOS in this plot as it, being on-policy, requires significantly more samples than the off-policy methods. While all methods perform equally well for standard discretization (denoted with 1/t"), ), our method is robust to interaction frequency and does not suffer a performance drop when we decrease fiin.
Figure 4:
How does tmin influence TACOS? As highlighted in Section I picking the right discretization for interactions challenging task. We show that TACoS can naturally alleviate this issue and adaptively pick the frequency of interaction while also being more computationally and tata-efficient. Moreover, we show that TACOS is robust to the choice of fmin which represents the minimal duration an action has to be applied, i.e., its inverse is the highest frequency at which we can control the system. In this experiment, we consider SAC-TACOS and compare it the standard SAC algorithm. TACOS adaptively picks the number of interactions and therefore during an episode of time T, it effectively collects less data than the standard discrete-time RL algorithm This makes comparison to the discrete-tine time setting challenging since environment interactions and physical time on the environment are not linearly related for TACOS as opposed to the standard discrete-time setting. Nevertheless, to be fair to the discrete-time method, for all environments, effectively resulting in the collection of more data for learning. Since the standard SAC algorithm updates the policy relative to the data amount, we consider version of SAC, SAC-MC AS more compute), which leverages the additional data it collects to perform more gradient updates. This version essentially performs more policy updates than SAC-TACOS and thus is computationally more expensive. Furthermore, to demonstrate the generality of our framework, we also combine TACoS with PPO (PPOSTACOS).
give SAC more physical time the system
We report the performance after convergence across different Emin in the first row of Figure 4 From our experiment, we conclude that SAC-TACOS and PPO- PPO-TACOS are robust to the choice of fmin
A standard RL algorithm would collect 7/tmin data points per episode.

---

and perform equally well when Emin is decreased, i.e., frequency is increased. This is in contrast to the standard RL methods, which have a sigmificant drop in performance at high frequencies. This observation is also made in prior work (Hafner et al. 2019). Crucially, this highlights the sensitivity the second row of Figure 3 we show of the standard RI. methods the frequency of interaction.
the learning curve of the methods for specific frequency 1/teu- From the curve, we conclude that SAC-TACOS achieves higher rewards with significantly less physical time on the environment. We believe this is because our method explores more efficiently (akin to Dabney et al. 2020 Eberhard representation of the underlying MDP.
ct al. 2022), and also learns a much stronger/continuous
Interestingly, at the default frequency used in the benchmarks 3/e", all methods perform similarly. However, slightly decreasing the frequency already leads to a drastic drop in performance for all methods. Intuitively, decreasing the frequency prevents us from performing the necessary fine-grained control and obtaining the highest performance.
While we have access to the optimal frequency 1/2" for these benchmarks, for general and unknown system it is very difficult to estimate this frequency. Furthermore, as we observe in our experiments, picking very high frequency is also not an option when using standard RI. algorithms. We believe this is where TACOS excels as adaptively picks the frequency of interaction, thereby relieving the problem designer of this decision.
5 Efficient Exploration for TACOS via Model-Based RL
In this section, we propose a novel model-based RL algorithm for TACOS called Optimistic TACOS (OTACOS). We analyze the episodic setting, where we interact with the system in episodes = ,..., N. In episode we execute the policy nne collect measurements and integrated rewards (mn.D, (n,0) ...0)...... kn bn, and prepare the data D, {(zn,1, 1 ..., (Zn,km yn.kn.) }}, def
where n = (mn.i-1, Un 1n. and Yn. (In,. bn.a). From the dataset Dı.m Usen Dr we build a model M, for the unknown function g such that it is well-calibrated in the sense of the following definition.
Definition (Well-calibrated statistical model of 4x Rothfuss et al. (023)). Let 2 def X x T. We assume xx E Enno Mn, with probability at least o, where statistical model Mn is defined as
Mn def 2 Rd.+1
de + 13 -z)= < Bn (
Here, pn, and Onj denote the j-th element the vector-valued mean and standard deviation functions phn and a respectively, and (o(5) E R2oo is a scalar function that depends the confidence level (0, 1] and which is monotonically increasing n.
Similar to model-based RL algorithms for the discrete- descrete-time setting (Kakade et al. 2020 Curi et al., 2020; Sukhija al., 2024), we follow the principle of optimism in the face of uncertainty and select the policy n,,, for both settings of TACOS (cf. Sections 3.1 and 3.2
by solving:
Th argmax max tEll
def
V (2o,
(8)
E.M,-1
where O {IC, BI} is the appropriate policy class from Section 3 Running OTACOS for N episodes, we measure the performance via the regret.
N
RN = Σ (Vr-, (2o, T) ".....((000
12ml
Here n² is the optimal policy from the class of policies we optimize over. Any kind of regret bound requires certain assumptions the regularity of the underlying dynamics (I.
Assumption 1 (Dynamics model). Given any norm I|- we assume that the drift f", and diffusion g" are L and Lg--Liipchitz continuous, respectively, with respect to the induced metric. We further assume supze ZEz ||g (2)]l A.
Assumption ensures the existence of the SDE solution under policy π. To provide bounds on the performance of OTACoS for settings Sections and we also need some assumptions on the noise and reward model.

---

Assumption 2 (Reward and noise model for Section 3.1 Setting). Given any norm II-II, we assume that running reward b Li-Lipschitz continuous, with respect to the induced metric. We further assume boundedness of the reward 0 & 6" (x, u.) S B, and interaction cost 0 S c( u) S C. The dynamics noise is independent and follows: w ~ N (0, (0,0² o (kktt))aa))
Assumption (Reward and noise model for Section 3.2 Setting). Given any norm |I-I1, we assume that the running reward is La-Lipschitz continuous, w.r. to the induced metric.
Finally, assume that learn well-calibrated model of the unknown flow 40'.
Assumption 4 (Well calibration assumption). Our learned model is an all-time wwtee-calibraate statistical model of &" i.e., there exists an increasing sequence of (8,(6)))22 such that our model satisfies the well-calibration condition, cf., Definition
Analogous assumptions are made for model-based RL algorithms in the discrete-time setting (Curi et al. 2020 Sukhija et al. 2024). This calibration assumption is satisfied 4p* can be represented with Gaussian Process (GP) (Williams and Rasmussen 2006 Kirschner and Krause 2018) models.
Theorem 2. Consider the setting from Section 3.1 and let Assumption
2 and Assumption 4 hold.
Then we have with probability least 1 6:
RN S O (3N-17312 VNIN
Now consider, the setting with bounded number of switches K, and let Assumption 7 and Assumption 4 hold. Then, we get with probability at least
RN S O KK +Ig-
(1+L,)TK V /NIN)
where D a constant. Here, with IN we denote the model-complexity after observing N points (Curi et al. 2020), which quantifies the difficulty of learning 4² For GPs, it behaves similar to the maximum information gain NN (Srinivas et al. 2009), i.e., implying sublinear regret for several common kernels (aailli al. 2027).
As proof of concept, evaluate OTACOS the pendulum and RC car environment for the interaction cost setting, As baselines, we adapt common model-based RL methods such as PETS (Chua et al. 2018) and planning with the mean to TACOS. We call them RETS-TACOS and MEAN-TACOS, respectively. The result reported Figure 5 From the figure, we conclude that OTACOS is more sample efficient than other model-based baselines and SAC- (SAC-TACOS requires circa 6000 episodes for the pendulum and 2000 for the RC car).
Pendulum Swing- Up [ouration=10s] SAC-TACOS
RC Car [Duration= 4s] PETS-TAGHE COSS
OTACoS
MEAN-TACOS
Episode Reward
N
100-
40
20-
0
10
0
20
40
60
80
100
Episodes
Episodes
Figure 5: We run OTACOS on the pendulum and RC car environment. We report the achieved reward averaged over five different seeds with one standard error.
Related Work
Similar to this work, Holt et al. (002)); and Jang (2022):
) Karimi (2023) consider continuous-
time deterministic dynamical systems where the measurements control input changes only
'The code is available htttttps///gtthb...m/m///ggg////

---

happen at discrete time steps. Moreover, Holt et al. (2023) proposes similar problem ours from Section [. where they specify cost on the number of interactions. However, their solution is based on heuristic, where measurement is taken when the variance of the potential reward surpasses a prespecified threshold. On the contrary, we directly tackle this problem at hand and propose a general framework for time-adaptive control that does not rely on any heuristics, [Karmi] (2023) adapt SAC (Haarnoja et al. 2018) to include a regularization term, which effectively adds a cost for every discrete interaction. Ni and Jang (2022) induce a soft-constraint on the duration of each action in the environment. However, all the atorementioned works propose heuristic techniques to minimize interactions, whereas formalize the problem systematically for the more general case of SDEs and show that it has an underlying MDP structure that any RL algorithm can leverage. In addition, we propose no-regret model-basee RL algorithm for this setting and analyze its sample complexity.
Temporal abstractions are considered also in the framework of options (Sutton et al.) 1999; Mankowitz ctt al. 2014 Mann and Mannor 2014 Harb et al.) 2018). However, a key difference to TACOS is that in the options framework, the agent measures the state even between the controller switches.
Learning to repeat actions Several works observe that repeating actions in the discrete-time MDPs problems such as Atari (Mnih et al., 2013 Braylan et al. 2015) or Cartpole (Hafner et al.] 2019)
significantly increase the speed of learning. However, the action repeat is fixed through the entire
rollout and treated as a hyperparameter. Durugkar et al. (2016); Vezhnevets et al. (2016); Srinivas
et al. 20177; Sharma al. 2017); Lee al. (2020); Grigsby et al. (2021); Chen et al. (202 Nam
et al. 2021): Yu et al. 2202); ; Biedenkapp et al. (2021); Krale et al. (003)) automate the selection of
action repeat, and show superior performance over the fixed number setting. Dabney et al. (2020)
empirically show that repeating the actions helps with the exploration, effectively having a similar
effect that colored noise exploration has over the standard white noise exploration (Eberhard et al.) 2022).
Continuous- time RL Following the seminal work of Doya (000) and the advances in Neural ODEs of Chen al. (2018), continuous- RL has regained interest (Cranmer et al. 2020 Grey danus et al.) 2019; Yildiz et al. 2021] Lutter et al.) 2021). Moreover, modeling in continuous time found to be particularly useful when learning from different data sources where each source is collected at a different frequency (Burns et al.) 2023 Zheng et al., 2023). An important line of work exists for modeling continuous dynamics for the case when states and actions are discrete, called
Markov Jump Processes (Kallianpur and Sundar 2014 Berger 1993 Huang et al. 2019 Seifner and
Sanchez 2023). Another line of work that is close ours event and self self-Triggered Control (Astrom
and Bernhardsson] [002] Anta and Tabuada 2010 Heemels et al. 2012 202T), where they model continuous time control systems by implementing changes to the input only when stability is at risk, ensuring efficient and timely interventions. [Teven et al. (2023) propose a no-regret continuous-timg model-basee -ased RL algorithm, which akin to OTACOS, performs optimistic exploration. They study the problem where controls can be executed continuously in time and propose adaptive measurement selection strategies. Similarly, we propose a novel model-based RL algorithm, OTaCoS, based on the principle of optimism in the face of uncertainty. We show that OTACOS has no regret for sufficiently smooth dynamics and has considerable sample-efficiency gains over its model-fres -feee-free counterpart.
7 Conclusion and discussion
We study the problem of time-adaptive RI. for continuous-time systems with continuous state and action spaces. We investigate two practical settings where each interaction has an inherent cost and where we have a hard constraint on the number of interactions. We propose novel RL framework, TACOS, and show that both of these settings result in extended MDPs which can be solved with standard RL algorithms. In our experiments, we show that combining standard RL algorithms with TACOS results in significant reduction in the number of interactions without having any effect on the performance for the interaction cost setting. Furthermore, for the second setting. TACOS achieves considerably better control performance despite having a small budget for the number of interactions. Moreover, we show that TACOS improves robustness to a large range of interaction frequencies, and generally improves sample complexity of learning. Finally, we propose, OTACOS, a no-regret model-based RL algorithm for TACOS and show that it has further sample efficiency gains.
9

---

Acknowledgments and Disclosure of Funding
This project has received funding from the Swiss National Science Foundation under NCCR Au- tomation, grant agreement 5INF40 180545, the Microsoft Swiss Joint Research Center, grant of the Hasler foundation (grant no. 21039) and the SNSF Postdoc Mobility Fellowship 211086.
References
Anta, A. and Tabuada, P. (2010). To sample or not to sample: Self-triggered control for nonlinear systems. IEEE Transactions automatic control, 55(9):2030-2042. 2042.
Astrom, K. and Bernhardsson, B. M. (2002). Comparison of riemann and lebesgue sampling for first order stochastic systems. In Proceedings of the s/ IEEE Conference Decision and Control, 2002., volume pages 2011 1-166. IEFE.
Berger, M. A. (1993). Markov Jump Processes, pages 121-138. Springer New York, New York, NY.
Biedenkapp, A., Rajan, R., Hutter, F., and Lindauer, M. (2021). Temporl: Learning when to act. In International Conference on Machine Learning, pages 914-924. PMLR.
Bobkov, S. G. and Götze, F. (1999). Exponential integrability and transportation cost related to logarithmic sobolev inequalities. Journal of Functional Analysis, 163(1):1-28.
Braylan, A., Hollenbeck, M., Meyerson, E., and Miikkulainen, R. (2015). Frame skip a powerful parameter for learning to play atari. In Workshops the twenty-ninth AAAI conference artificial intelligence.
Burns, K., Yu, T., Finn, C., and Hausman, K. (2023). Offline reinforcement learning at multiplc frequencies. In Conference on Robot Learning, pages 2041-2051. PMLR.
Chen, C., Tang, H., Hao, J., Liu, W., and Meng, 7.. (2021). Addressing action oscillations through learning policy inertia. In Proceedings the AAAI Conference Artificial Intelligence, volume 35, pages 7020-7027.
Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. (2018). Neural ordinary differential equations. Advances in neural information processing systems, 31.
Chua, K., Calandra, R., McAllister, R., and Levine, S. (2018). Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In NeurIPS.
Cranmer, M., Greydanus, S., Hoyer, S., Battaglia, P., Spergel, D., and Ho. S. (2020). Lagrangian neural networks. arXiv preprint arXiv:2003.046300
Curi, S., Berkenkamp, F., and Krause, A. (2020). Efficient model-based reinforcement learning through optimistic policy search and planning. Advances Neural Information Processing Systems, 33:14156 14170.
Dabney, W., Ostrovski, G., and Barreto, (2020). Teeperally-extended [yepsilon]-greedy explo- ration. arXiv preprint arXiv:2006. 01782.
Djellout, H., Guillin, A., and Wu, L. (2004). Transportation cost-information inequalities and applications to random dynamical systems and diffusions. The Annals of Probability,320 32(3))2702- 2732.
Doya, K. (2000). Reinforcement learning in continuous time and space. Neural computation, 12(1):219-22
Durugkar, I. P., Rosenbaum, C., Dernbach, S., and Mahadevan, S. (2016). Deep reinforcement learning with macro-actions. arXiv preprint 1606:v:600.046/ 04615.
Eberhard, O., Hollenstein, J., Pinneri, C., and Martius, G. (2022). Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In The Eleventh International Conference on Learning Representations.
10

---

Engquist, B., Li, X., Ren, W., Sanden-Eijnden, E., et al. (2007). Heterogeneous multiscale methods: review. Communications in Computational Physics, 2(3):667-43
Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., and Bachem, O. (2021). Brax differentiable physics engine for large scale rigid body simulation.
Greydanus, S., Dzamba, M., and Yosinski, J. (2019). Hamiltonian neural networks. Advances in neural information processing systems, 32.
Grigsby, J., Yoo, Y., and Qi. Y. (2021). Towards automatic actor-critic solutions to continuous control. arXiv preprint 206.::2066001166
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International conference on machine learning, pages 1861-1870 PMLR.
Hafner, D.,, Lillicrap, T., Ba, and Norouzi, M. (2019). Dream to control: Learning behaviors by latent imagination. arXiv preprint IrXiv:I912.011
Harb, J., Bacon, P.-..,, Klissarov, M., and Precup, D. (2018). When waiting is not an option: Learning options with deliberation cost. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32.
Heemels, W.. Johansson. K. H., and Tabuada, P. (2021). Event-triggered and self- ceff-triggered control. In Encyclopedia of Systems and Control, pages 724-730. Springer.
Heemels. W. P., Johansson, H., and Tabuada, P. (2012). An introduction to vent-triggered and self-triggered control. In 2012 ieee 51st ieee conference on decision and control (cdc), pages 3270-3285. IEEE.
Holt, S., Hüyük, A., and van der Schaar, M. (2023). Active observing in continuous-ting control. In shirty-seventh Conference on Neural Information Processing Systems.
Huang, Y., Kavitha, V., and Zhu, Q. (2019). Continuous-time markov decision processes with controlled observations. 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 32-39. IEEE.
Jones, D. S.. Plank, M., and Sleeman, B. D. (2009). Differential equations and mathematical biology. CRC press.
Kaandorp, G. C. and Koole, G. (2007). Optimal outpatient appointment scheduling. Health care management science, 10:217-229.
Kabzan, J., Valls, M. L, Reijgwart, J., Hendrikx, H. F., Ehmke, C., Prajapat, M., Bühler, A., Gosala, N., Gupta, M., Sivanesan, R., et al. (2020). Amz driverless: The full autonomous racing system. Journal of Field Robotics, 37(7): 126-1944
Kakade, S., Krishnamurthy, A., Lowrey, K., Ohnishi, M., and Sun, W. (2020). Information theoretic regret bounds for online nonlinear control. NeurIPS, 33:15312-15325..
Kallianpur, G. and Sundar, P. (2014). 266Jump Markov Processes. In Stochastic Analysis and Diffusion Processes. Oxford University Press.
Karimi, (2023), Decision frequency adaptation in reinforcement learning using continuous options with loon-loop policies.
Kirschner, J. and Krause, A. (2018). Information directed sampling and bandits with heteroscedastic noise. In Conference On Learning Theory, pages 358-384. PMLR.
Krale, M., Simão, T. D., and Jansen, N. (2023). Act-then-measure
reinforcement learning for partially observable environments with active measuring. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 33, pages 212-220.
Lee, J., Lee, B.JJ., and Kim, K.-F. (2020). Reinforcement learning for control with multiple frequencies. Advances in Neural Information Processing Systems, 33:3254-3264.

---

Lenhart. S. and Workman, J. T. (2007). Optimal control applied to biological models. CRC press.
Lillicrap, T. P., Hunt, J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv: (ttp:::99999977
Lutter, M., Mannor, S., Peters, J., Fox, D., and Garg, A. (2021). Value iteration in continuous actions, states and time. arXiv preprint
Mankowitz, J.J., Mann, T. A., and Mannor, S. (2014). Time regularized interrupting options. In Internation Conference on Machine Learning.
Mann, T. and Mannor, S. (2014). Scaling up approximate value iteration with options: Better policies with fewer iterations. In International conference on machine learning, pages 127-135. PMLR.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5600
Nam, H. A., Fleming, S., and Brunskill, E. (2021). Reinforcement learning with state observation costs in action-contingent noiselessly observable markov decision processes. Advances Neural Information Processing Systems, 34: 14:15550-15560
Ni, and Jang, E. (2022). Continuous control on time. In ICLR 2022 Workshop on Generalizable Policy Learning in Physical World.
Panetta, J. C. and Fister, K. R. (2003). Optimal control applied to competing chemotherapeutic cell-kill strategies. SIAM Journal on Applied Mathematics, 6366):155-1171..
Rothfuss, J., Sukhija, B., Birchler, T., Kassraie, P., and Krause, A. (2023). Hallucinated adversarial
control for conservative offline policy evaluation. In Uncertainty in Artificial Intelligence, pages 1774-1784. PMLR.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015). Trust region policy optimiza- tion. In International conference on machine learning, pages 1889-1897. PMLR.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347
Seifner, P. and Sanchez, R. J. (2023). Neural markov jump processes. arXiv:2305.19774.
arXiv preprint
Sharma, S., Srinivas, A., and Ravindran, B. (2017). Learning to repeat: Fine grained action repetition for deep reinforcement learning. arXiv preprint arXiv: 15l2::170.0055..
Spong, M. W., Hutchinson, S., Vidyasagar, M., et al. (2006). Robot modeling and control, volume 3, Wiley New York.
Srinivas, A., Sharma, S., and Ravindran, B. (2017). Dynamic action repetition for deep reinforcement learning, Proc. AAAI.
Srinivas, N., Krause, A., Kakade, S. M., and Seeger, M. (2009). Gaussian process optimization the bandit setting: No regret and experimental design. arXiv preprint urXiv:09/2.3995
Sukhija, B., Treven, L., Sancaktar, C., Blaes, S., Coros, S., and Krause, A. (2024). Optimistic active exploration of dynamical systems. NeurIPS.
Sutton, R. S., Precup, D., and Singh, S. (1999). Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2)181-211.
Tap, F. (2000). Economics-based optimal control of greenhouse tomato crop production. Wageningen University and Research.
Treven, I.., Hübotter, J., Sukhija, B., Dörfler, F., and Krause, A. (2023). Efficient exploration in continuous-ting time model-based reinforcement learning.
12

---

Turchetta, M., Corinzia, L., Sussex, S.. Burton, A., Herrera, J., Athanasiadis, I., Buhmann, J. M., and Krause, A. (2022). Learning long-term crop management strategies with cyclesgym. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems, volume 35, pages 11396-11409, Curran Associates, Inc.
Vakili, S., Khezeli, K., and Picheny, (2021). On information gain and regret bounds in gaussian process bandits. In AISTATS.
Vezhnevets, A., Mnih, V., Osindero, S., Graves, A., Vinyals, O., Agapiou, J., and kavukcuoglu, k. (2016). Strategic attentive writer for learning macro-actions. In Lee, D., Sugiyama, M., Luxburg, U.. Guyon, I,, and Garnett, R., editors, Advances Neural Information Processing Systems, volume 29, Curran Associates, Inc.
Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA.
Yildiz, C.,, Heinonen, M., and Lähdesmäki, H. (2021). Continuous-time model-based reinforcement learning, In International Conference Machine Learning, pages 12009-12018, PMLR.
Yu, H., Xu, W., and Zhang, H. (2021). Taac: Temporally abstract actor-critic for continuous control. Advances in Neural Information Processing Systems, 34:29021-22033.
Zheng, Q. Henaff, M., Amos, B., and Grover, A. (2023), Semi-supervised offline reinforcement learning with action-free trajectories. In International conference machine learning, pages 42339-424 42362. PMLR.
13

---

Contents of Appendix
Extended Theory
15
A.1 Transition Cost setting
15
Bounded number of transition
18
Additional Experiments
24
14

---

A Extended Theory
In this section, we prove Theorem 2 for OTACOS. We separate the section into two parts; proof for the transaction cost setting (Appendix A. and the proof for the bounded number of switches setting (Appendix A..)).
We start with the definitions of model complexity and sub- Gab-Sanssian random vector that we will use extensively in this section.
Definition 2 (Model Complexity). We define the model complexity as defined by Curi et al. 2020
IN
Σ
Σ
(9)
Definition 3. A random variable E R is said to be sub-Gaussien with variance proxy o² if]] 0 and we have;
222
E(e""] S
E
A random vector E R is said be sub Gaussian with variance proxy o² if any R' |ella 1 the random variable x e is o² sub Gaussian. We write subG (o²).
In the following, true state of the dynamical system æ. The augmented state at time step includes the true state of the system, 2k, the integrated reward by between k 1 and k. and the time to left go tg, i.c. 8% = a bR, tx]
will be distinguishing between the state of the augmented MDP and the
A.1 Transition Cost setting
We prove our regret bound for the transition cost case the following. We start with the difference lemma which adapts Sukhija et al. 22024, Lemma 2) to our setting.
Lemma (Difference lemma). Define V*,, + (x, T) as
k(r)-1 Σ k20
K(r)-1
Ex
x0 x where Σ
Tт (x) =
that is the total reward starting with time go and state for the policy and dynamics Q. Here the expectation w.r.t. π, & represents the expectation of the underlying trajectory induced by the policy π the dynamics T. Then have for all π,, Tb Φ.. By, T <,
Σ Vn, (#R+1, tx+1) - Vn Vn (x tets) kz0
Vx fn. (20, T) - Vv. (2o, T) = E.
(10)
where Øx+1 the state of 8kt1 = ( (Sx)) wa) and Øk+1 is the state of sk+1 = L aa.(s,, πI (sx), Wx).
Proof.
Vn. (wo, T) = Br.
Σ
r(8t, (TK)))
k20
En.50 $
r(s,, (So)) + Σ r(sk, T(Sx))
kz1
En, x- [r(8k, π (so)) + V* x- (x1,
En x- (r(81, π (8n)) + Vx r (i), (1) Vn (wo, T)] +
Em 4: V 2 (xo, T) V + (e) (1) + Vn x- (x) (1)
Vn g (Ko, T) + B .- V (2 41) Vx r (0 413
15

---

+ Bm 2. V*,.-(11,1))
Vr (21, t1)
Hence we have:
VA 4 (20, 1") - V V/ (zo,
F. x- [Vr, (21, Vx, (11,41)] En a- (V 00100(0111)) Vx (x1,4)1
By repeating the step inductively the result follows.
In the following, we leverage the result above to bound the regret of our optimistic planner w.r.t. the difference in value functions.
Lemma (Per episode regret bound). Let Assumption hold, then have with probability at least 1 8 for all n 2 O.
VT,, x- (xo, Va x (20, T) S Exr,,
Σ 120
Vin., th,, (nn.bi1,nn Vi,,, te, (In tn,
(11)
Proof. Since we choose the policy optimistically, we
F x. (Io, T) V=.. . (Io; T) S VT,, (Zo, T) V (zo. T).
Applying Lemma the result follows.
Now we derive an upper and lower bound on our value function. Lemma (Objective upper bound). Let π be any policy from the class II and consider any T > then have:
C C S Vn (x0, T) S BT.
min
Proof. Since rumning reward is bounded 0 S. ba u) S B, the number of steps K we can do in an episode is bounded with 0 < K
< T_ and switch cost is bounded 0 S c(æ, u) S C we have: Emin
T S Vn (x0, T) S BT.
mmin
A key lemma we use to bound the difference in value functions is the following from Kakade et al. (2020).
Lemma 6 (Absolute expectation Difference Under Two Gaussians (Lemma C.2. Kakade al. (020 Lel 21 ~ ((m, o²1) and 22 ~ ((u2, and oo any (appropriately measurable)
positive function g, it holds that:
B[g(z1)] B[g(<>)] S min {
||m |
Hell o²
(z)))
Furthermore, due to Assumption 4 we can also bound the distance between the next state prediction by the true system x. and the optimistic system o... Lemma 7. Let Assumption hold, then we have the following for all 2 0.
| B. S 22 de 9n ||a (Kn.k, ni (Xa. ta,k))
Proof.
||n,k+1
Bo.k+ || (ak,, n.. (Dn,ks kn,k)) )++))) + Wn.k (D Ti (Da,ki bnx) + wn,x))ll (x<", n.. (Rn.k, kn,k)) 4, (Kn,k Ti (won bn.k)
< 2 dd 8n-1 10 n (In,k kn
where the last inequality follows from the fact that x* Dn E Mn-1
16

---

Next, we relate the regret at each episode to the model epistemic uncertainty using Lemma and Lemma
Corollary 8. Lel Assumption 1 al least - 6.
2 and Assumption Folld, then have for all 2 0 with probability
2 2
C
Vn,, .. (Iv, T) - Vn x= (2u, T)
B
E
Σ
1
n 1
Tn n tn,,))l
tmin
420
(12)
Proof. From Lemma 4 we have:
v=
a. (xo. T) V (0.. T) s E
V (2, Vo,, &, (Ink+1,, (n,411) Σ k20
Lemma 6 can be applied to positive function g. We hence make a transformation and apply it to g(-) n... o (t, (,,1)
Eman T, which is positive due Lemma Moreover, Vas X:;
g(-) = Vr sen (s, (n,) + CT Bln +
C mii
TTBB C
bmin
4min
Applying Lemma we obtain:
T B+
7(B
C
VT th (In, (n,++) Va,,. t, (3n, I
E Il| Bn.4+1lll
Emin
Finally, applying Lemma we arrive at:
V. (20, T) Vx (wo, T) s 2 2o
C bnin
B+
E
Σ A20
1 (Zn,ks Tn bn,)))
Now we can prove our regret bound for the transition cost case.
Theorem Let Assumption 7
2 and Assumption 4 hold, then we have for all n 2 0 with probability
least 5.
Rx =
Σ
VT a- (Zoo, T) - V= (xo, T)
n=1
2 2 1N T8/2
B+ C
<
VNIN
02lmin
mmin
Proof. We compute:
N
R
(2o, T)
VTL x- (IO, T)
Tr.
n=1
T
C
B+ +
Σ
Bn-1E
B
E
1on (2 T (2n,ee tn
tmin
n=1
k00
2v S/d
C
N
B
I
Σ
J (20 Th ( bm.k)
tmin
p= 1 k20
2TT TT
TN
N
B
lo 1 (D T (2 11
tmin
Exmin
n= R20

---

2/ NN- -18//2
C
B+ VNIN Imin
Here the first inequality follows because of Corollary 8 the second inequality follows due to the monotonicity of sequence (Sa)nz0, the third inequality follows by Cauchy- Muuch-SSchwarz. and the last one by maximizing the term in expectation.
L.
Our regret Ry is sublinear if By v NIN is sublinear. For general well-calibrated models this is tough to verify. However, for Gaussian process dynamies, Zv equal to (up constant factors) the maximum information gain "/N (Srinivas et al. 2009) (c.f.. CCuri et al. (2020) Lemma 17)). The maximum information gain is sublinear for rich class of kernels (Vakili al. (02T]) i.e., yielding sublinear regret for OTACoS (see Sukhija et al. (2024) Theorem 2) for more detail).
A.2 Bounded number of transition
We overload the notation this section and add number of switches to the value function, such that we have V n..3 + (Io, T, 0) = V=.... (xo. T)
Lemma 10 (Per episode regret bound). We have:
V V= (2o, O)
7 a. (zo. T,0)
KK-
Σ
Van..,, (Zo lok41, Von,,,, Cen, +1)
where Bn,k+1 is the state of one step hallucinated component 8n,K+1 V Va, (8n,ks Tn (Sa,k) Wr,k) and kn, is the state of nn = D Pa (8n,ke Ta (8n,k), Wr.k).
Proof.
V .. (2op T,0) E
Σ r(8m,ks (8m, = E r(8 (n((n,))) Σ Er(sm TT( 120
E [r(8 nn (8n,0)) + V x- (2 1, 1)]
E [r(Sn, Tn (8n,0)) ))=)-))))))))))))))))) VT,,, V'T... &, (In, I 1, 1) Va,,, F (200, T,0}] +
+ E [V + (oo, T, 0) VT.,. Vm,. 4r (In, 1n 1) + VT.,, x- (Xn,1 tn,1 1)]
Vx., a, (mo: T,0) E [Vx,, 2, 1 1,,1, VT 0,, ( bn, 1)]
+ E [V..,. 11 1,,1, Vo,,, 4n (Fa,1 In,1,
Hence we have:
VT,,, T... (200 T,0) Vn D, (Eo, T,0)
-E[V 4 (1 1,,1, Va... T.. (I 1n,1, 1)] + [V>., .- 1,,,1, Vnn. t. (I 4n,1, 1)]
Repeating the step inductively the result follows and using VT., (In,K kn.K: K) = we prove the lemma.
A.2.1 Subgaussianity of the noise
In principle, we could assume that the noise wa is Gaussian and then with the same analysis obtain the regret bound. However, stochastic flows are in many cases not exactly Gaussian but only sub- Gaussian. For such noise we need can not apply Lemma aaa need to escort to different analysis. First we show that under mild assumptions on the SDE dynamics functions f and g" the resulting noise Wi is sub- Gaussian.
To derive this result we will follow the work of Djellout et al. (2004). We present the results in quite informal way, for more rigorous statements we refer the reader to Djellout et al. (2004).
Definition 4 (Wasserstein distance). Let (E, de) be metric space and let μ. be two probability measures on E. We define:
Wp(p,v)= inf E(ry)r [d(x, [d(x,))11 yET(wu))
18

---

Definition 5 (Kullback Leibler divergence). Let (E, de) be metric space and let be two probability measures on We define:
log
H(vlln)
du(m)
ifv & 1
100,
else
Definition 6 trappprasoootaioon cost information inequality). Let (E,dg) be metric space and let be a probability measure on &. We say that μ satisfy the LP-transportation cost information inequality, and for short write μ E [C),), if there exists a constant C such that for any measure on we have:
W, (2CH(v)
We now state an important theroem of Bobkov and Götze 1999) that we will use later.
Theorem 11 (From Bobkov and Götze (1999)). Let (E, de) be metric space and let μ be probability measure on & We have that μ (C) if and only if for any p-integrable and Lp- sipschitz (E, dg) + R and for any R we have:
(F(a))) fc,,
Next, we provide a condition under which ((x, u, 1) sub-uuuuan Gaussian random variable for any t E T.
Corollary 12 (Adjusted Corollary 4.1 Djellouu et al. (2004). Assume
sup 1 (z, u)ll S A, (af" - f ( i) & T (1 u) - (#: R)ll,
R uER
and denote the law of (E(x, u, t)TTTT on the space C(T, [Rd)) (space of continuous functions from ) by Pa. Then, there exist a constant C = C(A, Lge, ,T) T) such that P. (C) on the space C(T, R.de) equipped with the metric:
d(n, 72) = sup (1) - (())
tE[0,T]
Lets be a(ny) unit vector in Rd= and define:
Fe,: : C(T, ((T + +
Fa + 7t) Te
We have:
(f,)m))))e(((7))) (() - (2))
- 1l (2(1) ||e|| = (l) - 72(1)]
$ sup (t) - 72(t)] d(m, 72)
tET
Therefore for any e, the function Fe, L-Liscchitz. Since we have
E[| l (((T,
h(t) dP (7) = ut) x
the function Fe, is also Pa-integrable. Combining the latter observation with the Theorem II obtain that for any e Rd and any T we have:
Ez(a...) ( (t), el)] = Enpee
(Fealt))) < f0
Hence under the assumption of Theorem 2 for Bounded number of switches setting we have that for any E T the random variable E(x, u, [ax, u. 1)) ~ subG (C). The variance proxy C depends on A. T.
19

---

A.2.2 Lipschitness of the expected flow "
To apply analysis for the case when noise wy is any sub-Gaussian we also need to show that the dynamics function & is Lipschitz. We first start with some general results.
Lemma 13. Let f: : _ + R", A C [m] and denote B = A. If we have:
[f(EA, f( 2n)ll2 - LA 1|= Txll2
f( BB)ll2 S LB | ØBl2
then f is 2(LA + LB) Lipschitz.
Proof. We have:
[f(x)) - f(0)1l2 [f f(ū uB)1
= 1. f(žA. xB) + f(@A. f(0 @B)1l
< LA || - EAll2 + LB 2 EBl12
$ (LA + LB) (1#1 - TAll2 - | - 3B Bl
(axA TA
$ 2(LA + LB)
(B - BB
= 2(LA + LB) | - Ø|l2
Lemma 14 (Lipschitzness of f-). There exists a positive constant La, such that the flow a g- is La, -ipsschi....
Proof. We will first prove coordinate-wise Lipschitzness. We observe:
1. Lipschitness in time:
F (x.,, u)lds
//" (x." u)]ds S. (x,
$ f- (ax, t) g- (x. u, f)|l
/ E E1lff w]ll ds < F -7
2. Lipschitness in state æ: To prove this, consider the Ex, = (xx, E u. t). then we have
döx: = (f' u) (T, u))dt + (g" (x u) f (0t, u))dB
= 6f; dt + ög. dB..
Note that ||8 S Ly- |,x; and ||óg; S Lg ||اzق|| since both functions are Lipschitz. Define y/ óσx dæ; and use Ito's Lemma to get
dy. 20 (8f) di 8g dB.) + tr(6g) (óg]) Jddt
Moreover,
Ely.
2 85 + [[tr((6 J)]ss
<
2 [1 || |I] + E [8g |ds
/ (2L L |11 |ds
Note that y: = ||xx,112 so we can apply Grönwall's 's inequality to get
Sax [1
< ||ōæ 2 (2T.g )r
20

---

Moreover,
2 x=
21 g0 22
[ III - V B ||xxz11?
- |
|00
Hence we have:
24 y
1 (at, u, 1) - p (z, ue t)]l S x - 0
3. Lipschitness in action u. We denote óz, ( t) (xx, ū, t) and ou = ū Following the same steps in the proof of Lipschitzness in state we arrive at:
dy: 2dx; (ffī dt + ögî dB.) + tr(ógī (ōgi) )dt
Integration yields:
2E|0 8f
Jddss
<
2E |1 | + [[] Ids
< 2E [L |oc.1 0z || + | [)] + E 22 (1020.12 + [|ōse1()) do
$ I (3L.g- + 2T )B [. + (I.g- + 2T.2. ||ōze|| ds,
where used (a + 6)2 < 2a 26 and ab (a² (a2 + b²). Applying Grönwall's inequality results in:
E
$ |0uu11 2 (Ly- + 21 ) (3 +21..
$ |dul| 2 (Lg- + 21 )e (3 +21g. JT
Applying Lemma 13 on 2. and 3.
we have that (s:st) is
2 21.gr
31 +2r2
Ly. 2L
T
Lipschitz. Applying Lemma 13
on 1. and p (:,,tt and bounding 2
S
4 we finally obtain that o is
26
12 TT + Tig. + 2T2
g- 1262
4
T
F Lipschitz.
( (Cille Lipschitz, where
Corollary 15 (Lipschitzness of the 2.-). The cost flow 2 is O
C1 is a constant.
Proof. Same as in the proof of Lemma 14 we first show coordinate-wise Lipschitzness.
We first show Lipschitness in time:
J 6" (x,, ds
b (x, u, t) 2 (x, u,
E
J b" (xx, u)l ds
E
B(
8)] B(tt 8).
2. To obtain Lipschitzness in state observe:
(b- (x, u. t) Φb- (a, u.t) t) =
6 ("(,(,, s), -E'((( us 8), u)ds
21

---

b (3 x u, 8), u) - b=(E( u. s), u)
S L
[ u. 8) -3 us s)1l ds
S Lb '/"
3 [ u, 8)
[ u. 8)]l ds
2 + ds
1 Lb
x x
2
21igs +1 - 1) 1|ar 5ill
-1)
2. g
3. Finally, for Lipschitzness in action observe:
E 1 b' s) s)
1 (x, u, t) Db. (x, ū, ()]
$ E S (E(,
(x û, 8), u) ds
$ L 1 |E(x, u. - -xxx, ū. s)ll + 2l ēi|l ds
ull Lb (/' / [ | u. E(x, u, s)] ds
Lbit
Vllg. Leg. + 2 f
31 +1.1+2122 ds
1
ū|| + L1 ||--ii) ūill
++ 2 Lg + 21.
34 22,302223
+263
2 2 -1) 2 xl|l
3Lg- + 222
$ Lb- T+ 2v Lyr + 21g
32 g= +263
-11
-1)
1|u à
3 + 2L
Applying Lemma 13 result follows.
Corollary 16 (Lipschitzness of ②b)). The unknown function x is L Lay O D(T ( +1 Lipschitz, where D is constant.
Los
A.2.3 Regret bound
Lemma 17 (Per episode regret bound (general sub-Gaussian noise)). Consider the setting with bounded number of switches K, and let Assumption 3 and Assumption Ahold. Then, we get with
probability at least - 8:
Vi,,, (xo, T, T,K) - Va-. (No, T,K) <
<0 L
K- a K c +LL +1 E
Σ
Con-i(#n, Tn (n An.ks k))ll
k=0
Proof. Applying Lemma 5 of Curi al. 2020) the result follows.
Theorem 18. Consider the setting with bounded number of switches K. and let Assumption ! and Assumption 4 hold. Then, we get with probability at least 6:
N
RN
Va.. . (Do, T, T) - Vna x- (æo, T, K)
22

---

1 13 - 1 v /NNN
Proof. We apply Lemma 17 and Cauchy-Schward
Rg
Vmn d=
T,K, - V a- (xo,T,K)
n=1
N
K
< Σ O
Lo
K-
E
0n- (2m (an k))
n=1
0 [
PN CL1 - 23,0-11001 E
|10 1 ki Tn (Tn,k ti 11]]]12
k=0
0 (LK -
VK JKVNE NIN
Then we used the monotonicity of (Bn)n20 sequence. In the last step we first applied maximum over the collected data, then the definition of model complexity.
Here first applied Lemma 17
inequality and finally
23

---

B Additional Experiments
2.0
1.00
Instance reward
0.75
Time for action
10
Times for actions
State
Action
0.0
0.50
cost0)
sin(0)
0.25
Rewerds
-1.0
0.00
5
10
15
10
15
10
15
2 Action Stops
1
Time
Time
Time
2.01
Instance reward
lime actioon
limes for actions
0.5
0.75
State
Action
0.0
0.50
coste) Sin(e)
0.25
Rewards
-1.0
0.00
5
10
15
10
15
10
15
Time
Time
Time
Action Steps
Rewards
0.05-
0.025
Instance reward
MAM MAV
Action
0.020
cos(6)
sin(e)
0.05
0.015
T me
Time
Time
cos(()
Rewards
sin(e)
0,10
reword
0.045
State
1
0.05
0.040
0.033
0.00
10
15
0
15
15
Figure 6: Pendulum swing-down task. Row 1: interactions, optimized interaction times, Row 2: interactions, optimized interaction times, Row 3: 4 equidistant interactions, Row 4: 4 equidistant interactions.
24

---

1.00
Instarce reward
Time for acioon
0.5
0.75
State
cos(0)
0.5
0.25
Rewards
0.4
Timess for actions
-1.0
0.00
20
5
10
15
Action Stops
1.00
Time for sclion
Times actions
0.5
n
0.75
State
Action
0.6
-2.5
cos(8)
Inslance
0.4
0.25
-5.0
0.00
Rewordd
0.2
10
10
5
20
10
20
Time
Time
Action Steps
costel
2.0
0.5
pM An !
Instance reward
1.00
0.75
Tine for ecaiion
1.00
0.75
State
Action
0.0
Rewards
-2.5
-1.0
0.00
0.00
10
5 Time
10
5 Time
10
20
40
60
Time
Action Steps
Figure 7: When stochasticity of the environments increases, we need more interactions at the unstable equilibrium (Pendulum on top). The stochasticity scale goes from 0.1 to 0.5 to 1.0 from top bottom row respectively.
25

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? paper's contributions and scope?
Answer: [Yes]
Justification: We highlight the problem setting, algorithm, and theoretical and empirical results in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In Section 5 we highlight the assumptions of our work, which also correspond to the limitations of our theoretical analysis and also the setting for which our algorithm yields theoretical guarantees.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer: [Yes]
Justification: All theoretical results are accompanied by the relevant assumptions that are listed in Section 3 and we provide all proofs in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the code with all hyperparameters we used to run the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. .., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We provide the code with all the hyperparameters to run the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ptapp..cc/ubblic/uuu.ee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the experimental details either, minor part in the main paper, and the major part the accompanying code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All experiments are run with seeds and mean performance with standard error is reported in all our plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the details of compute in the readme.txt file as part of the enclosed code.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research conforms, in every respect, the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This paper proposes method improve exploration in reinforcement learning in the nonepisodic setting, and is not tied to specific applications. As such, it shares the many potential societal consequences that are associated with reinforcement learning and automation as whole, spanning from environmental impact to concerns on ethics and alignment.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
29

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release high-risk data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite all creators whose code we used in our experiments in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code is documented where applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "m7WW34JlMd", "reviewer": "Reviewer_XqGZ", "summary": "Reinforcement learning (RL) is effective for discrete-time Markov decision processes (MDPs), but many systems operate continuously in time, making discrete-time MDPs less accurate. In applications like greenhouse control or medical treatments, each interaction is costly due to manual intervention. To address this, we propose a time-adaptive approach, Time-adaptive Control & Sensing (TACOS), which optimizes policies that also predict the duration of their application. This results in an extended MDP solvable by standard RL algorithms. TACOS significantly reduces interactions while maintaining or improving performance and robustness. We also introduce OTACOS, a model-based algorithm with sublinear regret for smooth dynamic systems, offering further sample-efficiency gains.", "questions": "Thank you for the opportunity to review your paper. I have a few questions and observations outlined below.\n\n1. Real-world applications and global frequencies.\n\n- The discussion around the requirement for different global frequencies in real-world applications is intriguing (line 32), as it highlights the limitations of discrete-time sampling (lines 30-32). However, it seems this limitation is not fully addressed by continuous-time control (or the TACOS algorithm). The fundamental difference between discrete-time control and continuous-time control lies in the amount of environmental information included to compute the policy. For instance, within the total time interval $[0,T]$, discrete-time control computes the policy based on $K$ sampled data points {$t_1, t_2, \\cdots, t_K$}, resulting in an optimal policy only for those specific sampled times. In contrast, continuous-time control computes an optimal policy over the entire continuous duration $[0,T]$ and then applies it at discrete times (constrained by Equations (2) and (3)). Utilizing the policy derived from a continuous-time formulation in a discrete-time setting would provide better generalization ability to obtain higher rewards for $t \\notin$ {$ t_1,t_2,\\cdots,t_K$} but ***does not necessarily address the limitation mentioned in lines 30-32.***\n\n2. Definition of Notations:\n\n- The paper lacks precise definitions of certain notations. For example, the policy $\\pi: \\mathcal{X} \\to \\mathcal{U}$ maps states to control inputs, but the exact meaning of $\\pi_{\\mathcal{T}}$ is unclear. In line 80, it is stated that \"$\\pi_{\\mathcal{T}}$ is a policy that predicts the duration of applying the action.\" This suggests $\\pi_{\\mathcal{T}}$ is a prediction variable rather than a policy, as the term \"policy\" typically refers to decisions or actions taken. Additionally, while $t_{i} = \\pi_{\\mathcal{T}}(x_{t_{i-1}}) + t_{i-1}$ is mentioned, the role of $\\pi_{\\mathcal{T}}$ in the objective function needs further clarification.\n\nAlso have some minor issues.. \n\n- Line 65-66 and 68-69 discuss \"real-time inference\" or \"adaptive control approach,\" which is a known challenge in real-world reinforcement learning. Including references to related literature would help readers better understand these contributions as follows. \n\n[1] Dulac-Arnold, G., Mankowitz, D., and Hester, T.Challenges of real-world reinforcement learning.arXiv preprint arXiv:1904.12901, 2019.\n\n[2] Hyunin Lee, Ming Jin, Javad Lavaei, Somayeh Sojoudi, Pausing Policy Learning in Non-stationary Reinforcement Learning. ICML 2024 \n\n[3] Al-Shedivat, M., Bansal, T., Burda, Y., Sutskever, I., Mordatch, I., and Abbeel, P.Continuous adaptation via meta-learning in nonstationary and competitive environments.In ICLR, 2018.\n\n- Also please discuss what SDE stands for.", "limitations": "No potential negative societal impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- handling continuosu control is sort ot interesting since this also bridge the gap between real-world RL and RL in simulation.", "weaknesses": "My questions are listed as below", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and valuable feedback!\n\n# Questions\n \n1. *Real-world applications and global frequencies*: As you correctly described, the main difference between standard discrete-time control and our setting is that in the standard discrete-time setting, the interaction (measuring the system and changing the controller) happens only at equidistant points in time.  Our approach, also optimizes over the interaction times, that is our algorithm suggests times $t_1, \\dots, t_K$ at which we should exhibit control. In the case of the doctor and patient, the algorithm can then adapt interaction times such that the interaction frequency is not global (equidistant) but adapts to the patient's needs hence solving the problem we mentioned in lines 30-32. A good example of why this is the case are first 2 rows of Figure 7 in the pdf attached to the general response. There the task is to bring the Pendulum to stable equilibrium. We do several interactions in the beginning when the Pendulum is out of stable equilibrium (think of a sick patient) and don't do many interactions when the Pendulum is in stable equilibrium (think of a healthy patient). \n\n\n2. *Definition of notations:* We are happy to improve our notation. We agree, when we defined the policies, we were not consistent. We changed the notation in the update version of the paper as follows: The part of the policy that maps to the action space $\\mathcal{U}$ we denote as $\\pi_{\\mathcal{U}}$ and the part of the policy that maps into the time for how long we apply the policy we denote as $\\pi_{\\mathcal{T}}$. When the two parts of the policy are concatenated we write only $\\pi = (\\pi_{\\mathcal{U}}, \\pi_{\\mathcal{T}})$. \n\n3. *Additional related work:* Thanks for pointing out the additional literature that would help the reader to better understand the contributions. We added them in the updated version of the paper. \n\n4. *What does SDE stand for?* We already explained the acronym SDE in lines 22-23 of the paper. It stands for the Stochastic Differential Equation. \n\nHaving addressed your concerns,  we kindly ask you to consider revising our score. For any remaining questions, we are happy to provide further clarification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tcXarkC7Fo", "reviewer": "Reviewer_L1VF", "summary": "This paper proposes a novel time-adaptive RL method framework (TaCoS) for continuous-time systems with continuous state and action spaces. The framework shows that the settings of interactions having costs or a budget of interactions, can be formulated as extended MDPs, that can be solved with standard RL algorithms. The paper theoretically demonstrates this, and empirically verifies this. The paper also empirically demonstrates that TaCoS works across a range of interaction frequencies, and proposes OTaCoS a model-based RL approach.", "questions": "* Is the action applied from the policy kept constant until the next interaction? I presume the action is kept constant. Have you considered parameterizing the action and enabling non-constant actions or even continuous trajectories of actions between interaction times as a form of open-loop planning?", "limitations": "Yes, they are discussed in Section 5.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well written, and the intuitive explanations help the reader.\n* The problem of time-adaptive RL for continuous-time systems with continuous state and action spaces is of high significance to the community and is well-motived throughout.\n* The paper contributions are novel to the framework proposed by TaCoS and provide theoretical and empirical evidence.\n* The method works surprisingly well, with the surprise that TaCoS achieves better control performance even with a small budget of interactions.", "weaknesses": "* L154: “Intuitively, the more stochastic the environment, the more interactions we would require to stabilize the system.” the argument should either have a reference or be empirically verified.\n* Minor: Missing related work reference of (Nam et al. 2021).\n\nReferences:\n\n* Nam, HyunJi Alex, Scott Fleming, and Emma Brunskill. \"Reinforcement learning with state observation costs in action-contingent noiselessly observable Markov decision processes.\" Advances in Neural Information Processing Systems 34 (2021): 15650-15666.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the positive and valuable feedback!\n\n# Weaknesses\n\n1.  *Verification of “Intuitively, the more stochastic the environment, the more interactions we would require to stabilize the system.”:* We verify this empirically. In Figure 3 in the paper, bottom row, we analyze the setting where we increase the stochasticity of the environment. With larger stochasticity, the number of interactions increases (in the Pendulum swing-up task the number of interactions increases from 20 to 60). Does that answer your question? We also provide these 3 additional Figures where we plot trajectories at environment stochasticity scale 0.1, 0.5, and 1.0 for further intuition (look at Figure 6 in the pdf attached to the general response). We included these Figures in the appendix of the updated version of the paper.\n\n2. *Additional related work:* Thanks for pointing out to additional related work, we added it to the updated version of the paper.\n\n# Questions\n\n1. *Action between the interactions:* In all the experiments we conducted the action is kept constant between the two interactions. The case when we allow for open-loop planning between the interactions indeed makes a lot of sense and is very interesting. We leave it as an exciting future work. \n\nWe hope we addressed your concerns and questions. Please reach out if you need additional clarifications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ngR8DJJgyB", "reviewer": "Reviewer_BvQD", "summary": "The paper introduces a framework for reinforcement learning named Time-adaptive Control & Sensing (TACOS). The TACOS framework reformulates the problem of continuous-time RL into an equivalent discrete-time Markov decision process (MDP) that standard RL algorithms can solve. Additionally, a model-based version of TACOS, named OTACOS, is proposed to reduce the sample complexity.", "questions": "The model learned here differs from the typical models in model-based reinforcement learning, as it incorporates time as a transition variable, making the learning process more challenging. Could the authors demonstrate the learning performance of the model itself?", "limitations": "The paper does not show a discussion of limitations. Authors could enhance their discussion by explicitly addressing the potential limitations in real-world scenarios that involve complex, noisy, or non-stationary environments.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The introduction of the Time-adaptive Control & Sensing (TACOS) framework is a novel approach that creatively combines the challenges of continuous-time dynamics with the necessity of minimizing interactions due to cost considerations.\n2. The paper provides a strong theoretical foundation for the TACOS framework, with clear reformulation of continuous-time RL problems into an equivalent discrete-time MDP. \n3. The empirical results support the theoretical claims, with demonstrations of TACOS and OTACOS outperforming traditional discrete-time counterparts", "weaknesses": "1. Although the problem posed by the paper is novel and interesting, the solution simply involves incorporating time $t$ into the action space for learning, which a little lacks novelty.\n2. The empirical validation is somewhat limited in diversity, primarily focusing on controlled synthetic environments. This limitation might affect the generalizability of the results to more complex or noisy real-world systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, thank you a lot for your positive and valuable feedback! We will indeed incorporate the proposed ideas in the updated version of the paper.\n\n# Weaknesses\n\n1. *“Although the problem posed by the paper is novel and interesting, the solution simply involves incorporating time into the action space for learning, which a little lacks novelty.”:* We agree that augmenting the actions with time is very simple, and we see it rather as positive, since with little modification we can obtain an algorithm that hasn’t been extensively studied before and performs very well on standard deep RL benchmarks, outperforming the classical RL algorithms such as SAC and PPO. \n\n2. *“The empirical validation is somewhat limited in diversity, primarily focusing on controlled synthetic environments. This limitation might affect the generalizability of the results to more complex or noisy real-world systems.”:* We acknowledge that our algorithm has been tested only on synthetic environments and not on real-world systems. While implementation of the algorithm on real-world systems (e.g. robotic hardware) is a challenge on its own, we tried to showcase and evaluate our algorithm in simulation extensively. We tested the influence of the interaction cost on the algorithm, we showed the influence of requiring as little interactions as possible. We also studied the effects of stochasticity on our algorithm’s performance (c.f., Figure 3). Regarding the complexity of the systems, we tested our algorithm on the humanoid environment, where the dimensions of state and action space are 244 and 17 respectively and the dynamics are highly non-linear. The system is, to the best of our knowledge, considered as quite complex system in the community.\n\n# Questions\n\n1. *“The model learned here differs from the typical models in model-based reinforcement learning, as it incorporates time as a transition variable, making the learning process more challenging. Could the authors demonstrate the learning performance of the model itself?”*\nWe agree, learning the dynamics function that adds to the standard state and action also a time component as an input is more challenging. We think the best assessment of how good the model is, is whether the model-based RL algorithm converges, and in our case we can see that with an increasing number of episodes, the controller that uses the learned model obtaines higher and higher rewards on the true system, eventually converging to the best attainable reward. For example, in the Pendulum swing-up task the model is good enough to solve the task almost perfectly after only 8 episodes. \n\n# Limitations\n\n1. *“The paper does not show a discussion of limitations. Authors could enhance their discussion by explicitly addressing the potential limitations in real-world scenarios that involve complex, noisy, or non-stationary environments.”:* We added in the conclusion of the updated version of the paper that our algorithm has been only tested in the simulation and that we leave testing the algorithm on real-world environments for exciting future work. \n\nHaving addressed the reviewer’s concerns, we would appreciate if the reviewer would consider reevaluating their score for our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UQbkIxHhqd", "reviewer": "Reviewer_ZHjB", "summary": "The paper focuses on continuous-time reinforcement learning. Typically the control and the measurements of continuous-time systems occur at discrete time points. These interventions generally come with a cost. The paper proposes a time-adaptive approach to reduce the number of these interventions in an optimal manner. Therefore they formulate the problem as an MDP, for which standard RL algorithms can be used. Additionally the authors propose a model-based algorithm for their setting.", "questions": "1. Regarding the experiments for Figure 2: After what time fraction are the greenhouse and pendulum swing-down usually in the stable equilibrium? As described, if the systems are in the stable equilibrium, the discretized approach continues with interactions, while the time-adaptive approach needs less. It would be interesting to compare the interactions for the time until the stable equilibrium is reached.\n\n2. Figure 2, Pendulum Swing-up: Why does the equidistant approach perform so poorly? In Figure 1, the task could be solved with K=5 number of interactions, without high-frequency changes. Intuitively the equidistant approach should perform well with more than 10 interactions.\n\n3. How many iterations were used for learning the equidistant policy and the SAC-TaCos policy in Figure 2? It appears that learning the equidistant policy should be easier, as it has smaller input and output spaces.\n\n4. In the first row of Figure 3, the interaction cost increases, yet the reward does not decline significantly. It would be interesting to identify the point at which the reward starts to drop.", "limitations": "Limitations have been adressed adequately.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper describes an interesting setting. The problem of choosing the optimal measurement and control time-points for continuous-time systems is relevant for many processes in nature and industry.\n\n2. The reformulation of the original problems to discrete-time MDPs is well described.\n\n3. The various experiments visualize and describe the effects of the hyperparameters nicely.", "weaknesses": "1. The references in the paper lack consistency in formatting. Conference names are sometimes written in full, sometimes abbreviated, and occasionally omitted altogether.\n\n2. One major issue is the lack of details on the learning process. It would be interesting to compare the learning curves of the equidistant and the time-adaptive approach.\n\n3. The experimental section leaves some questions open -> See Questions\n\n\nMinor Weaknesses:\n- Figure 3 comes before Figure 2\n- Page 5, Line 134 says time discretization splits the whole horizon in T/K discrete time points. I think it should be K time points with length T/K\n- It is difficult to discern the decline in interactions for the greenhouse task in Figure 3 first row, which coincides with the drop in episode reward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and valuable feedback!\n\n# Weaknesses\n\n1. *“The references in the paper lack consistency in formatting. Conference names are sometimes written in full, sometimes abbreviated, and occasionally omitted altogether.”:* Thanks for spotting that, we updated the references in the updated version of the paper.\n\n2. *“One major issue is the lack of details on the learning process. It would be interesting to compare the learning curves of the equidistant and the time-adaptive approach.”:* We generally observed the trend that when we use our algorithm in a model-free fashion the learning is faster compared to the standard training. This phenomenon has been observed in several other works as well [1, 2]. We also provide learning curves in the bottom row of Figure 4. \n\n3. *“Figure 3 comes before Figure 2”:* Thanks for spotting that, we fixed the ordering in the updated version of the paper.\n\n4. *“Page 5, Line 134 says time discretization splits the whole horizon in T/K discrete time points. I think it should be K time points with length T/K”:* Indeed, thanks for spotting that. We changed it in the updated version of the paper.\n\n5. *“It is difficult to discern the decline in interactions for the greenhouse task in Figure 3 first row, which coincides with the drop in episode reward.”:* In this experiment the change in the dynamics is very slow, hence only few control changes over the entire horizon of 25h suffice. As soon as there is a certain cost for the interaction, the number of interactions needed significantly drops (you can see the drop on the left side of the plot), after the initial drop the number of interactions stays more or less the same with increasing cost, since interactions are already at the minimum of what is needed to successfully track the temperature. \n\n# Questions\n\n1. *“Regarding the experiments for Figure 2: After what time fraction are the greenhouse and pendulum swing-down usually in the stable equilibrium? As described, if the systems are in a stable equilibrium, the discretized approach continues with interactions, while the time-adaptive approach needs less. It would be interesting to compare the interactions for the time until the stable equilibrium is reached.”:* The Greenhouse and Pendulum swing-down systems we use in the experiment for Figure 2 preserve the energy (no friction or similar losses of energy) so technically they never reach exactly (due to numerical errors) the stable equilibrium and only reach its neighborhood. This is best illustrated with the first 2 rows of Figure 7 in the pdf attached to the general response, where we plot the trajectories of the Pendulum swing-down task. We show the trajectories with 4 and 5 interactions. With 4 interactions only, we can drive the system close to the stable equilibrium, while with 5 interactions we can drive it even closer. \nAt the same time, if we try to solve the task with equidistant control, the control frequency is too low, and with only 4 or 5 interactions over the horizon 15 seconds, we can not push the system close to the equilibrium, as can be seen in the bottom 2 rows of Figure 7 in the pdf attached to the general response.\n\n2. *“Figure 2, Pendulum Swing-up: Why does the equidistant approach perform so poorly? In Figure 1, the task could be solved with K=5 number of interactions, without high-frequency changes. Intuitively the equidistant approach should perform well with more than 10 interactions.”:* Comparison between the setting in Figure 1 and equidistant in Figure 2 is not as straightforward as it may seem at first. The first thing is that in Figure 1 we used a shorter horizon, i.e. 5 seconds compared to 10 seconds in Figure 2, which means that the same number of actions in Figure 2 means that each action lasts double the time. The second thing is that in Figure 1 we optimized the number of times between interactions, which results in some interactions taking 0.8 seconds and some interactions 1.3 seconds. With only equidistant interactions we need considerably more interactions to account for not being able to optimize the interaction times.\n\n3. *“How many iterations were used for learning the equidistant policy and the SAC-TaCos policy in Figure 2? It appears that learning the equidistant policy should be easier, as it has smaller input and output spaces.”:* For training in Figure 2 we used 200000 env interactions with 200000 gradient steps for SAC-TACOS. For equidistant training, we used the same amount of observations with the same amount of gradient updates. This results also in an equal amount of physical interaction time when comparing SAC-TACOS and equidistant training. Perhaps surprisingly, training with an additional degree of freedom, where we can choose the length of actions results in faster convergence as also observed in other works with similar settings [1, 2]. In our case, we choose 200000 iterations to make sure that both algorithms converge (for Pendulum swing-up and swing-down, we could in many cases obtain the same performance with only 20000 iterations). \n\n4. *“In the first row of Figure 3, the interaction cost increases, yet the reward does not decline significantly. It would be interesting to identify the point at which the reward starts to drop.”:* Thanks for pointing this out, we extended our experiments to cover that case as well. Look at Figure 8 in the pdf attached to the general response.\n\n[1] Ni, Tianwei, and Eric Jang. \"Continuous control on time.\" ICLR 2022 Workshop on Generalizable Policy Learning in Physical World. 2022.\n\n[2] Biedenkapp, André, et al. \"TempoRL: Learning when to act.\" International Conference on Machine Learning. PMLR, 2021.\n\nHaving addressed the reviewer’s concerns, we would appreciate if they could reevaluate their assessment of our work. We are happy to address any additional questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable and useful feedback. We have attached a pdf with 3 additional experiments that study the following setting:
1. *Figure 6:* We further show the effect of environment stochasticity on the frequency of interactions. We empirically show that with larger stochasticity the algorithm predicts interactions that last a shorter amount of time, on the Pendulum swing-up task.
2. *Figure 7:* We illustrated the difference between our algorithm and the algorithm with equidistant interactions around stable equilibrium in the Pendulum swing-down task. While our algorithm adapts and makes fast interactions at the beginning to bring the Pendulum down to the stable equilibrium and then doesn't interact with the system anymore, the algorithm with equidistant interactions can't adapt interactions time and can't even bring the systems to the stable equilibrium in one episode time. 
3. *Figure 8:* We extended part of the Figure 3 from the paper to the case with higher interaction cost so that the Reward [Without Interaction Cost] starts decreasing. 

We included the results of the additional experiment in the appendix of the updated version of the paper. We would be happy to further update the paper if there are any remaining questions or feedback from the reviewers.

Meta-review: The reviewers found the work novel, well-written, and well-supported by experiments. They all had questions that were properly answered by the authors, thus, they all raised their scores during the rebuttals. It would be great if the authors revise the paper to include these answers that properly addressed the reviewers' questions.
Decision: Accept (poster)