Beyond Information Gain: An Empirical Benchmark for  Low-Switching-Cost Reinforcement Learning
Authors: Shusheng Xu
Yancheng Liang
Yunfei Li
Simon Shaolei Du
Yi Wu
Venue: Accepted by TMLR ()

Abstract: A ubiquitous requirement in many practical reinforcement learning (RL) applications is that the deployed policy that actually interacts with the environment cannot change frequently. Such an RL setting is called low-switching-cost RL, i.e., achieving the highest reward while reducing the number of policy switches during training. It has been a recent trend in theoretical RL research to develop provably efficient RL algorithms with low switching cost. The core idea in these theoretical works is to measure the information gain and switch the policy when the information gain is doubled. Despite of the theoretical advances, none of existing approaches have been validated empirically. We conduct the first empirical evaluation of different policy switching criteria on popular RL testbeds, including a medical treatment environment, the Atari games, and robotic control tasks. Surprisingly, although information-gain-based methods do recover the optimal rewards, they often lead to a substantially higher switching cost. By contrast, we find that a feature-based criterion, which has been largely ignored in the theoretical research, consistently produces the best performances over all the domains. We hope our benchmark could bring insights to the community and inspire future research. Our code and complete results can be found at https: // sites. google. com/ view/ low-switching-cost-rl 

Published in Transactions on Machine Learning Research (01/2023)
Beyond Information Gain: An Empirical Benchmark for Low- Coit Reinforcement Learning
Shusheng Xu', Yancheng Liang', Yunfei Li', Simon S. Du', Yi W.
IIIS, Tsinghua University, 2 University of Washington, Shanghai Qi Zhi Institute {xuss20, liangyc19, Liyf20)0maili tsinghua. edu. cn
ssduêcs.
asdu@cs.washingee. washington. edu, jxwuyi@gmail, com
Reviewed on OpenReview: http:://pppaaeersr...ee////om/t2
Abstract
A ubiquitous requirement in many practical reinforcement learning (RL) applications is that the deployed policy that actually interacts with the environment cannot change fro- quently. Such an RL setting is called low-switcing-coot RL, i.c., achieving the highest reward while reducing the number of policy switches during training. It. has been cent trend in theoretical RL research to develop provably efficient RL algorithms with low switching cost. The core idea in these thcoretical works is to measure the information gain and switch the policy when the information gain is doubled. Despite of the theoretical ad- vances, none of existing approaches have been validated empirically. We conduct the first empirical evaluation of different policy switching criteria on popular RL testbeds, including a medical treatment environment, the Atari games, and robotic control tasks. Surprisingly, although informattion-gain-aaaes methods do recover the optimal rewards, they often lead to a substantially higher switching cost. By contrast, find that feature-based criterion, which has been largely ignored in the theoretical research, consistently produces the best performances over all the domains. We hope our benchmark could bring insights to the community and inspire future rescarch. Our code and complete results can be found at https: // sites. google. com/ vi view/ low- swt wwitchinn-coss-rl
1 Introduction
Reinforcement Learning (RL) has been successfully applied to solve sequential-ddecision problems in many real-world scenarios, such as medical domains (Mahmnud al. 2018), robotics (Gu et al. 2017 Kalashnikov et al. 2021). hardware placements (Mirhoseini et al.] 2017 2020), and personalized recommendation (Zheng et al. 2018h. When performing RL training in these scenarios, it is often desirable to restrict the agent from adjusting its policy too often due to the high costs and risks of deployment. For example, updating treatment strategy in medical domains requires a thorough approval process by human experts {Almi- rall et al.] 2012);; in personalized recommendations, it is impractical to adjust the policy online based on instantancous data and a more common practice is to aggregate data in a period before deploying a new policy (Zheng et al. 2018);; in problems where we use RT. to learn hardware placements (Mirhoseini et al. 2017), is desirable to limit the frequency of changes to the policy since it is costly to launch a largo-scale evaluation procedure on hardware devices like FPGA; for learning to control a real robot, it may be risky inconvenient to switch the deployed policy when an operation is being performed. In these settings, it is a requirement that the deployed policy, i.c., the policy used to interact with the enviromment, could keep unchanged as much as possible. Formally, we would like our RL algorithm to both produce a policy with the highest reward and at the same time reduce the number of deployed policy switches (i.e., a low switching cost) throughout the training process.
In low-switehing-cost RL, the central question is how to design criterion to decide when update the deployed policy. A good criterion should first ensure the optimal reward, then reduce the switching cost.

---

Published in Transactions on Machine Learning Research (01/2023)
Low-switchinggcost RL has recently become hot topic in theoretical RL research.
Many works have studied
low-switching-coss RT and its simplified bandit setting extensively (Auer et al. 2002 Cesa-Bianchi et al. 2013 Bai et al. 2019 Ruan et al. 2020 Gao et al.) 2021 "Zhang et al. 2020a b). The core notion in these thcoretical works is information goin. Specifically, they update the deployed policy only if the measurement of information gain is doubled, which leads to optimaaltt bounds for the final rewards. We suggest. the readers refer to the original papers for detailed thcoretical results. Algorithmic details will be presented in Section 3.2 However, no empirical evaluation has been conducted on whether these theoretically-guidee criteria are in fact effective.
In this paper, we aim to provide systematic benchmark studies on different policy switching criteria empiri- cally. We remark that our experiment scenarios are based on popular deep RL enviromments that are much more complex than the bandit or tabular cases studied in the existing theoretical works. Empirically, we find that, although information--aiin-aass
criteria do achieve the optimal rewards, they perform poorly roducing the switching cost. By contrast, a feature-basod critcrion, which is rarcly studied by recent the- oretical works, performs surprisingly well across all the experiment domains. We summarize our empirical findings and provide intuitive justifications for the nattrre-bssd method. We hope this benchmark could bring new insights to the community from both thcoretical and practical perspectives and inspire future RL research towards this largely ignored feature- based direction.
Our contributions are summarized below.
We conduct the first empirical study for low-switching-cost RL on environments that require modern RL algorithms, ie., Rainbow (Hessel et al.] 2018) and SAC (Haarnoja al.] 20018), including a medical environment, 56 Atari games"] and MuJoCo control tasks. We test theoretically guided switching criteria based on the information gain as well as other adaptive and non-adaptive criteria.
We find that a feature- based criterion produces the best overall quantitative performance, which largely outperforms the theoretically guided ones based on information gain. This suggests new research direction largely ignored by the existing literature.
We summarize our empirical findings from a practical RL perspective and provide initial justifications and intuitions for the feature- based criterion.
2
Related Work
Low switching cost algorithms were first studied in the bandit setting (Auer et al. 2002 Cesa-Bianchi al.
2013 Existing work RL with low switching cost is mostly theoretical. To our knowledge, Bai et al.
2009 is the first work that studies this problem for the episodic finite-horizon tabular RL setting. BBii et al.
(2019) gave a low-regret algorithm with an O OHH3 S log (K)) local switching upper bound where 5 is the number of stats, A is the number of actions, H is the planning horizon and K is the number of episodes the agent plays. The upper bound was improved in (Zhang et al. 2020b a). Recently, APEVE (Qiao et al., 2022) achieves a switching cost of O (HSA A log log (K)), which estimates the transition kernel, and adopts policy elimination by evaluating all policies using the estimated transition kernel. However, evaluating all policies is impossible in many real applications, and it is also non-trivial to estimate the transition kernel in many complex environments (model-based RL). In this paper, we focus on model-free RL and the design of the switching criterion, which is compatible with most existing DRL algorithms.
Offline RL (or Batch RL. can be viewed as a close but parallel variant. of low-switching-cost RI, where the policy does not interact with the environment at all and therefore does not incur any switching cost. Offline RL mothods typically learn from given dataset (Lange et al., 2012 Levine et al. (020) on a varicty of domains including education (Mandel el al.; 2014), dialogues (Jaques el al.) 2019) and robotics (Kumar) et al. 2020). However, previously provided high-quality dataset, which is typically generated by experts well trained policy, may not always be feasible in many practical situations. In contrast with offline
There are total of 57 Atari games. However, only 56 of them (excluding the "surround" game) are supported by the atari-py package, which we adopt as our RL training interface.
2

---

Published in Transactions on Machine Learning Research (01/2023)
RL, which optimizes reward subject to strong assumptions, i... given dataset and zero policy switch, low- switching-cost. RL aims to reduce the swilching cost while mainlaining similar sample efficiency and final reward compared to its unconstrained RL counterpart without any additional algorithmic assumptions. In thooretical literature, offinee RL Antos et al.] (2008); Tosatto et al. (2017): Xie & Jiang 2020); : Wang et al. (2000) and low-switching-cont RI. are also two parallel fields.
Some works interpolate offline and online RL, ie., semi-batch RL (Singh al,, 1995 Lange et al., 2012). which update the policy many times during training is not their focus. Matsushima et al. (2021) is perhaps the most related work to us, which repeats offline training for a fixed (i.e., 10) iterations. In each deployment iteration, the proposed algorithm collects transitions using a fixed deployed policy, trains an ensemble of transition models and updates a new policy for the next iteration. However, even though the proposed model-based RL method in Matsushima et al. (2021) outperforms a collection of offline RI. baselines, the final rewards are still substantially lower than online SAC even after consuming an order of magnitude more training samples. In our sctting, an effective policy switching criterion should preserve comparable overall sample efficiency and the final rewards to online RL. There are also works that aim to reduce the interaction frequency with the environment rather than the switching cost (Gu et al. 2017 Hu et al. 2021) which are parallel to our focus.
large batch of transitions. However, reducing the switching cost
3 Reinforcement Learning with Low Switching Cost
3.1 Notation
Markov Decision Process: We consider the Markov decision model (S,A, Y, P), where S is the state space, A is the action space, y is the discounted factor, r: S x A R is the reward function, po is the initial state distribution, and P(x |æ, a) denotes the transition probability from state x to state x' after taking action A policy क : S + A is a mapping from a state an action. An episode starts with initial state Zo Po- At each step h in this episode, the agent chooses an action On from T(z)) based the current state Thi receives reward r(zk, as)) and moves to the next state Th+1 ~ P(-(zn,,)) We assume an episode will always terminate, so each episode e = ((x%,a))) < h < He} will always have a finite horizon He. The goal of the agent is to find a policy n' which maximizes the discounted expected reward, -i o (r ] Ideally, we also want the agent to consume as few training samples
x² arg max, Ee
possible to learn ...
In standard RL, transition (o), an 2n) is always collected by single policy x. Therefore, whenever the policy is updated, a switching cost is incurred. In low-switching-cos RL, we have two separate policies, a deployed policy Taep that interacts with the environment, and an ouline policy Wont that is trained by the underlying RL algorithm. These policies are parameterized by 6dep and 6onl respectively. Suppose that we totally collect K samples during the training process, then each transition step k the agent is interacting with the environment using a deployed policy adep- After the transition is collected, the agent can decide whether to update the deployed "ep A++1 by the online policy *n ; i.e., replacing
Low-switching-cosst RL:
k+1
Odep with loal, according to some switching criterion J. Accordingly, the switching cost is defined by the number of different deployed policies throughout the training process, namely:
K-1
Cswitch = Σ Tmrdep
1-1 # mdopp.
(1)
In standard RL, Cswitch equals the number of policy updates during the training process, which can be millions when using DRL algorithms. Such a large Cswitch is unacceplable in the applications of medical domains and hardware placements. The goal of low-switching-com that learns x² using K samples while produces the smallest switching cost Cswitch- Particularly in this paper, we focus on the design of the switching criterion J, which is the most critical component that balances the final reward and the switching cost. The overall workflow of low-switching-oost RI. is shown in Algorithm
RL to design an effective algorithm
3

---

Published in Transactions on Machine Learning Research (01/2023)
In the following content, present collection of policy switching criteria and techniques, including those inspired by the information gain principle (Sec. 3.2 well other non-adaptive and adaptive criteria (Sec. 3.3 All the discussed criteria are summarized in Algorithm 2 of Appendix B.
A Algorithm General Workflow of Low-Switching-Con Cost RL 1: Initialize parameters ooll depp an empty replay buffer D, Cswitch +-
2: for + to K do
Select aj by (xx)) execute action og and observe reward Tki state 2k+1 Store (Iy,, ak, rk, Tk+11 in D
Update Poul using D and an off-policy RL algorithm
6: if [+) == true then
7:
Update Odep t Ooul, Cswitch & Cswitch +
3.2 Switching via Information Gain
Existing theoretical studies propose to switch the policy whenever the agent has gathered sufficient new information. Intuitively, if there is not much new information, then it is not necessary to switch the policy. The information gain is measured by the visitation count of each state-action pair or the determinant of the covariance matrix. We implement these two criteria as follows.
Visitation-based Switching: Following Bai et al. (2019), we switch the policy when visitation count of any state-action pair reaches an exponent (specifically 2 E in our experiments). Such exponential scheme is theoretically justified with bounded switching cost tabular cases. However, it not fcasible to maintain precise visitations for high-dimensional states, following count-based exploration (Tang et al., 2017), adopt random projection function map the states to discrete vectors by Φ(x) sign(A g(x)), and then count the visitation to the hashed states as an approximation. A is fixed matrix with i.dd entries from unit Gaussian distribution 1)0, and g is a flatten function which converts to 1-dimensional vector.
Information- Information-mmatri--eeer matrix-based Switching: Another algorithmic choice for achieving infrequent policy switches is based on the property of the feature covariance matrix (Ruan al. 2020; Gao et al. 2021). i.e., Aj = Ee:H,2r 24 (a "frh:aa)))/ (x af AI, where e denotes a training episode, h means the h-th timestep within this episode, and vi denotes mapping from the state-action space to feature space. For each episode timestep h. [Abbasi- Yadkori et al. 2011) switch the policy when the determinant of A% doubles. However, we empirically observe that the approximate determinant computation can be particularly inaccurate for complex RL problems. Instead, we adopt an effective alternative, i.e., switch the policy when the least absolute eigenvalue doubles. practice, we again adopt a random projection function to map the state to low-dimensional vectors, Φ(x) sign(A -g()), and concatenate them with actions to get (bx a) = [o(x), a).
3.3 Other Switching Criteria
The information-gain-bent criteria are proposed under bandit or tabular cases theoretically, and our exper- iment scenarios are much more complex. We further investigate some other criteria empirically.
Non-adaptive Switching Critcrion: This simplest strategy switches the deployod policy every n timesteps, which we denote as "FIX n" in our experiments. Empirically, we notice that "FIX 1000" is surprisingly effective criteria which remains effective in most of the scenarios without hyperparameter tuning. So we primarily focus on "FIX 1000" our non-adaptive baseline. In addition, We specifically use "None" to indicate the experiments without the low-switching-cos constraint where the deployed policy keeps synced with the online policy all the time.
Policy-based Switching Criterion: Another straightforward criterion switch the deployed policy when the action distribution produced by the online policy significantly deviates from the deployed policy. Specifically, in discrete action domains, sample batch of training states and count the number of states
4

---

Published in Transactions on Machine Learning Research (01/2023)
where actions taken by the two policies differ. We switch the policy when the ratio of mismatched actions exceeds threshold op- For continuous domains, we use KT.-divergence to measure the policy differences.
Beyond directly consider the difference of action distributions, an- other possible solution for measuring the divergence between two policies is through the feature representation extracted by the neural networks. Hence, we consider a feature-based switching criterion that decides switch policies according to the feature similarity between different Q-networks. Similar to the policy- based criterion, first sample batch of states B from the experience replay buffer, and then extract the features of all states with both the deployed Q-network and the online Q-network. It is noteworthy that there is an MLP with a single hidden layer between the feature representation we adopted and the final output of the w-network. For state I, the extracted features are denoted as faop(z) and [an) respectively. The similarity score between faep and foul on state x is clefined as
baased Switching Criterion:
(I)) fon((z)) [[ (a:)ll [[)m))
sim(x)
(2)
We then compute the averaged similarity score on the batch of states B
Ezes sim(x)
sim IB)
(3)
[B]
With hyper-parameter oy [0, 1J, the feature-based policy switching criterion changes the deployed policy whenever (B) S oy.
3.4 Implementation
In this section, summarize the implementation details of above criteria.
KeeeetCCheckinn Empiricallyy we also find an effective implementation enhancement, which produces lower switch costs and is more robust across different environments: we only check the feature similarity when an episode resets (i.e., a new episode starts).
Estimation of Feature Similarity & Action Difference: For policy-based and feature-based criteria, we uniformly sample 512 from recent 10,000 transitions, and compare the action differences or feature similarities between the deployed policy and the online policy on these sampled transitions. We also tried other sample size and sampling method, and there is no significant difference.
For the switching threshold (i.e., the mismatch ratio op in policy-based criterion and parameter of in feature-based criterion), we perform a rough grid search and choose the highest possible threshold that still produces a comparable final policy reward.
Switching Threshold:
4 Experiments
In this section, we conduct experiments to evaluate different policy switching criteria on Rainbow DQN (Hessel et al. 2018 and SAC (Haarnoja et al. [2018). For discrete action spaces, we study the Atari games and the GYMIC testbed for simulating sepsis treatment for ICU patients which requires low switching cost. For continuous control, we conduct experiments on the MuJoCo (ooorov et al., 2012) locomotion tasks.
The details of GYMIC, Atari and MuJoCo environments are introduced in Appendix C.
4.1 Evaluation Metric
For GYMIC and Atari games whose action spaces are discrete, we adopt Rainbow DQN to train the policy; for MuJoCo tasks with continuous action spaces, we employ SAC since is more suitable for continuous action spaces. We evaluate the efficiency among different switching criteria these environments. For the Atari and Mujoco environments, we conduct experiments over seeds. For the GYMIC environment, the
We list. the hyper- parameter search space Appendix C.
5

---

Published in Transactions on Machine Learning Research (01/2023)
experiments are repeated over 10 seeds due to the high variance in this environment. Implementation details and hyper-parameterss are listed in Appendix C.
We benchmark different policy switching criteria by measuring the accumulated reward as well as the switch- ing cost in all the environments. We report the average results of all the Atari games and detailed results of representative games in the main paper due to space limit. Results for every single Atari game can be found at our project website
To better quantitatively measure the effectiveness of policy switching criterion, we propose new evaluation metric, sennard-threshold Switching Improvement (RST), which takes both the policy performance and the switching cost improvement into account. Specifically, suppose the standard online RL algorithm (i.e.,
the "None" selting) can achieve an average reward of At with switching cost C Now, low-switching-con RL criterion J leads to a reward of Ry and reduced switching cost of Cj using the same amount of training samples. Then, we define RSI of critcrion J as
(1-sign( signi ( fd] log max (c:
RST(T) = Rg ( >
(4)
where [[:] is the indicator function and ØnsI is reward-lolerance threshold indicating the maximum allowed performance drop with the low-switching-co cost constraint applied. In experiments, choose fixed threshold parameter ØRSI 0.2. Intuitively, when the performance drop is moderate (i.e., within the threshold TsII), RSI computes the logarithm of the relative switching cost improvements; while when the performance decreases siguificantly, the RST score will be simply 0.
1
4.2 Results
We compare the performances of all the criteria presented in Sec. 3 including unconstrained RL ("None"), non-adaptive switching "FFix_1000") policy-based switching ("Policy"), feature-based switching ( ("Feature") and two information-gal variants, namely visitation- ("Visitation") and information-matte based ("Info") criteria.
None
Info
GYMIC: This medical environment is relatively simple, and all the criteria achieve similar learning curves unconstrained RL as shown in Fig. 1 However, the switching cost of visitation-based criterion is significantly higher - it almost overlaps with "None". While the other information-gale variant, i.c., information- matiiimmattttia-aaaaeee crito- rion, performs much better in this scenario. Overall, feasure-based criterion produces the most satisfactory switching cost without hurt to sample efficiency. GYMIC simulates the process of treating ICU patients. Each episode starts from random state of random pa- tient, which only provides binary reward (+15 or -15) at the end of each episode for whether the patient is cured or not. We notice that even a random policy can achieve an average reward of 13, which is very close to the maximum possible reward 15. Therefore, possible improvement by RL policics can be small this casc. However, we want to emphasize that despite similar reward performances, differ- ent switching criteria lead to significantly different switching costs.
Feature
Visitation
Policy
FIX, 1000
Reward
Switching Cost
104
10
0.0
0.1M
0.2M
0.3M
0.4M
0.5M
Figure 1: Results on GYMIC. Top: rewards vs. steps. Bollom: switching costs. The switching cost of " "Visita- tion" " almost overlaps with "None".
Atari Games:
We then compare the performances of different switching criteria in the more complex Atari games. The state spaces in Atari games are images, which are more complicated than the low-dimensional states in GYMIC. Fig. 2 shows the average reward and switching of different switching criteria across all the 56 games, where the feature-based solution leads to the best empirical performance. We also remark that the non-adaptive
hhttps://sites wow.lls.ooooooooooooooooooo woogle.com//aee wom/viee//eooo wiww//ow-switching-co We Ô here since some RI. algorithm may not update the policy every timestep. 5 We also tried variant. of RST that remove the log function. The results are shown in the appendix D.
6

---

Published in Transactions on Machine Learning Research (01/2023)
None
Feature
Policy
Info
Visitation
FIX 1000
Human Normalized Score
Normalizer aammllzzed Switching cost
30t
300%
10'
250%
200%
10:7
150%
10-2
10-4
50%
0.5
1.0
1.5
0.0
2.5
3.0
Step
Step
Figure 2: The average results on Atari games. We compare different switching criteria across 56 Atari games. Left:human normalized reward. Right: the average switching cost, which is normalized by the switching cost of "None" and shown in log scale.
baseline is particularly competitive in Atari games and outperforms all other adaptive solutions except the feature-based one. We can observe that information- gain variants produce substantially more policy switches. while the feature-based solution switches less frequently.
In addition, we notice that the policy-based solution is particularly sensitive to op in order to produce desirable policy reward with low switching cost, while feature- based solution is easier to tune, which suggests that the neural network features may change more smoothly than the action distribution. Peward
To validate this hypothesis, visualize the action difference and feature difference of the unconstrained Rainbow DQN the Atari game "Pong" throughout the training process in Fig. 3 Note that in this case, the deployed policy is synced with the online policy in every training step, so the difference is merely due to a single training update. However, even in this unconstrained setting, the difference of action distribution fluctuates significantly. By contrast, the feature change is much more stable.
Reward
Action Difference
Action Difference
Feature Similarity
MuJoCo Control: We evaluate the effectiveness of different switching criteria with SAC on all the 6 MuJoCo continuous control tasks. The results are shown in Fig. In general, we can still observe that the feature-based solution achieves the lowest switching cost among all the baseline methods while the policy-based solution produces the most unstable train- ing. Interestingly, although the non-adaptive baseline has relatively high switching cost than the feature-based one, the reward curve has less fluctuation, which also suggests a future rescarch direction on incorporating training stability into the switching criterion design.
Feeture Similarity
Steps
Figure 3: Action difference and feature sim- ilarity tested on Pong. Higher feature simi- larity or lower action difference implies that two policies are closer.
Average RSI Scores: We also report. the RSI scores of different policy switch- ing criteria on different domains. For each domain, we compute the average value of RSI scores over cach individual task in this domain. The results are re- ported in Table we can observe that the feature- based method consistently pro- duccs the best quantitative performance across all the 3 domains.
Avg. RSI
Feature
Policy
Info
Visitation
FIX 1000
GYMIC
9.30
3.93
8.39
0.0
6.91
Atari
3.60
3.22
2.33
1.83
3.25
Mujoco 8.26
5.26
4.69
1.90
6.68
Table 1: RSI (Eq ØRSI 0.2) of different criteria over different domains. We take unconstrained RI. (i.e., "None") performance the RSI reference, so the RSI value for "None" is always zero.
We list. the results øRsI Table and also evaluate RST using other dsst- See Appendix D) for details.

---

Published in Transactions on Machine Learning Research (01/2023)
Policy
Visitation
FOX_1000
Swimmer
HalfCheetah
Ant
50-
Reward
10000
25
5000
1.5
1.0
1.5
0.1
0.6
0.8
1.0
1e6
1c6
100
Switching
10a
10P
10a
10
10a
10'
1.5
1.5
0.6
0.8
1.0
Walker2d
Hopper
Humanoid
4000
Reward
5000
5000
2500
2000
2500
1.0
1.5
1.0
1.5
Switching
100
10P
10a
Cost.
10a
10a
10'
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
le6
166
166
Figure 4: The results on MuJoCo tasks.
In addition, we apply f-test for the switching costs among different criteria. The testing procedure is carried out for each pair of methods over all the experiment trials across all the enviromments and seeds. For Atari and MuJoCo, the results show that there are significant differences in switching cost between any two criteria (with p-value < 0.05). For GYMIC, "Nonc" and "Visited" is the only pair with no significant difference. It is also worth mentioning that RSI for "Visitation" for GYMIC is 0, which also shows the switching costs of "Visitation" and "None" are nearly the same.
Comparison with Offline RL:
QR-DQN Feature
QR-DQN
CQL
We also compare the low-switching- cost setting with offline RL empiri- cally. In Fig, 5 We adopt the im- plementations of a popular offline RL algorithm CQL. (Kumar et al. 2020) and its online counterpart QR-DQN (Agarwal et al. 2020b) provided by the authors] CQL, we adopt the same dataset with the author [5] We apply the feature-based switching criteria to QR-DQN and conduct the exper- iments on 4 Atari games consid- ered CQL. From Fig. 5 we ob- serve that feature-based criterion can also reduce the switching cost while maintaining the performance. CQL learns from fixed dataset, it switches just once and doesn't 't aim to recover the online performance, parallel to our focus.
Asterix
Breakout
Reward
6000
200
2000
0.
0.0
0.2
0.4
0.
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Switching
Cost
10*
10"
For
0.2
1.0
Pong
Qbert
1=2
Reward
20
7500
0-
5000
0.0
0.2
0.
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Switching
Cost
20.-
10"
10
0.2
0.5
0.0
Step
167
167
Figure Comparison with offline RL. Offline RL does not aim recover the online performance.
htttpe///githubb www.//vvraa.uuar2907///
8he datasel. Alari comes from hterr wtw.hhhb..cccooooleeeeeerheaaaaa

---

Published in Transactions on Machine Learning Research (01/2023)
In Fig. 6 further compare the low-sw www-switching-cost setting against several recent offiine RL methods Fujji
moto & Gu 2021 Fujimoto et al. 2019 K Kumar et al. 2020) on MuJoCo tasks. We use the author-provide
implementations for the offline algorithm H00 and evaluate these methods on the D4RL (Fu et al. 2020) benchmark. For TD3 (online) and TD3+ Feature, we remove the behavior cloning regularization term and the Maature-normaliiige process in the codebase of TD3+BC, which are designed for offline training. The results Fig. 6 indicate that the performances of offline methods are highly dependent on the dataset's quality. No offline algorithm obtains comparable performances with online training on the "random" "medium-replay" dataset. However, the low-switching- cost setting does not need dataset. It can recover online training performances and reduce switching costs by orders of magnitude.
(Online)
(Offline)
COL [Offline)
(Offline)
haffcheetah-random
halftheetah-medium-rro
haffcheetah-medium
halfcheetah-expert
expert
10000
10200
10000
Reward
7500
7500
7500
10200
5200
5000
5200
2100
2500
2100
Switching
Cost
10"
30
10"
104-
301-
106
100
100
0.2
367
hopper-random
mopper-medium-replay
hopper-medium
hopper-expert
hopper- soappermediuu-exxprr
Reward
3000
3000
3000
3000
9000
3000
2000
2000
3200
1800
1000
1.0
0.0
D.0
0.2
0.4
0.4
0.
1.0
10
Switching
109
309
100
300-
300
Cost
30F
30P
20a
207
0.2
1.0
1n7
367
3w7
walker2d-random
walleer2d-meduu--repaa
walker2d-mer 2d-medium
walker2d-expert
4200
4C00
Reward
4000
2000
2000
1200
1000
0.6
1.0
0.0
0.2
0.4
0.6
0.4
0.6
0.5
1.0
0.0
0.8
1.0
Switching
300
Cost
300-
30
200
20
307-
300
307
107
207
307
0.0
0.4
0.6
0.8
1.0
0.0
0.7
0.4
0.6
1.0
0.0
0.7
0/4
0.6
0.8
0.0
0.7
0.1
0.6
0.5
0.0
0.
0.1
0.6
0.8
1.0
Stop
Stop
Stop
Stop
Stop
Figure 6: Comparison with offline RL on the D4RL benchmark. The performances of offline methods are highly dependent on the quality of the datasct. "Random" uses 1M samples from a randomly initialized policy. "Expert" uses 1M samples from a policy trained to completion with SAC. "Medium" uses 1M samples from a policy trained approximately 1/3 the performance of the expert. "Medium-Replay" uses the replay buffer of a policy trained up the performance of the medium agent. "Medium- uses a 50-50 split of medium and expert data.
4.3 Findings and Suggestions
Information-gain-basee criteria empirically result in high switching cost. The thcoretically guided information-gain. criteria always recover the rewards but suffer from higher switching cost
htttp::///gthub..mmm/stjjjj/
htsp::///gttuub..om///tujj

---

Published in Transactions on Machine Learning Research (01/2023)
in practical benchmark tasks with modern RL algorithms. We remark that most theoretical works focus simplified bandits or tabular MDPs when analysing mathematical properties of Infornation-gain--aasel methods. These theoretical conclusions are carried out on much simplified scenarios compared with popular deep RL testbeds that we consider here. In addition, these theoretical analyses often ignore the constant lactors, which can be crucial for the practical use.
The difference in the output action distribution may not be a good indicator to measure policy variation. Many RL works utilize action difference (KL divergence) to mcasure the difference between policies Schulman et al. (2015); Galashov et al. (2019); Rudner et al. (022)). In our low-switching- cost RL experiments, we empirically observe that the policy-based criterion is particularly sensitive to the threshold parameter. Further analysis (Fig. 3 reveals that the output action changes drastically throughout the training process. Even after the reward converges, we can still observe a significant fluctuation of action difference between the online and the deployed policies. By contrast, feature similarity appears to be more stable. Therefore, we suggest that, when comparing two learning policies in practice, feature- based measurements can be strong candidates for stable policy difference estimates, in addition to popular action-based metrics. We remark that this finding could be applicable to the general RL use cases.
reature-based switching criterion empirically performs the best and should be worth more research efforts. Across all domains, feature-based criterion consistently achieves the lowest switching cost while maintains the similar reward to the case without the low-switching-cow constraint. Feature learning has been widely investigated from the theoretical perspective in other domains, such as supervised learning Allen-Zhin & Li (2020); : Du al. (2021): (Triprraenii al. (2020), constrastive learning Tian et al. (2021) and Agarwa et al. (2020a). However, there is little attention on feature-based methods in the thcoretical literature of low-switching-cost RL. We suggest that more research attention should be put on the feature- based direction. We also provide some intuitive justifications for the feature-basee criterion using some recent advances in representation learning theory and hope this insight could inspire future research.
To illustrate the insight, we consider the following setting. Suppose we want to learn f()), a representation function that maps the input to a k-dimension vector. We assume we have input-output pairs (æ, v) with y = (w, (x)) for some underlying representation function f*() and a linear predictor presentation, let us assume we know w, and our goal is to learn the underlying representation which together with gives us 0 prediction error. Suppose we have data sets Dj and D2. We use Dj to train an estimator of fr, denoted as fl, and D, D2 to train another estimator of f*, denoted as f2+2, The training method is empirical risk minimization, i.e.,
RR For ease of
min JEF [D (..u))D,
(y - f()))) and
F1 min JEF
Σ
(u - (w, ((x)))
[D1
U22
(x.))ED;UD,
where F is some pre-specified representation function class.
The following theorem suggests if the similarity score between f and fl+2 is small, then fl is also far from the underlying representation f
Theorem 1. Suppose f² and f2+2 are trained via aforementioned scheme. There exist dataset D1, D2. function class F and w such that if the similarity score between f and f1 on D142 smaller then a, then the prediction error of f1 on D142 is at least. 1
Theorem I suggests that in certain scenarios, if the learned representation has not converged (the similarity score is small), then it cannot be the optimal representation which in turn will hurt the prediction accuracy. Therefore, if the similarity score is small, we should change the deployed policy.
5 Conclusion
In this paper, we focus on the low-switching-cod cost reinforcement learning problem and take the first empir- ical step towards understanding how to design an effective solution for reducing the switching cost while
10

---

Published in Transactions on Machine Learning Research (01/2023)
maintaining good performance. By systematic empirical studies practical benchmark environments with modern RT algorithms, we find that there exists a large theory-practice gap in the information-gain-bae switching criteria. Meanwhile, the feature-based solution performs the best but has been rarely discussed by the thcoretical community. We raise this open question and provide some intuitive justification for the beature-based criterion in this paper. We hope this benchmark project could bring new insights for the community.
Limitation and Social Impact We remark that there is still a great research room towards designing more principled method for low-switching-cost RL. In addition to the feature- based criterion, another important direction is to give provable guarantees for the settings with a large state space or a function approximalor beyond existing works on labular cases (Bai et. al. 2019 Zhang el al. 2020b a). We believe our paper is just the first step on this important problem, which could serve as a foundation towards great future research advances. Medical domains are one potential application for this work, and we believe that it won't result in a worse negative social impact than traditional RL algorithms in this domain. Nevertheless, at the beginning of training, this policy may would not be optimal and should be applied with great caution. A possible direction is to initialize the policy using offline data, and to improve the policy using online RL algorithm with low switching costs, which we leave as future work.
References
Yasin Abbasi- Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In John Taylor, Richard S. Zemel, Peter L. Bartleut, Fernando C. N. Pereira, and Kilian Q Wein- berger (eds.), Advances in Neural Information Processing Systems 24: 25th Arnual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain, pp. (312-2320) 2011.
Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal- can, and Н. Lin (eds.), Advances in Newrol Information Processing Systems, volume 33, pp. 20095 20107. Curran Associates, Inc., 2020a. URL https::/prooceedinss www.eeddiinnnnuuus.c.cmmmm meerrpps.cc/pppr//220////. .pdf.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on oflline rein- forcement learning. In International Conference Machine Learning, pp. 104-114. PMLR, 2020b.
Zeyuan Allen-Zhu and Yuanzhi Li. Backward feature correction: How deep learning performs deep learning.
arXiv preprint Xki::2001.04413 2020.
Daniel Almirall, Scott N Compton, Meredith Gunlicks-Stoessel, Naihua Duan, and Susan A Murphy. Design- ing a pilot sequential multiple assignment randomized trial for developing an adaptive treatment strategy. Statistics in medicine, (1())):1887-19 1902, 2012.
András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with belmman-residual minimization based fitted policy iteration and single sample path. Machine Learning, 71 (1 129, 2008.
Peter Auer, Nicolo Cesa- and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256 2002.
Yu Bai, Tengyang Xie, Nan Jiang, and Yu- Yu-Xiang Wang. Provably efficient q-learning with low switching cost. In Advances in Neural Information Processing Systems, pp. 8002-8011, 2019.
Nicolo
Ofer Dekel, and Ohad Shamir. Online learning with switching costs and other adaptive adversaries. In Advances in Newral Information Processing Systems, pp. 1160 1168, 2013.
Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Loe, and Qi Lei. Few-shot learning via learning the representation, provably. In International Conference on Learning Representations, 2021. URL https: ///ppa/eriivv..t///oommttttt///p
11

---

Published in Transactions on Machine Learning Research (01/2023)
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning. aixie preprint axXi:2004.072199, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20 20145, 2021.
Scott. Fujimoto, David Meger, and Doina Precup. Off-policc deep reinforcement learning without exploration. In International conference on machine learning, pp. 2062-2062. PMLR, 2019.
Alexandre Galashov, Siddhant Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guil- laume Desjardins, Wojtek M. Czarnecki, Yee Whye Tch, Razvan Pascamu, and Nicolas Hoess. Information asymmelry in KI-regularized RL. In International Conference on Learning Represenlations, 2019. URL
Minbo Gao, Tianle Xie, Simon S Du, and Lin F Yang. A provably efficient algorithm for linear markov decision process with low switching cost. arXiv preprint arXiv:2101.00494 2021.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3389-3396. IEFE, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with a stochastic actor. In International Conference on Machine Learning, pp. 1861-1870 PMLR, 2018.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.
Han Hu, Kaicheng Zhang, Aaron Hao Tan, Michael Ruan, Christopher Agia, and Goldie Nejat. A sim-to-real pipeline for deep reinforcement learning for autonomous robot navigation in cluttered rough terrain. IEEE Robolics and Aulomation Lellers, 6(4):6569 6576, 2021. doi: (0.09//RR...0 http/RRR..222.1.0 2021.003551..
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off off-policy batch deep reinforcement learning of implicit human preferences in dialog. arv preprint arXic:1907.00 00456, 2019.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning scale. CoRR, abs/2104.08212. 2021. URL https::/aax.vv.or///bb/////// w0tp://///.aa/....././/./ 08212
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce-
ment learning. arXiv preprint arXiv:2006.00 042.0000077779 07779, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pp. 45-73. Springer, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.016/3 2020.
Mufti Mahmud, M. Shamim Kaiser, Amir Hussain, and Stefano Vassanelli. Applications of deep learning and reinforcement learning to biological data. IEEE Trans. Neural Networks Learn. Syst., 29(6):2063 2079, 2018.
Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications to educational games. In AAMAS, pp. 1077-1084, 2014.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment--ffeieen reinforeement learning via model-based offiine optimization. In International Conference on Learning Representations, 2021.
12

---

Published in Transactions on Machine Learning Research (01/2023)
Azalia Mirhoseini, Hieu Pham, Quoe V. Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean. Device placement optimization with reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2430 2439. PMLR, 2017.
Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Jiang, Ebrahim M. Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Sungmin Bac, Azade Nazi, Jiwoo Pak, Andy Tong, Kavya Srinivasa, William Hang, Emre Tuncer, Anand Babu, Quoc V. Le, James Laudon, Richard C. Ho, Roger Carpenter, and Jeff Dean. Chip placement with deep reinforcement learning. CoRR, abs/ abs/2004.10777 10746, 2020.
V. Mnih, K. Kavukeuoglu, D. Silver, Andrei A. Rusu, J. Veness, Marc G. Bellemare, A. Graves, Martin A. Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, S. Petersen, C. Beattie, A. Sadik, Toannis Antonoglou, H. King, D. Kumaran, Daan Wierstra, S. Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529 533, 2015.
Dan Qiao, Ming Yin, Ming Min, and Yu-Xiang Wang. Sample-officient reinforcement learning with loglog (t) switching cost. arXiv preprint arXiv:2902.0638. 2022.
Yufei Ruan, Jiaqi Yang, and Yuan Zhou, Linear bandits with limited adaptivity and learning distributional optimal design. arXiv preprint orXiv:2007.01900 2020.
Tim G. J. Rudner, Cong Lu, Michael Osborne, Yarin Gal, and Yee Whye Teh. On pathologies in KL- regularized reinforcement learning from expert demonstrations. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL https: httpaa/wwwws..ttttffoorrttttssss
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897) Lille, France, 07 09 Jul 2015. PMLR. URL htttps://procceednngg mlr. http://s........//3//3/3 hatarprrees//3///chhaaaaaa5 htmml
Satinder P Singh, Tommi Jaakkola, and Michael I Jordan. Reinforcement learning with soft state aggregation. In Advances in neural information processing systems, pp. 361-368 1995.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, F. Turck, and P. Abbeel. Exploration: A study of count-based exploration for deep reinforcement learning. ArXiu, abs/1611.04711, 2017.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pp. 10268-10278. PMLR, 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 (EEE/RS) International Conference on Intelligent Robots and Systems, pp. 5026 5033, 2012. doi: 10.1109 TTT.://.........6666666666 2012.6386109.
Samuele Tosatto, Matteo Pirotta, Carlo D'Eramo, and Marcello Restelli. Boosted fitted q-iteration. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3434-344 PMLR, 06-11 Aug 2017. URL httpss://proceedinng. wwwiiingg.mm....... www.press//70/tssattoi//// html
Nilesh Tripuraneni, Michael Jordan, and Chi Jin. On the theory of transfer learning: The importance of task diversity, Advances in Neural Information Processing Systems, 33:7852 7862, 2020.
Ruosong Wang, Dean P Foster, and Sham M Kakade, What are the statistical limits of offine rl with linear function approximation? Xix preprint or arXim:2000.1 11895, 2020.
13

---

Published in Transactions on Machine Learning Research (01/2023)
Tengyang Xie and Nan Jiang. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Jonas Peters and David Sontag (eds.), Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), volume 124 of Proceedings of Machine Learning Research, pp. 550 559. htt.eeeddinn......rreeess/eeeeee
PMLR, 03-06 Aug 2020. URL httpps:/proceeddnggs. mlr. hwt...pprees//1122xxx220.....
Zihan Zhang, Xiangyang Ji, and Simon S. Du. Is reinforcement learning more difficult than bandits? near-oplimal algorithm escaping the curse of horizon. arXiv preprinl arXiv:2009.1330 13500, 2020a.
Zihan Zhang. Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learning via reference- advantage decomposition, 2020b.
Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li.
DRN: deep reinforcement learning framework for news recommendation. In Pierre-Antoine Champin, Fabien Gandon, Mounia Lalmas, and Panagiotis G. Ipcirotis (ods.), Proceedings of the 2018 World Wide Web Conference World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018, pp. 167 176. ACM, 2018.
14

---

Published in Transactions on Machine Learning Research (01/2023)
A Project Statement
All of our code and detailed results can be found at our project website, including installation and running instructions for reproduciblity, and the results of individual games in Atari. The code is hosted at GitHub under the MIT license. We believe that there is not any potential negative societal impact of this project
All the environments are public accessible RL testbods. For experiments with MuJoCo engine, we register free student. license. For offline RL experiments, we conduct the experiments based on the codebase and dataset at httts:://gtthen
com wwwaaaaaaaaaaa9rr11a http://///0//////a/ttttt//////// and https://gitthut com/ woogooooeoereeeeeee batch_r") r1. respoctively, which are under Apache-2.0 2.0 license. We ensure that such a Atari game dataset does not contain any personally identifiblee information or offensive content. All experiments are run on machines with 128 CPU cores, 256G memory, and one GTX 2080 GPU card.
B Algorithm Details
B.1 Deep Off-policy Reinforcement Learning:
Deep Q-learning (DQN) (Mnih et al. 2015) is perhaps the most popular off-policy RL algorithm leveraging deep neural network to approximate Q(x,, a). Given the current state 2h, the agent seleets an action on greedily based on parameterized Q-function Qo(ure: a) and maintain all the transition dala in the replay For each training step, the temporal difference error is minimized over batch of transitions sampled from this buffer by ( = (1i41 max a'
max Qo(En+1,a') Qo(an, an)))
where I represents the parameters of the target Q-network, which is periodically updated from A. Rain- bow Hessel et. al. 2018) is perhaps the most famous DQN variant, which combines six algorithmic en- hancements and achieves strong and stable performances on most Atari games. In this paper, we adopt a deterministic version M of Rainbow DQN as the RL algorithm for the discrete action domains. We also adopt count-based exploration [Tang et al., 2017 as deterministic exploration bonus.
For continuous action domains, soft actor-critic (SAC) (Haarnoja et al. 2018 is the representative off-policy RL algorithun. SAC uses neural networks parameterized by 0 to approximate both Q(s, a) and the stochastic policy Ta Ta(a)s) Q-network is trained to approximate mntropy-regularin
expected return by minimizing
(o = El(rr + (Q2 - a log (a1()))1)
- Qo(an, on) Plaa ~ [+1[a++)]]
(6)
where a is the entropy coefficient. We omit the parameterization of since is not updated w.r.t Cq. The policy network ne is trained to maximize Cx Earr [Q(I, a) - log To (a(x))]
B.2 Detailed switching criteria
We summarize the switching critoria mentioned in Sec. 3 Algorithm
C Experiment Details
C.1 Environments
GYMIC GYMIC is an OpenAl gym environment for simulating sepsis treatment for ICU patients to an infection, where sepsis is caused by the body's response to an infection and could be life-threatening. GYMIC built an environment to simulate the MIMIC sepsis cohort, where MIMIC is an open patient EHR dataset from ICU patients. This environment generates a sparse reward, the reward is set to +15 if the patient recovered and 15 if the patient died. This environment has 46 clinical features and a 5 x action space.
11 Standard Rainbow adds random noise network parameters exploration, which can be viewed constantly switching policies over a random network ensemble. This contradicta the low-switching-ronk constraint.
15

---

Published in Transactions on Machine Learning Research (01/2023)
Algorithm 2 Switching Criteria (I in Algorithm 1) & Non-adaptive Switching input environment step counter k, switching interval n output bool(k mod == 0)
Policy-based Switching
input deployed and online policy ddeps Tonl, state batch IB, threshold Op Compute the ratio of action difference or KL divergence for deep and Toal on as 8. output bool(6 2 op)
Feature-basee switching input Encoder of deployed and online policy fdeps font; state batch B, threshold of Compute sBm(B) via (3)
output bool(sim(BB) - ar)
D Visitation-bart based Switching input the current visited times of state-action pair n(((zx), ax) output bool(n()(zz)) a&) E (1, 2, (....)
informationnmattinn based Switching input episode timestep h, current covariance matrix Ag, old A at previous switch time Compute the least absolute eigenvalues of and oy
output bool(uf, 2 x of))
Atari 2600 Atari 2600 games widely employed to evaluate the performance of DQN based agents. We evaluate the elficiency among different switching criteria on 56 games.
MuJoCo MuJoCo contains continuous control tasks rumning in a physics simulator, we evaluate different switching criteria on locomotion benchmarks.
For GYMIC and Atari games whose action space is discrete, we adopt Rainbow DQN to train the policy, for MuJoCo tasks with continuous action spaces, we employ SAC since more suitable for continuous action space.
C.2
of deterministic Rainbow
Table 2 lists the basic except that the experiments on GYMIC adopt Htarget 1K. Most of these hyper-parameters are the same those in the original Rainbow algorithm. For count-based exploration, the bonus is set 0.01.
of Rainbow. All of our experiments share these
Table 3 lists the extra lyper-parameter: for experiments GYMIC. Since there are 46 clinical features in this environment, we stack 4 consecutive states to compose a 184-dimensional vector as the input for the state encoder. The state encoder is 2-layer MLP with hidden size 128.
Table shows the additional parametrs for experiments on Atari games. The observations are grey- scaled and resized tensors with size 84 84, and consecutive frames are concatenated as single state. Each action selected by the agent is repeated for 4 times. The state encoder is composed of convolutional layers with 32, 64 and 64 channels, which use 8x8, 4x4, 3x3 filters and strides of 4, 1, respectively.
C.3 Hyper-pane parameers of SAC
We list the hyper-paramcters SAC Table 5 We adopt Adam as the optimizer, the learning rate is set 0.001 for MuJoCo control tasks except for Swimmer which 0.0003, and the temperature parameter a is set to 0.2 for Mujoco control tasks except for Humanoid which is 0.05. Other hyper-parametent are the
16

---

Published in Transactions on Machine Learning Research (01/2023)
Parameter
Value 20K
Hatart
learning rale
0.0000625
Huazet (Atari)
8K
Htarget (GYMIC)
1K
Adam
1.5 10 proportional
Prioritization type
Prioritization exponent s
0.5
Prioritization importance sampling
0.4 1.0
Multi-step returns
Distributional atoms Natoms
51
Distributional Vinis Vmax
[-10, 10]
Discount factor 7
0.99
Memory capacity
1M
Replay period
Minibatch size
32
Reward clipping
[1, 1]
Count-base bonus
0.01
Activation function 8
ReLU
Table 2: The basic hyper-parameters of Rainbow. used the Adam optimizer with learning rate 0.0000625 and e = 1.5 x 10-4 Before training the online policy, we let the initialized random policy run 20K steps to collect some transitions. The capacity of the replay buller is 1M. During the training process, we sample 32 transitions from the replay buffer and update the online policy every four steps. The reward is clipped into [-1, 1] and ReLU is adopted as the activation function. For replay prioritization we use the recommended proportional variant, with importance sampling from 0.4 to 1, the prioritization In addition, employ Natoms 51, Vmin 10, Vmax 10 for distributional RL and 3 for multi-step returns. The count-based bonus is set to 0.01.
0.5.
Parameter
Value
State Stacked
Number of layers for MLP
2
Hidden size
128
Table 3: Extra hype-parameters for the experiments in GYMIC, we stack consecutive states and adopt a 2-layer MLP with hidden size 128 to extract the feature of states.
same for these tasks. The update frequency "50/50" means we perform 50 iterations to update the online policy per 50 environment steps.
C.4 hyper-parameter. of different criteria
For the switching threshold in policy-based and Reature-based criteria (i.e., the mismatch ratio op in policy- based criterion and parameter oy in featuro-based critcrion), perform rough grid scarch and choose the highest possible Uhreshold that still produces a comparable final policy reward. For GYMIC, we tried op e (0.25, 0.5) and a E {0.97, 0.98, 0.99) and finally adopted op 0.5 and 0 0.97. For Atari games, we tried op E (0.25, 0.5) and o E {0.98, 0.99). For MuJoCo, we tried о, E (0.5, 1.0, 1.5 and o E {0.7, 0.7,
12 (0.5, 1.0, 1.5) is the thresholds of KT..
17

---

Published in Transactions on Machine Learning Research (01/2023)
Parameter
Value
Gray scaling
True
Observation
(84, 84)
Frame Stacked
Action repetitions
Max frames per episode
108k
Encoder channels
32, 64, 64
Encoder filter size
8 x 8, x 4, 3 x 3
Encoder stride
4, 2, 1
Table 4: Additional hyper-parameters for experiments Atari games. Observations are grey-scaled and rescaled 84 84. 4 consecutive frames are staked as the state and each action acted for four times. We limit the max number of frames in an episode 108K. The state encoder consists of convolutional layers.
Parameter
Value
Warm-up samples
10K
Learning rate
0.001
Optimizer
Adam
Temperature parameter
0.2
Discount factor y
0.99
Memory capacily N Number of hidden layers Number hidden nodes per layer
1M
2
256
Target smoothing coefficient
0.005
Update frequency
50/50
Target update interval
Minibatch size
128
Activation function
ReLU
Table 5: Hyper-parametee paramettrssss of SAC algorithms on MuJoCo control tasks.
D Discussion on RSI
D.1 RSI without log function
We choose log function because observe that switching costs of different criteria vary in orders of magni- tudes. We also tried variant of RSI that removes the log function, which is
max (c.)).
1 [[g > (1- sign( (i-ssg((A)onss R
The results are shown in the following Table B (ansı = 0.2 ). We think these numbers have exaggerated the differences among different criteria and we finally adopt the log function.
Feature
Policy
Info
Visitation
Fix 1000
GYMIC
10943
51
4408
1000
Atari
1028
184
93
57
643
MuJoCo
10826
136
128
10
833
Table 6: The results of RST variant that removes the log function, the results exaggerate the differences among different criteria.
18

---

Published in Transactions on Machine Learning Research (01/2023)
D.2 Evaluate RSI with different ØRSI
evaluate RSI using different onsT, and list the results in Table 7 Since the rewards obtained by different criteria on GYMIC are almost the same, RSI remains mostly unchanged when using different ØRsI. The only exception is when Ønsi = 0. this case, RSI of "Feature" and "Info" become 0. This because a positive RSI requires strictly better or equal reward performance than "None" criterion when øRsI = 0. For Atari and MuJoCo, the overall trend is that larger asI results a larger RSI, since a larger osss tolerates a wider performance range, And we can observe that for most ORST the conclusions remain unchanged.
GYMIC
ORSI
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Feature
0.0
9.30
9.30
9.30
9.30
9.30
9.30
9.30
9.30
9.30
Policy
3.93
3.93
3.93
3.93
3.93
3.93
3.93
3.93
3.93
3.93
Info
0.0
8.39
8.39
8.39
8.39
8.39
8.39
8.39
8.39
8.39
Visitation
FIX 1000
6.91
6.91
6.91
6.91
6.91
6.91
6.91
6.91
6.91
6.91
Atari
RSI
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Feature
1.68
3.06
4.00
3.90
4.50
4.70
5.02
5.13
5.42
5.54
Policy
1.60
2.74
3.22
3.66
4.15
4.50
4.81
4.81
4.81
4.88
Info
0.81
1.65
2.33
3.14
3.62
3.84
4.01
4.17
4.36
4.45
Visitation
0.84
1.14
1.83
2.02
2.11
2.11
2.17
2.17
2.17
2.25
FIX 1000
1.87
2.96
3.25
4.24
4.63
4.83
4.93
4.93
5.03
5.13
MuJoCa
RSI
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Feature
2.55
7.66
8.26
8.26
8.26
8.26
8.26
8.26
8.26
8.26
Policy
2.64
4.25
5.26
6.01
6.01
6.01
6.01
6.01
6.01
6.01
Info
2.96
4.53
4.69
4.83
4.83
4.83
4.83
4.83
4.83
4.83
Visitation
1.23
1.86
1.90
1.94
1.94
1.94
1.94
1.94
1.94
1.94
FIX 1000
4.60
6.22
6.68
6.91
6.91
6.91
6.91
6.91
6.91
6.91
Table 7: The RSI of different ØRsI
E
Proof of Theorem
Proof. We let w = .1.,,...,111 R be a k-dimensional all one vector. We let
F= (s f(x) (20 x)) 1,, 20 ((1((12)))))) x)) - 1,..., (0((v,, 1)} c { RR*
with o(.) being the ReLU activation function and v; E {ei, -ei where e; E B& denotes the vector that only the i-th coordinate is and others are O. We assume is an even number and ak is an integer for simplicity. We let the underlying f be the vector correspond to (e., We let Dj {(e1, 1). (e2, (E(--)))) 1)} and D2 = ((Gn-o) (ex, 1)). Because we use the ERM training scheme, it is clear that the training on D, U D2 will recover 1", i.e., fl+2 = because if it is not f is better solution (f* has 0 error ) for the empirical risk. Now if the similarity score between f² and fi+2 is smaller than a, it means for fl, its corresponding {u( {u(1-o]++:- .,..,, are not correct. In this case, f' prediction error is at least a on D1 U2, because it will predict 0 on all inputs of D2.
We define (00) 0.5
19


Reviews:
[{"review_id": "7id705BA9F", "reviewer": "Reviewer_5Vfo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is an interesting work which I believe may provide some useful insights for the RL community. Low-switching-cost RL is an important setting in many potential applications and gaining a better understanding of the empirical performance of common algorithms/paradigms can be a major step forward in landing some real-world impact for the field. My comments/questions below:\n- In the second to last paragraph in Section 3.2 (visitation-based switching), I suggest adding citation(s) related random projection and hashing.\n- 3 seeds is too small, I strongly recommend 5 at the very least (ideally 10). I think this is a critical to this paper since the work itself is empirical and it is important that we are given strong empirical evidence to support the conclusions. This is particularly problematic for the GYMIC environment which showed high variance across seeds.\n- Any ideas why non-adaptive methods worked so well on Atari?\n- Several questions related to Section 4.3\n  - Could you give some intuition or any conjectures you may have on the theoretical-practical discrepancies of information-gain based methods? One thing you mentioned is that the constant factor often gets ignored, do you have any sense what may influence the constant factor?\n  - In this work, you take the last layer of the Q network as the features and compute a similarity score. Since the policies themselves are induced (either directly or indirectly) by the Q network, on a high level, feature-based methods and policy-based methods are of a similar flavor. Could you comment on your intuition on why the former is more stable compared to the latter? I think overall some discussion on the connection between the three adaptive approaches (information gain, feature-based, policy-based) would be a nice touch\n- Could you give some brief comments on the infinite horizon setting? For example, are there any particular challenges in extending this work to the infinite horizon case?\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is an empirical study on the topic of low-switching cost RL — which is an RL paradigm that seeks to maximize the return of the agent as well as the number of times the deployed policy which interacts with the environment switches during training. The authors tested a wide range of switching criteria on a set of popular DRL benchmarks provided evidence on the empirical performance of these criteria both in terms of minimizing the switching cost and maximizing the reward.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As I mentioned previously, due to the empirical nature of this work, 3 seeds is too small to obtain any conclusive results, I suggest doing the experiments with a few more seeds (5 should be a bare minimum). My other suggestions are listed above", "broader_impact_concerns": "As suggested by the authors, one potential application of this work is within medical domains where low switching cost is an essential criterion for policy deployment. I suggest the authors further discuss the broader ethical and societal impacts of this application.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dC0rO9YIWf", "reviewer": "Reviewer_yPPd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: This paper conducts the first empirical study for low-switching-cost RL on environments that require modern RL algorithms, i.e., Rainbow (Hessel et al., 2018) and SAC (Haarnoja et al., 2018), including a medical environment, 56 Atari games1 and 6 MuJoCo control tasks. This paper finds that a feature-based criterion produces the best overall quantitative performance, which largely outperforms the theoretically guided ones based on information gain.\n\nWeakness: The paper seems has already been edited with red font. Overall, the result relatively good to me. The only thing I am concerning is that the Non-adaptive switching, policy-based switching, and feature-based switching seems are heuristic approaches and it would nice if some theoretical guidance can be conducted to validate those criteria, for example, under some weak assumptions, convergence of low-switching can be guaranteed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers low-switching-cost RL setting, i.e., achieving the highest reward while reducing the number of policy switches during training. It has been a recent trend in theoretical RL research to develop provably efficient RL algorithms with low switching cost. The core idea in these theoretical works is to measure the information gain and switch the policy when the information gain is doubled. This paper conducts the first empirical evaluation of different policy switching criteria on popular RL testbeds, including a medical treatment environment, the Atari games, and robotic control tasks. Surprisingly, although information-gain-based methods do recover the optimal rewards, they often lead to a substantially higher switching cost. By contrast, this paper find that a feature-based criterion, which has been largely ignored in the theoretical research, consistently produces the best performances over all the domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "One weakness I already mentioned in above. Another aspect is a related paper [1] is not discussed. I urge the authors to include this related paper in the revision.\n\n[1] Sample-Efficient Reinforcement Learning with loglog(T) Switching Cost, ICML 2022.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "y0DewOLJin", "reviewer": "Reviewer_wNAL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strong points:\n\n- Low-switching-cost RL is relatively well-presented and motivated in the introduction, even though I suggest later a dedicated section on motivating low-switching-cost RL (just for even better readability)\n- The paper is well-written and clear. The algorithms being compared are quite easy to understand from the paper.\n- The proposed method seems original, even if the paper mainly claims to be empirically comparing existing methods.\n- The experiments seem well-designed and consider real-world tasks (GYMIC). The fact that there are also Atari games and Mujoco tasks indicates a very general approach, and broad comparison, on many kinds of environments (with image observations, and continuous actions)\n- The comparison against CQL (batch RL) is welcome\n\nWeak points:\n\n- The definition of low-switching-cost RL is much clearer when read in combination with the examples given in the introduction. I think that listing some examples in Section 3.1 (very briefly) would help the reader understand what we try to optimize in this paper.\n- I did not find in the paper an explanation of how the dataset for CQL was generated, whether this was tune, and what happens if some other batch RL algorithms are used (Batch-Constrained Q for instance). Some recent paper on batch RL [1] seems interesting on its own, and lists much related work on batch RL.\n\n[1]: Fujimoto, S., & Gu, S. S. (2021). A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34, 20132-20145.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers the Reinforcement Learning setting in which the behavior policy should change (be updated to the currently-learned policy) as little often as possible. A motivation is given for when the behavior policy is implemented in an FPGA, in which case deploying a new policy can take a few hours. In this low-switching-cost RL framework, the authors review several approaches at choosing when to deploy a new policy, identify that information-gain-based approaches outperform the other ones, and propose a scalable feature-based way to approximate information gain, based on a matrix built on features, and analysis of its eigen values.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- A strong comparison against several recent batch RL methods (details about how the dataset are generated can be put in the appendix). This change is mainly to strengthen the paper, I think it is already acceptable as is.", "broader_impact_concerns": "No broader impact concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the reinforcement learning setting in which the policy should change as little as possible during training.
The authors empirically evaluate several criteria for choosing when switching the policy on a set of deep reinforcement learning benchmarks.
The reviewers agree that this paper addresses a relevant topic and that the empirical findings in the paper are significant.
The authors' responses have adequately addressed the issues raised in the reviews.
The new version of the paper has been considered satisfactory by the reviewers, who agreed to accept this paper.
Decision: Accept (poster)