Deep Q-Learning with Low Switching Cost
Authors: Shusheng Xu
Simon Shaolei Du
Yi Wu
Venue:  (2021)
Keywords: deep Q-network
DQN
switching cost
deep Q-learning

Abstract: We initiate the study on deep reinforcement learning problems that require low switching cost, i.e., small number of policy switches during training.  Such a requirement is ubiquitous in many applications, such as medical domains, recommendation systems, education, robotics, dialogue agents, etc, where the deployed policy that actually interacts with the environment cannot change frequently. Our paper investigates different policy switching criteria based on deep Q-networks and further proposes an adaptive approach based on the feature distance between the  deployed  Q-network  and  the  underlying  learning  Q-network.   Through  extensive experiments on a medical treatment environment and a collection of the Atari games, we find our feature-switching criterion substantially decreases the switching cost while maintains a similar sample efficiency to the case without the low-switching-cost constraint.  We also complement this empirical finding with a theoretical justification from a representation learning perspective.

Under review as a conference paper at ICLR 2021
DEEP Q-LEARNING WITH Low SWITCHING COST
Anonymous authors
Paper under double-blie blind review
ABSTRACT
We initiate the study on deep reinforcement learning problems that require low switching cost, i.e., small number of policy switches during training. Such a requirement ubiquitous in many applications, such as medical domains, recom- mendation systems, education, robotics, dialogue agents, etc, where the deployed policy that actually interacts with the environment cannot change frequently. Our paper investigates different policy switching criteria based deep Q-networks and further proposes an adaptive approach based on the feature distance between the deployed Q-network and the underlying learning Q-network. Through ex- tensive experiments on a medical treatment environment and collection of the Atari games, we find our feature-wwitching criterion substantially decreases the switching cost while maintains similar sample efficiency to the case without the low-switching-com constraint. We also complement this empirical finding with a theoretical justification from representation learning perspective.
INTRODUCTION
Reinforcement learning (RL) is often used for modeling real-world sequential decision-making problems such as medical domains, personalized recommendations, hardware placements, database optimization, etc. For these applications, oftentimes it is desirable to restrict the agent from adjusting its policy frequently. In medical domains, changing a policy requires a thorough approval process by experts. For large-scale software and hardware systems, changing a policy requires to redeploy the whole environment. Formally, we would like our RL algorithm admits a low switching cost. In this setting, it required that the deployed policy that interacts with the environment cannot change many times.
In some real-world RL applications such robotics, education, and dialogue system, changing the deployed policy frequently may cause high costs and risks. Gu et al. (2017) trained robotic manipulation by decoupling the training and experience collecting threads: Mandel et al. (2014) applied RL to educational games by taking data-driven methodology for comparing and validating policies offline, and run the strongest policy online; Jaques et al. (2019) developed an off- batch RL algorithms for dialog system, which effectively learn in an offline fashion, without using different policies to interact with the environment. All of these work avoid changing the deployed policy frequently they try train the policy offline effectively or validate the policy to determine whether to deploy it online.
For RL problem with low switching cost constraint, the central question is how to design criterion to decide when to change the deployed policy. Ideally, we would like this criterion to have the following four properties:
1. Low switching cost:
This the purpose of this criterion. An algorithm equipped with this policy switching criterion should have low switching cost.
2. High Reward:
Since the deployed policy determines the collected samples and the agent uses fewer deployed policies, the collected data may not be informative enough to learn the optimal policy with high reward. We need this criterion to deploy policies that can collect informative samples.
3. Sample Efficiency: Since the agent only uses a few deployed policies, there may be more redundant samples, which will not be collected if the agent switches the policy frequently. We would like algorithms equipped with a criterion with similar sample efficiency as the case without the low switching cost constraint.

---

Under review as a conference paper at ICLR 2021
4. Generality: We would like this criterion be effective not only specific task but also broadly effective on a wide range of tasks.
In this paper, we take a step toward this important problem. We focus on designing a principled policy switching criterion for deep Q-networks (DQN) learning algorithms, which have been widely used in applications. For example, Ahhn & Park (2000) apply DQN to control balancing between different HVAC systems, (Ao et al. (2019) propose thermal process control method based on DQN, and [Cenn et al. (2018) try to apply it to online recommendation. Notably these applications all require low switching cost.
Our paper conducts a systematic study on DQN with low switching cost. Our contributions are summarized below.
Our Contributions
We conduct the first systematic empirical study on benchmark environments that require modern reinforcement learning algorithms. We test two naive policy switching criteria: 1) switching the policy after a fixed number of steps and 2) switching the policy after increasing step with fixed rate. We find that neither criterion is generic solution because sometimes they either cannot find the best policy or significantly decrease the sample efficiency.
Inspired by representation learning theory, we propose new feature-based switching cri- terion that uses the feature distance between the deployed Q-network and the underlying learning Q-network. Through extensive experiments, we find our proposed criterion is a generic solution it substantially decreases the switching cost while maintains similar performance to the case without the low-switching-cos. constraint.
Along the way, we also derive a deterministic Rainbow DQN (Hessel et al. 2018), which may be of independent interest.
This paper is organized as follows. In Section 2 we review related work. In Sec- tion describe our problem setup and review necessary backgrounds. In Section 4 we describe deterministic Rainbow DQN with the low switching cost constraint. In Section 5 we introduce our feature-based policy switching criterion and its theoretical support. In Section [6] we conduct ex- periments to evaluate different criteria. We conclude in Section 7 and leave experiment details to appendix.
Organization
2 RELATED WORK
first studied in the bandit setting (Auer et al., 2002 Cesa- Bianchi et al. 2013). Existing work on RL with low switching cost is mostly theoretical. To our knowledge, Bai al. 2019) is the first work that studies this problem for the episodic finite-horizon tabular RL setting. Bai et al. (2019) gave low-regret algorithm with an O (H3S. SA log (K)) local the planning switching upper bound where S is the number of stats, A is the number of actions.
Low switching cost algorithms
horizon and K is the number of episodes the agent plays. The upper bound was improved in Zhang al. 2020b a).
The only empirical study on RL with switching cost is Matsushima et al. (2020), which proposed a concept of deployment efficiency and gave a model-bbaed algorithm. During the training process, the algorithm fixes the number of deployments, trains a dynamics model ensemble, and updates the deployed policy alternately. After each deployment, the deployed policy collects transitions in the real environment to enhance the models, and then the models optimize the policy by providing imag- ined trajectories. In other words, they reduce the number of deployments by training on simulated environments. Our goal is different: we design criterion decide when change the deployed policy, and this criterion could be employed by model-free algorithms.
There is a line of work offine RL (also called Batch RL) methods, where the policy does not interact with the environment directly and only learns from a fixed dataset (Lange et al.) 2012] Levine al. 2020). Some methods Interpolate offline and online methods, i.e., semi-batch RT.
2

---

Under review as a conference paper at ICLR 2021
algorithms (Singh et al. 1995 Lange et al. 2012). which update the policy many times on large batch of transitions. However, the switching cost is not their focus.
3 PRELIMINARIES
3.1 MARKOV DECISION PROCESS
Throughout our paper, consider the episodic Markov decision model (S, (S, In this model, S is the state space, A is the action space. H E Zt the planning horizon. P is the transition operator where P(x'|ar, a) denotes the transition probability of taking action from state a to state x'. r S xA -> is the reward function. A policy is mapping from a state to an action, S A. In this paper, we focus on deterministic policies as required by motivating applications.
The dynamics of the episodic MDP can be view as the interaction of an agent with the environment periodically. We let K be the total number of episodes the agent plays. At the beginning of an episode k E [K], the agent chooses a policy
The initial æf E S is sampled from a distribution, and the agent is then in step At each step E [H] in this episode, based on the current state o E S the agent chooses the action o = T (x The environment will give the reward for the step r(z% ) and move the agent to the next state 2h-1 P(-|z ).). The episode automatically ends when the agent reaches the step H
Q-function is used to evaluate the long-term value for the action and subsequent decisions. The Q- Q-fucttion of policy и at time step is defined as follows:
Σ r a (2:)) on x. an J
(T( (x)xx +E
The goal of this agent is to find policy x² which maximizes the expected reward, n² arg max; Er Σ Th Ideally, we want to use a few episodes (K) as possible to learn n²"
3.2 SWITCHING COST
The concept switching cost is used to quantify the adaptability of algorithms. The switching cost is defined as the number of policy changes of deployed policies in the running of the algorithm in K episodes, namely:
K Σ
N switch
The goal of this paper is to equip algorithm with a criterion that learns '²" using few episodes while the same time has small N Swwithh
3.3 DEEP Q-LEARNING
If we estimate Q-function for each state and action pair well, there is no difficulty in finding =. For example, we can always select a" = arg max, Q(x, a). However, it is not easy to learn Q value estimates for each state and action pair, especially when the state or the action space large.
In deep Q-learning (DQN), Mnih et al. (0015) combine deep networks and reinforcement learning successfully by using deep neural network to approximate (ar, a). Given the current state da, the agent selects action greedily based on the Q(In,, a), then the state move to 2h+1 and a reward Tz+1 is obtained. The transition (Iz, de, Tt+11 1x41) saved to the replay memory buffer. At each time, batch of transitions is sampled from this buffer, and the parameters of neural networks are optimized by using stochastic gradient descent minimize the loss
(Th+1 + 7h-1 max q8 (Z o²) - go(m), )2
where 7 is the chosen discount time step h + 1, 0 is the parameters of the online network, and 0 represents the parameters of the target network. The gradient of the loss is back-propagated
3

---

Under review as a conference paper at ICLR 2021
only update 8, while 8 is not optimized directly. DQN has been successful it led to super- human performance on several Atari games. Nevertheless, there are also several limitations this algorithm and many extensions have been proposed. Rainbow (Hessel al. 2018) combines six of these extensions and obtains an excellent and stable performance on many Atari games. In the Appendix A we review these tricks. In this paper, focus Rainhow DQN with low switching cost.
3.4 COUNT-BASED EXPLORATION
In many real-world scenarios that require low switching cost, it is also required deterministic policies, especially in applications mentioned above. Exploring strategies like e-greedy and noisy make the policy stochastic, which we cannot here. Count-based exploration algorithms are known to perform near-optimally when used in reinforcement learning for solving tabular MDPs. Tang et al. (2017) applied these algorithms into nigh-dimensional state spaces successfully. They discretized the state space with a hash function Φ S + Z. Then an exploration bonus (x) = (n()(w)) is added to the reward function and the agent is trained with the modified reward r + +rt. Note that with the count-based exploration, the policy deterministic.
4 DETERMINISTIC RAINBOW DQN WITH A POLICY SWITCHING CRITERION
In this section, we first introduce how to implement deterministic Rainbow DQN with policy switching criterion. This implementation combines six DQN tricks and the policy is always deter- ministic.
4.1 DETERMINISTIC RAINBOW DQN
We first discuss how to make Rainbow DQN always output deterministic policies. Recall that to explore more efficiently, Rainbow adopts the Noisy Net (Fortunato et al.] (018)) which makes the policy stochastic. To obtain a deterministic policy and keep exploring. we remove Noisy Net and employ the Count-Based exploration described in the previous section. When the deployed policy interacts with the environment, it selects the action which maximizes the Q-function. After obtaining the reward by taking this action, an exploration bonus y
is added into the reward.
v vn(((x)
4.2 DQN WITH A POLICY SWITCHING CRITERION
Besides, Rainbow updates the policies which interacts with the environment directly. These poli- cies usually switch millions of times during the training process (since the parameters updated millions of times). In our implementation, we add an online policy in addition to the deployed pol- icy that interacts with the environment. The deployed policy collects the data for the experience replay buffer, while the online policy is frequently updated when training. The deployed policy is replaced with the online policy when they meet the criterion we will discuss soon. The algorithm of deterministic switching Rainbow is shown in Algorithm I
4.3 POLICY SWITCHING CRITERIA
Here first introduce two straightforward policy switching criteria.
Fixed Interval Switching This is the simplest criterion, we switch the deployed policy with fixed interval. Under the FIX criterion, we switch the deployed policy whenever the online policy updated n times where a per-specified number. We will specify n in our experiments.
Adaptive Interval Switching This criterion aims to switch the deployed policy frequently at the first and reduce the switching speed gradually. Under the Adaptive_n2m criterion, we increase the deployment interval from n to m. The interval between the i-th deployment and the (i + 1)-th deployment min(( 1) x n, m). We will specify and m in experiments.
4

---

Under review as a conference paper at ICLR 2021
Algorithm Deterministic Switching Rainbow
Initialize arameters ooline deployed Orarget for online policy, deployed policy and target policy,
initialize empty replay buffer D
2: Denote the state encoder in online policy and deployed policy as fouline and ddepooyed
3: Set the step to start training Hstart, the step to end training Maxx and the interval to update the target policy Harget--
4: Set prccuunnlalee..updalee
= medeployment) =
5: for = to Hmax do
6: Select Λд arg max, Qdeploved (sh; a)
7: Execute action d, and observe reward rh and state #h+1
8: Compute the hash codes through for Th, ((xx)) sgn(Ag(zx))
9:
b A is a fixed matrix with i...d entries drawn from a standard Gaussian distribution (
and g is flat function
10:
Update the hash table counts, n(((2n)) n(((zn)) +1 Update the reward
11:
Th + Vn(o(ru))
12:
Store (Ih, Gh, The Th+1) in D if >> Hatart then
13:
Sample minibatch of transitions from D. Update Oonline by stochastic gradient descent on the sampled minibatch once. if h % Htarget
15:
then
17:
Update Orarget = follnne
Set n(accurmulated updates) = n naccumuulted updates) +
19:
end if
20:
if J (fdeployed: fonline D (ecccumllleduuupddeteeeerepppppo
= true then
21:
Update daeployed Oonline
22:
Update ((accumulaeed.uug updates) =
23:
Update n(deployment) n(deployment) +
24:
end
25:
end if
26:
end for
Algorithm Switching Criteria (I in Algorithm
Fixed interval switching FIX_i
input re naaccumulated _pddates) output bool (ool (aw.caaulatedd
2 n)
Adaptive interval switching Adaptive input ni naaccumulated updates), n (ddeployment)
output bool (a(acumullaeed updates) 2 min(n(deploymentt) + 1) n, m)
Feature-based switching F_a
input fdeployed, fouline. D
Sample minibatch B of transitions (2r)) from with probability Ph.
fdeploywa (2 Sontina (wr))
Compute the the similarity simm(xs) [Taeptoye X] fontine (#a
Compute the average similarity in sampled batch sim (B) E.ex sim(x) output (sim < a)
|||||
These two criteria and proposed new criterion summarized in Algorithm Unfortunately. as will be shown our experiments, these two criteria do not perform well. This requires us to design new principled criterion.
5

---

Under review as a conference paper at ICLR 2021
FEATURE-BASED SWITCHING CRITERION
In the section, we describe our new policy switching criterion based on feature learning. We first de- scribe this criterion, and then provide some theoretical justification from representation learning point of view.
Feature-based Switching Criterion We adopt the view the DQN learns to extract informative features of the states of environments. Our proposed criterion tries to switch the deployed policy according to the extracted feature. When deciding whether to switch the deployed policy or not, we first sample a batch of states B from the experience replay buffer, and then extract the feature of all states by both the deployed deep Q-network and online deep Q-network.
For state the extracted feature are denoted as deppooed (3r) and [a)line(I)) respectively. The similarity score between fdeployed and fouline on state x is defined as
(Saeploydd (z), fonline (2))
sim(x) = feppooo [=) x [[ (x)]l
We then compute the averaged similarity score on the batch of states
Exen sim(x)
sim [B
[[B]]
With hyper-parameter [0, 1]. the feature-based policy switching criterion is to change the deployed policy whenever simn(B) < a.
Theoretical Justification Our criterion is inspired by representation learning. To illustrate the idea, we consider the following setting. Suppose we want to learn f()), a representation function that maps the input to k-dimension vector. We assume have input-output pairs (x,, y) with y = (w, I" (xr) for some underlying representation function (*((-) and a linear predictor w E RR For ease of presentation, let us assume we know w, and our goal is to learn the underlying representation which together with w gives us 0 prediction error.
Suppose we have data D1 and D2- We use D1 to train estimator of f denoted as fl, and D1 U D2 to train another estimator of f", denoted as f12 The training method is empirical risk minimization, i.e.,
/ min JEF
Σ (x.0)ED1
(w, ((x)))² and f t min fef D1 D2
Σ
(y - (w, f(x)))
Di
(u.))DiiDs
where F is some pre-specified representation function class.
The following theorem suggests the similarity score between f and f1+2 small, then f² also far from the underlying representation. f"
Theorem 1. Suppose I and f2+2 are trained via aforementioned scheme. There exist datasel DD, D2. function class F and such that if the similarity score between f' and f1 on D D1+2 is smaller than a, then the prediction error of f¹ on DD+2 is
The proof is deferred to Appendix where we give explicitly constructions.
Theorem D suggests that in certain scenarios, if the learned representation has not converged (the similarity score is small), then cannot the optimal representation which turn will hurt the prediction accuracy. Therefore, if we find the similarity score is small, should change the de- ployed policy.
EXPERIMENTS
In this section, we conduct experiments to evaluate different policy switching criteria on DQN. We study several Atari game environments along and environment for simulating sepsis treatment for ICU patients. We evaluate the efficiency among different switching criteria these environments. Implementation details and hyper-parameters listed in the Appendix

---

Under review as a conference paper at ICLR 2021
6.1 ENVIRONMENTS
GYMIC GYMIC is an OpenAl gym environment for simulating sepsis treatment for ICU patients to an infection, where sepsis caused by the body's response to an infection and could life- threatening. GYMIC built an environment to simulate the MIMIC sepsis cohort, where MIMIC is an open patient EHR dataset from ICU patients. This environment generates a sparse reward, the reward is set to +15 if the patient recovered and -15 if the patient died. This environment has 46 clinical features and a 5 X 5 action space. For the GYMIC, we display the learning curve of 1.5 million steps of the environment, after which the reward converge. We choose this environment because is simulating a real-world problem that requires low switching cost. For Atari games, all the experiments were training for the environment stepping 3.5 million times.
Atari 2600 Atari 2600 games are widely employed to evaluate the performance of DQN based agents. We also evaluate the efficiency among different switching criteria on several games, such as Pong, Road Runner, Beam Rider, etc.
6.2 RESULTS AND DISCUSSIONS
For all environments, we evaluate a feature-hased criterion F_0.98, three fixed interval criteria cov- ering a vast range FIX _022 FIX_100 and F'IX-10% and an adaptive criterion increasing the deploying interval from 100 to 10,000. Besides the switching criteria we discussed above, we use "None" indicate an experiment without the now-switching-com constraint where deployed policy kept in syne with online policy all the time, notice that this experiment is equivalent to FIX_1.
None
FIX_10²
GYMIC As shown in figure I none of the switching criteria affects the performance, but they reduce the number of policy switches drastically, which indicates that reducing the switching cost in such medical environment could be feasible. In particular, we find FIX 10a and F_0.98 are the two best criteria to reduce switching cost. Finally, this criterion keeps good performance with the minimal switching cost.
F F_0.98
FIX_10"
FIX_102
Adaptive
14
Reward
13
Switching Cost
105
A Atrri 2600 GYMIC may be too simple, and we should compare the performances among differ- ent criteria in some more difficult environments. We evaluate the performance of different criteria when playing the Atari games, which image- based environments. In particular, the state space is much more complex. Figure 2 shows the sults of 6 Atari games. In each subgraph, the up- per curves are about the rewards of steps, while the curves blow is about switching cost.
102
0.0
0.2M
0.5M
0.8M 1.0M 1.2M 1.5M Step
Figure Results on GYMIC, "Step" means the number of steps of the environment. We show the learning curve of 1.5 million steps. The figure above is the learning cure of reward, while the figure below displays the switching cost. "None" means low-switching-cos con- straint, and the deployed policy always keeps sync with the online policy. Curves of reward are smoothed with a moving average over 5 points.
First observe that overall trend, higher switch- ing cost leads to better performance. In general Rainbow DQN with no switching cost constrain often gives the best performance. Also, FTX 102 enjoys better performance than FTX_10* and F1X_104. In some games such as Qbert and Riverraid, although FIX _1 and Adaptive lead to lower switching cost, they fail to learn how to play the games well. Therefore, they not desired generic policy switching criterion.
Secondly, we observe that changing the policy with an adaptive interval may be better than a fixed interval. Focusing on the criteria FIX_10* and 10 the adaptive criterion switches the online policy fast at first and decrease its switching speed gradually, and in the end, the adaptive criterion would have the same speed the fixed one. Therefore, there is significant difference
7

---

Under review as a conference paper at ICLR 2021
F_0.98
10
AIX beam ridder
MX_100
Adaptive 10'tx10*
battle zone
pong
20000
20
0-
-20
0-
10a
10a
10a
Switching
Cost
10a
10a
10a
10a
10a
107
qbert
riverraid
road runner
10000
40000
10000
20000
0
0
10a
10a
10a
Switching
Cost
10a
10a
10a
10a
10a
10a
1.0M
0.0
1.0M
2.0M
Step
Step
Step
The results the Atari games, compare different switching criteria six Atari games. "Step" means the number of steps of the environment. We constrain this 3.5 million steps for all environments. In each environment, we display the reward over the steps on the top and the switching cost in log scale at the bottom. "None" means switching criterion under which the deployed policy always keeps syne with the online policy. We evaluate bease criterion, three fixed interval criteria covering vast range, and an adaptive criteria increasing the deploying interval from 100 to 10,000. Curves of reward are smoothed with a moving average over 5 points.
Figure 2:
between the total switching cost of these two criteria. However, we could observe that the adaptive criterion's performance is better than the fixed one when playing Beam Rider, Pong, Qbert, and they obtain similar performances in the rest games.
Lastly, we find our proposed feature-based criterion (F 0.98) is the desired that satisfy all four properties we discussed Section It signiiicantly reduces the switching cost compared to "None", and is smaller than F/ X __022 While incurs higher switching cost than FIX 1_10_005_ F1X _100, and Adaptive_10'too10' on all environments feature-based criterion consistently perform as well as "None" the sense that it finds the optimal policy eventually, 2) has the same sample effi ciency as "None". On the other hand, other criteria sometime have significantly worse performance compared to "None", so none of them is generic solution.
CONCLUSION
In this paper, we focus on the concept of switching cost and take a step toward designing generic solution for reducing the switching cost while maintaining the performance. Inspired by representa- tion learning theory, proposed new feature-based policy switching criterion for deep Q-learning methods. Through experiment on one medical simulation environment and six Atari games, we find proposed criterion significantly reduces the switching cost and the same time enjoys the same performance as the case where there is no switching cost constraint.
We believe paper is just the first step this important problem. One interesting question is how to design principled policy switching criteria for policy-based and model-based methods. Another direction is give provable guarantees for these policy switching criteria that work for methods dealing with large state space contrast to existing analyses are all about tabular RL (Bai et al. 2019 Zhang al. 2020b
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ki Uhn Ahn and Cheol Soo Park. Application of deep q-networks for model-free optimal control balancing between different hvac systems. Science and Technology for the Built Environment, 26 (1):61-74, 2020.
Tao Ao, Jiong Shen, and Xichui Liu. The application of dqn in thermal process control. In 2019 Chinese Control Conference (CCC), pp. 2840-2845. IEEE, 2019.
Peter Auer, Nicolo Cesa-Bianche and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3)2235255 2002.
Yu Bai, Tengyang Xie, Nan Jiang, and Yu-Xiang Wang. Provably efficient q-learning with low switching cost. In Advances in Neural Information Processing Systems, pp. 8002-8011, 2019.
Marc G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learn- ing. ArXiv, abs// 0088777 2017.
Nicolo Cesa- Bisa-Bianchi, Ofer Dekel, and Ohad Shamir. Online learning with switching costs and other adaptive adversaries. In Advances in Neural Information Processing Systems, pp. I 1160-1168, 2013.
Shi- Yong Chen, Yang Yu, Qing Da, Jun Tan, Hai-Kuan Huang, and Hai-Hong Tang. Stabilizing reinforcement learning dynamic environment with application online recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 1187-1196, 2018
Meire Fortunato, Mohammad Gheshlaghi Azar, Piot, Jacob Menick, Ian Osband, A. Graves, Vlad Mnih, Rémi Munos. Demis Hassabis, O. Pietquin, Charles Blundell, and S. Legg. Noisy networks for exploration. ArXiv, 1006/ 10295, 2018.
Shixiang Gu, Ethan Holly, Timothy Lillicrap. and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEFE international confer- ence on robotics and automation (ICRA), pp. 3389-3396. IEEF, 2017.
H. V. Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In AAAI, 2016.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00450 2019.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643 2020.
Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications to educational games. In AAMAS, pp. 1077- 1084, 2014.
Tatsuya Matsushima. Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. arXiv preprint arXiv:2006.036647, 2020.
V. Mnih, K. Kayukcuoglu, D. Silver, Andrei A. Rusu, J. Veness, Marc G. Bellemare, A. Graves, Martin A. Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, S. Petersen, C. Beattie, A. Sadik, Ioannis Antonoglou, H. King, D. Kumaran, Daan Wierstra, Legg, and Demis Hassabis. Human- level control through deep reinforcement learning. Nature, 518:529-533, 2015.

---

Under review as a conference paper at ICLR 2021
T. Schaul, John Quan, Ioannis Antonoglou, and D. Silver. Prioritized experience replay. CoRR, abs/1511.05952, 2016.
Satinder P Singh, Tommi Jaakkola, and Michael Jordan. Reinforcement learning with soft state aggregation. In Advances in neural information processing systems, pp. 361-368, 1995.
R. Sutton. Learning predict by the methods of temporal differences. Machine Learning. 3:9 44, 2005.
R. Sutton and A. Barto. Reinforcement learning: An introduction. IEEE Transactions Neural Nenworks, 16:285-286, 2005.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, F. Turck, and Abbeel. Exploration: A study of count-based exploration for deep reinforcement learning. ArXiv, abs/1611.04717, 2017.
Ziyu Wang, T. Schaul, Matteo Hessel, H. V. Hasselt, Mare Lanctot, and N. D. Freitas. Dueling network architectures for deep reinforcement learning. ICML, 2016.
Zihan Zhang, Xiangyang Ji, and Simon Du. reinforcement learning more difficult than bandits? near-optimal algorithm escaping the curse of horizon. arXiv preprint arXiv:2009. 13503, 2020a.
Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learning via reference-advantage decomposition, 2020b.
10

---

Under review as a conference paper at ICLR 2021
A DETAILS OF EXPERIMENTS
A.1 DETAILED ALGORITIIM
For completeness, introduce the extensions of DQN and display our detailed algorithm.
Double Q-learning Conventional Q-learning is affected by an overestimation bias, due to the maximization step Equation 3
Double Q-learning (Hasselt et al. 2016) address this problem by
decoupling. They use the loss
(Th+1 + 1 arg go (xx41 - qo (z), an))²
(4)
This change reduce harmful overestimations that were present for DQN, which leads to a improve- ment.
Multi-step learning Q-learning accumulates a single reward, Sutton (2055) use forward-view
multi-step accumulated reward long ago, where a n-step accumulate ated reward is defined
(n)
(n)) Th Th+k+1
(5)
and the final loss is
(r/5)) + 2/ max yo ( ( )²
(6)
o'
Multi-step targets often lead to faster learning (Sutton & Barto 2005
Dueling networks Wang et al. (016)) splits the DQN network into two streams called value stream V and advantage stream A, these two strems share the convolutional encoder f. and action value Q(x, a) could be computed as:
Es A(f(z).
V(f())
Nactions
Prioritized replay DQN samples uniformly from the replay buffer, to sample more frequently from those transitions from which the policy Iearn more and quickly, Schaul et al. (2016) samples transitions with probability pa relative to the last encountered absolute TD error
Ph |ra41 + Mh+1 max (g a') qe (Zh, aa)
(
where is hyper-parameter, and new transitions always have the maximum priority when they enter the buffer to ensure a bias towards unseen transitions.
Distributional RL Instead of approximating the expected return as DQN, Bellemare et al. (2017) proposed a method to approximate the distribution of returns on a a discrete support z, is a vector with Natoms atoms and is defined
V mmax 2j Vuni (i-1)
Vmii Natoms
E 1,2, N Natoms
where Vmin and Vmaz is the minimal and maximal value in this support. The approximat- ing distribution d,, is defined
this support and the probability p ax) each atom i as dy = (2, (Ir,, (x)) (n))..7 .The goal is to update the trainable parameters to match this distribution with the actual distribution of returns. To learn the probability Pd for each with variant of Bellman's equation, they minimize the Kullbeck-Leibler divergence (x 2ddd [(dx) between the distribution dn and the target distribution d' := (rh+1 - 12, Pg (pp(#++1 1, arg max Gg (1h+1, a'))), where 42 is 1.2-projection from the target distribution the fixed support and q7 (1 a) =" p5(xN+1,
11

---

Under review as a conference paper at ICLR 2021
Nosiy Net To address the limitations of exploring using e-greedy policies, Fortumato et al. (2018) propose a noisy linear layer
y= b + Wa (Bnoisy O + (W noiey ① E e=")))
(10)
to replace the standard linear y = + Wx, where e and " are random variables, @ denotes the element-wise product.
Rainbow combines these extensions, To make policy deterministic but still keep explor- ing during training, remove the Noisy Net and adopt Count-Based exploration. The detailed algorithm is as follows.
Algorithm Detailed algorithm
1: >> Hyper for extensions
2: Distributional RL: Number of atoms Natomns min/max values Vinin max
3: Prioritization replay memory: exponent w, capacity
4: Multi-step: number of steps
5: b Initialization
6: Initialize prioritization replay memory D, state encoder Sonlines value stream Vontine and advan-
tage stream Aonliue for online policy Ponline
7: Initialize deployed policy Pdeployed and target policy Ptarget with parameters in P Pouline
8: Definition of Q-function
9: 2, Vinin + (i (4-11
Pmax
-mmin
Natomm
10: A'alline (x)) N. Nactions Da Aoulinn (fontin(2)) a)
p Pontino (It, a) = sof tmax (Vå Vantino (Soutine (x)) + A ( A Am (x))
12: Qontine (ar, () E3 PsPontine (1,, (I,aa)
13: Similarly, Qdephoyed (I, a) Ei ZiPeoploeed (Ir, a), Q tarrgtt (I,, =, ziPragor (2r ()
14:
Training process 16: set up environment
n( (accumulated )
n (ddeployment)
18: for h = 1 to Hmax do
if' Termination then reset environment end
20:
22:
select dд arg max, Qdeployed (In, a)
23:
Execute action as in emulator and observe reward Fa and state Th+1
24:
compute the hash codes through for 2h, Φ(xx) sgn(Ag(#x))
25:
a matrix with i.d, entries drawn from a standard Gaussian distribution N 1) and 9 a flat function
26:
update the hash table counts, n(((())) = n(((zn)) +
27:
update the reward Th = Th + Va(a(a)))
28:
store (Zh, ags The Th+1) in D
29:
if >> Hatat then
30:
sample minibatch of transitions (NW,, an, Wh', Thi, Th441 on Th/4n-1s Tb(4) from D
with probability Ph! wh is the importance-sampling weight for transitions at h
31:
32:
compute the multi-step reward ri = Σ k=0 7 r"rc+*
33:
a" arg max, Qontine (x 4 +n. a), m. 0. E 1,2, N Ntooms
34:
for j = to Natoms do
35:
T (r + "==]|_ccpppp Vminn Vmax)
36:
A= = (Vmux Vinin) Natoms - 1)
37:
by = (Tzj - Vmin) / / [bj], u= [by]
38:
my + Plarges (IM+n )u bj), mm mh Plargel (IN+ny a²) - 1)
39:
end for
40:
DELM - E, my donllee (x aw)
41:
update PM by [. update parameters in P Ponline by gradient descent step DELKS ww
42:
12

---

Under review as a conference paper at ICLR 2021
43:
nalaccumlated..uudaatts n[aceummlated.uppdaee +
44:
if h % Haarget then
45:
update Pearget by the parameters of Ponline
46:
end if [[a, fo: D, maacccumllated.uppdala a n(deployment)) update ddeployed by the parameters of Ponline
47:
true then
48:
ni (aaccumulaed..updeea
O, (ddeployment) n medeployment)
50:
end if
end if
52: end for
A.2
Table | lists the basic hyper- parametrsssters of the algorithm, all of our experiments share these hyper- parameters except the experiments on GYMIC adopt the Harget 1K. Most of these parameter are the same with raw Rainbow algorithm. For the count base exploration, the bonus is 3 set to 0.01.
Parameter
Value 20K 0,0000625 8K
Hstart
learning rate
Hargean (Atari)
Hargeet (GYMIC)
1K
Adam c
1.5 x 10-4
Prioritization type
proportional
Prioritization exponent w
0.5
Prioritization importance sampling
0.4 + 1.0
Multi-step returns n Distributional atoms N atoms Distributional Vnins Vimx
51
[-10, 10]
Discount factor 7
0.99
Memory capacity N
1M
Replay period
4
Minibatch size
32
Reward clipping
[-1, 1]
Count-base bonus
0.01
Activation function
ReLu
Table The basic hyper-parametert we used the Adam optimizer with learing rate a 0. 0.0000625 and = 1.5 10-4 before training the online policy, we let the initialized random policy make 20K steps collect some transitions and the capacity for replay buffer is 1M. During the training process, we sample 32 transitions from the replay buffer and update the online policy every four steps. The reward is clipped into [-1, I] and Relu is adopted as the activation function. For replay prioritization we use the recommended proportional variant, with importance sampling from 0.4 to 1, the prioritization w is set to 0.5. addition, we employ Natoms = 51, Vmin - 10, Vmax = 10 for distributional RL and
for multi-step returns. Finally the count- base bonus is set 0.01
Table 2 lists the rest hyper-parameterr for experiments GYMIC. Since there are 46 clinical fea- tures in this environment, we stack consecutive states to compose 184-dimensional vector as the input for the state encoder fouline (faploeee or ftarget). The state encoder is a 2-layer MLP with hidden size 128.
Table 3 shows the additional hyper-parameters for experiments Atari games. The observations are gcaled and resized to 84 x 84 tensor, and 4 consecutive frames are concatenated as a single state, each action selected by the agent is repeated for 4 times. The state encoder is composed of convolutional layers with 32, 64 and 64 channels, which use 8x8, 4x4, 3x3 filters and strides of 4, 2, respectively.
13

---

Under review as a conference paper at ICLR 2021
Parameter
Value 4
State Stacked
Number of layers for MLP
2
Hidden size
128
Table 2: Extra hype-parameten for the experiments in GYMIC, stack 4 consecutive states and adopt a 2-layer MLP with hidden size 128 to extract the feature of states.
Parameter
Value
Gray scaling
True
Observation
(84, 84)
Frame Stacked
Action repetitions
Max frames per episode
108k
Encoder channels
32, 64, 64
Encoder filter size
8 x 8, 4 4, 3 x 3 2,
Encoder stride
Table Additional hyper-parameters for experiments in Atari games. Observations are grey-scaled and rescaled to 84 x 84 consecutive frames are staked as the state and each action is acted four times. And we limit the max number of frames for an episode 108K. The state encoder consists of 3 convolutional layers.
A.3 REWARD AND SWITCHING COST
In the end, we list the value of the switching cost and reward of different criteria when the environ- ment take 1.5 million steps and million steps Table 4
B PROOF FOR SECTION
Proof of Theorem
1
We w (1, .,.,,,....
I2t be k-dimensional all one vector. We let
F= f(r) = (20(((1,,0) x)) (0 x)) - 1,..., 20 x)) - 1)} C [IR + R*
with (-) being the ReLU activation functior and vs E fes, -c;} where e; R denotes the vector that only the i-th coordinate is and others are O. We assume is an even number and ak is an integer for simplicity.
let underlying f be the vector correspond to (e1, E2 E2,,,.,..... Cs). We let D1 ((e1, 1), (e2, 1)...., (E(1-a)k: 1)} and D: = {(<[1-a)++1) 1),... (et, 1)]. Because D; U D: will recover f* i.e., f
use
the ERM training scheme, is clear that the training because it is not f is better solution (f" has O error for the empirical risk. Now the similarity score between f and f1+2 is smaller than a, it means for f its corresponding (v(1-))e+1>- 1..... vx)
are not correct. In this case, 's prediction error least 1 a D UD2 because it will predict 0 on all inputs of Dz.
We define (0)) 0.5
14

---

Under review as a conference paper at ICLR 2021 2
16970.0 + 5766.2
6759.74 + 2136.3
3947.5 + 1438.0
4182.2 925.8
28968. 0 + 4827.2 .2
261 15180.0 + 7161.5
12311.31 1. 6731.4
13791.0 3372.0
7052.3 +: 862.7
15940.0 4: 8234.1
Table 4: We list the value of the switching cost and reward of different criteria when the environment takes 1.5 million steps and million steps. "Reward" corresponds to the absolute value of the reward. and "Gap" denotes the difference between the reward under speeific criterion and "None." And "Switching Cost" corresponds the switching cost under a criterion this time step.
0.08 :: 6.7
17.43 + 5.4
F.0.98
529.2
-993..0
119.6
-896.2
1681
12.9
1268.5
-2122.8
899.5
-1271.0
1309.0
15702.0
4240.3
-6070.0
1233.4
-1439.9
2687.0
-0.5
1980.5
-3844.2
2325.5
-756.3
2736.6
8460.0
14330.0 = 4397.9
4078. 56 = 1819.8
-7.14 10.2
2681.0 = 1696.1
3267.6 1 705.2
19513.0 + 4751.0
12530.0 + 5096.0
8857. .14 1. 3173.9
19.02 0:02:22 2.2
4013.5 +: 938.3
8473.7
Adaptive
84.5
-12570.0
84.5
-3577.3
84.5
5.7
84.5
-3669.2
84.5
-2185.6
84.5
6247.0
122.5
19720.00
122.5
-493.3
122.5
1.1
122.5
13847.2
122.5
-3795.1
122.5
8226.0
15706.0
4417.4
4068. 26 = 1340.2
30620.0 0 = 6946.2
17830.0 + 4384.2
2120.0
2193.75 + 1629.5
35055.0 1: 7095.8
12.97 5.2
649.25 = 245.3
3152.6 6 658.6
3391.1 + 821.2
FIX 10
-8030.0
-3587.6
-5401.0
-2300.6
17354.0
114420.0
-8097.0
17.0 + 5.0
-14411.5
4417.5
-2425.0
36.5
36.5
36.5
-0.1
36.5
36.5
36.5
73.5
73.5
73.5
-1.0
73.5
73.5
73.5
18870.0
5654.16
12236. 026662226 5669.9 .9
17930.0 4472.7
7493. 76 2376.4
- 13.69 = 4.0
4359. 75 + 534.2
4177.0 + 870.0
2000.3
21290.0 8191.8
13836.0 +: 10328
7872.9 +: 939.1
36722.0 7119.7
20.35 + 2.3
1,500,000 steps
FIX_10"
369.0
-8970.0
369.0
-162.1
369.0
-0.8
369.0
-1690.5
369.0
-1276.2
369.0
1681.0
steps
744.0
-10960.0
744.0
-1514.6
744.0
2.4
744.0
-3849.2
744.0
64.3
744.0
-758.0
14947.0
3,000,000 000
23070.0 = 7691.9
7554. 26 = 2283.4
5073. 75 + 624.7
4602.8 8 693.8
26039.0 0 + 4605.4
23910.0 + 6289.8
11393.24 233.24111 1. 1802.1
17038.5 5 +: 2710.1
7840.3 +: 351.4
36118.0 = 6366.6
-.55 9.2
17.92 4.4
FIX 102
3687.5
-3830.0
3687.5
101.6
3687.5
104
3687.5
.976.5
3687.5
-850.4
3687.5
12773.0
7437.5
-8340.0
7437.5
-2358.0
7437.3
-0.0
7437.5
-646.8
7437.5
31.7
7437.5
-132200
26900.0 = 9445.1
7655.9 + 2021.2
12.87 + 8.1
6050. 25 + 2189.2
5453.2 = 787.6
13266.0 0 + 1513.4
32250.0 + 8119.6
6530.9 N/A
17.96 + 6.8
17685.2 25 +: 2186.0
7808.6 + 308.6
37/80.0 + 8429.9
368750.0
368750.0
368750.0
368750.0
368750.0
743750.0
743750.0
743750.0
743750.0
743750.0
743750.0
None
368750.0
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
13751.2
Switching cost Reward
Switching cost
wwithhing cost
Switching cost
Switching cost
Switching cost Reward
Switching cost
Switching cost Reward
wwitching cost Reward
Switching cost
Switching cost
Switching cost Reward
criteria
Reward
Reward
Reward
Reward
Reward
Reward
Reward
Gap
Gap
Gap
Gap
Gap
Gap
Gap
Gap
Gap
Gap
Gap
Gap
environment
zone battle zone
beam rider
road rumner
battle zone
beam. rider
road road_runnes runnerr
riverraid
riverraid
pong
qbert
pong
qbert
15


Reviews:
[{"review_id": "yEDt2Q8FmU-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "######################################\n\nSummary:\nIn many real world applications for RL such as medicine, there are limits on the number of policies from which we can simulate data. This paper proposes an approach that adaptively decides when to update the simulation policy, based on the difference between it and the current learned policy. Experiments on a medical treatment environment and Atari show that the approach obtains similar performance to on-policy RL with fewer changes of the simulation policy.\n\n######################################\n\nPros:\n1. The proposed approach is straightforward, adaptive, and achieves results comparable to the classical on-policy setting with fewer policy switches on all environments shown. It is also applicable to both model-based and model-free RL.\n2. The paper is organized well, and the algorithms are clearly explained.\n\nCons:\n1. I did not find the theoretical justification for the proposed approach to be very convincing for RL, since it is based on a construction in a simplified linear regression case. However, I think this is ok since the paper is application-focused.\n2. Based on the results on GYMIC, the proposed approach seems to have much greater variance than the other algorithms, especially at the early stages of training. This is often detrimental for the applications considered, such as medicine, in which robustness is also desirable.\n\n######################################\n\nOverall:\nI would lean toward accepting this paper. I am not completely familiar with the literature on RL with low switching cost, but the proposed approach appears to be novel. The experiments show that when combined with Rainbow DQN,  it effectively reduces the switching cost on a range of environments and is based on the training path, requiring less environment-specific hand-tuning than fixed or adaptive interval switching.\n\n######################################\n\nFurther comments and questions:\n1. How were the six Atari games chosen? How do the different approaches compare in other games?\n2. There are a number of typos, e.g. \"deno\" in the first paragraph of section 3.1. \n\n######################################\n\nUpdate after reading other reviews and author response:\n\nI have decided to lower my score from 6 to 5, as I agree with Reviewer 3 that more experimental analysis of the method is needed (ablations, sensitivities, etc.) given that the theoretical backing is not convincing. The authors also did not directly answer our questions.", "extra_scores": {}, "extra_text": {}}, {"review_id": "tmIpH-rYoMP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the RL context, this paper aims at designing a generic solution for reducing the number of policy switches during training (called switching cost) while maintaining the performance. This study is done in the context of deep reinforcement learning. A few generic baselines solutions are provided as well as a more complex solution that empirically outperforms the baselines.\n\nMotivation of the paper:\nThis paper studies an interesting question that is rarely studied in practice. The paper mentions a few applications domains but what would be a very concrete example where the switching cost is important? The paper would benefit from mentioning in what kind of specific application this can be useful to back up the sentences like \"Such a requirement is ubiquitous in many applications, such as medical domains, recommendation systems, education, robotics, dialogue agents, etc,\". \n\nThe approach\nThe paper investigates different simple switching criteria based on deep Q-networks that are used as baselines and it also proposes an adaptive approach based on the feature distance between Q-networks. I'm not fully convinced by the theoretical justification from a representation learning perspective. What are exactly the extracted feature denoted as f(x) beyond \"a representation function that maps the input to a k-dimension vector\" and that it is based on the Q-networks. Are these the features at the last layer of the Q-network? In fact, why are they referred to as representation at all in the case of DQN?\n\nExperiments\nIt seems that the paper does not mention the number of seeds (runs) that are used for the experiments. Otherwise, the experiments back up the claims.\n\nOther comments:\n- line 16 of the algorithm: do you mean \"h % H_{target}==0\" ?\n- The structure of the paper might be improved a bit and there are some typos (state encdoer, hyper-paramter, ...)", "extra_scores": {}, "extra_text": {}}, {"review_id": "mVi3Ck0SbD", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies RL with low switching cost under the deep RL setting. It points out several naive algorithms like switching after a certain number of steps and then propose a new heuristic. This heuristic learns a new policy offline using the experience replay the behavior collected and switches the behavior policy once the similarity of the feature embeddings of the current state by these two policies becomes large. The paper also makes an attempt to provide a theoretical justification for a better understanding of the heuristic. This method might outperform the naive algorithms by some margin, if any. It would be a more interesting manuscript if some stronger results could be provided from the perspective of any of theory, experiments, or applications.", "extra_scores": {}, "extra_text": {}}, {"review_id": "SvziB7Jx5yo", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Hi,\n\nFirst I want to thank authors for putting this manuscript together, and looking into an interesting issue. \n\n*Summary* : Authors proposed an algorithm for switching policies while using Deep Q-learning. The method is based on feature similarity (calculated by cosine similarity) between online and the deployed policy. In addition, they performed experiments in Atari and sepsis simulator.\n\n*Strength* : \n1. I believe the paper is well written and easy to follow. \n2. An important problem to tackle (well motivated)\n\n*Weakness* :\n1. Content : I believe the paper is a good first step, but for a publication analysis should go further, to expand the understanding of the method. For example, I'd like to see the effect of \"a\" for switching on the performance. It shows how stable/unstable the method will be. The effect of batch size \"|B|\" on how often we switch, and if we were to use smaller batches (higher variance in sim(B) score) are we going to see a large hit in the performance? \nIn addition, it is important to check (or discuss) other possible similarity metrics. For example, what if we look at inf norm of the two representation difference, or other similar metrics.\n\n2. Performance : It seems to me that FIX_10^3 has always lower switching cost, and also learns a good policy. I appreciate authors trying 3 different fix number, as mentioned above I will be curious to see effect of \"a\" as well.\n\nQuestions : I think that maybe another good criteria is to measure the distance between the deployed and the online policy, rather than feature representation. It may be the case that features distance determines the policy distance but that's not necessarily true. I was wondering what authors think about this?  (Of course it's hard to measure the distance between the two policy, but maybe using DRL we can measure the distribution distance of the two in a given batch?)\n\n\nAt the end, I would like to say that, the paper is a good a step, but for publication the analysis/ experiments should be more thorough and possibly give insights about how to \"formalize\" the problem. (which I believe should be the main focus, developing theoretical grounding for low-switching cost problems).\n\nHope it was helpful,\nThanks.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies RL with low switching cost under the deep RL setting. It provides new heuristics for doing so. The reviewers are worrying about whether the problem is important in practice, whether the policies obtained can be used in practice, and the theories might not be strong enough. The paper can be strengthened if better theory and more experiments are provided.  
Decision: Reject