Title: Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning
Authors: Sahil Sharma
Aravind S. Lakshminarayanan
Balaraman Ravindran
Venue: ICLR 2017
Keywords: Deep learning
Reinforcement Learning

Abstract: Reinforcement Learning algorithms can learn complex behavioral patterns for sequential decision making tasks wherein an agent interacts with an environment and acquires feedback in the form of rewards sampled from it. Traditionally, such algorithms make decisions, i.e., select actions to execute, at every single time step of the agent-environment interactions. In this paper, we propose a novel framework, Fine Grained Action Repetition (FiGAR), which enables the agent to decide the action as well as the time scale of repeating it.
FiGAR can be used for improving any Deep Reinforcement Learning algorithm which maintains an explicit policy estimate  by enabling temporal abstractions in the action space and implicitly enabling planning through sequences of repetitive macro-actions.  
We empirically demonstrate the efficacy of our framework by showing performance improvements on top of three policy search algorithms in different domains: Asynchronous Advantage Actor Critic in the Atari 2600 domain, Trust Region Policy Optimization in Mujoco domain and Deep Deterministic Policy Gradients in the TORCS car racing domain.


TL;DR: Framework for temporal abstractions in policy space by learning to repeat actions

Review discussion:
How do you handle the frames generated between action repeats? Are they discarded or are they fed into the network in some way?
Thanks for reviewing the paper.
In what follows | denotes the concatenation operator.
Suppose action decisions were taken at time steps (..., 20, 26, 39) The decision at time step 39 was to repeat the action a_{39} for 3 time steps. At time step 42 the observation presented to the agent is concatenation of the frame for current time step (42) with the last three action decision time steps's frames, that is f_{20}|f_{26}|f_{39}|f_{42}.
Hence it is the action decision points which decide the frames which are concatenated and presented as input. If the next action decision is to be taken at time step 50 (action repetition of 8 was chosen at time step 42), then the next input is f_{26}|f_{39}|f_{42}|f_{50}.
In short, the intermediate frames are indeed discarded. In the case of this example, the network just gets to see the frames on which it has either already made action decisions (frames 20, 26, 39), or those on which it wants to make action decisions currently (frame 42).
Please let us know if any further clarifications are required.
1. Do the policy for selecting the action a, and the policy for selecting the repetitions x, share any weights?
2. How does your proposed method relate to semi-MDPs?
3. What was the average action repetition (or distribution over action repetition counts) in the TORCS experiment?
Thanks for the questions.
Q1. For the paradigm to work, no assumptions are made about whether the policy for action selection ($\pi_{a}$) and that for action repetition share any parameters or not.
In practice, for the figar-a3c setup all but the final output layer are shared between the two policies and the critic (details on the architecture are in appendix A).
For figar-trpo no parameters are shared between the two policies.
For figar-ddpg all but final layers of the two policies are shared.
Q2.
Semi- MDPs  (SMDPs) :
SMDPs are MDPs with durative actions. The assumption in SMDPs is that actions take some "holding time" to complete. Typically, they are modeled with two distributions, one corresponding to the next state transition and the other corresponding to the holding time, which denotes the number of time steps between the current action from the policy until the next action from the policy. The rewards over the entire holding time of an action is the credit assigned to picking the action.
Relation to FiGAR:
In our framework, we naturally have durative actions due to the policy structure where the decision consists of both the choice of the action and the time scale of its execution.
Therefore, we convert the MDP to a semi-MDP trivially.
In fact, we give more structure to semi-MDP because we are clear that during the holding time, we "repeat" the chosen action, while in a semi-MDP, what happens during the holding time is not specified. One can think of the part of the policy that outputs the probability distribution over the time scales as a holding time distribution, and thereby, our framework naturally fits into semi-MDPs with the action repetition characterizing the holding time. As an additional note, we also ensure appropriate discounting factor exponent and sum of rewards over the holding time, both of which obey the semi-MDP framework.
Q3. The average action repetition was 2.
Although the average action repetition in this case is small, we believe that deep exploration in the form of the choice to execute larger action repetitions allows the agent to learn smoother policies.
Thanks for your replies.
Regarding parameter sharing, were these set ups tuned? Why not share any parameters for the TRPO policies?
Thanks for the additional questions.
The extent of parameter-sharing in the setups was not tuned. We merely followed one possible setup in each of the 3 sets of experiments. Other setups could potentially perform better or worse.
For TRPO, the decision to not sharing parameters (layers) was due to 2 factors:
i) In TRPO, the neural networks we used were rather shallow at only two hidden layers deep.
Hence, we believe that sharing of layers could potentially lead to negligible gains in terms of optimality of policy learnt.
ii) At the same time, we did want to experiment with what happens when different extents of weight sharing are enforced in the FiGAR setup. the FiGAR-TRPO experiments are also meant to demonstrate that in a zero-sharing setup, FIGAR still manages to learn sensible policies, outperforming the baselines in many tasks.
We can later add more experiments with layer-sharing for FIGAR-TRPO to be consistent in the comparisons to the other two setups.
This paper proposes a simple but effective extension to reinforcement learning algorithms, by adding a temporal repetition component as part of the action space, enabling the policy to select how long to repeat the chosen action for. The extension applies to all reinforcement learning algorithms, including both discrete and continuous domains, as it is primarily changing the action parametrization. The paper is well-written, and the experiments extensively evaluate the approach with 3 different RL algorithms in 3 different domains (Atari, MuJoCo, and TORCS).
Here are some comments and questions, for improving the paper:
The introduction states that "all DRL algorithms repeatedly execute a chosen action for a fixed number of time steps k". This statement is too strong, and is actually disproved in the experiments — repeating an action is helpful in many tasks, but not in all tasks. The sentence should be rephrased to be more precise.
In the related work, a discussion of the relation to semi-MDPs would be useful to help the reader better understand the approach and how it compares and differs (e.g. the response from the pre-review questions)
Experiments:
Can you provide error bars on the experimental results? (from running multiple random seeds)
It would be useful to see experiments with parameter sharing in the TRPO experiments, to be more consistent with the other domains, especially since it seems that the improvement in the TRPO experiments is smaller than that of the other two domains. Right now, it is hard to tell if the smaller improvement is because of the nature of the task, because of the lack of parameter sharing, or something else.
The TRPO evaluation is different from the results reported in Duan et al. ICML ’16. Why not use the same benchmark?
Videos only show the policies learned with FiGAR, which are uninformative without also seeing the policies learned without FiGAR. Can you also include videos of the policies learned without FiGAR, as a comparison point?
How many laps does DDPG complete without FiGAR? The difference in reward achieved seems quite substantial (557K vs. 59K).
Can the tables be visualized as histograms? This seems like it would more effectively and efficiently communicate the results.
Minor comments:
-- On the plot in Figure 2, the label for the first bar should be changed from 1000 to 3500.
-- “idea of deciding when necessary” - seems like it would be better to say “idea of only deciding when necessary"
-- "spaces.Durugkar et al.” — missing a space.
-- “R={4}” — why 4? Could you use a letter to indicate a constant instead? (or a different notation)
Thanks for reviewing the paper, the comments and questions! We believe addressing these questions will increase the quality of the work, and we will certainly do that.
-The introduction states that "all DRL algorithms repeatedly execute a chosen action for a fixed number of time steps k". This statement is too strong, and is actually disproved in the experiments — repeating an action is helpful in many tasks, but not in all tasks. The sentence should be rephrased to be more precise.
We agree, the statement in its current form is incorrect. We will change it to “many DRL algorithms execute a chosen action for fixed number of time steps k” in the next revision.
-In the related work, a discussion of the relation to semi-MDPs would be useful to help the reader better understand the approach and how it compares and differs (e.g. the response from the pre-review questions)
Definitely. We will add this to the next revision.
-Experiments:
-Can you provide error bars on the experimental results? (from running multiple random seeds)
The current set of experiments took us nearly 3 months to run. Running them for a significantly large number of random seeds (say 3 or 5) would be very difficult due to the limited nature of compute resources available to us.
-It would be useful to see experiments with parameter sharing in the TRPO experiments, to be more consistent with the other domains, especially since it seems that the improvement in the TRPO experiments is smaller than that of the other two domains. Right now, it is hard to tell if the smaller improvement is because of the nature of the task, because of the lack of parameter sharing, or something else.
We agree. It might take us some time to add those results since we do not have access to the compute resources right now. We will definitely try to add these to the final version.
-The TRPO evaluation is different from the results reported in Duan et al. ICML ’16. Why not use the same benchmark?
The evaluation procedure we have used is very similar to that used by Duan et al. ICML ‘16. The only difference is that instead of reporting average performance on training trajectories (the number of these trajectories used varies across training epochs), we report performance on a testing epoch consisting of a fixed number of trajectories, which has been inserted between every two consecutive training epochs. Note that this is to be consistent with the notion of “solving a task” as introduced by openai.com. We test for 100 episodes between every 2 training epochs.
-Videos only show the policies learned with FiGAR, which are uninformative without also seeing the policies learned without FiGAR. Can you also include videos of the policies learned without FiGAR, as a comparison point?
We have already included the videos for baseline as well. Probably youtube’s default player did not suggest the correct order for the videos. We will make sure that the next revision has a link to a playlist which contains all the videos.
We will additionally also add videos of Atari gameplay in the final version of the paper.
-How many laps does DDPG complete without FiGAR? The difference in reward achieved seems quite substantial (557K vs. 59K).
DDPG completes 2 laps without FiGAR. The complete task consists of 20 laps.
-Can the tables be visualized as histograms? This seems like it would more effectively and efficiently communicate the results.
Definitely. We will add the histograms in the main paper and shift the tables to the appendix in the final version.
-- Sahil & Aravind
This paper provides a simple method to handle action repetitions. They make the action a tuple (a,x), where a is the action chosen, and x the number of repetitions.
Overall they report some improvements over A3C/DDPG, dramatic in some games, moderate in other. The idea seems natural and there is a wealth of experiment to support it.
Comments:
- The scores reported on A3C in this paper and in the Mnih et al. publication (table S3) differ significantly. Where does this discrepancy come from? If it's from a different training regime (fewer iterations, for instance), did the authors confirm that running  their replication to the same settings as Mnih et al provide similar results?
- It is intriguing that the best results of FiGAR are reported on games where few actions repeat dominate. This seems to imply that for those, the performance overhead of FiGAR over A3C is high since A3C uses an action repeat of 4 (and therefore has 4 times fewer gradient updates). A3C could be run for a comparable computation cost with a lower action repeat, which would probably result in increased performance of A3C.
Nevertheless,  the automatic determination of the appropriate action repeat is interesting, even if the overall message seems to be to not repeat actions too often.
- Slightly problematic notation, where r sometimes denotes rewards, sometimes denotes elements of the repetition set R (top of page 5)
- In the equation at the bottom of page 5 - since the sum is not indexed over decision steps, not time steps, shouldn't the rewards r_k be modified to be the sum of rewards (appropriately discounted) between those time steps?
- The section on DDPG is confusingly written. "Concatenating" loss is a strange operation; doesn't FiGAR correspond to a loss to roughly looks like Q(x,mu(x)) + R log p(x) (with separate loss for learning the critic)? It feels that REINFORCE should be applied for the repetition variable x (second term of the sum) and reparametrization for the action a (first term)?
- Is the 'name_this_game' name in the tables  intentional?
- A potential weakness of the method is that the agent must decide to commit to an action for a fixed number of steps, independently of what happens next. Have the authors considered a scheme in which, at each time step, the agent decides to stick with the current decision or not? (It feels like it might be a relatively simple modification of FiGAR).
Thanks for reviewing the paper, the comments and questions! We believe addressing these questions will increase the quality of the work, and we will definitely do that.
- The scores reported on A3C in this paper and in the Mnih et al. publication (table S3) differ significantly. Where does this discrepancy come from? If it's from a different training regime (fewer iterations, for instance), did the authors confirm that running  their replication to the same settings as Mnih et al provide similar results?
The reason why the scores differ significantly is because of 3 reasons:
1. Mnih et al. publication [1] reports average scores on best 5 replicas out of 50 replicas that they started with. We did not mimic this setup because we do not possess the compute resources to run 50 different replicas for each game.
2. The evaluation method used was very different. They used human starts evaluation metric.
However, in the absence of the same human trajectories it would be very difficult to ensure a fair or repeatable evaluation setup.
3.
In fact we have found that the scores in general and evaluation setup in specific reported by Mnih et al. [1] are difficult to reproduce not only for us, but also researchers at Deepmind. In Unifying Count-Based Exploration and Intrinsic Motivation [2] the scores reported for A3C differ very drastically from those reported by the original A3C publication [1], even though the same evaluation metric (human starts) was followed, and hopefully the same set of human start trajectories was used (we do not know this for sure). The scores for many games are orders of magnitude lower for A3C in [2].
In conclusion we’d like to say that the training as well as testing setup of [1] are difficult to reproduce, which in turn makes it difficult to replicate the scores.
- It is intriguing that the best results of FiGAR are reported on games where few actions repeat dominate. This seems to imply that for those, the performance overhead of FiGAR over A3C is high since A3C uses an action repeat of 4 (and therefore has 4 times fewer gradient updates). A3C could be run for a comparable computation cost with a lower action repeat, which would probably result in increased performance of A3C.
Nevertheless,  the automatic determination of the appropriate action repeat is interesting, even if the overall message seems to be to not repeat actions too often.
It is true that for many games the lower action repetitions dominate in the sense that they are chosen for a large fraction of time.
However, the average action repetition (ARR) is a fairer metric to compare the computation cost since FiGAR still makes up by choosing large action repetition at other points in time. Table 5,6,7 in Appendix B seek to demonstrate the action repetition distribution and the ARR for all the games. It can be seen that for 28 out of 31 games, the average action repetition for FiGAR is greater than 4 (which is the ARR for A3C).
Concretely for the best 4 games by gameplay performance, the average action repetitions are (numbers taken from Table 7, page 18, Appendix B):
Atlantis: 7.2
Seaquest: 5.33
Asterix: 4.22
Wizard of wor: 9.87
- Slightly problematic notation, where r sometimes denotes rewards, sometimes denotes elements of the repetition set R (top of page 5)
Thanks for pointing this out. We will change this in the next revision.
- In the equation at the bottom of page 5 - since the sum is indexed over decision steps, not time steps, shouldn't the rewards r_k be modified to be the sum of rewards (appropriately discounted) between those time steps?
Thanks for pointing this out. The question is how should the reward for a macro action m = (a,x) be constructed. Should it be the discounted sum of intermediate rewards encountered during the execution of m or should it be the cumulative undiscounted sum of rewards? We went with the second formalism since we did not want to penalize the agent for choosing larger action repetitions. This we believe would encourage the agent to pick larger action repetitions.
- The section on DDPG is confusingly written. "Concatenating" loss is a strange operation; doesn't FiGAR correspond to a loss to roughly looks like Q(x,mu(x)) + R log p(x) (with separate loss for learning the critic)? It feels that REINFORCE should be applied for the repetition variable x (second term of the sum) and reparametrization for the action a (first term)?
Sorry for this. In DDPG, there is only a single loss function, the critic loss function. There is no loss function for the actor. The actor simply receives gradients from the critic. This is because the actor’s proposed policy is directly fed to the critic and the critic provides the actor with gradients which the proposed policy follows for improvement.
Hence, the actor does not really have a loss function per se, but only gradients provided by the critic. In FiGAR the total policy \pi is a concatenation of vectors \pi_{a} and \pi_{x}.
Hence the gradients for the total policy are also simply the concatenation of the gradients for the policies \pi_{a} and \pi_{x}. This is what we meant by the concatenation operator. We will make the section clearer in the next revision.
- Is the 'name_this_game' name in the tables  intentional?
It is. This is the name of a game in the Atari 2600 domain. Here is a video which shows gameplay in this game: https://www.youtube.com/watch?v=7obD1q85_kw. Note that this video is in no way related to FiGAR and only demonstrates general gameplay in this game.
- A potential weakness of the method is that the agent must decide to commit to an action for a fixed number of steps, independently of what happens next. Have the authors considered a scheme in which, at each time step, the agent decides to stick with the current decision or not? (It feels like it might be a relatively simple modification of FiGAR).
We agree with the reviewer and explain the need for and a possible solution for stopping macro-actions.
Atari, TORCS and MuJoCo represent environments which are largely deterministic with a minimal degree of stochasticity in environment dynamics. In such highly deterministic environments we would expect FiGAR agents to build a latent model of the environment dynamics and hence be able to execute large action repetitions without dying. This is exactly what we see in a highly deterministic environment like the game “Freeway”. Figure 1 (a) demonstrates that the chicken is able to judge the speed of the approaching cars appropriately and cross the road in a manner which takes it to the goal without colliding with the cars and at the same time avoiding them narrowly.
Having said that, certainly the ability to stop an action repetition (or a macro-action) in general would be very important, especially in stochastic environments. In our setup, we do not consider the ability to stop executing a macro-action that the agent has committed to.
However, this is a necessary skill in the event of unexpected changes in the environment while executing a chosen macro-action.
Thus, stop and start actions for stopping and committing to macro-actions can be added to the basic dynamic time scale setup for more robust policies. We believe the modification could work for more general stochastic worlds like Minecraft and leave it for future work.
We will also add this discussion to the conclusion section to reflect possible shortcomings of FiGAR.
[1] - Asynchronous Method for Deep Reinforcement Learning, Mnih et al, ICML 2016
[2] - Unifying Count-Based Exploration and Intrinsic Motivation, Bellemare et al, NIPS 2016
-- Sahil & Aravind
This paper shows that extending deep RL algorithms to decide which action to take as well as how many times to repeat it leads to improved performance on a number of domains. The evaluation is very thorough and shows that this simple idea works well in both discrete and continuous actions spaces.
A few comments/questions:
- Table 1 could be easier to interpret as a figure of histograms.
- Figure 3 could be easier to interpret as a table.
- How was the subset of Atari games selected?
- The Atari evaluation does show convincing improvements over A3C on games requiring extended exploration (e.g. Freeway and Seaquest), but it would be nice to see a full evaluation on 57 games. This has become quite standard and would make it possible to compare overall performance using mean and median scores.
- It would also be nice to see a more direct comparison to the STRAW model of Vezhnevets et al., which aims to solve some of the same problems as FiGAR.
- FiGAR currently discards frames between action decisions. There might be a tradeoff between repeating an action more times and throwing away more information. Have you thought about separating these effects? You could train a model that does process intermediate frames. Just a thought.
Overall, this is a nice simple addition to deep RL algorithms that many people will probably start using.
I'm increasing my score to 8 based on the rebuttal and the revised paper.
Thanks for reviewing the paper, the comments and questions! We believe addressing these questions will increase the quality of the work, and we will definitely do that.
- Table 1 could be easier to interpret as a figure of histograms.
Thanks for pointing this out. We will definitely add a histogram to the final version of the paper corresponding to Table 1.
- Figure 3 could be easier to interpret as a table.
We added a histogram version of this data (Figure3) because we wanted to illustrate that regardless of the action repetition set chosen, the rough "magnitude" of improvement is still the same. That all FiGAR variants continue to significantly outperform the baseline in the chosen games. We will definitely add a corresponding table of raw data in the appendix so that it can be looked up.
- How was the subset of Atari games selected?
The subset was chosen arbitrarily.
- The Atari evaluation does show convincing improvements over A3C on games requiring extended exploration (e.g. Freeway and Seaquest), but it would be nice to see a full evaluation on 57 games. This has become quite standard and would make it possible to compare overall performance using mean and median scores.
The results we have reported on 31 games was possible only after 3 months of computing. It might be difficult to report numbers on all 57 games, however we will definitely try to make the number of games on which we report results as large as possible (We already have results on 2 more games that we will add to the final version of the paper).
- It would also be nice to see a more direct comparison to the STRAW model of Vezhnevets et al., which aims to solve some of the same problems as FiGAR.
STRAW was run on a very small subset of games, namely 8 Atari games. The intersection of games on which both our work and STRAW report results is even smaller at 5 games. Such a comparison is likely to be skewed.
Having said that, we could definitely add scores reported by STRAW on the 5 games that we have evaluated on to Table 4.
- FiGAR currently discards frames between action decisions. There might be a tradeoff between repeating an action more times and throwing away more information. Have you thought about separating these effects? You could train a model that does process intermediate frames. Just a thought.
As pointed out in one of the comments:
"After learning is complete, did you try forward propagating through the network to find actions for every time-step as opposed to repeating actions? Concretely, if at t=5, action suggested by the network is a_3 with a repetition of 4, instead of sticking with a_3 for times t={5,6,7,8} perform action a_3 for just t=5, and forward prop through the policy again at t=6."
This is definitely an experiment worth trying out and we intend to do that and include results if they turn out to be significant.
Having said that, this only makes use of the discarded frames in the testing phase, not in the training phase.
A possible way to trade-off between discarding frames and action repeats is to construct a separate, second  network (consisting of a convnet followed by an LSTM) which processes every kth frame, much like A3C, and concatenate representations learnt by this network to those learnt by the usual A3C network, while making decisions on action selection. The reason one would like to do this is because as you rightly pointed out, skipped frames might also contain crucial information needed for finding out the optimal action as well as action repetition in the next action decision step. We will definitely explore this direction of work as future research. Thanks for the idea!
Thank you for the detailed response and the resulting corrections. I noticed that your reference to the STRAW paper is missing the first author of that work. The correct citation is actually:
"Strategic Attentive Writer for Learning Macro-Actions"
Alexander Vezhnevets, Volodymyr Mnih, John Agapiou, Simon Osindero, Alex Graves, Oriol Vinyals, Koray Kavukcuoglu
Thanks for pointing this out.
In fact a few of the citations in our paper are outdated (we cite the arxiv versions of the paper and not the conference versions). We will definitely correct this (including citations for STRAW and A3C) in the next revision.
We thank all the reviewers for asking interesting questions and pointing out important flaws in the paper. We have uploaded a revised version of the paper that we believe addresses the questions raised. Major features of the revision are:
1. We have added results on 2 more Atari 2600 games: Enduro and Q-bert. FiGAR seems to improve performance rather dramatically on Enduro with the FiGAR agent being close to 100 times better than the baseline A3C agent. (Note that the baseline agent performs very poorly according to the published results as well)
2. In response to AnonReviewer3’s comment about skipping intermediate frames, we have added Appendix F (page 23) by conducting experiments on what happens when FiGAR does not discard any intermediate frames (during evaluation phase). The general pattern seems to be that for games wherein lower action repetition is preferred, gains are made in terms of improved gameplay performance.
However, for 24 out of 33 games the performance becomes worse, which depicts the importance of the temporal abstractions learnt by the action repetition part of the policy (\pi_{\theta_{x}}). This does not address the reviewer’s question completely since at train time we still skip all the frames, as suggested by the action repetition policy. We have added a small discussion on future works section (section 6, page 10) which could potentially address this comment.
3. In response to AnonReviewer3’s suggestion to turn table1 into a bar graph we have done so (Figure 3, page 8) and it indeed does look much better.
4. In response to AnonReviewer3’s suggestion to compare directly to STRAW we have added Table 5 (Appendix A, page 14) which contains performance of STRAW models on all games which we have also experimented with. The general conclusion seems to be that in some games STRAW does better and in some games FiGAR does better.
5. In response to AnonReviewer4’s comment, we conducted experiments on shared representations for the FiGAR-TRPO agent. Appendix G (page 24) contains the results of the experiments. In general we observe that FiGAR-TRPO with shared representations does marginally better than FiGAR-TRPO, but not much better. The performance goes down on some tasks and improves on others. The average action repetition rate of the best policies learnt improves.
6. In response to AnonReviewer4’s comment on SMDPs we have added the relevant discussion to related works section (page 3).
7. In response to AnonReviewer2’s comment on the confusing nature of FiGAR-DDPG section, we have rewritten the section. It is hopefully clearer now.
8. In response to AnonReviewer2’s comment on the confusing notation ‘r’ for action repetition we have completely changed the notation for action repetition to the letter ‘w’.
9. In response to AnonReviewer2’s comment on  the potential weakness of the FiGAR framework, we have added a discussion on the shortcomings of the FiGAR in section 6 (page 10).
10.We have corrected several typos as pointed out by the reviewers.
One can think about the suggestion of not having to execute the repetitive plan but rather find the optimal action at each time step similar to Model Predictive Control. In MPC, at every time step,  the optimal control plans for the entire trajectory using the current state as the initial state but executes only the first action thereby discarding the remaining part of the planned trajectory. A plan is re-computed again with the next state as the initial state. This sort of a dynamic planning is useful to account for unexpected perturbations.
However, in our experiments, the Atari simulator is close to deterministic for most of the games.
Therefore, close to nil unexpected perturbations are potentially encountered in a test-run of the agent and a computed plan can be executed without having to re-plan (Here, a plan is just a "local" repetitive plan around the current state rather than an entire trajectory).
Going by papers that have pointed out the weaknesses of DQN / A3C (Reactive policies learned with a specific simulator) to random unexpected perturbations at test-time, FiGAR (or for that matter, STRAW) is expected to suffer from such unexpected perturbations and a re-planning is necessary for such cases similar to MPC.  It is a good avenue for future work to explore robust planning algorithms for non-deterministic simulators with discrete action spaces.

Meta-review:
role: PC
data: comment: The basic idea of this paper is simple: run RL over an action space that models both the actions and the number of times they are repeated. It's a simple idea, but seems to work really well on a pretty substantial variety of domains, and it can be easily adapted to many different settings. In several settings, the improvement using this approach are dramatic. I think this is an obvious accept: a simple addition to existing RL algorithms that can often perform much better.
 
 Pros:
 + Simple and intuitive approach, easy to implement
 + Extensive evaluation, showing very good performance
 
 Cons:
 - Sometimes unclear _why_ certain domains benefit so much from this

Decision: Accept (Poster)