Average-Reward Learning and Planning with Options

Abstract: We extend the options framework for temporal abstraction in reinforcement learning from discounted Markov decision processes (MDPs) to average-reward MDPs. Our contributions include general convergent off-policy inter-option learning algorithms, intra-option algorithms for learning values and models, as well as sample-based planning variants of our learning algorithms. Our algorithms and convergence proofs extend those recently developed by Wan, Naik, and Sutton. We also extend the notion of option-interrupting behaviour from the discounted to the average-reward formulation. We show the efficacy of the proposed algorithms with experiments on a continuing version of the Four-Room domain. 


Reviews and author responses:

The paper extends the options framework to the average reward reinforcement learning setting. The paper proposes methods to learn on SMDP level as well as the intra-option level. Additionally, the paper proposes method for planning and interrupting options under this setting.  The paper's structure is similar to the classical options paper: it proposes algorithms for all the settings presented in [1] while adapting them to the average reward setting. The paper is clearly written and flows nicely. The average reward setting is also important when we consider function approximation, and therefore extending options to this setting is a promising avenue.

Although the paper tackles multiple settings, it is not clear where some original contributions are being made (as opposed to simple extensions from the discounted setting to the average reward setting). A discussion on the originality and importance after each theorem (as is sometimes done in theory) would really help in understanding the merit of the paper.

The results for the inter-option algorithm seem to be the most original ones and as such the remarks following the theorem are appreciated. The empirical results compare to [2], however this paper is not cited in the introduction around L40, and as such is harder to situate in terms of existing work. The results presented in Figure 1-2 seem to paint a good picture for the proposed approach. However, the sensitivity in Figure 1 mentions that $\beta = 2^{-1}$. Why aren't there more values presented?

The results for the intra-option, planning and interrupting algorithms seem to be less important contributions, more discussion to clarify this would be appreciated. The empirical results also lack a bit of context. Although the discounted setting and the average settings are different, couldn't there be comparisons where a sensitivity to the hyperparameters are shown? In the current form, the paper brings the question as to why we should even bother extend the options to the average setting, as there is little motivation intuitively or through experiments. 

A small precision: In L211, it is said that all options are updated. However, in [1] it is clearly said that all consistent options are, which brings a key difference. 

[1] Sutton et al., Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
[2] Gosavi et al. Reinforcement Learning for Long-run Average Cost


=================Post-rebuttal===================
I appreciate the authors response. The theoretical derivations seem rigorous, however I am not sure how much novelty overall there is, except for theorem 1. Moreover, the authors did not provide convincing experimental results, and in their rebuttal do not intend to. This is quite negative for me. Although the theoretical contributions are present, but are not really surprising and as such I will keep my score as borderline. Limitations are not really discussed in a clear way.

No societal impact discussion.

 Thank you for the comments, we have responded to both:

**Comment**: “I was more interested in settings which you are able to motivate 'options' versus 'actions'... adding a discussion of this sort explicitly in the paper would be helpful”

**Response**: Indeed, we shall discuss the utility of options early in the introduction as part of revising the motivation early. As an example of options available in real-world applications, many real-world robots today (such as a vacuuming robot) have a combination of low-level actions such as torques of individual motors and high-level 'options' such as 'turning 90 degrees to the left', 'going forward till an obstacle is detected', or even 'go back to the charging dock'


**Comment**: “...some of the theorems and assumptions need to be presented more clearly”

**Response**: Thank you for the suggestions, we shall rephrase some of the theoretical results for clarity, especially the theorem and remark in Section 3.


 I thank the authors for their responses to my questions and points of confusion.  Your responses were helpful in my understanding of the paper.

 Thank you so much for responding to each of my comments directly.  I have a couple of follow up comments though:

1. "The motivation of the options framework is never explicitly described in the main paper"

I agree that there is a lot of motivation to move to the average reward setting versus the infinite horizon discounted or finite horizon, especially as multiple operational problems (queueing systems, caching systems, inventory control, etc) are more aptly modeled in the average reward setting.  I was more interested in settings which you are able to motivate 'options' versus 'actions', i.e. in what real-world problems do algorithm designers have access to a set of options policies versus just knowing the actions available in the system.  Especially as the experiments presented are simply toy grid-world examples without any real-world connection, adding a discussion of this sort explicitly in the paper would be helpful.

2. “Moreover, in section 3, the authors present the algorithm assuming that the option length is estimated and unknown. However, under remark 138 the authors then state the theorem assumes the expected option lengths are used in the equations.”

Thank you for clarifying this point.  I think this still highlights how some of the theorems and assumptions need to be presented more clearly.  Especially organizing the appendix more directly to mimic the organization of the main paper (highlighting in the main paper where the proofs in the appendix are located would help).  The remark makes it seem as though the theorem is only valid when the actual mean options lengths are used, but your comment suggests otherwise.  

 Thank you for taking the time to review the paper and write a detailed summary! We also appreciate your positive comments on the novelty, soundness, and clarity of the paper. We have responded to your comments one-by-one:

**Comment:** “Some of the theorem statements are presented rather informally (with a full presentation in the appendix)” 

**Response:** We tried to present intuitive versions of the theorem statements in the main text while reserving the precise formal statements for the appendix. We felt such a presentation would make the paper more accessible for a large fraction of the readers. 

**Comment:** “The motivation of the options framework is never explicitly described in the main paper.”

**Response:** There is a strong motivation throughout reinforcement learning for moving from the discounted objective to the average reward objective, primarily because, once genuine function approximation is employed, the discounted problem is no longer well defined and is deprecated (see Sutton & Barto 2018, Section 10.4; Naik et al. 2019). If we switch to the average-reward formulation for the basic problem, and we want the advantages of options in acting, learning, and planning, then we must extend options to the average-reward setting as well. One must be done along with the other. We realize in retrospect that we did not clearly present this motivation in the paper. Now that we see that it is missing, we will add it to the revision, early in the introduction.

**Comment:** “Moreover, in section 3, the authors present the algorithm assuming that the option length is estimated and unknown. However, under remark 138 the authors then state the theorem assumes the expected option lengths are used in the equations.”

**Response:** It seems there is a small misunderstanding; there is no contradiction. The option length is indeed unknown and estimated in the algorithm. $\hat{L}$ refers to the sampled option length (i.e., which is observed after actually executing the option) while $L$ refers to the estimated expected option length. As in (8), $L(\cdot,\cdot)$ is learned using $\hat{L}$. The remark on line 138 tries to say this. In addition, it points out that using the estimated expected option length $L$ in the algorithm yields solutions of (2), but this is not guaranteed to be true if the sampled length $\hat{L}$ is used instead.
Thank you for pointing out a source of potential confusion for readers; we shall reword the Remark to highlight this important distinction. 

**Comment:** “the introduction paragraph under section five doesn't help highlight the different approach presented in this section, being a model-based algorithm that specifically estimates the semi-MDPs transitions versus a Q-learning algorithm presented before.”

**Response:** Section 5 defines option-models in the average-reward case and provides algorithms to learn such models. We agree that specifying the topic clearly in the first paragraph of this section would improve the overall clarity of the paper.

**Comment:** “Have you performed a comparison between inter and intra option learning frameworks through an experimental simulation?”

**Response:** We have not performed this comparison because there is an obvious computational advantage of intra-option algorithms over inter-option algorithms: intra-option algorithms (1) not only make updates during the execution of an option, (2) they also update all possible options that could have resulted in the low-level transition at the given timestep. An interesting comparison as such would help verify this hypothesis but would not affect the main contributions of the paper.

**Comment:** “In line 120 - if the option length is known, the algorithm would just plug in the true value?”

**Response:** Yes, in (6) and (7), knowing the option length is equivalent to the estimated optional lengths being accurate. Using those would also lead to a solution guaranteed by Theorem 1.

**Comment:** “Related work section?”

**Response:** In section 1 (Introduction), we cover the relevant related work and place our contributions in the existing body of knowledge. Hence the paper does not have a separate ‘related work’ section.

We thank you again for your time. Please let us know if you have any follow-up questions or comments.

Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction. || 
Naik, A., Shariff, R., Yasui, N., Sutton, R. S. (2019). Discounted reinforcement learning is not an optimization problem.


 Thank you for taking the time to review the paper. We are glad that you agree with the importance of extending options to the average-reward setting and feel that the paper is well-written. We have responded to your comments one-by-one:

**Comment:** “Although the paper tackles multiple settings, it is not clear where some original contributions are being made (as opposed to simple extensions from the discounted setting to the average reward setting). A discussion on the originality and importance after each theorem (as is sometimes done in theory) would really help in understanding the merit of the paper.”

**Response:** We agree that the technical difficulties as well as the originality and importance of our contributions are not explicitly highlighted in the current manuscript. We will add more discussion about it in the revision. For instance:
- We generalize Wan et al.’s proof techniques to show convergence for both the cases of actions and options, as well as inter- and intra-option algorithms. We feel it is quite elegant to have such a unified proof for all the cases.
- The options case requires special treatment on the algorithms side. For example, in the case of primitive action, there is no need to learn the option length; there are no inter- and intra-option algorithms.

**Comment:** “The empirical results compare to [2], however this paper is not cited in the introduction around L40, and as such is harder to situate in terms of existing work.”

**Response:** Gosavi’s (2004) work is cited in the beginning of that paragraph (line 33). We also point out in the same paragraph that it is a convergent off-policy inter-option learning algorithm whose convergence proof requires a special recurrent state. 

**Comment:** “… the sensitivity in Figure 1 mentions that beta = -1. Why aren't there more values presented?”

**Response:** We indeed tested different values of alpha as well as beta (exact values specified in line 170). We found that the algorithm was not very sensitive to the choice of beta. This, in addition to the page limit, was why we presented the sensitivity w.r.t. alpha in Figure 2 for a single value of beta. As mentioned in line 187, we presented the sensitivity w.r.t. other values of beta in the Appendix B.1.

**Comment:** “The results for the intra-option, planning and interrupting algorithms seem to be less important contributions, more discussion to clarify this would be appreciated.” 

**Response:** We largely agree — in terms of innovation, the main insights are in the inter-option algorithms and theory; the intra-option, planning, and interruption contributions are more derivative in nature. But in terms of their implications, we think intra-option, planning, and interruption algorithms are crucial for maximizing the utility of the options framework. 
As per your suggestion, we will add more discussion in the introduction to clarify the relative importance and relationships between the contributions. It will indeed improve the quality of the final paper!

**Comment:** “Although the discounted setting and the average settings are different, couldn't there be comparisons where a sensitivity to the hyperparameters are shown?”

**Response:** We think the comparison is not appropriate. The key point is that the two problem settings are different — maximizing the rate of reward and maximizing the discounted sum of rewards. Our intent is to extend options to the average-reward setting, not compare the average-reward setting to the discounted one (in retrospect, we feel we did not make this sufficiently clear, and we plan to make it more so in the revision). From this point of view, a comparison with a solution designed for the discounted setting would be inappropriate and difficult to interpret.

**Comment:** “In L211, it is said that all options are updated. However, in [1] it is clearly said that all consistent options are, which brings a key difference.” 

**Response:** This is indeed a key difference — the use of the importance-sampling ratio makes our algorithm more general than Sutton et al.’s because our algorithm can handle the more general case of stochastic option policies (the remark starting L244 talks about this). When the option policies are deterministic (as Sutton et al.’s algorithm requires), our algorithm also only updates the 'consistent' options. 

We thank you again for your time. Please let us know if you have any more questions. We are happy to provide further clarifications. We hope you will consider re-evaluating your score after this discussion.


 Thank you for taking the time to review the paper. We are glad to see your positive comments on the novelty, quality, clarity, and significance of the paper. We have responded to your comments one-by-one:

**Comment:** “I think the significance of the paper would be strengthened through the use of more than one (relatively simple) domain.”

**Response:** The experiments in the paper currently serve to validate the theoretical contributions. We agree that further empirical demonstrations on domains of varying complexities would improve the confidence on the proposed algorithms’ efficacy — this is one of the most natural directions of future work.

**Comment:** “A discussion of why the goal state was changed between some of the experiments would also be helpful to understand the experimental setup.”

**Response:** Thank you for the suggestion, we shall add clarifications about why we chose the particular goals per experiment. In the inter-option experiments (Section 3), learning to reach G1 would demonstrate the ability to leverage options alongside primitive actions. In the intra-option experiments, we wanted to demonstrate that option values can be learned without ever executing any options. G2 can be reached using only options, so if the agent learns to do that without ever executing any options, the algorithm is indeed working as expected. Finally, you mentioned that the role of G3 was already clear.

**Comment:** “In equations (1) and (2), it might be helpful to add some notation (possibly as a superscript) to clearly indicate the purposes of the two q(s, o) functions so that they aren't accidentally misunderstood. And likewise for u^q(s, o) in (11) and (12).”

**Response:** Indeed, we thought about that but it’s a little tricky. Unlike the discounted case (where we could use $q_\mu$ and $q_*$ respectively), in the average-reward case, there are infinite solutions to the Bellman equations, all offset by a constant (Puterman 1994). In other words, there isn’t a unique solution like in the discounted case. To keep the presentation simple, we decided to go with the generic symbol $q$ to denote all the solutions to the Bellman equations. 

**Comment:** “I think the summation in line 290 should be over s',l and not r,l, correct?”

**Response:** You’re right, thanks for pointing that out!

We thank you again for the positive review. We hope we have addressed your concerns. Please let us know if you have any follow-up questions. 

Puterman, M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming

 Thank you for taking the time to review the paper. We have responded to your comments one-by-one:

**Comment:** “The proof techniques have been borrowed heavily from existing literature and the algorithm is a very natural extension of Q-learning to options framework”.

**Response:** We largely agree that the final inter-option Differential Q-learning algorithm is a natural extension of Differential Q-learning to the options framework. However, its development was quite challenging: our first 3 attempts of extending the Differential Q-learning algorithm — arguably even more natural than the current form — were not successful. We discussed these attempts in Section 3 (around line 115 as well as 138) and in Appendix C.1. Additionally, our convergence proofs indeed build on several existing proof techniques (Wan et al. 2021, Abounadi et al. 2001, Puterman 1994, Sutton et al. 1999 — attributed appropriately). 

We will try to add more discussion about the originality and importance of our algorithmic and theoretical contributions in the final paper. For instance:
- We generalize Wan et al.’s proof techniques to show convergence for both the cases of actions and options, as well as inter- and intra-option algorithms. We feel it is quite an elegant contribution to have such a unified proof for all the cases.
- The options case requires special treatment on the algorithms side. For example, in the case of primitive action, there is no need to learn the option length; there are no inter- and intra-option algorithms.

**Comment:** “... My question is then is this requirement of a special recurrent state only for theory?” 

**Response:** Indeed, Gosavi’s algorithm itself does not require the specification of a special recurrent state. But Gosavi’s proof does not guarantee convergence without assuming the existence of the special recurrent state. On the other hand, our algorithm is guaranteed to converge irrespective of the existence of such a state. The proof techniques have significant differences, for instance, we used a single-timescale ODE while Gosavi used a two-timescale ODE.

**Comment:** “Numerically, the authors said they observed that their algorithm had less variance than the baseline, what is the reason behind that?”

**Response:** Gosavi’s algorithm uses a sample average of rewards to estimate the reward rate; our method uses the TD error to estimate the reward rate. It is possible that the sample average estimate results in larger variance. Exercise 10.8 in Sutton & Barto’s (2018) textbook provides some evidence to this hypothesis: it shows when the reward-rate estimate is accurate, the TD error is zero while the sample average error is not zero. But further work is required to confirm this hypothesis.

We thank you again for the positive review. We hope we have addressed your concerns. Please let us know if you have any follow-up questions. 


This paper presents an options framework for average-reward MDPs. They present a Q-learning based algorithm for such setting. The authors then provide performance guarantees (almost-sure convergence) and then demonstrate on a Four-Room domain.  The main contribution of this paper is to provide a algorithm with performance guarantees (more generalized than earlier work). While discounted setting is relatively easier to handle (thanks to discount factor), the average-reward criteria needs more work. The author present a variant of Q-learning for average-MDPs with option framework. The proof techniques have been borrowed heavily from Wan et. al. (2021) and Abounadi et al (2001). The authors also presented experiment results on Four-room (a grid domain).  1. The proof techniques have been borrowed heavily from existing literature and the algorithm is a very natural extension of Q-learning to options framework.
2. The authors compare against Gosavi's work by arguing that its convergence proof requires the underlying SMDP to have a special state that is recurrent under all stationary policies. My question is then is this requirement of a special recurrent state only for theory? Because for experiments, when authors compared with it as a baseline, they did not choose any special state. I believe it might be an artifact of the proof because the proposed algorithm and the existing one are very similar. Numerically, the authors said they observed that their algorithm had less variance than the baseline, what is the reason behind that?

This paper addresses the problems of learning and planning in MDPs with options (i.e., higher-order policies) in the average-reward (rather than discounted reward) MDPs.  Learning and planning algorithms are proposed for (1) inter-option settings, where learning and planning occur between switches in the option followed by the agent, (2) intra-option settings, where the agent learns or plans between low-level actions, and (3) the interruption setting, where agents can actively choose to cease following an option during its execution.  Theoretical results establish the convergence of the learning and planning algorithms, and their performances are demonstrated on examples of the four room domain.   ===Originality===

Although much of this work extends previous results (with proper attribution), it finds novelty by doing so in a less-studied setting (average-reward MDPs).  Indeed, it provides the first intra-option average-reward algorithms and demonstrates their convergence to near-optimal policies, as well as improvements over existing algorithms in the inter-option and interrupted settings.  

===Quality===

The quality of this work is appropriate for a NeurIPS paper.  The mathematics used to develop the algorithms appear to be sound.  The theoretical results are accompanied by sufficient proofs (albeit in the appendices), and the empirical results are reasonably analyzed.  

===Clarity===

The paper was well written and relatively easy to follow.  It should be accessible to a general audience with some background in MDPs and RL, which is its target audience.

===Significance===

This paper offers several significant contributions to the literature on average-reward MDPs with options.  Novel algorithms are proposed for several settings, theoretical results establish their important properties, and empirical results demonstrate their performance.  Average-reward MDPs are not as commonly used as discounted-reward, although increased attention in the literature could change that.

I think the significance of the paper would be strengthened through the use of more than one (relatively simple) domain, and a discussion of why the goal state was changed between some of the experiments would also be helpful to understand the experimental setup (the use of room 3 in the interruption experiments was well explained).

===Minor Suggestions===

-- In equations (1) and (2), it might be helpful to add some notation (possibly as a superscript) to clearly indicate the purposes of the two q(s, o) functions so that they aren't accidentally misunderstood.  And likewise for u^q(s, o) in (11) and (12).

-- I think the summation in line 290 should be over s',l and not r,l, correct? Several limitations of the work were addressed in the final section.  I don't anticipate significant negative societal impacts of this research.

This paper considers reinforcement learning in an infinite horizon Markov decision process reward function $r$ and transitions $p_{s,a}$ for each $s$ and $a$ in $S \times A$.  The high level goal is to use the known model efficiently in order to learn a policy $\pi$ mapping states $s$ to actions $a$ which maximizes the cumulative reward experienced acrossed the trajectory.  In contrast to typical modeling assumptions, here the authors make two distinctions from prior work.  First, they consider the average cost setting instead of the discounted setting, where the goal is to instead maximize the limiting average rewards accumulated across the trajectory.  Second, they consider the so-called 'options' framework where instead of directly learning a policy over actions, the agent has access to a set of options $O$.  Each option consists of two components.  The first, is the policy $\pi$ mapping states to a distribution over actions.  The second is a probability distribution of the option's termination, mapping states to a value in $[0,1]$ for the probability the option terminates from that specific state.

Each option executes as follows.  First, upon visiting a state $s_t$ the agent choses an action according to the current option's policy.  The agent then observes the next state, picks an action according to the next state, and the option terminates with some probability.  The behavior of the agent is then classified according to the policy mapping states to options (instead of actions).  This process can be marginalized out to generate a new process whose actions correspond direction to options, but loses some of the Markovian properties to become a semi-MDP.  The goal in this framework is exhibited through three separate ways, all of which are addressed by the authors.

First the authors consider inter-option learning.  In this model the agent plans with options instead of primitive actions.  Given an option, the learning algorithm occur only after the options simulated execution is finished.  As such, these models are a direct form of Semi-MDP algorithm.  Here the authors provide an extension of differential learning and planning algorithms from average-reward MDPs to MDPs with options, which essentially is a form of $Q$ learning with the updates over options instead of actions, and an extra term scaling with the expected option length.  They show that their approach is consistent, leading to accurate estimates of the value of the policy, or that their algorithm is also consistent in learning the optimal policy as well.

Second the authors consider the intra option learning.  Different from inter-option, here the algorithms can update after each state-action transition within an options executed is completed.  From a single state-action transition, the algorithm is able to learn or plan to improve the value or policies for all options that may have also generated the same transition.  As such, they are more efficient than inter-option learning.  Here the authors again extend $Q$ learning based algorithms, and present a model based planning algorithm as well.  These algorithms are slight modifications of those for inter-option learning as they are able to update all options simultaneously after taking an action, potentially utilizing importance weights to fix the distribution shift issue.

Third, the authors investigate whether 'early termination' helps improves agent policy performance.  They tackle this issue by understanding the performance of a modified problem where each option is linked to a new option where the termination distribution is replaced via guaranteed termination whenever the value of $Q(s, o)$ is smaller than the value $V(s)$ for being in the state according to a policy $\mu$.  In Theorem 6 the authors show that this process allows for learning a policy which is no worse than the old one, and strict improvement whenever there is a non-zero probability of encountering an interruption.  In essence, this result highlights that allowing the agent to terminate an options once a state is visited where a separate option can allow for better performance guarantees policy improvement.

Lastly they complement these three theoretical directions with experimental results.  The experimental results are constructed to help highlight the theoretical convergence of their algorithm, serving more of a 'theoretical proof of concept' than an empirical performance benchmark.  This seems sufficient for there setting due to a lack of related work to offer comparisons with, and the experiments were chosen in a way to help highlight different aspects of the algorithmic and options framework.
  ### Originality:

The authors present a novel extension of the options framework to average cost MDPs.  While the approach presented seems like a straightforward modification of existing $Q$ learning algorithms for average-cost MDPs formulated in the options setting, the authors clearly express the new strategies required for modeling the options and theoretical tools needed.

### Quality:

The submission is technically sound and the theoretical and empirical claims are well supported.  The authors are honest and upfront on the new techniques used in the modeling, proof, and algorithm development, namely:
- access to an existing set of options.  The problem of option discovery is a tangential and related problem, and discussed in the conclusion as a direction for future work and
- discrete set of state  actions (and small enough to be computationally feasible to store and maintain estimates over)

### Clarity:

The submission is clearly written and well organized.  However, some of the theorem statements are presented rather informally (with a full presentation in the appendix).  Potentially the main paper should include some more of the technical details to help with readability.  At a high level, the authors never provide any concrete motivation for the options framework in contrast to the typical MDP model.

Moreover, in section 3, the authors present the algorithm assuming that the option length is estimated and unknown.  However, under remark 138 the authors then state the theorem assumes the expected option lengths are used in the equations.  This point should be clarified more and presented in Theorem 1 as it seems to contradict the algorithm presentation.

Similarly, the introduction paragraph under section five doesn't help highlight the different approach presented in this section, being a model-based algorithm that specifically estimates the semi-MDPs transitions versus a $Q$ learning algorithm presented before.

### Significance: 

The results presented in this paper provide a novel extension of $Q$ learning and model-based algorithms to average cost MDPs under the option framework.  While the novel theoretical basis is quite limited (the technical details are strong, but the model is not motivated at all in the paper). The work can be built upon by better investigating the simultaneous problem of learning options, and extending the algorithms for large state action spaces.

### Strengths:

The main strengths of the paper are as follows:
- the extension of average cost MDPs to the options framework
- the experimental results presented in the main paper and the appendix provide a detailed description of their algorithm's performance

### Weaknesses:

The main weaknesses of the paper are as follows:
- the motivation of the options framework is never explicitly described in the main paper
- some of the technical details are omitted in the main text

### Questions and Comments:
- Have you performed a comparison between inter and intra option learning frameworks through an experimental simulation?
- In line 120 - if the option length is known, the algorithm would just plug in the true value? 
- Related work section? The paper focuses on the theoretical foundations for developing optimal control and evaluation algorithms for average cost MDPs with options, and so it is expected that it will not have any potential negative impacts. The authors are also clear on the limitations of their work including:
- access to an existing set of options.  The problem of option discovery is a tangental and related problem, and discussed in the conclusion as a direction for future workand
- discrete set of state  actions (and small enough to be computationally feasible to store and maintain estimates over)

Meta-review: After a lot of discussion amongst the reviewers as well as with the authors, this paper was judged to be strictly borderline. Extending the options framework to the average reward setting is a fine line of inquiry, but as such the setup of this paper is rather weakly motivated as observed by multiple reviewers and the algorithms and analysis are relatively incremental in their novelty. Even for toy experiments, we would have at least liked to see the authors illustrate some capability afforded by the average reward formulation which the discounted formulation lacks, for instance. Perhaps the paper is more suited as a comprehensive journal submission, with a treatment of function approximation and/or option learning.

Decision: accept