A Hierarchical Bayesian Approach to Inverse Reinforcement Learning with Symbolic Reward Machines

Abstract: A misspecified reward can degrade sample efficiency and induce undesired behaviors in reinforcement learning (RL) problems. We propose symbolic reward machines for incorporating high-level task knowledge when specifying the reward signals. Symbolic reward machines augment existing reward machine formalism by allowing transitions to carry predicates and symbolic reward outputs. This formalism lends itself well to inverse reinforcement learning, whereby the key challenge is determining appropriate assignments to the symbolic values from a few expert demonstrations. We propose a hierarchical Bayesian approach for inferring the most likely assignments such that the concretized reward machine can discriminate expert demonstrated trajectories from other trajectories with high accuracy. Experimental results show that learned reward machines can significantly improve training efficiency for complex RL tasks and generalize well across different task environment configurations.

Reviews and author responses:

 Our experimental results show that by only using 10 and even less expert demonstrated trajectories the agent can attain high performance. Note that those trajectories are expert trajectories, not the training data for the agent policy networks. Similar results have been seen in other IRL works. For instance, in the GAIL paper ("Ho and Ermon. Generative adversarial imitation learning. NeurIPS 2016."), the performance is more or less the same when the numbers of demonstrations decreases from 10 to 1 in multiple tasks. Our experiments show similar results.

 7. The conjunctions and linear arithmetic can be understood as that the symbolic constraint is a conjunction of linear inequalities.

8. We cite the exploration driven RL in a different category, i.e., the **Reward Design**, as opposed to in the **IRL** category. 
Exploration driven RL agents rely on intrinsically generated rewards to attain high performance. 

9.  We describe GAIL and Bayesian GAIL in the Background section because GAIL is a well-known GAN-like IRL technique and our derivation in Section 5 adopts a similar GAN-like scheme.  The $J_{adv}$ loss term (the GAIL objective) introduced in Eq. (2) at the end of the Background section is used in the derivation of Eq. (5) and (6) in Section 5. We refer to Eq. (2) multiple times in Section 5 when we derive Eq. (5).We also directly compare with GAIL in the experiments. 
We will try to make the connection more explicit in the revision. 

9. The concatenation operators are introduced only to formalize Definition 2. As we mentioned in the paragraph below Proposition 1, we treat the trajectory, which is the concatenation of state-action pairs, as a list. 

10. The expressiveness of SRM is more related to symbolic automaton/transducer (SA, ST) than to standard FSA. 
SA and ST have been extensively studied in Veanes et al. (2012) as well as in the works that follow it. 
We are the first to adopt such a technique in RL and we start by using IRL to learn the unknown variables. 

11. Our experimental results show that by only using 10 and even less expert demonstrated trajectories the agent can attain high performance. Note that those trajectories are expert trajectories, not the training data for the agent policy networks. 
Similar results have been seen in other IRL works. For instance, in the GAIL paper ("Ho and Ermon. Generative adversarial imitation learning. NeurIPS 2016."), the performance is more or less the same when the numbers of demonstrations decreases from 10 to 1 in multiple tasks. 
Our experiments show similar results.

12. In our ablation study, we show with three MiniGrid tasks that when the parameter search space is $\mathbb{R}^n$, or the product $\mathbb{R}^{n_1}_{\geq 0} \times \mathbb{R}^{n_2}_{\leq 0}$, where $n$ and $n_1+n_2$ are the number of holes in the SRMs, our algorithm still efficiently achieves high performance. 
In contrast, as shown in our appendix, SRMs with randomly sampled parameters in those parameter spaces do not produce high performance. 

 * The KL-divergence in Eq(6) is decomposed into an entropy $H(q)$ and a $E_q[\log p(l)]$. We retain the entropy $H(q)$. The confusion here is at how we implement $E_q[\log p(l)]$. We note that in the paragraph above Eq(6), we define $p(l)$ as being $0$ where $l$ does not satisfy the symbolic constraint $c$ and uniform everywhere else in the space of $l$'s. Given this definition, we view $E_q[\log p(l)]$ as a barrier function, i.e. if there exists an $l$ such that $p(l)=0$ and $q(l)$ is larger than some constant value, then $q(l)\log p(l)$ goes to $\infty$; for all other $l$'s s.t. $p(l)>0$, we have $\underset{l: p(l)>0}{\sum} q(l)\log p(l)\propto \underset{l: p(l)>0}{\sum} q(l)$ since $p$ is uniform. In our implementation, instead of implementing this $E_q[\log p(l)]$, we use a heuristic in $J_{con}(q)$ by penalizing $q$ with a large loss if $\arg\max_l q(l)$ violates the symbolic constraint, otherwise we let the loss be $0$. We use this heuristic mainly for efficiency reasons, since we only use the most likely $l^*=\arg\max_l q(l)$ to train the agent policy $\pi_A$. As $\arg\max_l q(l)$ in fact corresponds to the Gaussian mean output by $q$, the heuristic makes loss $J_{con}$ **differentiable** w.r.t $q$. Otherwise, we would need to sample $l\sim q$.

* We now justify the relationship between this heuristic and the original $E_q[\log p(l)]$ by analyzing the allowable set {$l|p(l)>0$} and the distribution $q$. The symbolic constraint $c$ we consider in this paper is a group of linear inequalities of variables **$?_{id}$**, i.e. the holes. In the paper since we denote the assignment to those holes **$?_{id}$** as **$h_{id}$**, any allowable **$h_{id}$** should satisfy this family of linear inequalities in the real space. As mentioned in the paper, each assignment **$h_{id}$** corresponds to a unique concretized $l$. Hence, the allowable set {$l|p(l)>0$} is in turn determined by this family of linear inequalities. Meanwhile, $q$ specifies the distribution of $l$ by determining the distribution of **$h_{id}$**, which is a multi-variant Gaussian distribution with diagonal variance matrix. In this case, $\arg\max_l q(l)$ corresponds to the mean of **$h_{id}$**, which is directly output by $q$. Our heuristic, by penalizing $l^*=\arg\max_l q(l)$ that violates the symbolic constraint, is in fact optimizing $q$ such that the mean of **$h_{id}$** moves towards its allowable space as indicated by the family of linear equations aforementioned. 
Indeed, this $J_{con}$ is different and is weaker than the original  $E_q[\log p(l)]$. 
However, *the original KL-divergence $KL(q(l)||p(l))$ penalizes $q$ with infinite loss as long as the support of $q(l)$ only partially overlaps that of $p(l)$, which is not practical since the support of the Gaussian distribution determined by $q$ is the real space*. 
Besides, our ultimate goal is not to solve a distribution of $l$'s but rather to identify one single $l$ that not only leads to high performance of the agent policy but also satisfies the symbolic constraint. 

* As for the second question, we note that in our baselines, both AGAC and RIDE, which have achieved high performance in MiniGrid, use state visitation counts-based intrinsic reward to construct the reward function. It makes their reward functions **non-Markov** and this approach has been heavily leveraged by various other intrinsic reward RL methods as well. On the contrary, our approach does not rely on the state visitation counts to achieve high performance in the experiments. 

* In terms of the concern on the comparison with other baselines with the similar amount of prior knowledge on the reward structure, especially the IRL approaches using a linear combination of **hand-crafted** state features, 
we want to note that in the long list of papers on the MiniGrid Github page, we did not see any that employed such an approach and reported successes. 

* Besides, we want to point out that, in the conventional feature matching IRL approaches, the feature-based reward is a linear function of the weights. However, the reward output by SRM is not a linear function of the holes. Note that there are **holes in the predicates** on the transition edges in all our designed SRMs. Those predicates act like conditionals determining whether to execute the following $\lambda$-terms to produce rewards. 

* **We do recognize that it is important to evaluate the human-effort aspect of our approach.** 
Thus, we performed two ablation studies where one involved weakening/removing the symbolic constraint, and the other involved using random assignments to the holes (Research Question E and F in Appendix 1.1). 
As shown in Fig5(a) and (b), the performance of the "ablated" RL agents are far below those trained with SRMs concretized via Algorithm 1.    


The paper presents a method for inverse reinforcement learning for learning the non-Markovian (trajectory-based) rewards for symbolic transitions of a pre-speciefied (up to free variables that need to be learned) automaton. For this purpose, Symbolic Reward Machines (SRM) are introduced, which define symbolic states, transitions and rewards, whereby MDP-trajectories can be unambiguously mapped to SRM-trajectories. The method uses adversarial imitation learning, where the rewards that are computed by the SRM (for a given grounding of the free variables) are used by a discriminator to classify expert- and agent-trajectories. Following the framework of Bayesian GAIL, a distribution over groundings/rewards is learned. However, whereas Bayesian GAIL learns a particle approximation to the posterior, the submission learns a variational Gaussian distribution. The method is evaluted on (rather challenging) gridworlds, where it is able to learn rewards for concepts such as "picking up a key after finding the goal".
 
Merits:
- Inverse reinforcement learning with structured, symbolic rewards is a very promising topic as it can help to learn more generalizable and explainable reward functions by leveraging expert domain knowledge.

- The approach is original and has contributions both in the formalization of the SRM and the procedure for training them.

Weaknesses:
- I found the paper rather difficult to follow due to a bad presentation. I have some open questions, and also several suggestions regarding the structure / presentations, which are outlined below.

- I have several concerns regarding the technical soundness:
  * The method replaces the term J_con by a binary cross entropy loss that penalizes the constraint-violations of the mean hole-assignment. This objective is quite different from J_con as it does not take into account the covariance of the variational distribution, and does not reward its entropy. This simplification is also not well-motivated: Couldn't we use samples instead of the mean and reward the entropy of the Gaussian to get an objective much closer to J_con that would still be tractable?
  * The method changes the weighting between J_con and J_soft, which is somewhat hidden by the step-size scaling $\eta$ which is actually quite large (1e8).
  * It is not clear why the argmax of the variational distribution is used (line 3 in Algorithm 1) rather than computing the average reward based on samples as Bayesian GAIL.

Questions:
* I can not make sense of the definition of the dummy transition (last line on page 3, term over the arrow). I guess it should read as "For every outgoing transition, the guard is False", but I can not see how to parse it that way. What does the and-operator apply to? Should the $\neg{\phi}//0$-part be lowered (right now it looks like a superscript of $\delta$)?
* First line at page 4: Why is it "at any state $q \in \mathcal{Q}$" rather than $p$. The expression seems to state that for any L-state $q$, there is only one incoming transition. However, I think it should state that for any L-state $p$ there is only one outgoing transition. For example consider 3 L-states (0, 1, 2). Assume, the initial L-state is $0$ and there are two transitions (with $\phi$ = True), ending in state 1 and 2 respectively. This SRM does not seem to violate the definition of "deterministic", still any MDP-trajectory could be mapped to the L-trajectory 0->1 and 0->2.
* Page 8, Gridworlds: Fig. 3b) I assume the green key is for the green door. What is the difference between doors of different color and gray doors? For exampe, there is a red door, but no red key. Fig. 3c) The agent does not seem to be able to reach the blue target (both doors have obstacles in front).
* Why don't we see a difference in performance between 1 demonstration and 10 demonstrations (Fig. 3g-i). Is the task so simple that 1 demonstration is sufficient?
* I can't make sense out of the transitions in Appendix A2.1. For example in the sentence beginning with "The transitions are designed mostly based on high level human insights [...]" the transition seems to parse as "If the goal was reached, the agent must have unlocked a door after picking up a key". However, in the diagram in Fig. 8, the agent can reach the goal directly from the "Before Unlocking" state.
* In the experiments, it is not always clear what settings (RL vs IRL) were used. For example, Fig 3d): Is PPO(CNN) and AGAC(CNN) using a sparse MDP reward function (not using any demonstrations) and "Algo1+X" using the demonstrations (and not the sparse reward functions)?
* Is the LSTM only used for the policy or also for the value function? Since the learned reward is not Markovian in the MDP-states, I think the states for the RL algorithm should be given by the history of all states, and thus, the LSTM should be applied both for value function and policy, right?
* The free variables mainly show up as rewards. However, they also appear within guards, e.g. in the self transition to the "After Unlocking" state in Fig. 1b). Can the method also learn the dynamics of the abstract transitions? The self-transition is kind of a special case, because it only affects the reward in the end (If the rule is not applicable, the dummy transition is used which is also a self-transition). But in principle it should also be possible to transition to different states depending on the free variable. Did any of the SRMs make use of the possibility to use free variables to parameterize the "control flow" of the abstract machine?
* How does the approach compare to using standard GAIL/AIRL for matching expert-specified trajectory features. It would be possible to compute features $\phi(\tau_{1:t})$ for a partial trajectory $\tau_{1:t}$ of the expert/agent-trajectories and provide these features as input to the discriminator in order to learn a feature-based reward function $r_t = r(\tau_{1:t})$ for matching these features. Similar to the current work, the RL agent would learn in an MDP where the state is given by the complete history of state-actions. The features could be as expressive as the SRM-trajectories, e.g. a feature could indicate how often a door was closed and whether a key was dropped after unlocking a door. What are the main benefits of the proposed approach?

Suggestions:
* Section 3: Equations 1,2 should be better explained to make the paper more self-contained (the section makes little sense if you do not read the Bayesian GAIL paper). One or two sentence, that highlight that BGAIL uses a different discriminator objective compared to GAIL would already be very helpful.
* The fact that the learned reward is non-Markovian should be discussed. What implication does this have on the generator update (RL-part)?
* The guard of self transition to the "After Unlocking" state in Fig 1b) needs to be better explained (I saw that it is explained in the appendix, but when reading the paper it was quite confusing).
* Last paragraph of page 4: I suggest to move most of it to the caption of Fig 1b, to make the Figure more self-contained.
* The Gaussian noise $\epsilon$ needs to be motivated.
* The structure of the presentation of the optimization needs to be improved. I would mention the final objective (ELBO maximization) earlier, by mentioning that a variational approximation to the posterior is learned. It should be made more clear, that this objective is used for learning the reward function and variational distribution.
* The "binary cross-entropy loss" for replace $J_\text{con}$ needs to be better explained. The main paper (not the appendix) should state clearly that J_con is actually not used, and how the cross-entropy loss is defined. This part needs to be better motivated. If using samples from $J_\text{con}$ and and entropy objective (as I suggested earlier) did not work, this should be mentioned.
* Algorithm 1: It is not clear from the pseudo code how line 5 and 6 have any effect. Maybe remove them, or assign them to variables that are provided to J_soft. It needs to be better motivated why line 3 uses the argmax, rather than samples from the Gaussian variational approximation.
* The stepsize $\eta$ needs to be discussed
* The paper uses the discriminator structure of AIRL, which seems quite crucial, because for the GAIL-discriminator the learned reward would converge towards a constant function that would not generalize. I think this detail needs to be discussed better.
* Some of the questions I mentioned above should also be answered by the paper.

__Update__
I read the other reviews and the authors' rebuttal. I decreased the score from "accept" to "slightly above borderline", mainly for two reasons:
* The rebuttal did not properly address my concern with $J_\text{con}$. I was not talking about the sign in front of the objective, but I was criticizing that the actual objective was replaced without proper justification. While it is reassuring that at least an additional entropy objective was used (which was not mentioned in the initial submission, and also now only seems to be mentioned in the appendix), the objective is still significantly different than the original objective.

* I think, I was too generous in my initial review. The paper has several clear flaws, in particular with the weaknesses of empirical evaluation (other non-Markovian method, that include similar amount of prior knowledge on the reward structure should perform similarly well), and hence the paper should be assessed as borderline. However, I don't see a major problem in the current submission, so I still think that the paper should be accepted. The paper proposes an original approach to the interesting topic of inverse reinforcement learning of structured rewards. I have some concerns regarding the technical soundness, in particular related to the objective $J_\text{con}$. The presentation of the work needs to be improved.


The paper presents a new IRL algorithm built on top of a reward machine (FSA-like) representation of a reward function.  The machine is lifted to relational predicates that may have free variables that need to be inferred.  The IRL procedure takes in a set of expert trajectories that are used to ground (or “concretize”) these variables and an RL algorithm (such as PPO) that can be used to learn policies based on reward functions.  A GAN approach is then used until the policy converges to behavior that matches the experts.  Empirical results are shown in 3 mini-grid scenarios. UPDATE:
I thank the authors for their revisions and clarifications on several of the comments. The authors make a good point that mini-grid domains can be difficult. I do, however, remain concerned that the approach may not be scalable (for instance the inability to learn more than just a few parameters holes).  The authors’ comment about only needing 10 expert trajectories to learn a reward function hints that the domains may not be sufficiently complex to judge an IRL algorithm and multiple reviewers pointed out that not much changed from using 1 demonstration to 10.

I also remain concerned about the confusing comparisons of paradigms here.  I would really liked to have seem comparisons against more IRl algorithms rather than the cross-paradigm comparisons to RL and exploration techniques.

----------------------------------------

I am very happy to see work being done on relational representations for reward learning, which is an important topic for the field and a key for expanding domains where RL/IRL is applicable.  However, the empirical results in this paper on 3 mini-grid worlds are too small-scale and also compare with unreasonable baselines.  In addition, the current paper requires much more focus on specific problems and a more formal description and theoretical analysis of the proposed representation before it is ready for a top-tier publication.  I expanded on these issues below.

On the empirical side, mini-grid is a good domain for showing proof-of-concepts, but a work like this that purports to lift IRL to more realistic domains needs to at least show the approach is applicable in a more realistic setting.  More importantly, the baselines for comparison here are inappropriate.  The authors compare their IRL approach, which has the benefit of expert trajectories and an extremely expressive reward function that is only missing a few parameters, to RL algorithms like PPO that have to learn from scratch.  IRL and RL are fundamentally different problems, and comparing an algorithm that has access to trajectories and a powerful reward representation to one without such data and knowledge is bound to come out as the current results show.  The authors should instead focus on comparing strictly to other IRL algorithms, including linear IRL learners with relational features, or other IRL approaches that learn FSMs or relational representations (for instance see the paper “Inverse Reinforcement Learning in Relational Domains”)

The paper also lacks a clear and formal problem definition and focus on a specific tractable problem.  The paper is not clear up front about what parts of the representation need to be learned (it turns out just a few parameter holes need to be filled in?) and whether the learning agent is aware of the Reward Machine representation (how is the reward Markovian if the agent does not know about the FSM state?).  Many assumptions seem to also spring up out of necessity rather than being part of the problem constraints- for instance the assumptions about conjunctions and linear arithmetic on Page 6.  

The paper also contains many references to related work that tis not actually related, such as the discussion of exploration algorithms in RL, which is not relevant for an IRL algorithm.  There is also a significant amount of notation that is developed but never used.  This includes the GAIL terminology on Page 3, which could be left as a black box and the concatenation operators on page 4, neither of which seem to be used in the paper.

Finally, I was disappointed that the paper did not rigorously investigate the expressivity of the new representation.  If the main contribution here is a complex relational reward representation, what kinds of things can it represent?  What can it not represent?  The authors introduce operators like the # (counting) operator that seem to lift the expressivity of the reward machine beyond an FSA.  If so, is reasoning with this representation tractable?  Is it possible to learn any more than a handful of parameters?  How is all of this learning done with just 10 expert trajectories (unless the parameter space is very constrained)?  The paper should provide some theoretical investigation of the representation itself since that is the main contribution here. 
 The paper has a very interesting reward function representation but seems to confuse the IRL and RL cases and compares the two empirically.  The examples are also too small scale and the theoretical analysis of the new reward function is under-developed.

 We made the following revisions to our submission. 

* Reorganized the introduction on GAIL in the Background section to clarify how we embed the GAIL learning objective in our learning framework and how we implement the IRL baselines in our experiments.

* Emphasized in Section 5 that the GAIL objective is embedded in our learning objective 

* Fixed a few typos.

* Emphasized in the experimental section that we did compare the results of Algorithm 1 with IRL baselines and that we included the results of exploration driven RL algorithms for reference since they achieved SOTA in some of the tasks.

* Emphasized in the experimental section that we directly use the SRMs concretized in small environments to train RL agents in large environments and compare with training RL agents with default reward. By doing this, we show that our concretized SRMs can generalize to differently sized environments.

* Changed the legend of GAIL and GAN-GCL to GAN+PPO(CNN) and GAN-GCL+PPO(CNN) in Fig.3d to indicate that the GAIL and GAN-GCL baselines are implemented with PPOs.

* Added a curve for the experimental result of PPO(CNN) in Fig.3l.

* Emphasized in the experimental section that we implemented both CNN and LSTM versions of RL policies to tackle non-Markovian rewards, but we only reported the results of the ones that achieved higher performance in each task to make the plots less congested.


 We thank you for the insightful comments.
1. **Regarding questions on using cross-entropy instead of KL-divergence in the loss function** 
    
    * Since $p(l)$ in Eq.6 returns non-zero probability for $l$'s that do not violate the symbolic constraint and returns zero everywhere else, the KL-divergence becomes infinitely large when a sampled $l$ does violate the symbolic constraint. In order to lift the sampling process when estimating the KL-divergence, we adopt a heuristic here by substituting the KL-divergence with a cross-entropy loss. 
    
    * As described in our Appendix A.3 we use a ReLU function to ensure that when the constraints are not violated, this cross-entropy loss will not update the parameters of $q$.
    
    * We maximize $J_{con}$ because it includes the negative sign before the KL-divergence. 
    
    * Thanks for pointing out the entropy term.
    We did retain the entropy term extracted from the KL divergence in our implementation to handle the variance specified by $q$. The hyperparameter we used for the entropy term is $1E-2$. 
    
    * The reason for using a large $\eta=1.e8$ is that in PyTorch the cross-entropy loss has an upper-bound.
    
2. **Other issues**

    * In our definition of dummy transition, the $and$ operator is applied for everything after it before the // sign. The subscript below the $and$ operator specifies all the transitions from $p$ included in $\delta$.
    
    * The first line on page 4 should be $p\in Q$. Thank you for pointing out this typo.
    
    * The green key in Fig.3b is for the green solid door. All other hollow doors can be opened without keys.

    * It is not uncommon to see only a small difference between the result from using 1 demonstration and that from using 10 demonstrations. 
    For instance, in the GAIL paper ("Ho and Ermon. Generative adversarial imitation learning. NeurIPS 2016."), the performance is more or less the same when the numbers of demonstrations decreases from 10 to 1 in multiple tasks. 
    Our experiments show similar results.
    
    * The transition edge from "Before unlocking state" to "reach goal" is added in case of the situation where the goal is right in front of the agent after initialization, since the description of mini-grid does not rule out such a possibility.
    We note that the existence of this edge does not influence the results at all because the internal state transition will never pass this edge if this situation does not occur. 

    * Yes, in our experiment, PPO(CNN) and AGAC(CNN) use a sparse MDP reward function (e.g. the default goal-driven reward), and Algo1+X uses the demonstrations. We note that although we add PPO and AGAC to the baselines, our experiments do not aim at comparing IRL with RL. 
The goal of our experiments is to demonstrate the utility of SRMs in improving the performance of existing RL algorithms compared with them using the default goal-driven rewards (described in the first paragraph of Introduction and in Example 1), exploration-driven rewards (which can be viewed as a form of reward design) like in AGAC and RIDE, or rewards obtained from other IRL algorithms like GAIL.
Our results show that standard RL algorithms like PPO using our inferred SRMs significantly outperforms AGAC and other baselines using the default (or reshaped using exploration-driven heuristics) reward and PPO using GAIL or GAN-GCL.

    * An LSTM is always used in the reward function $f$. 
    We also experimented with using LSTM in both the PPO and AGAC policies to tackle the non-Markovian rewards, in which case LSTMs are used in both the value and policy networks. 
    However, the performance is not always better than that of the CNN version.

    * Since we have holes in the predicates on the transition edges, different hole assignments do possibly lead to different transitions. But the state space and the transition rules are not parameterized. 

    * The main reason that we choose GAIL and AIRL over IRL approaches based on linear combination of state features in our baselines is that GAIL and AIRL are more recent and more representative of the current SOTA in IRL. The second reason we did not include IRL with linear combination of features is that the latter requires handcrafting the state features which are not available in this benchmark. In the long list of papers on the MiniGrid Github page, we did not see any that employs such an approach and reports successes.
    
    * We sample SRMs in line 5 to estimate $J_{soft}$ as we mentioned in the last paragraph of section 5. 
    We add line 6 because we use a stochastic reward function $f+\epsilon$ as in equation 6.  
 
    * We use the mean returned by $q_\varphi$ to train agent policy rather than using sampled SRMs to save the sampling process and to avoid using diverged SRMs.
    
    * It is indeed crucial to use the discriminator structure of AIRL instead of GAIL. We will try to make this more explicit in our revision. 
 

 1. **Small-scale environments in the benchmarks.** First, we cannot agree with the reviewer on the comments that MiniGrid examples are too small-scale. These benchmarks are not the typical simple gridworld examples that we see as running examples in papers, but are in fact very challenging for RL. 
    * The environment is only partially observable as the agent only sees the 7x7 tiles in front of it and does not know its coordinate in the environment. The state space in the environment is in fact huge.
    * AGAC, one of the most advanced exploration driven RL algorithms, reportedly takes less than 10M steps to tackle a VizDoom task with high performance, but takes 200M steps to only achieve a moderate performance on the ObstructedMazeFull task in MiniGrid.
    * We pick the MiniGrid environment for the very reason that its state space is large, it requires the agent to perform a complex sequence of actions to reach the goal, and is generally very challenging for RL (the MiniGrid Github page has a long list of papers that aim at tackling this benchmark).

2. **Inappropriate baselines.** In the experiments, we did include IRL approaches, i.e. GAIL and GAN-GCL, in our baselines. 
We omit the lines for GAN-GCL and GAIL in some of the plots because they do not perform at all, i.e. close to 0 average return.
We noted this in the paragraphs where we describe results for different tasks.
Although we add PPO and AGAC to the baselines, our experiments do not aim at comparing IRL with RL. 
The goal of our experiments is to demonstrate the utility of SRMs in improving the performance of existing RL algorithms compared with them using the default goal-driven rewards (described in the first paragraph of Introduction and in Example 1), exploration-driven rewards (which can be viewed as a form of reward design) like in AGAC and RIDE, or rewards obtained from other IRL algorithms like GAIL.
Our results show that standard RL algorithms like PPO using our inferred SRMs significantly outperforms AGAC and other baselines using the default (or reshaped using exploration-driven heuristics) reward and PPO using GAIL or GAN-GCL.

4. **IRL based on linear combination of state features.** The main reason that we choose GAIL and AIRL over IRL approaches based on linear combination of features in our baselines is that GAIL and AIRL are more recent and more representative of the current SOTA in IRL. The second reason we did not include IRL with linear combination of features is that the latter requires handcrafting the state features which are not available in this benchmark. In the long list of papers on the MiniGrid Github page, we did not see any that employs such an approach and reports successes.

5. **The parameters to be learned.** We define in Definition 3 that the free variables in the `$\lambda$`-terms and the predicates are the parts to be learned. Completing programs with holes is among the most significant research topics in program synthesis (known as sketch synthesis). We take inspiration from that line of work and are the first to apply a similar idea to reward design in RL.


6. **Non-markovian reward.** The reward is Markovian in the product `$M\otimes L$` in Definition 2. 
Nevertheless, it is correct that the reward can be non-Markovian if the agent does not know about the SRM internal state. 
However, many real-world rewards are in fact non-Markovian, as mentioned in the paper `Gaon and Brafman. Reinforcement learning with non-markovian rewards. AAAI 2020.` 
In our experiments we include the testing on RL agents both with and without memory (CNN and LSTM respectively) and report 
their performances on handling non-Markovian rewards. 

 We thank you for the insightful comments.

1. *Difference between RMs and SRMs:*

    * The difference between RMs and SRMs is similar to the difference between finite-state automata (FSA) and symbolic automata/transducer (SA/ST). 
    As we defined in Definition 1, SRM is different from RM not because of unknown rewards, but
    because SRM is equipped with predicates and $\lambda$-terms on the transition edges.
    While the rewards on the transition edges of an RM are all constants, the rewards of SRM are returned by the $\lambda$-terms (both RM and SRM take trajectories as inputs).
    SA and ST have been extensively studied in Veanes et al. (2012) as well as in the works that follow it. 
    There are no unknown components in an SRM unless it is an **SRM with holes** as defined in Definition 3. The holes can be in the predicates or in the $\lambda$-terms.
    Existing works on RMs assume that human designers have manually provided the reward values on the transition edges. In contrast, we do not make this assumption.
    That is the very reason that we formulate the concretization problem in Definition 3 for concretizing **SRMs with holes**. 

2. *Comparisons with previous work:*

    * We compare our approach directly with approaches that have achieved the highest performance known at the time of our submission on the MiniGrid benchmarks. Those approaches do not strictly follow a Markov structure for the reward function. For instance, exploration-driven RL approaches generate intrinsic rewards in addition to the Markov reward returned by the environment. Those intrinsic rewards are dependent on the status of the RL agent (e.g. state visitation count in AGAC and RIDE), past trajectories and so on.

    
    * We also compare our results with representative inverse reinforcement learning algorithms such as GAIL and GAN-GCL. As shown in Section 6.1, those
    approaches do not produce any performance (i.e. close to zero average return) when combined with the same RL algorithm (e.g., PPO or LSTM) in any of the MiniGrid tasks. 
    
    * Regarding the comments on Toro-Icarte et al. [1], we would like to quote the following sentences from that paper `Finally, note that the definition of a perfect RM does not impose conditions over the rewards associated with the RM (i.e., $\delta_r $ ). This is why $\delta_r$ is a free variable in the model LRM. However, we still expect $\delta_r$ to model the external reward signals given by the environment. To do so, we estimate $\delta_r(u, l)$ using its empirical expectation over T (as commonly done when constructing belief MDPs.`
    This means the RM learning framework in that paper does not identify the reward values by solving the MILP. It estimates them empirically to match the environmental default reward. 
    That paper in fact does not provide any further detail on how those rewards are estimated. 
    Moreover, 
    Toro-Icarte et al. [1]
    proposes to learn an FSA but **fixes the number of states**, the set of atomic propositions, and the labeling function a priori, while our paper targets symbolic automata. 
    
    * For Shah et al. [2], and Vazquez-Chanlatte et al. [3], we note that both works aim at learning task specifications (mainly for interpretability reasons), and not reward functions.
    Given a temporal logic task specification, one still has to determine the appropriate reward value to be returned. 
    In the literature, such reward structures usually return a predetermined constant value (e.g. 1) or a random reward upon the satisfaction of the specification, or when reaching an acceptance state in the automaton translated from the specification. 
    There is no known literature that shows these approaches can work well in the MiniGrid environments.
 
 
    * The paper Toro-Icarte et al. [4] is about reward shaping on the RM from a predetermined concrete reward mapping  (e.g. default goal-driven reward).
    In contrast, our paper considers a reward mapping that contains unknown components.
    The difference between our paper and that paper is similar to that between IRL and reward shaping for RL.



 We thank you for the insightful comments.

* SRMs can work with continuous control tasks such as Mujoco in theory.
Our definition of SRM (Definition 1) allows both continuous state space and continuous action space. 
The predicates and the $\lambda$-terms on the edges are basically functions and can be designed to process continuous actions.
We plan to investigate this further in future work. 

* We want to additionally emphasize that while the Mini-Grid examples can look deceptively simple, they are in fact very challenging benchmarks for RL.
For instance, AGAC, one of the most advanced exploration driven RL algorithms, reportedly takes less than 10M steps to tackle a VizDoom task with high performance, while taking 200M steps to only achieve a moderate performance in ObstructedMazeFull task in mini-grid. 


* The time complexity of Algorithm 1 is equal to the time complexity of running a GAIL algorithm plus that of optimizing the KL-divergences in $J_{adv}$ and $J_{con}$. Since we use the logarithmic trick in $J_{adv}$, the time complexity will be $O(Km)$ where $K$ is the total number of concretized candidate SRMs and $m$ is total number of sampled trajectories. For the KL-divergence in $J_{con}$, 
which we simplified as we mentioned in the paper,
the time complexity is $O(1)$. The space complexity is equal to that of GAIL plus additional space for storing rewards returned by each candidate concretized SRM along each sampled trajectory, which amounts to $O(Km)$.


The paper presents Symbolic Reward Machines (SRM) which helps in achieving interpretable and explainable reward functions. The paper also proposes a hierarchical Bayesian framework to concretize Symbolic Reward Machines using expert demonstrations. The author evaluates the proposed method on a Mini-grid environment with varying complexity and compares against state-of-the-art methods. - The paper is well written with good details on SRM and proposed hierarchical Bayesian learning framework.
- The author presents good motivating example for SRM.
- The paper presents good literature survey and situates the proposed work well with current literature.
- The paper evaluates the proposed method on a grid-world which discrete problem. It would be good to see some experiments on continuous control tasks (e.x. Mujoco). At least the authors can comment on how the proposed method on continuous control tasks.
- The authors are also encouraged to comment on how these method can be extended on a real-robotics tasks.
- What is the time and space complexity of Algorithm 1?
 Already explained above.

This paper deals with the problem of learning non-Markov task specifications (or rewards) encoded as a reward machine by inferring the real-value parameters of a reward machine (typically the edge transition rewards in an RM). The authors adopt a Bayesian deep learning approach to simultaneously learn a reward approximator, a distribution over the parameters of the reward machine, and the policy to optimize the most-likely reward instance. 

The authors then go on to demonstrate in grid world environments that benefit from non-Markov rewards and policy representations that the inferred RM outperforms standard reward definition using standard RL approaches. And they further demonstrate that the inferred reward machines are beneficial  in larger instances within the same domain. **Major comments**:

*Differences between Reward machines and SRMs*: The authors propose SRMs as a novel framework, but I wasn't able to grasp how SRMs differ from regular reward machines with unknown edge transitions reward. Perhaps it would be a good idea for the authors to reframe their presentation to be contrastive in terms of the different representation in regular reward machines and symbolic reward machines. 

*Comparisons with previous work*: The primary comparisons provided by the authors seem to be with approaches that do not model the non-Markov structure of the reward function in learning from demonstration. Thus these approaches are bound to demonstrate poorer performance when confronted with tasks where the reward function and therefore the policy is dependent on the history. Reward machines are a way to succinctly summarize the history, and thus make the synchronous product Markov, thus allowing the use of MDP based RL algorithms. A true baseline for this task would include reward specifications learnt in non-Markov forms. Toro-Icarte et al. [1] learn reward machines directly (but as opposed to this paper, they learn both the structure and the reward parameters, whereas this work only focuses on the reward parameters, and assumes the strucuture to be given). In the same vein, Shah et al. [2], and Vazquez-Chanlatte et al. [3] infer LTL formulas that can be translated into sparse reward machines. In fact Toro-Icarte et al. [4] have also proposed reward shaping algorithms that transform sparse LTL-RM rewards into denser reward machines that are easier to learn. But the comparisons in this paper merely show that inferring structural reward information is helpful. That said, the proposed approach is different from prior published approaches, but a direct comparision is helpful towards a more complete study

*Clarity concerns:* This work appears to build significantly on Bayesian GAIL and the work by Fu et al. 2018. Certain concepts and notations from those works are built upon without adequate motivation or explanation in my opinion. It certainly helps the paper to be self-contained. Further the first three paragraphs of Section 5 will be served well with a major rewrite such that the mathematical reasoning flows better in the paper. 

*Results by varying number of examples:* I had quite a bit of trouble in intuiting how the change in number of training example would impact the solution quality. Are those curves essentially flat, or is it indicated in some examples that fewer examples is actually more benefecial?

[1] - Toro Icarte, Rodrigo, et al. "Learning reward machines for partially observable reinforcement learning." NeurIPS (2019): 15523-15534.

[2] - Shah, Ankit Jayesh, et al. "Bayesian inference of temporal task specifications from demonstrations." NeurIPS (2018).

[3] - Vazquez-Chanlatte, Marcell, et al. "Learning task specifications from demonstrations." NeurIPS (2018).

[4] - Icarte, Rodrigo Toro, et al. "Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning." arXiv preprint arXiv:2010.03950 (2020).
 I believe learning symbolic reward machines is an interesting problem framing, and the paper is enroute to making an important contribution. My primary concerns remain the positioning of the paper with respect to prior work and the difference of the formalism vis-a-vis the original definition of reward machines. I am inclined towards this paper but not in its present form, and I can be persuaded to upgrade my score if the presentation of the technical approach is improved, and the method is contrasted qualitatively or empirically with some of the prior works in learning non-Markov task specifications

Meta-review: The paper proposes a framework of Symbolic Reward Machine (SRM), an extension of Reward Machine (RM), for specifying interpretable and explainable reward functions. Then, a Bayesian IRL method is proposed to concretize an SRM using expert demonstrations. Experimental results demonstrate the effectiveness of SRMs in terms of training efficiency and generalization. The reviewers acknowledged that the problem of inferring symbolic rewards is important and that the proposed SRM framework is an important step in this direction. However, the reviewers pointed out several weaknesses in the paper and shared concerns, including (a) limited comparison with alternate approaches to tackle the problem and positioning w.r.t. the existing literature; (b) the domains seem rather simple since they require only a few demonstrations (implying that the holes being inferred might be quite small); (c) the novelty and theory around the SRM representation is not fully clear. I want to thank the authors for their detailed responses. Based on the reviewers’ concerns and follow-up discussions, there was a consensus that the work is not ready for publication. The reviewers have provided detailed feedback to the authors. We hope that the authors can incorporate this feedback when preparing future revisions of the paper.

Decision: Reject