SIGNIFICANCE AND ORIGINALITY: The authors propose to accelerate the learning of complex tasks by exploiting traces of experts. Unlike the most common form of imitation learning or behavioral cloning, the authors formulate their solution in the case where the expert’s state trajectory is observable, but the expert’s actions are not.  This is an important and useful problem in robotics and other applications.  Within this specific setting the authors differentiate their approach from others by developing a solution that does NOT estimate an explicit dynamics model ( e.g.,  P( S’ | S, A ) ). The benefits of not estimating an explicit action model are not really demonstrated in a clear way. The author’s articulate a specific solution that provides heuristic guidance rewards that cause the learner to favor actions that achieve subgoals calculated from expert behavior and refactors the representation of the Q function so that it has a component that is a function of the subgoal extracted from the expert. These subgoals are linear functions of the expert’s change in state (or change in state features). The resultant policy is a function of the expert traces on which it depends. The authors show they can retrain a new policy that does not require the expert traces. As far as I am aware, this is a novel approach to the problem.  The authors claim that this factorization is important and useful  but the paper doesn’t really illustrate this well. They demonstrate the usefulness of the algorithm against a DQN baseline on Doom game problems. The algorithm learns faster than unassisted DQN as shown by learning curve plots.  They also evaluate the algorithms on the quality of the final policies for their approach, DQN, and  a supervised learning from demonstration approach ( LfD ) that requires expert actions. The proposed approach does as well or better than competing approaches. QUALITY Ablation studies show that the guidance rewards are important to achieving the improved performance of the proposed method which is important confirmation that the architecture is working in the intended way.  However, it would also be useful to do an ablation study of the “factorization” of action values.   Is this important to achieving better results as well or is the guidance reward enough?  This seems like a key claim to establish. CLARITY The details of the memory based kernel density estimation and neural gradient training seemed complicated by the way that the process was implemented.  Is it possible to communicate the intuitions behind what is going on? I was able to work out the intuitions behind the heuristic rewards, but I still don’t clearly get what the Q-value factorization is providing: To keep my text readable, I assume we are working in feature space instead of state space and use different letters for learner and expert: Learner: S = \phi(s) Expert’s i^th state visit:  Ei = \phi( \hat{s}_i }  where Ei’ is the successor state to Ei The paper builds upon approximate n-step discrete-action Q-learning where the Q value for an action is a linear function of the state features: Qp(S,a) = Wa S + Ba where parameters p = ( Wa, Ba ). After observing an experience ( S,A,R,S’ ) we use Bellman Error as a loss function to optimize Qp for parameter p. I ignore the complexities of n-step learning and discount factors for clarity. Loss = E[    R + MAXa’ Qp(S’,a’)    -   Qp(S,a)   ]   The authors suggest we can augment the environment reward R with a heuristic reward Rh proportional to the similarity between the learner “subgoal" and the expert “subgoal" in similar states.  The authors propose to use cosine distance between representations of what they call the “subgoals” of learner and expert.  A subgoal is defined as a linear transformation of the distance traveled by an agent during a transition. The heuristic reward is proportional to the cosine distance between the learner and expert “subgoals" Rh = B  <   Wv LearnerDirectionInStateS,  Wv ExpectedExpertDirectionInStatesSimilarToS   > The learner’s direction in state S is just (S-S’) in feature space. The authors model the behavior of the expert as a kernel density type approximator giving the expected direction of the expert starting from a states similar to the one the learner is in.  Let < Wk S, Wk Ej > be a weighted similarity between learner state features S and expert state features Ej and Ej’ be the successor state features encountered by the expert. Then the expected expert direction for learner state S is: SUMj  < Wk S, Wk Ej > ( Ej - Ej’ )  Presumably the linear Wk transform helps us pick out the important dimensions of similarity between S and Ej. Mapping the learner and expert directions into subgoal space using Wv, the heuristic reward is Rh = B <   Wv (S-S’),  Wv SUMj  < Wk S, Wk Ej > ( Ej - Ej’ ) > I ignore the ReLU here, but I assume that is operates element-wise and just clips negative values? There is only one layer here  so we don’t have complex non-linear things going on? In addition to introducing a heuristic reward term, the authors propose to alter the Q-function to be specific to the subgoal. Q( s,a,g ) = g(S) Wa S + Ba The subgoal is the same as the first part, namely a linear transform of the expected expert direction in states similar to state S. g(S) =  Wv   SUMj  < Wk S, Wk Ej >  ( Ej - Ej’ )  So in some sense, the Q function is really just a function of S, as g is calculated from S. Q( S,a ) = g(S) Wa S + Ba  So this allows the Q-function more flexibility to capture each subgoal in a different linear space? I don’t really get the intuition behind this formulation.  It allows the subgoal to adjust the value of the underlying model?  Essentially the expert defines a new Q-value problem at every state for the learner?  In some sense are we are defining a model for the action taken by the expert? ADDITIONAL THOUGHTS While the authors compare to an unassisted baseline, they don’t compare to methods that use an action model which is not a fatal flaw but would have been nice.  One can imagine there might be scenarios where the local guidance rewards of this form could be problematic, particularly in scenarios where the expert and learner are not identical and it is possible to return to previous states, such as the grid worlds the authors discuss: If the expert’s first few transitions were easily approximable, the learner would get local rewards that cause it to mimic expert behavior. However, if the next step in the expert’s path was difficult to approximate, then the reward for imitating the expert would be lower. Would the learner then just prefer to go back towards those states that it can approximate and endlessly loop? In this case, perhaps expressing heuristic rewards as potentials as described in Ng’s shaping paper might solve the problem. PROS AND CONS Important problem generally.  Avoiding the estimation of a dynamics model was stated as a given, but perhaps more could be put into motivating this goal.  Hopefully it is possible to streamline the methodology section to communicate the intuitions more easily.