Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper considers an interactive RL ting where the agent and the human may have different sensory capacities. It then shows agents may learn only sub-optimal policies if they do not take into account human advice. A counterfactual agent is proposed to include the results of actions from humans. A new RL algorithm is also proposed considering that there is a budget constraint over the available amount of human advice.
Main review: The paper is generally well written and present interesting ideas within the specific MDPUC and DSCM models. My major concern is whether the foundation results, namely, Cor 6 and Thm 7, are indeed meaningful.
-	First, it is assumed that agent cannot access $U$, but human can. My question is why this is a rational setting? 
-	Second, in Thm 7, the condition “the human decision is affected only by observable variables” indicates that the human decisions cannot bring in more information than what the agent observes. In other words, as no more information is introduced, it is intuitive that human decisions are not useful if we consider the asymptotic performance.
-	Third, I do not find any condition on the human decision, which is a bit surprising to me. E.g., even if human can access $U$, but his/her decision is a constant (say, always 1), then it does not provide any information. 

Minor: $U^{(t1)}, U^{(t2)}$ seems not defined.

In summary, the paper presents interesting ideas, with the specific RL setting. However, it seems to me that whether the human can help is equivalent to whether there could be more information. In this sense, the contribution of the paper is weakened.
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper studies the role of human instructor in the interactive setting, and shows that it is helpful to keep human in the loop as long as it has information regarding to the task, though not playing it in an optimal way.
Main review: Significance:
This paper studies an interesting topic, i.e., what is the role of humans for interactive reinforcement learning settings. To formally study this, the paper defines the experimental agent and counterfactual agent and identifies the conditions that when human input is valuable to achieve optimal performance. The interesting part is that it shows when human has specific information for the task that is not accessible to the agent, though they do not perform it optimally, it could still boost the performance of the agent.

Quality/Clarity:
This paper is easy to read. By formally defining the related concepts in the causal language, the setup is clear and makes the paper easy to follow and understand. I appreciate the identification of the conditions that when human input can provide valuable information. It would be great (if possible) to characterize this in a more quantitative way. Intuitively the quality of human play should have some effect on the learning process of the agent, this might could be achieved by imposing some human players' policy and seeing how it affects the final performance. Another interesting part I like of the paper is the trade-off between optimality and autosomally through a polynomial optimization problem, though it would be great if the paper could discuss some scalability of the optimization problem. 

Minor comments:
Maybe I am missing something here, the paper mentions that the agents could still benefit from the human's advice though the reward and transition are fully known (with some features unobservable), is there any experiments showing this empirically?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: Summary: This paper considers the following setting of RL: one agent can only access part of the confounders in the system. Even this agent can access all the reward function and state-transition functions, the learnt policy may still be sub-optimal. Then the authors propose the so-called "counterfactual agent" and a hybrid approach to improve the regret.
Main review: Strengths:

The theory developed herein is relatively self-complete. The simulation studies are relatively comprehensive. The polynomial optimization formulation of the hybrid agent is quite novel and potentially has strong practical and theoretical relevance (from the algorithmic perspective). This may also open up the possibility of studying statistical-computational trade-offs in settings related to this paper. In particular, the hybrid setting is also related to recent literature on combining data with no unmeasured confounding with observational data with unmeasured confounding to dynamical settings. In this case, though, the trade-off between causal and non-causal can be balanced under the resource (access to human decision-makers) constraint.

Weaknesses/Other comments:

1. The paper did not reference many important works on time-varying causal inference (which is essentially RL + causality) in statistics: e.g. Murphy JRSSB 2003, Robins 2004, Dudík-Erhan-Langford-Li Stat. Sci. 2014.

2. From the perspective of causality, it is not at all surprising that the Experimental Agents (EA) as defined in this paper may learn sub-optimal policies when EA does not have access to all confounders. I believe this should be discussed in detail in the first section and the intuition by causal thinking should also be presented. I think it is also important to explain the intuition to readers why even when human decision-makers are inferior to random guessing, it is still beneficial to have this component in the algorithm.

3. As I mentioned, the hybrid setting seems to be closely related to the recent literature in causality that tries to combine real-world data for which strong ignorability fails to hold and data with no unmeasured confounding. Could the authors briefly comment on this?

4. In the hybrid setting, which algorithm is used for the polynomial optimization problem? How long does it take to run the algorithm? I had the impression that Sherali-Adams is fast but SDP is generally not scalable to even moderately big systems. It might be helpful if the authors can further comment on the practicality of these choices, in particular during the presentation of the simulation results.

5. MDP might not be a realistic assumption in particular in clinical applications. It might be helpful if the authors could make some further comments on how to extend their works to non-MDP settings in the Discussion section.

Overall, the flavor of causality in this paper is not very strong and the kind of idea borrowed from causality is rudimentary. But the optimization algorithms, the theoretical justification, and the detailedness of the authors' analyses are worth sharing with the causal inference community.

Minor typos:

Corollary 19 proof (Appendix page 17): $F_{CTF}$ should be $\Pi_{CTF}$.
review_title: null

Meta-review: Thank you for your submission. Reviewers appreciated this paper's use of counterfactual reasoning to analyze and characterize when human advice (including poor advice) based on state unobserved by an agent can inform a more optimal dynamic policy. The paper's setting is interesting, and the work provides meaningful results. Reviewers particularly appreciated the papers' formulation of the optimization problem and the trade-off between optimization and autonomy.

We appreciate the motivating example and intuition provided by the authors in their response; and encourage them to include this in a final revision, together with other details and clarifications from the author responses to reviewer questions.
Decision: Accept (Poster)