Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers the use of a factored Q-function representation for reinforcement learning that takes advantage of a factored action space in certain "appropriate" MDPs, where "appropriate" is characterized both theoretically and qualitatively/practically in the paper. The authors show empirically that their factored representation can help in low-data healthcare settings, even when it is not a perfect match for the environment. It is argued that this is because it allows for a better bias-variance trade-off and provides a structural basis to generalize outside of the empirical distribution.
strengths_and_weaknesses: This is a good paper. At first I was put off by the specificity of the representation---it really did seem to me that the factored action space representation used here was too specific to be of any practical use, but I felt that the authors did well to combat this and demonstrated applicability in healthcare at the very least. The presentation is clear and well done, and the development is logically consistent and flows well. While I found many of the theoretical bits "obvious" (e.g., props 2-4), I also felt that their presence adds to the overall depth of the paper as more or less a complete treatment of this "factored action space" representation of the Q-function. The fact that the abstraction used in Theorem 1 can be implicit is what makes this Theorem, and the paper as a whole, interesting, and sufficiently differentiated from, say, the literature on factored MDPs (but I'm not an expert in FMDPs). The empirical portions are well done---experiments are highly relevant, and the connection between the experiments and the method is plain---and the discussion is insightful.

I have seen, but not read in any detail, many works in multi-agent RL (and otherwise), which use factored action spaces (there are 69 results for "factored action spaces" on Scholar). I am trusting the author's lit review on the novelty point, and I hope that at least one of the other reviewers is sufficiently familiar with the literature to give a good opinion on novelty. To me, it seems sufficiently novel, but I lack broad knowledge of other work on factored action spaces.

I'm going to give this paper an "Accept" rating, since I don't see easy ways to improve it, and it feels complete. That being said, I do think the description for "Weak Accept" (which is pretty strong?) is more accurate, insofar as this paper is rather specific.
questions: A useful tactic in cases where we have an "approximately correct" structure, as you have in your experiments is to use a residual (even LSTMs / Resnets are examples of this, where identity is the approximately correct structure). So here, you could have $Q^\pi(s, \textbf{a}) = \sum q_d(s, a_d) + \mathcal{R}(s, a)$, where the residual function $\mathcal{R}$ can capture interactions between the components (and can be heavily regularized in cases of low data). It may be interesting for you to consider this approach in your applications. Not really a question, I know. 

I guess my most relevant question is: have you looked at the FMDP literature at all? I don't see it mentioned anywhere in the paper. I wonder if your Theorem can be related to some of their representation theorems by a simple transform of the state space (that entangles the previously disentangled state features). Could be interesting / worth looking into. 
limitations: I think this paper adequately addressed the limitations and potential negative societal impact of their work. The authors discuss the stringent conditions of their representations at length, and their experiments are performed in environments that *do not* satisfy their sufficient conditions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes the factored action spaces for offline RL in the applications of healthcare AI. The authors aim to leverage the factored action space in the form of linear Q-function decomposition to improve the sample efficiency compared with the baselines with combinational action spaces. The theoretical guarantees show that the linear Q-function decomposition can lead to zero bias under mild conditions. And even if the assumptions are violated, the method can still achieve good performances (nearly policy optimality). Empirical analysis verifies the proposed claims and effectiveness of the approach on simulated and real-world healthcare RL benchmarks. 
strengths_and_weaknesses: ## *Strengths*

#### **1. Technical soundness and significance** 

The paper proposes the factored action space, where the action space is expressed as a Cartesian product of a few sub-actions spaces. Though similar ideas and methods have been proposed in the past few years, this paper gives a detailed and comprehensive analysis in both theoretical and empirical manners. The theory parts are complete and technically sound, and evaluation with practical consideration also verifies the theoretical claims. With the analyzed theorems in this paper, factored approaches (either in state-action space or action space) in RL would be more broadly applicable. 

#### **2. Presentation** 

Though the presentation can be further improved for better readability (see the weaknesses points below), the overall logical flow is clear, and the concrete examples (Fig. 2-Fig. 3) are helpful for readers to understand the theorems. 

## *Weakness*

#### **1. The contributions on the algorithmic aspects are vague** 

As the authors mentioned in the related work section, several works have been proposed to explore the benefits of factored action space for both model-based and model-free RL. Yes, this work offers the solution for leveraging factored actions with value-based methods in 
offline RL. However, from the current version, I cannot tell the significant contributions of the algorithmic aspects of learning the factored space. The Cartesian product for action space has been used extensively in previous works (i.e., [1]). It would be better to explicitly list the contributions to the algorithmic aspects in the revised version. A table with comparisons of all related approaches would be helpful. 

#### **2. About evaluation** 

More complicated benchmarks (e.g., Mujoco control tasks, DOTA2, StarCraft, etc.) have been tested in other works using factored action spaces. Can this framework also be applicable for other commonly-used RL benchmarks? **Please note that running on these benchmarks during the rebuttal phase is not a must, but any discussion or analysis would be highly appreciated.**

#### **3. Possible directions to improve the writing for better readability** 

#### -> 3.1 Giving algorithmic frameworks into the main paper

The authors can consider adding the algorithmic frameworks as algorithm pseudo-code or figures. The framework can explain the pipelines of learning or exploiting the factored spaces. Moving some justification contents in Section 3.3.2 into the appendix can save room for this (since I feel like the propositions in 3.3.2 is already very clear). 

#### -> 3.2 Adding one background section on factored MDP and factored action space

The related work section in the appendix briefly introduces factored MDP and action space in RL. I think it is better to briefly give some formal definitions in the background or preliminary sections in the main paper. 


### **References**

[1] PIERROT, Thomas, et al. "Factored Action Spaces in Deep Reinforcement Learning." (2020).

questions: #### **1. About the autoregressive factorization**

Can authors give some analysis on both theoretical and empirical views on the scenarios where the actions are entirely in the autoregressive factorization? I notice that the theory part can be extended to this, and the empirical evaluation includes cases where a few actions are not independent. However, the entire autoregressive factorization, which is very common in RL with sub-actions, is not analyzed in the current version. In autoregressive factorization, the state would be augmented with the previously executed actions. Any discussion or analysis would be highly appreciated. 

#### **2. About the state space**

In the current version, each sub-action is based on the full state space. I think that why not leverage the local factorization or factored state space, where the state-action space is entirely factorized, and the state space of each sub-action would be much smaller. I think the sample efficiency would also be improved with the factored state-(sub)action space. **Please note that this is only a discussion. There is no need to run any experiment on this during the rebuttal phase.**

 ### **References**

[1] Pitis, Silviu, Elliot Creager, and Animesh Garg. "Counterfactual data augmentation using locally factored dynamics." Advances in Neural Information Processing Systems 33 (2020): 3976-3990.

[2] Huang, Biwei, et al. "Action-sufficient state representation learning for control with structural constraints." ICML 2022.

limitations: The limitations are more related to the algorithmic contribution and empirical evaluation (listed in the weakness section). I will increase my score if the authors give justifications during the rebuttal phase.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This study improved and extended the standard reinforcement learning (RL) to factored action space. A form of linear Q-function decomposition was proposed to handle factored action space. The novel method has been analyzed from a theoretical perspective, where the authors discussed the sufficient and necessary conditions for unbiases and studied its effect on variance reduction. The authors also demonstrated the proposed method through empirical experiments based on a simulation study and real-world MIMIC data analysis. The novel method provides both theoretical insights and empirical evidence for RL practitioners to consider this simple linear decomposition approach to factored action space. 
strengths_and_weaknesses: Strength: The research question is novel, and the authors found the unmet need of factored action spaces in the RL approach. Solving this problem will facilitate the use of offline RL in a broader application, especially in healthcare. The methodology is well-explained with details. This paper is complete with theoretical insights and empirical experiments. 

Weakness: The proposed method showed marginal improvement compared to the existing method. I also have some conservations on the applicability of the method (where many preconditions are required). The improvement has not addressed the key limitations of offline RL (i.e., how to improve exploration, discover high-reward regions and solve distributional shift problems). 

questions: 1.	Based on experimental evaluation (Figure 7), I was unable to observe significant improvement (in terms of clinical, real-world applicability) when compared with baseline BCQ and factored BCQ (90.44±2.44 vs 91.62±2.12).Therefore, it might be hard to conclude that the proposed factored BCQ outperforms the baseline.

2.	Based on Figure 6, the authors also claim better performance for a small sample size. I have concerns on this claim due to the lack of sufficient evidence. The variation is large for a small sample size, and the selection of samples would affect the result. Also, I couldn’t find the authors’ discussions on why the proposed method would get higher performance for small sample size, especially from the theoretical perspective. 

3.	The authors state that domain knowledge may be used to inform the applicability of the approach in practice. However, very little information in the main text address on how domain knowledge can be involved. This is an important question for real-world applications, thus would like to suggest the authors to provide more details for justifications. 

4.	If the authors claim that the method works well only for a limited sample size, I would suggest the authors do some analyses to determine the sample size cut-off (or give a quantitative/analytical formula to calculate). When the sample size is large enough, the empirical research shows that the proposed method is less useful and effective as compared to the baseline. Therefore, knowing the sample size cut-off is essential. 

5.	If possible, adding another real-world use case would be ideal to justify the generalizability and applicability of the proposed method. 

limitations: 1.	The performance improvement brought by the proposed method seems marginal. I suggest the authors add some other performance indicators and provide additional discussions.

2.	There is a lack of interpretability for the proposed method. The authors are suggested to elaborate on how to interpret the results at the individual level and how this can be applied to clinical practice.

3.	There are some preconditions required for using this proposed method. I would suggest the author provide some discussions and give more guidelines for future researchers who would like to use your method. 

4.	Please also add more on the importance of your method and how your method can address the unmet need of current literature.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes to decompose Q-function according to factorized action space. It conducts discussions on the bias-variance trade-off of the decomposed Q-function. Specifically, this paper provides sufficient conditions for zero bias and proves these conditions are not necessary. Furthermore, it shows that this form of Q-function decomposition leads to low variance because of the smaller lower bound on the empirical Rademacher complexity. Finally, it discusses what kinds of tasks can fulfill the sufficient conditions. The experiments on offline healthcare tasks demonstrate that the decomposed Q-function has the potential to help models improve their performance.
strengths_and_weaknesses: Strengths: The paper is well-structured and easy to follow. This paper studies the guarantee of the unbiasedness of Q-function decomposition, which may inspire the researchers to focus on safe decomposition and improve performance in healthcare tasks. Besides, the proofs of bias-variance trade-off are provided in detail. On the sufficiency and necessity of these conditions, it validates them either by theoretical analysis or examples. 
Weakness: 
1. According to Figure 1, Q-function decomposition can help reduce some calculation steps in the model. However, this paper does not demonstrate the optimization of computational efficiency, especially when it comes to experiments.
2. According to Section 3.4, it seems that the sufficient conditions are very hard to satisfy which may lead the model rather impractical. 
3. The lack of comparison to more algorithms makes the effectiveness of the decomposition somewhat less convincing.

questions: The title of the article points out that this is an efficient offline RL approach and the sample efficiency is verified in the paper. However, I am still wondering whether the running time and computational cost of factored BCQ are significantly reduced compared to the baseline BCQ?
The comparison of the models in the experimental evaluation part may be insufficient. Whether the validity of linear decomposition of the Q-function is applicable to other models of offline RL, such as CQL[1], PQI[2], REM[3]?
In Figure 8a, why does factored BCQ almost never recommend IV fluids 1L-2L, which is inconsistent with clinician behavior policy?
[1] Kumar, A., Zhou, A., Tucker, G., & Levine, S. (2020). Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33, 1179-1191.
[2] Liu, Y., Swaminathan, A., Agarwal, A., & Brunskill, E. (2020). Provably good batch off-policy reinforcement learning without great exploration. Advances in neural information processing systems, 33, 1264-1274.
[3] Agarwal, R., Schuurmans, D., & Norouzi, M. (2020, November). An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning (pp. 104-114). PMLR.

limitations: The authors have discussed the limitations of their proposed approach in the manuscript, which may cause risky results in healthcare.
review_title: null

Meta-review: metareview: Reviewers agree that the problem of factored action spaces in RL is important and that this paper makes novel contributions to this setting.  The reviewers were satisfied with the post-rebuttal discusion and have converged on an accept recommendation.

On revision, the reviewers request that the authors revise the paper according to the clarifications that occurred during post-rebuttal discussion.

Also, for context, it's important to note that the concept of factored action spaces goes back a long way in the factored MDP literature and I would request the authors to acknowledge this in their related work discussion as they prepare their final revision.  To the best of my knowledge, the first mention of factored action spaces is in a 1996 multiagent MDP paper:

Craig Boutilier.  Planning, Learning and Coordination in Multiagent Decision Processes. (1996)
https://www.cs.toronto.edu/~cebly/Papers/tark96.pdf

Somewhat more recently, the following paper presented a sequential hindsight method for compositional MDPs that is an upper bound approximation for (weakly) coupled MDPs.  I mention this specific paper since it discusses theoretical results relating to factored action MDP approximations and also presents a simple approximate decomposition methodology that I have found hard to beat empirically:

Aswin Raghavan, Saket Joshi, Alan Fern, Prasad Tadepallia, Roni Khardon.  Planning in Factored Action Spaces with Symbolic Dynamic Programming. (2012)
https://ojs.aaai.org/index.php/AAAI/article/view/8364

confidence: Certain
Decision: Accept