Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper looks at how to build deep RL agents that can perform more efficient learning by directly leveraging the factored structure of problems when that information is given. The approach proposed by the authors is to design a neural network architecture that explicitly honors the factored structure of the problem. The authors detail a version of MCTS that can leverage independent components of the state, a version of DQN that can leverage independent components of the reward, and a version of PPO that generates actions based on independent state components. I very much like the idea of making connections with the factored MDP literature. However, as the authors point out in the conclusion, an approach that actually discovers the causal factors of the problem would be much more interesting and generally applicable. In terms of leveraging the provided structure of the problem is concerned, the experiments seem convincing that the proposed approach by the authors has value. That said, it was not clear to me how common this setting is where the agent can directly leverage known structure about independent aspects of the state and reward like this. The experiments felt a bit contrived to me, which I think at least partially reflects this inherent difficulty of  finding an environment that naturally fits the setting the authors explore. Additionally, the environments considered of very low complexity. I do not usually like to focus on this, but see it as a potential issue in this case because it is unclear to me if the architecture implications of the factored neural network considered here would scale well for example to vision problems with complex feature spaces and CNN architectures. 

I am very much on the fence about this paper as I like the overall direction as a stepping stone towards a model that discovers factored state representation as well. However, I think the paper could really benefit from more evidence either theoretically or empirically that the factored neural network structure is always likely to lead to improvements when this structure is present. For example, could you say anything theoretically about the way that this leads to sample efficiency improvements during learning? Additionally, the authors should explain and potentially validate how their factored neural networks would extend to richer input streams like visual observation based environments. 

After The Rebuttal: I really appreciate the rebuttal and revisions submitted by the authors. They did a good job of detailing domains of interest where their approach may be applicable. I also agree with some of the points they made about the complexity of domains that they considered in their experiments. As such, I have revised my score and now lean towards acceptance of the paper. 
review_title: Nice Direction And Encouraging Results, But Not Quite Convinced About The General Appicability Of The Approach

final_score: rating: 6: Marginally above acceptance threshold
review_content: D4RL: Datasets for Deep Data-Driven Reinforcement Learning
review:

summarization:

In this paper, the authors consider using factored neural network (NN),
instead of directly using forward NN in model-based or model-free reinforcement learning.

Pros:
1. In general the project is nice and neat.
A simple algorithm is proposed,
and the performance is improved upon existing baselines.

2. Extensions on PPO, MCTS, DQN is studied and evaluated,
making the paper quite comprehensive and the conclusion proposed more robust and convincing.

3. The paper is well written and in general easy to understand.
The figures in this paper are very helpful as well.

4. The algorithm is also applicable to high dimensional problems such as humanoid.

Cons:
1. The algorithms were not combined with existing state-of-the-art algorithms,
which includes algorithms such as SAC, TD3, MBPO.
It would be interesting to see if the proposed method can lead to state-of-the-art performance.
 
Misc:
1. how efficient is it to train and evaluate (is it real-time?)
2. Does it mean that the structure has to be manually specified for each environment?
3. The paper reminds me a lot of NerveNet [1],
which also considers factored or graph NN in reinforcement learning.

Summary:

In general I believe the paper is nice and neat. I tend to vote for acceptance in this case.

[1] Wang, T., Liao, R., Ba, J., & Fidler, S. (2018, February). Nervenet: Learning structured policy with graph neural networks. In International Conference on Learning Representations.
review_title: review AnonReviewer2

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The authors present a method by which the underlying MDP describing RL tasks can be factored into independent components in combination with Deep RL solutions to improve the efficiency and tractability of the problem.  The graph factorization of state, action and reward components is realized via an adjacency matrix that adds masking across input and output components.  

They evaluate factored RL approaches for both model based and model free RL algorithms:  MCTS, DQN, PPO.  In the case of model based approach (MCTS) the state transition function is factorized.  For model-free approachs, DQN & PPO, decomposed Q-functions and policy functions are used.  These approaches are evaluated across a number of domains: Multi-Cartpole, Taxi, BitFlip and robotics tasks Ant, HalfCheetah and Humanoid from the RL Control Suite.

Overall, the paper is clear and well written.

Strengths & Weaknesses:

The authors detail the scope of the problem domain and embed their approach well in the existing literature. The scope of the problem that the authors are addressing is broad and relevant as it applies to a whole host of problems where the underlying MDP is well understood.  The novelty of the approach lies in the extension to Deep RL algorithms from prior approaches which relied on tabular or linear methods.  The authors also consider state and input/output masking in addition to action masking, an approach taken by earlier methods.  The author also seek to address and take advantage of causal inference in the fully observable RL environment setting.

The paper is divided nicely among the MCTS, DQN and PPO sections and the authors clearly describe the factored inputs and outputs in each case with the Multiple Cartpole used as an example.  The authors have also chosen a diverse set of tasks over which to evaluate and have demonstrated that there are strong gains to be realized in both sample efficiency and top-line performance when using the factored approach.

For me the greatest weakness of the paper is that there was not a great deal of insight given with respect to how scalable this approach might be in terms of other RL domains such as for instance in navigation or planning and the impact that a factored approach might have on representation learning where experience across diverse parts of the environment may be important to learn how to encode information over a diverse set of experiences.  Furthermore, the factorization algorithm needs to be  defined by hand for each task which may incur additional complexity and pose problems when the aim is to optimize for generalization in RL agents.  However the authors do state this explicitly and note that a future direction for this work could be in online discovery of factored graphs.


Other Points:

Figure 4a only contains one curve (Presumably this should be FactoredDQN rather than DQN?)


Recommendation:

I recommend a score of 6 and would be happy to see this paper accepted.   I believe the approach sets out a fruitful direction for Deep RL which could be particulary powerful if this factorization could be learned.  In these early steps I believe the authors have laid out the problem well and provided strong results to support their claim.
review_title: Review of "FactoredRL: Leveraging Factored Graphs for Deep Reinforcement Learning"

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a methodology for incorporating factor-graphs into model-based and model-free RL methods. The work starts by assuming access to a correct and factor graph showing the relationship between individual state factors, actions, and rewards. The authors propose to make use of this factor graph by using a Factored Neural Network - which is similar to the standard feed-forward MLP networks that would typically be used to parameterize a policy or Q-function - except that it masks out connections between input and output nodes that are not connected in the factor graph. Presumably this results in a sparser neural network which can lead to faster learning and better sample complexity. The authors demonstrate how these factored NNs can be incorporated with model-based MCTS as well as model-free DQN and PPO. In short - the algorithm remains unchanged and the only substition seems to be the Factored NN rather than a fully-connected NN. Experiments are performed on Multi-Cartpole (simultaneous control over several cartpoles), Taxi, BitFlip, and PyBullet's Ant, Half-Cheetah, and Humanoid. Each of the factored algorithms is compared with the un-factored equivalent and increased sample efficiency of learning is noted for the factored variants. The authors provide the manually-defined factor-graphs used for each of these environments in the Appendix.

On the positive side - I think the approach to creating factored-NNs to reflect the structure in the factor graph is sensible. Additionally, the results support the authors' claim that using these factored-NN can lead to greater sample efficiency. It's also nice to see that this approach - because it only modifies the underlying NN - is straightforward to integrate with many of the existing model-based and model-free algorithms.

In my opinion, the main drawback of this work is that many of the domains examined are either poor representatives of RL problems would ever be faced by an RL agent or simple enough to not require neural network function approximators. For example Multi-Cartpole is a highly-artificial environment which shows that current RL algorithms aren't equipped to simulatenously solve multiple MDPs. While this is a great motivating example - it's used extensively throughout the results. Similarly, Taxi and Bitflip are also toy domains that might not even require a neural network function approximators to solve (e.g. maybe the proper point of comparison for these domains is tabular methods built for factored MDPs?). The PyBullet environments are exceptions to this critique - but in these environments the performance improvement is not particularly notable.

More broadly, algorithms like DQN and PPO are commonly applied in environments featuring high-dimensional observation spaces such as pixel images, which don't admit easy factorization. Given that this paper is motivated by extending factorization techniques to Deep RL algorithms - the real question is "What environments admit easy factorization and require deep neural networks to solve?" It seems like the intersection of these criteria may leave vanishingly few opportunities for the community to apply the techniques in this paper.

In summary - the approach is reasonable, the results are impressive on toy domains but less impressive on realisitic domains. As pointed out in the future work, the real million dollar questions are 1) How to extract factor graphs from high dimensional pixel images and 2) How to infer latent structure of factor graphs from experience.

Minor Note - In Section 3 - PA(.) is not defined anywhere. Does it refer to the parents of a node?

Other related work - Working Memory Graphs; Loynd et al. has shown that using factored representations of state with transformer-based RL agents leads to improved sample complexity.

I have read the authors response and updated paper and appreciate the discussion of applications that admit factor graphs.
review_title: Commonsense Approach to Factored MDPs

Meta-review: The paper introduces variants of RL algorithms that can consume factored state representations. Under the assumption that actions only affect a few factors, these factored RL algorithms can learn more efficiently than their vanilla counterparts. Learning a factored dynamics model (to be used in a model-based algorithm) or representing factorized action-selection policies (to be optimized by a model-free RL algorithm) make intuitive sense in the problem settings that the paper considers. However, the paper should clarify the implicit assumptions being made about how the reward decomposes across factors. For instance, the factored DQN approach seems to require a linear reward decomposition across the factors.
The factored DQN approach is also reminiscent of the Hydra algorithm on MsPacMan (https://papers.nips.cc/paper/2017/file/1264a061d82a2edae1574b07249800d6-Paper.pdf Section 4.2) which assigns an RL agent to each factor ("ghost" in MsPacMan) to learn a factor-specific Q-function. The linear aggregator that they use is identical to the factored DQN in this paper.

The reviewers all rate the paper as borderline. All reviewers suggest that being able to learn the factor graph (or at least parts of it) will greatly widen the scope of applications where the approach can be fruitfully applied -- the paper acknowledges this as a compelling line of future work. The biggest weakness is originality -- the core message of the paper is just that, where factored representations of state/actions exist RL algorithms must use it. This is not a surprising or novel message. The paper advocates for incorporating the factorization information in the most straightforward way (state-masking, followed by action concatenation). Simple-in-retrospect is usually an excellent feature of an algorithm, not a bug; however, the proposal is literally the first idea a reader will likely think of. It might help to explore other ways of incorporating factorization information (e.g., rather than parameter sharing, have a separate network for each factor; rather than masking, have different width input layers to consume different number of parents in the DAG; etc.) and verifying that they are inferior to factored NN.
Decision: Reject