Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a factored adaptation framework for reinforcement learning in non-stationary environments. The paper first formalizes the notion of Factored Non-stationary MDP (FN-MDP), which augments a factored MDP with time-evolving latent change factors under the Markovian assumption. The generative model in AdaRL is adapted to the time-varying setting by introducing additional equations for the latent change factors update. The causal structure is captured by a DBN, where the edges are represented using binary masks. These masks can be inferred under certain identifiability assumptions.

The paper then proposes FN-VAE to model the dynamics of the latent change factors, the state transitions, and the reward function. It includes some sparsity and smoothness regularization terms, and can be adapted for continuous or discrete changes. The FANS-RL framework combines model learning with policy optimization. The policy depends on the states and change factors for the dynamics, and the graph structure is used to select only the dimensions which affect the reward.

Experiments are performed on four popular benchmarks, which have been modified to be non-stationary with both continuous and discrete changes. The results show that FANS-RL with FN-VAE performs better than some existing methods. The paper also includes ablation studies and experiments testing various aspects of the proposed method.
strengths_and_weaknesses: Learning effective and stable policies in non-stationary environments is an active area of research within the community. This is an important area of research and contributions in this direction can facilitate the application of deep RL to real-world scenarios. This work formalizes a non-stationary analogue of factored MDPs and proposes algorithms to model the environment and combine it with policy optimization to provide a general-purpose framework. The presented approach seems reasonable and theoretically sound. To the best of my knowledge, this represents novel and original work.

The writing is clear and concise. The factored generative model is quite complex with many moving parts, but the explanation and the diagrams make things fairly straightforward to follow. It is appreciable that both continuous and discrete changes are considered, and the framework is also adapted to image observations. The experiments are suitable and include comparison with recent works. The extensive results both in the main paper and the appendices are impressive. Notably, the authors perform significance tests for comparison with previous work.

The paper presents good quality and original work, with clear explanation of concepts and extensive results. The quality of submission can be enhanced by addressing some clarification questions and suggestions, provided below.
questions: 1. The inference networks infer $\theta_t$ from $\tau_{0:t}$ as described in the paper but in Eq. (4) $q_\phi$ also includes  dependence on $\theta_{t-1}$. This dependence is not clear, does this represent the time-evolving hidden state of the LSTM, $h_{t-1}$?

1. The one-step prediction encoders in Eq. (6) and (7) should have dependence on $\theta^s_{t+1}$ and $\theta^r_{t+1}$, respectively, as specified in the FN-MDP. Though Fig 2(d) shows that state and reward prediction help improve performance, predicting  $(s_{t+1},r_{t+1})$ from $(\theta^s_{t}, \theta^r_{t})$ seems to ignore the latent change factor dynamics. A clarification from the authors would be helpful.

1. The multiple loss functions terms each have an associated weight, which seems like a hyper-parameter tuning nightmare. This can hurt the applicability of the proposed model in practice. Could the author comment on this aspect? On a related note, Section D.4.1 specifies the values used for these weights but is missing a description of the hyper-parameter selection method.

1. The experiments verifying the values of learned reward CFs are insightful. It would be interesting to verify the dynamics CF in a similar experiment.

**Minor comments:**

- Eq. (2) seems to have a typing error, $r_t$ should be function of $s_t,a_t$ instead of $s_{t-1},a_{t-1}$.
- Line 87 typing error, $c$ should not have subscript $i$.
- Line 93-94 is confusing and seems to be making contradictory claims.
- Algorithm 1, initial values of $\theta_{old}$ not specified.
- Algorithm 1, line 3, what is $k$? What does $\tau^i_{0:k}$ denote?
- The smoothing technique applied is slightly unusual. Based on my reading, most works employ moving average-based smoothing.
- Figure 2(f) has incorrectly labelled axes.
- Appendix lines 190-191 state that CF inference networks are fully-connected, shouldn’t they be LSTMs?
- There is a lot of notation used in the paper, which naturally introduces scope of typing errors. I assume such issues will be fixed.
limitations: The paper provides a brief discussion on some limitations, namely the scalability and applicability of the approach to complex problems. The appendix provides a more detailed discussion on the effect of different assumptions or inductive biases, and acknowledges the lack of theoretical guarantees for the proposed method.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces FANS-RL, a factored adaptation approach that aims to generalize to non-stationary scenarios including changes across episodes and within episodes. They formalize FN-MDPs, and prove the causal graph of the transition and reward function is identifiable. The experiments show FANS-RL outperforms the state of the art.
strengths_and_weaknesses: **Originality:** Fair: The paper contributes some new ideas

**Quality:**		Good: The paper appears to be technically sound, but I have not carefully checked the details.

**Clarity:**		  Good: The paper is well organized but the analysis of the experiments' results could be improved.

**Significance:** Fair:  The paper is likely to have a moderate impact within a subfield of AI.

**Main Strengths:**

- It is a nice synthesis of causality and RL, leading to an intuitive design.
- It's fascinating to generalize a Factored-MDP to a FN-MDP, solving the non-stationary problem from a different perspective.
- If the results are robust then it could be an important and useful tool.

**Main Weakness:**

The paper is weak in a few ways. Mainly, the experiments section could include additional insight on the results as I will discuss in my question comments to the authors. As a minor point, the paper omits certain details such as a more thorough intro of AdaRL due to page limitations, which is understandable but would require more background knowledge from the readers. I suggest improving the readability in the revised version. 

questions: I list a few ways in which the paper can be improved below.

- The experimental chapter devotes a lot of space to the various environmental settings, and the experimental results seem to be insufficient. More detailed discussions on the experimental results in the main paper would be more appealing. 

-  The experiment section has compared different baselines with ablation experiments. However, a large number of results were presented in the Appendix section. From the main text, it's not very clear why the authors' approach is more effective than baselines, and ablation's analysis doesn't show what each component of FANS-RL does. It would be useful to extend the discussion in this section on why FANS-RL outperforms baseline methods.

- Figure 2 (c) shows that FANS-RL's reward is comparable to Oracle, even higher than Oracle. It is useful to explain why this is happening.
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors considered non-stationarity which is faced when RL is deployed into the real world. To model the non-stationarity across episodes or within an episode, which seems to be general scenarios, the authors disentangled the non-stationarity as two types of latent change factors and formalized FN-MDPs. These disentangled latent factors enable the proposed method called FANS-RL to estimate changes both in the environment dynamics and the reward function. The authors assumed the generative process of FN-MDPs containing these latent factors, and FANS-RL learns the generative process of FN-MDPs during training. FANS-RL can also infer the change factors in environment dynamics and reward function at the current time-step via the learned generative process of FN-MDPs. The experimental results showed that FANS-RL outperforms baseline algorithms in various simulation tasks. Especially, the ablation study showed the performance effect of their method, which has many loss functions and variables they considered.


strengths_and_weaknesses: This paper proposed FANS-RL that addresses non-stationarity both in environment dynamics and reward function. 

1. Strengths

(1-1) The authors formalized FN-MDPs that can deal with a general non-stationary RL setting including changes both across episodes and within an episode. It is a very important problem in RL.
(1-2) FN-MDPs also contain two latent change factors for the environment dynamics and the reward function, and FANS-RL can more explicitly handle the non-stationarity of the MDP due to the two change factors. 
(1-3) FANS-RL shows better adaptation for non-stationarity than other existing algorithms in various tasks. Also, the ablation study showed the effect of each component of FANS-RL architecture with respect to the expected return.

2. Weakness

(2-1) The proposed method showed the adaptation for the limited changes in the environment dynamics; for example, the changes of wind force or agent’s mass or gravity. So, the reviewer wonders if FANS-RL can adapt to changes in the agent’s mechanism; for example, the disabled joint of the agent due to aging.

(2-2) It seems difficult to find the fundamental difference between the architecture of FN-VAE and that of MiSS-VAE of AdaRL[1], even though they learn different MDPs. They have too similar components and loss functions.

(2-3) The paper relies much on the definitions and theorems of AdaRL [1] in literature. It would be better to add an additional explanation of the shared representation $s^{min}$ and the compact domain-specific representation $\theta^{min}$.  
Especially, in Line 74-84 in Appendix B.2, there exist sentences verbatim to those in  the proof of theorem1, page 18, [1].  I marked ethics alert  due to this point. Please check if this is OK.
The paragraph of concern: "We denote the variable set in the system ...."

3. minor points (typo)
(3-1) The first equation between line 72-73: $pa(\theta_{ij,t}^s)\to pa(\theta_{j,t}^s)$
(3-2) Figure 2-(f) in the main paper: the x and y labels need to correct.

[1] Biwei Huang et al., AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning, ICLR 2022


questions: FANS-RL can address non-stationarity, which is an important challenge in RL, and I think it is a great approach. However, it seems to rely on AdaRL, so I need additional explanation and experimental results to support the acceptance of the paper.

1. Why did the authors formalize FN-MDPs, not N-MDPs? Although the ablation study without structure shows improvement with respect to the expected return, I need a brief and intuitive explanation for understanding the effect of factored MDP.

2. As mentioned in “Weakness”,  the experimental results showed the adaptation to the limited changes in the environment dynamics. To find out the adaptation for the changes in the agent’s mechanism, I suggest additional experiments for these changes; for example, the increasing joint friction is introduced at a joint of Half-Cheetah.

3. To check the effect of disentangling the latent change factor as the two ones, I suggest adding an ablation study without disentangling latent factors in figure 2-(d).

4. Figure 2-(e,f) shows the distances of the learned $\theta^r$, similarly, I suggest plotting the distances of the learned $\theta^s$; for example, the $\theta^s$ estimates the changing wind force. It is helpful to identify that the learned latent factor can capture the true change in environment dynamics.

5. In Figure 2-(g), is the number of latent features of FANS-RL total number of dimensions of two latent features $\theta^s$ and $\theta^r$?

6. What is the fundamental difference between the FN-VAE and the MiSS-VAE of AdaRL?


limitations: I already mentioned the limitations of this paper and the suggestions for the proposed method.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel formalism to model non-stationary environments, Factored Non-stationary Markov Decision Process (FN-MDP), that models latent factors that affect the dynamics and rewards and evolves with time as in Dynamic Bayesian Network. Then, they propose FN-VAE, which is built on top of AdaRL, and is used to learn the parameters (latent factors, dynamics, rewards, masks) of the underlying FN-MDP. The method is evaluated in several robotic environments and compared with state-of-the-art algorithms tailored to deal with non-stationarity in RL.
strengths_and_weaknesses: Strengths:
- The proposed formalism is able to model, in a factored manner, non-stationary environments in which the factors that change the dynamics/reward also evolve over time. This is a novel idea not yet explored in the related literature.
- The authors compare the proposed approach with several state-of-the-art algorithms tailored to deal with non-stationarity, showing relevant performance improvements. Several ablation experiments were also presented.

Weaknesses:
- Some mathematical and algorithmic definitions are not very clear in the current version and should be clarified in the main text.

questions: Furthermore, I have the following questions and constructive criticisms:

- In Definition 1, I suggest clarifying that it is assumed that the action is a vector of m dimensions. It is ambiguous as m could denote the number of discrete actions.

- “Although the masks and noise are stationary, we allow the change of graph structure and noise distributions, whose changes are captured by $\theta$ instead.” 
If the masks are stationary, does it mean that if one of its values is 0 (denoting that one state variable does not influence some other state variable), then it will never have an effect, even if the dynamics change? In other words, to be able to capture all possible dynamics changes, would we need to have all masks with none zero elements?

- The proof of Proposition 1 in the Appendix ends with: “In this setting, identifiability of the graph G is trivial [6].” However, It is not clear why this is trivial. As this is an important theoretical result of the paper, I suggest elaborating this proof.

- The role of the state and reward encoders in Section 3 is not very clear. Why a single prediction network, conditioned on the masks, would be not enough to predict the next state? How are the encoders used?

- How can we obtain the adjustable parameters $w_1$ … $w_7$? In the total loss, how can we obtain $k_1$ … $k_5$?

- The total objective function in line 167 does not include $\mathcal{L}_{\text{rec-dyn}}$. Additionally, there is a typo here: the coefficient $k_5$ does not appear in the loss.

- “the policy parameters = $\psi = (\pi, Q)$ are the actor $L_\pi$ and critic loss $L_Q$.” I believe this sentence is not well formulated, how can a loss be a parameter?

- In Algorithm 1, how is $s^{min}_t$ and $\theta^{min}_t$ computed before following the policy in line 23?

- “The horizon in each episode is 50.” The horizon in these environments is generally higher, why this value was chosen?

- In Fig. 2(g-j) how were the values normalized? I suppose the value of 1.0 does not mean that the proposed method reaches the maximum possible reward, but that it is the maximum reference point used in the normalization.

- Because of the causality assumptions (see Section A of the Appendix), the proposed model can not model different types of non-stationarity at the same time (e.g. wind and gravity). Are any of the related work (e.g. LILAC) able to deal with these changes? If so, perhaps an experiment in such a setting would be interesting to observe whether the proposed method could still outperform the other baselines.

- For completeness of the first sentence of the Related Work section, the paper [“Minimum-Delay Adaptation in Non-Stationary Reinforcement Learning via Online High-Confidence Change-Point Detection.” In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems] is a more recent work on non-stationarity that detects changes that have already happened.

limitations: The authors appropriately discuss the limitations of the proposed approach. I would suggest, however, that the authors briefly include in the main text some of the assumptions of the method which are only discussed in Appendix A, as I believe they are very relevant and readers often do not check the Appendix.
review_title: null

Meta-review: metareview: The paper proposes a factored reinforcement-learning method to deal with non-stationary environments.
After reading the authors' rebuttals, the reviewers agree that this paper provides an original and sound contribution that deserves publication.
We recommend that the authors modify their paper as reported in their answers to the reviewers' comments.
confidence: Certain
Decision: Accept