Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces the problem setting where the environment has a “piecewise-stable context” (i.e. the context is constant for some stochastic length of time then changes abruptly to another context several times in an episode), called LS-MDP. Previous works tackle either extremes where either (1) only a single constant context is sampled for the episode or (2) there is a Markovian context at each timestep. The paper then proposed a method, SeCBAD, which performs joint inference for the segment structure (i.e. how long each context occurs for) the belief context given segment structure using variational inference techniques, and combines them in a mixture distribution to marginalize over the possible segment structures. The policy then is learned with the belief context as additional information (instead of conditioning on the exact context). 

Experiments were conducted both in a didactic grid-world example, as well as in larger school Mujoco environments, i.e. inferring ideal movement direction for ant locomotion and half-cheetah with changing wind direction. The inferred segment structure is also visualized in the half-cheetah environment to indicate agreement with the underlying context changes.
strengths_and_weaknesses: **Originality**: The paper was effective in contextualizing its problem formulation (pun intended) in the sea of non-stationary RL works, while clarifying that LS-MDP is a special case of POMDP. Given these additional assumptions in the problem setting, then we can exploit them in the solution. The proposed approach applies well-known tools like variational inference methods for this problem setting, but there is merit in showing that these existing tools can provide significant gains over methods that have more general/different problem assumptions (such as VariBAD). 

**Quality**: The paper mainly provides empirical results to analyze the proposed method to demonstrate that it can incorporate the piecewise-stable context for adapting to abrupt environmental changes. However, the limitations of the method was not discussed in detail, nor was there a section on potential negative societal impacts the work.  

**Clarity**: Overall the paper is relatively clear to read, with helpful conceptual diagrams for the method. There are several minor comments for improving the draft:

1. In the abstract, second sentence seems incomplete. 
2. Figure 4: should label the y-axis as ‘time step’ or something along that lines to help quickly understand how to read the plots. Based on the last paragraph on page 7 I was also expecting a plot for the vanilla inference behaviour but I do not see that in there. 
3. For the G_t plots, perhaps it’s also informative to plot the oracle $G_t^*$ value (i.e. the perfect zig-zag lines) to help compare that with the estimated $G_t^*$ by the proposed algorithm

**Significance**: The paper proposes a relevant problem setting and demonstrated that their method can yield noticeable improvement. There are still a lot of architecture improvements to the method (as mentioned about using Transformer instead of GRU in the recurrent encoder), and more likely also new approaches for the given problem setting. 
questions: Please see my questions below:
1. The experiments would be stronger with additional baselines in addition to the VariBAD baseline. For example, one of the other works assumes intra-episode context changes and models $c_t$ at each timestep. This will help to strengthen the result that the piecewise-stable prior helps over Markovian context type approaches in your environment settings. 
2. It would be good to provide empirical data to back up the claim that using $G_t^*$ with the highest probability in $p(G_t|\tau_{1:t}))$ and then using the corresponding belief lead to little performance loss. 
3. What is the sensitivity of the training schedule between training the encoder/decoder/policies? In Algorithm 1, it seems that they are all updated at the same pace, but I am wondering if you have also explored for example training the encoder/decoder for more steps relative to the policy optimization step, etc. 
4. More hypothetical, not asking for experimental data: Could the context transition also depend on the states reached? E.g. reaching a new terrain, etc. How would your method behave in those types of environments?

limitations: As mentioned above, the paper did not describe much limitations and potential negative societal impact of their work. Perhaps it is also worth to discuss about the potential sample complexity required to collect enough data to learn the encoder/decoder, or what happens if the underlying MDP has markovian contexts / one context, then would the approach still be able to also solve those situations just as well as other baselines (i.e. if there is a mismatch in the MDP assumption and the method).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Authors investigate the non-stationary RL problems defined as scenario MDP where there exists an unobservable context variable that could change abruptly and unpredictably but remain stable within a stochastic period.  The general idea of this paper is to develop a joint inference framework for the segment structure and the belief context. A variational inference method similar to variational auto-encoder (VAE) is proposed to infer the context given a segment of the past trajectory. An iterative inference method is also proposed for the segment length.  The infered context is then augmented to the observed state variable as inputs to RL algorithms. Experiments on grid world and Mujuco are conducted to verify the performance of the proposed method. 
strengths_and_weaknesses: Strength:

1. RL for non-stationary environment is an important research topic and advanced methods on this topic could potentially improve the applicability of RL algorithms to real world problems. 

2. The scenario MDP formulation proposed in this paper is interesting and could be a more realistic formulation than existing Markovian ones. The joint inference method of the segment structure and the belief context is also non-trivial and thus novel, to the best of the reviewer's knowledge. 

3. In general the paper is well motivated and clearly written and techincally sound.

Weakness:

1. Authors do not compare the proposed method to an important brunch of research on Partial Observable MDP (POMDP) in the main text and also in the experiment section. 

2. The RL part is fairly weak as the proposed method simply augment the learned context to the state and then feed them to any existing RL algorithms.  To me, the whole framework seems a two-stage approach where the connection between the two stages are quite weak.

3. The experiment section is weak. As stated as the motivation of the paper, non-stationarity could be the major issue for RL to be applied to real tasks, and I expect authors to demonstrate the performance of their algorithms on real tasks. However, experiments on real tasks are not presented.
questions: 1. What is the complexity of the proposed algorithms? According to Eq. 5, the posterior of the belief context has to sum over all possible segmentation and if it is the case, then the computational complexity seems to large for larget t.

2. If we look at the motivating example in Fig.1, it seems that the change of the context can be easily detected using the state variables, e.g. we can apply well developed "change point" detection algorithms to the observables to detect the change of the context. Do we really need such a complex method? I would like authors to further comment on this question.

3. In this paper, is the context variable a scaler? If we extend it to a high dimensional latent variable, whether the proposed method is able to solve the problem is not clear. 


limitations: The major concern of this paper is as stated "Authors do not compare the proposed method to an important brunch of research on Partial Observable MDP (POMDP) in the main text and also in the experiment section. "
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper tacles RL on non-stationary MDP. Based on the core assumtion that the environment context is piecewise stable, the authors prospose a Segmented Context Belief Augmented Deep (SeCBAD) RL method, which can jointly infer the belief context and segment structure from observed data. Experiments on a gridworld environment and Mujuco tasks show that the proposed method can quickly detect and adapt to abrupt context changes and outperform existing methods.
strengths_and_weaknesses: Strengths:

- The paper tackles a relavant problem, non-stationary MDP, and picks a simplified setting based on piecewise stable assumption, which has merits in some real-world scenarios (robotic and congestion control).
- The presentation of the paper is generally clear, despite some typos (e.g. line 5).


Weaknesses:

- **Limited scope**. The contributions of the proposed method depend entirely on the assumption that the non-stationary MDPs are piecewise stable, which inevitably limits the scope of real-world applications, e.g., greenhouse control given stochastic outdoor climate and quantitative trading in unpredictable stock market.  

- **Limited novelty**. The proposed algorithm employs the exact same formulation of approximate inference and training objectives of variBAD and makes incremental modifications to the framework to incorporate iterative inference for the segment length. The contributions to both the theoretical and algorithmic aspects of BAMDP are limited.

- **Lack of experiments**. The proposed method is only evaluated on 3 relatively simple environments (1 tabular + 2 MuJoCo) and compared to 1 baseline, variBAD, which is designed for stationary MDP. More baselines targeting non-strationary MDP [1][2] should be included to validate the effectivenss of the proposed method. Also more focused studies such as the robustness of SeCBAD to noise within segmented context could bring more insight to the readers.

Minor issues

- The paper seems a bit rushed. More details of the setup and experiments are needed, for example:
  - How many random seed/trials were performed to produce Figure 5 and 7?
  - What is $G$ defined in line 89?

[1]: Ren, H., Sootla, A., Jafferjee, T., Shen, J., Wang, J., and Bou-Ammar, H. (2022).  Reinforcement learning in presence of discrete markovian context evolution.arXiv preprint arXiv:2202.06557

[2]: Feng, F., Huang, B., Zhang, K., and Magliacane, S. (2022). Factored adaptation for non-stationary reinforcement learning.arXiv preprint arXiv:2203.16582.



questions: In principle, if the segment of the piecewise stable MDP lasts long enough, variBAD and its variants should be able to fully adapt to each segmented context. For instance, in Figure 4 of the original paper [1], variBAD adapts to a stationary MDP within 1 episode. However, both Figure 5 and Figure 7 show that they learn almost nothing, which is strange.

[1]: Zintgraf, L., Shiarlis, K., Igl, M., Schulze, S., Gal, Y., Hofmann, K., and Whiteson, S. (2020). Varibad: A very good method for bayes-adaptive deep rl via meta-learning. InInternational Conference on Learning Representation (ICLR).

limitations: N/A
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper provides a new method called SeCBAD (Segmented Contect Belief Augmented Deep RL), to deal with the segment structure of piecewise-stable context. Experiment demonstrates that this method can automatically detect the segment structure, which reflects when the context changes abruptly.
strengths_and_weaknesses: Strengths:
The discussed latent situational MDP with piecewise-stable context seems to be potentially useful to model certain stochastic environments. The approach proposed is neat.

Weaknesses:
1. Writing incomplete sentences, e.g. line 5: or assume Markovian context variables for?
2. Lack of theoretical analysis to compare to distribution shift in RL which seems to be a very similar topic.
questions: 1. How does this problem compare/relate to distribution shift? Should cite/discuss some distribution shift works.
2. Although it's termed context in this work, the formulation of the method looks quite similar to goal-conditioned RL. Can the authors provide some explanations on how the two are different?
3. I'm a bit confused by the Half-Cheetah Velocity case study. From my understanding, velocity should be the target/reward in this environment. However, it seems to be treated as the context in this paper?
4. Just curious, if this method can detect and adapt to abrupt changes, then can it also adapt to slow and insidious context changes?
limitations: 1. Seems to be a straightforward extension of variational inference framework.
2. Experiment is done in specifically modified environments. Therefore, its objectiveness and fairness remain to be confirmed. 
3. Clarity of experiment set up can be improved.
4. The paper claims that 'Piecewise-stable context is quite common in a wide range of real-world applications'. However, the authors did not give a concrete example of this.
review_title: null

Meta-review: metareview: This paper proposes an algorithm to deal with non-stationarity in RL, in settings where a latent context variable changes abruptly at discrete points in time (in contrast to previous work that focuses either on settings where the context is constant over an episode, or may change at each timestep)

This is very much a borderline paper, with reviews in the "weak reject" to "weak accept" range. Summarizing the main concerns raised by reviewers:
1. Some writing issues
2. Lack of comparison to other baselines
3. Lack of experiments on closer to real-world tasks
4. Limited novelty
5. Limited applicability (piecewise stable context)

The authors made significant improvements during the discussion period, in particular by:
1. Fixing the main writing issues
2. Adding comparisons to more baselines, in particular a POMDP algorithm and an algorithm for non-stationary RL
3. Adding a bandwidth control task that is closer to a real world setting (+ also additional MuJoCo tasks for a more comprehensive evaluation)

In my opinion, concerns #1-2-3 have thus been addressed in a satisfactory manner.

Regarding #4 (novelty), it is indeed somewhat limited since this work is very close to the prior VariBAD algorithm, and can be seen as an incremental improvement to make it handle intra-episode non-stationarity. That being said, I consider that tackling this "piecewise stable context" setting is a novel and valuable contribution, since it is definitely an under-explored setting in the current literature, and can have some interesting real world applications.

Concern #5 remains since this work is indeed specific to the piecewise stable context setting, and thus may (1) not be useful when there is no intra-episode non-stationarity, and (2) not work well in cases where the context changes frequently / continuously. However, I would not consider it a fatal flaw since I do believe this piecewise stable setting to be interesting and relevant.

I also note that the only reviewer recommending rejection (3SWX) did not reply to the authors' response (and neither did they update their review) to explain why they were still leaning towards rejection in spite of the above-mentioned improvements.

In conclusion, I believe the updated submission is now meeting the bar for publication, as it shows strong empirical results in a novel setting that is relevant to (some) real world applications.
confidence: Less certain
Decision: Accept