Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach for combining rule based policies and RL methods, such as TD3+BC and TD3. The proposed approach performs well (though I have no clear calibration of the tasks being benchmarked upon). The resulting policy attains good performance and can improve upon heuristics obtained through the rule based policy.
strength_and_weaknesses: Strength: While the idea of combining rule based learning with RL is not new, the paper presents an example with offline RL with a convincing downstream application (building control).

Weaknesses:

- Can the rule based policy also be learned, like parameters of a rule, to make it robust? More generally, how robust is this approach if the building changes? Can we see an ablation study where we compare performance of the method to standard offline RL when the rule based policy becomes worse and worse?

- Since TD3+BC is worse than CQL, as evident from the table, can this method also be applied to CQL to see how well it performs? 

- Would be good to do some qualitative analysis: when does the RL policy perform particularly much better than the rule-based policy? What happens if the data composition is different (more representative of real-world systems than random, medium, expert from D4RL)?

I like this paper overall, and would encourage authors to answer the questions above, after which I am happy to increase my score. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. Certainly the novelty in the idea is low, but I encourage authors to reformulate the wording of the paper to be more of an applications paper -- Applying offline RL for building control, and provide benchmarks for RL researchers to iterate on. I think that could be a big win of this paper if done right.
summary_of_the_review: Overall, I like the paper, but I am giving a reject score right now because I think the paper would be much stronger if my weaknesses are addressed. if the authors can address those weaknesses, I will move up to an accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a method to unify rule-based control with reinforcement learning for building management and control. The work provides both the online and offline setting variants. The idea is to extend the previous work of TD3 + BC by replacing the behavior policy with a dynamically weighted policy that chooses action either from the behavior policy or the rule-based policy. In TD3+BC, the policy update rule is updated to both maximize the value w.r.t. the policy, and also minimize the difference of the policy with a behavior cloning policy. In order to incorporate rule-based control, the method proposes to choose whichever policy with the maximum value from the collection of behavior cloning policy and rule-based policy. Experiments are performed in the offline setting and online setting. The offline setting demonstrates that the method has better data efficiency than other baselines, and is able to transfer from one weather condition to another with comparable performance. The online experiment also demonstrates that the method is better than TD3. 
strength_and_weaknesses: Strength: the paper is well written and contributions are appropriately stated, and the algorithm itself is easy to understand. The performed experiments show that the proposed algorithm is useful in some cases (medium and random buffer, etc.)

Weakness: (1) the figures are not designed carefully. For example, figure 1, it is not clear what each arrow mean, and it seems "solid lines" not always means "fixed actions". figure 2, the difference between different methods aren't that huge to distinguish one from another. Figure 3, it is not clear why this "RUBICON selects actions in a wider range" and "TD3+BC has a reward distribution of higher values", since it's hard to visually validate this, and better using quantitative approach if the author really want to prove this. 

(2) some results are mixed. sometimes the proposed algorithm performs better and sometimes not, the provided explanations are not very convincing. For example, why in hot-stochastic case, RUBICON performs better in table 1 with. expert data, while other cases, with expert data, RUBICON performs worse. Some results have 0.0 standard deviation, is that correct or error in data recording? is there a reason why it is exactly zero?

clarity,_quality,_novelty_and_reproducibility: Clarity: method is clear to understand, but the results and the presentation of results could be improved

quality and novelty: The method demonstrates good performance and the combination of rule-based control with reinforcement learning makes a lot of sense and potentially has huge impacts. the experiments are strong to demonstrate and validate the characteristics of the proposed method (its pros and cons)

reproducibility: code is not provided, so I am not sure if it could be reproduced. 
summary_of_the_review: Overall, I think the method is clear and strong, and results are good but could be improved. If the author could provide a reproducible work that would be even better.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method that integrates a rule-based control policy with online/offline reinforcement learning. The proposed method, RUBICON, is an extension of TD3+BC, which is one of the state-of-the-art offline RL algorithms. The basic idea is to switch the actor's loss function based on the state-action value function. The proposed method is evaluated on the HVAC control task. RUBICON outperformed the baseline methods in offline and online RL settings. 
strength_and_weaknesses: Strength
1. The proposed method utilizes the rule-based control policy to improve stability in offline and online RL settings. 
2. The experimental results show that the proposed method performs better than the baseline methods in offline and online RL settings.

Weakness
1. There is no theoretical justification. 
2. The experimental results of the offline setting show that the performance of the proposed method is competitive with CQL although CQL does not exploit the rule-based controller. 
3. The novelty of RUBICON in the online RL setting is unclear. For example, it should be compared with Residual Policy Learning.

clarity,_quality,_novelty_and_reproducibility: 1. Table 1 shows that RUBICON performed worse than CQL when the training data was collected by an expert agent. It suggests that dynamic switching of the actor's loss based on the value function does not work correctly. 

2. When the data collected by the rule-based controller is added to the buffer, do the previous offline RL methods improve performance? 

3. As described in Weakness, the proposed method should be compared with residual policy learning and TD3 with demonstration in the online setting. In particular, the latter also adds the behavior cloning loss function to the RL objective. 
Silver et al. (2018). Residual Policy Learning. arXiv. 
He et al. (2020). Deep Reinforcement Learning based Local Planner for UAV Obstacle Avoidance using Demonstration Data. arXiv. 

4. The titles of Figures 2 and 4 should be modified. For example, Eplus-5Zone-cool-continuous-stochastic-v1 is not understandable. I think it means the "cool-continuous" environment. Is it correct?

summary_of_the_review: Although the proposed method performed better than the baseline methods, I think the comparison is unfair because the baseline methods do not use the knowledge/data of the rule-based controller. In the offline setting, CQL performed better when the expert agent collected the training data. In the online setting, the proposed method should be compared with Residual Policy Learning and other baselines. Thus, I think this paper is not ready for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: To control HVAC building environments, rule-based control has been the default, but there is an expectation that learning based approaches should outperform them.  The authors introduce a hybrid approach that involves combining a rule-based scheme with learning-based approaches.  In particular, the authors want to improve upon online and offline RL approaches by incorporating a rule-based control policy for regularization.  

This is an important application domain, the paper is clear, and the proposed method performs reasonably well empirically, albeit in simulated environments.  Reviewer concerns focus on limited technical novelty and lack of theoretical justification for the approach.
justification_for_why_not_higher_score: Ultimately, this is a reasonable application paper.  However, the technical innovation is too limited and the lack of real-world evaluation of a deployed policy makes the evaluation and impact a bit niche for this venue.  I decline to endorse this for acceptance here, but I nevertheless commend the authors for their focus on contributing towards a practical solution for an important application area.
Decision: Reject