Rule-based policy regularization for reinforcement learning-based building control
Authors: Hsin Yu Liu
Bharathan Balaji
Rajesh Gupta
Dezhi Hong
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep reinforcement learning
batch reinforcement learning
buildings
HVAC

Abstract: Rule-based control (RBC) is widely adopted in buildings due to its stability and robustness. It resembles a behavior cloning methodology refined by human expertise. However, it is unlikely for RBC to exceed a reinforcement learning (RL) agent’s performance since it is challenging to ingest a large number of parameters during decision-making. In this paper, we explore how to incorporate rule-based control into reinforcement learning to learn a more robust policy in both online and offline settings with a unified approach. We start with state-of-the-art online and offline RL methods, TD3 and TD3+BC, then improve on them using a dynamically weighted actor loss function to selectively choose which policy should RL models learn from at each time step of training. With experiments across multiple tasks and various weather conditions in both deterministic and stochastic scenarios, we empirically demonstrate that our dynamically weighted rule-based incorporated control regularization (RUBICON) method outperforms representative baseline methods in offline settings by 40.7% in a reward settings consisting of the combination of thermal comfort and energy consumption and by 49.7% in online settings in building-RL environments.

Under review as a conference paper at ICLR 2023
RULE-BASED POLICY REGULARIZATION FOR REIN- FORCEMENT LEARNING-BASED BUILDING CONTROL
Anonymous authors Paper under double-blind review
ABSTRACT
Rule-based control (RBC) is widely adopted in buildings due its stability and robustness. It resembles a behavior cloning methodology refined by human ex- pertise. However, it unlikely for RBC to exceed reinforcement learning (RL) agent's performance as deep RL model constantly evolves and is scalable. In this paper, we explore how to incorporate rule-based control into reinforcement learn- ing to learn more robust policy in both online and offline settings with unified approach. We start with slate-of-the-ar online and offline RL methods, TD3 and TD3+BC, then improve on them using a dynamically weighted actor loss func- tion to selectively choose which policy RI. models to learn from each time step of training. With experiments across various weather conditions in both deter- ministic and stochastic scenarios, we empirically demonstrate that our rule-based incorporated control regularization (RUBICON) method outperforms representa- tive baseline methods in offline settings by 40.7% and by 49.7% in online settings for building-RI environments. We open-source our codes, baselines, and data for both RI. and building researchers to explore the opportunity to apply offline RL building domain.
INTRODUCTION
Most buildings implement rule-based control via building management systems, adjusting the set- points of actuators to co-optimize occupants' thermal comfort and energy efficiency. These rule- based control systems codify the problem-solving know-how of human experts, akin to behavioral cloning policy learnt from expert demonstration without randomness and uncertainty (Hayes-Roth) 1985). While stable, such control lacks the flexibility to evolve over time.
Much research has demonstrated that RL can outperform RBC in both online and offline settings. Zhang et al. (2019) developed framework for whole building HVAC (heating, ventilation, air- conditioning) control in online settings. Liu et al. (202)) incorporated Kullback-Leibler (KL) di- vergence constraint during training of an offline RL agent for stability, and deployed the policy in a real building. We focus on improving upon RL. algorithms for HVAC control buildings where rule-based policy already exists. We use established building-RI. simulation environments for experiments, in both online and offline settings (Jiménez- Raboso et al. 2021))
Reinforcement learning traditionally studied an online paradigm. However, in real-world prob- lems, configuring an accurate simulator or training environment dynamic models might be infeasible or time-consuming. Batch reinforcement learning (BRL, also known as offline reinforcement learn- ing) learns policies using only historical data without simulators to interact with during training. RL regularization methods are typically tailored specifically to online or offline setting. For example, online methods encourage exploration to either improve the estimate of non-greedy actions' values or explore better policy (Haarnoja et al. 2018, Ziebart et al. 2008 Haarnoja et al. (017). ) On the other hand, offline methods favor cxploitation since it is unlikely for BRL models to accurately estimate uncharted state-action values with a static dataset (Fujimoto et al.) 2019 Wu et al. 2019).
In our work, we explore the questions: Can we incorporate existing rule-based control policy into training of a reinforcement learning policy to improve performance? Can this method be im- plemented in both online and offline settings? TD3+BC (Fujimoto & Gu 2021) makes minimal changes to convert an online method TD3 (Fujimoto et al.) (018)) to offline mode with comparable performance as state-of-the ofette-artttttttt BRI methods only by adding behavior cloning term regularize

---

Under review as a conference paper at ICLR 2023
the policy. In TD3+BC, learning behavioral policy relies historical data. We build on this idea to regularize RL policy using an existing RBC policy combined with the behavioral policy. Our method can be incorporated into existing actor-critic RL algorithms with minimal changes.
RUBICON considers RBC as a safe policy based on which RL training can be improved. The actor selectively trains on either RBC behavioral policy, depending on which policy yields higher averaged Q-value in a mini-batch estimated by the critic network. Our proposed approach distinct from prior works in three aspects: (1) We develop unified regularization approach for both online and offline RL methods with minimal algorithmic modification. (2) Rule-based control policy is directly incorporated into the policy update step to provide stability and robustness. (3) We introduce dynamic weighting method in actor-critic settings. The actor loss is varied from time step to time step depending on the Q-value estimate of behavioral policy and RBC policy predicted from the value networks. We empirically demonstrate that our method outperforms state-of-the-ar methods in offline settings and improves on TD3 in online training in HVAC control environments. To our knowledge, previously RBC is only used as hard constraints or heuristics in RL settings, and we are the first to incorporate an existing reference policy directly into actor-critic algorithms.
2 RELATED WORK
Rule-based systems Rule-based system is one of the first artificial intelligence (Al) methods that solves many real-world problems. Planner (Hewitt, 1971) is a problem-solving language that em- beds real-world knowledge in procedures. It was used in robotic control and as a semantic base for English. MYCIN (Shortliffe. 1974) is rule-based problem-solving system to assist physicians with an appropriate therapy for bacterial infections. XCON, a production-rule ruuee-bbe--e-e system, is used to validate the technical correctness (configurability) of customer orders and to guide the actual assem- bly of these orders. has about 80K rules and achieved 95~98% accuracy. It saved $25M/year by reducing errors in orders, and gained increasing customer satisfaction (Kraft 1984).
RL + RBC The combination of RL. and RBC has been explored in many studies, where RBCs are primarily used auxiliary constraints or guiding mechanism. [ee et al. (2020) propose to two modules in their control flow, one for continuous control with RL agent and a discrete one con- trolled by RBC. [Wang al. (2019) improve RL with low level rule-based trajectory modification to achieve safe and efficient lane-change behavior. Zhu et al.) (202T) incorporate RBC for gener- ating the closed-loop trajectory and reducing the exploration space for RL pre-processing, Berenji (1992) use a learning process to fine-tune the performance of rule-based controller. Radaideh & Shirvan (02T)) first train RI. proximal policy optimization (PPO) (Schulman al. 2017) agents to master matching some of the problem rules and constraints, then RL is used to inject experiences to guide various evolutionary/stoocass RL methods. These previous methods incorporate RBC in the flow as heuristic or as hard constraints. Instead, we directly incorporate RBC policy in RL training in an algorithmic way.
algorithms. Likmeta et al. (020)) learn RBC parameters via
Online RL regularization The online baseline we will compare to in evaluation is state-of- the-art algorithm: TD3. It applics target policy smoothing regularization to avoid overfitting in the value estimate with deterministic policies. TRPO (Schulman al. 2015) uses trust region constraint based on KL--divergence between old and new policy distributions for robust policy up- dates. SAC (Haarnoja et al., (2018) uses soft policy iteration for learning optimal maximum entropy policies. Munchausen-RL (Vieillard et al. 2020) regularizes policy updates with KL-divergence penalty similar to TRPO, and adds scaled entropy term penalize policy that is far from uniform policy. Our method differs from these methods in that we incorporate an existing real-world policy for online RL training.
Offline RL regularization Offline RL is more conservative compared with online methods as it does not require interaction with the environment. It suffers from extrapolation errors induced by select- ing out-of-distrriutiin actions. Since offline RL policies are learnt entirely from a static dataset, it is unlikely for value networks to accurately estimate the values in regime where there is no sufficient state-action visitation. Thus, regularization methods become more prominent in offline settings. Batch-constie conechoonttrinee deep Q-learning (BCQ) (Fujimoto et al.] 2019), one of the pioneers of offline RL, ascribes extrapolation errors three main factors: absent data, model bias, and training mis- match. It mitigates the errors by deploying a variational autoencoder (VAE) to reconstruct the action given a state using the data collected by the behavioral policy. The offline baseline method we will
2

---

Under review as a conference paper at ICLR 2023
compare in our study is TD3+BC. It starts from online method TD3, adds behavior cloning term in the policy update to regularize the actor to imitate the behavioral policy and avoid selecting out- of-distribution actions. BRAC (Wu al. 2019) studies both value penalty and policy regularization with multiple divergence metrics (KI., maximum mean discrepancy (MMD), and Wasserstein) to regularize the actor's policy based on the behavioral policy. FisherBRC (Kostrikov al. 2021) incorporates a gradient penalty regularizer for the state-action value off'set term and demonstrates the equivalence to Fisher divergence regularization. CQL (Kumar et al. (2020) learns conserva- tive, lower-bound estimate in value network via regularizing Q-values. Model-based method, e.g. COMBO Yu al. 202I) regularizes the value function on out-of-support transitions generated via environment dynamic models' rollouts.
All of these prior works use data collected by a behavioral policy, and not assume access to any existing policy. The behavioral policy used in experiments is typically random agent or an RI. agent trained partially (medium agent) or to convergence (expert agent). In contrast, we assume direct access to robust behavioral policy in the form of rule-based control. While this assumption may not hold for robotic applications typically used to evaluate offline RL algorithms, rule-based control policies are routinely deployed in industrial control settings, such as building HVAC control.
The regularization methods in both online and offline settings regularize with ensemble models, divergence between partially trained policies, or with logged data. These algorithms still fall into the deadly triad of function approximation, bootstrapping, and off-policy training (Sutton & Barto) 2018). In case, incorporate robust reference policy to improve RL policy performance. The behavioral policy is selectively trained with actor only when the averaged Q-value estimate is higher than the rule-based control policy actions in every mini-batch. The rule-based control policy reduces uncertainty due to its deterministic behavior. On the opposite, deep learning model is af- fected by random initialization conditions, even if trained on the same dataset, as varied initialization conditions might lead to different policies.
3 BACKGROUND
In the reinforcement learning paradigm, an agent acts in an environment, sequentially selects actions based on its policy at every time step. The problem can be formulated as a Markov Decision Process (MDP) defined by a tuple A, R. P. 7), with state space S, action space A. reward function R, transition dynamics p. and discount factor r E [o, 1). The goal is to maximize the expectation of the cumulative discounted rewards, denoted by Re = "i 7'r(si, di, Si- ( Sutton & Barto (2018). The agent's behavior determined by a policy क : S - A. which maps states to actions either in deterministic way with probability distribution. The expected return following the policy from a given state s is the action-value function QT a) = E, 12 of RR41 s, 0, = al by taking action
Su
We use the building RL environment from Jiménez-Raboso et al. (2021). The objective of the agent to maintain comfortable thermal environment with minimum energy use. The state consists of indoor/outdoor temperatures, time/day, occupant count, thermal comfort, and related sensor data. The action adjusts the temperature setting of the thermostat. The reward is a linear combination of occupants' thermal comfort and energy consumption. The environment is a single floor building divided in zones, with 1 interior and 4 exterior rooms. See Appendix B for details about our building RL settings.
4 RULE-BASHE INCORPORATED CONTROL REGULARIZATION
Our goal is to improve an agent's ability to learn with the assistance of human expert's domain knowledge in both online and offline settings. In real-world problems, sometimes we have existing simulators as oracles so we can safely learn with online RL methods before deploying in the real environment, for example, in robot control (Todorov et al. 2012), Go (Silver et al. 2017). and video games (Mnih et al. 2013). However, in some real-world problems, it is time-consuming and requires domain expertise to build a functional simulation environment (e.g. building thermal simulations), or it can be dangerous trading). Offline RL algorithms, on the other hand, rely on historical data collected by an existing but unknown behavioral policy. The objective is to learn policy that improves the behavioral
risky to evaluate partially trained policy (e.g. healtheare and financial
3

---

Under review as a conference paper at ICLR 2023
policy measured through episodic rewards. In Fig. we illustrate the process of RL training that accommodates both the online and offline paradigms.
Offline approach
Online approach
Stani butfers (s),,
RBC policy Tep (5)
Simulator T(x (s, m)), r(s,
Save to bufler
Selectively learn from
Selecrively leam from
Dynamic bullers (s.a,'' r)-D,
transitions
RBC
evalusee
BRL models xz( (s.a)
Online-RI. modelss
Real enviroument
(5). (5,a)
Learn from bullir
Deploy environment
Deploy envircement
Figure 1: Our proposed method, RUBICON, it incorporates RBC into RL to improve stability in building HVAC control. It could be applied on both online and offline approaches.
Our algorithm builds on existing actor-critic algorithms TD3 and TD3+BC. We only modify the policy update strategy with the rule-based control policy, and use the critic as-is. Therefore, we focus our discussion on the policy update of the algorithm. TD3 starts from DDPG (Silver al. 2014) and mitigates the function approximation eггоr with double Q-learning and delayed policy updates. TD3+BC an offline RL algorithm adapted from TD3, and one of the state-of-the- art offiin RI methods evaluated with D4RL. datasets (Fu et al. (020). TD3+BC adds behavior cloning term to the policy update step to penalize the policy that is far away from behavioral policy (Eq. [p. The blue-colored terms indicate the changes from TD3 to TD3+BC.
arg max D( (s)( (=(s) -
(1)
n
N > (2) [Q a:)
In Eq. [ À is decided by the averaged mini-batch Q-estimate and hyperaparameter a to adjust between RL and imitation learning (Eq. 2).
Our method, RUBICON, dynamically weighs both TD3 and TD3+BC's policy update steps with either RBC policy or behavioral policy in each training iteration. Eq. 3 we replace the actions sampled from the buffers in Eq. with TQ@az (s) and add hyperparameter & to integrate TD3 and TD3+BC methods as one. Red-colored terms indicate the changes from TD3 and TD3+BC to our method. We replace the notation of sampled actions in TD3+BC with behavioral policy ms(s) to avoid confusion. Details of the hyperparameter settings in our work are in Appendix D
π = arg max EOWD (Q(s, - E(T(s) - Tmmax (8))
(3)
TQ mex (8) = arg max (Q(x, (+))) Q(+, (+)c
Every time when the policy is being updated, given the states of the sampled mini-batch, the behavioral policy (s(s) and the RBC policy Tybc(s) (s) select actions in a deterministic fashion. The state-action pairs' Q-values are estimated by the critic, the average of the Q-value estimations in the mini-batches are Q(s, TT(s)) and Q(,, (s)). We dynamically select the transitions with higher average Q-value to be regularized from in each policy update step, i.c. the actor loss function is dynamically weighted. In online settings, the behavioral policies are the older versions of the policy used to generate the buffer.
The reason we choose the average as the metric to decide which set of transitions to learn from instead of selecting each transition with higher estimated value (each batch is combination of T (s) and π. (s()) is that if we choose by each transition we will lose the information on which state- action visitations lead to worse values, the model will then suffer from the imbalanced data problem.
4

---

Under review as a conference paper at ICLR 2023
The credit/blame assignment is essential in RL learning convergence and the experience replay can help speed up the propagation process (Lin 1992). The algorithm of our method is given in Alg. [ Changes from baselines to method are highlighted in blue. d is the policy update frequency, the noise e added to the policy is sampled with Gaussian (N0, .) and clipped by c. In both online and offiine approaches, the policy update follows Eq. and and with different hyperparameter settings.
Our rule-based control algorithm is described in Alg. 2 is derived from the rule-based controller in Sinergym's is (Jiménez-Raboso et al.] 202T) example. For the purpose of computation efficiency and to fit the batch settings in our algorithm, we vectorize the original RBC policy. The rules are simple and intuitive, and can generalize well: First, we get the datetime information we need from the states. Then, we can get the seasonal comfort temperature zone for every transition. If the indoor air temperature (IAT) is below the lower bound of the comfort zone, then we set both cooling and heating setpoints degree higher (measured in Celcius degree). On the opposite, if the IAT is above the upper bound of the comfort zone, then we set both the heating and cooling setpoints a degree lower than the current setpoints. Finally, we examine if the current datetime is in the office hours. If not, then the setpoints are set to be (18.33, 23.33) ("C) for the purpose of energy reduction since occupants' thermal comfort is not important in these time periods assuming zero occupancy.
Algorithm 1: RUBICON
A Algorithm 2: Rule-based control policy
Initialize critic networks Qa Qs, ,actor network Ta. and RBC policy Thhe with random parameters 81, 82. Φ. target networks 82 + 01. 8-2 + 82, o' + o and replay buffer or load replay buffer B
Input Current daytime current dt, indoor air temperature I IAT, zone thermostat heating setpoint temperature us, and zone thermostat cooling selpoint temperature des all obtained from the states in the sampled mini-batch with size N Output: Actions selected by RBC for i in do
for = 1 to T do
if online then
Select action with exploration noise a a~ (s) c, ~ (N o)
Observe reward r and next state s'
season fom zone; = get season com fortreureent dt.)
Store transition (s, o, r, s in B
Sample mini-batch of N transitions (s, r,)s) from B
if 1 IAT > min msnssseson com fort zone;)
then
ā + ( + t,e y + +-r+ymm minn-112 u 2op (s'.a)
.c.)
Oh. dh.
deg Ge; 1 if IAT, < max ssaaorssssooomoooo cow.ooo..ooooool.ooooooo
Update critics
0₂ < argming, N Nu Qo (s, a))²
then
Ch. = ah. +1
if mod d then Update & by policy gradient:
dc, ac, +1
(ani des)) if current.dl, weekkdd 5 or
Policy update follows Fq. and
Calculate I(b)
current_dts .urrent_dt,hhour r2nge(22266) then
Update target networks:
0, = (18.33, 23.)
+ T0, + 0)0'
T + T)
5 EXPERIMENTS
In our experiments, there are two environment types: deterministic and stochastic. A Gaussian noise with μ=0 and o=2.5 is added to the outside temperature from episode to episode in the stochastic environments. And three weather types: hot, cool, and mixed. In the results, "hoor-ettrmmiiistic" indicates that the task is learnt and evaluated with hot weather condition and deterministic environ- ment. Similarly, we have all six combinations such as "cool-stochastic" and so on. More details about the RL setup is given in Appendix B All scores in tables and figures in this paper normal- ized with expert policy as 100 and random policy as 0.
5.1 OFFLINE APPROACH
First, we consider the offline approach, where no simulator exists but historical data is available. We follow the standard procedure for BRL evaluation (Fu et al. 2020): (L.) Train behavioral agents for 500K time steps, then compare the most representative algonthms DDPG, TD3, and SAC (learning curves shown in Appendix C). The online methods compare are described below:
5

---

Under review as a conference paper at ICLR 2023
Deep deterministic policy gradient is method that combines the actor-critic ap- proach and deep Q-network (DQN) (Mnih et al.) 2013). is capable dealing with con- tinuous action spaces problems via policy gradient in deterministic approach which out- performs stochastic policy method in high dimensional tasks.
DDPG:
SAC: Soft actor-critic, an off-policy maximum entropy RL algorithm to encourage explo- ration. They empirically show that SAC yields better sample efficiency than DDPG.
TD3: Twin delayed deep deterministic policy gradient algorithm, reduces overestima- tion with double Q Q-learning, combines with target networks to limit errors from imprecise function approximation.
(2.) Select the best agent as our expert agent and generate buffers with it for 500K time steps. A medium agent is trained "halfway", it means that an agent is trained most closed an agent with the evaluation performance half the performance as the expert agent. And a random agent which samples actions randomly and generate buffers.
(3.) Train BRL methods for 500K time steps and evaluate the policy every 25K time steps in all buffers mentioned above in step (2.). We show the detailed learning curves in Appendix C Normal- ized and averaged scores across runs are shown in Table T The offline methods we compare with are listed below:
TD3+BC: An offline version of TD3, it adds a behavior cloning term to regularize policy towards behavioral policy combined with mini-batch Q-values and buffer states normaliza- tion for stability improvement.
CQL: Conservative Q-learning, derived from SAC, it learns lower-bound estimate of the value function by regularizing the Q-values during training.
BCQ:
Batch-constrained deep Q-learning, implements a variational autoencoder (VAE)
(Kingma & Welling, 2013)) reconstruct the action given the state. And adds per- turbation in actor on the policy, the degree of perturbation and size of mini-batch can be adjusted in order behave more like traditional RL method imitation learning.
BC: Behavior cloning, we train a VAE to reconstruct action given state. It simply imitates the behavioral agent without reward signals.
In Table 1 we observe that RUBICON outperforms all other benchmarks in overall score across weather types, random seeds, and environment types. Other BRL methods show good performance either specific tasks with specific randomly initialized configuration; however, overall they are more unstable cf. RUBICON. Our method provides more robust and consistent performance across all variants and demonstrates the ability to generalize across various weather types and response modes of tasks. Also, as RUBICON improves on their best performance. Our method is stable since the standard deviation the least among the policies trained. We include the BRL learning curves with expert and random buffers in Appendix C
in Fig. learning from both medium buffer and RBC policy,
5.1.1 DATA RFFICIRNCY EXPERIMENT
We conduct the experiments with buffers of only one year of data (35, 040 transitions). Data ficiency is challenge for RL to yield accurate value estimation. In Table 2 we observe that our method still outperforms its baseline overall. Although it dominates with random buffers and has comparable performance with expert buffers, it does not learn well with medium buffers. The root cause is the similarity of the quality of actions between medium buffers and RBC policy, which causes the critic to misjudge which action pick between them. However, RUBICON still out- performs the baseline in other two types of buffers since the value estimation differences between (mo(s), s) and (8), s) are more prominent in these scenarios.
Some scores with standard deviation of 0 is caused by the round down of normalized scores, they negligible numbers.

---

Under review as a conference paper at ICLR 2023
Enviromment
Batter
RUBICON
TD:4C
COL
BCQ
BC
kee-dderminiiicc
Expert
86.13-17.83
99.72-0.42
10list
-3201-95446
-9.3414 S4
ke-ddterminiiic
Medium
49.58413.52
67,64532.85
26.74426.47
bee-iiterminiiic
Random
62.7-14.36
45,77
33.19-76.70
69.243361
-12.88474.61
mixed Jeterministc
Expert
81125:91
91.65E7366
10030
95. 15.75
Medium
36,25456.33
37.36486.8
27.82663.8
mised deterministi
Random
68.8324.93
13.71457.06
23.6683.61
65.29448888
103.42745
cool-cttminnisi
Expert
96e2278
8111116.8
100te0
29 29,74495.89
27.7 10
covl-detominiici
Medium
44.974364
45.44:49
8.6244531
coel-bttemiiisii
Random
-58.419.255
12.94433044
27.77+63.7
10.4070077
bot- sec-stchhastic
Expert
9500150.56
77,093344
99.49490.24
15,35484.51
-72 Mid
bot stochastic
Medium
59.72=5.29
14.84366.04
39.92356.67
62,245,25
31,22=66,22 26
Random
1.82773.31
36.61167.51
1.23168.86
10:455611
misel--stcchsiic
Experi
94.1655.12
96.62214
99.7780.21
108 08.3812.85
-02200:9 76
mised-stoo hiscc--ttccssiicc
Mediom
87.23e12.34
9,48481066
50.1 00.13+20.78
70.75-9
38,66-455 02
mixed stechadic
67.00a6.26
107.4144.6
coul-stchasss
Expert
53.58250.53
78.21431.85
9997790.32
-115885601.9
28.15=101.52
cool-stochisric
Medium
68,0700.46
16,099699 41
81.66110011
-11.55456.1 13
25.44-35.55
cool-stachssts
Random
67.55=11.14
46,33436,36
37.35=11.007
-53.92278.06
-50,37483399
1352.37=225.38
379.494714.77
960.988
-295.48+852.11
-537 93
Table I: BRL methods benchmark: Average normalized score over the final 5 evaluations and 3
random seeds. + corresponds to standard deviation over the last 5 evaluations across runs,
cool stochastic Medhum bulfers
cool det.dmmemiiicc
hot: stochssttc Medium bufters
Prediumm butters
300
-30
-30
300
-118
100000
200000
100000
400000
-150
500003
200000
300000
400000
-110
100000
230003
500000
420032
not-octermnnooc
MERDD stochast
miNDd Setermnsicc
Medium buffers
Medium butters
Madium buffers
300
1000
-10
100000
200000
300000
800000
-190
103003
200000
300000
400000
-116
100000
230003
200000
420032
Figure 2: Learning curves of RUBICON and the baseline method TD3+BC with medium buffers
5.1.2 ABLATION EXPERIMENT
In this experiment, we remove the dynamically weighted regularization. Instead we regularize the behavioral policy and RBC policy simultancously in every iteration of training (see Eq. [5) The experimental results are shown T We observe regularizing both the same time deteriorates policy training. Since in each iteration, one of RBC policy (s) and behavioral policy T,(s) yields a better action selection compared to the other. It emphasizes the necessity of dynamic weighting in the policy update steps.
arg max D (AQ(s. - (I(()) - a)² -(T(s) Triee (s)) it
(5)
5.1.3 TRANSFER EXPERIMENT
We consider a realistic scenario where in one weather condition we already have an existing buffer then we want to use it as prior knowledge to combine with RBC and transfer our model to another weather type where we have no data. We experiment with the medium buffers in stochastic envi- ronments. The results shown in Table Tindicate that method is capable of transferring from one weather condition to another with comparable performance without any hyperparameter changes. As the results indicate, due the diversity of the mixed weather, it improves the learning in cool
7

---

Under review as a conference paper at ICLR 2023
Environment
Butter
RUBICON
TDMBBC
Envarcament
Butter
RUBICON
RUBICON W/o DWW
bot-dzeemmnnticc
Expert
8945=9.12
74966112.38
bot-deterministic
Exper
86.13e17.83
-198863389
boe-daterminittic
-692678843
Mox-deterministte
47.26412.89
beedditemmiiiiic
Random
90A1n5.47
-24188-59.62
bet-doterwiiistic
Random
62.7a14.36
-1988663.8
mixed Jetermitisoo
Expect
4309180166
93.7728.71
desermitaxile
81825,94
75.0229.10
98.6742.28
determunustlc
mised detemminsttc
Random
82.89:10.19
72.5845.91
mixed deterministle
Random
68.8344.93
75.04229.46
cool-datentiniaas
Expert
89.52=11.28
86.4548.62
cock-dater determnntticc
Expeit
U8s2,78
41 34255
cool-detemmnnaiic
Medium
47.61a17.66
-ederminiiii
Medium
72.2x8.07
54467 84
ovel-litemiiistic
Random
35.02443.00
42.59472.43
woll-iieemiiiitic
66.5ee
41,3+20.55
hoe stoclaattii
Expert
80.35
89.0225.51
hot stochastic
Expext
99,01291.56
$7,68422.3
stochastie
Medium
5.3445,33
10.79456.47
stochastic
Medium
59.7245.29
26.266
stochastic
Expert
76.0k22577 78
85.25a12.96
mixed-suue stuchasiic
Espert
94.16s8.12
40,57144.91
mixed-stodii -ttchhrci
Medium
66.02+ 17
72.17e11614
prixto--tcchseiie
Medium
87.23e12.34
55.5+33.55
stechadic
cool stochattc
Expert
98.1867.56
39,08286.35
cook stochastic
Expest
53.582565.53
Ars 25.84827.44
cowl-stachratic
Medium
73.56:19.56
38,68+39.95
conk-stecluvtic
Medium
68.17:11466
8.01+61.1
cool-stachaetic
Random
69.76665.66
col-stocllatic
Random
67.55x1.14
408844227.46
Sum
1049.61552318
497.444
Sum
1362,876225.88
170.34686.
Table 2: Data reduction experiment
Table 3: Ablation experiment
Lavironmmntt
Frans. from
RU LIm
BUBICON
hot-stochisric
71,42+31
29
mixed- stochusicc
cool-tochastic
84.33666.92
87.23612.34
cool stochastic
hot stochastic
54,0720.85
65.07290.46
mased-stochastee
has-stehastte
02420.09
87.13=12.34
conl-stochsstcc
mixed- stcchuttcc
72.39.0.68
hot-stochisrie
mixed- stnchiatic
75.2643.3
50.72e5 79
439.09:44.44
23004476.18
Table Transfer experiment
and hot weathers. On the other hand, transfer from monotonic weather condition leads worse returns.
5.1.4 REWARD ANALYSIS EXPERIMENT
Since Q-value estimation is usually overestimated, want to use immediate reward as reference to examine the quality of the predictions of Q-functions. We pre-train a reward model (s, a) to predict reward given state and selected action with 200K iterations with the buffer training data. At each iteration of policy update, we record the policy =(s) and the predict rewards in each batch 2 R (s, (s)). We plot the distributions of reward in action spaces in Fig. 3 It demonstrates that RUBICON selects the actions in wider range cf. TD3+BC, however, with a reward distribution of higher values. The distribution shown is with 10% of data randomly selected from the entire training for better visualization.
04
04
650
0.5
050
-12
-025
-015
-025
-0
-100
-0.5
-100
-35
20
14
-10
-100
2.00
-0.
2.0
05
1.0
Figure 3: Reward distribution in action spaces of hot-continuous environment learns from medium buffer, from left to right: RUBICON buffer (0.908/0.11//-0.799), tuples indicate the (al range/a2 range/reward mean).
TD3+BC (1.534/1.322/-0.66) and
5.2 ONLINE APPROACH
In online approach, we assume an oracle exists for accurate simulation. real-world applications, researchers train online model in simulation before deployment in real building environments. We compare with TD3, the baseline we develop our method on. Experimental results comparing TD3 and method can be found in Table In five out of six tasks, method outperforms TD3
8

---

Under review as a conference paper at ICLR 2023
cool-stochaste
cool-dctominnize
Toc tochtochooblc
200
Enrestey
Teresto
Tereslyy
TD3
NUSIOON
10V
RUSIDON
103
MUSCON
hod-chteeminnccc
misae-ttochnniic
200
W
590000 CONEEE
100
Tirestop
Figure 4: Learning curves comparing RUBICON and TD3, solid line indicates averaged values and half- trt---aasppeentt region is the range of one standard deviation.
in averaged scores and with substantially smaller standard deviation across runs. The detailed learning curves are illustrated in Fig.
Environment
Fusimonment
bot determinssie
79.08612224
5183319993
hot deterministi
17.2412224
70,76120.25
2583168.3
deteministe
72.34:0.00
25.46:41.01
deterministic
72.M4:0.0
71.9240.22
71.26:0.05
66.52:0.00
66.3241.66
68.1624.69
bot sindlactic
X3.64:8.25
71.3:21:51
hot-stochiste
X364:8225
73.92420.224
68.1322000
mised-stochine stochssric
72.24x11.49
8.5e66.61
mixed- niredd-tneeassiic
72.24
72.2440.49
72.24.0.49
cool-stochestic
68.14+0.65
73.M616.61
cocl-stechastcc
68.14:0.65
Sum
441.99231.64
295 29:237
Sum
441.99:21.04
38755199774
14700
Table 5: RUBICON and TD3 comparison
Table 6: Hyperparameter experiment,
5.2. HYPERPARAMETER EXPERIMENT
Since we integrate TD3 and TD3+BC as one, we need to introduce the weighting À and its hyper- parameter a (see Eq. 2 and Alg. [ in the online settings. It is mentioned in TD3+BC paper that the value of decides if the model learns similar to RL (a=4) or imitation learning (a=1) and the default value set in TD3+BC is a =2.5. We experiment on the values {1, 2.5.44 4} to observe how affects the performance of our models in all tasks. The result (See Table shows that when a=1 1 the model gives the highest scores and the least variance. Which indicates that the actor policy should imitate RBC policy more in order yield traditional RI. configuration (o=4). In offline settings, we follow TD3+BC, use a=2.5.
stable and robust averaged score rather than
6 DISCUSSION
In this paper, we explore how rule-based control policy can be incorporated as regularization. Our method can implemented on the baseline method with minimal changes. We apply our method in building HVAC control simulation environments in both online and offline settings. We empirically demonstrate that our method outperforms several state-of-the-an batch reinforcement learning meth- ods and improves from its online baseline by substantial amount in building HVAC tasks where rule-based control is robust and standard real-world settings. We expect our study would en- courage both domain and RL experts to explore more opportunities for the combination of existing policies and RL and extend this concept to more real-world applications.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hamid R Berenji. reinforcement learning- based architecture for fuzzy logic control. Interna- tional Journal of Approximate Reasoning, 6(2):267-292, 1992.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
Department of Energy. Prototype Building Models.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep
data-driven reinforcement learning. arXiv preprint arXiv:200..072/9, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. JCML, pp. 2062-2062. PMLR, 2019.
Tuomas Haamoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference machine learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, pp. 1861-1870. PMLR, 2018.
Frederick Hayes-Roth. Rule-based systems. Communications of the ACM, (8(9):921-9322 1985.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Carl Hewitt. Procedural embedding of knowledge in planner. In IJCAI, pp. 167-182, 1971.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, and Jeff Braga. Cleanrl: High- quality single-file implementations of deep reinforcement learning algorithms. arXiv preprint arXiv:2111.08819, 2021.
Javier Jiménez-Rabose Alejandro Campoy-Nieves, Antonio Manjavacus-Lucas, Juan Gómez- Romero, and Miguel Molina-Solana. Sinergym: a building simulation and control framework for training reinforcement learning agents. In Proceedings the 8th ACM International Confer- ence on Systems for Energy-Fificcien Buildings, Cities, and Transportation, pp. 319-323, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 141iv:4412.699 69800,90, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. 1312.613 14, 2013.
arXiv preprint
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021.
Arnold Kraft. Xcon: An expert configuration system at digital equipment corporation. The Al Busi- ness: The Commercial Uses of Artificial Intelligence, The MIT Press, Cambrige, Massachusetts, 1984.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
10

---

Under review as a conference paper at ICLR 2023
Dacil Lee, Awwal Mohammed Arigi, and Jonghyun Kim. Algorithm for autonomous power-increase operation using deep reinforcement learning and a rule-based system. JEEE Access, 8:196727- 196746, 2020.
Amarildo Likmeta, Alberto Maria Metelli, Andrea Tirinzoni, Riccardo Giol, Marcello Restelli, and Danilo Romano. Combining reinforcement learning with rule-based controllers for transparent and general decision-maie mankin in autonomous driving. Robotics and Autonomous Systems, 131: 103568, 2020.
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8(3):293-321, 1992.
Hsin-Y Yu Liu, Bharathan Balaji, Sicun Gao, Rajesh Gupta, and Dezhi Hong. Safe hvac control via batch reinforcement learning. In 2022 ACM/IEEE 13th International Conference on Cyber- Physical Systems (ICCPS), pp. 181-192. IEEE, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5600 2013.
National Renewable Energy Laboratory. TMY3 Datasets. EyC8ost
https://www nrel wwrellgom http://wwwwwww./..ee..e.s.e 3gov/docs/
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Majdi Radaideh and Koroush Shirvan. Rule-based reinforcement learning methodology to inform evolutionary algorithms for constrained optimization of engineering applications. Knowledge- Based Systems, 106:16 2021.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Takuma Seno and Michita Imai. d3ripy: An offline deep reinforcement learning library. arXiv preprint arXiv:2111.0377 2021.
Edward Hance Shortliffe. Myein: rule-based computer program for advising physicians regarding antimicrobial therapy selection. Technical report, Stanford Univ Calif Dept of Computer Science, 1974.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pp- 387-395. PMLR, 2014.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lueas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-355,, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033 IEEE, 2012.
Stefan Van Der Walt, S Chris Colbert, and Gael Varoquaux. The numpy array: structure for efficient numerical computation. Computing in science & engineering, 13(2):22330, 2011.
Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Munchausen reinforcement learning. Advances in Neural Information Processing Systems, 33:4235 4246, 2020.

---

Under review as a conference paper at ICLR 2023
Junjie Wang. Qichao Zhang, Dongbin Zhao, and Yaran Chen. Lane change decision decision-making through deep reinforcement learning with rule-based constraints, In 2019 International Joint Conference on Neural Newwork (IJCNN), pp. 1-6. IEEE, 2019.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. 2019.
arXiv preprint
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances neural information processing systems, 34:28954-28967, 2021.
Zhiang Zhang, Adrian Chong. Yuqi Pan, Chenlu Zhang, and Khee Poh Lam. Whole building energy model for hvac optimal control: practical framework based on deep reinforcement learning. Energy and Buildings, 199:472-499 49072244-.400 2019.
Yuanyang Zhu, Zhi Wang, Chunlin Chen, and Daoyi Dong. Rule-based reinforcement learning for efficient robot navigation with space reduction. IEEE/ASME Transactions on Mechatronics, 27 (2)8846-857, 2021.
Brian Ziebart, Andrew I. Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008
12

---

Under review as a conference paper at ICLR 2023
A
EXPERIMENT DETAILS
Software
Python: 3.9.12
Pytorch: 1.12.1+cul13 (Paszke et al.
2019)
Sinergym: 1.9.5
Jiménez-Raboso et al. 2021
Gym: 0.21.0 (Brockman et al. 2016
Numpy: 1.23.1 (Van Der Walt et al. 2011
CUDA: 11.2
Hardware
CPU: Intel Xeon Gold 6230 (0.10 10 GHz)
GPU: NVidia RTX A6000
Benchmark implementations
DDPG: We adopt the DDPG implementation in TD3 author-provided implementation
TD3: Author-provided implementation
SAC: We adopt CleanRL (Huang et al. 2021
implementation due to software version
conflict with author-provided repository
TD3+BC: author-provided implementation
CQL: We adopt d3rlpy (Seno & Imai 2021
implementation due to software version con-
flict with author provorded repository
BCQ: Author-provided implementation
B BUILDING RL SETTINGS
In this section, we list all details about the MDP settings our problem.
Site outdoor air dry bulb temperature. site outdoor air relative humidity, site wind speed, site wind direction, site diffuse solar radiation rate per area, site direct solar radia- tion rate per area, zone thermostat heating setpoint temperature, zone thermostat cooling setpoint temperature, zone air temperature, zone thermal comfort mean radiant tempera- ture, zone air relative humidity, zone thermal comfort clothing value, zone thermal comfort Fanger model PPD (predicted percentage of dissatisfied), zone people occupant count, peo- ple air temperature, facility total HVAC electricity demand rate, current day, current month, and current hour.
State:
Action: Heating setpoint and cooling setpoint in continuous settings.
Reward: We follow the default linear reward settings, it considers the energy consumption and the absolute difference to temperature comfort.
Environment: A single floor building with area of 463.6m² divided in 5 zones, interior and 4 exterior. The HVAC system is a packaged VAV (variable air volume) (DX (direct expansion) cooling coil and gas heating coils) with fully auto-sized input. The control variables are the cooling and heating temperature setpoints for the interior zone, and the simulation period is a full year TTmenez--aiin are classified according to the U.S. Department of Energy (DOE) standard (Department of Energy). The weather type details and their representative geometrie locations are listed below:
et al.) 2021). The weather types
Washington, USA. The mean annual temperature and mean annual relative humidity are 9. and 81.1% respectively.
Cool marine:
Hot dry: Arizona, USA with mean annual temperature of 21.7%C and mean annual relative humidity of 34.9%
13

---

Under review as a conference paper at ICLR 2023
Mixed humid: New York, USA with mean annual temperature of C and mean annual relative humidity of 68.5%
based on TMY3 datasets (National Renewable Energy Laboratory)
C LEARNING CURVES AND ADDITIONAL EXPERIMENTS
C.I LEARNING CURVES OF MAIN EXPERIMENTS
BRL learning We illustrate the learning curves of BRI. methods learn from different qual-
ity of buffers for better visualization of comparison in Fig. 2 and 7
Transfer learning The learning curves of BRL transfer experiments are illustrated in Fig.
Behavioral agents The behavioral agents' learning curves are demonstrated in Fig. 9
Hyperparameter optimization Learning curves of hyperparameter experiments in online
RUBICON are shown in Fig. T0
C.2 ADDITIONAL EXPERIMENTS
Learn from RBC buffers To simulate more realistic scenario where we learn from buffers of real-world HVAC control dataset. It consists of transitions generated from rule- based control policy, which is widely used in building HVAC control. The learning curves are illustrated in Figure 11 The results in Table 7 indicate that even when learning with RBC buffers with RBC policy itself, RUBICON could still outperforms RBC policy due to its learning ability.
CQL RUBI+RU Since CQL demonstrates better performance compared to other meth- ods except RUBICON (see Table we conduct experiments combining CQL and our RU- BICON method to learn from random buffers since we expect the most improvement in this scenario. The results in Figure 12 and Table Sindicate that RUBICON also improves CQL. However, it does not consistently improve CQL's performance from task to task, which is not as we observe from TD3+BC to RUBICON learn from random buffers (see Figure Also, the improvement limited and not reach the RBC policy performance, thus did not continue exploring the possibility combining CQL with RUBICON.
Learn from a mixture of the original buffer and RBC buffer order to evaluate mixing the buffers (of RBC buffer and the original buffer to learn from) is equivalent to RUBICON, we conduct experiments mixing 50% of transitions in RBC buffer with 50% of transition in the original buffer to learn from. The result shown in Figure 14 and Table TO] It indicates our selective algorithm is necessary to dynamically decide RBC policy or the behavioral policy learn from instead of randomly trained both.
Learn from worsened RBC policies We run another ablation experiment to observe how the quality of RBC policy affects the performance compared with RUBICON and baseline TD3+BC. We design two worsened RBC policies: The first one is biased RBC where we modify the change in setpoints (ap, and a,,) from to in Algo. []] we name this method "RBC CB" in Figure 15] The other is replace RBC with random policy, it is named as "RBC Random". From the results in Table TT we could find that even with constantly worsened RBC policy it still improves from baseline, However, it is still too aggressive for the models to learn robust policy. And with random policy as a worsened RBC it is almost equivalent no reference policy, the performance similar our baseline TD3+BC.
All learning curves are normalized with random policy 0 and expert policy 100, averaged with 3 random seeds and the scores shown in tables are the average and standard deviation last evaluations unless mentioned otherwise.
D MODEL PARAMETERS
We list the hyperparameters used in this paper for reproducibility. Unless mentioned otherwise, keep the original hyperparameters setups as the implementations listed in Sec, since DRL methods
sensitive hyperparameter tuning (Henderson al. 2018) (see Table 12
13
14
and 15).
14

---

Under review as a conference paper at ICLR 2023
cool-stochastic Expert
cool-dete min stic Expert
Normalized apisode reward
Normalizze apicode roward
100
100
-50
-50
-100
-100
-150
-150
100000
280300
300000
400000
100000
400000
Timestop
Timestop
Export
Export
het-stochastic Expert buffers
hot-deterministic Expert buffers
Normalized apisode reward
100
Normalized apicode reward
-30-
1100
-100
-150
-150
100000
200000
300000
400000
100000
200000
300000
400000
Timastap
mimasta
TC33.
RUBICON
Expert
Experl
c-tcchastcc Expert buffers
mixed-dleted determiiitiiiiiiccc Expert buffers
Normalized apicede reward
100
Normalized apicede reward
100
50
0-
-50-
-50-
-100
100
-150
-150
100000
200000
300000
400000
100000
200000
300000
400000
Timastapp
mamstap
TC3, BC
RUBICON
Expert
TD3. BC
RUBICON
Expert
Figure 5: Learning curves of BRL models learn from expert buffers.
15

---

Under review as a conference paper at ICLR 2023
cool-stochastic
cool-deterministic
Normalized apisode reward
100
Normallzzed apicode roward
100
-50
1100
-150
-150
100000
202000
400000
100000
202000
Timestop
Timestop
COL
Medium
C0L
Medium
BCQ
BCQ
hot-stochastic
hot-dettrministic
Medium buffers
Medium buffers
Normalized apicode reward
100
Normalized apicode reward
100
50
-50
100
1100
-150
-150
100000
202000
300000
400000
100000
202000
300000
400000
Timestop
Timestap
RUBICON
COL
Medium
RUBICON
30L
Medium
TD3_BC
BCO
siixed-stochhstic
wwwed-deterministiie Medium buffers
Medium buffers
Normalized apicede reward
100
Normalized apicode reward
100
30
50
-50
-100
-100
-150
-150
100000
20C000
300000
400000
100000
202000
300000
400000
Timestap
Timestap
RUBICON
CQL
8C
Medium
RUBICON
3QL
BC
Medium
TD3,BC
Figure 6: Learning curves of BRL models learn from medium buffers.
16

---

Under review as a conference paper at ICLR 2023
cool-stochastic Random buffers
ccol-deterministic Random buffers
100
Normalized apisode roward
Normalized apicode roward
100
50
-50
-100
-100
-150
160000
100000
Timestep
Timestep
Rar dom
COL
COL
RBC
hot-stochastic
not-deterministic
Random buffers
Random buffers
Normalized apicode reward
100
Narmalized apicode reward
-50
550
1100
100
-150
-150
100000
200000
300000
400000
100000
200000
300000
400000
Timestep
Timestop
TD3 BC
RUBICON
Rai dom
TD3 BC
Rai don
wed-stochasttic Random buffers
dixeedddeeemmnistiicc
Random buffers
Normalized apicede reward
100
Normalized apicode reward
100
-50
-50
-100
-100
-150
-150
100000
200000
300000
402000
100000
200000
300000
400000
Timestep
Timestop
TD3,80 BC
RUBICON
Rar dom
TD3, BC
RUBICON
Rar dom
Figure 7: Learning curves of BRL models learn from random buffers
17

---

Under review as a conference paper at ICLR 2023
-tocsstch Medium
cool-stochastic Medium buffers
aontralieed spisoade rewero
80-
Nonnal bld spicade rawerdd
85
80
70
60
50
100000
200000
300000
4C0000
100000
200200
300000
400000
Timestap
Timestep
Tran. from hot
RUBICON_ DW
RBC buffer
Tran. from mixed
RLIBICON, DW
RBC buffer
hot-stochastic Medium buffers
not-stochaste
Medium buffers
85
Nonralized apiscade rowrre
Normalzzed spicaae roweted
70
es
co
100000
200000
300200
400000
100000
200000
300000
400000
Timestop
Timestepp
Tran. from cool
RUBI CON 2W
RBC buifer
Tran, from mixed
RUBICON, DW
RBC buffer
stocchassccc Medium buffers
mixed-stochastic Medium buffers
Normalzzed apisade roward
Normalzzed apicade rowntd
100
90
80
60-
60
10000C
200000
300000
4000C0
100000
200000
300000
400000
Timestep
Timestop
Tran. from cool
RUEICON, DW
RBC buffer
Tran. from hot
RUBICON, DDW
RBC buffer
Figure 8: Learning curves of BRL models transferred from other weather types
18

---

Under review as a conference paper at ICLR 2023
cool- stochastic
doal.deterministic
100
100
Episode reward
Episode reward
30
50
-50
-50
-100
-100
100000
200000
300000
400000
500000
100000
200000
300000
400000
500000
Timeslep
Timneslep
DDPG
TD3
SAC
DDPG
TD3
hot stochaschatic
hot-dettrministic
100
Episode reward
Episode reward
30
0
-50
100
-100
100000
200000
300000
400000
500000
100000
200000
300000
400000 500000
Timestep
Timestep
DDPG
TD3
SAC
DDPG
TD3
SAC
mixed-stochastic
mixed--eterministicc
100
100
Episode reward
Episode reward
50
-50
-50
-100
-100
100000
200000
300000 400000 500000
100000
200000
300000
400000
500000
Timestep
Timestep
DDPG
DDPG
Figure 9: Learning curves of behavioral model training, behavioral models are trained with 500K time steps before generating buffers.
19

---

Under review as a conference paper at ICLR 2023
cool- stochastic
determmmmiiiiii
100
Episade reward
50
Episade reward
-50
-100
-100
100000
200000
400000 Timestep
300000
500000
100000
200000
300000
400000 500000
Timestep
alpha=2.5
alpha=1
alpha=4
alpha=2.5
alpha=1
alpha=4
not-stochhstic
hot-detemmnisticc
100
Episode reward
50
Episode reward
50
0-
0-
-50
-100
-100
100000
200000
200000
400000
500000
100000
200000
300000
400000
500000
Timestep
Timestep
alpha=2.5
alpha=1
alpha=4
alpha=2.5
alpha=1 alpha=4
mixed-stochastic
dixee--detrministicc
100
Episode reward
75
Episode reward
75
50
50-
0-
0-
-25
-25
-30
100000
200000
300000
400000
560000
100000
200000
300300
400000
500000
Timestop
Timestop
alpha=2.5
alpha=1
alphe=4
elpha=2.5
alpha=1
alphe=4
Figure 10: Learning curves of online RUBICON hyperparameter optimization.
20

---

Under review as a conference paper at ICLR 2023
cool-stochastic buffers
cool-deterministic
RBC buffers
100
100
Normalized apicode reward
Normalized apicode roward
50-
0-
-50
-150
-15
100000
200000
300000
400000
100.00
2000000
300000
ACOCOn
Timestep
Timestap
KUBICUN ABC
RUL
HUBICUN HBC
MBC
hot-stochastic buffers
hoz-deterministic
buffers
100
100
Normalized epicode roward
Normalizzd epicode roward
-50
-50
-150
-150
100000
200000
500000
A00000
100200
200000
500000
A00000
Timestap
Timestap
KUBICUN ABC
RBC
RUBICUN RBC
RBC
mixed-stochastic RBC buffers
fiixed-deterministee
RBC buffers
100
100
Nomalized epicode raward
Normalized epicode roward
50-
-50
-50
100
-150
-150
100000
200000
100000
400000
100300
200000
100000
400000
Timestop
Timastap
RUBICON ABC
RBC
RUBICON RBC
RBC
Figure 11: Learning curves of RUBICON learns from RBC buffers.
21

---

Under review as a conference paper at ICLR 2023
cool-stochastic
cool-deterministe buffers
100
Normalized apisode roward
Normalized apicode roward
100
50
50
0
-50
-100
-100
-150
-150
202000
100000
200000
Timestop
Timestep
COL.
Randem
COL
Randam
RUBICON
RUBICON
hot-stochastir
hot-deterministic
Random buffers
Random buffers
100
Normalized apicode reward
100
Normalizee apicode reward
-30
-50
1100
-150
-150
100000
200000
300000
400000
100000
200000
300000
400000
Timestop
Timestep
COL
Rendum
COL
RUBICON
Rendam
RUBICON
RUBICON
mnixed-stochastee Random buffers
m deteeddtterministicc
Random buffers
100
Normalized apicede reward
Normalized apicode reward
100
50
50
0
-30
-50
-100
-150
-150
100000
200000
300000
400000
100000
200000
300000
400000
Timastap
Timastep
COL
RUBICON
Rendom
COL
RUBICON
Rendom
COL RUBICON
COL RUBICON
Figure 12: Learning curves of CQL, COL+RUBICON, and RUBICON learn from random buffers
22

---

Under review as a conference paper at ICLR 2023
cool-stochastic Medium buffers
cool-deterministic
Normalizzd apisode reward
100
Normalized apicode roward
100
0-
-50
-50
-100
-00
-150
-150
100000
202000
300000
400030
100000
200000
300030
400000
Timestop
Timestop
COL
Medium
Meditmn
COI_RLBICON
RBC
COL_RUBICON
RBC
hot-stochastic
hot-deterministic Medium buffers
Medium buffers
Normalized apicode reward
100
Normalized apicode reward
100
50
-30-
1100
1000
-150
-150
100000
200000
300000
400030
100000
200000
300000
400000
Timestop
Timastsp
COL
RUBICON
Mediunn
COL
RUBICON
Mediven
COL_RLBICON
COL_RUBICON
stochd-stochastic
m dedeeeeeeriiiittccc Medium buffers
Medium buffers
100
Normalized apicede reward
Normalized apicede reward
100
50
50
-30
-50
-100
-150
100000
200000
300000
400030
-150
100000
200000
300000
400000
Timestop
Timastop
COL
RUBICON
Medium
COL
RUBICON
Medium
RLBION
COL_RUBICON
Figure 13: Learning curves of CQL, CQL+RUBICON, and RUBICON learn from medium buffers
23

---

Under review as a conference paper at ICLR 2023
cool-stochastee Random burfers
cool-determi nistic Random buffers
Normal 2od ap sode roward
100
Normal zed ap sppicode reword
100
50
-50
-100
-150
100000
200000
30000C
4C0000
100000
20000C
300000
400300
RUBICON
TD3_BC
Random
RUBICON
TD3_Ba BC_BC
Random
TD3, BC Mixed
RBC
TD3, BC Mixed
RBC
hot-stochastic
hot-deterministe
Random buffers
Random buffers
100
100
Norma thed apicoda roward
Norma tred apicoda roward
-50
-100
100
100000
200000
30000C
400000
100000
20000C
300000
400000
Timestep
Timestep
RUBICON
TD3_BC
Random
RUBICON
TD3_BC
Random
TD3, BC, Mixed
RBC
TD3_ BC, Mixed
RBC
mixed-stochastic Random buffers
mixed-deter ministic Random buffers
100
Norm: linee apicote roward
100
Norm: lines apicote roward
-50
-100
-100
150
100000
200000
30000C
4D0100
100000
200100
300000
420100
RUBICON
TD3, BC
Random
RUBICON
TD3. BC
Random
TD3_ BC, Mixed
RBC
TD3_ BC, Mixed
RBC
Figure 14:
Learning curves comparing RUBICON and TD3+BC TD3+BC learns from mixture of 50% amount of transitions from the random buffer and 50% amount of transitions from the RBC buffer in stochastic environments
24

---

Under review as a conference paper at ICLR 2023
cool-stochastic Random buffers
cool-determi nistic Random buffers
100
100
Normal zed ap spicodd reward
Normall zod ap spicode reward
-50
-50
-150
-150
100000
200000
300000
400000
100000
200000
300000
400000
Timessep
Timestep
RJJBIDNN
RBC, Random
RBC
RUBICON
RBC, Random
RBC
R3C
BC
RBC
TD3 BC
hot-stochastic Random buffers
neeemeetrmiiistic Random buffers
100
100
Norma tred epicods reward
Narma brod opisods reward
-50
-50
100
-150
-150
100000
200000
300000
400000
100000
200000
300000
400000
Timestep
Timestep
RJBICON
RBC Random
ROC
RJBICON
RBC Random
RDC
R3C_CB
BC
Rancom
RaC_CB
TD3_BE BC
mticht-ttochastic Random buffers
mixed-deter determiitttiiiiiiic Random buffers
Norma line opicaze reward
100
Nirrmzllzes lined opicaze reward
50
-50
-100
-100
150
150
100000
200000
300000
400000
100000
200000
300000
408000
Timestep
Timestep
R.JBICON
RBC Random
RBC
RJBICON
RBC Random
RBC
RBC_CB
TD3_BC
R3C_CB
TD3_BC
Figure 15: Learning curves of RUBICON learns from worsened RBC compared with TD3+BC and RUBICON
25

---

Under review as a conference paper at ICLR 2023
Environment
RUBICON
RBC
hot-deterministic
67.92422.51
57.9
mixed-determmnisticc
73.68+1.96
50.12
coot-deteriinistic
72.28+6.53
59.15
hot-stochastic
53.83+0.8
57.92
mixed-stochastic
72.46+0.68
50.22
cool-stochastic
53.35420.36
58.48
Sum
393.53+52.85
333.79
Table RUBICON learns from buffers generated by RBC compared with RBC buffer performance
Environment
RUBICON
CQL.RUBICON
CQL
hot-deterministic
62.7+14.36
48.37=10.12
23..19476.76
mixei-dettemmniiticc
68.83+4.93
-2485.18
-33.46:88.61
coot-deterministic
66.5+0
88.98=13.24
12.98473.04
hot-stochastic
68.83+21.26
.47.04+45.48
36,64467.61
mixed-stochastic
67.03+6.26
38.08449.17
94.0445.87
cool-stochastic
67.55=1.114
-73.76420.51
-97.35+11.00
Sum
401.47247.98
52.644223.72
-0.22331.977
Table 8: CQL+RUBICON learns from random buffer compared with CQL and RUBICON
Environment
RUBICON
CQL.RUBICON
CQL
hot-deterministic
64.91=18.02
43.03155.26
67.64+32.83
dixerddeterministic
86.84#12.39
73.4137.59
37.36486.8
cooe-deteriinistic
72.2+8.07
85.24+17.33
55.44+49
hot-stochastic
59.72+5.29
9.39458.72
39.92456.67
mixed-stochastie
87.23+12.34
90.14+9.37
80.13420.78
cool-stochastic
68.07+0.46
91.05+11.39
81.56+18.01
Sum
438.98=56.59
392.26x1 189.69
362.074264.11
Table 9: CQL+RUBICON learns from medium buffer compared with CQL and RUBICON
Environment
TD3+BC_Mixed
RUBICON
TD3+BC
hot-deterministic
0.02+59.76
62.7+14.36
45.73+44.8
detedddetermiiistic
70.66+15.45
68.8314.93
-13.71457.00
cool-deterministic
59.01+44.92
66.520
-58.4+19.25
hot-stochastic
57.93+5.6
68.83+21.26
-1,82473.31
mixed-stochastic
74.08+8.7
67.03+6.26
28.01472.79
cool-stochastic
71.67435.04
67.55+1.14
.4..3++36.36
Sum
333.4+165.5
401.47247.98
135.984303.60
Table 10: TD3+BC learns from a mixture random buffer and RBC buffer compared with RUBICON learns from random buffer
Environment
RUBICON
TD3+BC
RBC.CB
RBC.Random
hot-deterministic
62.7+14.36
45.73:44.8
34.06+27.13
-47,72425.1
mixed-deterministicc
68.83+4.93
-13.71457.00
-33.89+38.41
36.39472.09
cool-deterministic
66.5+0
-58.4+19.25
70.6445.85
-48.84425.93
hot-stochastic
68.83421.26
-1.82473.31
33.81+36.99
-67.745.66
mixed-stochastic
67.03+6.26
28.01472.79
71.2242.64
4.07452.44
cool-stochastic
67.55+1.14
44.33436.36
65.8443.M 06
-5.9474.12
Sum
401.47+47.98
-33.984303.60
241.69
-137.85#255.38
Table 11: Comparison between RUBICON, TD3+BC, and worsened RBCs
26

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Value
Optimizer
Adam (Kingma 2014
Critic learning rate
3c
Actor learning
3e
Mini-batch size
256
Discount factor
0.99
Algorithm hyperparameters
Target update rate
5e
Policy noise
0.2
Policy noise clipping
(-0.5, 0.5)
Policy update froquency TD3+BC
2.5
RUBICON online
RUBICON offline a
RUBICON online
0 Qs(s, ma()) 2 Q8(s, (s)) else 1
RUBICON offline
Critic hidden dimension
256
Critic hidden layers
Critic activation function
ReLU
Network architecture
Actor hidden dimension
256
Actor hidden layers
Actor activation function
ReLU
Table 12: TD3, TD3+BC, and RUBICON hyperparameters
Hyperparameter Optimizer
Value
Adam
Critic learning rate
1e
Actor learning rate
3e "om"/
Mini-batch size
256
Discount factor
0.99
Target update rate
5e
Algorithm hyperparameters
Policy noise
0.2
Policy noise clipping
(-0.5,
Policy update frequency
SAC entropy auto-tuning
True
CQI threshold
10
CQI. conservative weight
5.0
CQL number of sampled actions
10
Critic hidden dimension
256
Critic hidden layers
Critic activation function
ReLU
Network architecture
Actor hidden dimension
256
Actor hidden layers
Actor activation function
ReLU
Table 13: SAC/CQL hyperparameters
27

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Value
Optimizer
Adam
Critic learning
1e
Actor learning rate
le-
Mini-batch size
64
Algorithm hyperparameters
Discount factor
0.99
Target update rate
1e-3
Policy noise
(0 0.1)
Policy noise clipping
(-0.5, 0.5)
Policy update frequency
Critic hidden dimension
400/300
Critic hidden layers
Critic activation function
ReLU
Network architecture
Actor hidden dimension
400/300
Actor hidden layers
Actor activation function
ReLU
Table 14: DDPG hyperparameters
Hyperparameter
Value
Optimizer
Adam
Critic learning rate
3
Actor Icarning rate
le
Mini-batch size
100
Algorithm hyperparameters
Discount factor
0.99
Target update rate
5e
Minimum weighting
0.75
Max perturbation
0.05
Critic hidden dimension
400/300
Critic hidden layers
Critic activation function
ReLU
Actor hidden dimension
400/300
Network architecture
Actor hidden layers
Actor activation function
ReLU
VAE hidden dimension
750
VAE latent vector clipping
(-0.5, 0.5)
Table 15: BCQ/BC hyperparameters
28


Reviews:
[{"review_id": "emYTiM4aes2", "reviewer": "Reviewer_r7Hj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a method to unify rule-based control with reinforcement learning for building management and control. The work provides both the online and offline setting variants. The idea is to extend the previous work of TD3 + BC by replacing the behavior policy with a dynamically weighted policy that chooses action either from the behavior policy or the rule-based policy. In TD3+BC, the policy update rule is updated to both maximize the value w.r.t. the policy, and also minimize the difference of the policy with a behavior cloning policy. In order to incorporate rule-based control, the method proposes to choose whichever policy with the maximum value from the collection of behavior cloning policy and rule-based policy. Experiments are performed in the offline setting and online setting. The offline setting demonstrates that the method has better data efficiency than other baselines, and is able to transfer from one weather condition to another with comparable performance. The online experiment also demonstrates that the method is better than TD3. ", "summary_of_the_review": "Overall, I think the method is clear and strong, and results are good but could be improved. If the author could provide a reproducible work that would be even better.", "strength_and_weaknesses": "Strength: the paper is well written and contributions are appropriately stated, and the algorithm itself is easy to understand. The performed experiments show that the proposed algorithm is useful in some cases (medium and random buffer, etc.)\n\nWeakness: (1) the figures are not designed carefully. For example, figure 1, it is not clear what each arrow mean, and it seems \"solid lines\" not always means \"fixed actions\". figure 2, the difference between different methods aren't that huge to distinguish one from another. Figure 3, it is not clear why this \"RUBICON selects actions in a wider range\" and \"TD3+BC has a reward distribution of higher values\", since it's hard to visually validate this, and better using quantitative approach if the author really want to prove this. \n\n(2) some results are mixed. sometimes the proposed algorithm performs better and sometimes not, the provided explanations are not very convincing. For example, why in hot-stochastic case, RUBICON performs better in table 1 with. expert data, while other cases, with expert data, RUBICON performs worse. Some results have 0.0 standard deviation, is that correct or error in data recording? is there a reason why it is exactly zero?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8USnXRr6P8S", "reviewer": "Reviewer_6eMZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach for combining rule based policies and RL methods, such as TD3+BC and TD3. The proposed approach performs well (though I have no clear calibration of the tasks being benchmarked upon). The resulting policy attains good performance and can improve upon heuristics obtained through the rule based policy.", "summary_of_the_review": "Overall, I like the paper, but I am giving a reject score right now because I think the paper would be much stronger if my weaknesses are addressed. if the authors can address those weaknesses, I will move up to an accept.", "strength_and_weaknesses": "Strength: While the idea of combining rule based learning with RL is not new, the paper presents an example with offline RL with a convincing downstream application (building control).\n\nWeaknesses:\n\n- Can the rule based policy also be learned, like parameters of a rule, to make it robust? More generally, how robust is this approach if the building changes? Can we see an ablation study where we compare performance of the method to standard offline RL when the rule based policy becomes worse and worse?\n\n- Since TD3+BC is worse than CQL, as evident from the table, can this method also be applied to CQL to see how well it performs? \n\n- Would be good to do some qualitative analysis: when does the RL policy perform particularly much better than the rule-based policy? What happens if the data composition is different (more representative of real-world systems than random, medium, expert from D4RL)?\n\nI like this paper overall, and would encourage authors to answer the questions above, after which I am happy to increase my score. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5SlI-kaQ22", "reviewer": "Reviewer_8sr1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method that integrates a rule-based control policy with online/offline reinforcement learning. The proposed method, RUBICON, is an extension of TD3+BC, which is one of the state-of-the-art offline RL algorithms. The basic idea is to switch the actor's loss function based on the state-action value function. The proposed method is evaluated on the HVAC control task. RUBICON outperformed the baseline methods in offline and online RL settings. ", "summary_of_the_review": "Although the proposed method performed better than the baseline methods, I think the comparison is unfair because the baseline methods do not use the knowledge/data of the rule-based controller. In the offline setting, CQL performed better when the expert agent collected the training data. In the online setting, the proposed method should be compared with Residual Policy Learning and other baselines. Thus, I think this paper is not ready for publication.", "strength_and_weaknesses": "Strength\n1. The proposed method utilizes the rule-based control policy to improve stability in offline and online RL settings. \n2. The experimental results show that the proposed method performs better than the baseline methods in offline and online RL settings.\n\nWeakness\n1. There is no theoretical justification. \n2. The experimental results of the offline setting show that the performance of the proposed method is competitive with CQL although CQL does not exploit the rule-based controller. \n3. The novelty of RUBICON in the online RL setting is unclear. For example, it should be compared with Residual Policy Learning.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject