ExID: Offline RL with Intuitive Expert Insights in Limited-Data Settings

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 02:20:57

Abstract: With the ability to learn from static datasets, Offline Reinforcement Learning (RL) emerges as a compelling avenue for real-world applications. However, state-of-the-art offline RL algorithms perform sub-optimally when confronted with limited data confined to specific regions within the state space. The performance degradation is attributed to the inability of offline RL algorithms to learn appropriate actions for rare or unseen observations. This paper proposes a novel domain knowledge-based regularization technique and adaptively refines the initial domain knowledge to considerably boost performance in limited data with partially omitted states. The key insight is that the regularization term mitigates erroneous actions for sparse samples and unobserved states covered by domain knowledge. Empirical evaluations on standard discrete environment datasets demonstrate a substantial average performance increase compared to ensemble of domain knowledge and existing offline RL algorithms operating on limited data.

Main paper: 
 introduction: 
 Offline RL [9, 1] , also referred to as batch RL, is a learning approach that focuses on extracting knowledge solely from static datasets. This class of algorithms has a wider range of applications being particularly appealing to real-world data sets from business [46] , healthcare [25] , and robotics [35] .However, offline RL poses unique challenges, including over-fitting and the need for generalization to data not present in the dataset. To surpass the behavior policy, offline RL algorithms need to query Q values of actions not in the dataset, causing extrapolation errors [21] . Most offline RL algorithms address this problem by enforcing constraints that ensure that the learned policy does not deviate too far away from the data set's state action distribution [13, 11] or is conservative towards Out-of-Distribution (OOD) actions [21, 20] . However, such approaches are designed on coherent batches [13] , which do not account for OOD states.In many domains, such as business and healthcare, available data is scarce and often confined to expert behaviors within a limited state space. For example, a sales recommendation system, where historic data may not contain details about many active users and operator gives coupon of higher value to attract sales. Learning on such limited data sets can curtail the generalization capabilities of state-ofthe-art (SOTA) offline RL algorithms, resulting in sub-optimal performance [23] . We illustrate this limitation via Fig 1 . In Fig 1a ) the state action space of a simple Mountain Car environment [27] is plotted for an expert dataset [32] and a partial dataset with first 10% samples from the entire dataset.shows the average reward obtained over these data sets and the average difference between the Q value of action taken by the under-performing Conservative Q Learning (CQL) [21] agent and the action in the full expert dataset for unseen states. It can be observed that the performance of the offline RL agent considerably drops. This is attributed to the critic overestimating the Q value of non-optimal actions for states that do not occur in the dataset while training. In numerous real-world applications, expert insights regarding the general behavior of a policy are often accessible [33] . For example, sales operators often distribute lower discount coupons to active users to maximize profit. While these insights may not be optimal, they serve as valuable guidelines for understanding the overall behavior of the policy. A rich literature in knowledge distillation [18] has shown that teacher networks trained on domain knowledge can transfer knowledge to another network unaware of it. This work aims to leverage a teacher network mimicking simple decision tree-based domain knowledge to help offline RL generalize in limited data settings.The paper makes the following novel contributions:• We introduce an algorithm dubbed ExID, leveraging intuitive human obtainable expert insights. The domain expertise is incorporated into a teacher policy, which improves offline RL in limited-data settings through regularization.• The teacher based on expected performance improvement of the offline policy during training, improving the teacher network beyond initial heuristics.• We demonstrate the effectiveness of our methodology on real sales promotion dataset, several discrete OpenAI gym and Minigrid environments with standard offline RL data sets and show that ExID significantly exceeds the performance when faced with limited data. 
related work: 
 This work improves offline RL learning on batches sampled from static datasets using domain expertise. One of the major concerns in offline RL is the erroneous extrapolation of OOD actions [13] . Two techniques have been studied in the literature to prevent such errors. 1) Constraining the policy to be close to the behavior policy 2) Penalizing overly optimistic Q values [24] . We discuss a few relevant algorithms following these principles. In Batch-Constrained deep Q-learning (BCQ) [13] candidate actions sampled from an adversarial generative model are considered, aiming to balance proximity to the batch while enhancing action diversity. Algorithms like Random Ensemble Mixture Model (REM) [2] , Ensemble-Diversified Actor-Critic (EDAC) [3] and Uncertainty Weighted Actor-Critic (UWAC) [42] penalize the Q value according to uncertainty by either using Q ensemble networks or directly weighting the loss with uncertainty. CQL [21] enforces regularization on Qfunctions by incorporating a term that reduces Q-values for OOD actions while increasing Q-values for actions within the expected distribution. However, these algorithms do not handle OOD actions for states not in the static dataset and can have errors induced by changes in transition probability.Integration of domain knowledge in offline RL, though an important avenue, has not yet been extensively explored. Domain knowledge incorporation has improved online RL with tight regret bounds [33, 4] . In offline RL, bootstrapping via blending heuristics computed using Monte-Carlo returns with rewards has shown to outperform SOTA algorithms by 9% [15] . Recent works improve offline RL by incorporating a safety expert [40] and preference query [44] , contrary to our work which improves imperfect domain knowledge. The closest to our work is Domain Knowledge guided Q learning (DKQ) [46] where domain knowledge is represented in terms of action importance and the Q value is weighted according to importance. However, obtaining action importance in practical scenarios is nontrivial.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The paper acknowledges the dependency on reasonable domain knowledge and coverage please refer to sec 6Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 A DRL setting is represented by a Markov Decision Process (MDP) formalized as (S, A, T, r, ρ 0 , γ).Here, S denotes the state space, A signifies the action space, T (s ′ |s, a) represents the transition probability distribution, r : S × A → R is the reward function, ρ 0 represents the initial state distribution, and γ ∈ (0, 1] is the discount factor. The primary objective of any DRL algorithm is to identify an optimal policy π(a|s) that maximizes E st,at [ ∞ t=0 γ t r(s t , a t )] where, s 0 ∼ d 0 (.), a t ∼ π(.|s t ), and s ′ ∼ T (.|s t , a t ). Deep Q networks (DQNs) [26] learn this objective by minimizing the Bellman resid-ual (Q θ (s, a) -B π θ Q θ (s, a)) 2 where B π θ Q θ (s, a) = E s ′ ∼T [r(s, a) + γE a ′ ∼π θ (.|s ′ ) [Q θ ′ (s ′ , a ′ )]].The policy π θ chooses actions that maximize the Q value max a ′ ∈A Q θ (s ′ , a ′ ). However, in offline RL where transitions are sampled from a pre-collected dataset B, the chosen action a ′ may exhibit a bias towards OOD actions with inaccurately high Q-values. To handle the erroneous propagation from OOD actions, CQL [22] learns conservative Q values by penalizing OOD actions. The CQL loss for discrete action space is given byEQUATIONEq. 1 encourages the policy to be close to the actions seen in the dataset. However, CQL works on the assumption of coherent batches, i.e., if (s, a, s ′ ) ∈ B, then s ′ ∈ B. There is no provision for handling OOD actions for s / ∈ B, which can lead to policy failure when data is limited. In the next sections, we present ExID, a domain knowledge-based approach to improve performance in data-scarce scenarios.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: Please refer to App A in the supplement material for theoretical analysis and proofs.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
problem setting and methodology: 
 In our problem setting, the RL agent learns the policy on a limited dataset with rare and unseen demonstrations. We define the characteristics of this dataset as follows:Definition 4.1. A reduced buffer B r is a proper subset of the full dataset B i.e., B r ⊂ B satisfying the following conditions:• Some states in B are not present in B r , i.e., ∃s′ ∈ B ∧ ∀(s, a, s ′ ) : (s, a, s ′ ) / ∈ B r• The number of samples N (s, a, s ′ ) for some transitions in B are less inB r i.e, ∃(s, a, s ′ ) ∈ B : N (s, a, s ′ ) Br < N (s, a, s ′ ) BWe observe, performing Q -Learning by sampling from a limited buffer B r may not converge to an optimal policy for the MDP M B representing the full buffer. This can be shown as a special case of (Theorem 1, [13] ) as p B (s ′ |s, a) ̸ = p Br (s ′ |s, a) and no Q updates for (s, a) / ∈ B r leading to sub-optimal policy. Please refer to the App. B for analysis and example.We also assume a set of common sense rules in the form of domain knowledge is available. Domain knowledge D is defined as hierarchical decision nodes capturing S → A as represented by Eq. 2.Each decision node T ηi is represented by a constraint ϕ ηi and Boolean indicator µ ηi function selects the branch to be traversed based on ϕ ηi . Action = a ηi if leaf µ ηi T ηi↙ (s) + (1 -µ ηi )T ηi↘ (s) o/w µ ηi (s) = 1 if s |= ϕ ηi 0 o/w (2)EQUATIONEq 3 incentivizes the critic to increase Q values for actions from π ω t and decreases Q values for other actions when argmax a (Q θ s (s, a)) ̸ = π ω t (s) for states that satisfy domain knowledge. Note that Eq 3 will only be 0 when argmax a (Q θ s (s, a)) = π ω t (s) for s |= D. It is also set to 0 for s ̸ |= D. However, since π ω t mimicking heuristic rules is sub-optimal, it is also important to incorporate learning from the data. The final loss is a combination of Eq. 1 and Eq. 3 with a mixing parameter λ ∈ [0, 1]:EQUATIONThe choice of λ and the warm start parameter k depends on the quality of D. In the case of perfect domain knowledge, λ would be set to 1, and setting λ to 0 would lead to the vanilla CQL loss. Mixing both the losses allows the critic to learn both from the data in B r and knowledge in D.Updating Teacher: Given a reasonable warm start, the critic is expected to give higher Q values for optimal actions for s ∈ D ∩ B r as it learns from data. We aim to leverage this knowledge to enhance the initial teacher policy π ω t trained on heuristic domain knowledge. For s ∼ B and s |= D, we calculate the average Q values over critic actions and teacher actions and check which one is higher in Algo 1 line 11 which refers to Cond. 6. For brevity E s∼Br∧s|=D is written as E. Carlo (MC) dropout, which involves including dropout before every layer of the critic network and using it during inference [14] . Following [42] , we measure the uncertainty of prediction using Eq 5.If E(Q θ s (s, a s )) > E(Q θ s (s,EQUATIONEq 5 estimates the variance of Q value Q(s, a) for an action a using T forward passes on the Q θ s (s, a)with dropout where Q(s, a) represents the predictive mean. We check the average uncertainty of the Q value for action chosen by the critic and teacher policy over the states that match domain knowledge in a batch. The teacher network is updated using the critic's action only when the policy expects a higher average Q return on its action and the average uncertainty of taking this action is lower than the teacher action. E(V ar T Q θ s (s r , a s )) < E(V ar T Q θ s (s r , a t )) indicates the actions were learned from the expert data in the buffer and are not OOD samples. The condition is summarized in cond. 6:EQUATIONWe update the teacher with cross-entropy described in Eq 7:EQUATIONwhere, π s (s, a) = e Q(s,a) a ′ Q(s,a ′ ) . When the critic's policy is better than the teacher's policy, L r (θ) is set to 0 Algo 1 Lines 11 to 13. Finally, the critic network is updated using calculated loss L(θ) Algo 1 Lines 17-18. We theoretically analyse the implications of using ExID in propositions 4.2 and 4.3.Proposition 4.2. Denote π as the policy learned by ExID, π u as any offline RL policy learned on B r and optimal Q function as Q * and V function as V * . Then it holds that η(π) -η(π u ) ≥ E s∼O|πu [V * (s) -Q * (s, π u (s))] -ρπ α Where α = E s∼O [V * (s) -Q * (s, π(s))], ρπ (s) = [ 1 |S π |(1-γ) , 1 1-γ ] (| S π |* (s, π ω t (s)) > Q * (s, π u (s)) for s ∈ D ∩ B r .Proof is deferred to App. A. In the next section, we discuss our empirical evaluations.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Yes all hyper-parameters and experimental setting have been clearly listed in the paper.Please refer to App F and sec 5.1.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
empirical evaluations: 
 We investigate the following through our empirical evaluations: 1. Does ExID perform better than combining D and offline RL algos on different environments with datasets exhibiting rare and OOD states Sec 5.2? 2. Does ExID generalize to OOD states covered by D Sec 5.4? 3. What is the effect of varying k, λ and updating π ω t Sec 5.5? 4. How does performance vary with the quality of D Sec 5.6?We evaluate our methodology on open-AI gym [5] , MiniGrid [6] and real sales promotion (SP) [30] offline data sets. All our data sets are generated using standard methodologies defined in [32, 31] except SP which is generated by human operators. All experiments have been conducted on a Ubuntu 22.04.2 LTS system with 1 NVIDIA K80 GPU, 4 CPUs, and 61GiB RAM. App. F notes the hyperparameter values and network architectures.Dataset: We experiment on three types of data sets. Expert Data-set [10, 16, 22] generated using an optimal policy without any exploration with high trajectory quality but low state action coverage.Replay Data-set [2, 13] generated from a policy while training it online, exhibiting a mixture of multiple behavioral policies with high trajectory quality and state action coverage. Noisy Data-set [12, 13, 22, 16] generated using an optimal policy that also selects random actions with ϵ greedy strategy where ϵ = 0.2 having low trajectory quality and high state action coverage. Additionally we also experiment on human generated dataset for sales promotion task.We do comparative studies on 10 baselines for OpenAI gym datasets. The first baseline simply checks the conditions of D and applies corresponding actions in execution. The performance of this baseline shows that D is imperfect and does not achieve the optimal reward. CQL SE is from [40] where the expert is replaced by D. The other baselines are an ensemble of D and eight algorithms popular in the Offline RL literature for discrete environments. These algorithms includeBehavior Cloning (BC) [29] , Behaviour Value Estimation (BVE) [16] , Quantile Regression DQN (QRDQN) [7] , REM, MCE, BCQ, CQL and Critic Regularized Regression Q-Learning (CRR) [41] .For a fair comparison, we use actions from domain knowledge for states not in the buffer and actions from the trained policy for other states to obtain the final reward. Hence, each algorithm is renamed with the suffix D in Table 5 .1.Limiting Data: To create limited-data settings for benchmark datasets, we first extract a small percentage of samples from the full dataset and remove some of the samples based on state conditions. This is done to ensure the reduced buffer satisfies the conditions defined in Def 4.1. We describe the specific conditions of removal in the next section. Further insights and the state visualizations for selected reduced datasets are in App H. Note : no data reduction has been performed on SP dataset to demonstrate a real dataset exhibits characteristics of reduced buffer.Our results for OpenAI gym environments are summarised in Table 5 .1 and Minigrid in Table 3 (App D). We observe the performance of offline RL algorithms degrades substantially when part of the data is not seen and trajectory ratios change. For these cases with only 10% partial data, ExID surpasses the performance by at least 27% in the presence of reasonable domain knowledge. The proposed method performs strongest on the replay dataset where the contribution of L r (θ) is significant due to state coverage, and the teacher learns from high-quality trajectories. Cart-pole Environment: For this environment, we use domain knowledge from [33] , which aims to move in the direction opposite to the lean of the pole, keeping the cart close enough to the center. If the cart is close to an edge, the domain knowledge attempts to account for the cart's velocity and recenter the cart. The full tree is given in the App. D Fig 10 (a) . We remove states with cart velocity > -1.5 to create the reduced buffer.Lunar-Lander Environment: We borrow the decision nodes from [34] and get actions from a sub-optimal policy trained online with an average reward of 52. SP dataset and environment [30] simulates a real-world sales promotion platform. The number of coupons and the discount the user received will affect his behavior. A higher discount will promote the sales, but the cost will also increase. The goal for the platform operator is to maximize the total profit. The horizon of the dataset is 50 days for the training and 30 days for the test. Domain knowledge ( [30] , App A] : Active users can be given more coupons with lower discount to maximize profit. We model this as order number > 60 ∧ Avg f ee > 0. without teacher updated, the sub-optimal teacher drags down the performance of the policy beyond the warm start, exhibiting the necessity of π ω t update. Also, the student converges to a sub-optimal policy if no L r (θ) is included beyond the warm start. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]Justification: The code is provided with the submission in a zip file with Readme for instructions.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion and limitation: 
 In this paper, we study the effect of limited and partial data on offline RL and observe that the performance of SOTA offline RL algorithms is sub-optimal in such settings. The paper proposes a methodology to handle offline RL's performance degradation using domain insights. We incorporate a regularization loss in the CQL training using a teacher policy and refine the initial teacher policy while training. We show that incorporating reasonable domain knowledge in offline RL enhances performance, achieving a performance close to full data. However, this method is limited by the quality of the domain knowledge and the overlap between domain knowledge states and reduced buffer data. The study is also limited to discrete domains. In the future, the authors would like to improve on capturing domain knowledge into the policy network without dependence on data and extending the methodology to algorithms that handle continuous action space.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification:The paper uses open source code to create the dataset and lists the modifications in details in the main paper and supplement material. Please refer to App F and sec 5.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
broader impact: 
 During the trial-and-error training phase, RL agents may exhibit irrational behavior, which can be risky and costly in real-world scenarios. As a more practical alternative to online RL, offline RL utilizes pre-existing collected data to eliminate the need for real-time interactions during training.However, a drawback of offline RL is its dependence on the quality and quantity of historical data, which, when sub-optimal, could adversely affect overall performance. Therefore, through this work, we use domain knowledge to suppress erroneous actions when available data is limited. However, this inclusion may facilitate harmful behavior in the presence of biased domain knowledge. Therefore, we advocate the use of well-regulated domain knowledge obtained from experts. Beyond this, we do not foresee any ethical impact on our work.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: All experiments have been run on 3 random seeds and the error bounds have been reported in Table 5 .1, Table 2 and Table 3 .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Please refer to the Experimental setup section in the main paper sec 5.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All codes and datasets used in this paper are under MIT licence and the original owners have been cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: No new assets have been introduced in this paper.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper did not require any crowdsourcing or human subject for experimentation.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? 
 
 Appendix: For any deterministic policy π the performance return is formulated asη(π) = Eτ∼π[ ∞ t=0 γ t r(st, at)]For any policy π, ρπ is the (unormalized) discounted visitation frequency given by ρπ(s) = ∞ t=0 γ t P (st = s)where s0 ∼ ρ 0 (s0) and the trajectory (s0, s1, . . . ) is sampled from the policy π and ρπ(s)∈ [0, 1 1-γ ]. ρπ(s) = sup{ρπ(s), s ∈ S} ∈ [ 1 |Sπ |(1-γ) , 1 (1-γ) ]We denote the regularized policy learned by ExID on Br as π and the unregularized policy as πu.We introduce the following Lemma required for our theoretical analysis.Lemma A.1. ( [44] ) Given two policies π1 and π2η(π1) -η(π2) = s∈S ρπ 1 (s)(Q * (s, π1(s) -V * (s))ds - s∈S ρπ 2 (s)(Q * (s, π2(s) -V * (s))dsProof. Please refer to Lemma A.1 Eq 17 in [44] Proposition A.2. (4.2) Denote π as the policy learned by ExID, πu as any offline RL policy learned on Br and optimal Q function as Q * and V function as V * . Then it holds thatη(π) -η(πu) ≥ E s∼O|πu [V * (s) -Q * (s, πu(s))] -ρπ αProof. According to [19] performance improvement between two policies if given byEQUATIONReplacing π1 by π and π2 by πu and by following Lemma A.1EQUATIONDividing the state space into in dataset domain states (I) and OOD states (O). The (11) s∈IEQUATION)Since the regularization loss facilitates visitation to OOD states via knowledge distillation we assume ρ π = ρπ u -∆i for s ∈ i and ρ π = ρπ u + ∆o for s ∈ o where ∆i ∈ [0, ρ πu(s) ] and ∆oEQUATIONEQUATIONUnder assumption in distribution action can be learned from the dataset due to conservatism of offline RLEQUATIONEQUATION)EQUATIONFurther loosening the lower boundEQUATIONEQUATIONCombining Eq 14, 17 and 19, andEQUATIONHence, Proposition 4.2 follows Q.E.D Proposition A.3. (4.3) Algo 1 reduces generalization error if Q * (s, π ω t (s)) > Q * (s, π(s)) for s ∈ D ∩ Br,where π is vanilla offline RL policy learnt on Br.Proof. Generalization error for any policy π as defined by [28] can be written as:EQUATIONHere, Eτ∼π represents sampling trajectories with policy π. Since the state space is continuous, we can represent the expectation as an integral over the state spaceEQUATION= -s∈S T t=0EQUATIONAnalysing with respect to s ∈ D ∩ Br we can break the integration into two partsEQUATION= -f (s|π) + s∈D T t=0EQUATIONFor a policy π learnt in Algo 1 the action for st = s ∈ D is regularized to be close to π ω t which either follows domain knowledge or expert demonstrations. Hence, it is reasonable to assumeEQUATIONNote for s / ∈ D, f (s|π) ≈ f (s|π). This is because the regularization term assigns max Q value to a different action for s ∈ D but maxa(Q(s, a)) remains sameEQUATIONHence, G π < Gπ Proposition 2 follows Q.E.DPerforming Q -Learning by sampling from a reduced batch Br may not converge to an optimal policy for the MDP MB representing the full buffer.Example (Theorem 1, [13] ) defines MDP MB of B from same state action space of the original MDP M with transition probabilities pB(s ′ |s, a) = N (s,a,s ′ ) s N (s,a,s) where N (s, a, s ′ ) is the number of times (s, a, s ′ ) occurs in B and an terminal state sinit. It states pB(sinit|s, a) = 1 when s N (s, a, s) = 0. This happens when transitions of some s ′ of (s, a, s ′ ) are missing from the buffer, which may occur in Br when Br ⊂ B. r(sinit, s, a) is initialized to Q(s, a). We assume that a policy learned on reduced dataset Br converges to optimal value function and disprove it using the following counterexample: Our methodology addresses these issues as follows:• For s ∈ D ∩ Br better actions are enforced through regularization using π ω t even when the transition probabilities are low for optimal transitions.• Incorporating regularization distills the teacher's knowledge in the critic-enhancing generalization.A visualization is shown in Fig 8 .The pseudo code of the algorithm is described in Algo 1.Algorithm 1 Pseudo code for EXID Update π ω t (s) using Eq 7 14:L r (θ) = 0 15:else 16:Calculate L r (θ) using Eq 3 17:end if18:Calculate L(θ) using Eq 419:Update Q θ s with L(θ) and softy update Q θ ′ s and τ 20: end for Minigrid Environments: Mini-grid [6] is an environment suite containing 2D grid-worlds with goal oriented tasks. As explained in the main text, we experiment using MiniGrid-LavaGapS7-v0 and MiniGrid-Dynamic-Obstacles-Random-6x6-v0 from this environment suite is shown in Fig 9 d ) and e). In MiniGrid-LavaGapS7-v0, the agent has to avoid Lava and pass through the gap to reach the goal. Dynamic obstacles are similar; however, the agent can start at a random position and has to avoid dynamically moving balls to reach the goal. The environment has image observation with 3 channels (OBJECT_ID, COLOR_ID, STATE). Following [31] experiments, we flatten the image to an array of 98 observations and restrict action space to three actions: Turn left, Turn Right, and Move forward. The results of minigrid environment are reported in Table 3 . Since this environment uses a semantic map from image observation, we collect states from a fixed policy with random actions to generate the teacher's state distribution. CQL on the full dataset achieves the average reward of Knowledge distillation is a well-embraced technique of incorporating additional information in neural networks and has been applied to various fields like computer vision [43, 36] , natural language processing [8, 38] , and recommendation systems [37] . [17] introduced the concept of distilling knowledge from a complex, pre-trained model (teacher) into a smaller model (student). In recent years, researchers have explored the integration of rule-based regularization techniques within the context of knowledge distillation. Rule regularization introduces additional constraints based on predefined rules, guiding the learning process of the student model [18, 45] . These techniques have shown to reduce overfitting and enhance generalization [38] . Knowledge distillation is also prevalent in the field of RL [47] and offline RL [39] . Contrary to prevalent teacher-student knowledge distillation techniques, our work does not enforce parameter sharing among the networks. Through experiments, we demonstrate that a simple regularization loss and expected performance-based updates can improve generalization to unobserved states covered by domain knowledge. There are also no constraints on keeping the same network structure for the teacher, paving ways for capturing the domain knowledge into more structured networks such as Differentiable Decision Trees (DDTs).We follow the network architecture and hyper-parameters proposed by [31] for all our networks, including the baseline networks. The teacher BC network π t ω and Critic network Q θ s (s, a) consists of 3 linear layers, each having a hidden size of 256 neurons. The number of input and output neurons depends on the environment's state and action size. All layers except the last are SELU activation functions; the final layer uses linear activation.π t ω uses a softmax activation function in the last layer for producing action probabilities. A learning rate of 0.0001 with batch size 32 and α = 0.1 is used for all environments. MC dropout probability of 0.5 and number of stochastic passes T=10 have been used for the critic network. The uncertainty check is performed every 15 episodes after the warm start to avoid computational overhead. The hyper-parameters specific to our algorithm for OpenAI gym are reported in Table F . The hyper-parameters specific to our algorithm for Minigrid environments are reported in Table 5 . We empirically evaluate the effect of λ In In this section, we discuss the intuition behind our data-limiting choices. We also visually represent selected reduced datasets for the OpenAI gym environments. However, this drop is not substantial. The performance also reduces on removing part of the dataset from the full dataset with states > -0.8. However, the worst performance is observed when both samples are reduced and data is omitted, attributing to accumulated errors from probability ratio shift contributing to an increase in generalization error. Our methodology aims to address this gap in performance.Removing part of the state space: Due to the simplicity of the Mountain-Car environment, we analyze the Mountain-Car expert dataset to show the effect of removing data matching state conditions of the different nodes in the decision tree in Fig 10 (c ). The performance for each condition is summarised in Table 6 . The most informative node in the tree is position > -0.5; removing states matching this condition causes a performance drop in the algorithm as the domain knowledge regularization does not contribute significant information to the policy. Similarly, removing data with velocity < 0.01 causes a performance drop. However, both performances are higher than the baseline CQL trained on reduced data. Based on this observation, we choose state removal conditions that preserve states matching part of the information in the tree such that the regularization term contributes substantially to the policy. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute. 


Decision: Reviewers agreed that this paper should be accepted, and praise the approach as simple, reasonable, and innovative.  However, I noticed that the paper does not mention model-based offline RL or include any model-based offline RL approaches as baselines.  I consider this a critically important ommission, since:
* The current related work mischaracterizes the field of offline RL, stating "Two techniques have been studied in the literature to prevent such errors. 1) Constraining the policy to be close to the behavior policy 2) Penalizing overly optimistic Q values [24]."  However, model-based approaches are distinct from these, as noted in [24]: "Conceptually, these methods resemble the conservative estimation approach described in Section 4.5, but instead of regularizing a value function, they modify the MDP model learned from data to induce conservative behavior."
* Model-based offline RL methods such as MOPO and MoReL were found to outperform approaches (1) and (2) over 4 years ago.  The papers introducing those methods were published at NeurIPS 2020 and are more highly cited than most of the works the authors reference in their related work section.  
* Concurrently with MOPO and MoReL, in "The Importance of Pessimism in Fixed-Dataset Policy Optimization", Buckman et al. noted theoretical reasons why existing approaches (with the exception of the model-based approaches) were fundamentally limited.

This makes me seriously question the strength of the baselines that are used in this work and the authors situation of their work within the literature, and reviewers were unable to reassure me on these points.  Thus I am unable to recommend acceptance at this point.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel technique ExID, a domain knowledge-based regularization method, that adaptively refines initial domain knowledge to boost performance of offline reinforcement learning (RL) in limited-data scenarios. The key insight is leveraging a teacher policy, trained with domain knowledge, to guide the learning process of the offline-optimized RL agent (student policy). This mitigates the issue of erroneous actions in sparse samples and unobserved states by having the domain knowledge-induced teacher network to cover them. And the initial domain knowledge would be improved when the student policy reaches a better perform than the teacher policy.  Empirical evaluations on standard discrete environment datasets demonstrate a substantial average performance increase compared to traditional offline RL algorithms operating on limited data
strength_weakness: strength_weakness: 1. Generalization to Continuous Domains: The paper is limited to discrete action spaces, which restricts its applicability to a broader range of RL problems involving continuous action spaces. This limitation is acknowledged by the authors.


2. Scalability: The scalability of ExID to more complex environments that requires a complex representation (e.g., a significant large tree) of domain knowledge is not thoroughly explored.  It would be beneficial to understand how the method performs in such settings and what challenges might arise because the challenging of updating the domain knowledge represented in a complex representation could hinder the learning process of the student policy in the proposed method ExID.
questions: questions: 1. Comparative Analysis: Can the authors provide a more detailed comparison with other domain knowledge-based methods? Specifically, how does ExID differ in its approach to leveraging domain knowledge compared to methods like DKQ and CQL SE that 
 are mentioned in Related Work?

2. Examples of Domain Knowledge Improvement: having some examples of the improved domain knowledge could make the paper stronger and more convincing.
limitations: limitations: The authors acknowledge several limitations of their work, including the reliance on the quality of domain knowledge and the focus on discrete action spaces. While these limitations are well-addressed in the paper, it may be worth to consider a broad evaluation:
   * Conducting experiments on a wider variety of environments that have larger state and action spaces, would provide a more comprehensive evaluation of the method's applicability.
paper_summary: paper_summary: This paper studies offline RL when data is limited. The authors propose a domain knowledge-based regularization technique to learn from an initial tracker network and limited data buffer. The experiments verified the effectiveness of the proposal, which outperforms the classic RL baseline methods.
strength_weakness: strength_weakness: 1. The technical novelty is limited. Despite the claimed use of expert knowledge, the method adopted by the paper is to directly train a policy from the knowledge, which assumes that the information provided by the domain knowledge is at the state-action level (a decision tree in this paper), which limits the feasibility of this method. Compared to the use of knowledge between latent concepts discussed in neuro-symbolic learning, I think it's more like traditional model distillation. 
2. In practice, limited offline data may come from domain knowledge-based strategies, such as human-designed rules, thus I have great concerns about whether these two can promote each other. Empirical studies on more real-world datasets or rigorous theoretical analysis will provide support to this issue and further improve this work.  
3. The introduction uses the sales task as an example, but the visualization is based on the Mountain Car dataset.
4. Definition 4.1 seems strange, why not directly define the offline dataset as a subset of the complete state spaces?
5. The $\eta$ in Proposition 4.2 is not well defined.
questions: questions: 1. Could the “Intuitive Expert Insights” be formally defined?
2. The difference between the proposal and the traditional model distillation.


---

After rebuttal, my concerns have been addressed. I decide to raise the score to 6.
limitations: limitations: The authors have provided a discussion about the limitations.
paper_summary: paper_summary: The paper introduces ExID, an offline reinforcement learning algorithm that enhances learning performance in limited data scenarios by combining domain knowledge in the form of simple decision trees with agent experience replay data.
strength_weakness: strength_weakness: * Discrete Action Space Limitation: The algorithm is currently limited to discrete action spaces, necessitating future extensions for continuous action domains.
* Hyperparameter Tuning Challenge: The need for precise hyperparameter tuning complicates the deployment of ExID in scenarios where extensive optimization is impractical.
* The paper does not have enough strong experiment comparisions. The methods of the paper is related with offline RL methods, such as SCQ[1], ReDS[2], A2PR[3], CPED[4]. But it lacks the experiments comparisions with offlien RL methods. I think adding some SOTA baseline methods will improve your paper. It is not required that experimental comparisons must be given, but at least add some discussion with these methods to the paper.  

References：

[1] Shimizu, Yutaka, et al. "Strategically Conservative Q-Learning." arXiv preprint arXiv:2406.04534 (2024).

[2] Singh, Anikait, et al. "ReDS: offline reinforcement learning with heteroskedastic datasets via support constraints." Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023.

[3] Liu, Tenglong, et al. "Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning." In International Conference on Machine Learning (ICML). PMLR, 2024.

[4] Zhang, Jing, et al. "Constrained policy optimization with explicit behavior density for offline reinforcement learning." Advances in Neural Information Processing Systems. 2023
questions: questions: * How does ExID perform when the true optimal policy deviates significantly from the provided domain knowledge? Can the algorithm recover from incorrect domain knowledge without additional corrective mechanisms?
* What measures are in place to ensure robustness when the domain knowledge contains errors or biases? How sensitive is the algorithm to such inaccuracies?
* The paper mentions a real sales promotion dataset, but how does ExID perform in other real-world scenarios? Are there any plans for more extensive real-world testing to validate the algorithm's practical utility?
* How should practitioners select the hyperparameters $\lambda$ and $k$ in the absence of extensive computational resources?
limitations: limitations: * The paper only conducts experiments in several simulated environments and a real-world sales promotion dataset, which may not fully verify the effectiveness and applicability of the algorithm in more diverse and complex real-world scenarios.
* The performance of the ExID algorithm heavily relies on the quality of the domain knowledge. If the domain knowledge is incomplete, inaccurate, or biased, it may mislead the learning process and result in suboptimal policies. Moreover, obtaining high-quality domain knowledge can be challenging and time-consuming in practice.
* The proposed method mainly concentrates on discrete action spaces, and its performance and applicability in continuous action spaces are not clear. This limits the algorithm's utility in many real-world control tasks that involve continuous action spaces.

Comments: title: Thank you for acknowledging the rebuttal
Please confirm you've read the author response
Thank you for your feedback
Thank you for discussion
Official Comment by Reviewer UWfE
Thank you for acknowledging the rebuttal
Additional baselines
Official Comment by Reviewer UWfE
comment: Dear reviewer NyjZ,

Thanks for your kind support and for helping us improve the paper! We will incorporate the additional discussions in the revised version of the manuscript.

Thank you again for your valuable comments and guidance.

Best,

Authors
Thanks for your response which addressed my concerns. I will raise my score to 6.
Dear reviewer, 
Can you please confirm that you've read the author's responses?
Given that the other review is now both vote for acceptance, it's important for you to voice any remaining concerns if you still believe the paper should not be accepted. 

Thank you!
Dear reviewer NyjZ,

Thank you for your insightful review comments and for asking important questions that helped us improve the work. As the discussion period approaches towards end, kindly let us know if we have addressed your concerns and if there are feedbacks for discussion. Otherwise we will be grateful if you could reconsider your score.

Thank you again for your efforts and we value your feedback deeply.

Regards,
Authors
Dear Reviewer UWfE,

Thanks for you for carefully reading the response and for helping us improve the paper! We will incorporate the additional comparisons in the revised version of the manuscript.

Thank you again for your valuable comments and guidance.

Best,

Authors
Thanks for your careful discussion, I have improved the score from 4 to 6.
Dear Reviewer Cy8u,

Thanks for your kind support and for helping us improve the paper! We will incorporate the additional discussions in the revised version of the manuscript.

Thank you again for your valuable comments and guidance.

Best,

Authors
We thank the reviewer for acknowledging the rebuttal and for raising important questions that strengthen our paper. Please find the below comparison with mentioned offline RL methods:

**ReDS: offline reinforcement learning with heteroskedastic datasets via support constraints**

The main contribution of RedS is to provided distribution constraints against a reweighted version of the behavior policy. This facilitates the policy to freely choose state-by-state how much the learned policy should stay close to the behavior policy for states seen in the the dataset.  This method is applicable to the datasets showing heteroskedastic distributions (different behavior actions for the same states) whereas our methodology employs action correction for  OOD states that are not observed in the dataset. Since our dataset is not heteroskedastic this methodology is not directly comparable with ExID.

**Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning**

A2PR trains a VAE similar to SCQ to identify high advantage that differ from those present in the dataset. The VAE is trained with $\log p{\psi}(a|s) \geq \mathbb{E}{q_{\phi}(z|a,s)} \left[ \mathbb{1}\{f(A(s,a)) > \epsilon_A\} \log p_{\psi}(a|z,s) \right] - \text{KL}\left[q{\phi}(z|a,s) \parallel p(z|s)\right]$ where $s \in \mathcal{B}r$. This method does not estimate actions for $s \notin {B}r$ which ExID does via knowledge distillation.


**Constrained policy optimization with explicit behavior density for offline reinforcement learning**

CPED uses a flow-GAN model to explicitly estimate the density of behavior policy. This facilitates choosing different actions which are safe for the for states in dataset. The flow GAN model is trained on the dataset generated by behavior policy and does not account for the states outside the dataset.

Please find the comparison of the baselines with ExID in the table below:

| Environment | CQL+$\mathcal{D}$ | SCQ | A2PR | CPED | EXID |
|-------------|------------------|-----|------|------|------|
| SP          | 679.25$\pm$ 35.02 | 708.44 $\pm$ 52.19 | 712 $\pm$ 32.09 | 715 $\pm$ 47.31 | 827.76 $\pm$ 43.79 |


In summary all these methods do not employ any action correction mechanism for OOD states outside the dataset leading to performance degradation in case of limited data. As a result these algorithms perform almost similarly on sales promotion dataset. ExID distills knowledge for OOD states from domain knowledge leading to performance enhancement over the baseline methods. We add this discussion to the revised manuscript.

 If there are any further questions we will be happy to answer them. Otherwise we will be grateful if you could reconsider your score.
Thanks for your clear explanation and careful answers, which have addressed my concern. I see you add the comparison experiments with SCQ. I expect you to add more comparison experiments or discussions with other SOTA offline RL methods ReDS, A2PR, and CPED as mentioned earlier in your paper. Then, I am willing to raise my score.
I thank the authors for the detailed response. My concerns have been addressed. And I would like to remain my score.