ExID: Offline RL with Intuitive Expert Insights in Limited-Data Settings

Authors: Briti Gangopadhyay, Zhao Wang, Jia-Fong Yeh, Shingo Takamatsu
Primary area: reinforcement learning
Keywords: Offline Reinforcement Learning, Knowledge distillation

Abstract:
With the ability to learn from static datasets, Offline Reinforcement Learning (RL) emerges as a compelling avenue for real-world applications. However, state-of-the-art offline RL algorithms perform sub-optimally when confronted with limited data confined to specific regions within the state space. The performance degradation is attributed to the inability of offline RL algorithms to learn appropriate actions for rare or unseen observations. This paper proposes a novel domain knowledge-based regularization technique and adaptively refines the initial domain knowledge to considerably boost performance in limited data with partially omitted states. The key insight is that the regularization term mitigates erroneous actions for sparse samples and unobserved states covered by domain knowledge. Empirical evaluations on standard offline RL datasets demonstrate a substantial average performance increase compared to ensemble of domain knowledge and existing offline RL algorithms operating on limited data.

Decision: reject
Reviewer scores: 3, 5, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1729847916724

thread_type:
official_review

<thread_object>
creator: reviewer_frm3

<content>
summary:
This paper proposes an offline RL algorithm, ExID, that can leverage domain knowledge-based heuristic rules to enhance small-sample performance. The proposed algorithm can be seen as combining conservative Q-learning (CQL) and Uncertainty Weighted Actor-Critic (UWAC), and adds a CQL-style domain knowledge-based regularize, i.e., push up Q-values for actions that comply with the domain knowledge and push down Q-values for policy-generated actions. For detailed, comments please refer to strengths and weaknesses.

strengths:
- Leveraging domain knowledge to enhance sample efficiency in offline RL is meaningful.
- The idea of modeling domain knowledge as tree-based heuristic rules and incorporating them using CQL-style value regularization is interesting. 
- The paper is easy to read. The results show good performance for tasks that have domain knowledge-based rules.

weaknesses:
- The literature review is insufficient, missing many recent offline RL works that focus on sample efficiency and OOD generalization.
- The proposed method has very stringent requirements for the domain knowledge input. Although we can design domain knowledge trees for simple tasks, it is not practical for complex tasks. Typically, in many real-world tasks, we might only be able to write very few heuristic rules for a task, which can only sparsely cover the state-action space. This severely limits the applicability of the proposed method for a wide range of problems. In the experiments, the authors are also only able to evaluate their method in very simple task environments. 
- When training the teacher policy $\pi_t^w$, a random action is used if a state is not covered by the domain knowledge rules. This can be very problematic, as it will inevitably lead to sub-optimality when regularizing the learned policy. Moreover, if only a small set of domain knowledge rules is given (common in most practical problems), then it will cause the teacher policy to learn lots of random behavior, which could damage policy learning.
- The algorithm is essentially a combination of CQL and UWAC, with an additional value regularization that penalizes the value for actions that are not consistent with the domain knowledge. Since CQL is already very conservative and does not generalize well. Adding additional conservative regularization will further distort the value function, which could impact the optimality of the learned value.
- Propositions 4.2 and 4.3 are meaningless and have huge theory-practice gaps. The proposed method combines both CQL and the uncertainty-based method, which makes it almost impossible to draw any reliable theoretical conclusion on the learned value function and the policy. In the proof of Proposition 4.2, the underlying analytical tools used work for any policy, the only thing related to the proposed method is the **assumptions** $\rho_{\hat{\pi}}=\rho_{\pi_{u}}-\Delta_i$ and $\rho_{\hat{\pi}}=\rho_{\pi_{u}}-\Delta_o$, with $\Delta_i$ and $\Delta_o$ assumed to be positive, as well as $Q^*(s,\hat{\pi}(s))-Q^*(s,\pi_u(s)) \approx 0$ for in distribution actions. First, these assumptions have no guarantees from your algorithm. Second, to prove something, you simply cannot introduce some arbitrary assumptions to achieve your purpose. Proposition 4.3 is also trivial and not very meaningful. It only relates to the optimal value function $Q^*$, rather than the learned value function $Q^{\theta}$. You can only learn $Q^{\theta}$, and the proposition provides no information to tell if the teacher policy indeed helps to improve the learned $Q^{\theta}$.
- There are also some inconsistencies in the symbols used in the paper. For example, the teacher policy is expressed as $\pi_t^w$ in the text, but written as $\pi_w$ in Figure 2. The action value function is written as $Q_s^{\theta}$ in the text, but written as $Q_{\theta}$ in Figure 2.

questions:
- How can the method be used if only limited domain knowledge rules are provided?
- Is it possible to not use random action for cases that are uncovered by domain knowledge rules?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730227306912

thread_type:
official_review

<thread_object>
creator: reviewer_n2mw

<content>
summary:
This paper work on incorporating domain knowledge into offline RL to improve OOD performance.
The method appears to be based on DQN, but incorporates a "teacher" policy network which is used to regularize the argmax Q-function policy.

strengths:
The use of domain knowledge in RL is worthy of study.

The results seem to indicate some empirical advantage of the method over baselines (although not consistently so).

weaknesses:
**Unconvincing motivation and contributions**
This seems to be an avenue of work that has received very limited attention, and I would appreciate a more thorough motivation for both the problem setting and the approach proposed.

The environments studied seem pretty simple, and I'm not convinced by the baselines.  
I'm skeptical that this approach can scale, as it is based on strong domain knowledge. 
The practical utility on simple environments is also unclear -- can the authors make a more clear and compelling case of when and why they expect this approach to yield practical value?
I would also expect more discussion of how well this approach can scale to more complicated environments, and what sort of future work might help answer that question.

**Other presentation issues**
- I found the presentation of the algorithm confusing and presentation to be weak overall.
- should use citep
- Definition 4.1 is confusing.
-- I think the notation in the first condition (bullet point) is used incorrectly.
-- The 2nd bullet point seems redundant (being guaranteed to hold if the first condition holds).


**Inadequate treatment of model-based RL**
The work needs a more thorough comparison with model-based offline RL methods.
Currently, there is only one experiment reported (in insufficient detail) in the appendix.
I would expect these baselines to be run for all of the environments and reported in main text.
As such, I'm not convinced the authors have made a fair effort to tune and compare their methods with baselines. 
Furthermore, the point of comparison is MOPO, a method from 2020 -- are there more modern methods which ought to be considered?

The submission also states that for such methods "performance highly depends on the accuracy of the learned dynamics."  But this claim is not supported.  These methods are meant to keep policies from seeking out states where the learning dynamics are inaccurate, and thus, we might expect performance to degrade gracefully in the presence of inaccurate dynamics.  On what basis are the authors claiming otherwise?

questions:
Are there more modern offline RL methods which ought to be considered?

Can your approach be viewed as a form of actor-critic?  Would an actor-critic method be more performant?

What are some alternative methods for 

Does D specify a complete policy (I.e. does it have support on the entire state space)?  If so, I think the notation should be pi_D.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730518147767

thread_type:
official_review

<thread_object>
creator: reviewer_341n

<content>
summary:
In this work, the authors propose a new transfer offline RL method ExID, which transfer the learned policy of the source domain into the target domain with a expert system and limit data of target domain.

strengths:
- This work forces on a new and important setting. This setting is make sence because the less the data is, the more the risk is.
- This work proposes a simple but effect architecture, which update the target network and the teacher at the same time. This architecture let the method can be used with expert systems with different qualities.
- The final result is good especially when the target data is noisy.

weaknesses:
- The presentation of this work is limit. For example, in Section 3, there is "(s,a,s')\in B, then s'\in B". What exactly the structure of the buffer is? Also, the Eq. 2 is hard to understand at the first time. 
- The environments used in this work are all simple and visible. Can we propose a useful expert system for more complex environments, such as halfcheetah?

questions:
- This work leverages a uncertainty based method to update the teacher network. Is this a stable standard? How many times the teacher network is updated in the learning process?
- How many feasible s of  Proposition 4.3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652786291

thread_type:
official_review

<thread_object>
creator: reviewer_mstz

<content>
summary:
This paper presents ExID, an offline RL method that enhances policy learning in limited-data settings by using a domain-knowledge-based regularization technique, significantly improving performance over traditional offline RL approaches across diverse datasets.

strengths:
1. Innovative to incorporate the domain knowledge. 
2. real-world application in sales promotion dataset and simglucose dataset.

weaknesses:
1. Training the teacher network highly requires domain knowledge. 
2. The method is heuristic and plug-and-play in different algorithms. 
3. The simulation tasks are simple.

questions:
1. Can this method be applied to a continuous environment? If yes, should the authors try difficult simulated tasks, such as d4rl? 
2. Can this method be applied to different base algorithms, e.g., BEAR, IQL.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684732286

thread_type:
official_review

<thread_object>
creator: reviewer_dypb

<content>
summary:
This paper aims to incorporate the domain knowledge in the sparse states where the offline data is not explored much. 
Firstly, the teacher policy $\pi_t^w$ is trained by using the domain knowledge in the heuristic manner. And then, the offline policy is trained using a proposed critic-regularization term to enhance performance.

strengths:
- The novel concept: It is not easy to incorporate some general domain knowledge to train the policy. They try to put somehow symbolic control in the discrete action offline RL.
Method

- The critic regularization term: To use domain knowledge in the offline RL setting, the proposed regularization term is intuitive and proper.

weaknesses:
- The method ‘Training Teacher’ is somehow naive, and Eq. (2) is not clear. (Actually, I don’t understance what it is)

- Limited domain: It is not applicable in the continuous action space.

- The hyperparameter $\lambda$, the coefficient constant for critic regularization. There is no exact way to decide the proper value of $\lambda$.

- Updating the Teacher policy: The explanation for updating the teacher policy, especially after line 226, becomes difficult to follow.

- Writing should be enhanced.

questions:
In the regularization term, are there any gradient stop or .detach() (in torch) in Q-network? 
Line 128, The definition of target network parameter $\theta'$ is missing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731664430502

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of rebuttal

comment:
We thank the reviewers for their time and constructive feedback and for highlighting the following strengths.

Strengths:

- The novel concept, intuitive and proper (Reviewer Dypb),
- use of domain knowledge, empirical advantage (Reviewer N2mw, frm3),
- important setting, simple architecture (Reviewer 341n)
- innovative incorporation of domain knowledge, validation on real dataset (Reviewer msTz)
- Easy to read (reviewer frm3)

We acknowledge the areas for improvement suggested by the reviewers and have made concerted efforts to address them. Below, we highlight our responses to the major concerns, followed by detailed responses to individual reviews.

**Simple environment (Reviewer 341n, msTz, N2mw)**

- We want to emphasize that our proposed method was validated on two continuous-domain sales datasets: **Sales Promotion** (business domain) and **Simglucose** (healthcare). These tasks are inherently complex and reflect the intricacies of real-world problems. For a detailed description of these tasks, please refer to [1] and [2].

- While we acknowledge that our methodology relies on domain knowledge—which may be challenging to design for high-dimensional robotics tasks such as those in d4rl (e.g., HalfCheetah)—domain knowledge is often more accessible in business and healthcare contexts. We believe that this type of problem setup deserves research attention, as offline RL is not solely intended for robotics. Its application to real-world business domains is a promising and motivating area of study.

[1] **Simglucose**: Jinyu Xie. Simglucose v0.2.1 (2018) [Online]. Available: [https://github.com/jxx123/simglucose](https://github.com/jxx123/simglucose).  

[2] Qin, Rong-Jun, et al. “NeoRL: A near real-world benchmark for offline reinforcement learning.” _Advances in Neural Information Processing Systems_ 35 (2022): 24753-24765.

**Presentation and other issues (341n, Dypb, N2mw, frm3)**

We revised the manuscript as follows:

- Provided a clearer explanation of Eq 2 as follows: 
Each decision node $T_{\eta_i}$ has constraint $\phi_{\eta_i}$ that determines its branching, a Boolean indicator $\mu_{\eta_i}$ selects the branch $(\swarrow or \searrow)$ to follow based on whether the constraint $\phi_{\eta_i}$ is satisfied.
- Explained the conditions of Def 4.1 in line 155 as follows:
The two conditions denote 1. some states in $\mathcal{B}$ do not appear in $\mathcal{B}_r$ 2. for certain transitions $\mathcal{B}_r$ contains fewer transition than $\mathcal{B}$.
-  used citep.
- Corrected symbolic inconsistencies in Fig 2.
- Add discussion to how the method is extended to continuous domain as follows:
Furthermore, we extend this to continuous domain by using the regularization in Eq 4 during critic $(Q_s^\theta)$ training for continuous domain and using actions from actor network $(\pi_s)$ for cross entropy loss in Eq 7. (line 273-275).
- Restructured updating teacher section.
- Removed proposition 4.3 due to limited value and moved proposition 4.2 to the appendix stating simplified assumptions.
</content>
</thread_object>

</thread>


Metareview: