Test-time Adapted Reinforcement Learning with Action Entropy Regularization
Authors: Shoukai Xu
ZihaoLian
Mingkui Tan
Liu Liu
Zhong Zhang
Peilin Zhao
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement Learning
offline
test-time adaptation
Primary area: reinforcement_learning->batchoffline

Abstract: Offline reinforcement learning is widely applied in multiple fields due to its advantages in efficiency and risk control. However, a major problem it faces is the distribution shift between offline datasets and online environments. This mismatch leads to out-of-distribution (OOD) state-action pairs that fall outside the scope of the training data. Therefore, existing conservative training policies may not provide reliable decisions when the test environment deviates greatly from the offline dataset. In this paper, we propose Test-time Adapted Reinforcement Learning (TARL) to address this problem. TARL constructs unsupervised test-time optimization objectives for discrete and continuous control tasks, using test data without depending on environmental rewards. In discrete control tasks, it minimizes the entropy of predicted action probabilities to decrease uncertainty and avoid OOD state-action pairs. For continuous control tasks, it represents and minimizes action uncertainty based on the normal distribution of policy network outputs. Moreover, to prevent model bias caused by overfitting and error accumulation during the test-time update process, TARL enforces a KL divergence constraint between the fine-tuned policy and the original policy. For efficiency, TARL only updates the layer normalization layer parameters during testing. Extensive experiments on popular Atari game benchmarks and the D4RL dataset demonstrate the superiority of our method. Our method achieved a significant improvement over CQL, with a 13.6% episode return relative increase on the hopper-expert-v2 task.

Test-Time/At Adapted Reinforcement Learning with Action Entropy Regularization
Shoukai Xu Zihao Lian * Mingkui Tan 134
Liu Liu?
Zhong Zhang²
Peilin Zhao
Abstract
1. Introduction
Offline reinforcement learning is widely applied in multiple fields due to its advantages in efîi ciency and risk control. However, a major prob- lem it faces is the distribution shift between offline datasets and online environments. This mismatch leads to
Reinforcement learning is an important research field in artificial intelligence and has achieved remarkable success. Offline reinforcement learning (Kumar al., 2020; An et al., 2021; Fujimoto & Gu, 2021; Kostrikov et al., 2021) offers significant advantages, enables efficient data utilization and low risks by learning promising policies from pre-collected datasets. Therefore, offline reinforcement learning has be- come an indispensable method for a wide range applica- tions. For example, offline reinforcement learning can learn an optimal driving policy from a large driving dataset to improve the safety and stability of autonomous driving (Yu et al., 2020a; Liu et al., 2023). Furthermore, offline re- inforcement learning can also be applied to robotics con- trol (Kalashnikov et al., 2018), game AI (AlphaStar; Ye et al., 2020), portfolio selection (Zhang et al., 2023; Xu et al., 2020), recommendation systems (Li et al., 2010; Gao et al., 2022) and other fields, where can improve the intel- ligence of robots agents by learning from large amounts of offline data.
(OOD) state-action pairs that fall outside the scope of the training data. Therefore, existing conservative training policies may not provide reliable decisions when the test environment deviates greatly from the offline dataset. In this paper, we propose Test- time Adapted Reinforcement Learning (TARL) to address this problem. TARL constructs un- supervised test-time optimization objectives for discrete and continuous control tasks, using test data without depending environmental rewards. In discrete control tasks, it minimizes the entropy of predicted action probabilities to decrease un- certainty and avoid OOD state-action pairs. For continuous control tasks, it represents and min- imizes action uncertainty based on the normal distribution of policy network outputs. Moreover, to prevent model bias caused by overfitting and error accumulation during the test-time update process, TARL enforces a KL divergence con- straint between the fine-tuned policy and the orig- inal policy. For efficiency, TARL only updates the layer normalization layer parameters during testing. Extensive experiments popular Atari game benchmarks and the D4RL dataset demon- strate the superiority of our method. Our method achieved a significant improvement over CQL, with a 13.6% episode return relative increase on the hopper-expert--2 task.
The dataset used in offline reinforcement learning can be col- lected through interaction with the environment based on a fixed policy, simulation runs or human demonstrations. The state-action pairs in the offline dataset are unable to cover all the possibilities of the state-action space. Consequently, the dataset inevitably exhibits a certain degree of distri- bution shift from the true state-action visitation frequency associated with the learned policy, as shown in Figure 1. During the deployment process, the policy may encounter novel state-action pairs, which subsequently leads to the predicament of being incapable of coping with unforeseen changes and dynamic environments practical applications. Specifically, offline reinforcement learning may face out- of-distribution (OOD) samples, i.e., samples not present in the dataset, leading to inaccurate value function estimation for unseen state-action pairs and resulting in significant ex- trapolation error (Fujimoto et al., 2019). For example, in an autonomous driving application, the vehicle controlled by such a policy may fail to respond appropriately to a rare combination of road conditions, endangering the safety of passengers and other road users.
"Equal contribution. This work is done when Zihao Lian works as an intern Tencent AI Lab. 'South China Univer- sity of Technology "Tencent AI Lab 'Pazhou Laboratory "Key Laboratory of Big Data and Intelligent Robot, Ministry of Educa- tion 'Shanghai Jiao Tong University. Correspondence Peilin Zhao <masonzhao wewpe/aaaaaeennttccooo.m tencent.com>> Mingkui Tan <mingkui- tan @ scut.edu.cn>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To address this issue, existing methods can be broadly classi- fied into two categories: conservative estimation (Wu et al.,

---

Test-Time Adapted Reinforcement Learning with Action Entropy Regularization
2019; Kumar al., 2020; Fujimoto& & Gu, 2021; Kostrikov al., 2021; Yu et al., 2021; Lyu al., 2022) and uncertainty estimation (Yu et al., 2020b; Wu et al., 2021; An et al., 2021). Conservative estimation methods enforce consistency be- tween the behavioral policy and the learned policy by adding a KL divergence constraint or a pessimistic penalty to the learned value function. This encourages the learned policy to maintain pessimistic estimate for out-of-distribution state-action pairs and avoid them in online environments. However, conservative estimation methods tend to be overly conservative by avoiding unknown state-action pairs and their performance is severely restricted by the quality of the offline dataset. Uncertainty estimation methods measure the uncertainty of state-action pairs using an uncertainty metric and then correspondingly adjust the value function estimation. This balances the risk and reward of unseen state-action pairs. However, uncertainty estimation methods face risks of inaccurate uncertainty quantification and poor generalization when dealing with unseen state-action pairs, leading to potential negative consequences in safety-critical applications such as autonomous driving. Both conservative estimation and uncertainty estimation methods remain con- strained by the closed and static offline dataset, and thus are incapable of effectively adapting to the environment.
Othe
Oel ea husdeyy
0.5-
0.5
0008
Action Range
Action Range
(e) State-action Pairs Distribution Shift
(b) State-action Pairs Distribution Shift Continuous Control
Discrete Control Tasks
Figure I. State-action pairs distribution shift. The horizontal axis represents the action value, the vertical axis denotes the action probabilities. The red color represents the actions in the offline data, and the blue color represents the actions in the online envi- ronment. The offline dataset and the dynamic online environment have a significant data distribution mismatch, called the out-of- distribution (OOD) problem. This problem hinders the application of reinforeement learning algorithms in real- world scenarios due to suboptimal policy performance.
We propose novel Test-time Adapted Reinforcement Learning method, referred to as TARL. This simple yet effective paradigm enables offline reinforcement learning methods to adapt to the real-world environ- ment, with only a few parameters in normalization layers requiring updates during testing. This method can be applied to various existing offline reinforcement learning methods. By integrating TARL, these existing methods can overcome the limitations imposed by the static offline datasets and better handle the uncertain- ties and variations in the real-world environment.
To overcome this limitation, we propose novel offline re- inforcement learning paradigm, called Test-time Adapted Reinforcement Learning (TARL). This paradigm empowers offline RL to establish an interface with the environment, enabling the efficient update of a few parameters. Through the utilization of test-time data in the testing stage, it be- comes possible to further fine-tune the model. As result, the performance of offline RL can be significantly enhanced, and the learned policy can be effectively optimized to adapt to the real testing environment. Note that since we do not conduct an exploration of the environment and do not re- quire feedback from the environment, this remains an of- fline rather than an online approach. TARL still retains the advantages of offline reinforcement learning in terms of effi- ciency and security. Specifically, we construct unsupervised test-time optimization objectives for discrete control and continuous control tasks separately. We fine-tune the param- eters in the normalization layers of the policy by minimizing the output entropy or output uncertainty of the policy. In this manner, we can make minor adjustments to the policy using only the unlabeled data from the test environment, enabling it to adapt to the online testing environment. More- over, we propose debiasing term by KL. divergence, which restricts the model parameters from becoming excessively large. This can effectively prevent the policy from overly focusing on the test-time data and thus avoid the bias prob- lem caused by overfitting and error accumulation. In this paper, our main contributions are as follows:
TARL combines the adaptability of test-time fine- tuning with the conservatism of offline learning. We separately design unsupervised optimization objectives for discrete control tasks and continuous control tasks to achieve test-time adaptation. By using unsupervised objectives, TARI adapts the offline policy based solely on the test data, avoiding the need to wait for envi- ronmental rewards. Therefore, TARL preserves the stability and safety of conservative offline reinforce- ment learning.
We conduct extensive experiments on the Atari and D4RL benchmarks to demonstrate the effectiveness of our proposed method. Our method achieves sta- ble improvement over the baseline on various discrete and continuous control tasks. Notably, on the hopper- expert-v2 task, our method achieved remarkable 13.6% relative increase in episode return compared to CQL. These results demonstrate the robustness and efficacy of our method in improving the performance of offline reinforcement learning on a variety of tasks.
2

---

Test-Time Adapted Reinforcement Learning with Action Entropy Regularization
2. Related Work
The test-time adaptation method can be used to solve various distribution deviation problems, including the distribution deviation between offline datasets and online environments in offline reinforcement learning. In this paper, we use the test-time adaptation method to address this problem and improve the performance of offline policies by incorporating online data, Our approach aims to overcome the distribution shift problem by adapting the offline policy with test-time adaptation using online data.
2.1. Offline Reinforcement Learning
Due to the distribution shift between the offline dataset and the online environment, offline reinforcement learning of- ten encounters issues related to out-of-distribttion (OOD) state-action pairs. Prior works have attempted to address this problem by adding KL-divergence (Jaques et al., 2019; Peng et al., 2019), Wasserstein distance (Wu et al., 2019), or MMD (Kumar et al., 2019) between the learned policy and the behavior policy to avoid the OOD state-action pairs. However, these methods often require a separately estimated model of the behavior policy and are limited by their ability to accurately estimate the unknown behavior policy, espe- cially when data is collected from multiple sources (Levine et al., 2020). Conservative Q-learning (CQL) (Kumar et al., 2020) does not require estimating the behavior policy, which is sought after by researchers.
3. Test-Time Adapted Reinforcement Learning
Notations. We represent the environment as a Markov Decision Process (MDP) consisting of a 5-tuple < S, P. A.. >, where S is the state space, A is the action space, P(s'[s, a) is the transition probability distribution, R: S x A + R is the reward function, and y E [0, 1] is the discount factor. Reinforcement learning (RI.) aims to seek policy x <(ls) from the set of policy functions π to maximize the expected cumulative discounted reward.
Previous approaches to offline RL have also attempted to estimate uncertainty determine the trustworthiness of Q- value predictions (An et al., 2021; Lee et al., 2022), but these methods have not been generally performant due to the high-fidelity requirements of uncertainty estimates in offline RL. (An et al., 2021 Jused Jusee a diversified q-ensemble to capture Q-value variance for uncertainty, yet it led to over- fitting with scarce or biased datasets, harming uncertainty reliability and decision-making in novel situations. (Lee bal-
Problem definition. In offline reinforcement learning, the offline dataset D {(s;, Qg Ti, {1+1)) is collected by a behavior policy u(-ls). The agent can only learn the offline policy gof (a|s) from the offline dataset D, without inter- acting with the environment to improve the policy. A ma- jor challenge arises from the pervasive presence of Out-of- Distribution (OOD) state-action pairs in real-world environ- ments, where offline policies encounter novel interactions beyond their training distribution. Due to the distribution shift between the state-action distribution dom a) of on- line samples and deff a) of offline samples, the learned offline policy gaf (als) cannot be well adapted to the online environment (Lee et al., 2022). This lack of adaptability can trigger a series of adverse effects, such as poor performance and erratic decisions in real-world environments.
et al., 2022) faced challenges in exploration-exploottaie ance. Its pessimistic bias to handle unseen states-action pairs sometimes made the agent overly avoid exploration, stalling learning and preventing optimal performance. Our approach is devised with the aim of empowering offline reinforcement learning methods to smoothly adapt to out-of-distribution (OOD) states-actin pairs in the actual environment, and notably, it manages achieve this without having fall back on online rewards.
Overview of TARL. To solve the problem, we develop Test-time Adapted Reinforcement Learning (TARL) to mitigate the distribution shift between the offline dataset and the online environment, as shown in Figure 2. Firstly, TARL. computes the probability entropy. We set a threshold for the entropy to exclude out-of nattoo-distributioo (OOD) samples. Subsequently, we use the selected highly confident samples and minimize the entropy loss to achieve unsupervised test- time fine-tuning. This process allows our model to adapt to the test-time data without relying additional supervision, leveraging the inherent information within the data itself. Secondly, we compute the debiasing term by enforcing the KL divergence between the original policy and the test-time updated policy. By constraining the update, we prevent the network from overfitting to the test-time data and ensure its generalization ability. Lastly, we combine the entropy minimization term and the debiasing term update only the layer normalization layers. By selectively updating these
2.2. Test-Time Adaptation
Test-time adaptation (TTA) aims to improve model accu- racy out-of-distribution (OOD) test data by adapting the model with test samples. Existing TTA methods, such as TTT (Wang et al., 2020) and TTT++ (Liu et al., 2021), rely on joint training of a source model using both supervised and self-supervised objectives, followed by adaptation with a self- supervised objective at test time. However, this pipeline assumes a specific manner of model training, which may not always be controllable in practice. To address this, fully test-time adaptation methods have been proposed, which adapt a model with only test data. These methods include batchnorm statistics adaptation (Nado et al., 2020), test- time entropy minimization (Wang et al., 2020; Niu et al., 2022), prediction consistency maximization over different augmentations (Zhang et al., 2022).
3

---

Test-Time Adapted Reinforcement Learning with Action Entropy Regularization
Test-Time Optimized Objectives
Train
Highly Confident
Test Samples
Known States
Wrong Action
Probability Entropy
Used
Filter
Offline Datasets
Offline Policy
Train
Test Domain
Minimize
Domain
Test
TARL
Optimizer
Data Distribution
Mismatch
Debiasing Term
Regularize KL Divergence
Test
Update Layer Normalization Layers
Distribution (OOD) States
Correct Action
Conv
LN
Conv
LN
Conv
LN
FC
Real-world Environments
Fine-tuned Policy
Figure 2. The overview of our Test-time Adapted Reinforcement Learning (TARL) method. TARL adjusts the trained policy during test time efficiently and effectively. Given a state from the test environment, our test-time adaptation strategy consists of three steps. First, we compute the probability entropy used sample filters and test-time lcarning objectives. Second, compute the debiasing term by enforcing the KI. divergence hetween the trained policy and test-time updated policy. Last, we combine the entropy minimization term and the regularization term update the layer normalization layers only.
layers, we can efficiently adapt the model the test-time environment while preserving the knowledge learned during the pre-training phase. When the pre-trained offline policy is deployed in the online environment, we use the test data to adapt it, which enables the offline policy to better align with the distribution of the online environment, thereby leading to higher returns.
to fine-tune the policy, further complicating the adaptation process. Therefore, it becomes crucial to enable the offline policy, which does not rely on real-time rewards from online environments, to adapt to real-world scenarios.
Motivated by this, we propose Test-time Adapted Rein- forcement Learning (TARL). Inspired by online Test-Time Adaptation techniques, we construct unsupervised objective functions for discrete and continuous control tasks respec- tively. These functions number parameters of layer normalization layers in the of- fline policy using test data. Specifically, our aim is to enable the policy to enhance the confidence of high-probability categories and avoid selecting low-confidence categories. Intuitively, the uncertainty associated with unknown out-of- distribution (OOD) actions, i.e., those actions not present in the offline dataset, is greater than that of known actions, which are the actions that exist within the offline dataset. Through the test-time min-tuning, the offline policy can be encouraged to select actions with higher confidence. This, in turn, helps to avoid some of the unknown OOD state- action pairs, thereby improving the policy's 's performance and adaptability in the face of distributional shifts. By imple- menting this approach, signals for policy enhancement and empower the offline policy to adapt to the online environment. The proposed TARL approach has the potential to deftly circumvent the inherent limitations plaguing offline reinforcement learn- ing. Through establishing interactions between the offline policy and the environment during test-time, our TARL pro- motes reinforcement learning to be more highly practical and efficacious in real-world scenarios.
3.1. Significance of Offline RL Test-Time Adaptation
employed to fine-tun minimal
In reinforcement learning, a prominent duito--disttibution (OOD) problem often exists between the offline dataset and the online enviromment, as shown Figure The offline dataset is usually collected under specific, limited condi- tions, like past experiments with fixed parameters. Con- versely, the online environment is dynamic and affected by unpredictable factors such as system state changes and evolving user behaviors. These differences cause signif- icant data distribution mismatch. As a result, deploying pre-trained offline policy directly the online environment may result in the agent taking unknown actions, leading to potential safety hazards due to the distributional shift between the online environment and the offline dataset. In safety-critical scenarios, such as autonomous driving, the agent taking unknown actions is not permissible. Therefore, risky to apply the original offline policy directly to the online environment. In contrast, online reinforcement learning enables the agent to interact directly with the envi- crucial
eliminate the reliance reward
ronment, where the environmental rewards feedback mechanisms for policy improvement. However, in some online environments, acquiring rewards can be time- consuming or even infeasible. Real-world scenarios may not provide real-time reward signals to motivate the agent

---

Test-Time Adapted Reinforcement Learning with Action Entropy Regularization
3.2. Unsupervised Test-Time Learning Objectives
task is as follows:
Our Test-time Adaptation Reinforcement Learning (TARL) is designed to establish interaction between offline reinforce- ment learning and online environments without relying environmental reward signals. To achieve this goal, we pro- pose unsupervised test-time learning objectives to update offline policies. The test-time objective Lene is to select the high confidence sample 8; to update partial parameters of the policy. We understand that discrete continuous control tasks and continuous control tasks have distinct characteris- tics. Discrete control tasks involve choosing from finite set of actions, like selecting a specific gear in vehicle. Continuous control tasks, on the other hand, involve actions that can take on any value within a certain range, such as adjusting the speed a robot. Therefore, we have tailored the rest-time learning objectives for these two types of tasks respectively.
Cale min [8) ( θ).
Continuous Control Tasks. Continuous control tasks are different from díscrete control tasks. The output of the pol- icy network is not about the probability of actions, thus we cannot directly use the minimum action probability entropy as the optimization goal. In continuous control tasks, the policy network outputs the specific value of each action. Across the entire action space, these outputs form a normal distribution, characterized by mean u and variance The action is then sampled by parameterization. So we adopt the following formula to represent the uncertainty:
1 2 (In (2xo²) + +1).
Econ (st )) =
(4)
Discrete Control Tasks. For discrete control tasks, we use probability entropy to quantify the uncertainty of the actions predicted by the offline policy (Niu et al., 2022). As shown in the following formula:
where o is the variance of the policy network, which repre- sents the uncertainty of the action mean u.
Due to the distribution shift between the offline dataset and the online environment, the policy network will inevitably output OOD actions for a certain state. In order to filter out this part of OOD samples, we use a replay buffer to temporarily store the test data 8,. When a certain amount of data is collected, we use Formula (4) to calculate the uncertainty of all samples in the replay buffer B {si J and select top k small variance sample data to fine-tune the strategy network. As shown in the following formula:
Edis (s; θ) = Σ (a log (a s), (1)
aEAA
where A is the action space, s is the state in the online environment. (s) is the probability entropy of the ac- tions predicted by the policy network, which represents the uncertainty of taking action.
A higher value of the entropy (s) indicates a greater level of uncertainty in the actions predicted by the offline policy. As this uncertainty increases, there is a higher prob- ability that the policy will execute unknown or incorrect actions, i.e., out-of-distribution (OOD) actions. In order to prevent the policy network from choosing those OOD actions with greater uncertainty, we filter out those samples with greater uncertainty by setting a threshold Eg:
feon = II (F))))
(5)
where Eg is the k-th smallest entropy of Frcm (8:)-
By minimizing the above Formula (4), the policy network can output more confident action mean u' with less un- certainty. Thus the test-time optimized objective for the continuous control task is as follows:
Cent min Seon (5) (s; θ).
(6)
(s) = (=)))
where T is the indicator function.
3.3. Test-Time Debiasing Regularization
Then, we use the selected samples to update the policy net- work. To improve the offline policy's 's adaptability to the online environment for discrete control tasks, we minimize the Edis (s) as the unsupervised optimization objective. By reducing the uncertainty of the predicted actions, we can enhance the confidence of the offline policy in predicting correct actions during the test-time. Consequently, this en- hances the policy's 's prediction accuracy and helps it better fit the online environment and handle new situations. What's more for effective adaptation, we only update a few train- able parameters C e of all layer normalization layers. Thus the test-time optimized goal for the discrete control
Since the test-time finetuning employs unsupervised objec- tives, the generation of errors is inevitable. The network relies on the inherent characteristics within the test data for optimization. When the threshold Eg is set higher, more online data are used to update the policy network. The more test data are involved, the more likely it is introduce issues such as data noise and data distribution shifts, exacerbating the generation of errors. As the training progresses con- tinuously and iteratively, these errors exhibit a cumulative effect. The accumulation of errors leads to a further negative impact on the model's performance. Moreover, the policy
5

---

Test- TimeTiie&Aa Adapted Reinforcement Learning with Action Entropy Regularization
Action Category
5
Algorithm 1 Training Method for TARL
-4
input The trained offline policy x xff, the online state the replay buffer B.
3-
2-
output The test-time updated policy vtta
1:
Initialize the replay buffer B to temporarily store the state.
2:
for episode 1, M do
20000
40000
600DD
80000
100000
120000
Steps
3: Get the initialized state from the online environment.
[a] Offine Policy Action Distribution
4:
for stept= t= 1, do
5:
Agent executes action d and observes state 8t. Store state 8, in B.
Action Category
5-
6:
7:
Filter out those states with greater uncertainty via Eqn.
3-
2-
(2) or Equ. (5).
8:
Compute the KL divergence between oof and л ttta by Eqn. (8).
20000
40000
60000
80000
100000
120000
9:
Use filtered samples to update the trainable parameters
Steps
Ô in π ta via Equ. (9).
(b) Test- Time Optimized Policy Action Distribution
10: end for
11:
end for
Figure 3. Policy network bias after test-time adaptation. When only using test-time learning objectives update the offline policy, the policy network will tend to output specific action category, such as the third category in the figure.
parameters so that the policy network can adapt to the online environment. The optimization objectives during iteration time step & are as follows:
network tends to over-emphasize the local features within the test data while neglecting the overall data distribution. As a result, the policy network is prone to the problem of overfitting. The accumulation of errors and the overfitting to the test data are intertwined issues. Together, they give rise to an increasingly severe bias problem. As shown in Figure 3, the model may output with high probability specific category. This bias can lead to suboptimal decision- making and a failure to generalize well to the real-world environment. In order to solve this problem, we propose to use the original pre-trained offline policy network of whose model parameters are frozen and perform KL di- vergence constraints between the output of the fine-tuned offline policy z!ta and the frozen policy network T.f, The KI. Divergence loss is as follows:
if A is discrete action space. Cdis
Cn =
(8)
ent if A is continuous action space.
Furthermore, we take into account the bias problem intro- duced during the test-time finetuning process. To address this issue, we proposed using the KL divergence loss as debiasing term to regularize the policy network. We present the whole loss function used in the debiased test- time adapted reinforcement learning:
L = Cen + A C
(9)
Ch KL(T= of m00)
(7)
where X is a hyperparameter that controls the degree of change of the output of the offline policy. When the A is large, the offline policy tends to the network that has not been updated; when the À is small, the offline policy parameter update step is almost determined by the online state 8t.
This debiased term by KI. divergence effectively restricts the update of model parameters. As result, the bias within the policy network is significantly reduced. With less bias and overfitting issues, the policy network can better generalize to different scenarios. During the test-time adaptation process, especially when dealing with out-of-distribution state-action pairs, the generalization ability of the policy is enhanced. It can make more accurate and reliable decisions, adapting well to various states and actions that may not be fully represented in the training data. This ultimately improves the overall performance and adaptability of the offline policy in real-world applications.
The overall method of our TARL is summarized in Algo- rithm 1. Leveraging the proposed debiased test-time learn- ing objectives, TARL can update an offline reinforcement learning network by using test data without reward. During the test phase. TARL can analyze the test data to adjust its internal parameters, optimizing the decision-makin strate- gies. TARL allows existing offline policies to effectively adapt to the environment. Consequently, offline policies integrated with TARL can maintain high performance and stability in the face of unexpected envirommental changes, significantly enhancing its generalization ability and practicality.
(OOD) state-action pairs in
3.4. Debiased Test-Time Adaptation for RL
We use the state data s, during the interaction with the online environment to further fine-tune the policy network
6

---

Test-Time Adapted Reinforcement Learning with Action Entropy Regularization
Table 1. Average episode return comparison of TARI. against baseline methods on Atari benchmarks over the 10 evaluations. The highest
mean scores are highlighted in bold.
Algorithm
Qbert
Seaquest
Asterix
Pong
Breakout
REM
914.38 + 6.81
335.42 = 9.72
387.00 + 4.68
200.23 + 0.10
3.27 = 0.44
REM (TARL)
920.18 + 15.82
339.26 + 9.38
394.73 + 8.53
200.20 + 0.06
3.58 + 0.55
QR-DQN
646.35 + 17.08
413.09 + 13.16
503.38
14.60
-88.48
0.12
3.89 + 0.42
QR-DQN (TARL)
672.73 + 14.44
424.57 + 11.91
819.24 30.74
-8.440 0.18
4.18 = 0.60
CQL
4334.64 259.67
257.80
9.25
504.59
13.98
1.02
1.22
8.26 = 1.17
CQL (TARL)
4602.67 + 264.11
270.40 8.42
524.76 13.98
2.08
0.92
8.53 1.15
4. Experiments
steps, and the average episode return used as the evalu- ation metric. For D4RL tasks, the agent interacts with the environment with 10 episodes, and we also use the average episode return as the metric. However, because the rein- forcement learning prediction is very unstable, we repeat the above process 10 times and use the mean and variance of 10 times as the evaluation metric.
To evaluate the effectiveness of Test-Time Adapted Rein- forcement Learning (TARL), we conduct experiments on both discrete control and continuous control tasks.
4.1. Datasets
Atari Benchmark. For discrete control tasks, we con- duct experiments Atari games (Bellemare et al., 2013). We evaluate TARL on five Atari tasks: Qbert, Seaquest, Asterix, Pong, Breakout. These games provide diverse challenges through distinct me- chanics, such as spatial reasoning in Qbert and reactive control in Pong.
4.4. Implementation Details
We implement the discrete control experiments in atari fol- lowing CQL (Kumar et al., 2020) and the continuous control tasks in the OfflieRL--Kit codebase (Sun, 2023)², We use two distinct sets of hyperparameters for discrete control and continuous control tasks, respectively. All environments of the same type of task share the same hyperparameters. For the D4RL benchmark dataset with continuous control tasks, the hyperparameters used for all tasks were a learning rate of le 1e-6, a buffer capacity size of 1000, and a selection of the top 10 small entropy samples to update the offline policy. The KL Divergence constraint was set to 1.0. For the Atari dataset with discrete control tasks, we set the hyper- parameters follows: a learning rate of 1e 1e= an entropy threshold Eg of 0.1, and a KI. Divergence constraint limit X of 1.5.
D4RL Benchmark. For continuous control tasks, we conduct experiments D4RL. benchmark (Fu al., 2020). We evaluate our algorithm in five settings: Expert (optimal/neearoppiiaal high-performing policy data, Fully Replay (training exclusively on fixed pre- collected datasets), Medium Policy (suboptimal policy- generated data), Medium Replay Buffer (mixed-quality data from training a medium policy), and Medium- Expert (hybrid expert-novice demonstrations). We evaluate TARL on three locomotion tasks: bipedal locomotion (HalfCheetah), monopedal jumping sta- bility (Hopper), and dynamic balance maintenance (Walker2D).
4.5. Offline RL on Discrete Control Tasks
Following (Kumar et al., 2020), we conducted experiments on discrete control tasks to compare the performance of our method with the classic offline reinforcement learning algorithms QR-DQN and CQL. As shown Table 6, our method achieved a greater performance improvement than QR-DQN and also outperformed the CQL method. In the Qbert task, TARL-enhanced REM (REM (TARL)) achieved an average episode return of 920. 18, while the original REM had a return of 914.38. For QR-QQN, the average episode return increased from 646.35 to 672.73 when combined with TARL (QR-DQN (TARL)). In the Asterix task, the average episode return of COL was 504.59, while CQL (TARL.
4.2. Baselines
For discrete control tasks, following (Kumar et al., 2020),
compare TARL with three classic algorithms: QR-
DQN (Dabney et al., 2018), REM (Agarwal et al., 2020)
and CQL (Kumar et al., 2020). For continuous control tasks,
we compare TARL with CQL (Kumar et al., 2020) and IQL (Kostrikov et al., 2022).
4.3. Evaluation Metrics
For the atari game, when deploying offline policy to the on- line environment, the agent interacts with the environment
'The source code for this project is publicly available at http://giihbb.com//sssoooaaaaaaa
for 125,000 steps, each episode is not more than 27,000
7

---

Test- Time Adapted Reinforcement Learning with Action Entropy Regularization
Table 2. Average episode return comparison of TARL. against hase- line methods on D4RL benchmarks. The highest mean scores are highlighted bold.
Table 3. The effect of Entropy Threshold Fs. The highest mean
scores are highlighted in bold.
Entropy Threshold
CQL.
TARL (Ours)
CQL Comparisons
0.9
1.02 + 1.22
199.98 + 0.11
Task Name
CQL
CQL (TARL)
0.7
1.02 1.22
-19.61 + 0.22
hopper-expert--2
99.72
113.34
0.5
1.02 + 1.22
-18.79 + 0.42
walkerr2d-ullyy-eepaa--22
95.31
97.99
0.3
1.02 = 1.22
-13.54 + 1.22
walker2d-expert-22
113.25
113.57
0.1
1.02 = 1.22
1.70 = 0.82
Table 4, Effectiveness of low-entropy selective training.
IQL Comparisons
Training Strategy
TARL
Task Name
IQL
IQL (TARL)
Global Entropy Minimization
77.74
Low-entropy Selective Training
82.95
Walker26-medium--2
79.92
82.43
High-entropy Selective Training
72.91
falker2d-expert-222
110.31
110.49
4.7. Ablation Study
reached 524.76. These results demonstrate the effectiveness of our method for discrete control tasks. Using test-time data and constructing unsupervised test-time optimization objectives, TARL fine-tunes the layer normalization layers of the offline policies. This allows the existing offline RL algorithms to better adapt to the distribution of the online environment. This further validates that our TALR can be applied to various existing offline reinforcement learning methods to make them adapt to online environments.
Effect of Entropy Threshold Eg. We verify the effect of entropy threshold Es on test-time adapted RL under the pong game in the Atari benchmark. The learning rate is le-9. From Table 3, we can clearly see that when the entropy threshold increases from 0.3 to 0.5, the performance of test-time adapted RL declines, which just shows that just minimizing the test-time optimized goal does not necessarily promote the offline policy to adapt to the online environment, it is necessary to have a suitable threshold to filter some samples with relatively high confidence to update the offline policy to adapted the online environment.
4.6. Offline RL on Continuous Control Tasks
Our experimental results in Table 2 demonstrate consis- tent performance improvements over both CQL and IQL baselines across D4RL benchmarks. The method achieves substantial gains 2.68 and 13.62 on CQL's walker2d-fully- replay-v2 and hopper-expert-22 tasks respectively, while maintaining stable superiority (0.32) even on expert-level walker2d-expeert-22. suggesting enhanced capability to han- dle policy mismatch across varying dataset qualitics. Similar improvements emerge with IQL baselines, where our ap- proach outperforms by 2.51 on Walker2d metrrim--mduu--- v22, while preserving a marginal but consistent advantage (0.18) Walker2d-expert-v2. This performance pattern reveals two critical insights: first, the method demonstrates stronger efficacy when tackling medium-quality datasets with pro- nounced distribution shifts. Second, the persistent albeit smaller improvements on expert-level tasks confirm the uni- versal existence of offline-online distribution discrepancies regardless of behavior policy quality. These findings col- lectively validate our approach's 's robustness in mitigating distribution shift challenges, particularly in suboptimal data regimes where conventional offline RL methods struggle. Extended experimental results are provided in the supple- mentary materials A, including comprehensive comparisons across more tasks.
Effect of Selective Training Low-entropy States. The experiments were conducted in the walker2d-medium-v2 task to evaluate the hypothesis that selective training on low-entropy states improves TTA performance. The experi- mental setup included three conditions:
Global Entropy Minimization, where all available data were used for test-time adaptation.
Low-entropy Selective Training, where only samples with entropy below a predefined threshold were used for training.
High-entropy Selective Training, where only samples with entropy over a predefined threshold were used for training.
Our results demonstrate that selective training on low- entropy states improves TTA performance more effectively than global entropy minimization. Meanwhile, if we only select high-entropy samples for tta, the performance actually becomes worse. This further indicates that selective training on low-entropy states enables beneficial knowledge transfer.
Effect of KL divergence A. To investigate the impact of KL divergence on offline policy constraints, we conducted experiments on the Pong game with a learning rate of 1e -9 and an entropy threshold of 0.5. Table 5 shows that when the
8

---

Test-Time Adapted Reinforcement Learning with Action Entropy Regularization
References
Table The effect of KL divergeence X. The highest mean scores are highlighted in bold.
Agarwal, R., Schuurmans, D., and Norouzi. M. An opti- mistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104- 114. PMLR, 2020.
CQL.
CQL (TARL)
0.5
1.02 + 1.22
0.92 = 1.08
1.0
1.02 + 1.22
1.32 = 1.39
AlphaStar, D. Mastering the real-time strategy game star- craft ii. URL: https://deepmmind. com/blog/alphastrr- mastering-read real-time-stratege game- starerafar..t--ii.
1.5
1.02 + 1.22
2.08 0.92
2.0
1.02 + 1.22
0.62 = 1.73
entropy threshold is set too high, more uncertain samples 8t are used to update the offline policy, which can make it difficult for the policy to adapt to the online environment. Incorporating an appropriate KL divergence constraint be- tween the offline policy and the frozen base policy can help the offline policy transition smoothly to the online environ- ment by setting a more suitable À in the optimization goal during the adaptation process. When À changes from 0.5 to 1.0, we can see that the adaptive performance is improv- ing, but when À is too large, the adaptive performance will deteriorate.
An, G., Moon, S., Kim, J.-H., and Song, H. O. Uncertainty- based offline reinforcement learning with diversified q- ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation plat- form for general agents. Journal of Artificial Intelligence Research, 47:253-2799 2013.
Dabney, W., Rowland, M., Bellemare, M., and Munos, R. Distributional reinforcement learning with quantile re- gression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
5. Conclusion
In this work, we study how to use the test-time data learn promising policy in the offline reinforcement learning task. To this end, we propose Test-time Adapted Reinforcement Learning (TARL) to efficiently and effectively adjust the parameters of the layer normalization layers. We first in- troduce an entropy minimization loss as the unsupervised training objective for policy parameter update. Then we pro- pose a debiasing term that regularizes the KL divergence of predictions between the original pre-traine pre-traine policy and the test-time updated policy to avoid model degeneration due to the overestimation of few test-time inputs. Extensive experiments on the widely-used Atari game benchmarks show the superior performance our method.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-nriven reinforcement learning. arXiv preprint urXiv:2004.07219, 2020.
Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 02000112221145 2021.
Fujimoto, S., Meger, D., and Precup, D. Off- Off-policy deep reinforcement learning without exploration. In Interna- tional conference machine learning, pp. 2052-2062. PMLR, 2019.
Acknowledgments
Gao, C., Xu, K., Zhou, K., Li, L., Wang, X., Yuan, B., and Zhao, P. Value penalized q-learning for recom- mender systems. In Amigó, E., Castells, P., Gonzalo, J., Carterette, B., Culpepper. J. S., and Kazai, G. (eds.), SI- GIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Mudrid, Spain, July 15, 2022, pp. 2008-2012. ACM, 2022. doi: 10.114553477449555531196 URL https: //doi. 1tt.:/11.1111////47/// 3531796.
This work was partially supported by the Joint Funds of the National Natural Science Foundation of China (Grant No.U24A20327), Key-Area Research and Development Pro- the Major Key Project of Peng Cheng Laboratory (PCL) PCL2023A08, and TCL Science and Technology Innovation Fund, China.
gram Guangdong Province
Impact Statement
Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, R. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 197iv:1907.00456 2019.
The proposed TARL enables offline reinforcement learning to interact with online environments using only test data. TARL allows existing offline policies to effectively adapt to Out-of-Distribttion (OOD) state-action pairs in the environ- ment. Consequently, offline policies integrated with TARL can maintain high performance and stability in the face of unexpected environmental changes, significantly enhancing its generalization ability and practicality.
Kalashnikov. D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D.,, Holly, E.. Kalakrishman, M., Van- houcke, V., et al. Scalable deep reinforcement learning

---

Test-Time Adapted Reinforcement Learning with Action Entropy Regularization
for vision-based robotic manipulation. In Conference on Robot Learning, pp. 651-673. PMIR, 2018.
Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 2019.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. arXiv preprint arXiv:2110.06169 2021.
Sun, Y. Offlinerl-kit: An elegant pytorch offline rein- forcement learning library. ht https://githe con/ y yyhhasuun124 /tt///offfnnnneeeee 2023.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. International Confer- ence on Learning Representations, 2022. URL https: //openreviex net/ forum? wed:iidd=6nnnnmm
Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. Tent: Fully test-time adaptation by entropy minimiza- tion. arXiv preprint arXiv:2006. 10726, 2020.
Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Wu, Y., Tucker, G., and Nachum, O. Behavior regu- larized offline reinforcement learning. arXiv preprint arXiv:1911.113 1etr.1911.111,,, 2019.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191, 2020.
Wu, Y., Zhai, S., Srivastava, N., Susskind, J., Zhang. J., Salakhutdinov, R., and Goh, H. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint a05.v:2105.0005
2021.
Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline- to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learn- ing. pp. 1702-1712. PMLR, 2022.
Xie, T., Jiang, N., Wang, H., Xiong, C., and Bai, Y. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34:27395-27444 2021.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint (XXi:20055006433 2020.
Xu, K., Zhang, Y., Ye, D., Zhao, P., and Tan, M. Relation- aware transformer for portfolio policy learning. In Bessiere, C. (ed.), Pruceedings of the Twenty-Ninth In- ternational Joint Conference on Artificial Intelligence IJCAI 2020, pp. 4647-4653, ijcai.org, 2020. doi: 10. 24963//C URL https://doi.co 10. 10..4993/ijccs 2020/ 641.
Li, L., Chu, W., Langford, J., and Schapire, R. E. A contextual-bandit approach personalized news article recommendation. In Proceedings of the 19th interna- tional conference on World wide web, pp. 661-670, 2010.
Liu, Y., Kothari, P., Van Delft, B., Bellet- Mordan, T., and Alahi, A. Ti++:: When does self-supervised test-time training fail or thrive? Advances Neural Information Processing Systems, 34:21808-2188 21820, 2021.
Ye, D., Liu, Z., Sun. M., Shi, B., Zhao, P., Wu, H., Yu, H., Yang, S., Wu, X., Guo, Q., Chen, Q. Yin, Y., Zhang, H., Shi, T., Wang, L., Fu, Q. Yang, W., and Huang, L. Mastering complex control in MOBA games with deep reinforcement learning. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances Artificial In- telligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 6672-6679. AAAI Press, 2020. doi: 10. 10.1609/AAALL org/10. 10090.1600//00 www.l/699aaa...... wtt.:////aaa...111..a. v34104. 11111.44..444,,
Liu, Z., Liu, L.. Wu, B., Li, L.. Wang. X., Yuan, B., and Zhao, P. Dynamics adapted imitation learning. Trans. Mach. Learn. Res., 2023, 2023. URL https: //openrevies net tant/ffrmnee forum? id= 3pq
Lyu, J., Ma. X., Li, X., and Lu, Z. Mildly conservative q- learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711-1724, 2022.
URL https:/ /doi.
Nado, Z., Padhy, S., Sculley, D., D' Amour, A., Lakshmi- narayanan, B., and Snoek, J. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10900, 2020.
Yu, F., Chen, H., Wang. X., Xian, W. Chen, Y., Liu, F,, Mad- havan, V., and Darrell, T. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceed- ings of the IEEE/CVF conference computer vision and pallern recognition, pp. 2636-2645, 2020a.
Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P., and Tan, M. Efficient test-time model adaptation with- out forgetting. In International conference on machine learning, pp. 11888-16903) PMLR, 2022.
Yu, T. Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. Mopo: Model-based
10

---

Test-Time Adapted Reinforcement Learning with Action Entropy Regularization
offline policy optimization. In Larochelle, H., Ran- zato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 14129-14142. Curran Associates, Inc., 2020b. URL htts:ss//prrcceeengggg...rrrr cc/ cc/pappeeffileee/pper////////// 622252cce0ff322244777c7bbboo Paper. pdf.
Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28998 2021.
Zhang, M., Levine, S., and Finn, C. Memo: Test time
robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 35:38629- 38642, 2022.
Zhang, Y., Zhao, P., Wu, Q. Li, B., Huang, J., and Tan, M. Cost-sensitive portfolio selection via deep reinforcement learning (extended abstract). In 39th IEEE International Conference on Data Engineering, ICDE 2023, Anaheim, CA, USA, April 3-7, 2023, pp. 3771-3772. IEEE, 2023. doi: 200009//CCDE55555522222.00000. URL https:/ // 200..org/0..1100//0555555... 00312.

---

Test-Time Adapted Reinforcement Learning with Action Entropy Regularization
SUPPLEMENTARY MATERIALS
A. More Comparisons on D4RL with CQL
We compared the performance of our method with the classic offline RL algorithm CQL on more tasks. As shown in Table 6, method outperformed CQL significantly. Moreover, observed that our method achieved greater performance improvement than the baseline when the offline dataset collected using medium strategy. For instance, the walker2d-m task, our method achieved return that was 5.51 higher than CQL. This demonstrates that our method has significant advantage over normal offline reinforcement learning algorithms when there is a large distribution shift between the offline dataset and the online environment. On the dataset collected using the expert strategy, our method performed similarly to CQL, but still showed some improvements. This indicates that regardless of how the behavior strategy collects data, there is always a certain distribution offset between the offline dataset and the online environment.
Table Normalized average score comparison of TARL against baseline methods D4RL benchmarks the 10 evaluations. The score ranges from 0 (random policy) to 100 (expert policy). The abbreviations used are m for medium-v2, m-r for medium-replay--2. and m-c for medium-expert-v The highest mean scores are highlighted bold.
Task Name
CQL
CQL (TARL)
halfcheetah-m
50.67 0.30
50.75 : 0.11
hopper-m
85.74 4.42
87.79 + 3.96
walker2d-m
5.544
82.95
halfcheetah-m44
46.82 0.26
46.92
hopper-m-r
101.47 0.11
walker2d-m-r
85.13
85.61 + 4.33
halfcheetah-m--e
94.36 +
95.12 + 0.39
hopper-m-e
4.27 106.18
107.63 = 3.29
walker2d-m-c
112.73 1.98
113.64 + 0.20
Avg. Score
84.52
85.79
B. More Discussion about the Difference between the TARL and Online RL
TARL and online reinforcement learning (Xie et al., 2021; Lee et al., 2022) operate at different levels. The fundamental distinction between TARL and online RL lies in their operational paradigms and feedback dependencies.
When offline RL methods run online environments, they will suffer from OOD issues. The core motivation of TTA that environmental dynamics and distribution shifts cause performance drops in offline-trained models. TARL aims to maintain stability amidst data distribution changes. TARL fine-tunes the policy during the test-time phase through entropy minimization, without needing environment feedback. It effectively adapts to changes in test data distribution and ensures efficient updates. In contrast, online RL algorithms depend on environment feedback for policy updates. However, in some environments acquiring rewards can be time-comsee consumonnuiing or even infeasible. Therefore, offline-to dffline-to-oning onlone RL cannot update policies.
12


Reviews:
[{"review_id": "WYYsMsx3AA", "reviewer": "Reviewer_3AG2", "summary": "This paper proposes TARL, a framework that minimizes action uncertainty at test time to mitigate distribution shift issues.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Implementing the TARL framework on top of other offline RL algorithms, such as IQL.\n\nA1. Thank you for your valuable feedback. The state-action out-of-distribution (OOD) issues stem from data distribution shifts between training and testing phases, rather than being specific to particular offline RL algorithm. Our proposed TARL method is specifically designed to mitigate OOD interference in offline RL algorithms, thereby significantly enhancing stability during testing phases.\n\nAs a foundational algorithm in offline reinforcement learning, IQL [1] will suffer from this fundamental challenge during the testing phase. In Table A, we present a comparative analysis of IQL performance on Walker2d-medium-v2 environment and Walker2d-expert-v2 environment before and after TARL. Our approach achieving improvement in terms of average episode return.\n\nTable A. Effectiveness of our TARL applied to IQL.\n| Task Name | IQL | IQL (TARL) |\n| -------- | -------- | -------- |\n| walker2d-medium-v2     |   79.92   |   **82.43**   |\n|walker2d-expert-v2|110.31|**110.49** |\n\n[1] Offline reinforcement learning with implicit q-learning. ICLR 2022.\n\n\n> Q2. Conducting experiments on other datasets included in the D4RL benchmark, such as the AntMaze environments and the replay and expert datasets in Gym environments.\n\nA2. Accoding to your suggestion, we study more extensively across multiple benchmark datasets, including the AntMaze environments and the replay and expert datasets in Gym environment, to rigorously validate the effectiveness of our method. From Table B, our TARL achieves better performance than original RL algorithms, with particularly notable improvements in the complex environment Antmaze-medium-diverse-v0 and the expert-level task Hopper-expert-v2. These results further demonstrate the advantages of our method in handling data noise, distribution shift, and high-complexity tasks.\n\nTable B. Average episode return comparison of TARL against baseline methods on D4RL benchmarks over the 10 evaluations.\n\nEnvironment| Task Name | CQL | CQL (TARL) |\n| --------| -------- | -------- | -------- |\n| Mujoco| walker2d-expert-v2    |   113.25   |    **113.57**  |\n| Mujoco| walker2d-fully-replay-v2     |   95.31   |**97.99**\n|Mujoco|hopper-expert-v2|99.72|**113.34**\n| Antmaze| medium-play-v0   |   6   |**7**\n| Antmaze| medium-diverse-v0      |    2  |**5**\n\n---\nWe sincerely hope our clarifications above have addressed your concerns.", "claims_and_evidence": "The authors conduct experiments on the D4RL and Atari benchmarks to validate the effectiveness of their framework.", "methods_and_evaluation_criteria": "Yes, the evaluation criteria adopted in this paper are standard in this field.", "theoretical_claims": "Yes, although the paper does not present major theoretical results, I have verified the correctness of the proposed methods.", "experimental_designs_or_analyses": "I suggest adding more experiments to further evaluate the effectiveness of the proposed framework. Specifically, implementing the TARL framework on top of other offline RL algorithms, such as IQL, could provide additional insights. Additionally, I recommend conducting experiments on other datasets included in the D4RL benchmark, such as the AntMaze environments and the replay and expert datasets in Gym environments, as this is a common practice in the field.", "relation_to_broader_scientific_literature": "This work falls within the area of offline-to-online RL algorithms, which aim to mitigate distributional shift issues when encountering out-of-distribution state-action pairs.", "essential_references_not_discussed": "I suggest adding more references to recent progress in this area.", "other_strengths_and_weaknesses": "Please refer to the Experimental Designs Or Analyses part.", "other_comments_or_suggestions": "No further comments.", "questions_for_authors": "No further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dR7rUnTApH", "reviewer": "Reviewer_xAD3", "summary": "This paper introduces Test-Time Adapted Reinforcement Learning (TARL), a method designed to help offline RL policies adapt to distribution shift during deployment by leveraging test data—without needing additional reward signals. The core idea involves (1) learning objectives that minimize policy entropy for newly encountered states to reduce uncertainty (discrete tasks) or output variance (continuous tasks), (2) filtering out high-uncertainty (out-of-distribution) states so only lower-uncertainty states are used for adaptation, and (3) a KL divergence regularization term that keeps the adapted policy close to the original offline policy to prevent overfitting or degenerate solutions. Empirical results on Atari (discrete) and D4RL (continuous) benchmarks show that incorporating TARL on top of standard offline RL algorithms (e.g., CQL, QR-DQN, REM) improves final performance on out-of-distribution evaluation settings. The paper argues that TARL can be integrated with multiple offline RL methods by only updating the parameters in layer normalization layers, thus making test-time adaptation both computationally cheap and stable.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1. Your paper seems to rest on the hypothesis that selective training on low-entropy states enables beneficial knowledge transfer to other similar states. The paper should provide ablation studies comparing selective entropy minimization against global entropy minimization.\n\nA1. Thank you for your valuable feedback. The rationale for focusing on low-entropy states operates on two levels:\n\n**1. Theoretical Motivation.** Low entropy directly reflects reduced action uncertainty. States with low-entropy correspond to more deterministic and reliable decision regions in the policy space. By prioritizing these low-entropy states during test-time adaptation, the policy can establish stable decision boundaries while avoiding OOD extrapolation.\n\n**2. More study on the advantage of selective training on low-entropy states.** The experiments were conducted in the walker2d-medium-v2 task to evaluate the hypothesis that selective training on low-entropy states improves TTA performance. The experimental setup included three conditions:\n\n* **Global Entropy Minimization**, where all available data were used for test-tiem adaptation.\n* **Low-entropy Selective Training**, where only samples with entropy below a predefined threshold were used for training.\n* **High-entropy Selective Training**, where only samples with entropy over a predefined threshold were used for training.\n\nOur results demonstrate that selective training on low-entropy states improves TTA performance more effectively than global entropy minimization. Meanwhile, if we only select high-entropy samples for tta, the performance actually becomes worse. This further indicates that selective training on low-entropy states enables beneficial knowledge transfer. \n\nTable C. Effectiveness of low-entropy selective training. \n|Global Entropy Minimization|Low-entropy Selective Training|High-entropy Selective Training|\n|-|-|-|\n|77.74|**82.95**|72.91|\n\n\n>Q2. Have you conducted experiments comparing TARL to simpler alternatives that also reduce policy entropy: a) argmax action selection, b) globally reduces temperature in the Boltzmann distribution? If so, what were the results? If not, why were these fundamental comparisons omitted?\n\nA2. Our study does not include comparative experiments between TARL and simpler entropy-reduction approaches like boltzmann or argmax action selection, primarily due to their incompatibility with test-time adaptation requirements.\n\n**1. Characteristics of Test-Time Adaptation**\n\nDuring the test-time phase, the model has already been trained. The primary goal is fine-tuning the model using test data to adapt to the actual environment. The key characteristics of this phase is that the model cannot access environmental reward signals or other forms of feedback. This means the model cannot rely on environment feedback to guide policy updates.\n\n**2. Limitations of Boltzmann and Argmax Policies**\n\nThe Boltzmann policy is an action selection method that uses the softmax function to convert action values into a probability distribution. During training, the Boltzmann policy requires environment feedback to adjust the temperature parameter and action value function. The argmax policy is an action selection method that directly selects the action with the highest value. During training, the argmax policy needs environment feedback to evaluate and update the action value function. \nIn the test-time phase, the lack of environment feedback makes them impossible to effectively update these parameters.\n\n**3. Advantages of TARL**\n\nTARL has no reliance on environment feedback. It uses entropy minimization as the objective function for policy updates during the test-time phase, effectively updating the policy. By minimizing entropy, TARL enhances the selection of high-confidence actions and reduces reliance on uncertain actions, better adapting to changes in the distribution of test data.\n\n>Q3. Can you provide controlled experiments in simpler environments that explicitly demonstrate the hypothesized learning transfer from low-entropy states to related states? \n\nA3. We have conducted ablation studies on the effect of low-entropy states selection in Section 4.7. $E_0$ controls the selection of samples for updating. Specifically, samples with entropy below this threshold are selected for updating. As $E_0$ decreases, the selected samples have lower entropy and fewer in number.\nWhen the entropy threshold increases, the performance of test-time adapted RL declines, which shows that it is necessary to filter some samples with relatively high confidence to update the offline policy to adapted the online environment.\n\n---\nWe sincerely hope our clarifications above have addressed your questions.", "claims_and_evidence": "The core hypothesis underlying the TARL approach appears to be that selective training on low-entropy states facilitates information transfer to other similar states in a way that cannot be achieved through simpler entropy minimization strategies. However, this fundamental hypothesis is neither explicitly articulated nor well tested in the paper.\n\nThe proposed state-selective entropy minimization approach lacks critical comparisons to two obvious alternatives during runtime:\n1. Simply decreasing the temperature globally in a Boltzmann policy\n2. Using argmax action selection on states with low entropy\n\nWithout these comparisons, it's impossible to determine whether the reported improvements stem from the claimed state-selective mechanism or could be achieved through much simpler approaches that also reduce policy entropy. The paper presents performance improvements without convincingly demonstrating that these improvements are causally linked to the proposed state-selective training mechanism rather than to a simpler entropy reduction effect that could be achieved with less complex methods.", "methods_and_evaluation_criteria": "While the paper's method of state-selective entropy minimization is coherent, the evaluation is fundamentally incomplete because it lacks the necessary experiments to validate the core hypothesis. To properly evaluate whether selective training on low-entropy states provides unique benefits, the following comparisons are essential:\n\n1. Direct performance comparison with global temperature reduction in Boltzmann policies\n2. Comparison with a policy that simply uses argmax action selection for states below an entropy threshold\n3. State-specific analysis showing exactly where and how state-selective training performs differently than these simpler alternatives\n\nThe paper should provide:\n- Controlled experiments in simplified environments where the hypothesized information transfer from low-entropy to related states can be explicitly tracked\n- Concrete examples identifying specific state types or scenarios where state-selective training provides advantages that global approaches cannot\n- Ablation studies comparing selective entropy minimization against global entropy minimization\n\nWithout these critical evaluations, it remains unclear whether the additional complexity of TARL is justified over simpler alternatives that achieve similar entropy reduction.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "As discussed in the \"Claims And Evidence\" and \"Methods And Evaluation Criteria\" sections, the experimental design cannot validate the central hypothesis because it lacks baselines for comparison. The experiments show that the method works in practice but fail to establish why it works or whether simpler alternatives might work equally well. The improvements shown could result from effective hyperparameter tuning of the entropy threshold rather than from the claimed state-selective mechanism. Refer to the \"Methods And Evaluation Criteria\" section for the specific comparisons and controlled experiments that would be needed to properly validate the paper's core hypothesis.", "relation_to_broader_scientific_literature": "The paper appropriately relates its contributions to existing offline RL approaches such as CQL, REM, and QR-DQN, as well as to existing test-time adaptation methods (e.g., TTT, TTT++, batchnorm adaptation).", "essential_references_not_discussed": "See \"Claims And Evidence\" section.", "other_strengths_and_weaknesses": "Strengths:\n- The implementation approach (only updating layer normalization parameters) is computationally efficient\n- The method demonstrates compatibility with multiple offline RL algorithms\n- The paper shows empirical improvements on standard benchmarks\n\nThe paper's primary weakness is its failure to validate the core hypothesis that training on low-entropy states transfers beneficial knowledge to other similar states in the environment. Without controlled experiments that explicitly track this knowledge transfer mechanism, it remains unclear whether the observed improvements stem from the claimed selective learning transfer or from simpler entropy reduction approaches.", "other_comments_or_suggestions": "L426 test-time data instead of test-time date\nL41 migrate -> mitigate\nL346: Impletement -> Implementation", "questions_for_authors": "1. Your paper seems to rest on the hypothesis that selective training on low-entropy states enables beneficial knowledge transfer to other similar states. Could you explicitly articulate this hypothesis and provide direct evidence for this specific transfer mechanism rather than just overall performance improvements?\n\n2. Have you conducted experiments comparing TARL to simpler alternatives that also reduce policy entropy:\n   a) A policy that simply uses argmax action selection for states with entropy below a threshold?\n   b) A policy that globally reduces temperature in the Boltzmann distribution?\n   If so, what were the results? If not, why were these fundamental comparisons omitted?\n\n3. Can you provide controlled experiments in simpler environments that explicitly demonstrate the hypothesized learning transfer from low-entropy states to related states? Ideally, this would include visualizations or state-by-state performance metrics showing how the selective training influences performance on states that weren't directly trained on.\n\n\nProviding such evidence would significantly strengthen the paper and my rating.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xJOtXfZXTZ", "reviewer": "Reviewer_Hvb2", "summary": "This paper presents Test-time Adapted Reinforcement Learning (TARL) for the distribution shift issue of offline RL. TARL creates unsupervised test-time objectives for different control tasks. Moreover, it uses a KL divergence constraint to avoid bias. TARL only updates layer normalization parameters during testing for efficiency. Experiments on Atari and D4RL datasets show its superiority over baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1. In Line 70, the authors claim that they propose a novel offline reinforcement learning paradigm. However, the method belongs to a test-time adaptation method. Is there a conflict between offline reinforcement learning and test-time adaptation? I suggest further clarification on this.\n\nA1. Thank you for your valuable feedback. Offline reinforcement learning and test-time adaptation are not conflicting concepts. \nTARL is a framework designed for test-time adaptation of offline reinforcement learning methods. Offline RL learns from pre-collected data without interacting with the real-time environment. TARL fine-tunes the offline-trained policy model using test data to better fit real-world conditions. It is important to note that TARL requires no environment interaction or feedback during adaptation. As a result, Offline RL algorithms optimized by TARL with test-time adaptation retain the offline characteristic, maintaining its efficiency and safety while improving real-world performance.\n\n\n\n>Q2. Continuous control tasks assume that the action obeys a normal distribution. However, the actions in real applications may not be strictly normal. Can the TARL method based on the normal distribution assumption still work?\n\nA2. In practice, TARL remains effective even when the action distribution is not a strictly normal distribution. The core principle of TARL is to reduce policy uncertainty during action selection, thereby improving stability and adaptability. This process does not depend on any specific state-action distribution. For non-normal distributions, entropy minimization still encourages the policy to favor more confident actions, leading to better performance.\n\n\n>Q3. This paper mentions that there is a distribution bias between offline training and online testing in the abstract. In the experiments, where is this bias specifically reflected?\n\nA3. In our experiments, the distributional shift primarily manifests as a mismatch between state-action distributions in offline training datasets and evaluation environments. This OOD stems from the sampling incompleteness problem: While the state-action space is theoretically infinite, offline datasets can only capture a finite subset through sampling. Consequently, the policy inevitably encounters unobserved state-action combinations during real-world deployment, highlighting the critical need for test-time adaptation mechanisms like TARL.\n\n\n>Q4. The details of the Atari and D4RL benchmarks are not clear. For example, the environment and the agent setting are unknown. It would be better to add more descriptions about the two benchmarks.\n\nA4. We provide additional details regarding the benchmarks used in our experiments.\n\n**Atari Benchmark:**  For **discrete control tasks**, we evaluate policy performance on five representative Atari environments: Qbert, Seaquest, Asterix, Pong, and Breakout. These games provide diverse challenges through distinct mechanics, such as spatial reasoning in Qbert and reactive control in Pong.\n\n**D4RL Benchmark:**  For **continuous control tasks**, we employ three standardized D4RL dataset configurations: Medium Policy (suboptimal policy-generated data), Medium Replay Buffer (mixed-quality trajectories), and Medium-Expert (hybrid expert-novice demonstrations). We evaluate TARL on three locomotion tasks: bipedal locomotion (HalfCheetah), monopedal jumping stability (Hopper), and dynamic balance maintenance (Walker2D).\n\n**Consistent Experimental Settings**\n\nOur implementation strictly follows original offline RL methodologies for fair comparison. For example, CQL(TARL) maintains identical network architectures and training hyperparameters to vanilla CQL.\n\n\n>Q5. The performance of TARL depends on selecting appropriate entropy thresholds ($E_0$) and KL divergence weight ($\\lambda$).\n\nA5. The selection of appropriate entropy thresholds $E_0$ and KL divergence weights $\\lambda$ is critical to the performance. This underscores the necessity of the data filtering mechanisms and debiasing regularization  introduced in our method. A suitable $E_0$ ensures that only high-confidence samples are retained, thereby preventing the introduction of noise from out-of-distribution data and preserving the stability and reliability of policy updates. Similarly, the selection of an optimal KL divergence weight $\\lambda$ plays a key role in facilitating a smooth transition of the offline policy to the online environment. \n\n---\nWe sincerely hope our clarifications above have addressed your questions.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "TARL enables offline RL to establish an interface with the environment during the testing stage. It uses test data to fine-tune the model without the need for environmental rewards, simply by using the inherent information within the test data itself.\nThe proposed methods and evaluation criteria make sense for the problem, but I still have some questions:\n\n1.\tIn Line 70, the authors claim that they propose a novel offline reinforcement learning paradigm. However, the method belongs to a test-time adaptation method. Is there a conflict between offline reinforcement learning and test-time adaptation? I suggest further clarification on this.\n2.\tContinuous control tasks assume that the action obeys a normal distribution. However, the actions in real applications may not be strictly normal. Can the TARL method based on the normal distribution assumption still work?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "TARL shows significant advantages in experiments across discrete and continuous control tasks. The experimental results support the claim that TARL can be applied to various existing offline RL methods, effectively enhancing their performance in online environments.\n\nThe experimental designs and analyses presented in the paper are sound and robust, but I still have some questions:\n1. This paper mentions that there is a distribution bias between offline training and online testing in the abstract. In the experiments, where is this bias specifically reflected?\n2. The details of the Atari and D4RL benchmarks are not clear. For example, the environment and the agent setting are unknown. It would be better to add more descriptions about the two benchmarks.", "relation_to_broader_scientific_literature": "TARL uses test-time data to fine-tune the model without environmental rewards. It helps the policy adapt to real-world test environments. \n\nTARL can be applied to existing offline reinforcement learning methods. By integrating TARL, these methods can overcome the limitations imposed by static offline datasets. It helps them better handle the uncertain environment. This indicates a broad application prospect for the proposed method.", "essential_references_not_discussed": "The authors clearly discussed key methodologies in the field of efficient attention.", "other_strengths_and_weaknesses": "**Strengths**\n\n1.\tDuring test time, TARL updates only a few parameters of the LN layers. Compared to updating the entire network, adjusting only the LN layer parameters requires far fewer computations. This speeds up adaptation and enhances overall efficiency.\n2.\tTARL uses the KL divergence constraint as a debiasing term. This effectively restricts the update of model parameters, reducing the bias within the policy network and preventing overfitting to the test-time data.\n\n**Weakness**\n\nThe performance of TARL depends on selecting appropriate entropy thresholds ($E_0$) and KL divergence weight ($\\lambda$).", "other_comments_or_suggestions": "In Line 359, the period is missing.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4nAKelbJg9", "reviewer": "Reviewer_k3Mg", "summary": "This submission proposes Test-time Adapted Reinforcement Learning (TARL) to address the transfer gap between offline learning and online testing. TARL has two main components: minimizing the entropy of action probabilities by filtering actions with high entropy and efficiently updating only the layer normalization parameters, along with a KL divergence constraint between the fine-tuned policy and the original policy. The authors empirically demonstrate the effectiveness of the proposed TARL.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1. The improvements reported on most tasks are relatively marginal. This raises questions about the practical significance and robustness under different conditions and task complexities.\n\nA1. Thanks for your valuable feedback. The marginal improvements should be interpreted through four aspects.\n\n1. Our performance reflects a trade-off between efficiency and performance. TARL updates only LN parameters during testing, requiring no gradient updates to the full network. \n\n2. Unlike online RL, TARL operates purely on test samples without reward signals, making it suited for safety-critical applications where exploration is prohibited. TARL preserves the stability and safety of conservative offline RL.\n\n3. TARL shows improvements across discrete (Atari) and continuous (D4RL) control tasks, which indicates its robustness. \n\n4. We study more extensively across multiple datasets to validate our effectiveness.\nFrom Table D, TARL achieves significant improvement.\n\nTable D. Comparison of TARL against the baseline.\nTask|CQL|CQL(TARL)|\n|-|-|-|\n|Walker2d-fully-replay-v2|83.38|**97.99**|\n|Hopper-expert-v2|99.72|**113.34**|\n|Antmaze-medium-play-v0|6|**7**|\n|Antmaze-medium-diverse-v0|2|**5**|\n\n>Q2. The motivation for only fine-tuning the layer normalization parameters is not entirely clear. Why was this particular choice made?\n\nA2. The test-time phase requires efficient model parameter updates to ensure the model can adapt to the new environment promptly. Thus, TARL only updates parameters in layer normalization layers of the model, ensuring computational efficiency without full model gradient updating. This allows for rapid adaptation while maintaining model stability.\n\n>Q3. In Table 1 and 2, are the baselines also fine-tuned with test data?\n\nA3. The baselines do not fine-tuned with test data.\n\n>Q4. This work addresses the issue of the transfer gap between offline learning and online environments. There are several offline-to-online algorithms[1][2] that seem very related to this work. Shouldn't these algorithms be compared within the study?\n\nA4. Comparing TARL directly with online RL methods isn't appropriate as TARL is an offline RL method. TARL and offline-to-online RL operate at different levels.\n\nWhen offline RL methods run in online environments, they will suffer from OOD issues. The core motivation of TTA is that environmental dynamics and distribution shifts cause performance drops in offline-trained models. TARL aims to maintains stability amidst data distribution changes. TARL fine-tunes the policy during the test-time phase through entropy minimization, without needing environment feedback. It effectively adapts to changes in test data distribution and ensures efficient updates.\nIn contrast, offline-to-online RL algorithms[1][2], depend on environment feedback for policy updates. However, in some environments acquiring rewards can be timeconsuming or even infeasible. Therefore, offline-to-online RL cannot update policies.\n\nWe will include and discuss relevant literatures in the revised manuscript.\n\n[1] Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. NeurIPS, 2021. \n\n[2] Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. PMLR, 2022. \n\n>Q5. While CQL is a classic offline RL algorithm, it would be valuable to apply the proposed method to more powerful algorithms, such as DT[3] and IQL[4].\n\nA5. Accoding to your suggestion, we present a comparative analysis of IQL[4] performance in Table E. Even on the expert, we still achieve improvement.\n\nTable E. Effectiveness of TARL applied to IQL.\n|Task|IQL|IQL(TARL)|\n|-|-|-|\n|Walker2d-medium-v2|79.92|**82.43**\n|Walker2d-expert-v2|110.31|**110.49**\n\nHowever, applying TARL to DT[3] would require higher workload due to the inherent differences in the Transformer architecture. Given the time constraints of this submission, we are unable to complete the experiments. We will explore this direction in future work.\n\n[3] Decision transformer: reinforcement learning via sequence modeling. NeurIPS 2021. \n\n[4] Offline reinforcement learning with implicit q-learning. ICLR 2022.\n\n>Q6. Curiou about the sensitivity of the proposed TARL to the hyperparameters. The detailed hyperparameter settings for different tasks should be included for clarity.\n\nA6. We use two distinct sets of hyperparameters for discrete control and continuous control tasks, respectively. All environments of the same type of task share the same hyperparameters.\nThis further highlights the strong generalization capability of TARL. We have show hyperparameter settings in Section 4.4 and would make them more detailed and clearly.\n\nFor important hyperparameters entropy threshold $E_0$ and the KL divergence $\\lambda$, we have conducted ablation studies in Section 4.7. The TARL is sensitive to $E_0$ because a small $E_0$ is crucial for selecting high-confidence samples. \n\n---\nWe sincerely hope our clarifications above have addressed your questions.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and the corresponding evaluation criteria are appropriate for addressing the problem.", "theoretical_claims": "This submission does not provide theoretical claims.", "experimental_designs_or_analyses": "The experimental design and analysis exhibit soundness and validity.", "relation_to_broader_scientific_literature": "This work primarily focuses on the field of offline reinforcement learning.", "essential_references_not_discussed": "This submission includes sufficient related references.", "other_strengths_and_weaknesses": "**Strengths**\n\nThe main issue discussed in this paper, the transfer gap between offline learning and online testing, is crucial in the field of reinforcement learning. The proposed TARL method is articulated clearly, offering a well-structured approach.\n\n**Weaknesses**\n\nThe improvements reported on most Atari and D4RL tasks are relatively marginal. This raises questions about the practical significance and robustness of the proposed method under different conditions and task complexities.", "other_comments_or_suggestions": "No further comments and suggestions. Please see the following questions.", "questions_for_authors": "1. The motivation for only fine-tuning the layer normalization parameters is not entirely clear. Why was this particular choice made? Could you provide more insight into the rationale behind this decision? \n2. In the comparison results of Table 1 and Table 2, are the baselines also fine-tuned with test data?\n3. This work addresses the issue of the transfer gap between offline learning and online environments. There are several offline-to-online algorithms[1][2] that seem very related to this work. Shouldn't these algorithms be compared within the study?\n4. While CQL is a classic offline RL algorithm, it would be valuable to apply the proposed method to more powerful algorithms, such as DT[3] and IQL[4].\n5. I am curious about the sensitivity of the proposed TARL to the hyperparameters. The detailed hyperparameter settings for different tasks should be included for clarity.\n\n[1] Xie T, Jiang N, Wang H, et al. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning[J]. Advances in neural information processing systems, 2021, 34: 27395-27407.\n[2] Lee S, Seo Y, Lee K, et al. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble[C]//Conference on Robot Learning. PMLR, 2022: 1702-1712.\n[3] Chen L, Lu K, Rajeswaran A, et al. Decision transformer: reinforcement learning via sequence modeling, 2 June 2021[J]. URL http://arxiv. org/abs/2106.01345, 2021.\n[4] Kostrikov I, Nair A, Levine S. Offline reinforcement learning with implicit q-learning[J]. arXiv preprint arXiv:2110.06169, 2021.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a framework that mitigates distribution shift in offline RL by fine-tuning policies during deployment without requiring reward signals. Reviewers found the method novel, lightweight, and broadly applicable, with demonstrated improvements across Atari and D4RL benchmarks. However, concerns were raised about the lack of comparisons to simpler entropy-reduction strategies (e.g., global temperature reduction, argmax action selection) and related offline-to-online RL algorithms, as well as the marginal performance improvements in many tasks, raising questions about practical significance. While the rebuttal addressed some issues, such as additional experiments on IQL and ablation studies, the paper still lacks critical baselines and more direct evidence for its core hypotheses. The authors are encouraged to address these concerns in the final version.
Decision: Accept (poster)