FANS: A Flatness-Aware Network Structure for Generalization in Offline Reinforcement Learning
Authors: Da Wang
Yi Ma
Ting Guo
Hongyao Tang
Wei Wei
Jiye Liang
Venue: NeurIPS 2025 poster ()
Keywords: offline reinforcement learning
generalization
structuration
Primary area: reinforcement_learning

Abstract: Offline reinforcement learning (RL) aims to learn optimal policies from static datasets while enhancing generalization to out-of-distribution (OOD) data. To mitigate overfitting to suboptimal behaviors in offline datasets, existing methods often relax constraints on policy and data or extract informative patterns through data-driven techniques. However, there has been limited exploration into structurally guiding the optimization process toward flatter regions of the solution space that offer better generalization. Motivated by this observation, we present \textit{FANS}, a generalization-oriented structured network framework that promotes flatter and robust policy learning by guiding the optimization trajectory through modular architectural design. FANS comprises four key components: (1) Residual Blocks, which facilitate compact and expressive representations; (2) Gaussian Activation, which promotes smoother gradients; (3) Layer Normalization, which mitigates overfitting; and (4) Ensemble Modeling, which reduces estimation variance. By integrating FANS into a standard actor-critic framework, we highlight that this remarkably simple architecture achieves superior performance across various tasks compared to many existing advanced methods. Moreover, we validate the effectiveness of FANS in mitigating overestimation and promoting generalization, demonstrating the promising potential of architectural design in advancing offline RL.

FANS: A Flatness-Aware Network Structure for Generalization in Offline Reinforcement Learning
Da Wang', Yi Ma'; Guo', Hongyao Tang', Wei Wei', Jiye Liang' 'School of Computer and Information Technology, Shanxi University,
(wanda, mayi, weiwei, S @xxu.edu.cn 2 Data Science and Technology, North University of China, 2024001 nuuccedu..c College of Intelligence and Computing, Tianjin University, bluecontra @tju.edu.cn
Abstract
Offline reinforcement learning (RL) aims to learn optimal policies from static datasets while enhancing generalization out-of-distribution (OOD) data. To mitigate overfitting to suboptimal behaviors in offline datasets, existing methods often relax constraints policy and data or extract informative patterns through data-driven techniques. However, there has been limited exploration into struc- turally guiding the optimization process toward flatter regions of the solution space that offer better generalization. Motivated by this observation, we present FANS, structured network framework that promotes flatter and robust policy learning by guiding the optimization trajectory through modular archi- tectural design. FANS comprises four key components: (I) Residual Blocks, which facilitate compact and expressive representations; (2) Gaussian Activation, which promotes smoother gradients; (3) Layer Normalization, which mitigates overfitting: and (4) Ensemble Modeling, which reduces estimation variance. By integrating FANS into a standard actor-critic framework, we highlight that this remarkably simple architecture achieves superior performance across various tasks compared to many existing advanced methods. Morcover, we validate the effectiveness FANS in mitigating overestimation and promoting generalization, demonstrating the promising potential of architectural design advancing offline RL.
generalization-orien
Introduction
Offline reinforcement learning (RL. (1] focuses learning policies from fixed, pre-collected datasets without access to online interactions with the environment, fundamental challenge in this paradigm the distributional shift between the offline dataset and the true environment dynamics encountered during deployment. This discrepancy often leads to unreliable generalization, particularly in out-of- distribution (OOD) regions, where value functions tend exhibit overestimation [2]. To mitigate this, prior work has introduced approaches such as policy constraints [3]
4 5 6 7 value function
regularization the learning process, ensuring that the learned policy avoids making unreliable decisions in regions of state space with overestimated values, thereby enabling the model derive more robust and effective
To M.I. and uncertainty estimation 12 13 [4] These methods aim constrain
policies from offline datasets.
In recent years, generalization has emerged as central focus in offline RI., driven by the need to ensure reliable policy performance beyond the narrow support of the training data. To enhance generalization, existing methods typically relax constraints the dataset (15), behavior policy (7).
or support S [6] during the learning process. In addition, some data-driven approaches [17
18 formulate generalization domain adaptation problem, treating the discrepancy between
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

training and deployment distributions as a shift between source and target domains. Despite these advances, relatively little effort has been devoted to exploring how neural network architectures might influence generalization. While most efforts in offline RI. have centered around data and optimization constraints, the role of model architecture promoting generalization remains relatively underexplored. Yet, architectural design can significantly influence the learning dynamics and the generalization behavior of the resulting policy.
Recent studies have demonstrated that specific architectural choices - such as residual connec- tions [9], layer normalization [20] 211, and smooth activation functions can implicitly bias optimization toward flatter regions of the loss landscape, which are often associated with improved generalization. While state-of-the the-trr-tt architectures like SimBa [22] have achieved remarkable scal- ability in deep RL, the most suitable architectural design for offline RL remains open question. Given the unique challenges of offline RI. particularly distributional shift and limited data coverage, which often lead to sharp, overfitted solutions incorporating architectural inductive biases that promote flatter minima represents promising and complementary direction for improving gener- alization. Motivated by these insights, we propose FANS (Flatness-Aware Network Structure), structural framework specifically designed to enhance generalization in offline RL. FANS comprises four key modules, each tailored to encourage flatter solutions and improve stability:
1. Residual Blocks:
Facilitate learning simple, clean mappings, enabling smoother gradient flow and mitigating the risk of overfitting to noisy or sparse data points.
2. Gaussian Activation Function: Replace traditional piecewise-linear activation (e.g-- ReLU) with smoother functions, promoting continuous gradients and flatter local loss landscapes.
3. Layer Normalization:
Regularizes feature distributions across layers, helping stabilize optimization dynamics and preventing sharp activations that could lead to overfitting
4. Model Ensemble: Aggregates multiple models reduce variance and bias, ensuring the learned policy is not overly sensitive to specific trajectories or regions in the training data.
Together, these components systematically bias the optimization process toward solutions located in flatter regions of the loss landscape, thereby enhancing the model's ability generalize to unseen OOD data while maintaining strong performance in well-covered regions. Importantly, by introducing the FANS framework into standard Actor-Critic (AC) architecture without modifying the objective function, we observe substantial performance improvements. This underscores the effectiveness of architectural design in addressing the unique generalization challenges inherent to offline RL.
In summary, our contribution is three-fold:
1. We propose structured network design framework for offline RL that integrates residual blocks, Gaussian activation function, layer normalization, and ensemble techniques to enhance generalization.
2. We validate the effectiveness of the proposed framework across multiple offline RL tasks, highlighting that our remarkably simple architecture leads to substantial performance gains.
3. We conduct a detailed analysis to elucidate how the FANS framework facilitates smoother optimization, reduces variance, and mitigates overfitting. thereby achieving significant improvements in OOD generalization performance.
2
Preliminary and Related Work
2.1 Offline RI.
Offline RL, also referred to as batch RL, aims to learn optimal policy solely from fixed dataset D= = [(s a, r, s')). collected by one or more behavior policies, without further interaction with the environment. The learning problem is typically formulated within the Markov Decision Process framework, defined by tuple (S,AA. A, r),, where S is the state space, A the action space, P(s'l ls, a) is the transition probability, r(s, a) is the reward function, and y (0, 1) is the discount

---

widely adopted state-of the- factor. The objective of offline RL is to learn policy r(als) that maximizes the expected cumulative reward over the data distribution induced by the behavior policy, formulated as
r'r(s,, ae) 80 = 8, ao a ].
(TT == (e (Q" a)] = E [a
E
(I)
where Q* a) denotes the expected discounted return starting from state 8, taking action and following policy x thereafter. Unlike online RI. [23]. where the agent iteratively interacts with the environment to refine its policy, offline RL faces the fundamental challenge of distributional shift: the learned policy π may generate actions and state-action pairs that are OOD concerning the dataset D. This can lead to overly optimistic value function estimates, hindering policy performance.
Generalization in offline RL. As attention to distribution shift increases in offline RL, various meth- ods have been proposed to address the challenge. Among them, traditional conservative approaches typically constrain learned policies to remain within the dataset's support, aiming to suppress the gen- eration of OOD actions. Representative approaches include explicit policy constraints 4
3 5
6. 24],
value function penalization 8 10 25 26), uncertainty quantification techniques [2
13 14, and the
integration of imitation learning [27 9. In most cases, these approaches favor conservative strategies to ensure the reliability and safety of the policies learned from limited offline data.
Building upon foundations laid by prior research, subsequent works have concentrated on refining methods to alleviate excessive conservatism, thereby enhancing generalization capabilities. For instance, MCQ [15] actively trains on OOD actions by constructing pseudo target values. SPOT [S] explicitly models the behavior policy's 's support using a VAE-based density estimator and intro- duces simple, pluggable density- bease regularization to effectively constrain offline RL policies. DOGE (28) leverages a learned distance function to guide policy learning beyond the data distribu- tion. TSRL. [29] exploits time-reversal symmetry in dynamics to improve representation learning and reliability estimation, enabling data-efficient and generalizable offline RI. from small datasets. POR [30] inherits the training stability of imitation-style methods while still allowing logical OOD generalization. STR [16] performs trust region policy optimization within the support of the behavior policy. Additionally, some data-driven approaches like PRDC 7 have found that regularizing policies toward the nearest sate-action pairs is more effective, enabling the learned policy to select actions outside the dataset for given state. Other studies [17] [8] innovatively model the OOD generalization challenge in offline RL from a distribution adaptation perspective. Orthogonal to existing methods, our approach leverages minimal architectural modifications to achieve impressive performance, offering a structural perspective largely overlooked in prior work.
2.2 Network Architecture Design and the Flatness of Optimization Landscapes
Early deep RL. largely overlooked network architecture design, often relying on simple MLPs 3T which, under RL's non- -tatiittttn and ftal-and-errov learning, exhibited optimization pathologies such as capacity loss 32), primacy bias [33], and plasticity loss [34). These issues worsen as model scale increases [35,, highlighting the urgent need for architectural innovations to alleviate training pathologies and enhance generalization.
Recent research has begun to address these challenges by leveraging network architecture design to steer optimization toward flatter minima in the loss landscape. At the macro level, techniques such as normalization and residual connections have demonstrated significant benefits. Methods like spectral normalization [36], batch normalization [37,, and the widely used layer normalization (20 38
effectively control gradient magnitudes and stabilize parameter updates. This regulation helps smooth the loss landscape by preventing excessively sharp or irregular surfaces, thereby facilitating more stable training dynamics and faster convergence. SEEM [39] identifies self-excitation mechanism causing Q-value divergence in offline RL and shows that can be effectively suppressed by LayerNorm. Additionally, BRO 400 and SimBa 22 demonstrate that incorporating residual blocks significantly improves training robustness and performance, with SimBa further enhancing stability by adding observation normalization layers, establishing art architecture in deep RL.
At the micro level, the choice of activation function also critically influences the geometry of the loss landscape. In particular, Gaussian- activations such as GELU (4), with their smooth curvature and continuous higher-order derivatives, have been theoretically and empirically shown to encourage
3

---

optimization trajectories toward flatter regions, thereby enhancing generalization performance A2] Compared to traditional piecewise linear functions like ReLU, these activations maintain nonlinear expressivity while mitigating gradient discontinuities that can destabilize training. Together, these micro-level designs complement macro-level architectural choices, jointly shaping more favorable optimization paths and improving model generalization.
3 FANS framework: Flatness-Awark Network Structure
To address the distributional shift challenge in of- fline RL -p-attiiulaayy the lack of generalization when encountering OOD states we propose a structure- enhanced network framework, FANS, Our framework leverages structurally guided design that incorporates several optimization-awaal architectural components to steer the training dynamics toward flatter regions of the loss landscape, thereby enhancing generalization.
Input x
S Standardization
Linear
LayerNom
Specifically, we employ residual connections, layer normalization, smooth Gaussian activation function, and model ensembling. An overview of the FANS architecture is illustrated in Figure
Residual Blacks
Ensemblee
Linear
Gaussian
Linear
Given an input vector x Rd,, the standardization process first computes the mean μ E Rd and standard deviation Rsd across the dataset as
+x
XN
LayerNorm
N
N
X
Σ N
x 1 +
Figure 1: FANS architecture.
where { denotes the of input vectors and E is small constant for numerical stability. The standardized output is then obtained by x x_ This transformation ensures that each input dimension has zero mean and unit variance, which improves optimization stability and prevents scale-sensitive biases in the learning process.
Residual Block. Each residual block in FANS is designed to encourage smoother optimization and more stable representation learning, addressing the sharp loss landscapes and overfitting risks inherent in offline RI. The architecture follows a pre-activation residual design, incorporating normalization, Gaussian nonlinearity, and linear projection within the skip-connected block. The residual path is shown in Table where W,, W E Rdd and b b2 E IRd are learnable parameters.
Table Residual Block Architecture in FANS, Each step operates on hidden vector of dimension d.
Step
Operation
Equation
Description
(1) Residual Save
rcs
Store input for residual connection Normalize input across features First linear transformation Smooth, non non-monotonic nonlincarity Second linear transformation
LayerNorm
h = LayerNorm(x) (x)
(3)
Linear Layer I
ha Wih + b
(4)
Gaussian Activation
h3 exp(-h3)
(5) Linear Layer 2
h4 W + b2
Residual Add
y
h4
Residual connection output
This structure preserves the identity mapping through residual addition, stabilizing gradient propaga- tion and promoting generalization through implicit regularization.
Gaussian Activation Function. The Gaussian activation function used in each residual block is:
duu) = exp(-u²),
(2)
where u is the output of a linear transformation. Unlike ReLU or other monotonic activations, the Gaussian function is smooth and bell-shaped, with bounded output in (0, 1). Its non-monotonicity enables richer functional representations and encourages localized sensitivity.

---

Layer Normalization. To stabilize training and enhance generalization, we adopt post-layer normal- ization after the final residual block. Formally, for an intermediate output y. we compute:
2 LayerNormmy).
Additionally, this operation reduces sensitivity feature scaling and enhances the smoothness of the optimization landscape, which is particularly beneficial when combined with activation functions such as the Gaussian function. Finally, it ensures consistent feature magnitudes across modules, which is important when the output is passed into a common prediction layer. Following normalization, the output is fed into linear layer, which maps it to the critic's value function predictions.
Ensemble of Networks. To further enhance stability and mitigate overestimation bias common in offline RI, FANS employs ensemble of M independently parameterized critic networks within the Actor-Critic framework. Each critic head Q(m²,, where 1, 1, ..... M, produces a separate estimate of the value function for a given state-action pair (s, a). The ensemble output is computed the average of these estimates:
Qussmbll (s, (8)
1 M
Σ
((tm)((s)))))))))))) (s)
By aggregating multiple value predictions, this ensemble approach captures epistemic uncertainty and helps prevent the overestimation of OOD actions. During actual implementation, FANS operates within a standard Actor- Critic (AC) framework. The actor π (als) is trained to maximize returns using value estimates from the critic ensemble.
Critic Target. Bootstrapped targets are computed using target networks Tigt and Qim²
rarrea (s, ss a) r(s,a) + min
-ii Qgr)) (s', (s)()')))
(5)
Critic Loss. Each critic minimizes MSE to the shared target. The ensemble critic loss is:
M
(g(m)ss Qungal (s,
Ceriic
(6)
m==1
Actor Loss. The actor is optimized to maximize the ensemble value:
Lactor
-EEDD
[Q Qensemble (s, [())]]
(7)
Experiments
In this section, we conduct experiments validate the effectiveness FANS. First, we compare FANS with various state-of-the-arr offline RL algorithms across multiple standard D4RL. benchmarks 43 to evaluate its overall performance improvements. Then, to verify the motivation behind FANS specifically, mitigating Q-value overestimation and enhancing generalization we design targeted experiments that assess these properties in controlled settings. Finally, we perform ablation studies to analyze the contribution of each component in the framework and validate their necessity. The appendix includes experiments combining FANS with other baselines, evaluations on additional tasks, analyses of FANS's structural extensions, and learning curves omitted from the main text. Code is available at hhtph:t::: //ttt//b.bttttt githtb..hh..... com www//omm///ooeeeeee DD//NG 1v6/FANS
4.1 Main Results
Experimental Setup. One of our main focuses is to demonstrate that FANS can yield substantial performance gains through minimal modifications the network architecture. To this end, we adopt the simplest actor-critic framework, TD3, as our base algorithm. It is important to note that we do not incorporate any offline-specific constraints such as behavior cloning (BC), order to purely isolate and showcase the effect of FANS. We evaluate the algorithm on MuJoCo locomotion tasks from the D4RI. benchmark. For simply, we abbreviate the names of the datasets all the tables as follows: [halfcheetah - ha, hopper + ho, walker2d
medium medium-replat
5

---

Table 2: Performance comparison on D4RL locomotion tasks over the final ten evaluations and five seeds (normalized scores). We bold the highest mean.
TD3
Tasks
TD3BC
AWAC
CQL
IQL
ReBRAC
SAC-N
EDAC
DT
+FANS
48.1
49.5
47.0
48.3
64.0
68.2
67.7
42.2
66.6
ha-m
+0.2
+0066
+0.2
+0.2
#0.7
+11.3
11.0
+0.33
+0.8
44.8
44.7
45.0
44.5
51.2
60.7
62.1
38.9
55.9
ha-mr
+0.6
+00.7
+0.3
+0.2
+0.3
++100
+11.
+00.5
+1.55
90.8
93.6
95.6
94.7
103.8
99.0
104,8
91.6
102.8
ha-me
+6.0
#0.4
#0.4
+0.5
#3.0
19.3
#0.6
=1.0
+3.4
60.4
74.5
59.1
67.5
102.3
40.8
101.7
65.1
104.6
ho-m
+3.5
+99.1
+3.8
+-3.8
+0.2
+.9.9
+0.3
+1.6
0.9
64.4
96.4
95.1
97.4
95.0
100.3
99.7
81.8
103.2
ho-mr
+22155
+55.3
+5.3
16.4
16.5
+0.8
+0.8
16.9
#1.1
101.2
52.7
99.3
107.4
109.5
101.3
105.2
110.4
113.3
ho-me
19.1
#37.5
100.9
=7.8
#2.3
111.6
11011
+0.3
#1.4
82.7
66.5
80.8
80.9
85.8
87.5
93.4
67.6
101.0
wa-m
+4.8
+:26.0
#3..3
#33.2
+0.8
+0.7
#1.4
22255
11.6
85.6
82.2
73.1
82.2
84.2
79.0
87.1
59.9
98.3
wa-mr
+4.0
=1.1
13.2
+3.0
+2.3
=0.5
#2.8
+2.77
+2.0
110.0
49.4
109.6
111.7
111.9
114.9
114.8
107.1
118.1
wa-me
+0.4
#38.2
+0.4
#0.9
#0.4
+0.4
#0.7
110
#0.4
Avg.
76.5
67.7
78.3
81.6
89.7
83.5
92.9
73.8
96.0
mr, medium-expert- -+ me]. For the baseline algorithms, we report results 1M gradient steps, the values reported in their
either by re-running the official implementations or directly adopting original papers. For our method, we conduct experiments using five random seeds and report the mean normalized score averaged over the final ten evaluations.
Baselines. We compare method with several representative state-of-the-art offline RL algo- rithms: (i) TD3+BC 3 combines TD3 with behavior cloning by adding supervised loss to constrain policy updates toward the dataset actions; (ii) AWAC 44 accelerates offline learning by weighting advantages in actor-critic updates to prioritize high-value actions; (iii) CQL (8) introduces a conserva- tive penalty on Q-values prevent overestimation for unseen actions offline settings; (iv) IQL [9] avoids explicit behavior cloning or importance sampling by selectively updating Q-values, V-values, and policy via expectile regression; (v) ReBRAC 45) enhances stability by regularizing the policy using behavior cloning and applying conservative f-function updates; (vi) SAC-N extends SAC with an ensemble Q-networks and conservative penalty to improve performance and robustness; (vii) EDAC [1] improves value estimation in offline RL by decorrelating gradients across Q-networks in an ensemble; (vii)) DT 46 reframes RL as a sequence modeling problem by training a transformer to predict actions conditioned on returns and past trajectories.
Table Arovidds comprehensive comparison of our proposed FANS method against several state- of-the-art offline RL algorithms discussed above. Bolded values represent the highest normalized scores achieved for each task, while the denotes the standard deviation computed over five seeds. The results indicate that integrating the FANS framework into the structurally simple Actor-Crine algorithm TD3 yields the best average performance (Avg.) across all evaluated tasks. Notably, TD3+FANS demonstrates superior performance in most scenarios, with particularly pronounced improvements observed the hopper and walker2d environments. Collectively, these results substan- tiate that employing succinct yet effective network architecture can markedly enhance algorithmic performance, underscoring the significant potential and practical utility of minimalist architectural modifications in offline RL.

---

4.2 Validation of FANS in Mitigating Overestimation
In offline RL, overestimation is common issue, particularly severe when encountering sparse OOD state-action pairs caused by distributional shifts in the data. To address this challenge, our FANS framework incorporates several architectural designs. Residual blocks promote the learning of stable and low-frequency value functions through identity mapping pathways, helping to reduce overfitting and overestimation. The smooth nature of Gaussian activation functions guides the optimization towards flatter regions the loss landscape, further reducing overestimation tendencies. Layer normalization stabilizes the training process by normalizing activations within each layer, mitigating risks of gradient explosion and vanishing, thereby improving value estimation accuracy. Finally, the ensemble method leverages multiple independent networks collaboratively evaluate values, effectively suppressing overly optimistic estimates from individual models and enhancing the robustness and stability of value predictions.
To evaluate whether FANS mitigates overestimation, measure value estimation accuracy by comparing each algorithm's 's Q-value predictions against the dis- counted Monte Carlo returns of its policy trajectories (ground truth). We quantify over- or underestimation by subtracting the ground truth from the predicted Q- value and dividing by the ground truth, where values greater less than indicate overestimation or un- derestimation. Averaged across six D4RL datasets (ha-m, ha-mr, ho-m, ho-mr, wa-m, wa-mr), FANS consistently achieves the most accurate Q-value esti- mation, with TD3BC also showing relatively precise predictions (Figure 2 This demonstrates that FANS and TD3BC provide superior value calibration avoiding the severe overestimation of vanilla TD3 and the strong underestimation of CQL. Notably, TD3BC's reasonable value accuracy does not translate to superior performance, indicating that accurate value estimation alone insufficient.
FANS
TD3BC
Relative Q estimatioo (96)
-20
-40
-00
-90
0.0
Training Stops (x220%)
Figure 2: Relative estimation.
waller22berediiumeesssyyyy
800
TD3
Ontine
100
800
Quallis
Nomallzd
1D3
TD3 FANB
200
N
0.2
0.1
0.6
0.0
0.2
0.4
0.8
0.0
Tra ning Sleps (x10%)
Tra ning Steps (x10%)
(a) Q-values for OOD data
(b) Training process
Figure 3: Evaluation of overestimation mitigation: Q-value estimates on OOD data and corresponding performance of TD3 and TD3 FANS.
Additionally, we conduct experiments comparing the Q-value estimations for OOD data. We randomly extract 100 groups (each containing 30 samples) from the walker2d--mediun-rrplle dataset in the D4RL. benchmark and use the remaining samples to train offline RI. models. These extracted samples serve as OOD data relative to the current training set. We run Soft Actor-Critic (SAC) [47 on the MuJoCo walker2d-v2 environment and save the Q-value function model at 3M steps, then estimate these OOD data obtain a standard Q-value (the curves of Online SAC in Figure 3 a)). Furthermore, we evaluate the TD3 algorithm and our proposed TD3+FANS on the dataset with OOD samples removed described above. We periodically assess the Q-value estimates on the OOD data, with the average values reported in Figure aa). The results demonstrate that the baseline TD3 consistently produces elevated Q-value estimates for OOD inputs, ultimately leading to training collapse due to severe overestimation. In contrast, TD3+FANS maintains consistently lower Q-value estimates on OOD data, effectively mitigating overestimation issues. This advantage further corroborated by the performance curves presented in Figure bb).
7

---

4.3 Revealing FANS's Fundamental Advantage: Generalization Control
4.3.1 Theoretical Analysis using Neural Tangent Kernel (NTK)
To investigate why FANS outperforms TD3BC despite similar value accuracy, analyze generaliza- tion patterns using Neural Tangent Kernel (NTK) [48]. When updating Q-values for (s, using TD learning, the change at any (s, @) is governed by the NTK kg (5, a). A simple derivation in 48 given in Appendix Section B We track the normalized NTK improvement based on the NTK of initialized network (training step=0) and explore three OOD query types of (5, a) during training:
(s,
(8) + 8 E e is noise
(d) E which is next state and next action (method of DR3
49
Due to the high dimensionality of V Qs a), the direct computation of E (s, a (s), 8, a)l
e,0~D
computationally prohibitive, method is adopted by approximating A(Ø) with the contribution solely
from the last layer parameters and obtain the E ((s,, (T)) (ss a)], where ((s, a) signifies the s.a~>
representation of state-action pairs, which is the output of penultimate layer of Q-network.
We present the average results across six D4RL, datasets (ha-m, ha-mr, ho-m, ho-mr, wa-m, wa-mr) logged every 100K step during 500K training steps (makes sure convergence) in Figure
The significantly lower NTK values in FANS demonstrate a fundamental suppression of patho- logical generalization patterns. This occurs be- cause the kernel k 0, a) acts general- ization amplifier: High values force TD-errors from ID) data (s, a) to distort OOD values Q(s, (88.0) a).
#is, a) TTs. T()s
FANS
TD38C
Reletve NTK valus (8))
COL
1500
During offline RI,, this propagates and amplifies extrapolation errors (especially dangerous for ac-
1000
tions T(s) near distribution boundaries). Our de- sign directly counteracts this through: (1) residual connections that maintain stable feature baselines via identity mappings, preventing chaotic error propagation; (2) layer normalization that constrain feature adaptation by enforcing per-sample fea- ture scale invariance and reducing sensitivity to aberrant activation patterns. These mechanisms collectively enhance the model's expressive capacity and mitigates harmful generalization errors, thus shows superior performance than baselines.
500
26-
M1A
Sleps
(K)
Figure 4: The NTK value of (s,)(s)), E D.
Other two modes are presented in the Appendix.
4.3.2 Comparison with generalization-centri methods
The FANS framework promotes smoother loss surfaces during training, encouraging models to converge in flatter regions, which leads to better performance on unseen data. This structural design enhances the model's generalization ability, even when facing distribution shifts between the training and test data. To validate this, first compare method with several representative approaches designed to improve generalization: DOGE (28, TSRL [29], SPOT (5), POR (30),
PRDC [7]. STR (16), DIFFUION-QL (25), and CQL+ADS [18] shown in Table 3 Among the
many advanced methods associated with improving generalization, our approach remains highly competitive, achieving the best average performance.
4.3.3 Validation under limited-data setting
Inspired by 29 we design small-sample experiments to test model performance with limited data to evaluate generalization ablility of FANS. We reduce the training dataset 5% of the full data and choose TD3 as the baseline algorithm. Note that FANS is implemented using simple AC framework, without incorporating any advanced offline RI. techniques such as behavior cloning (BC) constraints. The learning curves of different algorithms shown in Figure

---

Table 3: Performance comparison D4RL locomotion tasks the final ten evaluations and five seeds (normalized scores). We bold the highest mean.
DIFFUSION -QI.
CQL +ADS
TD3
Tasks
DOGE
TSRL
SPOT
POR
PRDC
STR
+FANS
ha-m 45.3
48.2
58.4
48.8
63.5
51.8
51.1
73.9
66.6
ho-m 98.6
86.7
86.0
78.6
100.3
101.3
90.5
101.0
104.6
wa-m
86.8
77.5
86.4
81.1
85.2
85.9
87.0
91.3
101.0
ha-mr
42.2
52.2
43.5
55.0
47.5
47.8
49.6
55.9
ho-mr
76.2
78.7
100.2
98.9
100.1
100.0
101.3
102.4
103.2
87.3
66.1
91.6
76.6
92.0
85.7
95.5
93.7
98.3
ha-me
78.7
92.0
86.9
94.7
94.5
94.9
96.8
93.5
102.8
ho-me
102.7
95.9
111.4
99.3
109.2
111.9
111.1
113.3
113.3
wa-me
110.4
109.8
112.0
109.1
111.2
110.2
110.1
112.1
118.1
Avg.
81.0
77.5
85.9
80.1
90.1
87.7
87.9
92.3
96.0
"afich mmadam-d
700 9039
100
Teppee-meeckmmae ma12
1600
walker milkar225-mduum-222
100 1000
100
10.5
Ma Murll uh sof pon p
10.0
20
Namee
16
80
2.
04
The performance of different algorithms under limited data. A ratio of 100% denotes training on the entire dataset, while 5% corresponds to using only mnettwentieth of the full data.
Figure 5:
Figure 5 clearly shows that when the training data is significantly reduced, FANS consistently demonstrates superior generalization ability across all tasks compared the baseline algorithms. is worth noting that, in the hopper and walker2d tasks, FANS trained on limited data even outperform the baseline algorithms trained on full dataset. These results strongly indicate that FANS's structured design enables more effective feature extraction in data-scarce environments, thereby enhancing model robustness and generalization performance.
4.4 Ablation Study
Table summarizes an ablation study evaluating the impact of each component in the proposed FANS architecture across multiple D4RI. MuJoCo tasks. The results demonstrate that every module contributes to the overall performance, though to varying degrees.
Notably, the residual block and the ensemble mechanism emerge as the most critical components of the FANS framework. The removal of the residual block leads to the most substantial degradation in performance, underscoring its essential role in stabilizing learning and enhancing value approximation, likely by improving gradient flow and increasing network expressivity. Similarly, the ensemble mechanism contributes significantly to overall robustness; its absence results in marked decline performance, suggesting that aggregating multiple sub sub-policies effectively reduces variance. The final column in Table 4 includes the parameter M, representing the number of ensembles in FANS. The M ranges from [2, 3, 5). and we report the best-performing M for each task. Different tasks exhibit varying needs for ensemble size.
The ablation of the Gaussian activation function also leads to noticeable performance drop, as its core function of introducing stochasticity during policy learning helps enhance generalization and reduce the risk of overfitting static offline datasets. In addition, layer normalization, while resulting in relatively smaller performance drops upon ablation, still provides meaningful benefits by stabilizing training dynamics and promoting smoother, more consistent learning behavior.

---

Table 4: Ablation study of FANS. M is the number of ensemble in FANS.
TD3+FANS w/o Residual
TD3+FANS w/o Gaussian
TD3+FANS w/o LayNorm
TD3+FANS w/o Ensemble
TD3 FANS
Tasks
Score
M
ha-m
66.4 +: 1.3
66.2 + 0.8
64.3 + 3.5
66.6 + 0.8
66.6 + 0.8
ho-m
21.0 19.3
62.9 = 30.5
99.6 = 6.9
79.6 - 26.6
104.6 0.9
wa-m
+ 4.3
7.7 + 7.8
91.9 2.3
6.7 + 0.4
101.0
ha-mr
55.1 0.5
+ 1.3
55.9 + 1.5
ho-mr
41.7 + 7.2
95.1 + 4.5
99.2 = 5.6
44.6 + 6.3
103.2 = 1.1
wa-mr
35.6 11.3
80.2
95.1
75.4 23.5
ha-me
64.5 + 21.1
102.6 + 3.1
51.1 + 4.7
28.4 + 1.4
102.8 3.4
ho-me
1.5 + 0.3
30.3 + 8.2
41.3 13.8
1.6 + 0.8
113.3 = 1.4
wa-me
-0.2 1 0.1
89.1 +: 21.0
115.2 + 0.4
16.5 10.7
118.1 0.4
Avg.
32.3
65.4
79.3
41.6
96.0
In addition, our method considers residual only in the critic network. We provide ablation experi- ments to demonstrate that incorporating residual in the actor is unsuitable for offline RL. As the results presented Table the Control Setup refers to considering residual connections both the actor and critic, which degrades the overall effectiveness. This degradation can be attributed to the residual connections in the ac- tor causing the action output to be constrained within a limited range, effectively restricting the learning capacity within narrow decision space. Consequently, this limitation hampers the actor's ability to learn optimal behaviors. Based these observations, adopt the design choice of applying residual connections exclusively in the critic network, where they facilitate stable value estimation without constraining the action space, thereby achieving better overall performance.
Table 5: Ablation study on residual placement: performance drops when residual blocks are added to the actor network.
Structure
Control Setup
TD3 + FANS
Actor + Residual
X
Critic + Residual
V
63.8 +
66.6 0.8
ho-m
21.8 + 19.8
104.6 0.9
50.3 35.2
101.0 1.6
5 Conclusion
In this work, we introduce FANS, novel network architecture framework designed to tackle the unique generalization challenges of offline RL. By integrating residual blocks, Gaussian activation functions, layer normalization, and model ensembling. FANS systematically steers optimization toward flatter minima, thereby enhancing stability and reducing overfitting. Our comprehensive analyses reveal the individual and combined effects of these components in promoting smoother optimization landscapes and lowering variance. These results highlight the critical role of architectural design complementary and effective approach to advancing offline RL performance, opening promising avenues for future research. Future work will explore architectures better suited for affline-to-onlin settings, aiming to achieve more stable and safe policies that can rapidly adapt the online environment.
Acknowledgments and Disclosure of Funding
The work is supported by the National Natural Science Foundation of China (No. U21A20473, No. 62276160), the Fundamental Research Program of Shanxi Province (No. 22250002121001. No. 202403021222153), the Shanxi Provincial Natural Science Foundation General Project (No. 20220302121 12294),, the Shanxi Provincial Overseas Study Fund Project (No. 20240002), the Scientific and Technological Innovation Programs of Higher Education Institutions in Shanxi (No. 2025L001).
10

---

References
[1] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems, 2020.
[2] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, pages 2052-2062. 2019.
[3] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 34, pages 20132-201433 2021.
[4] Rasool Fakoor, Jonas W Mueller, Kavosh Asadi, Pratik Chaudhari, and Alexander J Smola. Continuous doubly constrained batch reinforcement learning. In Advances in Neural Information Processing Systems, volume 34, pages 122001111200 10:00-11222 1273, 2021.
[5] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pages 11278-31299 2022.
[6] Jiachen Li, Edwin Zhang, Ming Yin, Qinxun Bai, Yuxiang Wang, and William Yang Wang- Offline reinforcement learning with closed-form policy improvement operators. In Proceedings of the 40th International Conference on Machine Learning, pages -05288, 2023.
[7] Yuhang Ran, Yichen Li. Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy regular- ization with dataset constraint for offline reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, pages 28701-28717, 2023.
[8] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 1179-1191, 2020.
[9] Ila Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In Proceedings of the 38th International Conference on Machine Learning, pages 5774-5783, 2021.
[10] Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. In Advances in Neural Information Processing Systems, volume 34, pages 19235-19247, 2021.
[11] Yixiu Mao, Qi Wang, Yun Qu, Yuhang Jiang, and Xiangyang Ji. Doubly mild generalization for offline reinforcement learning. Advances Neural Information Processing Systems, 37:51436- 51473, 2024.
[12] Gaon An, Seungyong Moon, Jang Hyun Kim, and Hyun Oh Song. Uncertainty- based of- fline reinforcement learning with diversified qeenseble.. In Advances in Neural Information Processing Systems, volume 34, pages 7436-7447, 2021.
[13] Chenjia Bai, Lingxiao Wang. Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-die offline reinforcement learning. In Proceedings of the Tenth International Conference Learning Representations, 2022.
[14] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdi- nov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, pages 1 11319-11328. 2021.
[15] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongging Lu. Mildly conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pages 1711-1724, 2022.
[16] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement learning. In Proceedings of the 40th International Conference Machine Learning, pages 23829-2511 23851, 2023.

---

[17] Han Qi, Yi Su, Aviral Kumar, and Sergey Levine. Data-driven offline decision measiion-mkinn via invariant representation learning. In Advances Neural Information Processing Systems, volume 35, pages 13226-13237, 2022.
[18] Da Wang, Lin Li, Wei Wei, Qixian Yu, HAO Jianye, and Jiye Liang. Improving generalization in offline reinforcement learning via adversarial data splitting. In Proceedings of the 41st International Conference on Machine Learning, 2024.
[19] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Muchine Learning, pages 1407-1416, 2018.
[20] Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney, Understanding plasticity in neural networks. In Procrereeeeeee Proceedings of the 40th International Conference on Machine Learning, pages 23190-23211, 2023.
[21] Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado P van Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning. In Advances in Neural Information Processing Systems, volume 37, pages 106440-106473, 2024.
[22] Hojoon Lee, Dongyoon Hwang. Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subrama- nian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning, 2024.
[23] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[24] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement learning. In International Conference on Machine Learning, pages 23829-23851. PMLR, 2023.
[25] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In Proceedings of the Tenth International Conference on Learning Representations, 2022.
[26] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regularization for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:40587-40609, 2023.
[27] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 18353-18363, 2020.
[28] Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya- fa-Qin Zhang, When data geometry meets deep function: Generalizing offline reinforcement learning. In Proceedings of the Eleventh International Conference on Learning Representations, 2023.
[29] Peng Cheng, Xianyuan Zhan, Wenjia Zhang, Youfang Lin, Han Wang, Li Jiang, et al. Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pages 7612-763. 2023.
[30] Haoran Xu, Li Jiang, Jianxiong Li, and Xianyuan Zhan. A policy- guided imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pages 4085-4098, 2022.
[31] Scott Fujimoto, Wei-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pages 61573-61624, 2023.
[32] Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning, 2022.
12

---

[33] Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Lue Bacon, and Aaron Courville. The primacy bias deep reinforcement learning. In Proceedings of the 39th International Conference Machine Learning. pages 16828-16847, 2022.
[34] Timo Klein, Lukas Miklautz, Kevin Sidak, Claudia Plant, and Sebastian Tschiatschek. Plasticity loss deep reinforcement learning: A survey, 2024.
[35] Johan Obando-Ceron Aaron Courville, and Pablo Samuel Castro. In value-based deep reinforce- ment learning, pruned network is good network. In Proceedings of the s1 International Conference on Machine Learning, pages 38495-38519, 2024.
[36] Nils Bjorck, Carla P Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learning with spectral normalization. In Advances in Neural Information Processing Systems, volume 34, pages 8242-8255, 2021.
[37] Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In Proceedings of the Twelfth International Conference on Learning Representations, 2024.
[38] Hojoon Lee, Hanseul Cho, Hyunseung Kim, Daehoon Gwak, Joonkee Kim, Jaegul Choo, Se- Young Yun, and Chulhee Yun. Plastic: Improving input and label plasticity for sample efficient reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pages 62270-62295, 2023.
[39] Yang Yue, Rui Lu, Bingyi Kang, Shiji Song, and Gao Huang. Understanding, predicting and better resolving q-value divergence in offline-rl. Advances in Neural Information Processing Systems, volume 36, pages 60247-60277, 2023.
[40] Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Mitoš, and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control, In Advances in Neural Information Processing Systems, 2024.
[41] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2016.
[42] Minhyeok Lee. Mathematical analysis and performance evaluation of the gelu activation function in deep learning. Journal of Mathematics, 2023(1): 2023.
[43] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. 2020.
[44] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets, 2020.
[45] Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pages 11592-11620, 2023.
[46] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in neural information processing systems, volume 34, pages 15084-15097. 2021.
[47] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, pages 1861-1870, 2018.
[48] Yi Ma. Hongyao Tang, Dong Li, and Zhaopeng Meng. Reining generalization in offline reinforcement learning via representation distinction. In Advances in Neural Information Processing Systems, volume 36, pages 40773-4078 2023.
[49] Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. In International Conference on Learning Representations, 2022.
13

---

A
Implementation details
All experiments run on a server equipped with an Intel Xcon@ Gold 6254 CPU @ 3.10GHz and NVIDIA GeForce RTX 3090 GPU.
Table 6: Detailed hyperparameter settings for the proposed FANS framework.
Hyperparameter
Value Range
Description
Network depth
Number of residual blocks in the critic.
Hidden dimension (d)
256
Width of hidden layers in both actor and critic. Activation function used the actor network.
Activation (actor)
ReLU
Architecture
Smooth, bounded nonlinearity encouraging flatter optimization landscapes.
Activation (critic)
Gaussian
Structure: LayerNorm Linear Gaussian + Linear (with skip connection).
Residual design
Pre activiivvtionn
Applied after the final residual block stabi- lize representation magnitudes.
Output normalization
LayerNorm
Optimizer
Adam
Optimizer used for both actor and critic.
Learning rate
10-4
Learning rate for both and critic.
Batch size
256
Samples used per optimization step.
Training
Discount factor (7)
0.99
Reward discount coefficient.
Training steps
IM
Total number of gradient update iterations.
Seeds
{0, 10, 100, 1000, 10000) Five seeds used for all reported averages.
Number independently parameterized critic networks; best M per task is reported.
Ensemble size (M)
(2, 3, 5
B Details of Neural Tangent Kernel (NTK)
Parameter update for (s,a). a):
o' = ( (oQo(s,
(8)
where T is the target operator (learning rate omitted).
Q-value change at (5.a) a) by Taylor expansion at the pre-update parameter o: Q (5, a) (=) (oQo Φ) 0(1A0))
(9)
Generalization via NTK by plugging the second equation into the first one:
Q5r(s, = = (8, + [ka a, s,), (TQu(s,a) Qo(s, 0(1A<11
(10)
where
ko(5,a,8,a) VaQo VoQo(s,
(11)
NTK interpretation:
High Ke = Prominent generalization. Q(3, a) changes significantly in syne with the TD error updating Q(s,a),, a).
Low hs
Minimal generalization (tabular-like behavior when Kg = 0).
Key Insight: The NTK (8, a, 8, a) quantifies how much updating Q(8, a) affects Q(5, a), acting generalization metric for TD learning.
14

---

0( d(s.mls) + noise)
d(s, a) (5.aaT60 dis
2500
2600
TD3BC
TD3BC
Relabive NTK (9)
COL
Relalive NTK (16)
COL
2000
2000
1600
1500
1000
1000
500
500
300K
Steps
Steps
Figure 6: The NTK values of (() + noise) and (s', (s',a').
C Additional Results
C.1 Additional Evaluations
In addition the integration of FANS with TD3 presented in the main text, also applied FANS to AWAC. As shown Table AWAC FANS demonstrates a significant performance improvement the original AWAC algorithm across range of offline RL tasks. Notably, in challenging tasks such as hopper-meduum-epeer and walkerr2d-meddim-experr AWAC FANS boosts the scores from 52.7 1 110.3 and from 49.4 109.6, respectively. These improvements also come with significantly reduced variance, indicating more stable and reliable policy behavior.
only substantial but
Moreover, the average performance across all tasks increases from 67.7 to 83.9, further highlighting the general and consistent enhancement brought by the FANS module. These results validate the effectiveness of our approach across diverse environments and demonstrate its potential general-purpose purpose enhancement to existing offline RL methods.
Table 7:
Performance comparison D4RL locomotion tasks over the final ten evaluations and five seeds (normalized scores). We bold the highest mean.
Tasks
AWAC
AWAC + FANS
halfcheetah-meddimm
49,5 + 0.6
48.9 + 0.5
halichectah-mmciiu--rpppyy
44.7 + 0.7
44.9 + 0.2
halfcheetah-mediim-eeye
93.6 0.4
94.4 + 0.4
hopper-medium
74.5 + 9.1
76.2 + 6.1
hopper-mediiun-repayy
96.4 + 5.3
99.8 +: 2.0
hopper-mediim-eepper
52.7 + 37.5
110.3 + 0.7
walker2d-medium
66.5 + 26.0
81.9 + 0.7
walker2d--mediu--rppaa
82.2 + 1.1
88.9 + 3.8
walker2d-meedium-xxper
49.4 + 38.2
109.6 + 1.2
Average
67.7
83.9
15

---

C.2 Analyses of FASN's Structural Extensions
We conduct a systematic scaling analysis of the critic network by varying its depth (1-4) and width (64-512). For width scaling, the critic depth is fixed at blocks; for depth scaling, the critic width is set to 256, following our default setup.
The results show that the best performance is achieved when the critic has depth = and width 256, yielding high returns with low standard deviations across all three tasks (halfcheetah-mmedimn hopper-medium, and walker2d-medium), indicating strong stability.
Overall, increasing the depth and width of the critic generally leads to performance improvements, suggesting that higher model capacity enhances the representational power of the value estimator.
However, we also observe significant instability under certain configurations (e.g., depth = 4 or width 64/512), particularly on the hopper-ment medpiu and walker2d-medine tasks, where the standard deviations are notably large. This highlights the trade-off between model capacity and training stability, and the importance of balancing expressiveness with generalization.
In summary, properly scaling the critic network can significantly boost performance, but must be done with care avoid instability.
halfcheeran nodiume2
hosgenmedium-v22
welker2d-mediumw2
100
1.00
Norolloo: Scort
62.
30
60.0
375
80
525
120
206
512
120
256
512
20
04
128
256
512
Wotth
Figure 7: Performance of TD3 with FANS by varying width for the critic network.
halichestah mediumm-22
hop
walker2d medium
:10
105.0
B Scoe
100
Score
102
90
37.5
Crttic Depth
Crtic
Crtic
Figure 8: Performance of TD3 with FANS by varying depth for the critic network.
C.3 Learning Curves
The learning curves of TD3 + FANS for all tasks corresponding to Table in the main text are shown in the figure below.
16

---

harichert---e med umy
halt
123
102
Normkkdd Scors
1
No
0
07
84
04
Tip ning Stoos (x10%)
Trakning Stope 1x10%%
Tra Stope 10
140
125
120
120
700
Narraked Srowe
Namokaad Bore
5
0
844
D6
110
022
0.4
56
0.2
D2 waforr2d-mact scperoo
Training
Traking
Normalaua Store
10
(x30%)
Training 10%
Trahiing (x109)
Figure 9: Performance of TD3 with FANS on different mujoco tasks.
NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT. please:
17

---

Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope, as we provide a clear overview of the research objectives, methodology, and key findings, which are consistently supported and elaborated upon throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations our method throughout the main text, experiments, and especially in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
18

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical discussions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
This paper provides sufficient methodological details, experimental settings, and evaluation protocols to allow reproduction of the main results that support the paper's core claims and conclusions.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
19

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: This paper provides open access to both the data and code, along with clear and detailed instructions in the supplemental material, enabling faithful reproduction of the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/put wwtpcccccppccccceee haspllic/giieee http:/gguuddeeeeeeuuuuuiieeeeee htt///ooeSubiiss... colomisssions ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: This paper clearly specifies all the relevant training and testing details.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: This paper appropriately reports error bars and provides other relevant sta- tistical information to correctly convey the uncertainty and statistical significance of the experimental results.
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: This paper provides sufficient information in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
This research conducted in the paper fully adheres to the NeurlPS Code of
Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
21

---

Justification:
This work will not have any potential negative impact on society.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimesss considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable, as the paper does not involve the release of data or models with a high risk of misuse, such as pretrained language models, image generators, or scraped datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators and original owners of all assets used in the paper, including code, data, and models, are properly credited, and the licenses and terms of use are explicitly mentioned and fully respected.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
22

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papprssitthheeeocc thoode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pase re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new assets introduced in the paper are well documented, and the docu- mentation, along with the code, is provided in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots. if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsoureing experiments or research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
23

---

This paper does not involve study participants or human subject research that would require such disclosures or approvals.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: This paper does not use large language models (LLMs) as a core component of the research methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy autppe//beurrpsccc//ooorrrrreeee for what should or should not be described.
24


Reviews:
[{"review_id": "tFzYZOTdIf", "reviewer": "Reviewer_RRU4", "summary": "This paper presents FANS,  a generalization-oriented structured network framework that promotes flatter and robust policy learning by guiding the optimization trajectory through modular architectural design. FANS comprises four key components: (1) Residual Blocks, (2) Gaussian Activation, (3) Layer Normalization,  and (4) Ensemble Modeling.", "questions": "* FANS includes four key modules, but I couldn’t find any ablation study that shows which parts contribute the most. Could the authors provide insights or experiments on the relative importance of each component?\n* Since a new network architecture is introduced, it would be helpful to analyze or report how its training cost compares with previous offline RL baselines. This would provide insight into the practical efficiency of the proposed approach.\n* The comparison baselines are not the SOTA methods. Can you add more experiments with recent offline RL methods? such as FQL, A2PR, SORL. Moreover, the related work section does not discuss the recent methods mentioned above. Could you consider adding some discussion on these newer approaches?\n\nReference:\n\nPark, S., Li, Q., & Levine, S.. Flow Q-Learning. arXiv preprint arXiv:2502.02538.\n\nLiu, T., Li, Y., Lan, Y., Gao, H., Pan, W., & Xu, X.. Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning. In International Conference on Machine Learning (pp. 31406-31424). PMLR.\n\nEspinosa-Dice, N., Zhang, Y., Chen, Y., Guo, B., Oertell, O., Swamy, G., ... & Sun, W.. Scaling Offline RL via Efficient and Expressive Shortcut Models. arXiv preprint arXiv:2505.22866.", "limitations": "* Lack of ablation studies to analyze the contribution of each component.\n\n* Missing experimental comparison with state-of-the-art (SOTA) methods.\n\n* Insufficient main experimental results to fully validate the effectiveness of the proposed approach.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n* FANS can easily plug in a standard actor-critic framework.\n* This remarkably simple architecture of FANS achieves superior performance across various tasks.\n### Weaknesses\n* This paper lacks some experimental comparison with the SOTA methods.\n* The experimental evaluation is limited, as it only includes D4RL MuJoCo tasks. The performance of the proposed method on more challenging environments such as AntMaze or Adroit remains unclear.", "was_revised": true, "final_justification": "We sincerely appreciate the authors for providing the additional experimental results during the rebuttal. These results help clarify the effectiveness of the proposed method and support some of the claims made in the paper. Based on the clarifications and additional experiments provided during the rebuttal, I will update my score accordingly.", "rebuttal": "We sincerely appreciate your thoughtful and professional insights, which have greatly improved the quality of our work. In response to your concerns, we provide the explanations and answers below.\n\n**> 1. Response to Lack of Ablation Studies:**\n\nWe have **conducted ablation studies in Section 4.4** that systematically evaluate the contribution of each module within FANS to the overall performance. Additionally, **Appendix B.2 provides further structural analysis**, verifying the effectiveness of each component through parameter variations. For **a more detailed explanation of the ablation experiments, please also refer to our response to Reviewer 6MWm (#R4), point 3**. We hope these clarifications help to better illustrate the relative importance of the components.\n\n**> 2. Response to the Time Complexity and Computational Overhead of FANS:**\n\n**(1) Residual Block Complexity:**\n\nEach residual block in FANS (as shown in Table 1 of the main paper) comprises two linear layers, a LayerNorm, a Gaussian activation, and a residual connection.\n\n- The dominant operations are the two linear layers, each with a time complexity of $\\mathcal{O}(d^2)$, where $d$ is the hidden dimension.\n- LayerNorm, the Gaussian activation ($\\exp(-x^2)$), and the residual addition are all element-wise operations with complexity $\\mathcal{O}(d)$.\n\nThus, each block maintains an overall complexity of **$\\mathcal{O}(d^2)$**, consistent with standard MLP layers used in typical offline RL networks. The additional operations increase the constant factor but do not alter the asymptotic complexity. Additionally, these Residual Blocks are only applied to the critic network, remaining lightweight and efficient.\n\n**(2) Ensemble Modeling:**\n\nWe acknowledge that ensemble modeling introduces additional cost, scaling linearly with the number of models $M$, resulting in **$\\mathcal{O}(M \\cdot d^2)$** complexity. However:\n\n- The ensemble is only used in the critic, not the actor;\n- We adopt a small $M$ (e.g., 2-5) while still achieving strong performance $-$ significantly fewer than SAC-N, which requires 100-500 Q-networks, and EDAC, which requires 20-50 Q-networks;\n\n**(3) Practical Impact:**\n\nDespite the slight increase in per-iteration cost, FANS demonstrates stronger generalization and stability, requiring fewer training epochs overall. In Figure 6 of the Appendix, we show that FANS converges faster than strong baselines under the same compute budget.\n\n**Summary:**\n\nIn summary, while FANS introduces moderate constant-factor overhead due to its structured architectural design, the **overall time complexity remains $\\mathcal{O}(d^2)$**, matching conventional offline RL networks. The improvements in generalization and training robustness justify this trade-off. Bellow the table \"Time Complexity and GPU Consumption Comparison\":\n\n| **Method**     | **Critic Architecture**                                         | **#Q Networks** | **Per-Step Time Complexity**     | **GPU Memory Footprint** | **Notes**                                                                 |\n|----------------|------------------------------------------------------------------|------------------|----------------------------------|---------------------------|----------------------------------------------------------------------------|\n| **CQL**        | Standard MLP                                                    | 2                | $\\mathcal{O}(d^2)$               | ★☆☆☆☆                    | No ensemble; low cost but prone to overestimation                         |\n| **EDAC**       | Ensemble + Gradient Decorrelation                               | 20–50            | $\\mathcal{O}(M \\cdot d^2)$       | ★★★★☆                    | Strong baseline but expensive ensemble                                    |\n| **SAC-N**      | Massive Ensemble                                                | 100–500          | $\\mathcal{O}(M \\cdot d^2)$       | ★★★★★                    | High Q-network redundancy, heavy computational burden                     |\n| **FANS (Ours)**| Residual Blocks + Gaussian Act + LayerNorm + Small Ensemble     | 2–5              | $\\mathcal{O}(d^2)$ + $\\mathcal{O}(M \\cdot d^2)$ | ★★☆☆☆         | Structured critic only; lightweight design; converges faster              |\n\nWe will include detailed time complexity analysis and empirical runtime comparisons in the revised version.\n\n**> 3. Response to Missing Experimental Comparison with state-of-the-art (SOTA) Methods:**\n\nThank you for the valuable suggestion. As for A2PR, we directly extracted their MuJoCo Gym results from their original paper. The comparison between FANS and A2PR is summarized in the table below, where FANS outperforms A2PR on 7 out of 9 tasks. We will include this comparison in the revised manuscript’s main text.\n\n|               | ha-m           | ho-m            | wa-m           | ha-mr          | ho-mr           | wa-mr          | ha-me          | ho-me           | wa-me           |\n|---------------|----------------|-----------------|----------------|----------------|-----------------|----------------|----------------|-----------------|-----------------|\n| A2PR          | **68.61±0.37** | 100.79±0.32    | 89.73±0.60     | **56.58±1.33** | 101.54±0.90    | 94.42±1.54    | 98.25±3.20     | 112.11±0.32    | 114.62±0.78    |\n| **TD3+FANS**    | 66.6±0.8      | **104.6±0.9**   | **101.0±1.6**  | 55.9±1.5       | **103.2±1.1**   | **98.3±2.0**   | **102.8±3.4**  | **113.3±1.4**   | **118.1±0.4**   |\n\nRegarding FQL and SORL, we note that neither method reports results on the standard MuJoCo benchmark, which serves as the primary evaluation suite in our study. As a result, a direct empirical comparison was not feasible under our current experimental setup. Nevertheless, we greatly value these contributions and will explicitly include a discussion of both methods in the Related Work section of the revised version, clarifying how they relate to our approach. We are grateful to you for pointing out these important works.\n\n**> 4. Response to Insufficient Main Experimental Results:**\n\nDue to the limited time available during the rebuttal phase, we have conducted a preliminary set of experiments on the AntMaze and Adroit tasks. The results are shown in the table below. More comprehensive results will be provided in the revised version of the paper.\n\n**(1) AntMaze:**\n\n|               | antmaze-umaze | antmaze-umaze-diverse | antmaze-medium-play | antmaze-medium-diverse | antmaze-large-play | antmaze-large-diverse |\n|---------------|---------------|-----------------------|---------------------|------------------------|--------------------|-----------------------|\n| TD3+BC         | 70.75±39.18 | 44.75±11.61         | 0.25±0.43         | 0.25±0.43            | 0.00±0.00        | 0.00±0.00           |\n| **TD3+FANS**          | **87.75±10.48**  | **72.25±7.54**           | **3.25±1.30**          | **5.75±0.70**             | 0.00±0.00        | 0.00±0.00           |\n\n**(2) Adroit:**\n\n|               | pen-human  | pen-cloned | door-human | door-cloned | hammer-human | hammer-cloned | relocate-human | relocate-cloned |\n|---------------|---------------|-----------------------|---------------------|------------------------|--------------------|-----------------------|--------------------|-----------------------|\n| TD3+BC         | -3.88±0.21 | 5.13±5.28    | -0.33±0.01   | -0.34±0.01  | 1.02±0.24       | 0.25±0.01         |-0.29±0.01      | -0.30±0.01         |\n| **TD3+FANS**          | **55.16±5.49**  | **70.43±6.31**     | **9.72±6.13**       | **10.37±6.73**             | **2.24±0.18**        | **1.52±0.38**           |**0.10±0.09**        | **0.17±0.59**  |"}, {"review_id": "hYrmsZexkU", "reviewer": "Reviewer_jCXx", "summary": "This work focuses on architectural design in offline reinforcement learning and introduces FANS (Flatness-Aware Network Structure) to optimize neural networks by leveraging the flatness of the loss landscape. The work proposes FANS to guide training toward flatter regions of the loss landscape, which often relate to better generalization performance. FANS combines residual blocks, Gaussian activation functions, layer normalization, and ensemble techniques to improve generalization. The paper provides empirical evidence demonstrating FANS's effectivness across various datasets.", "questions": "- How is the computational overhead of FANS?\n- The empirical results primarily emphasize performance. Is there a way to demonstrate that FANS effectively guides training dynamics toward flatter regions of the loss landscape? I understand it is difficult to visualize the loss landscape in RL, but if such results exist, they would strengthen this paper.\n- How many hyperparameters does the method introduce? How sensitive is the method to hyperparameter choices?", "limitations": "A more explicit discussion of the study's limitations would further enhance the paper's academic rigor.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- Instead of typical algorithmic development, the paper introduces architectural design to enhance neural network generalizstion in offline reinforcement learning, exploring a relatively under-investigated area.\n- The empirical performance of FANS is impressive, and ablation studies are included to confirm the effectiveness of each of the four components in FANS.\n- The proposed method is compatible with various offline reinforcement learning algorithms and can be easily integrated into existing ones.\n- The paper is well-written, with clear explanations of the method and results, making it easy to follow.\n\n## Weaknesses\n- In the limited data experiments (Figure 3), to test the generalization ability, FANS should be compared with other offline RL methods rather than the online method TD3, which is known to suffer in the offline setting.", "was_revised": true, "final_justification": "I thank the authors for their detailed response. The proposed method for measuring landscape flatness is reasonable, and the time complexity comparison is clear. I encourage the authors to include both the time complexity analysis and the loss landscape results in the revised version of the paper, as these additions would further enhance the completeness of the work. I think that FANS presents a simple yet effective architectural design for offline RL that achieves strong performance. Taking the rebuttal into account, I have updated my score accordingly.", "rebuttal": "We extend our sincere gratitude for your inspiring and professional insights, which are invaluable in enhancing the quality of our work. In response to your concerns, we report below the explanations and answer your questions accordingly.\n\n**> 1. Response to Computational Overhead of FANS:**\n\nWe provided a detailed analysis of FANS’s computational overhead in our **response to Reviewer RRU4 (#R1), point 2**. We hope this is helpful to you.\n\n**> 2. Response to Hyperparameter Count and Sensitivity:**\n\n**FANS introduces minimal additional hyperparameters compared to standard offline RL baselines**. Its architectural components are largely self-contained and do not require extensive tuning. The table below summarizes the key components introduced by FANS and whether they introduce new tunable hyperparameters.\n\n| **Component**        | **Hyperparameter**     | **Notes**                                                                 |\n|----------------------|------------------------|---------------------------------------------------------------------------|\n| Residual Block       | None                   | Structural replacement for MLP layers; no additional tunable parameters. |\n| Gaussian Activation  | None                   | Fixed as $f(x) = \\exp(-x^2)$; no hyperparameter introduced.               |\n| Layer Normalization  | None                   | Standard layer-wise normalization; no tuning typically required.         |\n| Ensemble Modeling    | Number of models $M$   | Primary additional hyperparameter; typically $M=2\\sim5$.                  |\n\n**FANS also exhibits low sensitivity to hyperparameter choices**. The use of residual connections and layer normalization enhances training stability and enables deeper architectures without introducing optimization instability. The fixed Gaussian activation promotes smoother gradients and helps reduce sensitivity to learning rates and batch sizes. Overall, FANS maintains robust performance across a broad range of hyperparameter settings, alleviating the need for exhaustive tuning.\n\n**> 3. Response to Flatter Regions of the Loss Landscape:**\n\nYour suggestion is highly valuable. Considering that we are unable to submit a PDF response, we outline below a proposed approach to illustrate whether FANS effectively guides training dynamics toward flatter regions of the loss landscape:\n\n(1) **Selection of Converged Policies:**  \n   We identify a point in training where the loss stabilizes and performance metrics (e.g., policy returns) no longer change significantly, indicating convergence. At this stage, we fix the parameters $\\phi$ of the policy network $\\pi_\\phi$.\n\n(2) **Local Surface Construction:**  \n   We randomly sample a batch of states from the replay buffer and apply small perturbations to each state. For each perturbed state, we compute the corresponding action using the fixed policy network $\\pi_\\phi$. The original (unperturbed) state's action is treated as the reference optimal action.\n\n(3) **Surface Visualization:**  \n   We construct a 2D surface where the *x*- and *y*-axes correspond to the direction and magnitude of perturbations in the state space, and the *z*-axis represents the normalized Euclidean distance between the perturbed action and the reference action. Concave points on this surface correspond to the optimal actions. A flatter surface indicates that the policy output is less sensitive to input perturbations, suggesting greater local robustness and a smoother loss landscape.\n\n(4) **Comparative Analysis:**  \n   We perform this analysis on policies trained *with* and *without* FANS. By comparing the resulting action surfaces, we aim to visually demonstrate that FANS leads to a flatter, smoother local landscape $-$ supporting the claim that it enhances training stability and generalization.\n\nAlthough directly visualizing the high-dimensional loss landscape in reinforcement learning is inherently difficult, this approach serves as an interpretable proxy. We will include such comparative plots in the final version of the paper to strengthen the empirical support for FANS.\n\n**> 4. Response to Weakness: Inappropriate Baseline in Figure 3:**\n\nThank you for your valuable feedback. Reviewer 6MWm (#R4) has raised a similar concern, and we kindly refer you to our detailed **response to #R4, point 1**, for your reference."}, {"review_id": "nXRJrF41Kv", "reviewer": "Reviewer_NuBS", "summary": "The paper proposes FANS (Flatness-Aware Network Structure), a plug-in critic architecture for offline actor-critic RL that swaps the usual MLP for residual blocks with smooth Gaussian activations, LayerNorm, and a small ensemble.", "questions": "Please respond to the weakness described above. I am happy to raise my score based on the further discussions.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strength\n\n* **Architectural novelty** – FANS is the first offline-RL approach that combats OOD value over-estimation *solely* through a flatness-aware critic architecture (residual blocks + Gaussian activations + LayerNorm + small ensemble), in contrast to prior work that inserts explicit conservative penalties such as CQL and IQL.\n* **Hyper-parameter-free simplicity** – because the vanilla TD3 loss functions remain unchanged, FANS eliminates the extra tuning knobs that regularizer methods require (e.g., CQL’s penalty weight, IQL’s expectile τ) and can be dropped into existing actor-critic pipelines without any objective-level modifications.\n* **Strong, complementary performance** – on all nine D4RL MuJoCo tasks the plain TD3 + FANS agent surpasses state-of-the-art conservative baselines, and when FANS is combined with those same baselines it delivers further gains, showing both superior standalone effectiveness and orthogonality to regulariser techniques.\n\n## Weaknesses\n\n* **On motivation**\n\n  I don't quite understand the motivation of the paper. While FANS clearly outperforms conservative-regularizer baselines, it is not obvious why an architectural change is the necessary remedy. Could the authors please spell out the fundamental limitations of regularizer-based methods (if any) and explain why these issues cannot be mitigated without altering the network architecture ? This would make the motivation much more compelling.\n\n* **On conceptual framing**\n\n  The paper’s central narrative is that FANS “steers optimization toward flatter minima,” which, in turn, mitigates OOD over-estimation. Yet the manuscript doesn't quantify flatness (e.g., Hessian trace) nor compares those metrics against regularizer baselines. As a result, how architecture impacts flatness and then further affects robustness remains unclear to me.", "was_revised": true, "final_justification": "Thank you for the clarification. I would like to increase my score to 4.", "rebuttal": "Thank you for your thoughtful and inspiring comments, which will help us greatly in improving the quality of our work. We provide discussions and explanations about your concerns as follows.\n\n**> 1. Response to Motivation and the Role of Architecture:**\n\nWe thank you for your insightful question. While regularizer-based methods (e.g., conservative Q-learning, behavior cloning constraints) have proven effective in offline RL by mitigating extrapolation errors and discouraging OOD actions, they primarily operate at the **objective level** by modifying the loss function. These methods aim to influence learning behavior via explicit penalties or constraints, often without considering the inductive biases introduced by the network architecture itself.\n\nHowever, a fundamental limitation of regularizer-based approaches is their inability to directly control the **geometry of the loss landscape** or the **sharpness of the learned solution**. In the offline RL setting, where learning relies heavily on bootstrapped targets and limited data coverage, sharp or brittle solutions can lead to unstable training and poor generalization. Regularizers may reduce certain types of estimation error but do not inherently promote **flatter minima** or **smoother optimization dynamics**.\n\nOur motivation for exploring an **architectural remedy** arises from this observation. The FANS framework is designed to structurally guide the optimization process toward **flatter and more robust regions** of the solution space. Components such as residual blocks, Gaussian activations, and layer normalization are known to promote gradient smoothness and stability, while lightweight ensemble modeling helps reduce estimation variance. These architectural choices introduce inductive biases that **complement, rather than replace,** regularization.\n\nImportantly, our goal is not to position architecture as a replacement for regularization-based methods, but rather as an **orthogonal and complementary** perspective. In principle, architectural and objective-level approaches can be combined. In this work, however, we isolate the architectural effects to highlight their **standalone benefits** and to motivate further exploration of **structure-aware design** in offline RL.\n\nWe will clarify this motivation more explicitly in the revised manuscript.\n\n**> 2. Response to Quantifying Flatness and the Conceptual Framing:**\n\nWe sincerely appreciate your insightful comments, which indeed highlight aspects not covered in the initially submitted manuscript. Thank you for bringing these to our attention. One of the core motivations of this work is to guide the optimization process toward **flatter minima** through **structural design**, thereby improving **generalization** in offline RL, especially under **distributional shift**. However, as the reviewer correctly noted, the current version of the paper does not explicitly include quantitative measures of flatness, such as the **Hessian trace**, **eigenvalues**, or **sharpness**, which is a limitation of our current work.\n\nWe chose not to include such analysis in the main paper for the following reasons: First, in RL, especially when using **bootstrapped targets** like Q-value estimates, the **loss surface is highly non-stationary**, making the direct computation of the **Hessian** unstable and computationally expensive. Second, this work focuses on understanding the role of **architecture itself** by isolating it from objective-level interventions. Therefore, we primarily evaluated improvements in **OOD generalization**, **estimation variance**, and overall policy performance as **indirect evidence** of improved flatness and robustness.\n\nWe fully agree that the causal link between **architectural design**, **loss surface flatness**, and **policy robustness** should be more rigorously demonstrated. Thus, in the revised version, we will supplement the paper with additional analyses to characterize the **flatness of the loss landscape**. Since we cannot include additional figures or appendices in the rebuttal, we have instead described our methodology for measuring landscape flatness in **Reviewer jCXx (#R2)’s comment 3** for reference."}, {"review_id": "6CPASivOkn", "reviewer": "Reviewer_6MWm", "summary": "This paper introduces FANS, a novel method for mitigating the Q-value overestimation problem in offline reinforcement learning through minimal network modifications. FANS integrates Residual connections, Gaussian Activations, Layer Normalization, and a model Ensemble. FANS combined with TD3 is shown to outperform other offline RL methods. Furthermore, the paper presents a series of ablation studies to validate the effectiveness of the FANS components.", "questions": "- Is there a specific reason why the four components of FANS are most effective when used together? It would be more convincing if you could provide an ablation study that demonstrates the synergy between these components, rather than just their cumulative benefit.\n- The paper's claim of reducing Q-value overestimation is not clearly substantiated against other offline algorithms. Could you please provide a direct comparison of the Q-values against other state-of-the-art offline RL methods to better illustrate the improvements?\n- To demonstrate the generalizability of your method, could you provide performance comparisons against other offline algorithms on the D4RL **AntMaze** and **Adroit** benchmark suites?\n- The FANS architecture does not seem to be inherently limited to the offline RL setting. Have you considered if this architecture could also improve performance in other reinforcement learning paradigms where distribution shift is a key problem, such as offline-to-online RL, continual RL, or multitask RL?", "limitations": "The authors addressed all limitations.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 1, "strengths_and_weaknesses": "### **Strengths**\n\n- **Clarity and Presentation:** The paper is exceptionally well-written, and the proposed method is presented with great clarity. This makes the core ideas and contributions easy for the reader to understand and follow.\n- **Simplicity and Effectiveness:** A key strength of this work is its simplicity. The FANS method achieves significant performance improvements in offline reinforcement learning by introducing what appear to be minimal, yet effective, modifications to the network architecture.\n\n### **Weaknesses**\n\n- **Insufficient Ablation and Analysis of Components:** The paper lacks a rigorous experimental analysis of the FANS components (Residual connection, Gaussian Activation, Layer Normalization, and Model Ensemble). The current experiments do not adequately explain *why this specific combination* is critical or how the components synergize to reduce overestimation. For instance, there is no analysis showing that this combination is superior to other possible combinations or that it maximally reduces overestimation. As it stands, the performance gains could be interpreted as merely the cumulative benefit of four independent techniques, rather than a cohesive, synergistic solution.\n- **Limited Scope of Evaluation Environments:** The empirical evaluation is conducted on a narrow set of environments. To robustly demonstrate the generalizability of FANS, it is crucial to include comparisons with other algorithms on more diverse and challenging offline RL benchmarks, such as the D4RL **AntMaze** and **Adroit** suites. It is common for algorithms to perform well on MuJoCo locomotion tasks but struggle with AntMaze or Adroit, so evaluation on these benchmarks is critical to validate the method's effectiveness.\n- **Unclear Justification in Overestimation Analysis (Figure 2):** The intent of the Q-value overestimation experiment shown in Figure 2 is difficult to interpret. While the goal seems to be showing that TD3+FANS reduces Q-function overestimation, comparing it to online RL algorithms like the standard TD3 and SAC is not a meaningful benchmark. The dynamics of Q-value learning are fundamentally different in online versus offline settings. To be convincing, the analysis should demonstrate the advantages of FANS relative to other leading *offline* RL algorithms.\n- **Inappropriate Baseline in Limited-Data Experiments (Figure 3):** The validation under a limited-data setting (Figure 3) suffers from a similar issue. Comparing an offline method to the online TD3 algorithm is not a meaningful evaluation of data efficiency. An online algorithm's performance is not a relevant baseline for an offline method. The experiment should instead demonstrate that FANS maintains a more stable and effective performance compared to other TD3-based *offline* algorithms when trained on a small fraction (e.g., 5%) of the available data.", "was_revised": true, "final_justification": "**Final Justification for Score (Score: 3)**\n\nAfter considering the authors’ rebuttal and reflecting on the discussion, I provide the following justification for my score.\n\n**Resolved Issues:**\n- The authors clarified the motivation behind comparing TD3 and TD3+FANS in Figures 2 and 3. I now have a better understanding that these experiments aim to demonstrate the effectiveness of the proposed structured enhancement.\n- Through additional baselines and experiments in diverse environments, the authors have shown that FANS performs well not only in MuJoCo but also in general offline RL settings, outperforming other algorithms.\n\n**Unresolved Issues:**\n- In the overestimation experiment (Figure 2), while the authors demonstrate that FANS reduces Q-value overestimation compared to TD3, they do not sufficiently address whether other offline RL methods would show similar improvements. It remains unclear why FANS should perform better than other offline RL algorithms specifically in mitigating Q-value overestimation.\n- The authors correctly state that simply lowering Q-values for OOD data does not guarantee better performance. However, they do not provide a clear conceptual or quantitative criterion for what constitutes “appropriate” Q-value suppression, nor do they convincingly demonstrate that FANS satisfies such a criterion.\n\n**Summary for Authors:**\n- The rebuttal helped clarify the motivation and design of FANS.\n- However, key concerns about baseline comparisons and the analysis of Q-value overestimation remain only partially addressed.\n- Given the partial resolution, I have increased my score to 3, reflecting a more favorable view—but I still have some remaining concerns.", "rebuttal": "Thank you for your insightful comments. To comprehensively address your concerns, we will explain the following aspects in order: **inappropriate baselines**, **methods to reduce Q-value overestimation**, **ablation studies**, **discussions on other paradigms**, and **additional experiments**. The order may differ from your questions, but we aim to thoroughly resolve your main concerns. We sincerely appreciate the time and effort you have invested during the rebuttal process.\n\n**> 1. Response to Inappropriate Baseline:**\n\nAs pointed out in the paper *Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction*, online algorithms such as TD3 and SAC often **struggle to perform effectively in offline settings**. This has led to the development of offline variants like TD3+BC and SAC-N. However, these methods are **fundamentally built upon online algorithms**, enhanced through additional regularization or structural modifications. Similarly, our proposed **FANS framework is a structured enhancement tailored for offline RL**. TD3+FANS and TD3+BC can thus be viewed as different extensions of the same base algorithm.\n\nIt is important to note that applying FANS on top of existing offline algorithms like TD3+BC would introduce **conceptual conflicts**. Specifically, BC regularization **enforces conservative behavior** by constraining the learned policy to stay close to the behavior policy, whereas FANS aims to **improve generalization beyond the behavior policy**. These objectives are inherently at odds. Therefore, we chose to apply FANS directly to the base online algorithm TD3 to **better isolate and evaluate the contribution of our structured framework**, without interference from other offline-specific techniques.\n\nThat said, to further address your concern, we have **added experiments using TD3+BC as a baseline**. The below results support our analysis and demonstrate the validity of our design choice.\n\n| |ha-m|ho-m|wa-m|ha-mr|ho-mr|wa-mr|ha-me|ho-me|wa-me|\n|-|-|-|-|-|-|-|-|-|-|\n| TD3+BC+FANS|64.4±2.8|63.5±43.1|13.0±2.4|54.1±1.3|73.6±25.9| 58.1±33.3|68.3±7.3|18.4±7.0|13.6±5.0|\n| **TD3+FANS** |**66.6 ± 0.8**|**104.6±0.9**|**101.0±1.6**|**55.9±1.5**|**103.2±1.1**|**98.3±2.0**|**102.8±3.4**|**113.3±1.4**|**118.1±0.4**|\n\nMoreover, we would like to highlight that our **simple combination of TD3 and FANS achieves new SOTA performance** compared to a wide range of baselines, including recent SOTA methods and those focusing on generalization. Following the suggestion from Reviewer RRU4 (#R1), we have also added comparisons with several of these latest methods. Please refer to our **response to #R1, point 3**, for more details.\n\n**Regarding the concern about the \"inappropriate baseline\" in Figure 3**. The primary purpose of this experiment is to provide an intuitive illustration of how FANS can significantly improve the generalization ability of a base algorithm (e.g., TD3) under limited data conditions. **Figure 3 shows that while TD3 almost completely fails in this setting, FANS still achieves stable and effective performance, highlighting the potential of our structured enhancement in low-data regimes**. We agree that comparing FANS with other TD3-based offline algorithms under limited data could provide a more direct evaluation of data efficiency. While we emphasize that this experiment is not intended as a comprehensive SOTA comparison, we will consider adding such comparisons in future versions to further demonstrate FANS’s effectiveness in small-data scenarios. That said, **the focus of this experiment is to serve as a qualitative analysis that emphasizes the stability and generalization benefits FANS brings to the base policy**. In other parts of the paper, we have already conducted systematic comparisons between FANS and various mainstream generalization methods.\n\n**> 2. Response to Reduce Q-value Overestimation:**\n\nThe purpose of Figure 2 is to demonstrate the effect of the FANS framework in reducing Q-value overestimation. One common misconception is to assume that lower Q-values for OOD data are always better. In reality, what matters is the relative magnitude of Q-values $-$ i.e., whether high Q-values for OOD data are properly suppressed. **Therefore, we should focus on whether the Q-value curve for OOD data exhibits signs of Q-value explosion**, which is a key indicator of overestimation. In Figure 2, it is evident that the online TD3 algorithm suffers from a Q-value explosion for OOD data, while introducing FANS to TD3 effectively prevents this issue. This highlights FANS’s ability to stabilize value estimation under distribution shift.\n\nIn addition, **directly comparing the absolute scale of Q-values is not always meaningful**, as the values can differ by orders of magnitude due to implementation-specific factors. In FANS, the use of residual connections helps retain the original input’s scale, and the second linear layer adds new feature representations, which tends to maintain or even amplify the output magnitude. Moreover, LayerNorm normalizes each sample to have zero mean and unit variance within a feature dimension. While it does not change the overall expressive capacity, it can compress intermediate feature scales, thereby indirectly affecting the final Q-value range.\n\n**Another common misconception is that lower Q-values for OOD data necessarily lead to better performance**. As shown in the MCQ paper, overly conservative handling of OOD data can suppress potential generalization, leading to suboptimal outcomes. Thus, appropriately controlling Q-values for OOD data may be more promising, although the optimal strategy remains an open question.\n\nIn summary, **the goal of our analysis is to show that FANS effectively reduces harmful Q-value overestimation for OOD samples**. For this reason, we directly compare TD3 and TD3+FANS in Figure 2. The online SAC curve, trained for 3 million steps, serves as a proxy for the \"true\" Q-value on OOD data. In contrast, directly comparing Q-values with other offline SOTA methods is not a suitable or reliable benchmark at this stage. **This experiment is also designed with reference to Sec. 5.2 \"Value Estimation Error\" from the paper *Improving Generalization in Offline Reinforcement Learning via Adversarial Data Splitting*.**\n\n**> 3. Response to Ablation Studies:**\n\nOur FANS framework is inspired by the well-established structured approach SimBa from the online setting. The structural differences between FANS and SimBa are as follows:\n\n|Comments|SimBa|FANS|\n|-|-|-|\n|Input Normalization Method|Running Statistics Normalization (RSNorm)|Dataset Mean Normalization|\n|Residual Feedforward Blocks|✔|✔|\n|Activation functions|ReLU|Gaussian|\n|Post-layer Normalization|✔|✔|\n|Ensemble|✘|✔|\n\nCompared to SimBa, FANS is specifically designed to enhance generalization in offline settings by additionally incorporating Gaussian activation functions and ensemble techniques. Therefore, on one hand, given that SimBa already provides a mature structured design for RL, there is no need to reiterate the analysis of the combination and effects of Residual Blocks and Layer Normalization.\n\nWe further supplement our work by presenting results of directly applying SimBa in the offline setting, i.e., TD3+SimBa, as shown in the table below:\n\n||ha-m|ho-m|wa-m|ha-mr|ho-mr|wa-mr|ha-me|ho-me|wa-me|\n|-|-|-|-|-|-|-|-|-|-|\n|TD3+SimBa|64.4±2.8|63.5±43.1|13.0±2.4|54.1±1.3|73.6±25.9|58.1±33.3|68.3±7.3|18.4±7.0|13.6±5.0|\n|**TD3+FANS**|**66.6±0.8**|**104.6±0.9**|**101.0±1.6**|**55.9±1.5**|**103.2±1.1**|**98.3±2.0**|**102.8±3.4**|**113.3±1.4**|**118.1±0.4**|\n\nThe additional components introduced on top of SimBa $-$ the Gaussian activation function and ensemble techniques $-$ are well demonstrated in the ablation experiments in Table 4 of this paper. For instance, in the ho-me task, while the full FANS achieves 113.3, removing any single component causes severe performance drops (best ablated score: 41.3 without LayerNorm), revealing **nonlinear synergies between components**. Moreover, different tasks rely on distinct components (e.g., Gaussian noise is crucial for ho-m but LayerNorm dominates wa-m), proving **our components dynamically collaborate rather than providing fixed incremental gains**. This emergent synergy explains why the complete FANS consistently outperforms all partial implementations across diverse tasks.\n\nMoreover, while SimBa applies residual blocks to both the actor and the critic, our FANS framework further analyzes the effects of applying residual blocks separately to the actor and the critic in offline environments, as detailed in Table 5 of this paper.\n\n**> 4. Response to Discussions on other Paradigms:**\n\nThis work focuses on offline RL and does not specifically discuss other RL paradigms. However, we believe that some of the design principles embodied by FANS $-$ such as guiding the optimization process toward flatter regions through structured design and reducing estimation variance to enhance generalization $-$ may offer valuable insights for other paradigms that are also heavily affected by distribution shifts. In offline-to-online RL, continual RL, or multitask RL, a core challenge is improving the policy’s adaptability to dynamically changing environments or tasks. The modular structure of FANS provides a potential architectural framework for further exploration of these challenges.\n\nIt is important to emphasize that applying FANS to these paradigms would require appropriate extensions or adaptations tailored to their specific problem characteristics, such as incorporating task-specific memory mechanisms, adaptive modules, or dynamic sharing structures, rather than directly applying the existing architecture. Therefore, systematic investigations of FANS in other paradigms are left for future work.\n\n**> 5. Response to Additional Experiments:**\n\nWe have supplemented the results on AntMaze and Adroit; please refer to our **response to Reviewer RRU4 (#R1), point 4**."}]

Author rebuttal:

Meta-review: This paper proposes a network architecture framework for improved out of data generalization in offline reinforcement learning. Unlike other offline RL papers, this paper focuses on new network architectures allowing the method to be easily integrated into standard actor-critic methods without altering the loss function. Empirical results on D4RL benchmark show that TD3 + FANS achieves consistent improvements over other baselines.

Strengths:

The focus of the paper is not to propose another offline RL loss function but rather simple architecture change.

The method achieves strong performances with a lot less Q network ensembles with only 2-5 Q networks compared to 20-25 in EDAC.

Weaknesses:

Flatness claims are primarily speculative and has no theoretical insight.

Experiments are reported only on D4RL benchmark. During the rebuttal additional experiments were added on more challenging AntMaze/Adriot environments.

During the rebuttal, the authors provided ablation studies clarifying the role of each component, extended comparison to newer baselines, additional results on more challenging environments, detailed analysis of computational efficiency. Importantly, the authors demonstrated that FANS achieves comparable or better generalization with 2–5 Q-networks versus 20–50 in EDAC, highlighting its efficiency. They also provided Q-value over/underestimation experiments and NTK-based generalization analyses. Several reviewers raised their scores after these clarifications, and consensus moved toward acceptance.

The decision is to accept. The proposed framework offers a simple yet effective architectural contribution for offline RL with strong empirical support. For the camera ready version, I strongly encourage the authors to 1) include the Q-value over/underestimation results, NTK analysis presented during rebuttal, 2) expand the experimental results for AntMaze/Adroit, 3) more explicitly connect architectural design to flatness claims.
Decision: Accept (poster)