Title: FANS: A Flatness-Aware Network Structure for Generalization in Offline Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: offline reinforcement learning

generalization

structuration
Abstract: Offline reinforcement learning (RL) aims to learn optimal policies from static datasets while enhancing generalization to out-of-distribution (OOD) data. To mitigate overfitting to suboptimal behaviors in offline datasets, existing methods often relax constraints on policy and data or extract informative patterns through data-driven techniques. However, there has been limited exploration into structurally guiding the optimization process toward flatter regions of the solution space that offer better generalization. Motivated by this observation, we present \textit{FANS}, a generalization-oriented structured network framework that promotes flatter and robust policy learning by guiding the optimization trajectory through modular architectural design. FANS comprises four key components: (1) Residual Blocks, which facilitate compact and expressive representations; (2) Gaussian Activation, which promotes smoother gradients; (3) Layer Normalization, which mitigates overfitting; and (4) Ensemble Modeling, which reduces estimation variance. By integrating FANS into a standard actor-critic framework, we highlight that this remarkably simple architecture achieves superior performance across various tasks compared to many existing advanced methods. Moreover, we validate the effectiveness of FANS in mitigating overestimation and promoting generalization, demonstrating the promising potential of architectural design in advancing offline RL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FANS, a novel method for mitigating the Q-value overestimation problem in offline reinforcement learning through minimal network modifications. FANS integrates Residual connections, Gaussian Activations, Layer Normalization, and a model Ensemble. FANS combined with TD3 is shown to outperform other offline RL methods. Furthermore, the paper presents a series of ablation studies to validate the effectiveness of the FANS components.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n- **Clarity and Presentation:** The paper is exceptionally well-written, and the proposed method is presented with great clarity. This makes the core ideas and contributions easy for the reader to understand and follow.\n- **Simplicity and Effectiveness:** A key strength of this work is its simplicity. The FANS method achieves significant performance improvements in offline reinforcement learning by introducing what appear to be minimal, yet effective, modifications to the network architecture.\n\n### **Weaknesses**\n\n- **Insufficient Ablation and Analysis of Components:** The paper lacks a rigorous experimental analysis of the FANS components (Residual connection, Gaussian Activation, Layer Normalization, and Model Ensemble). The current experiments do not adequately explain *why this specific combination* is critical or how the components synergize to reduce overestimation. For instance, there is no analysis showing that this combination is superior to other possible combinations or that it maximally reduces overestimation. As it stands, the performance gains could be interpreted as merely the cumulative benefit of four independent techniques, rather than a cohesive, synergistic solution.\n- **Limited Scope of Evaluation Environments:** The empirical evaluation is conducted on a narrow set of environments. To robustly demonstrate the generalizability of FANS, it is crucial to include comparisons with other algorithms on more diverse and challenging offline RL benchmarks, such as the D4RL **AntMaze** and **Adroit** suites. It is common for algorithms to perform well on MuJoCo locomotion tasks but struggle with AntMaze or Adroit, so evaluation on these benchmarks is critical to validate the method's effectiveness.\n- **Unclear Justification in Overestimation Analysis (Figure 2):** The intent of the Q-value overestimation experiment shown in Figure 2 is difficult to interpret. While the goal seems to be showing that TD3+FANS reduces Q-function overestimation, comparing it to online RL algorithms like the standard TD3 and SAC is not a meaningful benchmark. The dynamics of Q-value learning are fundamentally different in online versus offline settings. To be convincing, the analysis should demonstrate the advantages of FANS relative to other leading *offline* RL algorithms.\n- **Inappropriate Baseline in Limited-Data Experiments (Figure 3):** The validation under a limited-data setting (Figure 3) suffers from a similar issue. Comparing an offline method to the online TD3 algorithm is not a meaningful evaluation of data efficiency. An online algorithm's performance is not a relevant baseline for an offline method. The experiment should instead demonstrate that FANS maintains a more stable and effective performance compared to other TD3-based *offline* algorithms when trained on a small fraction (e.g., 5%) of the available data.\n\nQUESTIONS:\n- Is there a specific reason why the four components of FANS are most effective when used together? It would be more convincing if you could provide an ablation study that demonstrates the synergy between these components, rather than just their cumulative benefit.\n- The paper's claim of reducing Q-value overestimation is not clearly substantiated against other offline algorithms. Could you please provide a direct comparison of the Q-values against other state-of-the-art offline RL methods to better illustrate the improvements?\n- To demonstrate the generalizability of your method, could you provide performance comparisons against other offline algorithms on the D4RL **AntMaze** and **Adroit** benchmark suites?\n- The FANS architecture does not seem to be inherently limited to the offline RL setting. Have you considered if this architecture could also improve performance in other reinforcement learning paradigms where distribution shift is a key problem, such as offline-to-online RL, continual RL, or multitask RL?\n\nLIMITATIONS:\nThe authors addressed all limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes FANS (Flatness-Aware Network Structure), a plug-in critic architecture for offline actor-critic RL that swaps the usual MLP for residual blocks with smooth Gaussian activations, LayerNorm, and a small ensemble.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n\n* **Architectural novelty** – FANS is the first offline-RL approach that combats OOD value over-estimation *solely* through a flatness-aware critic architecture (residual blocks + Gaussian activations + LayerNorm + small ensemble), in contrast to prior work that inserts explicit conservative penalties such as CQL and IQL.\n* **Hyper-parameter-free simplicity** – because the vanilla TD3 loss functions remain unchanged, FANS eliminates the extra tuning knobs that regularizer methods require (e.g., CQL’s penalty weight, IQL’s expectile τ) and can be dropped into existing actor-critic pipelines without any objective-level modifications.\n* **Strong, complementary performance** – on all nine D4RL MuJoCo tasks the plain TD3 + FANS agent surpasses state-of-the-art conservative baselines, and when FANS is combined with those same baselines it delivers further gains, showing both superior standalone effectiveness and orthogonality to regulariser techniques.\n\n## Weaknesses\n\n* **On motivation**\n\n  I don't quite understand the motivation of the paper. While FANS clearly outperforms conservative-regularizer baselines, it is not obvious why an architectural change is the necessary remedy. Could the authors please spell out the fundamental limitations of regularizer-based methods (if any) and explain why these issues cannot be mitigated without altering the network architecture ? This would make the motivation much more compelling.\n\n* **On conceptual framing**\n\n  The paper’s central narrative is that FANS “steers optimization toward flatter minima,” which, in turn, mitigates OOD over-estimation. Yet the manuscript doesn't quantify flatness (e.g., Hessian trace) nor compares those metrics against regularizer baselines. As a result, how architecture impacts flatness and then further affects robustness remains unclear to me.\n\nQUESTIONS:\nPlease respond to the weakness described above. I am happy to raise my score based on the further discussions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on architectural design in offline reinforcement learning and introduces FANS (Flatness-Aware Network Structure) to optimize neural networks by leveraging the flatness of the loss landscape. The work proposes FANS to guide training toward flatter regions of the loss landscape, which often relate to better generalization performance. FANS combines residual blocks, Gaussian activation functions, layer normalization, and ensemble techniques to improve generalization. The paper provides empirical evidence demonstrating FANS's effectivness across various datasets.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- Instead of typical algorithmic development, the paper introduces architectural design to enhance neural network generalizstion in offline reinforcement learning, exploring a relatively under-investigated area.\n- The empirical performance of FANS is impressive, and ablation studies are included to confirm the effectiveness of each of the four components in FANS.\n- The proposed method is compatible with various offline reinforcement learning algorithms and can be easily integrated into existing ones.\n- The paper is well-written, with clear explanations of the method and results, making it easy to follow.\n\n## Weaknesses\n- In the limited data experiments (Figure 3), to test the generalization ability, FANS should be compared with other offline RL methods rather than the online method TD3, which is known to suffer in the offline setting.\n\nQUESTIONS:\n- How is the computational overhead of FANS?\n- The empirical results primarily emphasize performance. Is there a way to demonstrate that FANS effectively guides training dynamics toward flatter regions of the loss landscape? I understand it is difficult to visualize the loss landscape in RL, but if such results exist, they would strengthen this paper.\n- How many hyperparameters does the method introduce? How sensitive is the method to hyperparameter choices?\n\nLIMITATIONS:\nA more explicit discussion of the study's limitations would further enhance the paper's academic rigor.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents FANS,  a generalization-oriented structured network framework that promotes flatter and robust policy learning by guiding the optimization trajectory through modular architectural design. FANS comprises four key components: (1) Residual Blocks, (2) Gaussian Activation, (3) Layer Normalization,  and (4) Ensemble Modeling.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n* FANS can easily plug in a standard actor-critic framework.\n* This remarkably simple architecture of FANS achieves superior performance across various tasks.\n### Weaknesses\n* This paper lacks some experimental comparison with the SOTA methods.\n* The experimental evaluation is limited, as it only includes D4RL MuJoCo tasks. The performance of the proposed method on more challenging environments such as AntMaze or Adroit remains unclear.\n\nQUESTIONS:\n* FANS includes four key modules, but I couldn’t find any ablation study that shows which parts contribute the most. Could the authors provide insights or experiments on the relative importance of each component?\n* Since a new network architecture is introduced, it would be helpful to analyze or report how its training cost compares with previous offline RL baselines. This would provide insight into the practical efficiency of the proposed approach.\n* The comparison baselines are not the SOTA methods. Can you add more experiments with recent offline RL methods? such as FQL, A2PR, SORL. Moreover, the related work section does not discuss the recent methods mentioned above. Could you consider adding some discussion on these newer approaches?\n\nReference:\n\nPark, S., Li, Q., & Levine, S.. Flow Q-Learning. arXiv preprint arXiv:2502.02538.\n\nLiu, T., Li, Y., Lan, Y., Gao, H., Pan, W., & Xu, X.. Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning. In International Conference on Machine Learning (pp. 31406-31424). PMLR.\n\nEspinosa-Dice, N., Zhang, Y., Chen, Y., Guo, B., Oertell, O., Swamy, G., ... & Sun, W.. Scaling Offline RL via Efficient and Expressive Shortcut Models. arXiv preprint arXiv:2505.22866.\n\nLIMITATIONS:\n* Lack of ablation studies to analyze the contribution of each component.\n\n* Missing experimental comparison with state-of-the-art (SOTA) methods.\n\n* Insufficient main experimental results to fully validate the effectiveness of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}