Title: Hadamax Encoding: Elevating Performance in Model-Free Atari
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Atari

Encoder Design
Abstract: Neural network architectures have a large impact in machine learning. However, in the specific case of reinforcement learning, network architectures have remained notably simple, as changes often lead to small gains in performance. This work introduces a novel encoder architecture for pixel-based model-free reinforcement learning. The Hadamax (\textbf{Hada}mard \textbf{max}-pooling) encoder achieves state-of-the-art performance by max-pooling Hadamard products between GELU-activated parallel hidden layers. 
Based on the recent PQN algorithm, the Hadamax encoder achieves state-of-the-art model-free performance in the Atari-57 benchmark. Specifically, without applying any algorithmic hyperparameter modifications, Hadamax-PQN achieves an 80\% performance gain over vanilla PQN and significantly surpasses Rainbow-DQN. For reproducibility, the full code is available on GitHub.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Hadamax encoder, a pixel-based neural network architecture for model-free reinforcement learning that combines max-pooling down-sampling, Hadamard products between parallel hidden layers, and GELU activations. Integrated into the Policy Quantization Network (PQN), Hadamax raises median human-normalized scores on Atari-57 by roughly 80% over vanilla PQN, surpasses Rainbow-DQN while training more than an order of magnitude faster, and transfers well to C51.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The reported 80% performance gain over the PQN baseline and the outperformance of Rainbow-DQN with >10× shorter training times demonstrate clear practical impact.\n- The encoder’s design is conceptually simple—replacing strides with max-pooling and inserting an element-wise Hadamard branch—so practitioners can drop it into existing codebases with minimal engineering effort, but enjoy massive performance gains.\n- The ablation study cleanly isolates the contributions of max-pooling, Hadamard products, and GELU activations, strengthening the causal link between the architectural decisions and the observed gains.\n\nWeaknesses\n- All experiments are confined to the ALE 200M benchmark, leaving open whether the encoder generalizes to domains that demand fine-grained spatial reasoning or continuous actions, such as robotics or locomotion related environments such as DMC, Maniskill and Adroit.\n- The study compares mainly to PQN, an algorithmic rather than architectural baseline, and omits direct comparison to architectural strategies such as SoftMoE [1] and network pruning [2] based approaches.\n- The architecture exploration is shallow: residual connections are excluded, depth is limited, and modern components such as attention or normalization variants are not considered.\n- Computational analysis focuses on wall-clock training time but omits memory usage and inference latency.\n\n[1] Mixtures of experts unlock parameter scaling for deep rl, Obando-Ceron et al., ICML 2024\n[2] In value-based deep reinforcement learning, a pruned network is a good network, Obando-Ceron et al., ICML 2024\n\nQUESTIONS:\n- An exploration of how Hadamax integrates with data augmentation techniques such as DrQ, as well as with pruned or mixture-of-experts encoders, could help determine the compatibility of the proposed architecture with modern modular approaches.\n- It would be helpful to understand the parameter allocation between the encoder and the policy/value head, and how this distribution compares with other architectural configurations.\n- An evaluation of Hadamax on continuous-control benchmarks, such as DMC or Adroit, could offer insight into its generalization capabilities in domains requiring fine-grained spatial reasoning.\n- Clarification on the exclusion of residual connections and deeper networks would be appreciated, along with discussion on whether such design choices were influenced by concerns over diminishing returns or optimization instability.\n- Investigating the effects of substituting max-pooling with strided convolutions while preserving Hadamard interactions—or vice versa—could further isolate the contribution of each architectural component.\n- A quantitative analysis of GPU memory consumption and inference throughput in comparison to Rainbow-DQN would provide a more complete picture of the computational trade-offs introduced by the Hadamard operations.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors focus on the problem of model free reinforcement learning applied to the Atari video game domain. To this end, the authors propose an architectural modification to the image based encoder of PQN. They apply max pooling in addition to Hadamard products to create what they deem \"Hadamax.\" The authors then evaluate their approach on the Atari video game dataset. They compare against regular PQN as well as the Rainbow algorithm, showing improvement over baselines. Furthermore, they also perform an ablation analysis showing the ablative effects of max pooling, Hadamard operations, and and GELUs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The approach is very simple and easy to implement. I believe that I would have few issues attempting to reproduce the results in the paper. \n\n2. The paper is well written, clear, and easy to understand.\n\n3. Performance over baselines is quite impressive considering the simplicity of the approach (an architectural change to the image encoder).\n\nWeaknesses:\n\n1. The scope of the evaluation is very narrow. The experiments focus only on Atari-57; there are no other domains used for evaluation. It is thus difficult to understand how well the performance improvement could generalize across other datasets.\n\n2. The technical novelty is quite limited. The approach is largely a tweaking of the image encoder within the existing PQN framework. \n\n3. The scope of models is also narrow. It is unclear how this approach could improve image encoders for other RL architectures or even encoders for other tasks such as image recognition.\n\nQUESTIONS:\n1. Can you please elaborate on the technical novelty of the approach?\n\n2. How could the approach be applied to different domains or model architectures? Do you believe that performance improvements may be gained with other datasets/models?\n\nLIMITATIONS:\nYes, the authors have discussed limitations in Conclusions and Future Work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new network architecture for pixel-based model-free RL called Hadamax encoder. It replaces ResNet as the visual encoder in the PQN algorithm. It achieves a new SOTA performance on Atari-57, while being much faster in terms of computation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n\n**Quality**\n- Paper written with high quality, \n- It is nice the performance improvement can be achieved by only changing the network architecture. \n- The insights such as that max-pooling might be overlooked in RL, \"average pooling achieves the opposite\" are interesting\n- Ablation is nice. \n- Many technical details are provided.\n\n**Clarity**\n- Paper is overall clear and easy to follow. \n\n**Significance**\n- The resulting performance and wall-clock time improvement are quite significant and convincing. \n\n**Originality**\n- While the proposed changes are relatively simple, it is novel to show that they can lead to very significant performance gain, which also brings new insights. \n\nWeaknesses\n\nMajor: \n- Paper can greatly benefit from further analysis of why certain components such as max-pooling can help so much? Currently there is only empirical result and the arguments are entirely based on the performance comparisons. Does it provide better visual features? Does it make training more stable? \n\n\nMinor:\n- The proposed method is faster but does it also take less computation per forward pass? I would suggest in the intro mention whether Hadamax encoder is faster per forward pass, or is it slower per forward pass but finish training at a smaller number of steps and thus leads to a smaller wall-clock time. Good to show comparison of total computation. \n- Can benefit from testing on more benchmarks such as any MuJoCo benchmarks with visual input. Less important. The results on Atari are already quite convincing and might be too much work.\n\nQUESTIONS:\n- Will such an architecture change also improve performance in e.g. standard computer vision benchmarks? Can it compete with the newest visual encoders? Why or why not?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new network architecture for RL on image-based inputs (Hadamax), which builds on PQN by using GELU activations, Hadamard products, and max pooling. The author(s) evaluate on Atari-57, comparing with and significantly outperforming Double DQN, C51, Rainbow, and PQN. Using Atari-10 the author(s) also show that Hadamax architecture is similarly effectively applied on top of C51, demonstrating that its benefits are not restricted specifically to PQN or the parallel training setting more generally.\n\nEffects on the representation learned at various layers of the network are empirically analyzed and an ablation study is performed moving between PQN and Hadamax PQN.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* As this is essentially “just” a network architecture change, and would be quite simple to implement and reproduce, the magnitude of performance improvement is extremely strong. This is two strengths, (1) very simple to implement / reproducible and (2) strong empirical performance.\n* The empirical work is also quite thorough, comparing with multiple baselines in the primary results and conducting well-chosen secondary experiments to supplement those results.\n* The paper is well written and easy to follow.\n\nWeaknesses:\nThese are really not significant weaknesses, and more just potential areas for improvement.\n\n* This is such a generally applicable architecture that not demonstrating it on more domains is arguably the biggest weakness, and missed opportunity, in the paper. This is simply because only evaluating on Atari begs the question if this is somehow especially well suited to that one domain. At the same time, when others try it on other image-based domains and (I expect) find that it does work, that finding won’t really get published. And as a result, in the best case scenario where this is the new architecture everyone wants to use, the knowledge that it works broadly will get delayed much longer because it wasn’t communicated in a published paper. If this gets in, please include in your talk/poster additional domains to get the word out.\n* After writing the above rant I realize the same logic applies to the type of base algorithm used with Hadamax: I would really like to see this used with an actor-critic / policy gradient algorithm.\n* There is some degree of muddying the waters with the focus in the writing on vectorized RL. I don’t think that is the main point, nor should it be entirely downplayed, but because of how the paper is structured it leads the reader in that direction for long enough to distract from the core contributions.\n* Figure 2 is great and the math is clear enough to implement, but I would have benefitted from another diagram/visual to dive deeper into the Hadamard product followed by max pooling component.\n\nQUESTIONS:\n* What do you think the biggest weakness or limitation of Hadamax is?\n* Could you explain why the contributions would necessarily be specific to pixel-based inputs?I suppose this is more a question about the Hadamard product component, but nonetheless I’ll rephrase as: Do you think a related approach could similarly benefit non-pixel-based domains and if not why is your reasoning?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}