Hadamax Encoding: Elevating Performance in Model-Free Atari
Authors: Jacob Eeuwe Kooi
Zhao Yang
Vincent Francois-Lavet
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Atari
Encoder Design
Primary area: reinforcement_learning

Abstract: Neural network architectures have a large impact in machine learning. However, in the specific case of reinforcement learning, network architectures have remained notably simple, as changes often lead to small gains in performance. This work introduces a novel encoder architecture for pixel-based model-free reinforcement learning. The Hadamax (\textbf{Hada}mard \textbf{max}-pooling) encoder achieves state-of-the-art performance by max-pooling Hadamard products between GELU-activated parallel hidden layers. 
Based on the recent PQN algorithm, the Hadamax encoder achieves state-of-the-art model-free performance in the Atari-57 benchmark. Specifically, without applying any algorithmic hyperparameter modifications, Hadamax-PQN achieves an 80\% performance gain over vanilla PQN and significantly surpasses Rainbow-DQN. For reproducibility, the full code is available on GitHub.

Hadamax Encoding: Elevating Performance in Model-Free Atari
Jacob E. Kooi jacobkooi92@t gmail.com Department of Computer Science Vrije Universiteit Amsterdam
Zhao Yang z.yang 3 @ vu.nl
Department of Computer Science Vrije Universiteit Amsterdam
Vincent François-Lavet vincent. www.coosllaeeeteveeeeee vu.nl Department of Computer Science Vrije Universiteit Amsterdam
Abstract
Neural network architectures have large impact in machine learning. However, the specific case of reinforcement learning, network architectures have remained notably simple, changes often lead small gains performance. This work introduces a novel encoder architecture for pixel-based model-free reinforcement learning. The Hadamax (Hadamard max-pooling) encoder achieves state-of-the-arr performance by max-pooling Hadamard products between CELU-activated parallel hidden layers. Based on the recent PQN algorithm, the Hadamax encoder achieves state-of- the-art model-free performance in the Atari-57 benchmark. Specifically, without applying any algorithmic hyperparameter modifications, Hadamax -PQN achieves 80% performance gain over vanilla PQN and significantly surpasses Rainbow-DQN. For reproducibility, the full code is available on GitHub
Introduction
Ever since reinforcement learning (RL)
Runtime (GPU hours)
algorithms (53] surpassed human play-
ers on the Atari-57 benchmark [6] 37
DON
DDON
38), progress has been driven mainly
by various algorithmic innovations [5 50
Hadamax-PON
PQN (Resnet-15)
Compared with the field supervised learning (SL., the deep learning compo- nents of RL have remained relatively simple, usually consisting of few convolutional layers (for image-based tasks) followed by fully connected lay- So far, the most common
PON
Median Human-Normalized Score
Figure 1: Performance versus GPU hours in the full Atari- 57 domain at 200M environment frames. The application of our Hadamard max-pooling encoder on PQN yields significant performance improvements of-the-art model-free method, Rainbow, while remaining more than an order of magnitude faster.
ers 38 27 encoder modification in image-based RL tasks has been the integration ResNet encoder [13), inspired by its wide use in supervised learning architec-
current state-
*Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

tures [25]
Several further approaches
have been explored to scale the deep learning architecture, but findings indicate that scaling pixel- based RI. remains significant challenge [1 42). and finds greater success in either low-dimensional
state-based continuous control 35
24
or complex model-based architectures 47 23
In this work, we revisit the assumption that modifications to deep learning architectures can not lead to significant improvements in RL. We build on top of the recent Parallelized Q-Network (PQN), which reinvented DQN to function without the replay buffer and target network, while profoundly increasing performance [7]. This is done by combining recent advances Hadamard representations
31 with max- max-pooling found in the ResNet encoder structures [25 13] Specifically, we augment the state-of- the-art art PQN algorithm with a Hadamard maxpooolln (Hadamax) encoder. The contributions
can be summarized as follows:
A novel deep learning architecture is proposed to improve usual pixel-based convolutional encoder architectures for model-free RI. This design shows alternative direction of encoder synthesis in RI., as compared to the widely used deeper ResNet architectures.
Without applying any algorithmic or hyperparameter modifications, Hadamax- PPN achieves 80% performance gain in the full Atari-57 suite over the recent PQN baseline [17. These changes allow Hadamax- -QQN to significantly surpass Rainbow-DQ2 [27] while remaining more than an order of magnitude faster, setting new state-of-the-ar for model-free RL on Atari.
Related Work
Different neural network architectures are applied in RL to enhance the performance in online
settings [37 5] [3] 27 [7] [5] as well as offline limited data settings [7 49,
10] 50 39]. In this
work, we focus on agents in the high-dimensional Atari-57 domain (6]. diverse and commonly-used challenging benchmark with discrete actions and pixel-based input.
Network development in RL for Atari: Deep Q-learning (DQN) [37 38) achieves human-level performance on Atari games for the first time in the RL history by using three convolution layers (with RcLU) followed by fulle-connected layers. Due to its simplicity and efficiency. this classic architecture is used for many later works, such as Double DQN (DDQN) [55], Ducling DQN (58),
Noisy DQN [4], Categorical DQN (C51) 5 and Sainbow-DQN 27.. The recent Parallelized
Q-Network (PQN) 17 algorithmically simplifies DQN and uses LayerNorm [4) to provably stabilize
optimization. C51 and R2D2 [29] enhance the output layer using categorical distributions
and a recurrent network, respectively. In the context of model-based RL, Recurrent State-Space Models (RSSM) 20
21 23), image augmentation (34) forward prediction [49 40. residual
architectures I 46 and transformers ( have also been explored to solve Atari Impala [33
introduces a deeper ResNet-15 encoder structure with residual blocks, allowing for high data efficiency under distributional training. BBF (50) further widens the Impala encoder, achieving state-of-the-art performance on the Atari-100k benchmark. SPR (49. using DQN's architecture with a self- celf-prediction objective, also improves data efficiency. For model- based methods, residual architectures 60 577, transformers [6]] and diffusion models are being increasingly leveraged to boost sample efficiency. Our work focuses on model-free agents in the Atari-57 benchmark, where relatively modest algorithmic architectures are used, and a large amount of environment interactions is allowed.
Speedups in RL: Since the development of JAX ()) parallel and vectorized training of reinforcement learning (RI.) agents has become a promising area of research, offering significant performance and scalability improvements. Physics simulation engines and tools that are compatible with JAX have emerged to support this paradigm, including Brax [16], a physics simulation engine optimized for high-speed differentiable environments; Gymnax (33 a lightweight, JAX-based version of classic Gym environments; Jumanji 8 suite of combinatorial and decision-makee makkin environments tailored for JAX; and EnvPool [59]. a high- gighthhuouuhhut environment execution engine with up to 20x speedup compared to Python. To complement these environments, growing ecosystem of reinforcement learning libraries built entirely in JAX has been developed. PureJaxRL (36] implements standard RL algorithms entirely end-to-end in JAX, enabling parallel execution across thousands of environments. JaXMARI. (45) focuses on multi-agent reinforcement learning, demonstrating strong acceleration of existing algorithms. Additionally, cleanr! (28). a library providing high-quality and reproducible RL

---

implementations, also includes several JAX-based implementations. Our work builds upon PQN [77] which leverages EnvPool and PureJaxRL, achieving greater efficiency compared conventional Py Tyrch-bbasd implementations. With the Hadamax encoder, we further architecturally improve PQN to the point that it significantly surpasses Rainbow-DQN, while remaining more than an order of magnitude faster.
3 Preliminaries
As background, we briefly explain general value-based RI and the recent PQN algorithm, which is extended with our proposed encoder.
3.1 Reinforcement Learning and Value-based Methods
We consider a Markov Decision Process (MDP), defined by the tuple < S, R. >> with state space S, action space A. transition function P, reward function R. and discount factor 7 E [0, 1). An agent in state E S at timestep t, taking action a, E A observes reward r2 R(s,, as) and next state $1+1 ~ (s,,, a.). The goal is learn an optimal policy x² S + A that can maximize the expected return ((s,)
> 8 = over all possible trajectories. Unlike policy- based or actor-critic methods 48 19 that optimize the policy, value-based methods [37 learn state-action value function ((s, a) Once the optimal Q-function is learned, the optimal policy is implicitly defined by selecting greedy actions x (s) argmax, (s, a). Q-learning is the most widely used value-based algorithm. learns Q(s, a) through temporal difference (TD) learning. The update rule is:
|>
((s,, a.) + Q(2,, ax) + air + w...aaeeeaaCC....
- ((s at)],
(1)
where is the learning rate. Over time, this iterative process allows the Q-function to converge the optimal value function Q" (s, a). from which the optimal policy can be derived.
Deep Q-Network (DQN) 37 extends Q-learning by using deep neural network to approximate the Q-function. The network is trained to minimize the difference between the predicted Q-values and the target values, typically using loss function such as mean squared eπrоr:
(()) Es, -u
e. ; Q(s a, B))
(2)
where 8 and f are the parameters of the Q-network and are the parameters of target network that is periodically updated. D is the experience replay buffer from which mini-batchnn are sampled.
3.2 Parallelized Q-Network (PQN)
PQN is a simplified deep online Q-learning algorithm. By parallelizing vectorized environments and normalizing neural networks (LayerNorm), PQN can stabilize the training even without target network and replay buffer. Moreover, it is compatible with pure-Gil UPU training, leading efficient training on Atari tasks. More specifically, PQN makes the following modifications compared to the original DQN:
Unlike the original DQN uses 1-step return, PQN leverages more stable X-return. The loss in Equation (2 thus becomes:
X-return:
(( Emzjs((re + (AG + (1 (1
Q(8,, ()))
where G is the X-return. When À 0 will be similar to Q-learning, and if À 1 is equivalent to the Monte Carlo update, which uses the full return until the episode ends.
LayerNorm: PQN adds LayerNorm for the output of convolution MLP layers before the ReLU activation functions, which helps stabilize the training process.
Removal of replay buffer and target network: Since the whole training process happens on GPU, removing the replay buffer can largely reduce memory and thereby accelerate training. As result of the training stability, the target network is also eliminated.
3

---

DQN
PQN
Hadamax
IMPALA
Input
Input
Input
Conv
ReLU
Conv
Comv
LayerNorm
Convv
ReLU
Conv
LayerNorm
GELU
MaxPool
ReLU
ReLV
Residual
x
MaxPool
Residual
1
LayerNorm
MIPP
RelU
ReLU
LayerNorr
MUP
MIl P
GELU
Re
MIPP
Figure 2: Encoder architectures of DQN, PQN. the proposed Hadamard max- max-pooling (Hadamax) encoder and the Impala 15 encoder (from left to right). the Hadamax encoder, down- sampling is facilitated by max-pooling operators. Furthermore, we apply Hadamard product between parallel representation layers. The implementation is straightforward and can be found in Appendix B These changes allow for substantial increase in algorithm performance, while keeping general encoder structure, convolutional depth and algorithmic hyperparameters unchanged.
Hadamax Encoder
The first human-level performance in the Atari-57 domain was achieved with the 'Nature' DQN encoder design [38]. The general effectiveness of this architecture, as well as the problems with scaling in deep RL, has led this architecture's 's use even in the modern state-of-the-art algorithms such as PQN [17]. In this section, we provide the reasoning and implementation of the proposed Hadamard max- pax-pooling augmentation of the original DQN encoder. For reproducibility purposes, we refer the reader to detailed implementation of the proposed architecture in Appendix B
4.1 Design Choice 1: Down-sampling by Max-pooling
As pixel-based observations are high-dimensional, the encoder must effectively compress the state representation enable the downstream RL algorithm to converge within reasonable number of updates. In the conventional DQN encoder, this compression is achieved by the convolutional operations (See Fig. 2 where the compression is determined by the convolutional kernel size and stride. In contrast, when examining the well-known and widely used Impala ResNet-15 encoder RL max-pooling responsible for the bulk of feature compression. The resulting effect that minimizing convolutional strides and adding max max-pooling allows for the selection of a more dense representation convolutional features, and subsequently emphasizes the strongest signals. Additionally, the use of max-pool max-pooling adds slight translation invariance to the important features. We therefore hypothesize that the use of max-poot max-pooling in RL is, although widely implemented in supervised learning, relatively overlooked. In the Hadamax encoder, convolutional down-sampling is therefore replaced by max- max-pooling operators. Furthermore, in contrast to the average- sarrage-pooling used by the original supervised learning ResNet architecture (25). the Hadamax encoder max- max-pools the final features before flattening to the linear layer. Since value functions RL should be able to show strong correlations with the most important features, pverage-pooling before the linear layer will achieve the opposite, as it smoothens out feature importance.
The max-poot max-pooling design choices; max- max-pooling and downsampling instead of convolutional down- sampling, followed by max-poolle max-pooling without wown-sampling before flattening, are thus respectively influenced by the ResNet-15 15 (Impala) RI. encoder and the original ResNet. However, in stark contrast both residual encoders mentioned, the Hadamax encoder remains shallow (3 convolutional layers), and therefore no residual connections need to be applied.

---

4.2 Design Choice 2: Application of Hadamard Representations
Although multiplicative interactions have been commonly used in Deep Learning architectures [52] (2, [I], their application in RL remains limited. Recent work however has shown that the effective rank (ER [32 [8] and downstream performance improved when training deep RL in the Atari domain, by defining hidden layers as Hadamard products [3]] Hadamard products between hidden layers enable richer high-dimensional interactions within the representation space, without increasing hidden layer dimensionality. This leads to more network capacity without explicitly scaling the network, which often unstable in RL. Specifically, any hidden layer 2 E Z, with layer depth j. will be the Hadamard product of two parallel layers connected to the preceding hidden layer 2²-1;
2² f - o f(=
(4)
where A is weight matrix, f(*) is nonlinear activation and the bias layers are left out for simplicity. As PQN employs layer normalization for training stability, and every representation is max max-pooled, the final Hadamax representation layers be defined
2 MP MMM(((LLVNNNN (E(F((LN(--111)) (LN))))
(5)
Where LN and MP represent layer normalization and max max-pooling, respectively. It is worth noting
that contrary recent work on Hadamard representations [31]. we show the possibility of successfully applying Hadamard products to zero-saturating activation functions such as ReLU or GELU [26]. We believe this is possible due the relative training stability increase of PQN over DQN, result of applying LayerNorm and the removal of the target network and replay buffer. This training stability correlates with minimal amount of dead neurons in the representation (51), which even leads the ability to do dementtwwsst multiplication of gero-saturating (sparse) neurons without increasing dead neurons.
4.3 Design Choice 3: Gaussian Error Linear Unit
The Gaussian Error Linear Unit (GELU) is used in various neural network architectures, the most notable applications being in transformer-base architectures such as BERT [22 and GPT 44
is defined as:
GELU(z)= = (x)
where (xx) is the cumulative distribution function of the standard normal distribution. Equivalently, it can be expressed using the error function as:
ReLUU
1.5
GELU
GELU(x) = 0.5x 1 erf
x
In contrast to the ReLU, which converts negative inputs to zero, GELU permits small negative values to pass through in softened form (See Fig. 3). allowing more stable gradient flow for negative inputs. Overall, GELU has been shown to
0.0
improve performance in various deep learning tasks, including computer vision and natural language processing [26]. the
Figure 3: ReLU and GELU.
Hadamax encoder, we therefore replace all the original ReLU activation functions with the GELU.
5 Experiments
We compare our agent against widely used model-free RL baselines across 57 Atari games. Through experiments, aim to answer: (i) do agents equipped with Hadamax encoders outperform those using conventional encoders? (ii) what are the reasons behind Hadamax 's superior performance?
(iii) what is the impact of each proposed design choice?
Baselines: We compare our method with the following baselines: (1) DQN 37
pioneer RI.
method that uses deep neural network play Atari, achieving human performance. (2) C51 (5),
5

---

Rainbow [27] state-of-the-an model-free method, combining various algorithmic and architectural techniques together. (3) PQN [7]. recent novel parallel Q-learning network without replay buffer and target network. In terms of performance, PQN is on par with C51, while remaining algorithmically less complex than DQN. Our final baseline is (4) PQN (ResNet-15), which combines PQN with the more complex Impala CNN architecture, used throughout modern state-of-the-ard RL
algorithms as a drop-in replacement for the conventional Nature encoder [33 50)
The full 57-game Atari domain 6 is used standardized benchmark for eval- uating our algorithm's 's performance. In line with best practices in the field, we focus on the me- dian human-none normalzze score over all 57 games 38
Environments:
To manage computational load, ablations are done on 40M frames, while compar- ison with baselines is done at the official 200M frame scores. Note that there can be relative dif- ferences between performances in 40M and 200M frames, as the epsilon-greets coefficient is scaled down over the total training time. An algorithm seed initialized to run for 40M frames will therefore have different convergence curve towards 40M than the same algorithm initialized for 200M frames seed. We refer the reader to more detailed descriptions of environments and implementations of baseline agents in Appendix C.
27 21 17..
Figure 4: The Atari-57 domain.
5.1 Hadamax-PONi Results
The full 200M frame training curves for PQN, PQN (ResNet-15) and Hadamax- -PON are shown in Fig. (left). The Hadamax encoder clearly yields benefits over the widely used Impala ResNet-15 encoder [13, and causes PQN to significantly surpass Rainbow-DQN [27]. Although the original paper shows that PQN able to beat Rainbow-DQN when training for around 260M environment frames [17]. Hadamax-PON PQN reaches this score at around 90M frames. Another commonly used scoring method, the Atari-57 score profile, can be seen Fig. (right). Note that the scores used in this research for DDQN, C51 and Rainbow have been taken from the original papers, and are generally higher than their practical implementations on various GitHub repositories. For details on how to compute the median duman-normalizer score and the Atari score profile, we refer the reader to Appendix
Atari (57 Games, 200M)
Atari (57 Games, 200M)
100
Median Numan-Normalisn Score
Hadamax PQN
3.0
90
PQN
PQN (ResNet-15)
2.5
Rainbow
80
Rainbow
70
2.0
C51
A
60
1.5
g
DDQN
50
1.0
PQN Hadamax PQN
40
30
PQN (ResNet-15)
0.0
20
0.0
0.5
1.0
1.5
2.0
3 Normalized Score (r)
Env Frames
1e8
Figure 5: Median Human- Normalizdr performance training PQN, PQN (Resnet-15) and Hadamax- PQN in the Atari domain over 57 games, 200M frames and 5 seeds (left), and the Atari-57 score profile (right). The Atari-57 score profile illustrates the percentage games that exceed the normalized score threshold on the x-axis.

---

The effect of the Hadamax encoder on the baseline PQN on per per-game basis can be seen in Fig. 6 The results show a significant performance increase over the baseline, with over 17 games having more than 100% improvement, compared to only one single game having more than 50% decrease in performance. The per-game improvements over the Rainbow-DON baseline can be seen in Appendix E.4 For each individual game's training curve and the final 200M frame score table, we refer the reader to Appendix F To the best of our knowledge, the implementation of the Hadamax encoder is one of the biggest recorded non-algorithmic improvement over a recent competitive RL baseline, and it does not involve any complex hyper-parameter tuning.
Improvement over PQN (200M Frames) Log Scale
10a
56 Improvement
107
10
-10
-10P
-10*
Busiing
Salrre
Freeney
Skine
Wecterurattie Pemeenateeeana
Boxing
rebodank
koul
Qbert
Summuns
CrsyCimecs
Rieeraid
Breu sout
Namet Come
Prozuato
Hore
Zantoo
Assmit
jam coon
Proseite
Deleddrr
Allen
See cat
Siequest
Chopoer Venture
Figure 6: Per-game improvement of Hadamax-PQN over PQN (Log Scale).
5.2 Does Hadamax Generalize Beyond PQN?
(Atari-10, 40M)
100
The Hadamax encoder only enhances the performance of PQN, but also works effectively with other reinforcement learn ing agents. To showcase this, the C51 algorithm is evaluated on the Atari-10 10 benchmark for 40M environment frames. As shown Figure 7 direct implementation of the Hadamax en-
w/ Hadamax
Original
80
Atari-10 Score
60
coder to the C51 algorithm boosts the performance by approx- imately 70% on Atari 10 2 Similar to PQN, the algorithmic hyperparameters for Hadamax-C51 remain exactly the same as for the C51 baseline from cleanrl 28 These improvements suggest that the Hadamax encoder is able to be implemented as strong default encoder for multiple algorithms in the Atari domain. For more information on the Atari-10 10 benchmark and corresponding score normalization metrics, we refer the reader Appendix D.3
40
0
1
3
Env Frames
1e7
Figure 7: C51 with and without Hadamax encoder on Atari-10.
5.3 Effective Rank and Dead Neurons
order to obtain clues about the stabilizing effects the proposed Hadamax encoder, the effective rank of the hidden layers is investigated during training [8). well as the amount of dead neurons 5II. The effective rank of a feature matrix for a threshold (8 = 0.01), denoted as sranks (Ø),
- 6). where (o.($)} are the singular values of & in decreasing order, i.e., .... Fa 0. The effective rank portrays measure of network capacity i.e. the amount information that can be approximated in a certain hidden layer.
given by srankis (+) = min A:
3 21-1 (+) (+))
>* a.(4)
We investigate the differences in effective rank between the baseline PQN and Hadamax- PPNN. To find clues for Hadamax's 's strong improvements on certain environments, the differences are visualized on random subset of 5 high-improvemmen environments from Fig. 6 The effective rank of the
7

---

55
Encdiive Rank
Encctive Kank
Effectow Bank
45
FhatoMa
ExI
Ery Fratness
Figure 8: Effective rank 32 of the 4 hidden layers for both the baseline PQN and the Hadamax- PQN setting. Although there is visible difference between the first and final layer, the deeper convolutional layers show a lower effective rank in the baseline setting, as well as stronger rank decay during training.
encoder's representation layers while training for 200M frames can be seen in Fig. The plots show that there minimal differences in effective rank in the first and last hidden layer of the encoder. However, the baseline PQN encoder, the deeper convolutional layers show a more prominent decay in rank during training, well reduced initial effective rank. As mentioned in Section 4, the increase effective rank in the deeper convolutional layers can largely be credited to the use of Hadamard representations. A further look is taken at the convolutional channel cosine similarity [54], where a low cosine similarity indicates that the channels are extracting dissimilar or uncorrelated features from the input data, which is desirable and suggests diversity among the channels. An ablation of the max-pool and Hadamard components' effect on both effective rank and channel cosine similarity be seen in Table The Hadamax encoder improves both the effective rank and channel cosine similarity, when compared to the baseline encoder. This indicates that Hadamax extracts more expressive, uncorrelated features from the pixel inputs.
Table 1: Channel Cosine Similarity & Effective Rank
Metric
Baseline
Maxppol
+ Hadamard
Both (Hadamax)
Effective Rank
Base
+10% +20%
+10% -90%
+ +10-00%
Channel Cosine Similarity
Base
-50%
Further investigation into the penultimate layer's fraction of dead neurons shows small decrease from the baseline (see Fig. 9). The percentage of dead neurons in the final hidden layer is calculated by finding neurons that have variance of less than 10-4 over the batch dimension. In practice, this metric generalizes well any activation function (ReLU, GELU, Tanh). After training for 200M frames, both the baseline PQN and Hadamax-PQN have less than 8% dead neurons, which remains extremely low compared to DQN
Dead Neuron Fraction (MLP)
0.20
Fraction of Dead Neurons
Hadamax-PQN
0.15
0.10
[5]] We therefore do not expect substantial correlation between the small reduction in dead neurons and the perfor- mance. However, in contrast to recent work on Hadamard representations [31], who showed that the DQN algorithm exhibits instability when multiplying ReLtvvccie neu- rons, we show that it is possible to use Hadamard products on zero-saturating activations. We believe inherent stabil- ity of the PQN algorithm and its corresponding low fraction of dead neurons allows for successful Hadamard multiplica- tion of lincar-unit activations like ReLU or GELU.
0.05
Frames
1e8
Figure 9: Fraction of dead neurons over 200M frames.

---

5.4 Which Design Choice is most Important?
As described in Section 4 the Hadamax encoder differs from PQN's conventional Nature CNN encoder in three areas: (1) applying max- max-loooing (2) using Hadamard representations and (3) GELU- activated hidden layers. The precise influence of each component of the Hadamax encoder remains to be determined. An ablation analysis over these areas is therefore done 40M environment frames in the full Atari-57 suite. The ablations are defined as implementation subtractions from the original Hadamax architecture in Fig- The result of the ablation study is shown in Fig. T0a Next to the ablations, the effects of direct additions our design choices on the baseline PQN are investigated. The results of the addition analysis can be seen in Fig. 10b
Hadamax PQN
PON Hadamard
No GELU
Abiation
Addition
PQN Max-pool
Hadamard
PON GELU
No Max-pool
PON
0.6
2.0
1.2
1.4
0.50
0.85
0.90
0.95
3.00
1.05
110
Median Human-Norralliin Score
Median Human-Normalizen Score
(a) Hadamax ablations.
(b) Naive PQN additions.
Figure 10: Ablations of Hadamax-PQN each represented as a subtraction from the full Hadamax architecture (a), and naive architectural additions the baseline PQN (b).
Over training period of 40M frames, the subtraction of max max-pooling leads to the largest decay in performance. Note that when max-pooling is subtracted from our architecture, we return the convolutional strides their original values, in order still retain feature compression. The importance of down-sampling with max-pooling strengthens our hypothesis that selection of the most prominent features key when working with high-dimensional observation spaces in the Atari domain. The use of convolutional Hadamard representations is also an important component, showing that the increase in effective rank paired with other benefits such as high-order interactions [[], have strong correlation with downstream performance. Finally, the GELU activation has the lowest importance, although its contribution as compared to the ReLU still remains substantial for such small architectural component. Notably, if the ablations are compared to the effects of directly implementing a single design choice on the baseline (see Fig. [Ob). becomes clear that the overall combination of all three components is key factor. For experimental analysis with two deeper Hadamax encoders, we refer the reader to Appendix E.2
5.5 VizDoom
Additional experiments have been conducted on the pixel-based VizDoom enviromment [30] Training RI. VizDoom is different from Atari it works with 3D environments, semi-realistic physics, and stochastic elements demanding more advanced reasoning. On VizDoom Deathmatch, the baseline convolutional encoder converted Hadamax encoder, following the same shallow convolutional architecture with filter sizes of 32-64-128. The RL baseline used is the actor-critic PPO algorithm 43 48). Without changing any other hyperparameters, the results after training for 40M frames can be seen in 2
Table 2: VizDoom Deathmatch performance
Method
10M
20M
30M
40M
Hadamax
-1.62 + 0.20
8.08 + 1.61
19.36 + 1.82
29.38 + 10.31
Baseline
-2.81 + 0.40
1.27 = 0.80
3.10 - 0.89
5.21 = 1.27

---

The strong performance increase over the baseline encoder suggests that Hadamax is applicable on actor-critic architectures, as well wider variety of pixel-based based environments.
Conclusions and Future Work
This paper introduced the Hadamax encoder architecture, augmenting the conventional pixel- bxsed-bsedd Nature CNN architecture with Hadamard representations, while down-sampling using max- maoxloooing instead of convolutional strides. Furthermore, the Gaussian Error Linear Unit activation was im- plemented to improve training stability. The application of these fundamental changes to the PQN baseline encoder, while preserving its original shallow structure, allowed for profound increase in performance over several model-free baselines. Specifically, we reach an almost two-fold perfor- mance gain over the baseline PQN setting, and surpass Sainbow-DQNe official 200M frame score after just 90M frames, while remaining an order of magnitude faster. Additional results on C5I I and PPO/VizDoom show that the Hadamax encoder remains effective across variety of algorithms and across other pixel-based environments.
Due to computational constraints, this paper includes only limited testing of the Hadamax encoder on complex algorithms such as Bigger-Bettee Beet--etttt-sssee (BBF) [50] on the Atari- 100k benchmark or on tate-of-theear model-basee algorithm such Dreamerv 1-v3 by the performance improvement on C51 in Fig. 7 and PPO/Vizdoom in Table 2
22 23 However, as seen we do expect a certain degree of generalization across algorithms and/or environments. Another limitation is that the Hadamax encoder, due to its increased architectural complexity, accounts for some extra computa- tional overhead compared PQN's conventional Nature CNN architecture. For completeness, the training durations are therefore reported in Fig. [and the inference durations are reported in Table 3 Inference time is however usually not key issue in the RL context.
Table 3: Inference times
Architecture
Inference time (milliseconds)
Rainbow
0.59
PQN
0.39
PQN (Impala)
1.40
Hadamax-PQN
1.75
All all, we believe this paper takes an important step forward in functional encoder synthesis for RL, discovering an alternative for the usual deep and complex ResNet architectures to optimize performance, An interesting avenue for future work would be to investigate scaling of the Hadamax encoder, as it already achieves significant performance improvements using only 3 convolutional layers and the classic 32-64-64 filter dimensions. Finding successful ways to scale the Hadamax encoder in either width or depth could yield even stronger improvements and more insights into architecture synthesis. Another promising avenue would be to explore the integration of MoE-style prediction heads in the Hadamax encoder, since common implementations of MoE do not necessarily affect the base encoder [42). Furthermore, as Hadamax-PQV does not come with any algorithmic or hyperparameter changes, it can be used new baseline to build other algorithmic improvements upon. Specifically, since hard-exploration games are generally not suited for PQN's epsilon-greeds exploration regime, augmenting PON-Hadamax with novel exploration techniques might further bridge the gap in performance between compute-light model-free and compute- heavy model-bar based algorithms such as DreamerV3 Muzero (3 47
Acknowledgements
We would like to thank Prof. Mark Hoogendoorn for his helpful guidance during this project. We also thank SURF (www.surf.nl) for the support in using the National Supercomputer Snellius. This work used the Dutch national e-infrastructure with the support of the SURF Cooperative using grant no. 1385-1
10

---

References
[1] P. Agarwal, S. Andrews, and S. H. Kahou. Learning to play atari in a world of tokens. arXiv preprint arXiv:2406.01361, 2024.
[2] M. Aitchison, P. Sweetser, and M. Hutter. Atari-5: Distilling the arcade learning environment down to five games. In International Conference on Machine Learning, pages 421-438 PMLR, 2023.
[3] E. Alonso, A. Jelley, V. Micheli, A. Kanervisto, A. Storkey, T. Pearce, and F. Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757-58791, 2024.
[4] J. L., Ba, J. R. Kiros, and G. E. Hinton. Layer normalization, arXiv preprint arViv: 1607.06500
2016.
[5] M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective reinforcement learning. In International conference on machine learning, pages 449-458, PMIR,, 2017.
[6] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of artificial intelligence research, 47:253-279, 2013.
[7] A. Bhatt, D. Palenicek, B. Belousov, M. Argus, A. Amiranashvili, T. Brox, and J. Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. arXiv preprint 1902.
2019.
[8] C. Bonnet, D. Luo, D. Byrne, S. Surana, S. Abramowitz, P. Duckworth, V. Coyette, I. Midgley, E. Tegegn, T. Kalloniatis, O. Mahjoub, M. Macfarlane, A. P. Smit, N. Grinsztajn, R. Boige, N. Waters, M. A. Mimouni, U. A. M. Sob, R. de Kock, S, Singh, D. Furelos-Blance V. Le, A. Pretorius, and A. Laterre. Jumanji: diverse suite of scalable reinforcement learning environments in jax, 2024.
[9] J. Bradbury, R. Frostig. P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, and Q. Zhang. JAX: composable transformations of
J. VanderPlas, S. Wanderman- Python+NumPy programs, 2018
[10] X. Chen, C. Wang, 7. Zhou, and K. Ross. Randomized ensembled double q-learning: Learning fast without a model, arXiv preprint arXiv:2101. 10.98292 2021.
[11] G. G. Chrysos, Y. Wu, R. Pascanu, P. Torr, and V. Cevher. Hadamard product in deep learn- ing: Introduction, advances and challenges. arXiv preprint arXiv:2504.13112 13112, April 2025. arXiv:2504.1 1311311112 12v1 [cs.LG]..
[12] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo- gies, volume 1 (long and short papers), pages 4171-4186, 2019.
[13] I.. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pages 1407-1416. PMLR, 2018.
[14] M. Fortunato, M. G. Azar, B. Piot, Menick, I. Osband, A. Graves, V. Mnih, R. Munos, D. Has- sabis, O. Pietquin, et al. Noisy networks for exploration. arXiv preprint 17X.v:1706.10295, 2017.
(15) V. Frangois-Lavet, P. Henderson, R. Islam, M. G. Bellemare, J. Pineau, et al. An introduction to deep reinforcement learning. Foundations and Trends0 in Machine Learning, 11(--4):219-34, 2018.
[16] C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax a differentiable physics engine for large scale rigid body simulation, 2021.
[17] M. Gallici, M. Fellows, B. Ellis, B. Pou, I. Masmitja, N. Foerster, and M. Martin. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.0488 2024.
[18] C. Gulcehre, S. Srinivasan, J. Sygnowski, G. Ostrovski, M. Farajtabar, M. Hoffman, R. Pascanu, and A. Doucet. An empirical study of implicit regularization in deep offline RI.. Transactions on Machine Learning Research, 2022.

---

[19] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05903 2018.
[20] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pages 2555-2565. PMLR, 2019.
[21] D. Hafner, Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprini arXiv:2010.02/99, 2020.
[22] D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba. Mastering Atari with Discrete World Models. 10 2020.
[23] D. Hafner, J. Pasukonis. J. Ba, and T. Lillicrap. Mastering diverse domains through world models, arXiv preprint arXiv:2301 1.4404,4 2023.
[24] N. A. Hansen, H. Su, and X. Wang. Temporal difference learning for model predictive control. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8387-8406. PMLR, 17-23 Jul 2022.
[25] K. He, X. Zhang, S. Ren, and Sun. Deep Residual Learning for Image Recognition. In Proceedings of 2016 IEEE Conference on Computer Vision and Puttern Recognition, CVPR '16, pages 770-778. IEEE, June 2016.
[26] D. Hendryeks and K. Gimpel. Gaussian Error Linear Units (GELUs). 2016.
[27] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot,
M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
[28] S. Huang, R. F. J. Dossa, C. Ye, J. Braga, D. Chakraborty, K. Mehta, and J. G. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
[29] S. Kapturowski. G. Ostrovski, J. Quan, R. Munos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
[30] M. Kempka, M. Wydmuch, G. Runc, Toczek, and W. Jaškowski. ViZDoom: A Doom-based AI research platform for visual reinforcement learning. In IEEE Conference on Computational Intelligence and Games, pages 341-348, Santorini, Greece, Sep 2016. IEEE. The best paper award.
[31] J. E. Kooi, M. Hoogendoorn, and V. François- Lavet. Hadamard representations: Augmenting hyperbolic tangents in rl, 2024.
[32] A. Kumar, R. Agarwal, D. Ghosh, and S. Levine. Implicit mnder-parameteriizaion inhibits data- efficient deep reinforcement learning. In International Conference on Learning Representations, 2021.
[33] R. T. Lange. gymmax: A JAX-based reinforcement learning environment library, 2022.
[34] M. Laskin, Srinivas, and Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pages 5639-5650. PMIR, 2020.
[35] H. Lee, D. Hwang, D. Kim, H. Kim, J. Tai, K. Subramanian, P. R. Wurman, J. Choo, P. Stone, and T. Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410009754. 2024.
[36] C. J. Kuba, A. Letcher, L. Metz, C. Schroeder de Witt, and J. Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 15:16555-16668, 2022.
[37] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXiv: 13/2.6002, 2013.
[38] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforeement learning. Nature, 5 518(7540):529-533 02 2015.
12

---

[39] M. Nauman, M. Ostaszewski, K. Jankowski, P. Miloš, and M. Cygan. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. arXiv preprint arXiv:2405. 16158, 2024.
[40] T. Ni, B. Eysenbach. E. Seyedsalchi, M. Ma. C. Gehring, A. Mahajan, and P.-L. Bacon. Bridging state and history representations: Understanding self-predietive rl. arXiv preprint arXiv:2401.08898, 2024.
[41] J. Obando-Ceron, A. Courville, and P. S. Castro. In deep reinforcement learning, a pruned network is good network. arXiv preprint httv:2402//2479) 2024.
[42] J. Obando-Ceron, G. Sokar, T. Willi, C. Lyle, J. Farebrother. J. Foerster, G. K. Dziugaite,
D. Precup, and P. S. Castro. Mixtures of experts unlock parameter scaling for deep rl. arXiv preprint arXiv:2402 08609, 2024.
[43] A. Petrenko, Z. Huang, T. Kumar, G. S. Sukhatme, and V. Koltun. Sample factory: Egocentric 3d control from pixels at 100000 FPS with asynchronous reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 7652-7662. PMLR, 2020.
[44] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. OpenAl, 2018.
[45] A. Rutherford, B. Ellis, M. Gallici, J. Cook, A. Lupu, G. Ingvarsson, T. Willi, A. Khan, C. de Witt, A. Souly, S. Bandyopadhyay, M. Samvelyan, M. Jiang, R. T. Lange, S. Whiteson, B. Lacerda, Hawes, T. Rocktaschel, C. Lu, and J. N. Foerster. Jaxmarl: Multi-agent environments in jax. arXiv preprint (...xx33311100090 2023.
[46] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis. T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588 (888773)):604-600 2020.
[47] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis. T. Graepel. Lillicrap, and D. Silver. Mastering atari, go, chess and shogi by planning with a learned model, 2019. cite 1exiv:1911.000 1.08265.
[48] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: /tf.::/777.33333 2017.
[49] M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman, Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929. 2020.
[50] M. Schwarzer, J. S. O. Ceron, A. Courville, M. G. Bellemare, R. Agarwal, and P. S. Castro. Big- ger, better, faster: Human- level atari with human-level efficiency. In International Conference on Machine Learning, pages 30365-30380, PMLR, 2023.
[51] G. Sokar, R. Agarwal, P. S. Castro, and U. Evei. The dormant neuron phenomenon in deep reinforcement learning- In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning. volume 202 Proceedings of Machine Learning Research, pages 32145-32168. PMLR, 23-29 Jul 2023.
[52] R. K. Srivastava, K. Greff, and J. Schmidhuber. Training very deep networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama. and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[53] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
[54] Y. Tang, Y. Wang, Y. Xu, Y. Deng, C. Xu, D. Tao, and C. Xu. Manifold regularized dynamic network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5018-5001 002888 June 2021.
[55] H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[56] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, I., u. Kaiser, and
I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach,
13

---

R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[57] S. Wang, S. Liu, W. Ye, J. You, and Y. Gao. Efficientzero v2: Mastering discrete and continuous control with limited data. arXiv preprinl arXiv:2403.00552 2024.
[58] Z. Wang. T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas. Dueling network architectures for deep reinforcement learning. In International conference machine learning, pages 1995-2003. PMLR, 2016.
[59] Weng, M. Lin, S. Huang, Liu, Makoviichuk, V. Makoviychuk, Liu, Y. Song, Luo, Y. Jiang. 7. Xu, and S. Yan. EnvPool: A highly parallel reinforcement learning environment execution engine. In Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 22409-22421.
Curran Associates, Inc., 2022.
[60] W. Ye, S. Liu, T. Kurutach, Abbeel, and Y. Gao. Mastering atari games with limited data. Advances in neural information processing systems, 34:25476-25488, 2021.
[61] W. Zhang, G. Wang, J. Sun, Y. Yuan, and G. Huang. Storm: Efficient stochastic transformer based world models for reinforcement learning. Advances in Neural Information Processing 2023.
36:
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflect our paper's contributions and scope, which is the creation of new state-of-the-art baseline via encoder synthesis for model-free RI..
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: the Conclusions and Future Work section, we dedicated full paragraph to view the limitations of this work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification: As this paper focuses on neural network architectures reinforcement learning, it does not contain any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The full, easily implementable JAX-code for the Hadamax encoder is provided in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: As stated 4, we provide open access to the code exact convolutional architecture used for the main results in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ btiaps..ccpuulic/guuu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Our experimental setup is equal to that of the baselines that are used. We specify the hyperparameters in Appendix C.I.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our experiments are based on the full Atari-57 suite, which consists of 57 different pixel-based environments. Following conventional score metric visualization, use the median-hur huw-hhhummmaaaaaa scores over 57 environments, which are conventionally not accompanied by error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Figure 1, we show the compute times of all the algorithms used for single 200M environment frame seed.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't i make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttpp///heuris...ccpuubb/////// cet.eiieeeeeiicccccceeeeeeeee
Answer: [Yes]
Justification: This paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This work is not tied to particular applications, and serves to optimize RL algorithms in the simulation domain. We therefore do not expect any socictal impacts of this research.
Justification:
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our data and models used not have high risk for misuse, We use openly available data from the Atari-57 domain.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The creators of the baseline models have been properly credited in our research by way of GitHub repository links and several citations.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code of the proposed Hadamax encoder in this paper is clearly provided in Appendix B.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Tis paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs have not been used for core components of our paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

Appendix
Table of Contents
A Impact Statement
23
Hadamax Encoder Code
23
Experiment Details
24
Hyperparameters
24
Environments
24
Baseline Implementations
24
C.4 Compute Usage
24
D Metries
25
D. Median Human- Human-Norman Normalized Score
25
Stari-57 Score Profile
................................
25
D33
Atari-3 and Atari-10
25
E
Additional Experiments
26
E.1
Memory Usage
26
E.2
Deeper Hadamax Networks
26
E.3
Hadamax with Other Agents
2
Per-game improvement over Rainbo-DDQN
27
Individual Game Scores
28
22

---

A Impact Statement
This work shows that architectural innovations like the Hadamax encoder can drive significant progress in reinforcement learning. By enabling more efficient and accessible AI, it encourages broader adoption and exploration of learning systems across diverse real-world domains.
B Hadamax Encoder Code
We provide the full JAX-based code of the Hadamax encoder for reproducibility purposes.
Input imput_obs, frame stacked Atari observation
jnp transpose (input_obs, (0, 2, 3, 1))
x = x 255.0
# First block
x1 Conv (32, kernel_size- 8), strides-(1, 1), padding " SAME"
kernel_init-m. initialieess caalizers.xavee.noorra.. mavier_normaal (x)
x2 nn. Conv (32, kernel,size=(8, 8), strides=(1, 1), padding=" SAME""
()) (x)
x1 - normalizexx1) # Normalize before activation
normalize(x2) # Normalize before activation
11 x1 nn. nn.gelu
Apply activation
12 x2 = mn (x2) Apply activation
13 x1 x2 # Hadamard product
max_pool (x, window_shape-(4, 4), strides-(4, 4), padding-"SAME 15 Second block
16 x1 nn. (64, kernel_size=(4, 4), strides=(1, 1), padding="
17
ternel_initan i intiiaaiiiittiiieeeee.... narai
x2 = (64, kernel_size=(6 4), strides=(1, 1), padding= SAME",
vermel_init-mn www.iniiieiii...ii.a.a 20 x1 normalize(x1) tenhelaatttmmmiiullllle.ee aaw.liieee..avier_oooaaaall ()) (x)
Normalize before activation
21 x2 (x2) # Normalize before activation
22 x1 = nn
x2 (x1gel((x1) # Apply activation
(x2) Apply activation
24
# Hadamard product 35 x max max_pool( (x, vindov_shape=(2, 2), strides=(2, 2), PAdding=""AME
26 Third block
x1 = Conv kernel_size=(3, 3), strides=(1, 1), Badding="SAME
kernel_ htt.:iiiiiiiiiiiaaiieee...et initializerii wwwpeerrr...viiie.ee.e.ee.eeeee favier_normal (x)
29 x2 nn Conv C6nv(64, kernel.size-(3, 3), strides-(1, 1), padding SAME".
30
termel_init-m..ing htw.temmm.ittttllzess..eoo navierroravilrrnormmllll (x)
31 x1 - normalize # Normalize before activation
x2 = normalize(x2) # Normalize before activation Apply activation
(x1) Apply activation
34 x2 nn gelu(x2)
35 x xi x2 # Hadamard product
36 = max_pool (x, window_shape-(1. 3), strides-(1, 1), "adding="SAME")
Flatten for MLP layer
38 x. reshape ((x. stapeha -1))
39 X nn. Dense (512, kernel_init-ng initializers. www.:eerr.eeeeeoeeee normal ()) (x)
8 normalize(x)
= nn. mn.
Dense (self action_diz. name= actaaactiio.. tame="action acase (x) # Final Q-Values
23

---

C Experiment Details
C.1 Hyperparameters
Table 4: Atari Hyperparameters for PQN, PQN (ResNet-15) 15) and Hadamax-PQN. These hyperparam- eters are equal to the original hyperparameters from the PQN baseline [7].
Parameter
Value
NUM ENVs
128
NUM SIM_STEPS
APS_START
1.0
APS_FINISH
0.001
EPS DECCY
0.1
NUM_EPOCHS
NUM LUM_MINIBATCHIEE
32
NORM NORM_INPUT INPUT
False
NYPE_TYPE TYPE
layer norm
LR
0.00025
MAX GRAD
FRAD_NORM NORM
10
LR LINEAR DECCY
False
GAMMA
0.99
LAMBDA
0.65
OPTIMIZER
RAdam
C.2 Environments
We run experiments on the Atari-57 suite, where there are different games in total. No per-game tuning is allowed and the same agent architecture, hyper- prper-pprameters and pre-processing needs to run on every game. The suite contains varying games that can be used to examine different properties of RL agents, c.g. long- mor-zhoizonn credit assignment, partial observability, hard exploration, etc.
Each observation consists of 4 grayscale images of the game state stacked together, i.e. (4, 64, 64). The action space is discrete, and each action represents a different operation in the game. The reward function depends on the environment chosen. More details each game can be found https://alh farama. org
Atari-3 and Atari-10:
We examine C51, DQN and Rainbow on Atari-3 or Atari-10 (2) which are small but representative subset of the full Atari-57 suite. Atari-3 includes Battle Zone, Name This Game and Phoenix. Atari-10 10 includes Amidar, Bowling, Frostbite, Kung Fu Master, River Raid, Battle Zone, Double Dunk, Name This Game, Phoenix and QeBert.
C.3 Baseline Implementations
PQN: We use the official codebase of PQN and default hyper-parameter settings.
Rainbow, C51, DQN:For the Fig. 12 training results we use implementations from cleanrl and default hyper-parameter settings. The scores for DDQN, C51 and Rainbow in figures ] and have been taken from their respective official papers.
Since the whole PQN codebase is in Jax, we implement the Hadamax encoder for PQN in Jax as well. As Implementations of Rainbow, C51 and DQN from cleanrl are in Py PyTorch we also implement the Hadamax encoder for these agents PyTorch.
Hadamax encoder:
C.4 Compute Usage
We run all our experiments on a HPC cluster equipped with A100 GPUs. Each run of Hadamax-PQN needs around 45 minutes for 40 millions frames and PQN needs around 20 minutes.
htttp:://gtthab..o.//atttg/////p htttp:://github.cco//wwws
24

---

D Metrics
D.1 Median Hluman-Normalized Score
For each game, compute the average score 2; across multiple independent seeds. Then compute the normalized score Z; as:
7: h r
where zy is the raw score, and and h2 are the random and human scores for game i, respectively (see T values). After computing the normalized scores for all 57 games seeds, they sorted and the median value is computed.
D.2 Atari-57 Score Profile
X-axis:(r Normalized Score). Represents the threshold score (e.g., Human-Normalized Score). Higher values mean better performance.
y-axis: 7% = fraction of games above Shows the fraction of games for which the agent's 's normalized score greater than т. For example, at r 1, the y-value represents what fraction of games the agent beats human performance In other words, represents the percentage of games that has scores higher than т.
D.3 Atari-3 and Atari-10
The Atari-3 and Atari- 10 scores approximate the median normalized score across the full 57-game Atari benchmark using subsets of 3 and 10 games, respectively 2 The computation involves the following steps:
1. For each game in the subset, compute the normalized score Z; as:
Z, 100
h4 TV
where n; is the raw score, and r, and h; are the random and human scores for game i, respectively (see Table values).
2. Apply the log transform:
(2:) = -10110 + max(0, Zz))
3. Compute the weighted sum f - Gā((;), where is the subset of games and c; the subset-specific coefficients.
4. Obtain the predicted median score as:
i 10
For Atari-3, the subset comprises Battle Zone, Name This Game, and Phoenix, with coefficients [0.3706, 0.5133, 0.1015).
For Atari-10, 10, the subset includes Amidar, Bowling, Frostbite, Kung Fu Master, River Raid, Battle Zone, Double Dunk, Name This Game, Phoenix, and Q*Bert, with coefficients [0.0825, 0.0559, 0.0691, 0.0986, 0.0486, 1888, 0.0852, 0.1787, 0. 0.1643, 0.0592).
25

---

E
Additional Experiments
E.1
Memory Usage
Table 5: Memory usage and batch sizes for different architectures
Architecture
Training Update Memory (MB)
Batch
Inference Memory (MB)
Inference Batch
Rainbow
198.77
185.33
PQN
254.54
256
155.87
8
PQN (Impala)
725.18
256
143.46
8
Hadamax-PQN
2247.26
256
233.14
8
E,2 Deeper Hadamax Networks
As network scaling has become topic of interest in the field of RL [35 50 42). we provide experiments using deeper versions of our encoder: 5-layer and 7-1ayer Hadamax-PQN Specifically, the second and third convolutional layers in the original 3-layer encoder are duplicated, and we refrain from max- poollng the duplicates to avoid excessive compression. Similar to the ablations, the deep networks are tested on the full 57-game Atari suite for 40M environment frames. The results can be seen Fig.
Atari (57 Games, 40M Frames)
Hadamax-PQN
Depth
Depth
Median Human-Normalized Score
Figure 11: Hadamax encoder depth Ablations.
Simply using deeper convolutional Hadamax encoders does not seem to improve performance. Although there are more promising ways scale the Hadamax encoder both in depth and width, the computational cost was the limiting factor in pursuing this in more detail. As discussed in the main paper, we leave this as promising research area for future work.
26

---

E.3 Hadamax with Other Agents
We modify the encoders of the widely-used cleanrl [2 implementations of C51, DQN, and Rain- bow to demonstrate that the Hadamax encoder can generalize across various model-free agents. See Figure [2, on Atari-10, Hadamax improves the performance of the original C51 by 70%. and on Atari-3, it boosts DQN and Rainbow by 20% and 30%, respectively. These substantial gains, achieved by simply replacing the encoder, suggest that Hadamax could serve as new default encoder for model-free reinforcement learning methods Atari.
DQN (Atari-3, 40M)
C51 (Atari-10, 40M)
Rainbow (Atari-3, 20M)
160
w/ Hadamax Original
Original
Hadamax
M
160
w/ Hadamax Original
A Score
120
Atari-10 Score
Atarl-3 Score
120
80
80
40
40
1
3
2
0.0
0.5
1.0
1.5
2.0
Env Frames
1e7
Env Frames
1e7
Env Frames
1e7
Figure 12: Performance gains of DQN, C51 and Rainbow with Hadamax encoders on a subset of Atari-57.
E.4 Per-game improvement over Rainbow-DON
Improvement over Rainbow Log Scale
207
-107
-107
Pronayye
St Ing
Saltr S
Freethi
Kanganc
Duutt uni
Bos ing
Rovenng
Ping
Graeter
Indesttnns
Conta Cl mine"
De Demonitian Nemorttack stack
1OU1
Halkoman
Zacton
areatou
Alan
Stuclle
Centivede
Saars unoer umee
Seove ga-
Delunder
commard
Game
27

---

Individual Game Scores
Alreagg
Amisanys
Ascouthus
Arteria--5
Acternideuy5
Attamtis-o
va
BatticZone
BeanRidervs
Berzeric
Bowllng
Bouing
CraryCimbrewth
DaublebunkovS
Bating
Freemay-uS
Frestiious
Gopher-v5
Gr
Horo
vs vs
amerbord vs
Kongarco vS
Krufl vs
Normalized ized Score
KingfuMaecteno
MaderraatBeennnee
MaParmanwh
Name Cammea
Phoenisws
Ponqus
PrivatoSyow6
Obertee
Riverriduus
Roadfonnerrs
Robotankus
Scanuest us
Skinggv
Sobrisss
Spaceinvaders vs
Starfunnervv
uu
Tennisus
Timefiotu5
Itanknamws
UpNDown- -5
Venture
Videofinball
WizardDfWooss
Hadamax PON
PQN
PON (ResNet- 15)
1e8
1e8
1e8
Environment Frames
28

---

Table 6: Final 200M frame scores.
Game
Hadamax -PNN
PQN (Resnet-15)
PQN
Alien-v5
20045.4
16935.0
29T6.3
5midar-v5
1774.5
944.1
740.3
Assault-v5
26426.7
11160.5
15089.7
Asterix-v5
274915.0
95400.0
287697.6
Asteroids-v5
39328.0
4232.7
21047.6
Atlantis-v5
715750.0
516357.8
831884.7
BankHeist-vS
1260.2 2.0
1446.1
1336.2
BattleZone-v5
92951.2
55106.5
44130.8
BeamRider-v5
49480.9 9
16315.8
18131.7
Berzerk-v5
57497.4
19597.8
6061.3
Bowling-v5
29.7
30.2
42.5
Boxing-v5
99.8
96.3
98.3
Breakout-v5
607.4
470.3
489.6
Centipede-v5
17901.7
9266.5
8178.2
ChopperCommand-v5
203593.5
26974.7
11688.8
CrazyClimber-v5
202048.2
162776.7
168732.4
Defender-v5
360287.5
48761.1
66381.0
DemonAttack-y. Attack-v5
135450.5
125870.4
131320.0
DouhleDunk-v5
-1.8
-1,2
-1.2
Enduro-v5
2323.4
462.7
2284.6
FishingDerby-v5
46.6
31.2
45,4
Freeway-v5
33.7
21.4
33,8
Frosthite-v5
7689.5
6537.2
5623.8
Gopher-v5
67829.0
26859.5
40834.5
Gravitar-v5
1547.2
514.4
1107.3
Hero-vS
30617.4
24912.9
18099.9
Icellockey-v5
16.3
-2.4
-1.4
Jamesbond-v5
4244.8
1285.8
1942.8
Kangaroo-v5
13177.3
8728.6
13992.5
Krull-v5
10554.4
9497.9
9802.2
KungFuMaster-v55
36751.9 .9
24102.8
38233.3
MontezumaRevenge-v5
0.0
MsPacman-Vv
6968.3
4584.7
4909.7
Name TameeeiGGmmm-55
21334.2
18754.3
16437.0
Phoenix-v5
267080.2
41001.4 .4
120959.5 .5
Pitfall-v5
-43.5
-34,4
-50.5
Pong-v5
21.0
20.8
21.0
PrivateEye-vS
3.6
3,9
7.5
Qbert-v5
25970.2
21818.4
22449.6
Riverraid-v5
29423,9
18669.8 8
24133.3
RoadRunner-v5
190019.6
52925.0
76600.9
Robotank-v5
71.5
66.1
68.3
Seaguest-v5
129408.8
43559.8
11554.4
Sküng-v5
..9971.3
-29479.8
-29972.3
Solaris-v5
1884.2
863.5
2189.4
SpaceInvaders-v5
22258.0
13800.3
15125.0
StarGunner-v5
549350.4
215397.7
264413.1
Surround-v5
7.5
6.3
Tennis-v5
23.8
22.9
-1.0
TimePilot-v5
17946.0
11924.0
12320.1
Tutankham-v5
258.7
216.8
248.0
UpNDown-v5
191857.2
82743.3
270833.7
Venture-v55
940.7
0.0
18.1
VideoPinhall-vS
522510.3
416690.8
463022.1
WizardOrWor-vS
21526.1
13130.5
22214.2
Yars Veveneve
444710.8
119951.1
111611.7
Zaxxon-v5
31400.8
14229.4
17644.0
29


Reviews:
[{"review_id": "NGeE5Niabw", "reviewer": "Reviewer_GuDX", "summary": "This paper introduces a new network architecture for RL on image-based inputs (Hadamax), which builds on PQN by using GELU activations, Hadamard products, and max pooling. The author(s) evaluate on Atari-57, comparing with and significantly outperforming Double DQN, C51, Rainbow, and PQN. Using Atari-10 the author(s) also show that Hadamax architecture is similarly effectively applied on top of C51, demonstrating that its benefits are not restricted specifically to PQN or the parallel training setting more generally.\n\nEffects on the representation learned at various layers of the network are empirically analyzed and an ablation study is performed moving between PQN and Hadamax PQN.", "questions": "* What do you think the biggest weakness or limitation of Hadamax is?\n* Could you explain why the contributions would necessarily be specific to pixel-based inputs?I suppose this is more a question about the Hadamard product component, but nonetheless I’ll rephrase as: Do you think a related approach could similarly benefit non-pixel-based domains and if not why is your reasoning?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* As this is essentially “just” a network architecture change, and would be quite simple to implement and reproduce, the magnitude of performance improvement is extremely strong. This is two strengths, (1) very simple to implement / reproducible and (2) strong empirical performance.\n* The empirical work is also quite thorough, comparing with multiple baselines in the primary results and conducting well-chosen secondary experiments to supplement those results.\n* The paper is well written and easy to follow.\n\nWeaknesses:\nThese are really not significant weaknesses, and more just potential areas for improvement.\n\n* This is such a generally applicable architecture that not demonstrating it on more domains is arguably the biggest weakness, and missed opportunity, in the paper. This is simply because only evaluating on Atari begs the question if this is somehow especially well suited to that one domain. At the same time, when others try it on other image-based domains and (I expect) find that it does work, that finding won’t really get published. And as a result, in the best case scenario where this is the new architecture everyone wants to use, the knowledge that it works broadly will get delayed much longer because it wasn’t communicated in a published paper. If this gets in, please include in your talk/poster additional domains to get the word out.\n* After writing the above rant I realize the same logic applies to the type of base algorithm used with Hadamax: I would really like to see this used with an actor-critic / policy gradient algorithm.\n* There is some degree of muddying the waters with the focus in the writing on vectorized RL. I don’t think that is the main point, nor should it be entirely downplayed, but because of how the paper is structured it leads the reader in that direction for long enough to distract from the core contributions.\n* Figure 2 is great and the math is clear enough to implement, but I would have benefitted from another diagram/visual to dive deeper into the Hadamard product followed by max pooling component.", "was_revised": true, "final_justification": "Strong paper and continue to advocate for acceptance. The rebuttal slightly raised my estimation and will discuss with reviewers.", "rebuttal": "Thank you for your positive review—we really appreciate your high rating and enthusiasm for the simplicity, reproducibility, and strong empirical performance of Hadamax. We'll address your stated weaknesses and questions below:\n\nWeaknesses:\n\n- We agree with the reviewer that implementations across different image-based domains would present more opportunities in terms of the academic contribution and transfer of knowledge. However, as much as we would like to have included, for example, the full pixel-based MuJoCo domain, the compute requirements are a limiting factor. The full 57 game Atari suite for 200M frames (as is used in the original DQN, DDQN, Rainbow, C51 papers) is already a very demanding benchmark. \n\nHaving said this, we do see that environment diversity is a recurring theme amongst the reviewers and wanted to address some of these concerns. We have therefore conducted preliminary experiments using the **actor-critic** based PPO, on a pixel-based VizDoom (Deathmatch) environment for 40M frames:\n\n\n\n| VizDoom Deathmatch | 10M | 20M | 30M | 40M |\n|---------|--------------|-------------|--------------|---------------|\n| Hadamax | **-1.62** ± 0.20 | **8.08** ± 1.61 | **19.36** ± 1.82 | **29.38** ± 10.31 |\n| Baseline | -2.81 ± 0.40 | 1.27 ± 0.80 | 3.10 ± 0.89 | 5.21 ± 1.27 |\n\nNote that, as in the PQN framework, we only substituted the original encoder with a Hadamax encoder. Although Vizdoom is usually trained for longer, there is already a clear performance boost over the baseline encoder, and we will add the results, as well as the code, in the camera-ready version.\n\n- As for your comments on the poster/presentation, if our work gets accepted, we will be sure to try and include as many use-cases as possible.\n\n- About the emphasis on vectorized RL: We see your point and apologize if this muddied the waters. We'll slightly revise the structure of the manuscript for a clearer storyline in the camera-ready version.\n\n- We have already included the precise code of the encoder in Appendix B. Or do you mean that readers could have benefited from it in a more visual way? If it’s the latter, we are more than willing to create a more in-depth diagram for the camera-ready version.\n\nQuestions:\n- The biggest limitation of Hadamax is that it increases the parameter size of the encoder and adds additional training time, in comparison to a less complex baseline encoder. Furthermore, the Hadamax encoder, in this form, has been designed for pixel-based environments only where a CNN feature extractor is followed by a bottleneck (Linear layers).\n- We think that, because pixel-based inputs (especially in Atari) are high-dimensional, the encoder synthesis differs from that of non-pixel domains. We assume Hadamax works so well here because it improves the feature extraction process. In non-pixel-based domains, the hidden layers are commonly larger than the original input spaces, so in that case we see the MLP encoder as more of an ‘optimization vessel’ rather than a true (compressing) encoder. From what we’ve seen, these MLP encoders on state-based inputs generally have different design requirements.\n\nWe thank you once again for your detailed review, and praise of our work. We will implement your suggestions as well as the VizDoom results in the main paper, in order to strengthen the camera-ready version."}, {"review_id": "NslS3pZPHu", "reviewer": "Reviewer_J3f7", "summary": "This paper introduces a new network architecture for pixel-based model-free RL called Hadamax encoder. It replaces ResNet as the visual encoder in the PQN algorithm. It achieves a new SOTA performance on Atari-57, while being much faster in terms of computation.", "questions": "- Will such an architecture change also improve performance in e.g. standard computer vision benchmarks? Can it compete with the newest visual encoders? Why or why not?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n\n**Quality**\n- Paper written with high quality, \n- It is nice the performance improvement can be achieved by only changing the network architecture. \n- The insights such as that max-pooling might be overlooked in RL, \"average pooling achieves the opposite\" are interesting\n- Ablation is nice. \n- Many technical details are provided.\n\n**Clarity**\n- Paper is overall clear and easy to follow. \n\n**Significance**\n- The resulting performance and wall-clock time improvement are quite significant and convincing. \n\n**Originality**\n- While the proposed changes are relatively simple, it is novel to show that they can lead to very significant performance gain, which also brings new insights. \n\nWeaknesses\n\nMajor: \n- Paper can greatly benefit from further analysis of why certain components such as max-pooling can help so much? Currently there is only empirical result and the arguments are entirely based on the performance comparisons. Does it provide better visual features? Does it make training more stable? \n\n\nMinor:\n- The proposed method is faster but does it also take less computation per forward pass? I would suggest in the intro mention whether Hadamax encoder is faster per forward pass, or is it slower per forward pass but finish training at a smaller number of steps and thus leads to a smaller wall-clock time. Good to show comparison of total computation. \n- Can benefit from testing on more benchmarks such as any MuJoCo benchmarks with visual input. Less important. The results on Atari are already quite convincing and might be too much work.", "was_revised": true, "final_justification": "The rebuttal addressed most of my concerns, I still believe the paper can further benefit from more analysis, but I also see the paper has pretty strong empirical result and is well-written. The new results in the rebuttal are also helping to strengthen the paper's significance. I maintain my score of 4: Borderline accept.", "rebuttal": "We sincerely thank you for your supportive review. Your appreciation for the writing quality, insightful components (e.g., max pooling's overlooked role), thorough ablations, and performance gains is encouraging. We will respond to your points made.\n\n- We agree with your point of looking beyond performance analysis for certain components. As a result, we have done additional experiments analyzing two feature metrics for the isolated Hadamard and Maxpooling components. In addition to the effective rank, we have now also analyzed convolutional channel cosine similarity, as done in [1]. A low cosine similarity indicates that the channels are extracting dissimilar or uncorrelated features from the input data, which is desirable and suggests diversity among the channels.\nThe results are interesting. Our paper already shows increased effective rank (Fig. 8) in the deeper convolutional feature space (layer 2 & 3) as compared to the baseline. However, investigating both Hadamard and Maxpooling separately shows that Maxpooling increases effective rank as compared to the baseline, but also slightly increases channel cosine similarity. Meanwhile, the Hadamard component also slightly increases effective rank but massively decreases channel cosine similarity by up to 90% as compared to the baseline. This gives further insights into why the combination of Maxpooling and Hadamard could work so well, as Hadamard strongly compensates against the increase in channel cosine similarity caused by Maxpooling. We will add these results to the main paper, along with a detailed explanation. A table summarizing the results is seen below:\n\n| Metric                          | Baseline | + Maxpool       | + Hadamard              | Hadamax                          |\n|---------------------------------|----------|--------------------------|----------------------------------|----------------------------------|\n| Convolutional Effective Rank    | Base     | +10%              | +10%                    | **+10-20%**      |\n| Channel Cosine Similarity       | Base     | +20%      | **-90%**   | *-50%*  |\n\n- We have analyzed the forward pass computation time for both Hadamax encoder, baseline encoder, and the Rainbow encoder. The inference times are as follows:\n\n\n| Architecture | Inference time (seconds) |\n|----------------|----------------|\n| Rainbow | 0.00059 |\n| PQN | 0.00039 |\n| PQN (Impala) | 0.00140 |\n| Hadamax-PQN | 0.00175 |\n\nThis shows that due to setting the convolutional stride to 1, and adding maxpooling elements as well as Hadamard interactions, the inference time of the encoder will increase. Hadamax inference time is slightly higher than Impala inference time. We will be sure to add these inference times to the main paper in the camera-ready version.\n\n- We are grateful for the reviewer acknowledging the scope of the Atari-57 benchmark. We have however ran an additional evaluation on a PPO baseline in the pixel-based Vizdoom environment: Deathmatch. This encoder was similar to the Atari encoder, which allows us to directly implement the Hadamax encoder here. The results are as follows:\n\n| VizDoom Deathmatch | 10M | 20M | 30M | 40M |\n|---------|--------------|-------------|--------------|---------------|\n| Hadamax | **-1.62** ± 0.20 | **8.08** ± 1.61 | **19.36** ± 1.82 | **29.38** ± 10.31 |\n| Baseline | -2.81 ± 0.40 | 1.27 ± 0.80 | 3.10 ± 0.89 | 5.21 ± 1.27 |\n\nThe results show a significant increase over the baseline encoder, and will be added to the camera-ready version to improve the environment diversity. \n\n- We have not run any tests in standard computer vision benchmarks. We do think that the Hadamax architecture, or maybe a revised, more complex version thereof, can also improve performance in standard computer vision benchmarks. However, this is always hard to predict, because if we look at the other way around, the newest computer vision encoders generally do not transfer well to pixel-based RL. This is a part of the reason that a new state-of-the-art algorithm such as PQN still uses the 'nature' DQN encoder from 2015.\n\nThank you again for your insights. We will implement the more detailed convolutional feature analysis, the VizDoom results and the inference times in our manuscript, in order to strengthen the camera-ready version of the paper.\n\n[1] Manifold Regularized Dynamic Network Pruning, Tang et al, 2021."}, {"review_id": "rLoBDGBa4z", "reviewer": "Reviewer_PtqA", "summary": "In this paper, the authors focus on the problem of model free reinforcement learning applied to the Atari video game domain. To this end, the authors propose an architectural modification to the image based encoder of PQN. They apply max pooling in addition to Hadamard products to create what they deem \"Hadamax.\" The authors then evaluate their approach on the Atari video game dataset. They compare against regular PQN as well as the Rainbow algorithm, showing improvement over baselines. Furthermore, they also perform an ablation analysis showing the ablative effects of max pooling, Hadamard operations, and and GELUs.", "questions": "1. Can you please elaborate on the technical novelty of the approach?\n\n2. How could the approach be applied to different domains or model architectures? Do you believe that performance improvements may be gained with other datasets/models?", "limitations": "Yes, the authors have discussed limitations in Conclusions and Future Work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The approach is very simple and easy to implement. I believe that I would have few issues attempting to reproduce the results in the paper. \n\n2. The paper is well written, clear, and easy to understand.\n\n3. Performance over baselines is quite impressive considering the simplicity of the approach (an architectural change to the image encoder).\n\nWeaknesses:\n\n1. The scope of the evaluation is very narrow. The experiments focus only on Atari-57; there are no other domains used for evaluation. It is thus difficult to understand how well the performance improvement could generalize across other datasets.\n\n2. The technical novelty is quite limited. The approach is largely a tweaking of the image encoder within the existing PQN framework. \n\n3. The scope of models is also narrow. It is unclear how this approach could improve image encoders for other RL architectures or even encoders for other tasks such as image recognition.", "was_revised": true, "final_justification": "The authors have addressed my concerns in the rebuttal. I therefore upgrade my score to 4.", "rebuttal": "Thank you for your detailed review and for highlighting the strengths: the simplicity, reproducibility, clarity, and impressive performance from architectural changes. We would like to respond to the weaknesses and questions raised.\n\n- We understand that the ALE 200M benchmark might seem like a limited evaluation. However, we would like to kindly reiterate that we have performed experiments on all of the 57 games for 200M frames in the Atari suite, similar to the original DQN, DDQN, C51, Rainbow papers etc. This still remains the largest, most diverse and challenging testbed in visual (discrete) RL. However, in order to increase environment diversity and take your concerns seriously, we have ran additional preliminary experiments on another pixel-based environment “VizDoom: Deathmatch”, using the actor-critic PPO algorithm:\n\n| VizDoom Deathmatch | 10M | 20M | 30M | 40M |\n|---------|--------------|-------------|--------------|---------------|\n| Hadamax | **-1.62** ± 0.20 | **8.08** ± 1.61 | **19.36** ± 1.82 | **29.38** ± 10.31 |\n| Baseline | -2.81 ± 0.40 | 1.27 ± 0.80 | 3.10 ± 0.89 | 5.21 ± 1.27 |\n\nAlso here, we show a remarkably strong increase over the baseline by replacing the encoder. We will add the results to our updated manuscript, extending our testbed beyond Atari.\n\n- The Reviewer has a point in that there is no thorough technical analysis or proof as a precursor to the encoder design. The Hadamax design has been the product of a generalist look on all of the recent work in non-algorithmic RL, in order to strongly improve the widely used, older ‘nature’ encoder that has been used in pixel-based RL since 2015. To the best of our knowledge, the only alternative encoder used in literature was the Impala encoder. We therefore believe that we bring novelty in the way that encoders are structured, also showing that deeper encoders such as Impala are not necessarily needed for the state-of-the-art performance gains that Hadamax can bring. Adding to this, we have now done additional experiments further analyzing the effect of the isolated Hadamard and the Maxpooling operators on the convolutional feature space. Next to the effective rank already discussed in the paper, we have analyzed the feature space channel cosine similarity, as done in [1]. A low cosine similarity will indicate that convolutional channels are extracting uncorrelated features from the input data, which is positive shows a feature diversity. Maxpooling has shown to increase effective rank but also slightly increases the channel cosine similarity. Meanwhile, the addition of the Hadamard component shows a strong decrease in channel cosine similarity by up to **90**%. This gives further insights into why combining Maxpool and Hadamard works so well, as the Hadamard interaction compensates against the increase in channel cosine similarity caused by Maxpooling. We will make sure to add these results to the main paper, strengthening  our reasoning behind the proposed architecture. A table summarizing the results is seen below:\n\n| Metric                          | Baseline | + Maxpool       | + Hadamard              | Hadamax                          |\n|---------------------------------|----------|--------------------------|----------------------------------|----------------------------------|\n| Convolutional Effective Rank    | Base     | +10%              | +10%                    | **+10-20%**      |\n| Channel Cosine Similarity       | Base     | +20%      | **-90%**   | *-50%*  |\n\n- In our paper (fig. 7) we show that Hadamax is also effective on the distributional C51 architecture. Next to this, we have now shown Hadamax to also be working on the PPO actor-critic architecture on VizDoom. We therefore believe that this approach can be applied to different domains and model architectures, as long as there is convolutional downsampling involved. Besides this, we believe that the practical effectiveness of the Hadamax encoder might also allow researchers working in e.g. Image Recognition to try and implement these architectural changes in their research on encoder synthesis.\n\nNonetheless, we thank you for the review on our work. We would however kindly ask to reconsider the given score, as we have now added strong results on an additional hard pixel-based environment (Vizdoom) in the paper, as well as more insights into the effect of Maxpooling and Hadamard on the convolutional features. Furthermore, we would like to emphasize that we believe we have made a very strong academic contribution in encoder synthesis in pixel-based RL. Most recent algorithms are still employing the ‘nature’ DQN encoder from 2015- we believe this indicates that successful encoder synthesis for Reinforcement Learning is very hard, due to the unstable nature of optimization as compared to Supervised Learning. To the best of our knowledge, Hadamax-PQN is now a state-of-the-art RL algorithm on Atari when looking at wall-clock time and performance.\n\n[1] Manifold Regularized Dynamic Network Pruning, Tang et al, 2021."}, {"review_id": "n8TrIPORrH", "reviewer": "Reviewer_kiFq", "summary": "This paper introduces the Hadamax encoder, a pixel-based neural network architecture for model-free reinforcement learning that combines max-pooling down-sampling, Hadamard products between parallel hidden layers, and GELU activations. Integrated into the Policy Quantization Network (PQN), Hadamax raises median human-normalized scores on Atari-57 by roughly 80% over vanilla PQN, surpasses Rainbow-DQN while training more than an order of magnitude faster, and transfers well to C51.", "questions": "- An exploration of how Hadamax integrates with data augmentation techniques such as DrQ, as well as with pruned or mixture-of-experts encoders, could help determine the compatibility of the proposed architecture with modern modular approaches.\n- It would be helpful to understand the parameter allocation between the encoder and the policy/value head, and how this distribution compares with other architectural configurations.\n- An evaluation of Hadamax on continuous-control benchmarks, such as DMC or Adroit, could offer insight into its generalization capabilities in domains requiring fine-grained spatial reasoning.\n- Clarification on the exclusion of residual connections and deeper networks would be appreciated, along with discussion on whether such design choices were influenced by concerns over diminishing returns or optimization instability.\n- Investigating the effects of substituting max-pooling with strided convolutions while preserving Hadamard interactions—or vice versa—could further isolate the contribution of each architectural component.\n- A quantitative analysis of GPU memory consumption and inference throughput in comparison to Rainbow-DQN would provide a more complete picture of the computational trade-offs introduced by the Hadamard operations.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The reported 80% performance gain over the PQN baseline and the outperformance of Rainbow-DQN with >10× shorter training times demonstrate clear practical impact.\n- The encoder’s design is conceptually simple—replacing strides with max-pooling and inserting an element-wise Hadamard branch—so practitioners can drop it into existing codebases with minimal engineering effort, but enjoy massive performance gains.\n- The ablation study cleanly isolates the contributions of max-pooling, Hadamard products, and GELU activations, strengthening the causal link between the architectural decisions and the observed gains.\n\nWeaknesses\n- All experiments are confined to the ALE 200M benchmark, leaving open whether the encoder generalizes to domains that demand fine-grained spatial reasoning or continuous actions, such as robotics or locomotion related environments such as DMC, Maniskill and Adroit.\n- The study compares mainly to PQN, an algorithmic rather than architectural baseline, and omits direct comparison to architectural strategies such as SoftMoE [1] and network pruning [2] based approaches.\n- The architecture exploration is shallow: residual connections are excluded, depth is limited, and modern components such as attention or normalization variants are not considered.\n- Computational analysis focuses on wall-clock training time but omits memory usage and inference latency.\n\n[1] Mixtures of experts unlock parameter scaling for deep rl, Obando-Ceron et al., ICML 2024\n[2] In value-based deep reinforcement learning, a pruned network is a good network, Obando-Ceron et al., ICML 2024", "was_revised": true, "final_justification": "I have reviewed the comments given by the reviewers and find that many of my concerns have been resolved and thus maintain my positive score.", "rebuttal": "We thank Reviewer kiFq for the extensive review, and acknowledging the practical impact of our encoder design. As the Reviewer’s stated weaknesses are similar to the questions, we will stick to addressing the questions 1 by 1.\n\n- We have done tests integrating Hadamax and the baseline with a mixture-of-expert encoder head, specifically the SoftMoE head [1]. Using a SoftMoE head, we noticed that  the performance dropped. We were hesitant to report this, as we could have made an implementation error, which would not have been fair to the SoftMoE authors. Furthermore, the authors of SoftMoE and RL network pruning [2] are working with, to the best of our knowledge, a different Atari domain with sticky-actions, and mostly experiment on a subset of the 57-game suite. It is therefore not possible to do objective comparisons using scores from their papers. In order for us to compare to the official results in the Rainbow and PQN paper, we follow the 57-game, 200M frame implementation on Atari without sticky actions, and report the median Human-normalized scores. Having said this, we do see strong potential in future work integrating e.g. SoftMoE with Hadamax, as Hadamax is focused on the convolutional part of the encoder, while the Mixture of Expert models are focused on the bottleneck part of the encoder. \n- We have calculated the parameter allocation between the encoder and the policy/value head:\n\n| Architecture | Encoder Parameters | Head Parameters | Encoder % of Total | Head % of Total | Total Parameters |\n|--------------|--------------------|-----------------|--------------------|-----------------|------------------|\n| Hadamax | 4,123,072 | 9,234 | 99.78% | 0.22% | 4,132,306 |\n| Baseline | 1,685,472 | 9,234 | 99.45% | 0.55% | 1,694,706 |\n| Rainbow | 77,984 | 7,418,770 | 1.04% | 98.96% | 7,496,754 |\n| Impala | 596,312 | 4,626 | 99.23% | 0.77% | 600,938 |\n\n- We agree that a Hadamax implementation on continuous control benchmarks would offer more insights. However, pixel-based continuous control is, like Atari-57, very compute intensive, as it requires a lot of samples to converge as opposed to state-based continuous control. Even though we concur that testing the capabilities in fine-grained spatial reason would be very interesting, we respectfully leave the integration of the Hadamax encoder in the pixel-based continuous control domain for future work. We did however run an additional evaluation experiment on the actor-critic architecture PPO in Vizdoom (Deathmatch), where we replaced the baseline encoder with Hadamax. While not continuous control, VizDoom also demands spatial reasoning in a dynamic FPS environment. Hadamax also achieves strong performance benefits over the baseline encoder in VizDoom, as shown in the following table:\n\n| VizDoom Deathmatch | 10M | 20M | 30M | 40M |\n|---------|--------------|-------------|--------------|---------------|\n| Hadamax | **-1.62** ± 0.20 | **8.08** ± 1.61 | **19.36** ± 1.82 | **29.38** ± 10.31 |\n| Baseline | -2.81 ± 0.40 | 1.27 ± 0.80 | 3.10 ± 0.89 | 5.21 ± 1.27 |\n\n\n- In Appendix E.1 we have an ablation where we ‘naively’ scale Hadamax networks in depth, which shows that simply scaling in depth yields decreasing returns for Hadamax (and for the baseline, but that is not shown here). Regarding the residual connections, IMPALA's architecture separates channel increases and downsampling into initial convolutions and pooling layers within the Conv sequences, allowing the residual blocks to operate on constant channels and spatial sizes for identity skips without needing projections. Our Hadamax encoder does not have this, creating channel mismatches that prevent direct residuals. We therefore did not experiment with residual connections. If you would prefer, we are willing to add an ablation to the camera-ready showing (projected) residual connections on the Hadamax encoder.\n- Sorry for the confusion, but do you mean, taking the full Hadamax encoder, but substituting the max-pooling with strided convolutional layers, thereby increasing the encoder’s convolutional depth? If that is the case, we are also able to add this ablation to the camera-ready version.\n- We have tested the inference times on the Pytorch models:\n\n| Architecture | Inference time (seconds) |\n|----------------|----------------|\n| Rainbow | 0.00059 |\n| PQN | 0.00039 |\n| PQN (Impala) | 0.00140 |\n| Hadamax-PQN | 0.00175 |\n\nAnd also the GPU memory of the Pytorch models:\n\n| Architecture   | Training Update Memory (MB) | Training Batch | Inference Memory (MB) | Inference Batch |\n|-----------------|----------------------|---------------------|----------------|----------------------|\n| Rainbow | 198.77          \t| 32             \t| 185.33    \t| 1               \t|\n| PQN| 254.54          \t| 256            \t| 155.87    \t| 8               \t|\n|PQN (Impala)  | 725.18          \t| 256            \t| 143.46    \t| 8               \t|\n| Hadamax-PQN | 2247.26         \t| 256            \t| 233.14    \t| 8               \t|\n\n\nThese tests were done on an Nvidia 3080 Laptop GPU. As expected, Hadamax-PQN uses more GPU memory than Rainbow. However, we kindly emphasize that these numbers are still only a fraction of the available GPU memory on any recent GPU. Furthermore, as also stated in the original PQN paper, PQN saves 26GB of replay-buffer RAM compared to Rainbow.\n\nWe thank you for your detailed review, and will implement the additional clarifications / ablations in the main paper and appendices of the camera-ready version, as they will certainly strengthen the paper.\n\n[1] Mixtures of experts unlock parameter scaling for deep rl, Obando-Ceron et al. 2024.\n\n[2] In value-based deep reinforcement learning, a pruned network is a good network, Obando-Ceron et al, 2024."}]

Author rebuttal:

Meta-review: The reviewers agree that the paper presents surprising empirical evidence that small tweaks to the existing PQN algorithm can result in significant performance increases. They found the paper to be clearly presented and the hypotheses to be well-evaluated.
Decision: Accept (poster)