Hadamard Representations: Augmenting Hyperbolic Tangents in RL

Authors: Jacob Eeuwe Kooi, Mark Hoogendoorn, Vincent Francois-Lavet
Primary area: reinforcement learning
Keywords: Representation Learning, Reinforcement Learning, Activation Functions

Abstract:
Activation functions are one of the key components of a deep neural network. The most commonly used activation functions can be classed into the category of continuously differentiable (e.g. tanh) and piece-wise linear functions (e.g. ReLU), both having their own strengths and drawbacks with respect to downstream performance and representation capacity through learning (e.g. measured by the number of dead neurons and the effective rank). In reinforcement learning, the performance of continuously differentiable activations often falls short as compared to piece-wise linear functions. We provide insights into the vanishing gradients associated with the former, and show that the dying neuron problem is not exclusive to ReLU's. To alleviate vanishing gradients and the resulting dying neuron problem occurring with continuously differentiable activations, we propose a Hadamard representation. Using deep Q-networks, proximal policy optimization and parallelized Q-networks in the Atari domain, we show faster learning, a reduction in dead neurons and increased effective rank.

Decision: reject
Reviewer scores: 5, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1730668412758

thread_type:
official_review

<thread_object>
creator: reviewer_vagz

<content>
summary:
The paper investigates the role of activation functions on performance in reinforcement learning. Continuously differentiable, such as Tanh, can have advantages, including bounded output and smooth gradients. Practically, however, ReLU activations are more preferred. The authors investigate the reason and identify that the number of saturating tanh activations for all inputs increases with experience throughout the training. This decreases the network’s representation power (measured by the effective rank). To resolve this problem, the authors introduce Hadamard representations (HR) with Tanh. The authors showed that HR with Tanh experiences less saturation and mitigates loss in effective rank, which also reflects on performance. Through a series of experiments, the paper demonstrates the effectiveness of HR with Tanh.

strengths:
This paper presents a novel approach to have more effective representations for reinforcement learning. Based on insights from previous works on dying/dormant neurons and their relationship to reduced network capacity, the authors present the Hadamard representations that experience less dormancy and thereby maintain the network representational capacity. I think this research is fundamental and important, and the proposed method is very simple to use and integrate into a wide range of methods, further enhancing its potential impact.

weaknesses:
- Some definition inaccuracies:
  - The authors claim that the dying neuron problem is not exclusive to ReLU but also extends to Tanh and provide a definition where the activation output $|\alpha_i| \approx 1$. This definition is problematic for two reasons: 1) Intuitively, the neuron is not ‘dead’ if its output is not zero, and 2) the definition doesn’t work for ReLU. Instead, I suggest making this definition about saturating neurons, which would only work for tanh and sigmoid. The figures measuring the fraction of dead units (e.g., Figure 6a) need to reflect the fraction of dead or saturated units instead.  
  - The authors mentioned that the definition of a dead/saturating neuron applies when we test the neuron in all data in the buffer. Is this quantity actually being measured in the experiments? I am asking this because going through all the data in the buffer at each evaluation time seems very computationally expensive, so I was wondering if some approximation is made in measuring this quantity in the experiments section (e.g., sampling a fixed small number of samples from the buffer instead).
- Relatively limited empirical evaluation:
  - The empirical evaluation could be more substantial. The authors used 8 Atari environments with only five independent runs (with some overlapping error bars). I encourage the authors to use a larger number of independent runs (e.g., 10) to have more statistically significant results. Adding more environments would be great, although I recognize the hurdle of running extra Atari games.
  - More relevant activation function baselines are needed. Specifically, concatenated ReLU has been shown to improve plasticity in reinforcement learning [1] in addition to adaptive rational activations [2]. I think adding these two baselines is very relevant to understanding the effectiveness of HR Tanh against other recently investigated activation functions. 

**Minor issues:**
- The discussion in lines 207-212 needs revising.
- In line 103, $r_t$ should be $r_{t+1}$.
- In line 205, $h^\prime(x)$ should be $z^\prime(x)$.
- “Stronly” in line 357 is a typo

&nbsp;
&nbsp;

Overall, I would like this paper to be accepted. There are a few fixable issues, so I’m willing to increase my score if the authors address those issues based on my feedback in a paper revision.

&nbsp;
&nbsp;

**References:**

[1]. Delfosse, Q., Schramowski, P., Mundt, M., Molina, A., & Kersting, K. (2024). Adaptive Rational Activations to Boost Deep Reinforcement Learning. International Conference on Learning Representations. 

[2]. Abbas, Z., Zhao, R., Modayil, J., White, A., & Machado, M. C. (2023). Loss of plasticity in continual deep reinforcement learning. Conference on Lifelong Learning Agents (pp. 620-636).

questions:
- The authors mentioned in line 367 that they considered addition operation instead of Hadamard product and showed it performs poorly. There was no reason or postulate for why this is the case. Can the author provide a probability analysis similar to the one given on page 5?
- If the score is normalized with respect to the ReLU activation baseline score, why doesn’t the final ReLU baseline reach 1 at the end of training in Figure 7a and Figure 9a but slightly less than 1?
- Why did the authors use HR Tanh only for the last layer? What happens if you use it in all layers? The authors did mention that the success might not translate to convolutional layers. Can the authors present an experiment showing that and suggest what the reason might be? Additionally, why wasn't HR Tanh applied for the last two fully connected layers in the architecture?
- In line 13, “linear-unit functions” should be “piece-wise linear functions”
- I don’t see the reason for the paragraph starting at line 467. It is not relevant to the conclusion of the paper, and hence it should be removed.
- In some environments, such as Seaquest and SpaceInvaders, HR hurts performance compared to ReLU. Is there a reason for why?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682707919

thread_type:
official_review

<thread_object>
creator: reviewer_3nuw

<content>
summary:
The paper proposes a novel activation function based on Hadamard representations to address the limitations of traditional activation functions in reinforcement learning. It highlights issues like vanishing gradients and dying neurons, demonstrating that the proposed method improves learning efficiency and representation capacity in deep Q-networks and proximal policy optimization.

strengths:
1. The proposed Hadamard representation effectively addresses dead neuron issue with existing activation functions, particularly in reinforcement learning contexts.
2. The experimental results show clear improvements in learning speed and performance metrics, providing solid empirical evidence for the claims made.
3. The paper presents a well-defined problem and clearly articulates its contributions, making the paper easy to understand.

weaknesses:
1. The lack of a diverse range of network architectures tested limits the generalizability of the findings. The paper should test HR on various networks and test HR on different layers. It would be better if HR were a universal activation function suitable for all layers, rather than only suitable for the last layer.
2. Absence of detailed comparisons with other novel activation functions that have emerged recently.
3. The paper did not conduct tests on diverse tasks, e.g. MuJoCo, DMControl. using only Atari does not demonstrate that this is a universal issue in reinforcement learning.
4. As mentioned in Line 279, the experiments are all performed on 8 Atari games. Testing on only 8 Atari games does not sufficiently demonstrate a stable improvement in performance.
5. In the PREVENTING DYING NEURONS section in Line 191, the analysis is not related to reinforcement learning task at all. So, claiming that HR is an activation function particularly suitable for reinforcement learning feels somewhat forced.

questions:
1. How do Hadamard representations compare with other recent activation functions not covered in the paper?
2. What potential challenges might arise when applying this method to different neural network architectures?
3. What is the result if HR is tested on various tasks, e.g. MuJoCo, DMControl, and more tasks on Atari?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697316724

thread_type:
official_review

<thread_object>
creator: reviewer_hmtr

<content>
summary:
This work aims to counteract the dead neuron and rank-collapse phenomena in RL. It proposes the 'Hadamard representation': simply duplicating the final encoder layer in the encoder of the 'Nature DQN' architecture and taking the elementwise product between the two outputs. Empirically, the results show an overall moderate increase in learning speed, together with lower dead neurons and higher rank-collapse as averaged over 8 Atari games.

strengths:
1. Overall, I found the paper well-written and easy to parse through.

2. Already in the introduction, the paper concisely introduces the problem settings and effectively conveys the tradeoff between ReLU and Tanh with results (fig. 1) and toy examples (fig. 2).

3. Evaluation is conducted both with canonical off-policy and off-policy architectures/algorithms (Nature DQN and PPO) widely used in the broader literature.

weaknesses:
Major:

1. The experiments are conducted on a very small fraction of only 8 hand-picked Atari games. While the performance of the proposed method is slightly higher than ReLU on average, this does not hold consistently even across these 8 games (as reported in the Appendix). Moreover, while these environments are notably sensitive to stochasticity, evaluation is carried out with only 5 random seeds I believe this to be very much insufficient for a paper with a simple architectural contribution that should be thoroughly validated. Thus, I am very much not convinced of the generality of these results, even across the full set of Atari environments, and would strongly encourage to expand this section. In case Atari is too demanding, I would suggest considering other more computationally-friendly benchmarks such as the 16 environments in the highly-parallelizable Procgen benchmark.

2. Even in the results of this paper, there does not seem to be a clear relationship between dead neurons/effective rank and performance, (e.g., ReLU outperforms Tanh while having a higher number of dead neurons). Thus, even analytically, I feel the paper in its current states leaves many unresolved questions and provides very little novel information.

3. I believe the claims about the probabilities of neurons 'dying' such as: "Taking a product of hyperbolic tangent activated neurons thus reduces the probability of neuron saturation from p to p^2." (ln242-243) are not correct, and potentially misleading, as they make independence assumptions that are not even reported by the paper's own results in Appendix C.3.

Minor:

Figures 3 and 5 show 32 very small graphs each with the value distribution of 16/512 hand-picked neurons at two hand-picked points during training. I did not find these figures convey much meaningful information and would suggest replacing them with a graph showing the percentage of dead neurons for tanh/relu/and Hadamard representations throughout training.

I would suggest adding Subsection numbers.

questions:
I would appreciate it if the authors could address, in their response, the criticism raised above. In addition:

- In the MLP block of several modern LLMs such as Llama, the hidden representation is computed as (using this paper's notation) f(A_1x + B_1) * (A_1x + B_2). According to the paper's own analysis in Section 3, this simpler version should entirely prevent the dead neuron phenomenon with respect to the input activations. I wonder if the authors have considered and/or have this version.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730748487631

thread_type:
official_review

<thread_object>
creator: reviewer_s1dp

<content>
summary:
This paper studies the dying neuron problem with tanh-type activations. The paper's first result is that bounded activation functions like tanh perform worse than ReLU-type activations. It is argued that the poor performance is probably due to the vanishing-gradient/dying-neuron problem faced by the tanh-type activations. The paper then proposes to solve the dying-neuron problem using Hadamard representations. Finally, it is shown that Hadamard representations outperform the standard DQN architecture.

strengths:
The paper is generally well-written. The proposed solution is simple to use and novel. The usefulness of the proposed method is shown to be more effective than standard DQN.

weaknesses:
The biggest issue with the paper is the weak empirical evaluation—particularly the following two points.
* The number of hidden units in Hadamard Representations is twice that of the base system. However, the comparison with the base system with 1024 hidden units is not done correctly. The base system with 1024 hidden units is not tuned. At least the learning rate of the base system should be tuned. I suspect that something like 5e-5 could be a good value for the learning rate for the base system with twice the number of hidden units. Using default hyper-parameters is insufficient when the architecture is changed by making it twice as wide. The paper has to convincingly show that  Hadamard representations provide a benefit over a well-tuned base system with twice as many hidden units. 
* Missing baseline. The paper does not compare Hadamard representations to resetting baseline like Redo [1] and continual backprop[2]. Why not just reset the units that have saturated in the tanh network? 

[1] Sokar et al. The dormant neuron phenomenon in deep reinforcement learning. ICML 2023
[2] Dohare et al. Loss of plasticity in deep continual learning. Nature 2024

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732194575217

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Message to all Reviewers.

comment:
We thank all reviewers for their reviews. Based on your reviews, we have run additional experiments and made significant changes (highlighted in red) to our manuscript. The most important changes are summarized here:

To adress concerns regarding comparison with a valid baseline, we have added a comparison with the ICLR 2024 Rational [1] Activation function in our original 8-environment DQN setting. This activation function has been added to the main plotting results in the Experiments section, as well as the per-game score in Appendix D.2.

To address multiple reviewers’ concerns about the lack of Atari environments, we have run the (nearly) full Atari suite of 51 games for 5 seeds on a new parallelizable version of DQN called “PQN” [2]. Here, we compare with ReLU, tanh, and the tanh Hadamard. As per standard practice, we have added the median-human normalized scores to our experiment section, showing consistent results across the nearly full Atari suite.

Finally, as reviewer HMtR pointed out, there seems to be a slight gap in understanding why the ReLU might be less affected by performance issues correlating with dead neurons, as opposed to the hyperbolic tangent. We have added an additional section in the main paper on page 9 to give insights into this problem. Specifically, we show that dead hyperbolic tangents transform weights into biases, while dead ReLU’s resemble network pruning.

We will additionally reply to each reviewer individually.

[1] Gallici et al. Simplifying Deep Temporal Difference Learning. Arxiv 2024

[2] Delfosse et al. Adaptive Rational Activations to Boost Deep Reinforcement Learning. ICLR 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1732791390474

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final message to all reviewers (Nov 28)

comment:
Dear reviewers,

Based on the reviews, we have done a significant amount of additional experiments since the first manuscript. We have compared against the [Rational](https://openreview.net/forum?id=g90ysX1sVs) activation in DQN on 8 Atari games, and compare against the [CReLU](https://proceedings.mlr.press/v232/abbas23a/abbas23a.pdf) in PQN on 51 Atari games. In PQN, we significantly outperform CReLU, which, like the Hadamard, has nearly double the parameters of the baseline. Interestingly, in PQN, switching from normal hyperbolic tangents to Hadamard hyperbolic tangents more than **doubles** the performance, without any hyperparameter changes. Individual games for PQN can now also be seen in Appendix D.2.

Furthermore, we have added a large section on Page 9 explaining the difference in the effects on the next layer for dying ReLUs and dying tanhs. Specifically, we show that dying tanhs essentially turn weights into biases.

Based on each reviewer, we have also made minor changes throughout the text. We have highlighted all changes in Red.

We hope to have clarified the reviewers' concerns, specifically in the empirical evaluation of the Hadamard representation.
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this paper, the authors proposes a novel activation function based on Hadamard representations (HR) to address limitations of traditional activation functions in reinforcement learning, such as vanishing gradients and dying neurons. The authors demonstrate improved learning efficiency and representation capacity using HR in Deep Q-Networks (DQN) and Proximal Policy Optimization (PPO).

The major concerns raised by the reviewers for the first draft lies in 

1, Limited evaluation and comparison: the empirical study in the first draft lies in limited Atari games, and mising the other activation functions. 

2, Weak justification in RL: the analysis in the "Preventing Dying Neurons" section is not related to RL tasks, making the claim that HR is particularly suitable for RL feel forced.

In the rebuttal period, the authors completed more Atari games comparison, partially addressed the first weakness. However, it is not clear whether this is universal applicable for other environments, without experiemnts on MuJoCo and DMC suite. Meanwhile, it is not clear whether the "Dying Neurons" is only for RL and supervised learning. I suggest the authors further investigate the issue and improve the draft.