Hadamard Representations: Augmenting Hyperbolic Tangents in RL

## Abstract

Abstract Activation functions are one of the key components of a deep neural network. The most commonly used activation functions can be classed into the category of continuously differentiable (e.g. tanh) and linear-unit functions (e.g. ReLU), both having their own strengths and drawbacks with respect to downstream performance and representation capacity through learning (e.g. measured by the number of dead neurons and the effective rank). In reinforcement learning, the performance of continuously differentiable activations often falls short as compared to linear-unit functions. We provide insights into the vanishing gradients associated with the former, and show that the dying neuron problem is not exclusive to ReLU’s. To alleviate vanishing gradients and the resulting dying neuron problem occurring with continuously differentiable activations, we propose a Hadamard representation. Using deep Q-networks and proximal policy optimization in the Atari domain, we show faster learning, a reduction in dead neurons and increased effective rank.

## 1 Introduction

Figure: Figure 1: Normalized performance training DQN in the Atari domain where the nonlinear activation function of the representation, defined as the final hidden layer, is either ReLU or tanh. A strong performance discrepancy due to a relatively small architectural change can be observed.
Refer to caption: x1.png

Out of all activation functions, the Rectified Linear Unit (ReLU) (Fukushima, ; Nair & Hinton, ) and its variants (Xu et al., ; Klambauer et al., ) have emerged as the most widely used and generally best-performing activation functions up until this day (Jarrett et al., ; Goodfellow et al., ). The strength of the ReLU activation lies in its ability to naturally avoid vanishing gradients when used in deeper networks, in contrast to the continuously differentiable activation functions, such as the sigmoid and the hyperbolic tangent (Glorot & Bengio, ).

A common drawback of using the ReLU activation is its limited expressivity in the context of shallow networks (see Fig. 2), as well as the phenomenon known as the dying ReLU problem (He et al., ; Lu et al., ). As training progresses, the number of dying ReLU’s tend to increase, resulting in a dying network and loss of network capacity (Dubey et al., ).

In reinforcement learning (RL) (Sutton & Barto, ), even though training results in a large number of dying ReLU’s (Gulcehre et al., ; Sokar et al., ), the ReLU function still remains the most popular activation for performance reasons (Henderson et al., ).
Similar to supervised learning (Teney et al., ), continuously differentiable activation functions such as the hyperbolic tangent are therefore not commonly used in RL (see Fig. 1). However, one might argue that their symmetrical, bounded shape and smooth gradient landscape offer optimization advantages that the ReLU lacks. Recent findings also indicate that a hidden layer activated by a hyperbolic tangent displays a high effective rank and thus a high layer expressivity (Kumar et al., ; Gulcehre et al., ). Despite being a theoretically sound candidate, its lack of success in RL has not been thoroughly investigated.

We therefore aim to provide insights into the hyperbolic tangent’s suboptimality, revealing that the vanishing gradient problem leads to dying neurons and under-utilization of the full network capacity. Based on these insights, we mitigate said effects by augmenting the original architecture. Specifically, we focus our research on the activations of the compressed representation of an encoder, defined as the final hidden layer, and provide an alternative to the conventional parameterization of this representation. Our contributions can be summarized as follows:

- •
We show that, in reinforcement learning, dying hyperbolic tangents are a phenomenon of a similar scale as the dying ReLU problem.
- •
A Hadamard representation (HR) is proposed, defining a latent representation as the Hadamard product of two separate, individually parameterized activation vectors.
- •
We empirically show that, without hyperparameter tuning or the use of auxiliary losses, a Hadamard representation yields significant performance gains over a standard hyperbolic tangent in the Atari domain, and reveal how it decreases dying neurons and increases the internal representations’ effective rank.

## 2 Preliminaries

Figure: Figure 2: A regression of three shallow neural network architectures on a random complex sinusoidal function ($y=10*torch.sin(7*x)+15*torch.sin(10*x)+5*torch.cos(5*x)$). The Tanh (HR) network emerges as the strongest function approximator, even while having less trainable parameters (501 vs 601 for Tanh & ReLU). To make a fair comparison, the Tanh and ReLU networks have one single hidden layer of size 200, while the Tanh (HR) network has a hidden layer of size 100. For the Tanh (HR) network however, we use two parallel linear layers preceding the hidden layer in order to be able to use the single hidden layer as the Hadamard product of two activations (see Section 3). For experiments comparing deeper networks, we refer the reader to Appendix C.1.
Refer to caption: x2.png

We consider an agent acting within its environment, where the environment is modeled as a discrete Markov Decision Process (MDP) defined as a tuple $(\mathcal{S},\mathcal{A},T,R,\gamma)$. $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $T:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$ is the environment’s transition function, $R:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{R}$ is the environment’s reward function and $\gamma\in[0,1)$ is the discount factor. A replay buffer $B$ is used to store visited states $s_{t}\in\mathcal{S}$ that were followed by actions $a_{t}\in\mathcal{A}$ and resulted in the rewards $r_{t}\in\mathcal{R}$ and the next states $s_{t+1}$. One entry in $B$ contains a tuple of past experience $(s_{t},a_{t},r_{t},s_{t+1})$. The agent’s goal is to learn a policy $\pi:\mathcal{S}\rightarrow\mathcal{A}$ that maximizes the expectation of the discounted return $V^{\pi}(s)=\operatorname{\mathbb{E}}_{\tau}[\sum_{t=0}^{\infty}\gamma^{t}R(s_{
t},a_{t})\mid s_{t}=s]$, where $\tau$ is a trajectory following the policy $\pi$.

Furthermore, we examine the setting where a high-dimensional state ($s_{t}\in\mathbb{R}^{v}$) is compressed into lower-dimensional activations $z_{t}\in\mathcal{Z}=\mathbb{R}^{w}$ where we call $\mathcal{Z}$ the representation space with $w\leq v$. This is done by means of a neural network encoding $e:\mathcal{S}\rightarrow\mathcal{Z}$ where $e$ represents the encoder.

## 3 Augmenting Hyperbolic Tangents

Continuously differentiable activations such as the hyperbolic tangent ($tanh$) and the sigmoid ($\sigma$) activations are fundamentally different than the $ReLU$ or its Linear-Unit descendants, which are non-symmetric and have a large part of the input space mapped to zero (leading to sparsity).
The hyperbolic tangent and the sigmoid output values in the ranges $[-1,1]$ and $[0,1]$, respectively. These functions are defined as:

$$ $tanh(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}},\qquad\sigma(x)=\frac{1}{1+e^{-x}}$ (1) $$

Both functions have the advantage of being differentiable everywhere, as well as being bounded. Furthermore, the sigmoid is well suited for output probabilities, while the tanh is convenient when requiring a zero-centered symmetrical output. However, both functions exhibit the vanishing gradient problem for saturating activations (Glorot & Bengio, ; Goodfellow et al., ).

### Dying Hyperbolic Tangents

Although common literature has focused on the dying ReLU problem (He et al., ; Lu et al., ; Gulcehre et al., ; Sokar et al., ), we find that hidden layers activated by hyperbolic tangents similarly show strong dying neuron behavior. When using a hyperbolic tangent in a deep neural network, a single neuron $\alpha_{i},i\in\mathbb{R}^{w}$, with $w$ the layer dimension, can be classified as dying if:

$$ $|\alpha_{i}|\approx 1\;\forall\;s_{t}\in B$ (2) $$

Where $s_{t}$ is an observation in buffer $B$. Note that an absolute value is used, as a hyperbolic tangent can either saturate towards -1 or 1.
Given that the hyperbolic tangent is an asymptotic function near its saturation point, an approximate equality must be considered, as the classification of its saturation will always remain qualitative ($|\alpha_{i}|\neq 1\;\forall\;s_{t}\in B$)
. To approximate the condition given in Eq. 2, the amount of dying hyperbolic tangents is calculated by using a kernel density estimation (KDE) (Silverman, ) on the activations $\alpha_{i},i\in\mathbb{R}^{w}$ of each individual neuron in the activation layer. In order to visualize activations in a hidden layer, a fixed subset of the KDE’s of the neurons $\alpha_{i}$ is taken. A clear visualization of dying hyperbolic tangents during training in the Atari Breakout environment can be seen by analyzing sixteen individual neuron KDE’s in Fig. 3.

Figure: (a) Tanh - $10^{6}$ iterations
Refer to caption: x3.png

More details on the KDE calculation and dying neuron classification can be found in Appendix B.

### Hadamard Representations (HR)

As Fig. 3 indicates that the activation of $z_{t}$ with a hyperbolic tangent leads to saturation and dying neurons, an augmentation of the representation architecture is proposed. In the conventional encoder setting, the networks’ final hidden layer is defined as $z^{enc}(x)=f(A_{1}x+B_{1})$, with $A_{i}$ and $B_{i}$ representing weight and bias parameters, the function $f()$ representing a nonlinear activation function while $x$ is the set of activations from the previous layer.
In order to reduce the information dependence on a single set of neurons, we propose using a Hadamard representation that augments the original representation with a parallel representation layer $z^{*}$. This can be interpreted as using a single highway layer with a closed carry gate (Srivastava et al., ). The final representation is defined as the Hadamard product between the aforementioned activations $z(x)=z^{enc}(x)\cdot z^{*}(x)$, where $z^{*}(x)=f(A_{2}x+B_{2})$. A visualization of the proposed architecture can be found in Fig. 4.

Figure: Figure 4: A visualisation of the Hadamard Representation architecture combined with the nature DQN architecture applied on a snapshot of the ‘SpaceInvaders’ Atari environment (Mnih et al., ). Vertical and horizontal bars represent convolutional and linear layers, respectively. A parallel linear layer providing independently parameterized hidden layer activations is integrated, where the Hadamard product of the two activations represents the final latent representation $z_{t}$ used for downstream learning.
Refer to caption: x5.png

#### Preventing Dying Neurons

Our key hypothesis is that the Hadamard representation can prevent saturation, hence alleviating vanishing gradients and dying neurons (See Eq. 2). To support our hypothesis, we investigate the derivative of a product of two functions. For the product of two arbitrary functions $g(x)\cdot h(x)$, the derivative is defined as $\uline{\color[rgb]{0.875,0.26171875,0.8984375}{\color[rgb]{0,0,0}g^{\prime}(x)
h(x)}}+\uline{\color[rgb]{0.296875,0.76171875,0.83203125}{\color[rgb]{0,0,0}g(
x)h^{\prime}(x)}}$.
In the context of using a sigmoid activation function for $f(x)$, the derivative of $z(x)$ becomes:

$$ $\displaystyle z^{\prime}(x)=\uline{\color[rgb]{0.875,0.26171875,0.8984375}{ \color[rgb]{0,0,0}A_{1}\sigma(A_{1}x+B_{1})(1-\sigma(A_{1}x+B_{1}))\sigma(A_{2 }x+B_{2})}}\color[rgb]{0,0,0}$ $\displaystyle+\uline{\color[rgb]{0.296875,0.76171875,0.83203125}{\color[rgb]{ 0,0,0}A_{2}\sigma(A_{1}x+B_{1})\sigma(A_{2}x+B_{2})(1-\sigma(A_{2}x+B_{2}))}}$ $$

If a neuron from $f(A_{1}x+B_{1})=0\hskip 2.84526pt\forall\hskip 2.84526ptx$, the gradient of the product becomes 0 while if a positive saturation is experienced i.e. $f(A_{1}x+B_{1})=1\hskip 2.84526pt\forall\hskip 2.84526ptx$, $z^{\prime}(x)$ can remain nonzero. For a product of two hyperbolic tangent functions, the derivative is defined as:

$$ $\displaystyle h^{\prime}(x)=\uline{\color[rgb]{0.875,0.26171875,0.8984375}{ \color[rgb]{0,0,0}A_{1}\text{sech}^{2}(A_{1}x+B_{1})\tanh(A_{2}x+B_{2})}} \color[rgb]{0,0,0}+\uline{\color[rgb]{0.296875,0.76171875,0.83203125}{\color[ rgb]{0,0,0}A_{2}\text{sech}^{2}(A_{2}x+B_{2})\tanh(A_{1}x+B_{1})}}$ $$

where $\operatorname{sech}^{2}$ is the derivative of a hyperbolic tangent. When using a hyperbolic tangent, the key difference with a sigmoid is that for any saturating value of $g(x)$, it will consistently keep the second part of the equation nonzero and dominated by $h(x)$, due to the hyperbolic tangents’ saturation into strictly nonzero values. This means that during saturation of a neuron in $g(x)$, the non-saturated function $h(x)$ is still able to maintain a non-trivial gradient in the product derivative $h^{\prime}(x)$, thus providing a mechanism to avoid vanishing gradients. We visualize the kernel densities during training with a Hadamard representation in Fig. 5. The individual representations before taking the Hadamard product can be found in Appendix B.

Taking a more formal approximation of neuron collapse, we start by defining the probability of a single neuron saturating as $\mathbf{p}$. Furthermore, in the case of a sigmoid or hyperbolic tangent, we assume symmetric saturation probabilities to both ends, defining the probability of a neuron saturating to one end of the spectrum as $0.5\mathbf{p}$. Under these assumptions, we show that interpreting a neuron as the product of two individual neurons will change saturation probabilities depending on the neuron’s activation function.

Figure: (a) Tanh (HR) - $10^{6}$ iterations
Refer to caption: x6.png

##### Hyperbolic Tangent:

In the case of the hyperbolic tangent, product saturation only occurs if strictly both neurons are saturated. This results in a probability of $\mathbf{p}\cdot\mathbf{p}=\mathbf{p}^{2}$. Taking a product of hyperbolic tangent activated neurons thus reduces the probability of neuron saturation from $\mathbf{p}$ to $\mathbf{p}^{2}$.

##### Sigmoid:

For the sigmoid function, product saturation occurs in two scenarios: Either one of the neurons is saturated towards zero or both neurons are saturated towards 1. The probability that a single neuron does not saturate towards zero is $(1-0.5\mathbf{p})$, and subsequently the probability that neither neuron saturates towards zero is $(1-0.5\mathbf{p})^{2}$. The probability that at least one of the two neurons saturates to zero is therefore $1-(1-0.5\mathbf{p})^{2}=\mathbf{p}-0.25\mathbf{p}^{2}$. Adding the probability that both neurons saturate towards 1, which is $(0.5\mathbf{p})^{2}=0.25\mathbf{p}^{2}$, the final probability of the neuron product saturation is $\mathbf{p}-0.25\mathbf{p}^{2}+0.25\mathbf{p}^{2}=\mathbf{p}$. Taking a product of sigmoid activated neurons therefore does not reduce the probability of neuron collapse.

##### Rectified Linear-Unit:

In the case of a ReLU activation, we also assume the probability of a single neuron dying to be $\mathbf{p}$. As we look at the product of two neurons, the probability that one of the two neurons does not saturate is therefore $1-\mathbf{p}$, and the probability that both neurons do not saturate is $(1-\mathbf{p})^{2}$. The probability that at least one neuron saturates is thus equal to $1-(1-\mathbf{p})^{2}=2\mathbf{p}-\mathbf{p}^{2}$. As the ReLU saturation results in strict zeroes, this results in the product also being zero. Taking a product of ReLU activated neurons therefore increases the final neuron saturation probability from $\mathbf{p}$ to $2\mathbf{p}-\mathbf{p}^{2}$. For an overview, we refer the reader to both Table 1 and the corresponding empirical evidence in Appendix C.3.

**Table 1: Predicted dying neuron probabilities with and without a Hadamard representation.**
| Activation Function | Dying Neuron Probability | Probability with HR | Difference |
| --- | --- | --- | --- |
| Hyperbolic Tangent | $\mathbf{p}$ | $\mathbf{p}^{2}$ | $-(\mathbf{p}-\mathbf{p}^{2})$ |
| Sigmoid | $\mathbf{p}$ | $\mathbf{p}$ | 0 |
| ReLU | $\mathbf{p}$ | $2\mathbf{p}-\mathbf{p}^{2}$ | $+(\mathbf{p}-\mathbf{p}^{2})$ |

## 4 Experiments

The main experiments and discussion are conducted with the value-based DQN algorithm (Mnih et al., ) in Atari, and complemented by an evaluation on the policy-based proximal policy optimization algorithm (PPO) (Schulman et al., ). In all experiments, we build on the implementations from cleanrl  (Huang et al., ). For more details on the PPO and DQN implementation, we refer the reader to Appendix A.1. For both DQN and PPO, we define the representation $z_{t}\in\mathbb{R}^{512}$ as the last hidden layer of the network, which serves as the compressed representation of the original pixel observation $s_{t}$ (see Fig. 4).

We perform all experiments on 8 common, non-exploration driven Atari games. We aim to show the effect of a Hadamard representation on the representation’s fraction of dying neurons, its effective rank and its downstream performance. For more information on the dying neuron and effective rank calculations, we refer the reader to Appendix. B.1 and Appendix. B.2, respectively. The performance is aggregated over 8 environments and normalized using the ReLU baseline scores.

Figure: (a) Dead Neurons
Refer to caption: x8.png

#### Mitigating dying Hyperbolic Tangents

As mentioned in Section 3 and defined in Eq. 2, the number of dead neurons is equal to the amount of neurons that display the same saturated output for any given observation $s_{t}$. The amount of dead neurons over time when training on Atari can be seen in Fig.  6(a). For the ReLU activation, around 60% of the neurons in the representation $z_{t}$ die during training, while for the sigmoid and hyperbolic tangent activation this number is around 40%. When using a hyperbolic tangent Hadamard representation, a reduction in dead neurons as compared to using a single hyperbolic tangent can be observed. We credit this to the inherent ability of a Hadamard product of hyperbolic tangents to minimize long-term activation saturation, as explained in Section 3. Quantitative results of dead neurons can be found in Table 2, which is correlated with earlier predictions from Table 1.

**Table 2: Dying neuron fractions with and without a Hadamard representation (HR).**
| Activation Function | Dead Neuron Fraction | Dead Neuron Fraction (HR) | Difference |
| --- | --- | --- | --- |
| Hyperbolic Tangent | 0.39 | 0.30 | -23% |
| Sigmoid | 0.44 | 0.45 | +2% |
| ReLU | 0.62 | 0.73 | +18% |

For more dying neuron graphs related to the ReLU and the Sigmoid activations, we refer the reader to Appendix C.3.

#### Increasing Effective Rank

We additionally investigate the effective rank (Kumar et al., ) of the representation $z_{t}$ during training, which can be seen in Fig. 6(b). As observed by Gulcehre et al. (), a representation activated by a hyperbolic tangent or a sigmoid, already has a relatively high effective rank compared to a representation activated by a ReLU. Furthermore, similar to the results in our supervised learning experiments (see Fig. 2), using a Hadamard representation with hyperbolic tangents significantly improves the effective rank, which is strongly correlated to a network’s ‘expressivity’. Employing a Hadamard representation with ReLU activations significantly decreased the effective rank of the representation, as was expected. For more information on the effect of a Hadamard representation on a ReLU, we refer the reader to Appendix C.4.

#### Analyzing Performance in Atari

The influence of a Hadamard representation on downstream performance can be seen in Fig. 7. Correlating with the reduction in dying neurons and an increase in effective rank, a significant improvement of nearly $60\%$ over the standard hyperbolic tangent baseline is obtained, as well as an improvement over the default ReLU baseline. Note that no hyperparameter or architectural tuning has been employed in our experiments.

Figure: (a) Normalized Score
Refer to caption: x10.png

Figure: Figure 8: Additional ablations run on DQN in the Atari domain. Tanh (+) represents an addition rather than a Hadamard product, and Tanh (2HR) uses a triple Hadamard product.
Refer to caption: x12.png

Examining further ablations in Fig. 8 shows that using a sophisticated Linear-Unit function such as the SELU activation (Klambauer et al., ) or using an addition rather than a product of hyperbolic tangents seems detrimental to performance. Furthermore, taking a product of 3 hyperbolic tangents (2HR) also appears to enhance performance, though there seems to be a negative effect in the early stages of training as compared to using a single Hadamard product. We hypothesize that this is the result of increased contracting behavior in the early stage of training due to increasing multiplication of hyperbolic tangent activations whose absolute values are $<1$. For human-normalized scores, we refer the reader to Appendix D.1.

#### Evaluating Hadamard representations on PPO

To evaluate results on a policy-based algorithm, additional experiments are run using the PPO algorithm (Schulman et al., ). For PPO, the internal architectural difference with DQN is that the compressed representation $z_{t}$ precedes both a critic and an actor network, and thus receives policy and value gradients (see Appendix A.2). The results can be found in Fig. 9. Similarly to the results on DQN, the Tanh (HR) exhibits the highest effective rank and the best performance. The Tanh (HR) and the ReLU also seem to have consistent strong performance across both algorithms, whereas the pure hyperbolic tangent and sigmoid activations can be relatively unreliable across algorithms.

Figure: (a) Normalized Score
Refer to caption: x13.png

Interestingly, it seems that the ReLU activations’ performance is less correlated to its low effective rank and high amount of dying neurons than the continuously differentiable activations. As also indicated by Teney et al. (), the exact reasons for the broad success of the ReLU activations are not yet fully understood. A qualitative overview of our results can be found in Table 3.

**Table 3: High-level summary of the strengths of a Hadamard representation.**
| Activation function | Performance | Few dead neurons | Effective rank |
| --- | --- | --- | --- |
| Tanh (HR) | ++ | ++ | ++ |
| Tanh | - | - | + |
| ReLU | + | – | – |
| Sigmoid | - | - | - |

### Increasing Representation Dimensionality

A Hadamard representation increases the amount of trainable parameters in the encoder without increasing the dimension of the representation $z_{t}\in\mathbb{R}^{512}$. This is an important feature as simply increasing the dimension of the representation to $\mathbb{R}^{1024}$ is not often effective (see Fig. 7(b)). Even though the fraction of dying neurons decreases and the effective rank increases when using $z_{t}\in\mathbb{R}^{1024}$, it seems that the ’relative’ effective rank, defined as the ratio $\frac{rank}{dim(z_{t})}$, actually goes down (See Appendix. C.2). We therefore hypothesize that reducing dying neurons and increasing a representation’s effective rank is only superficially solved by expanding representation dimensionality.

## 5 Related Work

##### Network Capacity in RL.

Liu et al. () investigated the need for sparse representations in the continuous control domain. Gulcehre et al. () analyzed network expressiveness in RL by measuring the effective rank (Kumar et al., ) of the compressed representation, and found that hyperbolic tangent representations generally maintain high rank while not suffering strongly from rank decay as training continues. Related work used normalization techniques and action penalization to counteract high variance in pixel-based robotic control (Bjorck et al., ). Other work by Lyle et al. () investigated capacity loss in RL and similarly found that, as training progresses, the inherent network capacity of RL algorithms decays. Further research by Nikishin et al. () used network resets to counteract the primacy bias and Sokar et al. () evaluated and mitigated the dying ReLU phenomenon in DQN, both operating in the sample efficiency setting. Nikishin et al. () further studied plasticity injection for long-term training. In another related direction, recent work has investigated network sparsity in RL, showing that a large part of network capacity might be unnecessary when training reinforcement learning (Arnob et al., ; Graesser et al., ; Sokar et al., ; Tan et al., ; Obando-Ceron et al., ). This provides further insights into why a ReLU can achieve strong performance despite resulting in a significant number of dead neurons.

##### Network Architecture in RL

The origin of network optimization problems with hyperbolic tangents and sigmoids were empirically investigated by Glorot & Bengio (), where, according to the authors, a lot of mystery still surrounds the subject. Work by Srivastava et al. () in supervised learning first looked at the idea of using products of hidden layers together with a ’gate’ that determined the amount of information flow (Hochreiter & Schmidhuber, ). Using these ideas, the Resnet was invented (He et al., ) and also showed strong performance in combination with RL (Espeholt et al., ). Further work by Henderson et al. () showed differences in RL performances over different network architectures and nonlinear activations. Work by Abbas et al. () successfully applied ReLU concatenation (Shang et al., ) to improve continual learning while keeping a similar performance when training from scratch. Finally, recent work by Grooten et al. () investigated raw pixel masking for distractions in RL using a parallel CNN input layer.

## 6 Conclusions and Discussion

This paper analyzed issues with continuously differentiable activations in RL and demonstrated that these activation functions also suffer from the dying neuron problem. Based on this analysis, we propose a novel representation architecture, the Hadamard Representation (HR), which enhances an encoder’s final hidden layer by taking the Hadamard product with a parallel, independently parameterized activation layer. We further discussed and empirically showed that applying the Hadamard representation with hyperbolic tangents reduces the occurrence of dead neurons in the representation and increases layer expressiveness. This approach significantly improved performance in Atari games compared to both standard representation parameterizations and merely increasing the representation dimension.

As investigated by Kumar et al. (), a compressed representation activated by a hyperbolic tangent attains a much higher effective rank than when activated by a ReLU, and will therefore yield stronger network expressiveness. In light of this, the hyperbolic tangent can potentially replace the ReLU in a variety of architectures where expressiveness is required without using deeper networks.

Future work could focus on further identifying the intricacies of hyperbolic tangents in an attempt to push the potential of continuously differentiable activations in reinforcement learning or even in a supervised learning setting (see Fig. 2). Also, we believe that an implementation of Hadamard-style architectures in a continual learning setting as in Abbas et al. () could be promising.

## 7 Limitations

Our research has focused on the encoders’ final hidden layer in reinforcement learning. Both literature and preliminary experiments suggest that convolutional layers have a very strong preference for rectified activations (Jarrett et al., ), possibly due to a low-frequency bias (Ramasinghe & Lucey, ; Teney et al., ). We have therefore not conducted experiments integrating Hadamard representations in the convolutional section of our encoder architectures.
In addition, a hyperparameter search could further strengthen our results, since the baseline is specifically tuned for the ReLU activation. Furthermore, as we perform experiments on DQN and PPO, integration of a Hadamard representation into more sophisticated algorithms and architectures such as Rainbow (Hessel et al., ) and Impala (Espeholt et al., ) would be interesting.

## Reproducibility Statement

Constructing a Hadamard representation is a relatively straightforward process, of which the Pytorch pseudocode can be found in Appendix A.1. The DQN and PPO algorithms used are adopted from the compact cleanrl baselines (Huang et al., ). Finally, the details on the KDE calculation and the effective rank calculation can be found in Appendix B.1 and Appendix B.2.

## Appendix A Implementation Details

### A.1 Hyperparameters

To evaluate, 8 different Atari environments are tested, using 5 different random seeds. For the mean scores, we take the mean over the eight environments. Our normalized score is calculated according to our baseline, the original implementation using a ReLU activation.

All the hyperparameters used in our experiments for DQN and PPO, respectively, are as reported in cleanrl (Huang et al., ). The hyperparameters can be found in Table 1 and Table 2.

**Table 4: DQN Hyperparameters**
| Hyperparameter | Value | Description |
| --- | --- | --- |
| Learning Rate | $1\times 10^{-4}$ | Learning rate for the optimizer |
| Discount Factor $(\gamma)$ | 0.99 | Discount for future rewards |
| Replay Memory Size | 1,000,000 | Size of the experience replay buffer |
| Mini-batch Size | 32 | Number of samples per batch update |
| Target Network Update Frequency | 1000 | Update frequency for the target network |
| Initial Exploration | 1.0 | Initial exploration rate in $\epsilon$-greedy |
| Final Exploration | 0.1 | Final exploration rate in $\epsilon$-greedy |
| Final Exploration Frame | 1,000,000 | Frame number to reach final exploration |
| Exploration Decay Frame | 1,000,000 | Frames over which exploration rate decays |
| Action Repeat (Frame Skip) | 4 | Number of frames skipped per action |
| Reward Clipping | [-1, 1] | Range to which rewards are clipped |
| Input Dimension | 84 x 84 | Dimensions of the input image |
| Latent Dimension | 512 | Dimension of the latent representation |
| Input Frames | 4 | Number of frames used as input |
| Training Start Frame | 80,000 | Frame number to start training |
| Loss Function | Mean Squared Error | Loss function used for updates |
| Optimizer | Adam | Optimization algorithm used |
| Optimizer $\epsilon$ | $10^{-5}$ | Adam Epsilon |

**Table 5: PPO Hyperparameters**
| Hyperparameter | Value | Description |
| --- | --- | --- |
| Learning Rate | $2.5\times 10^{-4}$ | Learning rate for the optimizer |
| Discount Factor $(\gamma)$ | 0.99 | Discount factor for future rewards |
| Number of Steps | 128 | Number of steps per environment before update |
| Anneal LR | True | Whether to anneal the learning rate |
| GAE Lambda | 0.95 | Lambda parameter for GAE |
| Number of Minibatches | 4 | Number of minibatches to split the data |
| Update Epochs | 4 | Number of epochs per update |
| Normalize Advantage | True | Whether to normalize advantage estimates |
| Clipping Coefficient | 0.1 | Clipping parameter for PPO |
| Clip Value Loss | True | Whether to clip value loss |
| Entropy Coefficient | 0.01 | Coefficient for entropy bonus |
| Value Function Coefficient | 0.5 | Coefficient for value function loss |
| Maximum Gradient Norm | 0.5 | Maximum norm for gradient clipping |
| Target KL | None | Target KL divergence between updates |
| Latent Dimension | 512 | Dimension of the latent representation |
| Optimizer | Adam | Optimization algorithm used |
| Optimizer $\epsilon$ | $10^{-5}$ | Adam Epsilon |

#### Hadamard Implementation

Constructing a Hadamard representation is a straightforward process that only requires an additional, parallel linear layer. Starting from the flattened convolutional features in DQN, the Pytorch pseudocode is defined as follows:

#### Reinforcement Learning

In DQN, the action $a_{t}$ is chosen following an $\epsilon$-greedy policy. With probability $\epsilon$, a random action is selected, and with $(1-\epsilon)$, the action maximizing the Q-value is chosen. The target $Y_{t}$ is defined as:

$$ $Y_{t}=r_{t}+\gamma Q^{\prime}(z_{t+1},\underset{a\in\mathcal{A}}{\operatorname *{arg\,max}}Q(z_{t+1},a)),$ (3) $$

where $Q^{\prime}(z,a)$ denotes the target Q-network, an auxiliary network that stabilizes the learning by providing a stable target for $Q(z,a)$. The parameters of $Q^{\prime}$ are updated less frequently to enhance learning stability. The loss function for training the network is:

$$ $\mathcal{L}_{Q}=\big{|}Y_{t}-Q(z_{t},a)\big{|}^{2}.$ (4) $$

Proximal Policy Optimization (PPO) operates on a different principle, utilizing policy gradient methods for policy improvement. PPO seeks to update the policy by maximizing an objective function while preventing large deviations from the previous policy through a clipping mechanism in the objective’s estimator. The clipped policy gradient loss $L^{CLIP}$ is defined as:

$$ $L^{CLIP}(\theta)=\mathbb{E}\left[\min(r_{t}(\theta)\hat{A}_{t},\text{clip}(r_{ t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t})\right],$ (5) $$

where $r_{t}(\theta)$ represents the ratio of the probabilities under the new policy versus the old policy, and $\hat{A}_{t}$ is the advantage estimate at timestep $t$. This clipped surrogate objective ensures gradual and stable policy updates.

### A.2 PPO Architecture

Figure: Figure 10: A visualisation of the Hadamard representation (HR) architecture combined with the PPO architecture applied on a snapshot of the ’SpaceInvaders’ Atari environment. A parallel linear layer providing an independent representation is integrated, where the Hadamard product of the two parallel representations represents the final representation $z_{t}$.
Refer to caption: x15.png

## Appendix B Kernel Density Estimations

As discussed in Section 3, we hypothesize that the differences between a hyperbolic tangent with and without an HR are due to the increased ability of the product of hyperbolic tangents being able to negate dying neurons. We further see this phenomenom when plotting a random selection of neurons from both the mask and the base representation in Fig. 11(a).

Figure: (a) Tanh (HR) - $5\cdot 10^{6}$ iterations
Refer to caption: x16.png

Figure: (a) Tanh - $10^{7}$ iterations
Refer to caption: x19.png

### B.1 KDE calculation

Firstly, to stabilize the KDE computation and avoid singularity issues, a small noise $\epsilon$, following a normal distribution, is added to each neuron’s activations:

$$ $\alpha^{\prime}_{i}=\alpha_{i}+\epsilon,\quad\epsilon\sim\mathcal{N}(0,\sigma^ {2})$ $$

where $\sigma^{2}=1\times 10^{-5}$. The bandwidth for KDE, crucial for the accuracy of the density estimate, is calculated using Scott’s rule, adjusted by the standard deviation of the jittered activations:

$$ $bw=n^{-\frac{1}{5}}\cdot\text{std}(\alpha^{\prime}_{i})$ $$

where $n$ is the number of samples in $\alpha_{i}$. The density of activations is then estimated using a Gaussian kernel:

$$ $f(x)=\frac{1}{n\cdot bw}\sum_{j=1}^{n}K\left(\frac{x-\alpha^{\prime}_{ij}}{bw}\right)$ $$

Here, $K$ denotes the Gaussian kernel function. In order to finally determine if a neuron is dead, the maximum value of the estimated density function $f(x)$ is compared against a predefined threshold:

$$ $\max(f(x))\geq\omega$ $$

where $\omega$ represents the predetermined threshold. In practice, after analyzing the individual neuron KDE’s, using an $\omega$ of 20 provides a strong approximation of actual dead neurons.

### B.2 Effective Rank calculation

In line with Kumar et al. (), the effective rank of a feature matrix for a threshold $\delta$ ($\delta=0.01$), denoted as $srank_{\delta}(\Phi)$, is given by $srank_{\delta}(\Phi)=\min\left\{k:\frac{\sum_{i=1}^{k}\sigma_{i}(\Phi)}{\sum_{
i=1}^{d}\sigma_{i}(\Phi)}\geq 1-\delta\right\}$,
where $\{\sigma_{i}(\Phi)\}$ are the singular values of $\Phi$ in decreasing order, i.e., $\sigma_{1}\geq\cdots\geq\sigma_{d}\geq 0$. Intuitively, the effective rank of a feature matrix represents the number of “effective” unique components that form the basis for linearly approximating the resulting Q-values. The calculation in Python is done as follows:

## Appendix C Additional Experiments

### C.1 Shallow and Deep Function Approximation

To further showcase the effect of activations on complex function approximation, we compare the single hidden layer Tanh (HR) network from Fig. 2 with a deep ReLU and Tanh network containing three hidden layers each. The comparison with shallow networks can be found in Fig. 13(a) and a comparison with deep networks can be found in Fig. 13(b).

Figure: (a) Comparison of shallow networks for a nonlinear regression task. The Tanh and ReLU networks have a single hidden layer of 200 neurons, while the Tanh (HR) has a single hidden layer of 100 neurons but two preceding linear layers. The Tanh and ReLU networks have 601 parameters, while the Tanh (HR) network has 501 parameters. As found by Gulcehre et al. (), a shallow network activated by ReLU has a lower effective rank and consequently reduced network expressivity as compared to a Tanh activated network. Using a Hadamard representation, we achieve better function approximation while using less parameters.
Refer to caption: x23.png

### C.2 Increasing Representation Parameters

Figure: (a) Performance
Refer to caption: x25.png

### C.3 Validating dying neuron probability derivations

As discussed in Section 3, the effect of using a Hadamard representation strongly depends on the activation function. These derivations are empirically validated by the results in Fig. 15. In practice, since a neural network prefers symmetry, a sigmoid saturates slightly faster to 0 than to 1. This could explain the very slight increase in dead neurons when using an HR with activations.

Figure: (a) Sigmoid with and without HR
Refer to caption: x28.png

### C.4 ReLU activated Hadamard representation

Additional Atari experiments are provided comparing a ReLU activation with and without an HR. The normalized scores, dying neurons and the effective rank during training can be seen in fig. 16.

Figure: (a) Normalized Score
Refer to caption: x31.png

## Appendix D Atari

### D.1 Evaluation Details

We normalize performance with respect to the ReLU baseline in cleanrl  Huang et al. (). The minimum and maximum score of the ReLU baseline are taken for each environment, and the normalized score for each environment is calculated as follows:

$$ $\text{Normalized Score}=\frac{\text{Score}-\text{Min Score}}{\text{Max Score}- \text{Min Score}}$ (6) $$

where Score refers to the raw performance score of the model being evaluated, Min Score is a single value representing the lowest score obtained by the ReLU baseline (usually equivalent to random policy or even slightly worse), and Max Score is a single value representing the highest score achieved by the ReLU baseline in the same environment. To average, we sum the normalized scores for every run and take the mean.

The more official Human-Normalized Score, as referenced in Mnih et al. (), is calculated similarly but using human and random performance benchmarks:

$$ $\text{Human-Normalized Score}=\frac{\text{Score}-\text{Random Score}}{\text{ Human Score}-\text{Random Score}}$ (7) $$

where Human Score and Random Score refer to the scores recorded by human players and random agents, respectively. Calculating our performance according to the Human-Normalized Score leads to the plot seen in Fig. 17. Due to taking a subset of the full atari domain, the VideoPinball environment is extremely dominant in the Human-Normalized Score calculation. For a more realistic comparison of the methods, we therefore decided to use baseline-normalized scores in the main paper.

Figure: Figure 17: Human-Normalized performance (in multiples) with the standard deviation over the means in the Atari domain for 10M iterations (40M Frames).
Refer to caption: x33.png

### D.2 Individual Environment Scores

Figure: Figure 18: DQN Performance comparison on the individual Atari Environments. Plotted lines represent the mean taken over 5 seeds, with the standard deviations expressed as the shaded region. Eight popular Atari games in the RL community that are non-exploration driven were chosen due to computational limitations. No other games were tested.
Refer to caption: x34.png

Figure: Figure 19: PPO Performance comparison on the individual Atari Environments. Plotted lines represent the mean taken over 5 seeds, with the standard deviations expressed as the shaded region. Eight popular Atari games in the RL community that are non-exploration driven were chosen due to computational limitations. No other games were tested.
Refer to caption: x35.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See Weaknesses
soundness: 2
strengths: The paper is generally well-written. The proposed solution is simple to use and novel. The usefulness of the proposed method is shown to be more effective than standard DQN.
summary: This paper studies the dying neuron problem with tanh-type activations. The paper's first result is that bounded activation functions like tanh perform worse than ReLU-type activations. It is argued that the poor performance is probably due to the vanishing-gradient/dying-neuron problem faced by the tanh-type activations. The paper then proposes to solve the dying-neuron problem using Hadamard representations. Finally, it is shown that Hadamard representations outperform the standard DQN architecture.
weaknesses: The biggest issue with the paper is the weak empirical evaluation—particularly the following two points.
* The number of hidden units in Hadamard Representations is twice that of the base system. However, the comparison with the base system with 1024 hidden units is not done correctly. The base system with 1024 hidden units is not tuned. At least the learning rate of the base system should be tuned. I suspect that something like 5e-5 could be a good value for the learning rate for the base system with twice the number of hidden units. Using default hyper-parameters is insufficient when the architecture is changed by making it twice as wide. The paper has to convincingly show that  Hadamard representations provide a benefit over a well-tuned base system with twice as many hidden units. 
* Missing baseline. The paper does not compare Hadamard representations to resetting baseline like Redo [1] and continual backprop[2]. Why not just reset the units that have saturated in the tanh network? 

[1] Sokar et al. The dormant neuron phenomenon in deep reinforcement learning. ICML 2023
[2] Dohare et al. Loss of plasticity in deep continual learning. Nature 2024

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: I would appreciate it if the authors could address, in their response, the criticism raised above. In addition:

- In the MLP block of several modern LLMs such as Llama, the hidden representation is computed as (using this paper's notation) f(A_1x + B_1) * (A_1x + B_2). According to the paper's own analysis in Section 3, this simpler version should entirely prevent the dead neuron phenomenon with respect to the input activations. I wonder if the authors have considered and/or have this version.
soundness: 2
strengths: 1. Overall, I found the paper well-written and easy to parse through.

2. Already in the introduction, the paper concisely introduces the problem settings and effectively conveys the tradeoff between ReLU and Tanh with results (fig. 1) and toy examples (fig. 2).

3. Evaluation is conducted both with canonical off-policy and off-policy architectures/algorithms (Nature DQN and PPO) widely used in the broader literature.
summary: This work aims to counteract the dead neuron and rank-collapse phenomena in RL. It proposes the 'Hadamard representation': simply duplicating the final encoder layer in the encoder of the 'Nature DQN' architecture and taking the elementwise product between the two outputs. Empirically, the results show an overall moderate increase in learning speed, together with lower dead neurons and higher rank-collapse as averaged over 8 Atari games.
weaknesses: Major:

1. The experiments are conducted on a very small fraction of only 8 hand-picked Atari games. While the performance of the proposed method is slightly higher than ReLU on average, this does not hold consistently even across these 8 games (as reported in the Appendix). Moreover, while these environments are notably sensitive to stochasticity, evaluation is carried out with only 5 random seeds I believe this to be very much insufficient for a paper with a simple architectural contribution that should be thoroughly validated. Thus, I am very much not convinced of the generality of these results, even across the full set of Atari environments, and would strongly encourage to expand this section. In case Atari is too demanding, I would suggest considering other more computationally-friendly benchmarks such as the 16 environments in the highly-parallelizable Procgen benchmark.

2. Even in the results of this paper, there does not seem to be a clear relationship between dead neurons/effective rank and performance, (e.g., ReLU outperforms Tanh while having a higher number of dead neurons). Thus, even analytically, I feel the paper in its current states leaves many unresolved questions and provides very little novel information.

3. I believe the claims about the probabilities of neurons 'dying' such as: "Taking a product of hyperbolic tangent activated neurons thus reduces the probability of neuron saturation from p to p^2." (ln242-243) are not correct, and potentially misleading, as they make independence assumptions that are not even reported by the paper's own results in Appendix C.3.

Minor:

Figures 3 and 5 show 32 very small graphs each with the value distribution of 16/512 hand-picked neurons at two hand-picked points during training. I did not find these figures convey much meaningful information and would suggest replacing them with a graph showing the percentage of dead neurons for tanh/relu/and Hadamard representations throughout training.

I would suggest adding Subsection numbers.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. How do Hadamard representations compare with other recent activation functions not covered in the paper?
2. What potential challenges might arise when applying this method to different neural network architectures?
3. What is the result if HR is tested on various tasks, e.g. MuJoCo, DMControl, and more tasks on Atari?
soundness: 2
strengths: 1. The proposed Hadamard representation effectively addresses dead neuron issue with existing activation functions, particularly in reinforcement learning contexts.
2. The experimental results show clear improvements in learning speed and performance metrics, providing solid empirical evidence for the claims made.
3. The paper presents a well-defined problem and clearly articulates its contributions, making the paper easy to understand.
summary: The paper proposes a novel activation function based on Hadamard representations to address the limitations of traditional activation functions in reinforcement learning. It highlights issues like vanishing gradients and dying neurons, demonstrating that the proposed method improves learning efficiency and representation capacity in deep Q-networks and proximal policy optimization.
weaknesses: 1. The lack of a diverse range of network architectures tested limits the generalizability of the findings. The paper should test HR on various networks and test HR on different layers. It would be better if HR were a universal activation function suitable for all layers, rather than only suitable for the last layer.
2. Absence of detailed comparisons with other novel activation functions that have emerged recently.
3. The paper did not conduct tests on diverse tasks, e.g. MuJoCo, DMControl. using only Atari does not demonstrate that this is a universal issue in reinforcement learning.
4. As mentioned in Line 279, the experiments are all performed on 8 Atari games. Testing on only 8 Atari games does not sufficiently demonstrate a stable improvement in performance.
5. In the PREVENTING DYING NEURONS section in Line 191, the analysis is not related to reinforcement learning task at all. So, claiming that HR is an activation function particularly suitable for reinforcement learning feels somewhat forced.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - The authors mentioned in line 367 that they considered addition operation instead of Hadamard product and showed it performs poorly. There was no reason or postulate for why this is the case. Can the author provide a probability analysis similar to the one given on page 5?
- If the score is normalized with respect to the ReLU activation baseline score, why doesn’t the final ReLU baseline reach 1 at the end of training in Figure 7a and Figure 9a but slightly less than 1?
- Why did the authors use HR Tanh only for the last layer? What happens if you use it in all layers? The authors did mention that the success might not translate to convolutional layers. Can the authors present an experiment showing that and suggest what the reason might be? Additionally, why wasn't HR Tanh applied for the last two fully connected layers in the architecture?
- In line 13, “linear-unit functions” should be “piece-wise linear functions”
- I don’t see the reason for the paragraph starting at line 467. It is not relevant to the conclusion of the paper, and hence it should be removed.
- In some environments, such as Seaquest and SpaceInvaders, HR hurts performance compared to ReLU. Is there a reason for why?
soundness: 3
strengths: This paper presents a novel approach to have more effective representations for reinforcement learning. Based on insights from previous works on dying/dormant neurons and their relationship to reduced network capacity, the authors present the Hadamard representations that experience less dormancy and thereby maintain the network representational capacity. I think this research is fundamental and important, and the proposed method is very simple to use and integrate into a wide range of methods, further enhancing its potential impact.
summary: The paper investigates the role of activation functions on performance in reinforcement learning. Continuously differentiable, such as Tanh, can have advantages, including bounded output and smooth gradients. Practically, however, ReLU activations are more preferred. The authors investigate the reason and identify that the number of saturating tanh activations for all inputs increases with experience throughout the training. This decreases the network’s representation power (measured by the effective rank). To resolve this problem, the authors introduce Hadamard representations (HR) with Tanh. The authors showed that HR with Tanh experiences less saturation and mitigates loss in effective rank, which also reflects on performance. Through a series of experiments, the paper demonstrates the effectiveness of HR with Tanh.
weaknesses: - Some definition inaccuracies:
  - The authors claim that the dying neuron problem is not exclusive to ReLU but also extends to Tanh and provide a definition where the activation output $|\alpha_i| \approx 1$. This definition is problematic for two reasons: 1) Intuitively, the neuron is not ‘dead’ if its output is not zero, and 2) the definition doesn’t work for ReLU. Instead, I suggest making this definition about saturating neurons, which would only work for tanh and sigmoid. The figures measuring the fraction of dead units (e.g., Figure 6a) need to reflect the fraction of dead or saturated units instead.  
  - The authors mentioned that the definition of a dead/saturating neuron applies when we test the neuron in all data in the buffer. Is this quantity actually being measured in the experiments? I am asking this because going through all the data in the buffer at each evaluation time seems very computationally expensive, so I was wondering if some approximation is made in measuring this quantity in the experiments section (e.g., sampling a fixed small number of samples from the buffer instead).
- Relatively limited empirical evaluation:
  - The empirical evaluation could be more substantial. The authors used 8 Atari environments with only five independent runs (with some overlapping error bars). I encourage the authors to use a larger number of independent runs (e.g., 10) to have more statistically significant results. Adding more environments would be great, although I recognize the hurdle of running extra Atari games.
  - More relevant activation function baselines are needed. Specifically, concatenated ReLU has been shown to improve plasticity in reinforcement learning [1] in addition to adaptive rational activations [2]. I think adding these two baselines is very relevant to understanding the effectiveness of HR Tanh against other recently investigated activation functions. 

**Minor issues:**
- The discussion in lines 207-212 needs revising.
- In line 103, $r_t$ should be $r_{t+1}$.
- In line 205, $h^\prime(x)$ should be $z^\prime(x)$.
- “Stronly” in line 357 is a typo

&nbsp;
&nbsp;

Overall, I would like this paper to be accepted. There are a few fixable issues, so I’m willing to increase my score if the authors address those issues based on my feedback in a paper revision.

&nbsp;
&nbsp;

**References:**

[1]. Delfosse, Q., Schramowski, P., Mundt, M., Molina, A., & Kersting, K. (2024). Adaptive Rational Activations to Boost Deep Reinforcement Learning. International Conference on Learning Representations. 

[2]. Abbas, Z., Zhao, R., Modayil, J., White, A., & Machado, M. C. (2023). Loss of plasticity in continual deep reinforcement learning. Conference on Lifelong Learning Agents (pp. 620-636).

Meta-review: In this paper, the authors proposes a novel activation function based on Hadamard representations (HR) to address limitations of traditional activation functions in reinforcement learning, such as vanishing gradients and dying neurons. The authors demonstrate improved learning efficiency and representation capacity using HR in Deep Q-Networks (DQN) and Proximal Policy Optimization (PPO).

The major concerns raised by the reviewers for the first draft lies in 

1, Limited evaluation and comparison: the empirical study in the first draft lies in limited Atari games, and mising the other activation functions. 

2, Weak justification in RL: the analysis in the "Preventing Dying Neurons" section is not related to RL tasks, making the claim that HR is particularly suitable for RL feel forced.

In the rebuttal period, the authors completed more Atari games comparison, partially addressed the first weakness. However, it is not clear whether this is universal applicable for other environments, without experiemnts on MuJoCo and DMC suite. Meanwhile, it is not clear whether the "Dying Neurons" is only for RL and supervised learning. I suggest the authors further investigate the issue and improve the draft.

Decision: Reject