Adaptive Regularization of Representation Rank as an Implicit Constraint of Bellman Equation
Authors: Qiang He
Tianyi Zhou
Meng Fang
Setareh Maghsudi
Venue: ICLR 2024 poster (2024)
Keywords: representation learning
deep reinforcement learning
Primary area: reinforcement learning
TL;DR: We introduce BEER, a regularizer based on the Bellman equation, to adaptively control the representation rank in Deep Reinforcement Learning, leading to improved performance in complex control tasks.

Abstract: Representation rank is an important concept for understanding the role of Neural Networks (NNs) in Deep Reinforcement learning (DRL), which measures the expressive capacity of value networks. Existing studies focus on unboundedly maximizing this rank; nevertheless, that approach would introduce overly complex models in the learning, thus undermining performance. Hence, fine-tuning representation rank presents a challenging and crucial optimization problem. To address this issue, we find a guiding principle for adaptive control of the representation rank. We employ the Bellman equation as a theoretical foundation and derive an upper bound on the cosine similarity of consecutive state-action pairs representations of value networks. We then leverage this upper bound to propose a novel regularizer, namely BEllman Equation-based automatic rank Regularizer (BEER). This regularizer adaptively regularizes the representation rank, thus improving the DRL agent's performance. We first validate the effectiveness of automatic control of rank on illustrative experiments. Then, we scale up BEER to complex continuous control tasks by combining it with the deterministic policy gradient method. Among 12 challenging DeepMind control tasks, BEER outperforms the baselines by a large margin. Besides, BEER demonstrates significant advantages in Q-value approximation. Our code is available at https://github.com/sweetice/BEER-ICLR2024.

Published as a conference paper at ICLR 2024
ADAPTIVE REGULARIZATION OF REPRESENTATION RANK AS AN IMPLICIT CONSTRAINT OF BELLMAN EQUATION
Qiang He', Tianyi Zhou", Meng Fang", Setareh Maghsudi
Ruhr University Bochum, UUniversity of Maryland, College Park, University of Liverpool (Qeang.He, Sttareh.Maghsuii Maaghsudd@@ruubdcceee @rub.de, tianyi umd.edu, Meng. @ liverpool.ac.uk
ABSTRACT
Representation rank is important concept for understanding the role of Neural Networks (NNs) in Deep Reinforcement learning (DRL), which measures the expressive capacity of value networks. Existing studies focus on unboundedly maximizing this rank; nevertheless, that approach would introduce overly complex models the learning, thus undermining performance. Hence, fine-tuning repre- sentation rank presents challenging and crucial optimization problem. To address this issue, we find guiding principle for adaptive control of the representation rank. We employ the Bellman equation theoretical foundation and derive an upper bound on the cosine similarity of consecutive state-action pairs representations of value networks. We then leverage this upper bound to propose novel regularizer, namely BEllman Equation-based automatic rank Regularizer (BEER). This regular- izer adaptively regularizes the representation rank, thus improving the DRL agent's s performance. We first validate the effecttiveness of automatic control of rank on illustrative experiments. Then, scale up BEER complex continuous control tasks by combining it with the deterministic policy gradient method. Among 12 challenging DeepMind control tasks, BEER outperforms the baselines by large margin. Besides, BEER demonstrates significant advantages in Q-value approxi- mation. Our code is available at
INTRODUCTION
Deep reinforcement learning (DRL), empowered by the large capacity of neural networks (NNs), has made notable strides variety sequential decision-makien making tasks, ranging from games (Mnih et al., 2015; Silver et al., 2018; Vinyals et al., 2019), to robotics control (Haarnoja et al., 2018) 8), and large language models (Christiano al., 2017; OpenAI, 2023). Yet, despite these advancements, NNs within DRL systems are still largely treated black-box function approximators (Schulman et al., 2017; Lillicrap et al., 2016; Fujimoto et al., 2018; Haarnoja et al., 2018) 8). Investigating the specific roles played by NNs in DRL. may offer insights into their function, enabling further optimization of DRL agents from the aspect of NNs. Recent work (Kumar et al., 2021; Lyle et al., 2022; He et al., 2023a) has begun to shed light on the role of NNs in RL, proposing to leverage some properties (e.g.. capacity (Lyle et al., 2022) and plasticity (Lyle et al., 2023; Nikishin et al., 2022; Sokar et al., 2023)) of NNs to improve DRL agents.
A core concept to study the properties of NNs in the DRL setting is the representation rank (Kumar al., 2021). It measures the representation capacity of NN by performing singular value decomposition on the output of its representation layer, which serves as the input representation for value functions. Previous studies (e.g., InFeR (Lyle al., 2022) and DR3 (Kumar et al., 2021)) heavily focus on unbounded maximizing this metric. Yet, that approach might cause overfitting by constructing an overly complex function approximation, thus inhibiting the model's ability to generalize to new data. The resulting complexity demands more data and makes the model susceptible to noise that deters sampling efficiency and robustness in RL. As such, an indiscriminate maximization of representation rank is not advisable. A more desirable method is adaptively control the representation rank. However, empirically fine- tuning the balance of representation rank can be tricky. An excessively complex model induces the issues mentioned above, whereas overly

---

Published as a conference paper at ICLR 2024
simple models lose the capability obtain the optimal policy. As result, it is imperative to search for a guiding principle that allows for the adaptive control of the representation rank.
A primary discovery of this paper is that adaptive control of the representation rank can be derived from Bellman equation (Sutton & Barto, 2018) as an implicitly imposed constraint on NNs. Specifically, this can be achieved by upper bound on the cosine similarity between the represen- tations of consecutive state-action pairs, determined by factors like the discount factor, the weights of the last layer of the neural network, and the representations norms, Cosine similarity measures the linear dependent relationship between the composition vectors, thus affecting the rank. The implication is profound: The cosine similarity is constrained, thereby restraining the representation rank itself (Kumar et al., 2021; Lyle et al., 2022; 2023). However, a previous work (Kumar et al., 2021) shows that the dynamics of NNs result in feature co-adaptation, which potentially makes DRL, agents fail to hold the bound. Thus, these derived upper bounds provide a criterion to adaptively control the representation rank. Motivated by these findings, we introduce a novel regularizer, namely BEllman Squation-based automatic rank Regularizer (BEER), designed to control the representation rank by regularizing the similarity between adjacent representations of the value network. That allows DRL agents adaptively control the rank based on the constraint of the Bellman equation and preserve desirable representation ranks. Specifically, the BEER regularizer only reduces the similarity (thereby increasing the representation rank) when the similarity exceeds the upper bound. Numerical experiments, e.g., in Figure I, show that our method outperforms existing approaches that do not control or overly strengthen the representation rank. For instance, we demonstrate that algorithms like InFeR, which keep maximizing the representation rank without constraint, produce high approximation errors. In contrast, our proposed regularizer, BEER, functions significantly better by adaptively regularizing the representation rank. We further scale up BEER to challenging continuous control suite DMcontrol (Tunyasuvunakool al., 2020), where BEER performs better than the existing methods DR3 and InFeR.
Our main contribution is three folds:
1. We discover the implicit constraints on representations the Bellman equation and establish an upper bound on the cosine similarity.
2. We find theoretical principle to maintain the representation rank adaptively. We design novel, serreticalll-grounnee regularizer called BEER. controls the representation rank by adhering to the constraints imposed by the Bellman equation.
3. The empirical experiments validate that the BEER regularizer can help models maintain a balanced representation rank thereby further enhancing the DRL agents' performance. We scale up BEER to 12 challenging continuous control tasks, and BEER outperforms existing approaches by a large margin.
Representation Rank of Network
Aporoximation Error of Q Function
Perforance Curves
BEER
5 Rotum
DON
Approxiration
Approdiin
InFoR
InFaR
Time Steps (x40K)
Time Steps (x40K)
Time Steps (x40K)
(a) Lunar Lander
(b) Representation Rank
(c) Approximation Error
(d) Performance
Figure 1: Illustrative experiments on the Lunar Lander environment, with results averaged over ten random seeds. The shaded area represents half a standard deviation. (a) snapshot of the Lunar Lander environment. (b) Comparison of representation ranks. BEER exhibits more balanced rank compared to InFeR and DQN. (c) Approximation errors of different algorithms. BEER displays a lower approximation error compared to both DQN and InFeR in the latter stage (0.9 to 1 x40K time steps). (d) Performance curves substantiating the superiority of BEER.
2
PRELIMINARIES
This paper employs a six-tuple Markov Decision Process (MDP) (S, A, P.Y. Po) to formalize RL, where S represents state space, A denotes an action space, R S A + I reward function,
2

---

Published as a conference paper at ICLR 2024
P x - p(s) is transition kernel, y E [0, serves as a discount factor, and pu specifies an initial state distribution. The objective of RL is to optimize the policy through return, defined as Rg = Ei "(s;, a.). The action value (Q) function measures the quality of an action given state s for policy x, defined as
(") = Erwa, [R s, dg a),
(1)
where r is a state-action sequence (so, dg $1, 41, $2, 02 ...) generated by the policy æ and transition probability P. The Q value adheres the Bellman equation (Sutton Barto, 2018):
(7(s, a) r(s,a) + ,a ,')],
where s' P(-ss, a) and (-|ss). In scenarios where the state action space is prohibitively large, leveraging conventional tabular methods for value storage becomes computationally intractable. Function approximations such NNs are commonly employed for value approximation in these cases.
It is instructive to adopt matrix view for elaborating the MDP (Agarwal et al., 2019). Let Q E (((SS-IA)) denote all Q values, and as vectors of the same shape. We extend notation pT to represent a matrix dimension (IS) [A]) x (18)- [Al), where each entry is P (s.o).(s',a)) = (s a))((")s'), where pd is induced by a stationary policy т. It is straightforward to verify the matrix form of the Bellman equation (Agarwal et al., 2019)
Q th =
(3)
Representations of value functions. In the DRI. setting, we express the Q function Q" (s. a) o"(s (s, a a)w, where d(s, a) E Nx1 denotes a feature representation of dimension N and w E IR Nx1 weight vector. Note that is not related to the input pair (s, as.a). In this setting, ø(s, can be considered as the output of the penultimate layer of a neural network, and w corresponds to the weight of the network's final layer. To assess the expressiveness of the value function representation, adopt the concept of the "representation rank" as introduced by Lyle et al. (2022). The representation rank is formally defined follows.
Definition 1 (Numerical representation rank). Let Φ S x A RRN be representation mapping. Let X,, C S x A be set of n state-action pairs samples drawn from a fixed distribution The representation rank of the value function is defined as
Rank(á, Cnic, = <x for E SpecØ( Vn X,)) : |o| > c]|,
where Spee denotes the singular spectrum containing the singular values, produced by performing singular value decomposition on matrix ((X,).
We set to be a uniform distribution over S x A for the scope of this paper. In the context of finite state space X, and 0, the numerical representation rank hereafter referred to as the "representa- tion rank"- corresponds to the dimension of the subspace spanned by the representations. Selecting 0 filters out the small singular values. Therefore, one can determine the representation rank by calculating the singular spectrum of representation matrix, which is constructed by concatenating the sampled state- action pairs
METHOD
In pursuit of principled approach to regulating the representation rank, we deviate from the common practice of directly manipulating phenomena such as primacy bias (Nikishin et al., 2022) or dormant neurons (Sokar et al., 2023) within NNs. While such heuristics offer valuable insights, they seldom provide generalizable theoretical underpinnings. One of the challenges in deriving theoretical insights from NNs is their inherent flexibility, which often leaves little room for rigorous analysis. To circumvent this limitation, we incorporate the Bellman equation in the context of NNs for DRL. The intersection Bellman equations and NNs offers a fertile analysis ground, thereby facilitating the extraction of solid guarantees. We commence our discussion with an analytical examination of the Bellman equation and then derive an upper bound for the inner product of two consecutive representations of value networks. This upper bound is immediately transformable into new
3

---

Published as a conference paper at ICLR 2024
bound that constrains the cosine similarity of the representations. Notably, the cosine similarity implicitly restricts the representation rank. Motivated by the theoretical analysis, we introduce the BEER regularizers to adaptively control the representation rank. We consider value function approximation problem with constraints. Then the BEER regularizer serves as a penalty to solve the optimization problem. Contrary methods that unboundedly maximize the representation rank (Lyle et al., 2022; Kumar et al., 2021; He et al., 2023a), our strategy imposes well-calibrated constraint, which is instrumental in achieving a more nuanced and effective regularization.
3.1 UPPER BOUNDS ON SIMILARITY MEASURES FOR ADIACENT REPRESENTATIONS
We examine the Bellman equation through the interplay between Q values and their underlying representations within DRL. The view based on the Q value can be described as an inner product between the representation vector and corresponding weight vector. This allows us to rewrite the Bellman equation as
(5) Inspired by Equation (3), isolate the term involving the representation c one side of the equation. This manipulation yields an inter-tempont relationship between 4 and its successor, i.e.,
w r + + 7 w.
+ 7 $ wu
Taking L2 norm of both sides results in
I[ T p + wwll2 = ||x||2-
Given the intractability of finding an optimal policy DRL if either the representation function or the weight vector becomes trivial (e.g., zero vector), we introduce the following assumption. Assumption 1. Both the representation function @ and the weight vector w are non- con-trivial; specifically, is not constant function, and w is not zero vector.
With Assumption and by utilizing the definition of the operator norm (Debnath & Mikusinski, 2005), we arrive
|r||
$ T P
op
|
(8)
where || - llop symbolizes the operator norm. Equation (8) shows an implicit requirement of the representation & originating from the Bellman equation, which inspires us to re-consider the Bellman equation in Equation (2). We derive the following theorem constraining the inner product of two adjacent representations of value functions.
Theorem 1. Under Assumption 1, given Q value Q(s, a) a(s,a) Tw,
|||| || 27
1| (o(s,a), ((s) d') S (1o((s,) a) 7 a')l1
where () represents the inner product, & is state-action representation vector, is weight, and o(s', a²) Eg o(s' a') denotes the expectation of the representation of the next state action pair.
To keep the consistency, defer all proofs to the Appendix. Theorem I states the relationship of representations at adjacent time steps. We derive the theorem from the Bellman equation as a necessary condition for optimality associated with the optimal value function. Hence, the upper bound in Equation (9) is also an interesting result concerning the optimal value networks. We can directly obtain the cosine similarity between the current representation and the one the next step by dividing both sides of the Equation (9) by their norms.
Remark 1. Under Assumption 1, given Q value Q(s, a) = (0ss (ls,,)"= aTT,, then
||||
1
cos(Ø(8, 008' a')) < ([lo(s, a)l + 10(8 a²il
(10)
w 2 a) ll (lllo((s','))) a')ll
Remark 1 discusses the cosine similarity between two adjacent representations of the value function. The representation rank, as defined in Definition 1, is determined by computing the singular spectrum, which quantifies the number of filtered singular values. Under these conditions, the cosine similarity
4

---

Published as a conference paper at ICLR 2024
is intrinsically linked to the representation rank. For example, if all vectors in the representation matrix are orthogonal cosine similarity is zero, then they are linearly independent. That means that the matrix is full rank. If some vectors of the representation matrix are linearly dependent, which means the cosine similarity taking 1, then they will contribute to a lower rank. Explicitly controlling cosine similarity allows for adaptive adjustment of the representation rank, methodology also echoed in prior research (Kumar et al., 2021; Lyle et al., 2022).
3.2 ADAPTIVE REGULARIZATION OF REPRESENTATION RANK
Previous work (e.g., InFeR, DR3) generally unboundedly maximizes the representation rank by regularizing the representation explicitly. However, this strategy neglects subtle yet critical trade-off' inherent in DRL. A large representation rank can lead to an excessively complex representation function that can hinder the learning process (Goodfellow al., 2016). Such complex models may not only demand larger dataset for successful training but also exhibit high sensitivity to noise, impeding sample efficiency and overall performance.
Motivated by the relationship between cosine similarity and representation rank elucidated in Sec- tion 3.1, we propose an adaptive regularization technique. Our method uses the upper bounds derived in the preceding subsection to constrain the representation rank, conforming to the limitations stipulated by Equations (9) and (10). To this end, we introduce a hard constraint within the value function's 's optimization process. Formally,
min A = 2 [ w -SSG''' SGOTTw) w)²
|r| |w| 271lo(s, ())l, a'il
s.t. cos(o(s, a). SC ( a')) S (1lo(s, a)ll ('s"'''' +
where L denotes standard value approximation loss and SG is the stopping gradient, reflecting the semi-gradient nature of the Bellman backup (Sutton & Barto, 2018). One can interpret that equation as adding an explicit desirable hard condition to the learning process of the value network. The introduced constraint effectively ensures that the representations satisfy conditions derived from the original Bellman equation, thereby aiding the value network's learning process. Applying the Lagrange multiplier technique to solve this optimization problem is challenging because the original problem is not convex. Hence, we cannot solve the problem using the primal-dual method like Haarnoja et al. (2018) does. Consequently, utilize penalty regularizer. We introduce the following regularizer into the value network learning process:
R(0)= = ReLU cos(((ss SGo(s'
|r| 1
(12)
(G]oosss a))l "00('''
+
a')))
27110(8, a) a)llll(''''""))
where ReLU(z)) (0,x) Incorporating this regularizing term offers generalized applicability across various DRL algorithms that utilize value approximation. Formally, the loss function can be rewritten
(())= = (0) + BR(0).
(13)
where 3 serves as a procedure. The working mechanism of the regularizer as follows. When the input the ReLU function is negative, it indicates the condition in Equation (9) is fullfilled. Under such circumstances, the regularizer ceases to influence the optimization trajectory, thereby allowing the original value approximation loss, Lv to exclusively govern the optimization process. Conversely, when the ReLU input is positive, the regularizer contributes to optimizing the value function and minimizes the cosine similarity to ensure compliance with Equation (9). To empirically validate the utility of the proposed BEER regularizer, we integrate it with DQN (Mnih et al., 2015) for problems with discrete action spaces and with the Deterministic Policy Gradient (DPG) algorithm (Silver et al., 2014) for those involving continuous action spaces. We summarize BEER based on DPG Algorithm 1.
controlling the regularization effectiveness on the learning
3.3 AN ILLUSTRATIVE EXAMPLE
5

---

Published as a conference paper at ICLR 2024
Does very large representation rank harm the learning process of value function? To validate the adaptive regularization of the representation rank of BEER, we perform illustrative experi- ments on the Lunar Lander task. In this task, a spacecraft shall safely land on a designated landing pad. A policy with a return greater than 200 is considered optimal. We evaluate three al- gorithms: DQN, InFeR, and relaxed version of BEER, which we integrate into the DQN frame- work without additional modifications. Compre-
Algorithm 1 BEER (based on DPG)
1: Init actor x, critic Q, targets n', Q' with ran- dom parameters 0, sp, 0', 88','' replay buffer R, noise distribution (0, or), regularization co- efficient B, total timestep T
2: Init environment, get state
3: for t= T, do
4: a4 (Ts) C
5: Execute ar get Tt, 8t-1
6: Store (8t, (ty I'y, 814.1 in R
hensive details of the experimental setup appear
7: Sample batch from R
in Appendix C. The primary goal of a value net- work is precise approximation of the optimal
8: Compute target value:
9:
Yi =y + yQ (5 Tor (8:+1))
value function. Hence, we choose approxima- tion error metric defined the absolute difference between the estimated and the true value functions- as the principal criterion for model evaluation. This measure assesses the models' quality directly and precisely.
10:
Update critie parameters & by minimizing
(())
L.( Qp(a,, a,)) + (R(())
(Equation (13))
11:
Update policy parameters O using DPG
12:
VoJ Rs N Σ VaQu a,) (s)
13:
Update target network 8' 8''6'
14:
0' 4- 7 + -7)0', p' <- TG+ ()
Figure shows the results. Initially, we quantity the representation rank for each algorithm as appears in Figure 1(b). We observe that BEER's representation rank is lower than those of both InFeR and DQN, thereby suggesting that the model learned with BEER less complex. Then, we compute the approximation error. During the latter stage of the training process (0.9 to 1 x 400K time steps), BEER displays a substantially lower approximation error compared to both DQN and InFeR; that is, BEER generates more accurate and reliable value function model. Figure ((d) further confirms the superior performance of BEER. In brief, the experimental results verify that BEER improve the complexity of the model as originally envisioned by adaptively controlling the representation rank.
15:
end for
4 EXPERIMENTS
Representation Rank of Network
Approximation Error of Q Function
Number of Steps to Reach Sy
Approsimation Errrr
BEER
BEER
ST
InFiR
InFoR
DQN
DON
Rank
BEER
InFoR
Time Stops (X25X)
Time Stops (X25K)
Time Stops (x258)
(a) Grid World
(b) Representation Rank
(c) Approximation Error
(d) Performance
Figure 2: Illuttrative experiments on the Grid World task. We report the results over twenty random seeds. The shaded area represents a half standard deviation. (a) The grid world task. The initial state follows uniform distribution over state space. The objective is to arrive at state Sy, which results in reward (=10); Otherwise, it is zero. (b) Representation rank of tested algorithms. Our proposal, BEER, has the highest representation rank compared InFeR and DQN. (c) Approximation error. The error of BEER is lower than that of InFeR and DQN. (d) The BEER algorithm requires the least time to reach Sr, i.c., it Iearns faster than the benchmarks.
In this section, we evaluate BEER concerning its regularization of representation rank and its empirical performance. Our evaluation focuses on three primary research questions. BEER algorithm tackles overly complex models with a potential high representation rank. Thus, one is interested in i) How does BEER perform in relatively straightforward tasks, such the grid world environment? ii) Can BEER scale effectively to over- wwrappatmtttiizeed high-complexity tasks? By over over- weer-paramttrizeed tasks, specifically refer agents on DMControl tasks (Tunyasuvunakool et al., 2020), where the

---

Published as a conference paper at ICLR 2024
agents generally wer-parameteeizee (Lutter et al., 2021; Kumar et al., 2021). iii) How does the approximation error of BEER compare to that of existing methods? A lower approximation error indicates superior model in terms of complexity and performance. To ensure the reproducibility and fairness of our experiments (Henderson et al., 201 8), all tests are based on ten random seeds. Besides, BEER implementation does not incorporate any engineering trick that could potentially boost performance. In all experiments, we refrain from fine-tuning the hyperparameter P.. We consistently use a value of le We elaborate the experimental configurations in Appendix C to save space, The influence of the regularization coefficient on BEER's 's performance, as well as the experimental results of BEER on relatively simpler tasks, are presented Appendices and G, respectively. The impact of cosine similarity rank representation is discussed in Appendix D. Whether the effects of the BEER regularizer are implicitly present the value approximation algorithm deliberated upon in Appendix E.
4.1 EFFECTIVENESS OF BEER ON SIMPLE ENVIRONMENTS
Here, we study the effectivvnees of BEER in simple environment with the grid world task as shown in Figure 2. As illustrated Figure 2(b), BEER achieves a higher representation rank than the cxisting methods such InFeR and DQN. Additionally, shown in Figure 2(c), the approximation error of BEER is observably lower than other algorithms. Consequently, BEER also outperforms these methods concerning the number of steps required reach the terminal state ST (see Figure 2(d)). Combined with the illustrative experiment in Section 3.3, these results demonstrate the adaptive regularization capability of BEER both on simple and complex tasks, which reflects our primary objective to design BEER.
acrobot-swingup
hopper-hop
hopper-stand
humanoid -tannd
400
Average Return
800
400
150
BEER
300
DR3
300
100
InFeR
200
SAC
200
100
humanoid-run
humanoid walk
pendulum-swingup
Swimmer- swimmer
100
Average Retur
80
200
200
20
200
100
Swimmer-swimmrr15
finger-spin
fish-swim
cartpole-swingu sparse
400
Average Retum
800
400
400
600
400
200
200
100
100
0.00 0.25
0.50
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
3.00
Time Steps (x 1e6)
Time Steps (x 1e6)
Time Steps (x 1e6)
Time Steps (x 1e6)
Figure 3: Performance curves for OpenAl gym continuous control tasks on DeepMind Control suite. The proposed algorithm, BEER, outperforms other tested algorithms significantly. The shaded region represents half of the standard deviation of the average evaluation over 10 seeds. The curves are smoothed with moving average window of size ten.
4.2 SCALING UP TO COMPLEX CONTROL TASKS
DMControl (Tunyasuvunakool et al., 2020) serves as standard benchmark suite for evaluating the capabilities of DRI. agents in complex, continuous control tasks (Laskin et al., 2020b;a). We further
7

---

Published as a conference paper at ICLR 2024
extend BEER's applicability to continuous control tasks and demonstrate the performance of the BEER regularizer on the 12 challenging (Yarats & Kostrikov, 2020) DMControl tasks, where we combine BEER with the deterministic policy gradient method (Silver et al., 2014; Lillicrap et al., 2016). Our primary objective this study to investigate the BEER's s efficacy concerning automatic regularization of representation rank. We select the following baseline algorithm: DR3 (Kumar et al., 2021), which unboundedly maximizes representation rank by maximizing the inner product of two adjacent representations of the Q network; ii) InFeR, which consistently maximizes the rank with the help of designed auxiliary task; iii) TD3 (Fujimoto et al., 2018) and iv) SAC, both of which have already shown robust performance a spectrum of tasks. The performance curve, illustrated in Figure 3, demonstrates the sample efficiency and superior performance of BEER. We report the best average scores over the million time steps in Table I, where the best performance of BEER outperforms the other tested algorithm by large margin. These results demonstrate the sample efficiency and performance of BEER.
Table 1: 10-Seed peak scores over IM Steps on DMC. BEER demonstrates the best performance on the majority (12 out of 12) tasks by a large margin. Specifically, the BEER algorithm outperforms DR3, InFeR, SAC, and TD3 by 140%, 96.8%, 60%, and 101.9%, respectively. The best score is marked with the blue color box. + corresponds standard deviation over trials.
Domain
Task
BEER
DR3
InFeR
SAC
TD3
Acrobot
Swingup
260.5 42,9
46.2 14.0
18.9 = 13.6
43.2 63.5
32.3 +: 27.4
Hopper
Hop
383.6 115.8
22.0 50.0
58.1 37.6
149.0 73.6
91.2 + 46.9
Hopper
Stand
929.3 29.7
465.6 + 276.3
563.4 256.5
650.5 = 238.5
443.0 208.6
Humanoid
Stand
471.3 92.5
7.4 = 1.0
50.8 87.9
418.1 278.7
58.0 + 151.6
Humanoid
107.1
1.1
12.4 32.9
43.4
15.1 +:
Humanoid
Walk
393.2 38.2
55.8 107.2
39.0 108.9
329.0 202.0
33.5 + 92.0
Pendulum
Swingup
833.2
331.9 228.8
456.4
270.7
Swimmer
Swimmer6
398.0 123.8
321.1 + 100.4
332.8 125.3
243.8 74.4
321.9 + 148.4
Swimmer
Swimmerl5
345.5 110.2
320.6 + 165.6
283.8
314.1 198.5
226.7
Finger
Spin
983.6 6.8
715.2 +: 387.5
966.0 + 21.8
956.5 43.0
957.9 +: 26.9
Fish
Swim
573.2 103.4
377.5 123.4
335.7
418.2 127.1
316.0
Cartpole
Spinge
750.8 61.8
15.0 39.1
148.8 + 235.2
147.6 = 295.5
235.0 + 356.9
Average
Score
535.8
223.3
272.2
335.0
265.4
acrobot-swingup
hopper-hop
pendulum-swing swingup
finger-spin
Approximation Error
5
BEER
DR3
3
SAC
2
TD3
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Time Steps (x 1e6)
Time Steps (x 1e6)
Time Steps (x 1e6)
Time Steps (x 1e6)
Figure 4: Approximation error curves. The results demonstrate that the approximation enror of BEER empirically minimal when compared to other algorithms.
4.3 BEER REDUCES APPROXIMATION ERROR.
The results discussed above establish the superiority of BEER that stems from the adaptive regular- ization of representation rank. We continue to validate the quality of the learned models in terms of approximation eтror. We select four representative tasks to measure the models, where BEER has better performance three and matching performance one of the tasks (finger-spin). As demonstrated Figure 4, BEER maintains lower approximation error than the baselines. The results consistent with previous theoretical analysis, thus validating its utility.
8

---

Published as a conference paper at ICLR 2024
5 RELATED WORK
In this section, we dive into the intersection of RL with NNs and the growing emphasis on NNs-centric DRL. The differences between our work and previous studies are also discussed this section.
5.1 RL WITH NNS
Prior to the era of DL., function approximators in RL are either tabular (Sutton & Barto, 2018) or utilized learnable weights based on handcrafted features. (Barreto et al., 2017). DQN (Mnih et al., 2015), which stimulates the tremendous development of DRL, leverages NNs to approximate Q functions. The subsequent work (Schulman et al., 2017; Lillicrap et al., 2016; Fujimoto et al., 2018; Haarnoja et al., 2018; He & Hou, 2020), which focuses the RL side, utilizes NNs general function approximators. As representation learning techniques (Devlin et al., 2019; Oord et al., 2018; He et al., 2020; Chen al., 2020a; He et al., 2022) continue evolve, an increasing number of these methods are being incorporated into the field of RL (Jaderberg et al., 2016; Guo et al., 2020; Ghosh & Bellemare, 2020; Laskin et al., 2020b; Yarats et al., 2021a;b; Laskin et al., 2020a; Oord et al., 2018), because the learned representations by NNs with these methods are beneficial for RL agents and also the downstream tasks. Our work is distinct because it considers the properties that RL should possess within the context of DL settings. Unlike some approaches that predominantly draw upon intuitions from the field of DI., our work aims for an integrated understanding of both domains.
5.2 NNS-CENTRIC DRL
Some studies focus on leveraging NNs to enhance the capabilities of DRL. Examples include approaches that employ NNs to learn multiple Q-values (Anschel al., 2017: Lan et al., 2019; Chen et al., 2020b), simultaneously learn state value and advantage value (Anschel et al., 2017), and deep successor features (Barreto et al., 2017). Researchers have gradually recognized that the integration of NNs with RL enables specific characteristics, such the distinguishable representation property (He et al., 2023a) and the domination of top-subspaces in the learning process (Lyle et al., 2021; He et al., 2023b). However, this integration also introduces particular challenges that hinder DRI, such the catastrophic forgetting (Khetarpal et al., 2022), the capacity loss (Lyle et al., 2022), the plasticity loss (Lyle al., 2023), dormant neurons (Sokar et al., 2023), primacy bias (Nikishin al., 2022), and etc. These studies aim to identify and rectify NNs' attributes that are detrimental to RL. For instance, (Lyle et al., 2022: 2023) assert that NNs become less adaptive new information as learning progresses, thereby experiencing a capacity/plasticity loss that can be alleviated by increasing the neural network's representation rank. Recent literature discusses how to maximize representation rank in bandit settings and RL settings (Papini et al., 2021b;a; Zhang et al., 2023), which cannot inherently bound the cosine similarity in this work.
In contrast, our work diverges from these studies by originating from the inherent properties of RL rather than focusing solely on the NNs component. We establish a connection with representation rank and introduce a novel method for adaptively controlling it, providing fundamentally different starting point compared to previous research. In terms of the form of the regularizer, both previous works Kumar al. (2021): He et al. (2023a) and ours involve the inner product form of the rep- resentation. Nevertheless, both Kumar et al. (2021) 1) and He et al. (2023a) utilize an unboundedly maximizing form. In contrast, our work starts with the Bellman equation and introduces adaptive regularizer that will not unboundedly maximize the representation rank.
6 CONCLUSION
In this study, we discussed a significant issue of representation rank in DRL, which the current literature neglects a great extent: How to adaptively control the representation rank of DRL agents. By rigorously analyzing an inherent property of the Bellman equation in the DRL setting, we introduced the theoretically grounded regularizer BEER, a novel approach to adaptively control the representation rank. Our theoretical and empirical analyses demonstrated the effectiveness of BEER, outperforming the existing methods that focus on the representation rank. In our future works, we plan to validate the efficacy of BEER other benchmarks. Our work opens new avenues for understanding the role of representation rank in DRL and offers an adaptively practical tool to control the rank and thus improve agent performance.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The authors thank anonymous reviewers, the area chair, and the senior area chair for fair evaluations and professional work. This research was supported by Grant 011S20051 and Grant 16KISK035 from the German Federal Ministry of Education and Research (BMBF).
REFERENCES
Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle. WA, USA, Tech. Rep, pp. 10-4, 2019.
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In International conference on machine learning, pp. 176-185. PMLR, 2017.
André Barreto, Will Dabney, Rémi Munos, Jonathan Hunt, Tom Schaul, David Silver, and Hado van Hasselt. Successor features for transfer in reinforcement learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4055 4065, 2017. URL https: http:/::///cccceeeeeeeoeoo mewrdddiiiiiinnnnnnooeee..e..nrr cc/ httpppppppppp//ae/e//ppp 2ttp:/2200/hhhhh. //hah/ 3550b00l.661112223344aaaaaaaa333 -bsttbbbaabbbbbaccttt.aa html.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020a.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W Ross. Randomized ensembled double q-learning: Learning fast without model. In International Conference on Learning Representations, 2020b.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus. S. V. N. Vishwanathan, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4299-4307, 2017. URL https: / http:///ooooeeeeeeeeeeeee/c/s.ee www.ingg...nurrrrr neurips. wtt.nnis....pppprrr/aa..... #bs20aaadd03ee11f91444oooooodddd
wwwpppppe///11////ee.oo //ahhhh Abstrratt html.
Lokenath Debnath and Piotr Mikusinski. Introduction to Hilbert spaces with applications. Academic press, 2005.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- doi: 10. 10.18653//1/111-14423 URL
4186. Association for Computational Linguistics, 2019.
https: htttps://do..org/// wt1/10orrg11...1 1tt....1153///1n1111111
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmsmässan. Stockholm, Sweden, July 10-15, 15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591. PMLR, 2018.
Dibya Ghosh and Marc G Bellemare. Representations for stable off-policy reinforcement learning. In International Conference on Machine Learning, pp. 3565. PMLR, 2020.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
10

---

Published as a conference paper at ICLR 2024
Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, Jean- Bean-Bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap latent- intent-predictivy representations for multitask reinforcement learning. In International Conference on Machine Learning, pp. 3p.5575-3886 PMLR, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässsnn Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1856-1865. PMLR, 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022.
Qiang He and Xinwen Hou. Wd3: Taming the estimation bias in deep reinforcement learning. In 2020 IEEE 32nd International Conference Tools with Artificial Intelligence (ICTAI), pp. 391-398. IEEE, 2020.
Qiang He, Huangyuan Su, Jieyu Zhang, and Xinwen Hou. Frustratingly easy regularization on representation can boost deep reinforcement learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 20215-20225, 2023a.
Qiang He, Tianyi Zhou, Meng Fang. and Setareh Maghsudi. Eigensubspace of namics and how it improves value approximation in reinforcement learning. CoRR, abs/2306. 16750, 2023b. doi: 10.46550/arxxi.2366667553 16750. 2306. 16700.
dy-
URL pttps:://di... 10t5//....../////...........
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intelli- gence, 2018.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint /r1iv: 1.05397, 2016.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: review and perspectives. Journal of Artificial Intelligence Research. 75:1401-1476. 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. In International Conference on Learning Representations, 2021.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learming: Controlling the estimation bias of q-learning. In International Conference on Learning Representations, 2019.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforeement learning with augmented data. In Hugo Larochelle, Marc' Aurelio Ranzato. Raia Balcan, and Hsuan- Sesuan-Tien Lin (eds.), Advances in Neural Information Pro- Hadsell, 33: Annual Conference Newral Information Processing Systems 2020, NeurlPS cessing Systems
2020, December 6-1 2020, virtual, 2020a. URL / www.rroceediingnn..eoe wig.eeiiiin....... neur. www.inurr........... ips.ppp..oo cc/ paper/2020 hash/ahhhheeeeaa htte//665522aaae 61 6 61166811ee2222 wdd.2daddaa8800eee.a Abst Abstaat0taaaa-aas....aoa.. html.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pp. 5639-5650. PMLR, 2020b.
11

---

Published as a conference paper at ICLR 2024
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforeement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http::/arxtit org/ abs 15059.940 02971.
Michael Lutter, Leonard Hasenclever, Arunkumar Byravan, Gabriel Dulac-Arolld Piotr Trochim, Nicolas Heess, Josh Merel, and Yuval Tassa. Learning dynamics models for model predictive agents. arXiv preprint 1401v.2109.14311, 2021.
Clare Lyle, Mark Rowland, Georg Ostrovski, and Will Dabney, On the effect of auxiliary tasks on representation dynamics. International Conference on Artificial Intelligence and Statistics, pp. 1-9. PMLR, 2021.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Ávila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, Pp. 13190-23211 PMLR, 2023. URL https: //proceedings ml press v20v2222211222222
e23b. html.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 5 518(7540):529-53,, 2015.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Lue Bacon, and Aaron C. Courville. The primacy bias in deep reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 16828-16847 PMLR, 2022. URL https: html.
//proceedings... mlr wwwpppressseeeeee http///////iiiihhiee
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03744, 2018.
OpenAl. GPT-4 technical report. CoRR, abs/23033087711111 2023. doi: 10.4855WaaXxx.22333.87774 URL httphtttt///oi.org//0000 wtt.:oog/........... wtt.a/wwww..........
Matteo Papini, Andrea Tirinzoni, Aldo Pacchiano, Marcello Restelli, Alessandro Lazaric, and Matteo Pirotta. Reinforcement learning in linear mdps: Constant regret and representation selection. Advances in Neural Information Processing Systems, 34: 14:163311-1633 2021a.
Matteo Papini, Andrea Tirinzoni, Marcello Restelli, Alessandro Lazaric, and Matteo Pirotta. Lever- aging good representations linear contextual bandits. International Conference on Machine Learning, pp. 8371-83888 PMLR, 2021b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. arXiv preprint
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. 2014.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1 1140-1144 2018.
12

---

Published as a conference paper at ICLR 2024
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon in deep reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Confer- ence on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 32145-32166 PMLR, 2023. URL https: https//pppcceeiingg.e
wal...peess/vee https///v2//ssss//ee sok0r2/ atmm.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm-control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)350-354 2019.
Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation pytorch, 2020.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. URL https: / ///opnrreveew.. n wwwp:/nntt/otteieeeeeeeee FoormmiddcGe 6- Tva..
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021b. URL httph:t///// /tttpp:://pppnnee hpwp//epeeeeeeeeeeeeeeeeeeeeeeee hwtp:/nnttooort//eeeeeee
6
Weitong Zhang, Jiafan He, Dongruo Zhou, Q Gu, and Zhang. Provably efficient representation selection in low-rank markov decision processes: from online to offline rl. In Uncertainty in Artificial Intelligence, pp. 2497. PMLR, 2023.
13

---

Published as a conference paper at ICLR 2024
A PROOF OF THEOREM
Theorem 1. Under Assumption 1, given value Q(s, = á(s,a) Tu, w,
|||| | 2
( (1o((s ( 1lo'''
(9)
where (..:) represents the inner product. & is state-action representation vector, is weight, and
o(s', a') or a') denotes the expectation of the representation of the next state action pair.
Proof. We start with the Bellman equation for Q values, i.e.,
((s r(s, a) + at) (s(()) a')].
(14)
Substituting the representation Q(s, a) = o(ss a) T w into the equation, we get
( a) T = r 70(8', a') w).
(15)
Taking the norm of both sides gives us
o(s,a) w a0(s',a)) [(({(s,)) a0) )wl |||||
w = r,
(16)
(17)
a)) 70(8'aa) ||] 2 ||rl]
(18)
||]
al)s, vo(s',
(19)
||w||
where the third line follows by the Cauchy-Schwarz inequality. Taking the square of both sides, we
obtain
|||
(1)(s,a) (')''aa')
(20)
|w
Expanding the square of the norm, we find
l10(, a) a')'1a''i11
(21)
=(6(s, a) - -0((s (0) -
(22)
|||
=I|o(s, a)l2 27 -( + 7² "110('' >
(23)
|
Finally, rearranging the terms yields
|||||
S (10((, + + a0i12
(24)
|| 2 27
which concludes the proof.
B SIMILARITY UPPER BOUND UNDER THE BELLMAN OPTIMALITY EQUATION
We discuss the similarity upper bound under the Bellman optimal equation. That induces another BEER regularizer, which is slightly different from Equation (12). We first introduce an upper bound on the representations of two consecutive representations based on the Bellman optimality Equation. We then derive the related bound the cosine similarity. Afterward, we introduce BEER regularizer based on the bound combined with the DQN algorithm. Such a regularizer is used Figures and 2.
14

---

Published as a conference paper at ICLR 2024
Theorem 2. Under Assumption 1, given value Q(s, a) 0(s, a) Tw. w.
( (ls (0''s((')))) a')) & (lo(s, a)ll + 1= a')l |||1 | 2
(25)
where (:,-) represents the inner product, & is a state-action representation vector, w is weight. And o(s', a') = ==)=p(//) a') with a' arg m (0(ss'a'),
The proof is similar to Appendix A.
Proof. We begin with the Bellman optimality equation for Q values, i.c. Q a) = r(8, a) + 7B [max Q a')]
(26)
Y'EA
Substituting the representation Q(s, a) o(s, a) T w into the equation, we get
((s.a) T w r(s,a) + 7o(s',a') w
(27)
Taking the norm of both sides yields
6(s,0) w a0) = [((((s T)T - a')
(28)
T
)wl
|||||
(29)
| a)T 70(8', a²)
||| |||l
(30)
|||
(1)(, - - a²)
(31)
|w]
where the third line follows from the Cauchy-Schward inequality. Squaring both sides, we find
|||||2
( a) ro(s',a) d²) 112
(32)
||
Expanding the square of the norm, we find
lIo(s a) 70(s',a')11"
(33)
=(((8,0) - ((, - 70(8', (il'',a') d'))
(34)
|||
= =Ilø(s,a)] 27(0(s, a), 088 al)) + 22100(' a2)1 >
(35)
|
Finally, we obtain the inequality stated in equation 25:
|||||
(o(s, a), ((s'sa'))) a()) < (116(s, 2 0')1
(36)
|| 27
which completes the proof.
The following remark arises from Theorem 2.
Remark 2. Under Assumption 1, given Q value Q(s, a) = 0(s, a)
w. then
||||2
cos(o(s, 0(s' d')) < (I
+ 70 ('))1' | 2r a) (Iloss))a)))
(37)
According to Theorem 2 and Remark 2, obtain the following regularizer:
R(0) = ReLU cos(o(s, a). (Go(s',
|r|
(38)
(Ilo(s, a)ll + 7 110(s', a')ll |
27110(s, a)ll]((s'',
which is used in Figures and 2.
15

---

Published as a conference paper at ICLR 2024
C ADDITIONAL DETAILS REGARDING EXPERIMENTS
In this section, intend to furnish additional insights and specifications concerning the experimental setups employed to derive the results presented in the main manuscript. This systematic presentation of our experimental procedures aims to promote transparency, facilitate replication, and enable future research to build upon these foundational insights.
Algorithm Design. Our goal is analyze the properties of the BEER regularizer. As previously claimed, it can be integrated with DRL algorithms that involve value approximation. In addition to combining it with DQN Figures 1 and 2), we also integrate the BEER algorithm with DPG (Silver et al., 2014), as listed in Algorithm 1. The purpose of this integration is to focus on studying the intrin- sic properties of BEER itself, thereby demonstrating the superiority of the BEER regularizer. DPG propagates gradients through the value network when optimizing the policy network. A high-quality value network leads to a high-quality policy network. Our BEER regularizer is primarily designed to achieve better value network by adaptively controlling the representation rank. Therefore, we chose to combine the BEER regularizer with DPG. In our implementation of BEER based on DPG, exclusively use pure DPG. The exploration scheme of BEER+DPG involves adding Gaussian noise to the policy network, as done in the TD3 paper (Fujimoto et al., 2018). It's worth noting that, for more accurate measurement of the BEER algorithm, we have not included any additional performance-enhancinn techniques such as double Q clipping or target noise smoothing for the BEER algorithm based on DPG. Comparing BEER based on DPG with TD3 serves natural ablation experiment. This is because BEER is combined only with DPG without any other tricks, while TD3 has already been widely proven to be one of the best DPG algorithms across a broad range of scenarios (Yarats et al., 2021b; Fujimoto et al., 2018; Haarnoja et al., 2018).
Implementation Consistency. our quest to maintain scientific rigor, we have imposed deterministic controls by fixing all random seeds, encompassing libraries such as PyTorch, Numpy, Gym, Random, and CUDA. Unless otherwise noted, all algorithms under study are assessed across 10 predetermined random seeds. Our code is available at
Computational resources. We conducted all experiments on a single GPU server. This server is equipped with 8 GeForce 2080 Ti GPUs and has 70 CPU logical cores, allowing for the parallel exccution of 70 trials. Computational tasks for 70 runs can be completed within eight hours.
Figure 1. We conducted the experiments concerning this figure on the Lunar Lander-1 I environment. We calculated the ground truth Q-values through Monte Carlo methods Fujimoto et al. (2018). The error bars in the figures represent half of the standard deviation across multiple trials. Unless specified otherwise, such error bars in all subsequent figures denote half of a standard deviation across 10 unique seeds. The implementation of the three algorithms is nearly identical, except for the portions unique to each algorithm. For specific hyperparameters, please refer to Table 2.
Figure 2. The grid world is shown in Figure 2(a). If the agent arrives at Sr, it gets a reward of 10, and other states get a reward of O. We present the remaining Table 3.
for the grid world in
Figure 3. Note that we select 12 challenging tasks from dmcontrol (Tunyasuvunakool et al., 2020). Our selection is based on the data reported by Yarats & Kostrikov (2020), where we opt for the twelve tasks with the highest level of challenge to present in the main text. Results for simpler tasks are also shown in Table 6. For Figure 3, we examine the algorithmic performance on continuous control tasks in the DeepMind Control Suite. The interaction protocol employs the gym Python library (Brockman et al., 2016). The shaded regions depict half a standard deviation around the mean evaluation score from 10 different seeds. Additionally, the performance curves are smoothened using a moving average window of 10. Evaluations were conducted over million timesteps, with average returns recorded at intervals of 10,000 timesteps over 10 episodes. We present the remaining Syper-parameterr for the DMControl in Table 4.
The approximation error is normalized relative to the true Q value; i.c., approximation error orignal approximation error where le 6 is a small number to stabilize the Q++
Figure 4.
computational process. The purpose of normalization is to mitigate the impact of the value scale on the approximation eтror, which arises due to performance disparities among various algorithms so that we can compare the approximation error of different algorithms. The performance of the
16

---

Published as a conference paper at ICLR 2024
algorithm under test is evaluated across 1 million timesteps. We assess the approximation error intervals of 10,000 timesteps. For clarity in visualization, the y-axis is limited to the range [O, 5].
Table 1. First, calculate the average return obtained by the algorithm across ten episodes within one million steps. We then select the best average value from this set. Next, we choose ten different random seeds and calculate similar values for each seed. The final data obtained by averaging these ten values.
Table 2: Hyper-parameters settings for lunar lander-vl task (Figure 1). For InFeR, we use its default hyper-parameters. For every 5e3 step, we evaluate the algorithms.
Value
Shared hyper-parameter,
Dimension of state space
8 Discrete(4): up, down, left, right 0.99 10a
Action space
Discount (?)
Replay buffer size
Optimizer
Adam (Kingma & Ba, 2015)
Learning rate for Q-network
1 1e
Number of hidden layers for all networks Number of hidden units per layer
3
64
Activation function
ReLU
Mini-batch size
64
Random starting exploration time steps
10a
Target smoothing coefficient (7))
0.005
Gradient clipping
False
Exploration method
Gesilyn-Greedy
(Exploration)
0.1
Evaluation episode
10
Number steps
4c4
for rank computing
0.01
BEER
BEER coefficient (33)
5e-3
InFeR
Number of head (k)
10
B
100
a
0.1
17

---

Published as a conference paper at ICLR 2024
Table 3: Hyper-parameters settings for Grid World experiments. For every 5e3 step, evaluate the algorithms.
Hlyper-paramee
Value
Shared hyper-purameters
State space
integer: from t 19 Discrete(4); down, left, right 0.99
Action space
Discount (7)
Replay buffer size Optimizer
10a
Adam (Kingma & Ba, 2015)
Learning rate for Q-network
1 10
Number of hidden layers for all networks
2
Number of hidden units per layer
32
Activation function
ReLU
Mini-batch size
64
Random starting exploration time steps
103
Target smoothing coefficient (7))
0.005
Gradient clipping
False
Exploration Method
Epsilon-Greedy
(Exploration)
0.1
Evaluation episode
10
Number of timesteps
2.5e4
for rank computing
0.01
BEER
BEER coefficient (3)
5e
18

---

Published as a conference paper at ICLR 2024
Table 4: Hyper-parameterr settings for DMcontrol experiments (Figures 3 and 4 and Table 1).
Hyper-parameter
Value
Shared hyper-parametters
Discount (n)
0.99
Replay buffer size
10
Optimizer
Adam (Kingma & Ba, 2015)
Learning rate for actor
10 10-4
Learning rate for critic
3 10-4
Number of hidden layers for all networks
2
Number of hidden units per layer Activation function
256
ReLU
Mini-batch size
256
Random starting exploration time steps
2.5 104
Target smoothing coefficient (1)
0.005
Gradient clipping
False 0.01
e for rank computing
TD3
Variance of exploration noise
0.2
Variance of target policy smoothing
0.2
Noise clip range
O.5,, 0.5]
Delayed policy update frequency
2
Target update interval (d)
2
SAC
Target entropy
dim of
Learning rate for
1 10 2
Target update interval (d)
InFeR
Number of head (k)
10
100
O
0.1
DR3
Regularization coefficient (cp)
5e-3
BEER
BEER coefficient (8)
1 10 10-3 0.2
Variance of exploration noise
19

---

Published as a conference paper at ICLR 2024
Metrics Over Time
Mean Cosine Similarity
Representation Rank
Mean Cosine Similarity Representation Rank
0.0
2.5
5.0
12.5 Time Step x (100)
7.5
10.0
15.0 17.5
Figure 5: Relationship between cosine similarity and representation rank when minimizing cosine similarity. Our experiment demonstrates the effective control of cosine similarity for representation rank.
D EMPIRICAL VALIDATION OF COSINE SIMILARITY'S CONTROL ON REPRESENTATION RANK
In Section 3.1, we discussed how cosine similarity can control representation rank. Here, we empirically validate the control effect of cosine similarity representation rank to gain intuitive understanding. The results of the experiment are shown in Figure 5. In this experiment, we first generate matrix with very high cosine similarity, resulting in very low representation rank. We randomly sample a mini-batch of row vectors from this matrix and minimize the cosine similarity between these vectors. We then track the relationship between cosine similarity and the representation rank we' 've defined, as displayed in Figure 5. For better visual clarity, we use a logarithmic scale on the y-axis better observe the relative change trends between these two metrics. We find that a slight reduction in cosine similarity immediately leads to an increase in representation rank, demonstrating the effective control of cosine similarity for the representation rank.
Specifically, we first generate 256 x 256 dimensional matrix and use the Adam optimizer with learning rate of 5e-3. We sample row vectors from the matrix each time. Every 100 training steps, we record the cosine similarity among the vectors in the matrix well as the representation rank, using an epsilon value 0.05 for calculating the latter. The experiment is conducted over 2,000 steps. In the later stages of the experiment, the cosine similarity becomes too small to be within the display range. At this point, the representation rank almost full rank.
20

---

Published as a conference paper at ICLR 2024
acrobot-swingup
hopper-hop
pendulum-swingup
finger-spin
0.0
1,0
-25
-00
0.5
-5.0
BEER
DR3
-75
-100
-0.5 InFeR
10.0
125
TD3
-12.5
-175
150
1.00
0.50
Time Steps (x 1e6)
Time Steps (x 1e6)
Time Steps (x 1e6)
Time Steps (x 1e6)
Figure 6: The results demonstrate that the tested algorithms don't 't hold the bound across all the tasks. However, BEER makes the agent hold the bound. Combined with the performance results (see Figure 3), these demonstrate that BEER effectively ensures the satisfaction of this bound, thereby adaptively regularizing the rank and ultimately improving performance.
E
IS THE BEER REGULARIZER IMPLICITLY INCORPORATED INTO VALUE APPROXIMATION ALGORITHMS?
Our theory and regularizers are built upon the Bellman equation. Almost all the value-based DRL algorithms leverage the Bellman equation. Thus, would suspect that this regularizer effect would also be implicitly satisfied by these algorithms without BEER since these algorithms also use the Bellman equation. Kumar et al. (2021) shows that the dynamics of NNs result feature co-adaptation. Such ro-adaptation makes the inner product of representations grow fast, which potentially makes DRL agents fail hold the bound. To corroborate this claim, we conduct additional experiments. During the training process, we collect data every two thousand training steps to ascertain whether the bound Equation (9) is preserved or not.
We define
||| | | 2
C o(s,)) (1lo ()ll(s
(39)
C== means the bound in Equation (9) holds, otherwise it does not. The results of the experiments are presented in Figure 6. The results demonstrate that the tested algorithms don't 't consistently hold the bound. While BEER makes the agent hold the bound.
Specifically, BEER maintains this bound across the first three tasks, thereby significantly outperform- ing other algorithms these tasks (see Figure 3). In contrast, alternative algorithms under test, those (DR3 and InFeR) specifically designed to increase the representation rank, fail to maintain this bound, resulting in inferior performance. On the finger-spin task, all algorithms meet this bound, and there is no discernible difference in performance among them. This demonstrates that our designed algorithm effectively ensures the satisfaction of this bound, thereby adaptively regularizing the rank and ultimately improving performance. These experimental findings are consistent with our theoretical analysis.
21

---

Published as a conference paper at ICLR 2024
F THE COEFFICIENT OF BEER
In this section, we investigate the selection of hyper-parameter B. We continue to use the four representative examples and consider eight different
environments identified in the main values for the hyperparameter 3. Our experimental results indicate that starting from 0.001, as the hyperparameter increases, the algorithm's performance gradually deteriorates due to the increasing interference of the regularization effect on the normal learning of the value function. The results suggest that 3 value range less than or equal 0.001 is reasonable. Performance differences at lower 3 values are not significant. Therefore, in the main text, we consistently use le - 3 as the hyperparameter.
Table 5: 10-Seed peak scores 1M Steps DMC. The best is marked with the blue color box. The results suggest that a 3 value range less than or equal to 0.001 reasonable. Performance differences at lower values are significant. Therefore, in the main text, we consistently use le 3 as the hyperparameter.
Domain
Task
0.0001
0.0005
0.001
0.005
0.01
0.1
1.0
10.0
Acrobot
Swingup
232.8
191.7
243.3
171.9
171.8
148.2
148.9
81.1
Hopper
Hop
376.1
404.5
365.7
282.3
362.7
217.9
0.4
0.6
Pendulum
Swingup
830.0
829.8
830.0
829.1
828.8
616.6
660.2
25.5
Finger
Spin
974.8
978.2
979.2
975.1
981.7
764.4
384.1
1.3
Average
Score
603.4
601.0
604.6
564.6
586.2
436.8
298.4
27.1
22

---

Published as a conference paper at ICLR 2024
G ADDITIONAL EXPERIMENTAL RESULTS ON DMCONTROL SUITE
In Section 4, we report BEER performance 12 challenging tasks. In Table 6, we report the performance of the tested algorithms on easier DMC tasks. The results show that BEER has slight performance advantage over the other algorithms, although not as large as the advantage on the other challenging tasks. This is because these tasks are relatively simple according the results reported by Yarats & Kostrikov (2020). So the performance difference between the algorithms is not too large. Less capable algorithms are also able perform very well.
Table 10-Seed peak scores over 1M Steps on 9 easy DMC tasks. these tasks, BEER has slight performance advantage over the other algorithms, although not as large as the advantage on the other challenging tasks. This is because these tasks are relatively simple, so the performance difference between the algorithms is not too large. Less capable algorithms are also able to perform very well. corresponds to a standard deviation over ten trials.
Domain
Task
BEER
DR3
InFeR
SAC
TD3
Cartpole
Swingup
879.2 -
875.8 =
875.1
869.5 + 5.7
872.3
Cheetah
Run
883.4 + 12.2
863.9 = 23.7
857.3 37.7
874.5 1 19.8
853.4 37.6
Finger
Turnasy
976.5 1 6.5
961.2 1 50.5
902.1 1 89.3
949.2 1 47.8
975.6 1. 12.2
Finger
TurnHard
949.2 37.1
796.8 123.6
816.9 158.4
896.5 97.6
967.9 t 10.0
PointMass
Easy
893.7 +
815.3 270.7
903.0 12.3
902.3 12.5
903.1 12.3
Reacher
Easy
985.4 3.6
982.5 3.9
984.2 2.8
984.5 4.0
984.2 3.5
Reacher
Hard
975.4 2.0
976.9 + 1.2
976.4 5.2
980.7 0.9
976.6 + 0.1
Walker
983.6 3.9
988.0
987.2 2.5
988.2
985.3 + 2.7
Walker
Run
793.6
= 12.6
653.7 129.5
697.7 61.8
791.0 26.1
680.8 108.4
Average
Score
924.4
879.3
888.9
915.2
911.0
23

---

Published as a conference paper at ICLR 2024
H COMPUTATIONAL OVERHEAD OF BEER
In this section, we assess the computational impact of our proposed method, BEER, compared to other algorithms. Our evaluation focuses main aspects: the number of learnable parameters and the runtime analysis.
Learnable Parameters. Unlike methods such InFeR, which introduce multiple additional heads to the value networks, thereby increasing the number of learnable parameters, BEER functions as a regularizer without adding any learnable parameters the base algorithm. This is significant advantage as maintains computational efficiency.
Runtime Analysis. To evaluate the runtime efficiency of BEER, conduct comparative experiments against algorithms such as InFeR, TD3, SAC, and DR3. These experiments were performed on GPU server equipped with an Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz and NVIDIA GeForce RTX 2080 Ti GPUs. Each task is executed sequentially under the same evaluation protocol. Table 7 presents the runtime in hours for each algorithm across various environments. As evident from Table Table 7, BEER demonstrates competitive runtime efficiency, further highlighting its computational viability for practical applications in deep reinforcement learning.
Table 7: Runtime comparison of BEER and other algorithms.
Environment
BEER
INFER
TD3
SAC
DR3
Pendulum-Swinn Swinngu
1.43H
1.79H
1.20H
1.68H
1.34H
Acrobot-Swingup
1.46H
1.83H
1.20H
1.71H
1.36H
Hopper-Hop
1.48H
1.83H
1.18H
1.74H
1.30H
Finger-Spinn
1.48H
1.81H
1.25H
1.85H
1.29H
Average Time
1.47H
1.81H
1.21H
1.74H
1.32H
I CLARIFICATION OF SG DEFINITION
We provide a detailed explanation about SG in this section. The term SG, standing for 'Stop Gradient', refers concept intrinsic the semi-gradient nature of the Bellman backup, as elaborated in section 9.3 of Sutton & Barto (2018).
To elucidate, formally define the SC operator follows.
Definition (Stop Gradient). The SG operator is applied to a differentiable function f RR R within the context of gradient- based optimization. When this operator applied, generates new function SG()), such that for any input vector x E IR", the output during the forward computation remains the same as that of f. i.e., (x) = fxx). However, during the backward computation phase, where gradients are typically calculated through backpropagation, the gradient of SG(J) with respect to x is explicitly set to zero. This is irrespective of the actual gradient of f at x. Formally, this concept is encapsulated by the following equation
Vx E R", SG(f)(x) f(x) and (GG())( 0.
(40)
J CLARIFICATION ON STATE- PAIR COVERAGE IN BEER
We discuss the state-action pair coverage of BEER in this section since this affects the regularization of rank.
A critical aspect of our approach is the significance of the reachability of the agent. Specifically, the representation of state-cction that are not reachable by any policy deemed irrelevant. This perspective directly influences our approach to handling state-action pairs in BEER.
Sampling Mechanism: BEER incorporates a uniform sampling strategy from the replay buffer, designed to theoretically encompass the entire spectrum of possible state-action pairs. This approach ensures that our representation accounts for a comprehensive set these pairs.
24

---

Published as a conference paper at ICLR 2024
Exploration Enhancements: In addition to the uniform sampling, BEER incorporates exploration tricks, such as adding noise to actions, to further ensure extensive coverage of ftateacction spaces.
Practical Implementation of Sampling. In practical terms, our implementation involves extensive sampling iterations, amounting to one million, with each batch comprising 256 samples. This substantial sampling framework important in enhancing the likelihood of covering all possible state-action pairings, thereby addressing concerns regarding limited coverage. The specifics of this process can be found Algorithm
K CONTRAST WITH PRIOR THEORETICAL WORK
Recent literature discusses how to maximize representation rank in bandit settings and RL set- tings (Papini et al., 2021a;b; Zhang et al., 2023). We discuss the relationship between our work and the theoretical foundations established in Papini et al. (2021a;b); Zhang et al. (2023). Our approach, while acknowledging the importance of maximizing representation rank, presents distinct perspective compared to these references.
Our work presents notable contrast to the contributions of Papini et al. (2021a;b); Zhang al. (2023). These works focus on selecting (pre-given) representations for ppcision-makinng problems while research centers on learning good representations in DRL. The high representation rank assumption Papini et al. (2021b;a cannot inherently bound the cosine similarity derived in our paper. Our approach, grounded in the structure of the Bellman equation and neural networks, uniquely bounds this similarity.
The diversity assumption in Papini et al. (2021b). The settings and definitions in Papini et al. (b021b) differ significantly from ours. In Papini et al. (2021b), the definition of representation u(s, a) << a(,,a), >, where μ is the true reward vector, 0 is learnable parameters vector. Thus, é is static representation matrix for computing reward. However, in our settings, 6(s8, a) is learnable representation for directly computing Q values. Let's first ignore the difference in settings. The answer is No. Their main theory is to achieve good regret, the optimal representations o' must span the whole representation space RRd where is the dimension of representation ő. It's a trivial fact that the rank of & cannot additionally bound the mean cosine similarity of its column vectors. For example, consider the matrix
1 + x 1- 1=(1+
1 + 1-x)
Here, A being full rank (x.2 - y22 * 0) does not bound the cosine similarity. The cosine similarity (C) is
- + (1=))) (1- v)
C v -1-=))+(11
(41)
and we know that lim = 1. In the case of matrix A with full rank, the cosine similarity can be infinitely close to I when 2 2 + O, which demonstrates that the rank cannot inherently bound the cosine similarity.
The UniSOFT assumption in Papini et al. (2021a). UniSOFT a necessary condition for constant regret in any MDP with linear rewards. Specifically, it shows that the span of the representation on (s, a) over all states and actions that are reachable under any policy π is the same as the span of the optimal representation (7,(s) over all states that have non-zero probability of being reached under the optimal policy x². This implies that the representation map captures all the necessary information represent the optimal policy. UniSOFT assumption cannot bound the cosine similarity derived in our paper. Because UniSOFT only considers the span of the representation map. UniSOFT doesn't t further consider the interrelationships between column vectors of the representation map, allowing high average cosine similarity (<l) for the column vectors. The UniSOFT condition can be satisfied even with high cosine similarity (can be infinitely close when x? y² # O. is full rank), as primarily concerns the span of representations.
25


Reviews:
[{"review_id": "9iqA4bzklx", "reviewer": "Reviewer_ykJX", "summary": "The paper proposes a novel method to control the representation rank of neural networks in deep reinforcement learning (DRL), which measures the expressive capacity of value networks. They argue that existing methods either ignore or unboundedly maximize the representation rank, which can lead to overfitting or underfitting problems.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method is well-founded. The authors establish an upper bound on the cosine similarity between the representations of consecutive state-action pairs, using the Bellman equation1. They demonstrate that this bound indirectly restricts the rank of the representation and offers a criterion for adaptive control.", "weaknesses": "It is recommended to thoroughly investigate the computational overhead of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HVDcXsy9h4", "reviewer": "Reviewer_Joqj", "summary": "This work studies how to learn a low-rank representation in reinforcement learning (RL). Following several previous works, the authors claimed that RL favors a representation with moderate rank, and they proposed a new regularization method based derived from the Bellman equation. In detail, such a regularization method controls the complexity of the learned representation, and it encourages the rank of the representation to be small empirically. Combined with the regularization method, several baseline methods such as DQN and DDPG outperform existing baselines.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The presentation of this paper is clear. \n2. The experiment setup is described very clear. \n3. The literature review is complete.", "weaknesses": "There are several things that can improve the paper. For instance, \n1. On page 4, the definition of $\\bar{\\phi}(s',a')$ seems not clear since $(s',a')$ serves as both the input of $\\bar{\\phi}$ and the random variable which is going to be integrated. \n2. On page 5, what is the exact definition of $\\mathbb{SG}$? \n\nI do not quite get why the authors need to design the regularization term as in (12). It seems more natural to me to set the regularization term as in (19), which avoids a calculation of the gradient of an inverse term $1/|\\phi(s,a)|$ which might hurt the stability of the optimization process. \n\nThe logic behind the superior performance of BEER is not clear to me. The authors tried to claim that 'representation rank affects the model performance (approximation error), while BEER explicitly controls the rank, thus BEER outperforms other baseline methods'. However, according to figure 2, InFeR has a similar representation rank as BEER, while it performs worse than BEER. Is there any explanation why such a phenomenon happens? Meanwhile, in simpler tasks (Grid World), the rank of BEER is higher than other baselines, while in complex tasks (Lunan radar), the rank of BEER is lower than other baselines. Due to the inconsistency, it is doubtful whether the performance gain is due to the better control of representation rank. If so, it indeed deserves more explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gi4FpM8avH", "reviewer": "Reviewer_dvZZ", "summary": "This paper studies an implicit regularization of representation rank in deep RL. The authors draw the intuition between the representation rank and the cosine similarity between adjustment state-action pairs.", "questions": "- Maximizing the representation rank has always been an important part of the literature. Several theoretical works [1, 2, 3] show that as long as all the representations are in the span of the covariance matrix (i.e. $x \\in E[xx^\\top], the diversity assumption in [1] or the UniSOFT assumption in [2]), the performance can be improved. It would be beneficial for authors to comment on the relationship between this paper and these theoretical results, e.g. will the cosine similarity be well-bounded under these assumptions?\n\n[1] Papini, Matteo, et al. \"Leveraging good representations in linear contextual bandits.\" International Conference on Machine Learning. PMLR, 2021.\n[2] Papini, Matteo, et al. \"Reinforcement learning in linear mdps: Constant regret and representation selection.\" Advances in Neural Information Processing Systems 34 (2021): 16371-16383.\n[3] Zhang, Weitong, et al. \"Provably efficient representation selection in low-rank Markov decision processes: from online to offline RL.\" Uncertainty in Artificial Intelligence. PMLR, 2023.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well written and easy to follow\n- The experiment shows a good performance and backup the theoretical insights", "weaknesses": "- The argument that the cosine similarity of the ** adjustment** state-action pairs $(s, a)$ and $(s', a')$ does not rigorously lead to the rank. This is because one can only control the similarity between limited state-action pairs, instead of all possible state-action pairs. Think of an extreme case that the representation in adjustment state-action pairs is iterating between $(0, 1, 0, \\cdots, 0)$ and $(1, 0, 0, \\cdots, 0)$, the implicit regularization will fail.\n- It seems that in the experiment (Figure 1), the representation rank of the proposed method is dropping. It would be beneficial if the authors can provide some explanation on this", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s6OnWIqpL3", "reviewer": "Reviewer_ef2F", "summary": "This paper studies an important issue of representation rank in deep reinforcement learning (DRL).  To tackle this problem, the authors propose BEER, which leverages the Bellman equation to derive an upper bound on the cosine similarity between consecutive state-action pair representations. This bound is then used as a regularizer to adaptively control the representation rank during training. The empirical studies on DeepMind control tasks validate the high efficiency of the proposed algorithm.", "questions": "Please refer to the above Weaknesses section.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and easy to follow. The proposed adaptive regularization is new and effective.\n\nThe empirical results clearly validate the high efficiency of BEER, which is impressive. Since I am not an expert in this specific area and haven't followed this line of recent literature, I will refer to others reviewers' opinions on the experiments.", "weaknesses": "I have some minor questions:\n\n1. The authors intuitively explained the intrinsic similarity between the cosine similarity and the representation rank with examples. I am curious about if there is some formal statement or proof on this topic, since it serves as a very important property of your paper. There is some parameter in the definition of the representation rank (e.g. $\\epsilon$) in Definition 1, but I didn't see it in the following discussion.\n\n2. When the authors transform the constraint in Eqn. (11) to the penalty regularizer with ReLU in Eqn.(12), is there any theoretical support for this transformation? And also is there any theoretical analysis on how to choose the value of $\\beta$? I check the ablation study on this hyperparameter on Appendix, but I feel its choice is still quite important but unclear in practice.\n\nFor the Eqn. (5), should the $\\phi$ be $\\Phi$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)