On the Geometry of Reinforcement Learning in Continuous State and Action Spaces
Authors: Saket Tiwari
Omer Gottesman
George Konidaris
Venue: Submitted to ICLR 2023 (2023)
Keywords: geometry
deep reinforcement learning
manifold

Abstract: Advances in reinforcement learning have led to its successful application in complex tasks  with continuous state and action spaces. Despite these advances in practice, most theoretical work pertains to finite state and action spaces. We propose  building  a theoretical understanding of continuous state and action spaces by employing a geometric lens. Central to our work is the idea that the transition dynamics induce a low dimensional manifold of reachable states  embedded in the high-dimensional nominal state space. We prove that, under certain conditions, the dimensionality of this manifold is at most the dimensionality of the action space plus one. This is the first result of its kind, linking the geometry of the state space to the dimensionality of the action space. We empirically corroborate this upper bound for four MuJoCo environments.We further demonstrate the applicability of our result by learning a policy in this low dimensional representation.   To do so we introduce an algorithm that learns a mapping to a low dimensional representation, as a narrow hidden layer of a deep neural network, in tandem with the policy using DDPG. Our experiments show that a policy learnt this way perform on par or better for four MuJoCo control suite tasks.

Under review as a conference paper at ICLR 2023
ON THE GEOMETRY OF REINFORCEMENT LEARNING IN CONTINUOUS STATE AND ACTION SPACES
Anonymous authors Paper under double-bling review
ABSTRACT
Advances in reinforcement learning have led to its successful application in complex tasks with continuous state and action spaces. Despite these advances in practice, most theoretical work pertains to finite state and action spaces. We propose building a theoretical understanding of continuous state and action spaces by employing a geometric lens. Central to our work is the idea that the transition dynamics induce a low dimensional manifold of reachable states embedded in the high-dimensional nominal state space. We prove that, under certain conditions, the dimensionality of this manifold is at most the dimensionality of the action space plus one. This is the first result of its kind, linking the geometry of the state space to the dimensionality of the action space. We empirically corroborate this upper bound for four MuJoCo environments. We further demonstrate the applicability of our result by learning a policy in this low dimensional representation. To do so we introduce an algorithm that learns mapping to low dimensional representation, as a narrow hidden layer of a deep neural network, in tandem with the policy using DDPG. Our experiments show that policy learnt this way perform on par or better for four MuJoCo control suite tasks.
INTRODUCTION
The goal of reinforcement learning (RL) agent is to learn an optimal policy that maximises the return which is the time discounted cumulative reward (Sutton & Barto 1998). Recent advances in RL research have lead to agents successfully learning in environments with enormous state spaces, such as games (Mnih al. 2015 Silver al., 2016), and robotic control in simulation (Lillicrap
et al. 2016, SS6ulman et al.) 2015 2017a) and real environments (Levine et al.) 2016 Zhu et al. 2020 Deisenroth & Rasmussen 2011). l). However, we do not have an understanding of the intrinsic complexity of these seemingly large problems. For example, in most popular deep RL algorithms for continuous control, the agent's 's policy is parameterised by a a deep neural network (DNN) (Lillicrap let al. 2016 (Schulman et al.] 2013 (21 but we do not have theoretical models guide the design of DNN architecture required to efficiently learn an optimal policy for various environments. There have been approaches measure the difficulty of RL environment from a sample complexity perspective (Antos et al. 2007] Munos & Szepesvari 2008 Bastani 2020) but these models fall short of providing recommendations for the policy and value function complexity required to learn an optimal policy.
We view the complexity of RL environments through geometric lens. We build on the intuition behind the manifold hypothesis, which states that most high-dimension real real-world datasets actually lie on nww-dimensional manifolds (Tenenbaum 1997 Carlsson et al., 2007 Fefferman al., 2013] Bronstein et al.) 202T); for example, the set of natural images are a very small, smoothly-vary subset of all possible value assignments for the pixels. A promising geometric approach is to model the data as diw-dimensional structure- manifold-smbedeee space. In supervised learning, especially deep learning theory, researchers have shown that the approximation error depends strongly on the dimensionality of the manifold (Shaham al. 2015 Pai et al.] 2019 [Cenn et al., 2019; CConingerr & Klock] (020). thereby connecting the complexity of the underlying structure of the dataset to the complexity of the DNN.
in a high-dinensional ambient
As in supervised learning, researchers have applied the manifold hypothesis in RL-RRL-i i.c. hypothesized that the effective state space lies on low dimensional manifold (Mahadevan 2005 Machado et al.

---

Under review as a conference paper at ICLR 2023
2017 2018 Banijamali et al. 2018 Wu et al. 2019 Liu et al. 2021). Despite its fruitful applications, this assumption of a low-d cww-dienniooal underlying structure has never been theoretically and empirically validated in any RL setting.
Our main result provides a general proof of this hypothesis for all continuous state and action RL environments by proving that the effective state space is manifold and upper bound its dimensionality by, simply, the dimensionality of the action space plus one. Although our theoretical results are for deterministic environments with continuous states and actions, we empirically corroborate this upper bound on four MuJoCo environments (Todorov et al. 2012). with sensor inputs, by applying the dimensionality estimation algorithm by Facco et al. (2017). Our empirical results suggest that in many instances the bound on the dimensionality of the effective state manifold is tight. To show the applicability and relevance of our theoretical result we empirically demonstrate that a policy can be learned using this low-dimensional representation that performs as well as or better than policy learnt using the higher dimensional representation. We present an algorithm that does two things simultaneously: 1) learns mapping to low dimensional representation, called the co-ordinate chart, parameterised by a DNN, and 2) uses this low-dimensional mapping to learn the policy. Our algorithm extends DDPG (Lillicrap et al. 2016) and uses it as baseline with gigeer-dimeniional representation as the input. We empirically show a surprising new DNN architecture with a bottleneck hidden layer of width equal dimensionality of action space plus one performs on par better than the wide architecture used by [illicrap et (2016). These results demonstrate that our theoretical results, which speaks to the underlying geometry of the problem, can be applied to learn low dimensional or compressed representation for learning in a data efficient manner. Moreover, we connect DNN architectures effectively learning a policy based on the underlying geometry of the environment.
BACKGROUND AND MATHEMATICAL PRELIMINARIES
We first describe the continuous time RL. model and Markov decision process (MDP). This forms the foundation upon which our theoretical result is based. Then provide mathematical background on various ideas from the theory of manifolds that we employ in our proof's and empirical results.
2.1 CONTINUOUS-TINE TIME REINFORCEMENT LEARNING
We analyze the setting of continuous-ting time reinforcement learning in a deterministic Markov decision process (MDP) which is defined by the tplle M = A. fr. Sq, A) over time t E [o, T). S c Rd, is the set of all possible states of the environment. A C R is the rectangular set of actions available to the agent. f S x A R+ + S and f & C a smooth function that determines the state transitions: s' = f(8 a, T) is the state the agent transitions to when it takes the action a at state for the time period т. Note that f(s, a, 0) = s, meaning that the agent's 's state remains unchanged if an action is applied for duration of = O. The reward obtained for reaching state is fr). determined by the reward function fr : S + R. 8t denotes the state the agent is at time and a, is the action it takes at time f. So is the fixed initial state of the agent
0, and the MDP terminates at f = T. The agent does not have access to the functions f and fr. and can only observe states and rewards given time [0,
The agent is equipped with policy, म S + A. that determines its decision making process. We denote the set of all the possible policies by II. Simply put, the agent takes action T()) at state 8. The goal of the agent is to maximise the discounted return J(z) Shd ff where Sitc = f(t, T (st),e) e) for infinitesimally small and all tlltt E [O, We define the action tangent mapping, g S x A Rdd for an MDP
f(s.a,c) 0f(s,a,c)
g(s, a) lim --- E
de
Intuitively, this captures the direction in which the agents state changes at state upon taking action a. For notational convenience we will denote g(s, π (s)) as Jn : S - Rd and name it the action flow of the environment defined for policy Note that 9r, is well defined function. Intuitively, 9n is the direction of change in the agent's 's state upon following policy π at state s for an infinitesimally small time. The curve in the set of possible states, or the state-trajyctory of the agent, is differential
2

---

Under review as a conference paper at ICLR 2023
equation whose integral form is follows,
+ L 9x (x( (dl)).
This solution also unique (Wiggins 1989) for a fixed start state, $o, and policy, т. The above curve is smooth curve if the policy is also smooth. Therefore, given MDP, M, and smooth deterministic policy, т E II, the agent traverses a continuous time state-trajectory or curve H,,= : [0, T) - S.
The value function time for policy is the cumulative future reward starting at time t:
0" (s)) S. e - fx(F
(2)
Note that the objective function, J (Tx), is the same as 2n (30). Our specification is very similar to classical control and continuous time RL (Cybenko 1989 Doya 2000) with the key difference being how we define the transition function, f.
2.2 MANIFOLDS
MDPs, in practice, have a low-dimensional underlying structure resulting in them having fewer degrees of freedom than their nominal dimensionality. In the Cheetah MujoCo environment, with image observations, the goal of the RL agent is to learn a policy to make the Cheetah move forward plane. The actions available the agent
fast possible, where the Cheetah is constrained providing torques at each one of the 6 joints. In the case of learning from visual input in a MuJoCo environment like 2D cheetah, one can describe the cheetah's 's state by its "pose" and position instead of the 128 x 128 pixels of the image. The idea of a low dimensional manifold embedded in a high dimensional state space formalises this.
A function h X + Y. from one open subset X C R to another open subset Y C R²/, is diffeomorphism h is bijective, and both and h-1 are differentiable. Intuitively, a low dimensional surface embedded in a high dimensional Euclidean space can be parameterised by differentiable mapping, and if this mapping is bijective we term it a diffeomorphism. Here X said to be diffeomorphic to Y. A manifold is defined as follows.
Definition 2.1. A subset M C R* is called smooth m-dimensional submanifold of R² (or m- manifold in RR) iff every point p E M has an open neighborhood U C Ik* such that U n M is diffeomorphic to an open subset O C R A diffeomorphism, & U N + O is called a coordinate chart of M and the inverse, ==
M is called a smooth parameterisation of M.
U
ga
Figure 1: The surface of an open cylinder of unit radius, denoted by S2, in RR3 is a 2D manifold embedded in 3D space. More formally, S {zx y, 2) + y² 1, (-h, h)} where the cylinder's height is 2h. One can smoothly parameterise S as (0, b) (sin 0, cos 0, b). The
coordinate chart is ő(x, y, z) = (sin-1 x, z).
We illustrate this with an example in Figure If AM C R is non-empty smooth m-manifold then m < k reflecting the idea that a manifold of lower or equal dimension than its ambient space. A smooth curve Y infinitely differentiable for all The derivative of y at is denoted as ý(t). The length of curve y: M is defined as I(>>) = Jr [lt()) where || denotes the vector norm. In the Euclidean
M is defined from an interval I C R the manifold M function that is
3

---

Under review as a conference paper at ICLR 2023
space the distance between two points is the length of the unique straight line connecting them. Similarly, the geodesic distance between two points a, E M is defined as the length of the shortest such curve,
b. We denote the geodesic distance [0, 1] + M, such that 7(0) = and (1) between a, b E M on the manifold M is denoted by the function d (a, b). The set of derivatives of the curve time f, (t), for all possible smooth y, form set that is called the tangent space. The langent space characterises the geometry of the manifold and it is defined as follows.
Let M be an m-manifold TR. and p E M be a fixed point. vector u E RR.h is called tangent vecior of M at p if there exists smooth curve y - M such that y(0) p. p.)(0) = v. The set 7,M- == (5))) : -> M is smooth, (0)) = p} of tangent vectors of M at p is called the langent space of M at p.
Definition 2.2.
Continuing our example, the tangent space of a point p in S is the vertical plane tangent to the cylinder that point. For small enough and a vector E T2,5² there exists unique curve 7: [[e,e] + 82 such that 7(0) p and ((0)
3 STATE SPACE GEOMETRY
The state space is typically thought of as a dense Euclidean space in which all states lie, but it is not necessarily the case that all such states are reachable by the agent. Two main factors constrain the states available to an agent: 1) the transition function and the actions that are available to an agent, and 2) the start state 8g. We therefore define S. as the effective set of states. Under the assumptions that IT is the set of all smooth policies, S + A. for a continuous time MDP, M,, and those made in section 2.1 we define the effective set of states as follows.
Definition 3.1. For MDP, M, the effective set of states, Se, is defined as the union of the sets of states of all possible continuous curves traversed by the agent for the set of all smooth policies. Formally, Se = Usss H HM, (t) for some E (0,T)), where II is the set of everywhere smooth policies with domain S, and M (0,t) + S is the curve defined by the policy a given the MDP M.
We make three additional assumptions:
1. Full rank Jacobian: f has a full rank Jacobian, in variables [a, t), for all s E S,a E A and t E (0, T), and for a fixed policy ı the solution for the equation s7 s, if it exists, is unique in L.
2. No small orbits: there exists solution for the equation s7 = s, if it exists, is unique in t.
0 such that for all t and fixed policy the
3. Action space restriction: there exists an r and an open restriction A', of A, for every state S., such that A', C AA and for all 0 < e < if we have ((s,a1,E) = [(s, 02 e) for some fixed 01 42 E then d1 = 02
We provide further explanations for these assumptions and how they restrict our study in Appendix A Under these two assumptions for the setting of continuous RL as in Section 2.1 we have the result stated below.
Theorem 3.2. The set of effective states, S., is smooth manifold of dimensionality at most d. + 1.
We provide the proof in Appendix B Intuitively, the action set limits the directions in which the agent can go. For example, a single action executed for finite time interval makes the agent traverse a ID curve. The union of all such curves is the only subset of the state space the agent can reach. The key technical idea is to construct a coordinate chart, and therefore diffeomorphism, from subset of low dimensional euclidean space to a subset the state space. We state and consider the function A r). + S such that á(a, = fs, e). This result pertains the global geometry of the state manifold, meaning the state manifold everywhere has this low dimensional structure.
Our result formalises the long-held idea that there is in fact lower dimensional structure to the state manifold; in particular, this is true when d, + 1 < d,, which almost always the case for RL environments. It also formalises the idea that the agent interacting with the environment using the set of actions at its disposal. Henceforth, we refer to S, as the state manifold, We also present an immediate corollary of Theorem 3.2
only observe data for states reachable by
4

---

Under review as a conference paper at ICLR 2023
T,S.
S.
Figure 2: Consider hypothetical scenario as in the image above where upon taking an action even though the agent moves locally state in the direction of the red arrow (along In (s)) as time progresses it ends up travelling along the green arrow which is on the manifold S.. There some constraint that restricts the state space "available" to agent. Thus, the agent only reach the states that have a valid solution for the Equation for any smooth policy.
Corollary 3.3. For every E A and fixed 8 E Se. the action tangent mapping, g(8, a). maps from the set of actions A to TyS. and therefore there exists vector Va E T Se such that Va g(3,0) a) for all E A.
This implies that locally an agent is moving along tangent in the tangent space of a state in the state manifold acts upon the environment. The corollary follows from Definition 2.2 and its proof is in Appendix B Intuitively, Corollary 3.3 implies that locally the agent's 's state changes in constrained manner dictated by the actions available and the local geometry of the manifold, i.e.. the tangent space and its projection onto the state manifold. We illustrate this in Figure 2 We note that the idea of actions mapping to the tangent space was assumed earlier by Liu et al.) (2021 for constrained RL but Corollary both proves this assumption and shows that it holds in general for continuous RL environments.
4 CONNECTIONS BETWEEN CONTINUOUS TIME DETERMINISTIC RL AND EMPIRICAL RL
We have presented our results in the continuous time RL setting, which is an underutilized theoretical tool in the study of RL. Here, and in the experimental section, we argue that it in fact useful model for theoretical analysis in the continuous state and action, discrete time, setting. The primary intuition that, the context of simulated robotic control problems, our work approximates the agents behavior during the transition period (t, + 1) the action a changes to a++1 smooth manner and similarly for s, to $141- In this context, the discrete-time observations can be viewed time-uniform samples from underlying continuous time process. This is in keeping with with robotic control, in a robot with various joints we can only measure the joints over intervals which are then attributed to discrete time measurements. Similarly, we can actuate the motors up frequency and those can be considered as discrete control signals, even though the underlying physical process continuous time.
More formally, consider a discrete time MDP with continuous state and action spaces, a discrete state trajectory is the sequence of states {st)=1 and similarly for actions, {ar)7=1- Let & Ax [0, 1]xAA + AA be function that acts as smooth transformation between two successive actions a,, a++1 such that the agent transition from 8, to 8 in the continuous time model, as described in Section 2.1 In other words, U(a,, 1, 4++1)= = 04-1 is the discrete to continuous time transformation of actions. We postulate that there exists an operator Hg. dependent on the MDP M, such that a discrete trajectory can be transformed into a continuous time trajectory. Proving such an augmentation exists, given that the underlying physical process as described in Section 2.1 and the discrete trajectory is sampled at discrete time intervals described above, is beyond the scope of our current work. Intuitively, since the discrete time trajectories are temporally spaced out measurements of continuous trajectorics the existence of such operator can be considered an "inversion" of the sampling process.
5

---

Under review as a conference paper at ICLR 2023
Predited Dmm
Prof-tted D3m
Predrced
50000
100000
25000 75000
4000 6000
(a) Walker2D
(b) Cheetah
(c) Reacher
(d) Swimmer
Figure 3: State manifold dimensionality, blue, close to da + (horizontal red line) and far below d, (horizontal green line) for various environments, estimated using the algorithm by Facco et al. (2017).
Finally, we address the assumption of deterministic transitions. For our empirical analysis we use the popular MuJoCo environments for robotic control with sensor inputs (Todorov al. 2012) and these environments are deterministic for all practical intents and purposes (see Appendix C We argue further that our theoretical model has broader applicability. One way to model transitions is to assume that the underlying state transitions are deterministic but the observations the agent receives have additive noise. More formally, a simplistic way to model stochastic transition F(s, a, f(s c) NN(,, ar) is the stochastic transition function such that there is additive noise to this deterministic transition. We postulate that the effective set of states can be modeled as a manifold with each observation the agent makes lying certain distance to the state manifold and the distance normally distributed. This is also the idea behind many manifold learning paradigms: the data lies at distance to a low-dimensional manifold and this distance is distributed according to some probability law (Fefferman et al.) 2013 Pal et al. 2019).
EMPIRICAL VALIDATION
Our empirical validation is two fold. First, we show that the bound on the manifold dimensionality as in Theorem 3.2 holds in practice. Second, demonstrate the practical relevance of our result by learning a "bottleneck" representation using which an RL agent can learn effectively.
5.1 EMPIRICAL DIMENSIONALITY ESTIMATION
To empirically corroborate our main result (Theorem 3.2) we perform experiments in the MuJoCo domains provided in the OpenAl Gym (Brockman et al. 2016). These are all continuous state and action spaces with de < de for simulated robotic control tasks. The states are typically sensor measurements such as angles, velocities or orientation, and the actions are torques provided at various joints. We estimate the dimensionality the state manifold S., To sample data from the manifold, we record the trajectories from multiple evaluation runs of DDPG across different seeds (Lillicrap) ctt al. 2016), with two changes: we use GELU activation (Hendrycks
Gimpel 2016) instead of single hidden layer network of width 400
ReLU. in both policy and value networks, and also instead hidden layers for both the networks. This is inline with the assumptions for Theorem 3.2 in Section 3 that the policy is smooth. Performance is comparable to the original DDPG setting (see the Appendix [ For background on DDPG refer to Appendix[B FF We then randomly sample states traeectorics the dimensionality with 10 different subsamples of the same size to provide an error region for the estimates.
from the evaluation trajectories to obtain a subsample of states, D {s S.. We estimate
We employ the dimensionality estimation algorithm introduced by Facco et al. (011)) which estimates the intrinsic dimension of datasets characterized by non-uniform density and curvature, to empirically сorroborate Theorem 3.2 Further details about the dimensionality estimation procedure are presented in Appendix The estimates for four MuJoCo environments are shown in Figure 3 For all environments the estimate remains in the neighbourhood of d, + in keeping with Theorem 3.2
5.2 LEARNING VIA THE LOW-DIMENSIONAL MANIFOLD REPRESENTATION
We now validate the relevance of Theorem 3.2 using a popular policy gradient method, deep deter- ministic policy gradient (DDPG) Lillicrap al. 2016), which discrete time. DDPG is framework

---

Under review as a conference paper at ICLR 2023
for learning in environments with continuous action spaces, and the policy and value function are parameterised by DNNs. Let 0" be the parameters of the policy DNN. The agent learns by updating the policy parameters with respect to the discrete time discounted return, J(0T):
0 + tt + a oo=
where ox is the learning rate for the policy and J(0T) is the discounted return objective. For further details and background on DDPG please see Appendix [F] We do so by learning mapping to a low dimensional manifold of size da + 1 from the control input space that feeds into the policy. The central idea is to show that this compressed representation can be used to learn a control policy with RL without any loss-and possibly even a gain- of performance.
Our goal in validating the applicability of this low dimensional representation is to learn an isometric coordinate chart from the high dimensional representation for states in Se a low dimensional Euclidean space IRm where m S do + 1 as noted in Theorem 3.2 and then compare learning a policy using this compression of the state to learning from the full state. We denote the coordinate chart by &h S. - Rd.+1 parameterised by 8V For a coordinate chart to be isometric it must preserve the geodesic distance on the manifold Se. meaning for $1, 82 E Se we have that ds, ($1, $2) = [1yb(s); - $20")[22. where ds,, (2) is the geodesic distance between 81 and 82 on the manifold S. and || |/2 denotes the Fuclidean norm. Note that imposing isometry on a coordinate chart is stronger condition than it being diffecmorphism because it is distance preserving, in addition to being a bijection and differentiable. Such isometric mappings have been used in the past to learn the underlying manifold for digeniinnnsi data (Tenenbaum et al., 22000 Basri & Jacobs 2017] Pai et al., 2019). 9). This is done to ensure tractability of the objective that we introduce below. Given dataset of states D = {s:) C SS we minimize the loss
(00) =
1 N
Σ BL,0,ED
(ds. (81, - [1bb(11;*)) (1200))))2)
which is similar to the loss for learning low-dimensional manifold embeddings introduced by Tenen- baum et al. (2000). The computation and estimation of the geodesic distance, ds. is particularly challenging. One approach is to use a graph. as a discrete approximation of a manifold, to calculate this geodesic distance (Yan et al. 2007 Dong et al. 2011). The graph is constructed with each datapoint as node and an edge in between two datapoint if one of them is km -ncarestt neighbor of the other in the dataset. In addition to that there a distance attribute associated with every edge that is the Euclidean distance between the two points. Therefore, the geodesic distance between two datapoints can be approximated the sum of these edge-wise attributes for the edges constituting the shortest path between the two points. Another practical challenge in the empirical estimation of the loss Ly is sampling pairs of states for which we can obtain approximate geodesic distances with reasonable compute time. In summary, we calculate ds, in three steps: 1) sample data from replay buffer and therefore the manifold S.. 2) construct a -m-nnst neighbor graph with edge attributes, and 3) estimate the distance between two states using the sum of edge attributes of the shortest path between these edges. The exact procedure employed by us is detailed in Appendix and illustrated with simple example in Figure I0 further algorithmic details are also provided in Appendix [
Although there has been significant work in learning isometric mappings Paa et al. 2019 Basri & Jacobs 2017) and embeddings (Tenenbaum et al.) 2000 Zha & Zhang 2003). for data sampled from a manifold, these prior works assume that the data distribution remains static throughout the training process. However, the state distribution changes with the policy in RL and this mapping feeds into the policy itself, effecting the agents performance. We learn this isometric mapping, v/ in tandem with the policy to account for the distribution shift. To do so we introduce an intermediate hidden layer of size da + 1 in the policy DNN. The output of this layer is then trained by performing gradient descent on the loss Lig. In this architecture 0 C f" and the gradient is as follows:
8Th 8 + a J(=") - a V Lo.0
where an and ag are the learning rates for the policy and the coordinate chart respectively. Note that update for parameters 0t is then 0 + 0 a. V Lv, in addition to the update with respect to the objective for increasing the discounted return (See Appendix I and Algorithm T for further details). This is because for all 0 E 8² 8th we have Vo,L(8%) O. We illustrate the architecture and how the representation feeds into the loss, for the policy network, in Figure 4 Note that the architecture and gradient updates for the DNN parameterising the Q function remains the same as
7

---

Under review as a conference paper at ICLR 2023
Our Architocture
DDPG Architecture
VanJ
m(e)
300 Naurona
300 Neurons
51(6)
Figure 4: We introduce a bottleneck hidden layer of width da + which is the output the coordinate chart, v., Le. the green colored base of the DNN. The output of this coordinate chart is fed into the manifold loss (Equation
Moas Eos setume
1500
DORG
Wein Eva Rature
8020
DOPG
-50
Retums
2.0
3020
B420
-75
500
Eve
-10
1000
-125
2030
150
-500
250
520
500
250
500
750
Epech
Esoch
Epeci
Epoch
(a) Walker2D
(b) Cheetah
(c) Reacher
(d) Swimmer
Figure 5: For all the environments we use ai the ppermparaeeters are the same as reported by Lillicrap et al. (2016), over 6 random seeds.
10-5 in comparison ag 10-4, and the rest of
used by Lillicrap et al. (2016), for control inputs, which is two hidden layer DNN with 400 neurons in the first hidden layer, 300 in the second one and a one dimensional output.
We present results for different MuJoCo environments: Cheetah, Walker2D, Swimmer and Reacher in Figure 5 and for three out of four of these environments the performance either the same or better than the DDPG baseline. Our results strongly suggest that compressed isometric representation of dimesionality de + 1, learned using gradient updates can be used for learning a policy. This furthers our argument: there is low dimensional structure RL problems and it can be employed to learn more efficiently. Algorithmic details are given in Appendix [ Ablation studies for the newly introduced hyper-parameter au in Appendix [ and comparison to training absence of manifold loss is provided in Appendix M due to space limitations. In addition to this, we observe that the manifold loss (Equation 3 also drops steadily as training progresses, meaning that the representation being learnt, sx f.). is a low-dimensional isometric representation which retains the manifold geometry. The graphs for how the manifold loss, Lyp, evolves in each case is given with the interpretation. We explain the reasons for Reachers' failure for our algorithm in Appendix K In summary, to learn an isometric mapping to the Euclidean space the agent might require more than (Sm) dimensions, for the case of Reacher. The additional constraint that place, that the learnt coordinate chart has to be isometric, is detrimental in this case. As has been done in manifold learning literature, need methods for learning the low-dimensional mapping that do rely on learning an isometry (Roweis & Saul 2000 Zhang & Zha 2004). We demonstrate the effects of changing the width of the bottleneck layer the Cheetah domain in Figure [2] in the Appendix. We also need better methods to approximate the geodesic distance between data points in the very high-dimensional setting. We also report additional results for learning with soft actor critic algorithm (Haarnoj et al. 2018) in conjunction with manifold representation learning in Appendix
[igure6]alone
6 RELATED WORK
There has been significant empirical work that assumes the set of states to be manifold in RL. The primary approach has been to study discrete state spaces as data lying on a graph which has an underlying manifold structure. Mahadevan & Maggioni (2007) provided the first such framework to utilise the manifold structure of the state space in order to learn value functions, Machado et al. (2017) and Jinnai al. (2020) showed that PVFs can be used to implicitly define options and applied
8

---

Under review as a conference paper at ICLR 2023
1,4
0.40
0.04
Haahog L29
Myntta LOAN.
0.35
Meritodd tove
0,025
0.00
2.0
0.30
0.023
0.02
0.25
0.6
0.20
0.03
0
230
500 Epoch
750
0
250
500
750
250
500
750
250
120
750
spach
Esoch
Epach
(a) Walker2D
(b) Cheetah
(c) Reacher
(d) Swimmer
Figure 6: We observe that the the loss Ly gradually decreases in all instances except Reacher where the performance is sub-par. For the Walker2D environment we see increase in manifold error that matches the downward spike in performance in Figure 5 but the return for our method still surpasses the baseline in this case.
them to high dimensional discrete action MDPs (Atari games). Wu et al. (2019) provided an overview of varying geometric perspectives of the state space in RL and also show how the graph Laplacian is applied to learning in RL. Another line of work, that assumes the state space is manifold, is focused on learning manifold embeddings or mappings. Several other methods apply manifold learning to learn compressed representation in RL (Bush & Pineau 2009; Antonova et al. |0200] Liu et al.)
202I). Jenkins & Mataric (2004) extend the popular ISOMAP framework (Tnenbaum, 1997) to spatio-temporal data and they apply this extended framework to embed human motion data which has applications in robotic control. [Bowling et al. (2005) demonstrate the efficacy of manifold learning for dimensionality reduction for robot's position vectors given additional neighbourhood information between data points sampled from robot trajectories. At the same time, continuous RL has been applied continuous robotic control (Doya 2000 Deisenroth & Rasmussen 2011 Duan al. 2016). We apply continuous state, action and time RL as a theoretical model to study the geometry of popular continuous RL environments for the first time.
More recently, in various papers that take a theoretical approach to deep learning the intrinsic dimension of the data manifold and its geometry play an important role in determining the complexity of the learning problem (Shaham et al. 2015 Cloninger & Klock 20200] Goldt et al. 2020 Paccolat al. 2020, [Buhhanan et al. (021). SchmidttHieberr (2019) shows that, under assumptions over the function being approximated, the statistical risk deep ReLU networks approximating a function can be bounded by an exponential function of the manifold dimension. [Basr & Jacobs (2017) theoretically and empirically show that SGD can learn isometric maps from figh--imensional ambient space down to mn-dimensional representation, for data lying on an -i-dimensional manifold, using a two-hidden layer neural network with ReLU activation where the second layer is only of width m. Similarly, Ji et al. (2022) show that the sample complexity of off-policy evaluation depends strongly on the intrinsic dimensionality of the manifold and weakly on the embedding dimension. Coupled with result, these suggest that the complexity of RL problems and data efficiency would be influenced more by the dimensionality of the state manifold, which upper bounded by d, 1, as opposed to the ambient dimension. Finally, our work could be applied in conjunction with recent work that studies RL algorithms in light of the underlying structure of deep Q-functions (Kumar al., (021).
DISCUSSION AND CONCLUSION
We have shown that that the dimensionality of the manifold is upper bounded by d, + 1 and have empirically verified it (Figure 3). This proves that the popular manifold assumption (Mahadevan 2005) Machado et al., 2017; 2018 [Banjaaali et al. 2018; Wu et al., 2019 Liu et al.) 2021) holds under certain conditions in continuous-tim time reinforcement learning. It also shows that there is an underlying lower dimensional structure to the MDPs. Additionally, we demonstrate the applicability of this result in practical setting by showing that DDPG agent can learn efficiently in this highly compressed, low-dimensional space. Our newly introduced architecture and simultaneous low-dimensional representation learning along with policy learning performs on par or better than the baseline DDPG approach, for MuJoCo environments with sensor inputs. Overall, we show theoretical bound on intrinsic dimensionality of continous RL problem and also show the efficacy of this www-imenssional representation in learning policy. This opens up room for new theoretical and empirical advances paving way for better DNN architecture design and representation learning algorithms.

---

Under review as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
We offer the following details for all the experiments we have performed, in the main body or appendix: hyperparameters, sample sizes, GPU-hours, CPU-hours, code, neural network architectures, Python libraries used, input sizes, and external code bases. We also provide the code with instructions on running in the supplementary material. All the details run the code and its location are in Appendix O All the hyperparameters, for our experiments in Section 5.2 can be found in appendices I anddn
REFERENCES
Alessio Ansuini, Alessandro Laio, Jakob H. Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. In NeurlPS, 2019.
Rika Antonova, Maksim Maydanskiy, Danica Kragic, Sam Devlin, and Katja Hofmann. Ana- lytic manifold learning: Unifying and evaluating representations for continuous control. ArXiv, abs/2006.0871 18, 2020.
András Antos, Csaba Szepesvari, and Rémi Munos. Learning near-optimal policies with bellman- residual minimization based fitted policy iteration and a single sample path. Machine Learning, 71:89-129, 2007.
Francis Bach and David M. Blei (eds.). Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, 2015. JMLR.org. URL
ngs. ml press/
v377
Ershad Banijamali, Rui Shu, Mohammad Ghavamzadeh, Hung Hai Bui, and Ali Ghodsi. Robust locally-linear controllable embedding. In AISTATS, 2018.
Ronen Basri and David W. Jacobs. Efficient representation of oin-dimesiional manifolds using deep networks. ArXiv, htss/100.004223, 2017.
Osbert Bastani. Sample complexity of estimating the policy gradient for nearly deterministic dynamical systems. ArXiv, abs/1901 108562, 2020.
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents (extended abstract). In IJCAI, 2013.
Michael Bowling, Ali Ghodsi, and Dana Wilkinson. Action respecting embedding. Proceedings of the 22nd international conference on Machine learning, 2005.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. ArXiv, abs/1606.015400, 2016.
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Velivckovi'e. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. ArXiv, abs/2104.1344 2021.
Sam Buchanan, Dar Gilboa, and John Wright. Deep networks and the multiple manifold problem. ArXiv, abs/2008. 1245, 2021.
Keith Bush and Joelle Pineau. Manifold embeddings for model-based reinforcement learning under partial observability. NIPS, 2009.
G. Carlsson, T. Ishkhanov, V. D. Silva, and A. Zomorodian. On the local behavior of spaces of natural images. International Journal of Computer Vision, 76:1-12, 2007.
Minshuo Chen, Haoming Jiang, Wenjing Liao, and Tuo Zhao. Efficient approximation of deep relu networks for functions on low dimensional manifolds. ArXiv, abs/1908.018... 2019.
Alexander Cloninger and Timo Klock. Relu nets adapt to intrinsic dimensionality beyond the target domain. ArXiv, abs/2008. 02545,. 2020.
10

---

Under review as a conference paper at ICLR 2023
G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2:303-314, 1989.
Marc Peter Deisenroth and Carl Edward Rasmussen. Pilco: A model-based and data-efficient approach policy search. In ICML, 2011.
Wei Dong, Moses Charikar, and Li. Efficient k-nearest neighbor graph construction for generic similarity measures. In WwW, 2011.
Kenji Doya. Reinforcement learning in continuous time and space. Neural Computation, 12:219-245, 2000.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and P. Abbeel. Benchmarking deep reinforce- ment learning for continuous control. In ICMI., 2016.
Elena Facco, Maria d' Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimen- sion of datasets by a minimal neighborhood information. Scientific Reports, 7, 2017.
C. Fefferman, S. Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. arXiv: Statistics Theory, 2013.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop Representation Learning Graphs and Manifolds, 2019.
Sebastian Goldt, Mare Mézard, Florent Krzakala, and Lenka Zdeborová. Modelling the influence of data structure on learning in neural networks. ArXiv, abs/1909. 1500, 2020.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. Proceedings of the 22ndl ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
Tuomas Haarnoja, Haoran Tang. P. Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In ICML, 2017.
Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018.
Matthew J. Hausknecht and Peter Stone. The impact of determinism on learning atari 2600 games. In AAAI Workshop: Learning for General Competency in Video Games, 2015.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv: Learning. 2016.
Odest Chadwicke Jenkins and Maja J. Mataric. A spatio-temporal extension to isomap nonlinear dimension reduction. In ICML '04, 2004.
Xiang Ji, Minshuo Chen, Mengdi Wang, and Tuo Zhao. Sample complexity of nonparametric off-policy evaluation on low-dimensional manifolds using deep networks. ArXiv, abs/2206.02887. 2022.
Yuu Jinnai, Jee Won Park, Marlos C. Machado, and George Dimitri Konidaris. Exploration in reinforcement learning with deep covering options. In ICLR, 2020.
Steven G. Krantz and Harold R. Parks. The implicit function theorem. 2002.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60:84 90, 2012.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under- pataettzaaattnn inhibits data-efficient deep reinforcement learning. ArXiv, abs/2010.14498, 2021.
Sergey Levine, Chelsea Finn, Trevor Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. ArXiv, Abs/15544000700 2016.
11

---

Under review as a conference paper at ICLR 2023
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs/1509.02971, 2016.
Puze Liu, Davide Tateo, Haitham Bou-Ammar, and Jan Peters. Robot reinforcement learning on the constraint manifold. In CoRL, 2021.
Marlos C. Machado, Mare G. Bellemare, and Michael Bowling. A laplacian framework for option discovery in reinforcement learning. ArXiv, abs/1703.00956, 2017.
Marlos C. Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Mur- ray Campbell. Eigenoption discovery through the deep successor representation. ArXiv, Abs/1710.11089) 2018.
Sridhar Mahadevan. Proto-value functions: developmental reinforcement learning. Proceedings of the 22nd international conference on Machine learning, 2005.
Sridhar Mahadevan and Mauro Maggioni. Proto-value functions: A laplacian framework for learning representation and control in markov decision processes. J. Mach. Learn. Res., 8:2169-2231 2007.
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In ICLR, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):59-5533 2015. doi: 10.1038/nature 14236. URL
https://doi. ort/:..orgg/0.. http:///...00///auaeeeee/e/e
Rémi Munos and Csaba Szepesvari. Finite-time bounds for fitted value iteration. J. Mach. Learn. Res., 9:815-8571 2008
John Nash. C1 isometric imbeddings, Annals of Mathematiccs, 60(3):383-396, 1954. ISSN 0003486X.
URL http://wwwit http:///wwwww.toooroooooooo http://t..oggg/ttaaae/////a3//a.
Jonas Paccolat, Leonardo Petrini, Mario Geiger, Kevin Tyloo, and Matthieu Wyart. Geometric compression of invariant manifolds in neural networks. Journal of Statistical Mechanics: Theory
and Experiment, 2021, 2020.
Gautam Pai, Ronen Talmon, Alexander M. Bronstein, and Ron Kimmel. Dimal: Deep isometric manifold learning using sparse geodesic sampling. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 819-828, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, deep learning library. H. Wallach, H. Larochelle, Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8033 8035. Curran Associates, Inc., 2019.
Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel. M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2-22825-8800. 2011.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: online learning of social representations. Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, 2014.
12

---

Under review as a conference paper at ICLR 2023
Sam T. Roweis and Lawrence K. Saul. Nonlinear dimensionality reduction by locally linear embed- ding. Science, 290 5500:2323-6 2000.
Johannes Schmidt-Hieber Deep relu network approximation of functions on a manifold. ArXiv, abs/1908.0065 00695, 2019.
John Schulman, S. Levine, Abbeel, Michael I. Jordan, and Moritz. Trust region policy optimiza- tion. ArXiv, 1bs/11502.05477, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, hass/1777003477 2017a
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 2e//17070064477 2017b. URL http::/arxiv. org. abs/
1707.0634
Uri Shaham, Alexander Cloninger, and Ronald R. Coifman. Provable approximation properties for deep neural networks. ArXiv, abs/1 00335, 2015.
David Silver, Aja Huang. Chris Maddison, Arthur Guez, L. Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-489, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. CoRR, abs/ 409.1566, 2015.
Richard Sutton and Andrew Barto. Introduction to reinforcement learning, volume 135, MIT press Cambridge, 1998.
J. Tenenbaum, V. de Silva, and Langford. A global geometric framework for nonlinear dimension- ality reduction. Science, 290 5500:2319-23, 2000.
Joshua B. Tenenbaum. Mapping a manifold of perceptual observations. In NIPS, 1997.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based based control. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
George E. Uhlenbeck and Leonard Salomon Omsteen. On the theory of the brownian motion. Physical Review, 36:823 841, 1930.
Stephen Wiggins. Introduction applied nonlinear dynamical systems and chaos. 1989.
Yifan Wu, G. Tucker, and Ofir Nachum. The laplacian in rl: Learning representations with efficient approximations, ArXiv, abs/1810.04555 2019.
Shuicheng Yan, Dong Xu, Benyu Zhang, Hongliang Zhang, Qiang Yang, and Stephen Lin. Graph embedding and extensions: A general framework for dimensionality reduction. IEEE Transactions on Pattern Analysis and Machine Intelligence. 29:40-51, 2007.
Greg Yang and Edward J. Hu. Feature learning in infinite-width neural networks. 2os/2011.14522. 2020.
ArXiv,
Hongyuan Zha and Zhenyue Zhang. Isometric embedding and continuum isomap. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 864 871, 2003.
Zhenyue Zhang and Hongyuan Zha. Principal manifolds and nonlinear dimensionality reduction via tangent space alignment. SIAM J. Scientific Computing, 26:3 13:313-338, 2004.
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of real-world robotic reinforcement learning. ArXiv, abs/2004. 1at/20004.157000 15700, 2020.
13

---

Under review as a conference paper at ICLR 2023
A ASSUMPTIONS
We list all the assumptions by section and consequently by theorems. We also provide some intuition what these assumptions mean and how they limit study.
Assumption made in Section 2.1 about continuous- time RL:
1. We assume that the transition function, is infinitely differentiable smooth. This is done to ensure that the manifold it forms is also smooth. This might not always hold in practice but the transitions assumption in the construction of diffeomorphism in the definition of a manifold.
be O smooth, meaning t-times differentiable. We use this
2. The set of actions, A, is rectangular and open. This is pretty standard practice for continuous environments. This much less assumption but done to simplify the analysis and avoid any confusion. The only reason we need the open part is to construct a diffeomorphism.
We do not state the entirety of the assumptions made in Section but we provide further intuition for them:
1. We also use assumption that the Jacobian of f is full rank, in (a, 1]. This assumption is needed for applying the implicit function theorem in the proof of Proposition B.2
2. The small orbits assumption is strong one but it is essential to proving Theorem 3.2. This might not hold strictly in practice but it can be argued that holds almost always with probability I, essentially the probability of agent making orbit is almost surely 0. Moreover, as long as there is a positive such that there are no orbits for time < this condition is satisfied. In other words, this condition is akin to saying there is reversibility, of state transitions, in infinitesimally small time intervals. It is naturally true in environments with physical dynamics, e.g., acceleration, velocity displacement cannot be instantancously reversed.
3. The action space restriction condition helps us construct a bijection for proof of Theorem 3.2 Intuitively, the idea is that if two actions have the same local effect how the agents state changes in some state then they can be collapsed into a single action for fixed state s.
B PROOF OF THEOREM 3.2
We first state the implicit function theorem (for more details see theorem 3.3.1 in Chapter by Krantz & Parks (2002) and its proof using the inverse function theorem). In the statement of the implicit function theorem we use F to denote a general function, that satisfy the conditions stated in the theorem.
Theorem B.1. (Implicit Function Theorem) Let F Rditds + Rdd be a continuously differentiable function. Let d2 have coordinates yx,y) fix point (xo (ra,1:--.. Fo,dy Vo, yo.dg with Fo) where 0 E 12 The appropriate Jacobian is defined as:
0F (No.
aF Noo 8w3g
(Fs(x,y)
8F82 (xu
8 Byag
(x)
Gyn
If this Jacobian matrix is invertible then there exists an open set V C Rdd containing Xo such that there exists a uniquely continuous differentiable function h V + Rd such that h(xo)= y and F(x, (xxx)) 0 for all V. Additionally, h is continuously differentiable in the h-image of V.
Using the implicit function theorem we first prove our main result for the exact case where the dimensionality of the effective state space, S., is d, + 1. We do so under additional assumptions in the following proposition.
Proposition B.2. Under the assumptions of Section 2.7 and 3 with the added assumption that the transition function. f. is injective in A x (0, L), for some L, for fixed E S the effective state space,
S. C RRd,, is manifold of d.
14

---

Under review as a conference paper at ICLR 2023
Proof. Given any state E Se we can construct an open neighbourhood that maps to an open subset of I. +1, Since we know that s E Se there exists a state s' E Se such that f(s', O, e) = s for some E A and c E (0, L). Now. for fixed s' consider the map V. A x (c 7, + m) + S. such that (. 1) [(6', a, where O < 1) < 6, such that 6 + 1 < I,, and a E A.
We need to show that this map, ths,, is a diffeomorphism and maps to an open subset of Se. Since the transition function, f. is smooth we have that u),- is also smooth in its range. Also note that f is injective A x (0, L) for fixed s' and therefore maps A (e (E-1); ++))) uniquely to set U. This set U is a subset of S. by definition. Note that U is open because A (E-)),,,++)) is open because a bijection maps open sets open sets. This leads to the conclusion that v A (E - 7. + 11) + C Se is a smooth bijection. Thereby providing us smooth parameterisation from R to U. This means that its inverse, ,/7" exists. Now to show that this inverse differentiable we apply the implicit function theorem.
Let F(x, y. =x (y. t), note that s' is fixed, and as in above the variables are x, y and t. The function F is restricted to the domain such that E Se. y E A and & E (e (e- 7,, + 11). Since the Jacobian of f is full rank, by assumption, the appropriate Jacobiaa, JF.1.. as defined in Theorem B.I invertible and F is also continuously differentiable. Similarly, the condition F(x. y,t) = 0 holds for x 8. h and an open neighhourhood V containing (a. c] E Rdd such that F(h(y, t). y. t) = 0 and h is differentiable. Since h is unique we need to show that h t/a for the domain V (n(Ax (E (6-17,
and t= c. Therefore, by the implicit function theorem there exists a unique function
The proof follows from uniqueness of We have, for the domain Vn (A x (e - -,,77,55 + n)), F( (v, (),,,)) () f(s', whe (y, l) 0. Therefore since h is unique and f is injection for fixed s' we have h = th, which means v is continuously differentiable in its domain By Theorem B.I
We can similarly construct these diffeomorphisms from Rd to U C Se for all E Se. Finally, by Definition 2.1 we have that S. is a da + +1 dimensional manifold.
Now we prove the general case without the assumption of injectivity on f. To do so we construct an surjection f' S x A' x R+ + S where A' C Rd is an open set such that d' < dg. We state the two additional assumptions made in Section 3] The first one being that for any policy E II we do not have any critical points or loops in the differential field defined by Jr. Formally, we have the following conditions (=))) F 0 for all 8 E S and for a fixed policy the solution for the equation ss = s, if it exists, is unique in f. The second one is that if two actions have the same outcome at one state then they have the same outcome at all states. More formally, there exists an r > 0 such that for all 0 < e < r, if we have f(s, .1:E) f(s, 02 t) for some fixed d1 02 E A and any one E S then it holds for all S.
Under the assumptions of Section 2.1 and the ones stated above restate the main theorem below before presenting the proof.
Theorem 3.2. The set of effective states, Se, is smooth manifold of dimensionality most d, + 1.
Proof. With assumption 3 for every state we have a restriction A', that essentially turns f(s I) into bijection for every which is a restriction of I's domain to A', x (0, T) for a fixed 8'. Now for a fixed 8, we can construct an open neighbourhood using ب. : A'. x (E ( 77. + 17) constructed as in Proposition B.2 (a, t) = f'(s', a, t) such that ( = s for some E A's' and e > 17 > 0 and also e + 1 < T. where 7 is as in assumption 2. Now we know that this is a bijection from assumptions Section 3.1 as long as we choose and 1) small enough to satisfy the requirements for Proposition B.2 The restricted set A'. ensures that each action, for fixed time and s, maps to a unique state. Further, the assumption on no orbits ensures that for a fixed action a E A', there is unique solution to f(s', a, = s" for a fixed a, s' and s" in t. Therefore, we constructed a full rank smooth bijection ahs which maps open set A' x (c - 1, + 17) to an open subset of S. for every now apply the same arguments of Proposition B.2 to argue that there exists a diffcomorphism vha form an open set in A', x (e 77, 6 + 1) to open set in S... This holds true for all
A'c Now consider a function s' : S. x A', (e - 7, 7)
and 3 in
Se. We can
15

---

Under review as a conference paper at ICLR 2023
Table 1: Transition Variances for MuJoCo Environments
Environment
std std(S.)
Cheetah
9.4 X 10
Walker2D
2.7 x 10-16
Reacher
7.8 10-16
Swimmer
1.04 10-15
Finally, we conclude by stating that the dimensionality the manifold is equal to the of dim((A' x 7/5 + v)) + which is less than equal d, + 1.
Finally, we prove Corollary 3.3. The proof is fairly straightforward following Definition 2.2 and Definition 3.1. Since the manifold S. is defined as the union of all continuous curves from all possible smooth policies we can find curve such that for any E Se and E A for some policy TTs) in the neighborhood of S. n (s. <) where (ss, <) is a Euclidean ball of radius c > 0 centered s. This would mean that for this policy, starting at 8o, there is curve such that for some time and some interval // > O we have y()) = a and therefore +(()) = e where u E TyS.. This vector v can now be uniquely mapped to the action and therefore can be labeled va in Corollary 3.3. We also note that this vector is unique independent of the choice E or 17.
C EMPIRICAL VALIDATION OF OUR ASSUMPTIONS
We validate of assumptions. We first demonstrate that the MuJoCo environments, for which we present results in Section are deterministic for all practical intents and purposes. Second, we show that the Jacobian of the tranistion function with respect to the action for fixed time and state full rank for all of these environments, thereby suggesting that assumption 1 in Section 3 is "partially satisfied".
C. DETERMINISTIC TRANSITIONS IN MuJoCo ENVIRONMENTS
To estimate the stochasticity of the transitions in the environments we estimate the transition standard deviation:
std(S.) E,--u[s))a-_[A11 [(s' B(s')) S -',, A,
where [r[-) represents the uniform distribution over set. This captures how much variance is in the transition dynamics. We estimate this value by randomly and uniformly sampling states, from an agents trajectories over the learning process. We then obtain randomly and uniformly sample actions, from the set of actions A. Then for fixed (s, we take observe the one step transition that the environment returns: s', 100 times. For fixed s we sample 30 such actions a. Therefore, estimate the quantity std(S.) using approximately 6 million samples, for each environment. The results, divided by the standard deviation of states, are presented in Table suggest that for all practical intents and purposes these environments are deterministic.
C.2 FULL RANK JACOBIAN ASSUMPTION
To validate assumption from Section we empirically estimate the Jacobian. Our assumption states that the function (s, a, full rank Ja, for all s S. Since we are working in practical framework the value of t is fixed to be 1, meaning the environment simulates the application of action for single unit of time and returns the next state s'. Therefore, we are only able estimate the Jacobian of f(,, 1) for the variable a, see that is fixed to 1. To do so, we sample ~ [[S..,, as described in the previous section. We then sample a ~ U[AA]. as described in the previous section. Finally we estimate the Jacobian for the variable and the function f(s, a, 1), see Theorem B. for definition of the Jacobian. We do so using the scipy function approx_ fprrmmeee Finally, once we have the d, x dg Jacobian matrix est estiin immat eeatterrnkka rank. We tabulate all the results in Table 2 This procedure comes as close as we can to validating assumption in Section
estimate the rank of this matrix using the scipy function
16

---

Under review as a conference paper at ICLR 2023
Table 2: Transition Variances for MuJoCo Environments
Environment
da
Rank
Cheetah
6
6
Walker2I)
6
6
Reacher
2
2
Swimmer
2
2
(a)
(b)
(c)
(d)
Figure 7: Frames from the Atari game Seaquest. We present 4 frames that are temporally successive from left to right.
D CONNECTION TO DISCRETE ACTION ENVIRONMENTS
One of the most popular RL environments with discrete states and actions are the Atari games from the arcade learning environment (Bellemare et al.] (013). Mnih et al. (2013) provided the first result in learning policies with only high-dimensional images the input DNN. Atari games deterministic given a fixed policy (Hausknecht & Stone 2015). In case of images, there are two considerations when it comes the underlying structure of data:
1. Underlying structure of images: the states, which are frames from video game, in Atari games have low dimensional underlying structure. Even though the frames are 210 x 160 pixel images there is low dimensional underlying structure them. For example, consider the frames from the Atari game Seaquest in Figure 77] These frames can be identified uniquely with far fewer variables such as: position and heading of the submarine, position and heading of the sharks, oxygen capacity, the scores, the number of lives, position of the bullet and the score.
2. Underlying structure from the transitions: Since know that the agent can only change its state and observation by taking actions limits the states available to it. In the Seaquest example, we know that the agent can only access some subset of all the possible configurations of the low-dimensional manifold of Seaquest images. Meaning only a subset of configurations of shark positions, submarine positions etc. are realizable.
We further illustrate these ideas in Figure 8 These two factors combined together impose low- dimensional structure on the state manifold. Development such a theoretical model, which accurately captures these restrictions dual structures, is beyond the scope of current work where we primarily deal with the structure imposed by the transitions.
E DIMENSIONALITY ESTIMATION BY FACCO ET AL. (2017)
We describe the algorithm for dimensionality estimation in context of sampled data from the state manifold S.. Let the dataset be randomly sampled points from manifold S, embedded in Idd, N
denoted by D {s: 1=1- For point from the dataset D let (ri,1, Ta,2, r.... be sorted list of distances of other points the dataset from 5; and they set To = 0. Then the ratio of the two nearest neighbors is 14; r where ri, is the distance to the nearest neighbor in D of 8; and r..2 is the distance to the second nearest neighbor. Facco et al. (2017) show that the logarithm of the probability distribution function of the ratio of the distances two nearest neighbors is distributed inversely proportional to the degree of the intrinsic dimension of the data and we follow their algorithm for estimating the intrinsic dimensionality. We describe the methodology provided by Facco et al.
17

---

Under review as a conference paper at ICLR 2023
Figure 8: The structure imposed by transitions is illustrated above. Suppose the agent starts at the red state. the agent were to execute the down action it gets to the blue state. There are three "directions" in which the agent (and the submarine) can transition. If the agent chooses execute the action left it gets to the yellow state, similarly green state on right and white state on down. The agent traverses along this surface of admissible images, which are constrained by the underlying structure of the images, given that they are from the Atari game Seaquest and from the set of possible transitions along this manifold. Here discrete actions are represented by continuous curves, in black, on a manifold.
(2177) in context of data sampled by an RL agent from a manifold. Without loss of generality, we N' assume that {s; /==1 1= are in the ascending order of r,. We then fit a line going through the origin for (log(44), log( (1
/ The slope of this line is then the empirical estimate of dim(S.). We refer the reader to the supplementary material provided by Facco et al. (2017) for the theoretical justification of this estimation technique. The step by step algorithm is restated below.
1. Compute 1 and r2, for all data points i.
2. Compute the ratio the two nearest neighbors μ4: r2,2/Ti.1-
3. Without loss of generality, given that all the points in the dataset are sorted in ascending order of μ4 the empirical measure of edf is i/N.
4. We then get the dataset Dicasity {log(u)), log(1 i/N)} through which a straight line passing through the origin fit.
The slope of the line fitted as above is then the estimate of the dimensionality of the manifold.
F DDPG BACKGROUND
agent trained with the DDPG algorithm learns in the discrete time but with continuous states and actions. With abuse of notation, discrete time and continuous state and action MDP is defined by the tuple M = A, P, fr. so, A), where A, 5o and fr are the state space, action space, start state and reward function as above. The transition function P S x A x S is the transition probability function, such that P(s, a, s') = Pr(S++1 = s'|S, = s, Ar = a). is the probability of the agent transitioning from to s' upon the application of action for unit time. The policy, in this setting. is stochastic, meaning defines a probility distribution over the set of actions such that (s, a) Pr(A, alS, s). The discount factor is also discrete in this setting such that
18

---

Under review as a conference paper at ICLR 2023
analogous state value function is defined as
(8t) Esg B
Σ (e
which the expected discounted return given that the agent takes action according to the policy T, transitions according to the discrete dynamies P and 8; is the state the agent is at time Note that this is a discrete version of the value function defined in Equation 2 The objective then is to maximise (TT) rx (so). One abstraction central learning this setting is that of the state-action value function Q" S x A R, for policy x, is defined by:
T
Q" = Bai,arnm
X frs,,, Σ
which is the expected discounted return given that the agent takes action a, at state 8t and then follows policy и for its decision making. An agent, trained using the DDPG algorithm, parametrises the policy and value functions with two deep neural networks. The policy, π S + A. is parameterised by DNN with parameters HT and the action value function, S x A - R,is also parameterised by DNN with ReLU activation with parameters 8², Although, the policy has an additive noise, modeled by an Ornstein-Uhlenbeck process (Uhlenbeck & Ornstein, 1930), for exploration thereby making stochastic. Lillicrap et al. (2016) optimise the parameters of the Q function, 02, by optimizing for the loss
1 N
LQ =
W - 04; ((;)));)0))))
where is the target value set as = + 10( (8 0T) 02). The algorithm updates the parameters & by 0 <- 82 + aq oe Lo, where Li is defined as in Equation 5 The gradient of the policy parameters is defined as
V(r N
(c) ( (o (=)
(6)
and the parameters 0 are updated in the direction of increasing this objective.
G DDPG MODIFIED ARCHITECTURE COMPARISON
We provide the comparison between single hidden layer network and multiple hidden layer network because our results in section 4 are for single hidden layer. The same architecture is used by Lillicrap) al. (2016) for the policy and value function DNNs which two hidden layers of width 300
and 400 with ReLU activation. Here we provide the comparison to single hidden layer width 400 and MUP Yang Hu/ 2020) with GELU activation for the architecture used by Lillicrap et al. (2016). We provide this comparison in Figure 9 for both the architectures. All results are averaged over 6 different seeds. We use a Py Porch based implementation for DDPG with modifications for MUP parametrisation and the of GELU units. The base implementation of the DDPG algorithm can be found here: https::/gituu..com//// The hyperparameters are in the base implementation.
and note that the performance remains comparable
(a) Walker2D
(b) Cheetah
(c) Reacher
(d) Swimmer
Figure 9: Comparison of single hidden layer (blue) and multiple hidden layer (red) architectures for DNNs.
19

---

Under review as a conference paper at ICLR 2023
H SAMPLING STRATEGY AND GEODESIC DISTANCE ESTIMATION
We use graphs as a discrete abstraction of manifolds to sample efficiently whilst preserving local properties of the sampled nodes. In many representation learning aprroaches the main objective is to learn to map individual data points to dense vectors in a low dimensional space via stochastic gradient descent (Mikolov al. 2013 Perozzi et al. 2014] Grover & Leskovee 2016). One of the primary objectives is to preserve relationships between data points. Graphs, with individual data points mapping to nodes and the relationships between them ecnoded as the edges, are used as an abstraction to represent the whole dataset with all the charecteristic relationships. In our case we seek to preserve local isometry in the learnt mapping, v. Meaning, given a point & E D C S, we would like to preserve the geodesic distance between and the points in its "neighbourhood" upon being mapped to a lower dimensional representation.
We first define the graph that is discrete representation the state manifold, Se, and state the above objective more formally. An undirected graph 9 = (V, E) consists of nodes and edges, where V = [u are the individual nodes and an edge e = (e;, 25) E E implies that there is a an edge between the nodes vn and Uj for some i, Since the graph, G, is undirected (02,2)) E E = (1), Us) E E. Given a node, Vi, the one-hop neighborhood of this node is defined as all the nodes U/ E V such that (vy, vy)) E. Now we define graph, deonted by GD = (D) that forms discrete abstraction over the manifold S. using the dataset D. We first set all the ndoes to be the states in our dataset, D. A datapoint's s one hop neighbourhood as the km points nearest by the Euclidean distance in the dataset D. This means that for every state s; E 'D we have ki edges to Amn distinct nearest points in D. This is a kim--nean -neaest neighbors construction of the graph from a dataset D, sampled from a manifold (Yan) eet al.] 2007 Dong et al. (001T) We augment the edges of this graph Go with an additional set of edge attributes, the Euclidean distance between two points (8;; 85) E ED i.e. ||8, 8,llg. Therefore, the augmented graph now becomes GD (D, ED. AD) where Ap is an ordered set with the Euclidean distances between the two nodes in ED.
For most practical applications, the number nodes of this graph GD is in thousands and consequently the number of edges are km times the number of nodes. Since we learn the mapping ib using SGD we can sample batches of state pairs and the distances between them. To do so we randomly sample a subset of states from D. We then construct a subgraph which consists of nodes 42 hops away from these randomly subsampled points. We then use this randomly sampled subgraph to compute the pairwise distance between them using breadth first search. We denote this procedure by the function Random-K- -eaeen-------bboaaee Hoo-Subgraphh(S (GD. kn). Since this randomly sub-sampled graph is much smaller in size whilst preserving the distances we postulate that the distance thus obtained between pairs of states, as the sum of the distance attributes of the shortest paths, is a good approximation of the geodesic distance ds.- We remind the reader that this geodesic distance is needed for the manifold loss introduced in Equation 3 This three step procedure: 1) sampling data on the manifold, 2) constructing knn -nn-esssstt neighbors graph from this data, and 3) randomly sampling kn.hhop subgraphs, is illustrated with an example of the Swiss roll manifold in Figure I0
I ALGORITHMIC DETAILS
An RL agent, trained using the DDPG algorihm, collects data from trajectories as tuples: (s, a, s', r) where 8 is the state at which the agent takes an action a and transitions to state s' and obtains the reward The algorithm stores a buffer of these tuples and therefore we have access to the set of available states sampled from the state manifold S.. As described in the previous section, our algorithms requires a dataset sampled from the manifold to estimate the manifold loss in Equation 3. Therefore our algorithm subsamples the states from this buffer to obtain the aforementioned dataset, D, at every episode. Then our algorithm (Algorithm [ uses this dataset to obtain a sample of pairs of states and geodesic distances between these states, as described in the previous section, to perform SGD on the parameters of the mapping to the low-dimensional space, 0g. on the loss Lgg All the results have the same values for km 6, ki. = 4 and we use batch size of 128 for the subsampling the graphs as described in Appendix forr the function Random-K KK K-Hop-Sun Suob-uhra. These were obtained after manual tuning. The other parameters, are set to the defualt ones for DDPG T = 0.01, x= = 0.99, ag = 0.001 and the replay buffer size is 10 We provide details for running the code in Appendix O
20

---

Under review as a conference paper at ICLR 2023
10
10
-3
-3
20
10
5
-10
(a) First we obtain set of points sampled from (b) In the second step, edges are drawn between manifold.
the km nearest points.
10
-5
10
10
-10
(c) Finally, nodes and edges (in red) are subsam- pled based on the hú, 2 hop neighbourhood of randomly sampled nodes.
Figure 10: The three step sub-sam sampling for estimating geodesic distances and batched learning in SGD illustrated with the example of the Swiss-roll manifold. We use the scikit. Wilt-learm rn package (Pedregosa al.] 2011) for sampling the data, torch torch-geomet.rl package for generating Ang nearest neighbors graph and also for obtaining the -no subgraphs (Fey & Lenssen 2019).
21

---

Under review as a conference paper at ICLR 2023
Algorithm 1 DDPG with Manifold Representaion Learning
T, Q parameterised by 0", 04 respectively. 8*, 8- Pr(0)
Initialize the parameters of the target 0" 88' +- ,0,0000 A
Initialise replay buffer B
for Episode to Max Episodes do
Initialise the Ornstein- Omstein-Uhllnbeck process for exploration noise: X
for Time step t, to do
Set action de x (8t; + Xt-
Execute action d₃ and observe Tt, 8r+1 storing it in
Sample minibatch of N transitions (84, Ug Taa 8141) from B
Compute the loss LQ and update the parameters 0
Construct the dataset D from all the states s; from the sampled N tuples Construct the graph GD using -neaest neighbors
Subsample graph J/ (D', E
Kandom-K-Hoo K--Ho--Sugrrahhe GD, kn)
Obtain the set of state pairs and geodesic distances, from GbD, to calculate Li Update the policy as Equation
tTh + 8" - ax Vgr.(( - ap o Lsb
Update the target networks:
00' +708 + (1 T)
0x -- (1 7)
end for
end for
0.05e-5
0.1e-5
1200
0.25e-5
0.5e-5
1000
1.0e-5
1.5e-5
Manifoid Loss
3.0e-5
800
6.0e-5
600
400
200
200
400
600
800
Epoch
Figure We observe the effect of increasing the hyperparameter az on the discounted return the Walker2D environment.
ADDITIONAL EXPERIMENTS AND ABLATION STUDIES
All results reported here that are reported are an average over different seeds. For but the Reacher environment
the manifold loss decrease training progresses, expected. Meaning the agent is able to learn a low-dimensional isometric representation, v, as well as a policy that operates on this low low-dimensional input. We observe that for the Reacher environment Algorithm is unable
22

---

Under review as a conference paper at ICLR 2023
8000
7000
6000
Mean Eval Returns
5000
4000
3000
2000
10
1000
200
400
600
800
Epoch
Figure 12: The discounted return for varying the width of the bottleneck layer for the Cheetah domain with d, + 7. We the performance peaks width
0.020
-80
Mertildd Lawa
-100
0.016
0012
-140
200
400
600
800
200
4000
600
800
Ezoch
Epoch
Mean Evaluation Returns
(b) Manifold Loss
Figure 13: We observe that the manifold loss decreases as we increase the width of the bottleneck layer and the performance improves. All the hyper-parmaeters are the same as in Appendix
to simulataneously learn a low dimensional mapping and policy. This is a true despite searching across all the hyperparameters.
We present the ablation study over the hyperparameter in Figure II increasing the learning rate, agg, improves performance the agent, case of walker but begins to detireorate after certain point, at = 1.5
This suggests that further
Finally, we demonstrate the effects of changing the width of the bottleneck layer on the Cheetah domain in Figure Most importantly, we observe that the performance peaks at width equal to 8, with both and have similar returns. From the results for dimensionality estimation, Figure 3 we know that the estimate lies between 7 and 8. This suggests that in case of Cheetah there is optimal isometric "compression" that allows the agent to perform optimally and better than the baseline. This furthers argument that RL agent utilising the underlying structure. The error margins are omitted for the clarity of exposition since there multiple curves the same graph.
leam efficiently this low dimensional manifold by
23

---

Under review as a conference paper at ICLR 2023
K EXPLAINING FAILURE IN REACHER ENVIRONMENT
Here we explain the failure case of the Reacher environment. As noted in Section 5.2 isometry is stronger condition that mere diffeomorphism. The Nash embedding theorem states that the an m-dimensional C² manifold can be embedded isometrically into a Euclidean space of dimensionality most m(m + 1) or (3m 12)/2 Nash 1954), therefore the embedding dimension required for learning an isometric embedding for Se might be greater than d, + 1. For example, a circle which 1D manifold in 2D Euclidean space cannot be isometrically embedded into ID. Therefore, the objective we train on is stronger than learning coordinate chart. We hypothesize that for the reacher environment the agent is unable to learn this isometry. As reported in Figure [3], as we increase the width, and therefore the embedding dimension, the manifold loss decreases and the agents performance as measured by mean discounted return improves. As we increase the width to the performance is par with the baseline approach. The error margins omitted for the clarity of exposition since there are multiple curves on the same graph.
L LEARNING VIA LOW-DIMENSIONAL REPRESENTATION FOR SOFT ACTOR CRITIC
The soft actor critic (SAC) algorithm (Haamoja a al. 2018) provides method for learning policy in a more stable manner compared to previous algorithms like DDPG (Lillicrap et al.) (0166). TRPO (Bach & Blei 2015), and PPO (Schulman et al. 2017b).
L.I BACKGROUND SOFT ACTOR CRITIC
The goal of the SAC algorithm is to train an RL agent acting in the MDP M = (S,A, 80, X), which is as described in Appendix (F)
The SAC agent optimises for maximising the modified objective:
J(0
Σ
E Rerau ()),)))))) + H(T(( 86; 07))),
where H term is the entropy of the policy æ. This additional entropy term improves exploration (Schulman et al. 72l Haarnoja et al. (017). Haarnoja et al. (2018) optimise this objective by learning 4 DNNs: the (soft) state value function V(s; 0' ). instances of the (soft) state-action
value function:
((s1, ax; 8?²) where i E {1,2], and a tractable policy a(s,, ae; (7). To do so they maintain dataset D os citteaatioo-reward-ttaaa tuples: D = {(s,, as $.)). The soft value function is trained to minimize the following squared residual error,
Jv (v EgnD (v(s; 0 - Faw [Q(s, a; 09) log az as)))))))))
(7)
where the minimum of the values from the two value functions Qi is taken empirically estimate this expectation. The soft Q-function parameters can be trained to minimize the soft Bellman residual
1
JQ(0 E.
- -
(8)
where 0 are the parameters the target value function, which are updated at a slower rate compared to the parameters @V, as is also done in Algorithm The policy parameters are learned by minimizing the expected diveeieeeee
J(8T) = EEND DKL (0)) xxp(Q(s.:;09) Zoo (s)
)
(9)
where Zo4 (s) normalizes the distribution. In addition to this we add the manifold loss as described in Equation to the policy objective. In keeping with the notation above, the learning rates for the functions V. Q. π and v are agg ar and ag respectively. Algorithm details how the agent performs this modified learning.
24

---

Under review as a conference paper at ICLR 2023
Algorithm 2 SAC with Manifold Representation Learning
Ti Qi V parameterised by 0 0 ov respectively. 0r, 00, or Pr(O)
Initialize the parameters of the target @V + 0V. Initialise empty dataset ! for Episode to Max Episodes do
for Time step t, 0 to 'I' do Sample action at (ss; 0-).
Observe successive state 8t+1 P(sc+1l8t,0) at).
Append to dataset D +- D U {8L, at 81+1)].
end for
Construct the graph GD using -earest neighbors Subsample graph P (D', ED: A ) Bandonn-KHHop-SSbbrrhhff., ks.)
Obtain the of state pairs and geodesic distances, from D calculate L for Gradient steps do Update the value function parameters:
0¹ -000 - av Vor
where Jy in Equation Update the Q-function parameters:
0 aq VJ (1,2),
where Jo is as in Equation Update the policy following the objective in Equation g
g
8 - Tm + a V (( a L..
Update the value function target network:
gv + T + (1 7)0%
end for end for
25

---

Under review as a conference paper at ICLR 2023
Our Architecture
SAC Archilecture
900 Neurons
300 Neurans
300 Neurons
300 Neurons
Figure 14: Our architecture for SAC, which similar to the architecture in Figure
Maan Seal Bature
SAC
Taturns
100
54C
6300
4320
-75
2000
250
503
750
1000
250
500
750
1000
250
530
750
1000
(a)
Walker2D
(b) Cheetah
(c) Reacher
(d) Swimmer
Figure 15: For all the environments we use 0gp 7.5 x 10-5 in comparison to O'r = 3 x 10-1, and the rest of the hyper-parameters are the same as reported by |Haarnja et al. (2018), over random seeds.
L.2
DETAILS
The discount factor, A, is set to 0.99. The learning rates for the Q-functions, aq. the V-function, ov, and the policy, Opi are set to x 10-1 The update parameter of the value function, т, set to x 10-3 33 The batch size is 256. The learning rate for the manifold loss, ag is set to 7.5 x 10-5, All the parameters and for learning the manifold representation are the same described in Appendix []
L.3 EXPERIMENTAL RESULTS
We provide the results for our architecture (as described in Figure [4). with manifold learning, in comparison to "vanilla" SAC for the four algorithms. These results are for architecture and algorithm similar to described Section 5.2 except with SAC algorithm as opposed to DDPG. All the mean discounted rewards are reported in Figure and the corresponding manifold loss is reported in Figure 16 we observe that our agent performs at par with the baseline in case of Walker2D, Cheetah and Reacher and slightly worse the Swimmer environment. This result was obtained without an extensive hyperparameter tuning by varying the value of ag very small range.
M COMPARISON WITHOUT MANIFOLD LOsS
We provide another comparison where we compare the same bottleneck architecture for the policy network with and without the manifold loss. This demonstrates the efficacy of the manifold loss
0.20
8.02
0.3
0.15
Mertoill
3035
0.2
5005
3034
9032
0.002
250
500 Eaoch
1000
0:25
270
750
1000
500
250
1000
Eooch
bauc
Booch
(a) Walker2D
(b) Cheetah
(c) Reacher
(d) Swimmer
Figure 16: We report the manifold loss as above. Except Cheetah every other domain behaves as expected.
26

---

Under review as a conference paper at ICLR 2023
Noon Exsl seture
1500
Ours DORG
Wein Pval Reture
6030
DUrG
Outs
Foe Seeume
130
2000
6020
520
4030
100
2020
150
DDPQ
-500
250
500 spach
0
250
500
750
250
500
750
0
250
500
750
Esoch
Epech
Spach
(a) Walker2D
(b) Cheetah
(c) Reacher
(d) Swimmer
Figure 17: We compare the scenario where we use the same architecture for DDPG as illustrated on the left side of Figure 4 with bottleneck layer of width d, 1, with, labeled "DDPG", and without, labeled "Ours", the manifold learning loss. All the results are averaged over 6 random seeds.
4002
Mest Exa Pouu
10000
2000
Cun
60
SAC
-50
A0
2003
-75
Near
230
750
1000
250
520 Spech
750
1000
500 750
1000
250
500
1000
Epach
Spech
Spech
(a) Walker2D
(b) Cheetah
(c) Reacher
(d) Swimmer
Figure 18: We compare using the same architecture, as in Figure 14 for the two implentations the SAC algorithm with (labeled "Ours") and without manifold loss (labeled "SAC").
described in Equation 3, In Figure 17 we observe that for all the environments our algorithm performs better except for the Reacher environment where both the algorithms perform sub-optimally to the wide network baseline. We attribute the higher variance to changing representation of yt. Ansuini et al. (2019) showed that DNNs implicitly learn low-dimensional manifold structure at varying depths when trained with SGD in a supervised manner. Their results for various popular image classification models like ResNet (He et al. (016)) AlexNet (Krizhevsky et al. 2012) and VGG-16 (Simonyan & Zisserman 2015). This speaks to the efficacy our manifold loss Lg that the addition of this loss improves performance over the implicit low-dimensional manifold representation learnt by a Deep Rel.u network with bottleneck layer. We report a similar comparison between SAC algorithms with the same DNN architecture, in Figure 14] with and without the manifold loss. We observe that with the manifold loss the performance is better for Walker2D, Swimmer and Cheetah environments. For the Reacher environment we observe that SAC algorithm ends up finding the optimal policy with low variance and with far fewer episodes in both cases.
N COMPUTE REQUIREMENTS
For each one of our experiments we perform 6 different runs with varying seeds to obtain the results. Since experiments in (..tion[5.Ioor only require trajectories we reuse the trajectories sampled form ReLU and GELU comparison experiments in Appendix E. For all the experiments in Appendix E we required approximately 180 hours of processing time on NVIDIA GeForce RTX 3090 GPUs which have 24 GB of RAM. We also run two runs of the DDPG algorithm simultancously on each GPU, owing to the 24 GB RAM availability per GPU, which cuts our run time into half.
O STEPS FOR RUNNING THE CODE AND GPU HOURS
All code is in python and was tested for python 3.6. We use the pytorch library for the ease of defining and training DNNs (Paszke et al. 2019) and pytorch geometric p (Fey & Lenssen 2019) for all graph operations.
We provide the implementation of DDPG using GELU units in the code/rlkizg folder of supplementary material To install the environment recommend installation using the setupy.pt file present in the folder. For running the experiment, there are multiple files in the code/ rlki teti/een tetp//xxxlle////eeeeemm httpmppples/mooth_dhaaeoo www.otooooggo/ooooooooooo /ddp///dpp...//// Py, each uses different architec-
27

---

Under review as a conference paper at ICLR 2023
ture and environment details of which can be found within the file. To execute the code, e.g. for adpg_arch__ 2. py, rum the following command from the rlkit folder:
python examples/ ssamlea/smooth, http:/stoooohhoeeee wwpp//dpg.ppppppppoopppppggggg.p arch 2 py 115 gelu
where "115" is the seed and "gelu" argument means the DNN thus instantiated uses gelu activa- tions. Our code samples trajectories and the location of the folder be specified in the file path_collector..
The code for dimensionality estimation using neighborhood data is fairly simple and only requires version 1.1.1 of the python package scikit-lears (Pedregosa et al. 201T). It is provided in code/da aaprocess /ggeg// hg/dimmeste mate. The main file is dim-est i te -y. It expects a pickle file which is an array of all states sampled for an agent. The code for cleaning the samples, from runs of the DDPG code as described above, and acquiring them in the required format can be found in the file run_data_ htt.caat_procsssinggg
The code for the simultaneous dimensionality reduction and policy learning can be found in the folder podee/ikit/exxmppeee http://oooodeeaaanngg///o/////// hearning/ddpss implementation for graph creation and sampling procedure using pytorch geometric is in the file code//lkit/rikie httii///o///////teeeteeeeee// http:/mmnnfoommmmmdddmod//d various architectures used for the policy network can be found in the folder
The
The
Finally, the code for obtaining the illustrations in Figure are the folder http://kkii/llusstaatttiot
For the results and all its ablations in Section 5.2 we ran multiple instances of the modified DDPG algorithm and the baseline DDPG algorithms for 6 seeds each for 1000 epochs on cloud instances with Nvidia 3090 Ti GPUs with 24 GB memory, and CPUs with 8 cores and 16 GB RAM. Each run takes about 3 hours each. This means utilised about 700 GPU hours, including hyperparameter tuning and auxiliary experiments presented in the Appendix. The results Section were obtained from the trajectories sampled from various DDPG runs and took about 20 CPU hours on an 8 core machine.
28


Reviews:
[{"review_id": "pNl_f0CpKx", "reviewer": "Reviewer_RjLF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a theoretical analysis to understand the intrinsic dimension for deterministic continuous-time State and action space problems. This method is motivated by understanding the representations required to train optimal policies in these types of spaces will help us better understand the structure of networks that can be used to solve these types of tasks. The method performs some theoretical analysis to indicate that a diffiomorphism can be found between the continuous time optimization problems such that the intrinsic space is only the size of the action space + 1. Two different experiments are shown to try and empirically justify the results that analyze the reproducibility of the theoretical results in some of the mujocu simulated robotic control environments.", "summary_of_the_review": "The work forms a theoretical analysis to indicate the intrinsic dimension of a continuous time control system. The motivation and scope of the contribution need more details before the score can be raised.\n\n---- Updated score\nAfter more clarity on the method and a few updates to the framework.", "strength_and_weaknesses": "pros\n- The first pro for this method is that it does appear to have a rather helpful proof in showing how the dimensions for a control problem are also heavily connected to the action space for the problem.\n- The analysis also includes two different types of empirical experiments to better understand how close do the theoretical analysis is to a practical comparison.\n\ncons\n- The analysis, including the theory in the paper, is limited to a very small subset of possible planning problems. While the analysis may be correct, the fact that it only works on continuous state and action problems that are deterministic might say more about the limited applicability of the intrinsic dimensionality findings.\n- This is also reflected in some of the experiments that analysis or it would be helpful to include analysis where the state space is image-based rather than the already dense information from the robot poses.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "my_BJuaB3d", "reviewer": "Reviewer_HpSc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the manifold of effective (or reachable) states in MDP with continous states and continuous actions. Based on the continuous-time modeling, the authors present their main theorical contribution, i.e., the theory that proves the set of effective states is a smooth manifold of dimensionality at most the dimensionality of action space plus 1. Then, the authors provide empirical validation from two aspects: 1) the empirical estimation of the effective state manifold, and 2) the performance evaluation of a DDPG variant equipped a low-dimensional manifold representation by learning from the knn-graph estimated geodesic distance.", "summary_of_the_review": "According to my detailed review above, I think this paper is below the acceptance threshold mainly due to my concerns on the theory and the empirical results (see concrete content above).\n\nI am willing to raise my rating if my concerns are well addressed. \n", "strength_and_weaknesses": "$\\textbf{Strengths:}$\n+ I appreciate the authors’ efforts in study the manifold of effective states in MDP. Personally, I am interested in it and I think it is fundamental to efficient RL in complex problems.\n+ The writing and presentation of the proposed method is almost clear. The organization of this paper is good.\n+ The theoretical results and the empirical validation match well to some extent.\n\n&nbsp;\n\n$\\textbf{Weaknesses (and Questions): }$\n\nThe main theoretical contribution is Theorem 3.2. By referring to Appendix B, I am confused on the proof of Proposition B.2. Concretely, since $\\phi_{s^{\\prime}}: A \\times (\\epsilon - \\eta, \\epsilon + \\eta) \\rightarrow U \\in S_{e}$, the inverse $\\phi_{s^{\\prime}}^{-1}$ should be a mapping from state to action and time interval, right? This turns to be confusing when the authors derive the equivalence between $h$ and $\\phi_{s^{\\prime}}^{-1}$. Do I misunderstand some part?\n\n\n\n&nbsp;\n\n\nMy other concerns are on the empirical validation.\n\n\n\nFor the empirical dimensionality estimation, I feel a gap between theory and practice, i.e., the policies (DDPG policies) used to sample state data are poor in performance in MuJoCo and thus should only cover a very small part of all possible smooth policies.\n\nTherefore,  I question the validity and generality of the results of estimated manifold dimensionality.\n\nIn addition, what is the exact number of the size $n$ of the sampled states used to estimate the dimensionality in Section 5.1? I am also curious about how different numbers of $n$ affect the dimensionality estimated.\n\n&nbsp;\n\nSimilarly, personally, I am not satisfied with the results in Section 5.2, because I am very familiar with the performance of different representative DRL algorithms in MuJoCo and I do not think DDPG is a good baseline algorithm in MuJoCo.\n\nI recommend the authors to conduct the expriments based on TD3 or SAC which will be much more convincing, at least to me.\n\nFor the comparison in Figure 5, do the authors have the results of a baseline that keeps the $d_a + 1$ bottleneck structure while not uses $L_{\\phi}$ (i.e., train as the original DDPG). I think this baseline will help a lot in understanding the efficacy of $L_{\\phi}$.\n\n&nbsp;\n\n\nFor some additional discussion, I wonder the authors’ opinion on how this work, i.e., the manifold of effective state space, relates or differs to recent works on studying low-rank state (representation) space in DRL, e.g., [1,2].\n\n&nbsp;\n\n\nReference:\n\n[1] Yuzhe Yang, Guo Zhang, Zhi Xu, Dina Katabi. Harnessing Structures for Value-Based Planning and Reinforcement Learning. ICLR 2020\n\n[2] Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, Sergey Levine. Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning. ICLR 2021\n\n&nbsp;\n\nMinors:\n- The reference in the second paragraph in Section 5.2 is broken.\n- The figures are in low fidelity. The font size of the texts in the figures are too small to read.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QDnxYI0sst", "reviewer": "Reviewer_R4ss", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper shows that, under certain conditions, the set of reachable states is a smooth manifold with dimension at most the dimension of the action plus one. The paper then proposes a DNN architecture with a bottleneck and show its competitive performance in numerical experiments.", "summary_of_the_review": "The analysis on the geometry of effective state space is interesting, but the theoretical results only works for a very limited class of problems. Numerical experiments may be a bit limited for validating the low-dimensional effective state space property.", "strength_and_weaknesses": "Strength\n- The paper provides an interesting analysis on the geometry of the state space. For a deterministic environment satisfying the three conditions stated in the paper, the effective/reachable state space is shown to be a low-dimension manifold whose dimension is determined by the dimension of the action space.\n\n- The upper bound on the effective dimension is shown to be close to the empirical estimate, and this effective dimension could potentially help the architecture design of value and policy networks in RL algorithms.\n\nWeaknesses\n- The analysis only works for deterministic environments. In fact, with noise in the transition dynamics, the dimension of the effective state space will be the entire state space for a general problem. There is a paragraph briefly discussing the possible extension to stochastic environments with additive Gaussian noise, but no actually analysis for the stochastic case.\n\n- The three assumptions seem pretty restrictive. They are essential for the proof as discussed in Appendix A, but they basically restrict the model to be rather simple and not likely to hold in interesting problems like those MuJoCo environments. Does any of the MuJoCo environment (no randomness) satisfy these assumptions?\n\n- In numerical experiments, only one architecture with a bottleneck of $d_a + 1$ is considered. If the goal is to show that $d_a + 1$ is the effective dimension and is sufficient for state representation in a RL algorithm, one may need to try multiple architectures with different bottleneck dimensions and compare their performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4xCnu4f09Y", "reviewer": "Reviewer_cmmy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Reinforcement learning has led to empirical success in complex tasks with continuous state-action spaces in the contexts of game-based and robotics simulations.  Theoretically current tools seem to suggest poor performance - due to the \"exponential\" scale of the state space representation, even when using strong function approximators like a neural network which is used in practice.  To some extent - these empirical successes have been justified due to the so-called \"manifold hypothesis\" - that most high-dimensional real-world datasets actually lie on low-dimensional manifolds.  For example, the set of all possible pixel representations of an image is clearly much later than the set of \"feasible\" images.  Representing the \"feasible\" region is a low-dimensional manifold, and highlighting that the theoretical guarantees should scale with respect to this manifold instead of the latent space, could be seen as some justification for the results along this lens.  At a high level, this paper essentially proves the manifold hypothesis - stating that the \"feasible\" state space (i.e. set of states reachable by a smooth policy) has dimension at most the dimension of the action space plus one.  This is applicable for robotics tasks where the state space is the space of all possible images (e.g. 128 x 128 pixels) whereas the actions is a simple 4 dimensional manifold for controlling torque of the four joints.\n\nTo be more concrete, the authors consider a continuous-time reinforcement learning with deterministic transitions.  The MDP is represented as $(S, A, f, f_r, s_0, \\lambda)$ where $S$ and $A$ are $d_S$ and $d_A$ dimensional subspaces, $f$ is the transition function, $f_r$ the reward function, and $\\lambda$ the exponential time-discount factor.  The agent is tasked with learning a policy $\\pi$ which is a smooth mapping from states to actions with the goal of maximizing their discounted return, now an integral over time of $f_r(s_t)$ with respect to $t$ where $s_t$ evolves along a curve dictated by the smooth policy.  \n\nUnder this notation - the authors then define $S_e = \\cup_{\\pi \\in \\Pi} ( s | s = H_{\\pi}(t) \\text{ for some } t )$ as the set of effective states, where $H_\\pi$ is the continuous time trajectory or curve of policy $\\pi$ in the state space.  Intuitively, this set can be thought of as the set of \"reachable\" states in this continuous time MDP model.  Note that any state $s$ which is reachable by \"some\" policy $\\pi$ in the policy class $\\Pi$ then belongs in $S_e$.  With this, the main result is the following:\n\n**Theorem 3.2** $S_e$ is a smooth manifold of dimension at most $d_a + 1$.\n\nThe authors then complement this theoretical discussion with empirical results.  They compare DDPG on four MuJoCo tasks with two network architectures.  The first is a standard one, first discussed in the DDPG paper. The second is a novel architecture which has a bottleneck output of $d_a + 1$ neurons (for learning the manifold) and the rest is a standard layer as in the original architecture. They introduce a modified loss (to ensure that the output serves as a manifold).  The empirical results show similar performance on the original DDPG algorithm and the one with the modified network architecture.\n\n## Questions\n- Do prior works which consider manifold learning on top of developing an RL algorithm essentially just tune the resulting manifold dimension?\n- Discussion on top of page 6 is confusing - state that you can find such a $v$ but then state that the existence is beyond the scope?\n\n## Minor Comments\n- Last sentence of introduction is unclear and could be rewritten and clarified.\n- \"This solution\" on top of page 3 - solution to what?\n- Superscript of $\\pi$ missing in $s_l$ in Equation 3\n- First paragraph of section 2.2 is a bit unclear\n- \"start\" instead of \"starting\" in first paragraph of section 3\n- Restate definition of $H_\\pi$ in Definition 3.1\n- Theorem ?? on page 7\n- Plots are hard to read in grey-scale\n\n", "summary_of_the_review": "The authors include strong theoretical contributions on state-space reduction.  However, the empirical results are not convincing, and there are not any concrete theoretical justification on the benefits of their main result (Theorem 3.2).", "strength_and_weaknesses": "## Strengths\n1. Model + Theoretical Results: The theoretical results presented in section 3 is novel to the reinforcement learning community.  In fact, this low-dimensional manifold assumption has been made in several prior works - and so this paper can be seen as a proof for the assumptions of those results (although the paper could have explicitly included what those are).\n2. Quality of Results: The theoretical results presented in section 3 help highlight the fact that the effective size of solving an MDP problem can be related to the dimension of the action space.  The neural network architecture is novel - combining manifold learning alongside the typical DDPG losses.\n3. Relation to Existing Literature: The authors do a good job relating the current analysis to the current literature on manifold learning (highlighting that it has been tested empirically in the past), and neural network architecture design for reinforcement learning.\n\n## Weaknesses\n1. Empirical Results: The empirical results show that the performance of the two network architectures (original one + one which uses manifold learning) are statistically similar (due to overlapping confidence intervals).  Potentially there are more extensive empirical results in prior work that could be cited as well for further justification.\n2. Theoretical Benefits: The authors should include more theoretical justification of what Theorem 3.2 allows for current RL convergence results.  For example, taking typical function approximation algorithms in the theoretical reinforcement learning community - does knowledge of reduced dimension of $S_e$ versus the state space allow for reduced regret guarantees?  In isolation, the writing and results seem to suggest this fact but the connection is not formally made.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject