Title: Periodic agent-state based Q-learning for POMDPs
Venue: NeurIPS 2024
Accepted: yes
Keywords: POMDPs

RL

Q-learning

non-stationary policies

non-Markovian environments
Abstract: The standard approach for Partially Observable Markov Decision Processes (POMDPs) is to convert them to a fully observed belief-state MDP. However, the belief state depends on the system model and is therefore not viable in reinforcement learning (RL) settings. A widely used alternative is to use an agent state, which is a model-free, recursively updateable function of the observation history. Examples include frame stacking and recurrent neural networks. Since the agent state is model-free, it is used to adapt standard RL algorithms to POMDPs. However, standard RL algorithms like Q-learning learn a stationary policy. Our main thesis that we illustrate via examples is that because the agent state does not satisfy the Markov property, non-stationary agent-state based policies can outperform stationary ones. To leverage this feature, we propose PASQL (periodic agent-state based Q-learning), which is a variant of agent-state-based Q-learning that learns periodic policies. By combining ideas from periodic Markov chains and stochastic approximation, we rigorously establish that PASQL converges to a cyclic limit and characterize the approximation error of the converged periodic policy. Finally, we present a numerical experiment to highlight the salient features of PASQL and demonstrate the benefit of learning periodic policies over stationary policies.

Periodic agent-state based Q-learning for POMDPs

Abstract

The standard approach for Partially Observable Markov Decision Processes (POMDPs) is to convert them to a fully observed belief-state MDP. However, the belief state depends on the system model and is therefore not viable in reinforcement learning (RL) settings. A widely used alternative is to use an agent state, which is a model-free, recursively updateable function of the observation history. Examples include frame stacking and recurrent neural networks. Since the agent state is model-free, it is used to adapt standard RL algorithms to POMDPs. However, standard RL algorithms like Q-learning learn a stationary policy. Our main thesis that we illustrate via examples is that because the agent state does not satisfy the Markov property, non-stationary agent-state based policies can outperform stationary ones. To leverage this feature, we propose PASQL (periodic agent-state based Q-learning), which is a variant of agent-state-based Q-learning that learns periodic policies. By combining ideas from periodic Markov chains and stochastic approximation, we rigorously establish that PASQL converges to a cyclic limit and characterize the approximation error of the converged periodic policy. Finally, we present a numerical experiment to highlight the salient features of PASQL and demonstrate the benefit of learning periodic policies over stationary policies.

1 Introduction

Recent advances in reinforcement learning (RL) have successfully integrated algorithms with strong theoretical guarantees and deep learning to achieve significant successes [Mni+13; Sil+16]. However, most RL theory is limited to models with perfect state observations [SB08; BT96]. Despite this, there is substantial empirical evidence showing that RL algorithms perform well in partially observed settings [Wie+07; Wie+10; Hau00; HS15; Gru+18; Kap+19; Haf+20; Haf+21]. Recently, there has been a significant advances in the theoretical understanding of different RL algorithms for POMDPs [Sub+22; KY22; Sey+23; DRZ22] but a complete understanding is still lacking.

Planning in POMDPs. When the system model is known, the standard approach [ ˚ Ast65; SS73; CKL94] is to construct an equivalent MDP with the belief state (which is the posterior distribution of the environment state given the history of observations and actions at the agent) as the information state. The belief state is policy independent and has time-homogeneous dynamics, which enables the formulation of a belief-state based dynamic program (DP). There is a rich literature which leverages the structure of the resulting DP to propose efficient algorithms to solve POMDPs [SS73; CKL94; CLZ97; Cha07; Zha09; PGT+03; SS04; SV05]. See [KWW22] for a review. However, the belief state depends on the system model, so the belief-state based approach does not work for RL.

RL in POMDPs. An alternative approach for RL in POMDPs is to consider policies which depend on an agent state { z t } t ≥ 1 , where Z t ∈ Z , which is a recursively updateable compression of the history: the agent starts at an initial state z 0 and recursively updates the agent state as some function of the current agent-state, next observation, and current action. A simple instance of agent-state is frame stacking , where a window of previous observations is used as state [WS94; Mni+13; KY22]. Another example is to use a recurrent neural network such as LSTM or GRU to compress the history of observations and actions into an agent state [Wie+07; Wie+10; HS15; Kap+19; Haf+20]. In

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

⋅

⋅

⋅

⋅

⋅ ⋅

fact, as argued in [DVZ22; Lu+23] such an agent state is present in most deep RL algorithms for POMDPs. We refer to such a representation as an 'agent state' because it captures the agent's internal state that it uses for decision making.

When the agent state is an information state, i.e., satisfies the Markov property, i.e., P ( z t +1 | z 1: t , a 1: t ) = P ( z t +1 | z t , a t ) and is sufficient for reward prediction, i.e., E [ R t | y 1: t , a 1: t ] = E [ R t | z t , a t ] (where y t is the observation, a t is the action, and R t is the per-step reward), the optimal agent-state based policy can be obtained via a dynamic program (DP) [Sub+22]. An example of such an agent state is the belief state. But, in general, the agent state is not an information state. For example, frame stacking and RNN do not satisfy the Markov property, in general. It is also possible to have agent-states that satisfy the Markov property but are not sufficient for reward prediction (e.g., when the agent state is always a constant). In all such settings, the best agent-state policy cannot be obtained via a DP. Nonetheless, there has been considerable interest to use RL to find a good agent-state based policy.

One of the most commonly used RL algorithms is off-policy Q-learning, which we call agent-state Q-learning (ASQL). In ASQL for POMDPs, the Q-learning iteration is applied as if the agent state satisfied the Markov property even though it does not. The agent starts with an initial Q 1 ( z, a ) , acts according to a behavior policy µ , i.e., chooses a t ∼ µ ( z t ) , and recursively updates

̸

where γ ∈ [0 , 1) is the discount factor and the learning rates { α t } t ≥ 1 are chosen such that α t ( z, a ) = 0 if ( z, a ) = ( z t , a t ) . The convergence of ASQL has been recently presented in [KY22; Sey+23] which show that under some technical assumptions, ASQL converges to a limit. The policy determined by ASQL is the greedy policy w.r.t. this limit.

Limitation of Q-learning with agent state. The greedy policy determined by ASQL is stationary (i.e., uses the same control law at every time). In infinite horizon MDPs (and, therefore, also in POMDPs when using the belief state as an agent state), stationary policies perform as well as nonstationary policies. This is because the agent-state satisfies the Markov property. However, in ASQL the agent state generally does not satisfy the Markov property. Therefore, restricting attention to stationary policies may lead to a loss of optimality!

As an illustration, consider the POMDP shown in Fig. 1, which is described in detail in App. A.2 as Ex. 2. Suppose the system starts in state 1 . Since the dynamics are deterministic, the agent can infer the current state from the history of past actions and can take the action to increment the current state and receive a per-step reward of +1 . Thus, the performance J ⋆ BD of belief-state based policies is J ⋆ BD = 1 / (1 -γ ) . Contrast this with the performance J ⋆ SD of deterministic agent-state base policies with agent state equal to current observation, which is given by J ⋆ SD = (1+ γ -γ 2 ) / (1 -γ 3 ) < J ⋆ BD . In particular, for γ = 0 . 9 , J ⋆ BD = 10 which is larger than J ⋆ SD = 4 . 022 .

We show that the gap between J ⋆ SD and J ⋆ BD can be reduced by considering non-stationary policies. Ex. 2 has deterministic dynamics, so the optimal policy can be implemented in open-loop via a sequence of control actions { a ⋆ t } t ≥ 1 , where a ⋆ t = 1 { t ∈ D 1 } . This open-loop policy can be implemented via any information structure, including agent-state based policies. Thus, a non-stationary deterministic agent-state based policy performs better than stationary deterministic agent-state based policies. A similar conclusion also holds for models with stochastic dynamics.

The main idea. Arbitrary non-stationary policies cannot be used in RL because such policies have countably infinite number of parameters. In this paper, we consider a simple class of non-stationary

policies with finite number of parameters: periodic policies . An agent-state based policy π = ( π 1 , π 2 , . . . ) is said to be periodic with period L if π t = π t ′ whenever t ≡ t ′ (mod L ) .

To highlight the salient feature of periodic policies, we perform a brute force search over all deterministic periodic policies of period L , for L = { 1 , . . . , 10 } , in Ex. 2. Let J ⋆ L denote the optimal performance for policies of period L . The result is shown below (see App. A.2 for details):

The above example highlights some salient features of periodic policies: (i) Periodic deterministic agent-state based policies may outperform stationary deterministic agent-state based policies. (ii) { J ⋆ L } L ≥ 1 is not a monotonically increasing sequence. This is because Π L , the set of all periodic deterministic agent-state based policies of period L , is not monotonically increasing. (iii) If L divides M , then J ⋆ L ≤ J ⋆ M . This is because Π L ⊆ Π M . In other words, if we take any integer sequence { L n } n ≥ 1 that has the property that L n divides L n +1 , then the performance of the policies with period L n is monotonically increasing in n . For example, periodic policies with period L ∈ { n ! : n ∈ N } will have monotonically increasing performance. (iv) In the above example, the set D 0 is chosen such that the optimal sequence of actions 1 is not periodic. Therefore, even though periodic policies can achieve a performance that is arbitrarily close to the optimal belief-based policies, they are not necessarily globally optimal (in the class of non-stationary agent-state based policies). Thus, the periodic deterministic policy class is a middle ground between the stationary deterministic and non-stationary policy classes and provides us a simple way of leveraging the benefits of non-stationarity while trading-off computational and memory complexity.

The main contributions of this paper are as follows.

Motivated by the fact that non-stationary agent-state based policies outperform stationary ones, we propose a variant of agent-state based Q-learning (ASQL) that learns periodic policies. We call this algorithm periodic agent-state based Q-learning (PASQL).

We rigorously establish that PASQL converges to a cyclic limit. Therefore, the greedy policy w.r.t. the limit is a periodic policy. Due to the non-Markovian nature of the agent-state, the limit (of the Q-function and the greedy policy) depends on the behavioral policy used during learning.

We quantify the sub-optimality gap of the periodic policy learnt by PASQL.

We present numerical experiments to illustrate the convergence results, highlight the salient features of PASQL, and show that the periodic policy learned by PASQL indeed performs better than stationary policies learned by ASQL.

2 Periodic agent-state based Q-learning (PASQL) with agent state

2.1 Model for POMDPs

A POMDP is a stochastic dynamical system with state s t ∈ S , input a t ∈ A , and output y t ∈ Y , where we assume that all sets are finite. The system operates in discrete time with the dynamics given as follows: The initial state s 1 ∼ ρ and for any time t ∈ N , we have

where P is a probability transition matrix. In addition, at each time the system yields a reward R t = r ( s t , a t ) . We will assume that R t ∈ [0 , R max ] . The discount factor is denoted by γ ∈ [0 , 1) .

Let ⃗ π = ( ⃗ π 1 , ⃗ π 2 , . . . ) denote any (history dependent and possibly randomized) policy. Then the action at time t is given by a t ∼ ⃗ π t ( y 1: t , a 1: t -1 ) . The performance of policy ⃗ π is given by

The objective is to find a (history dependent and possibly randomized) policy ⃗ π to maximize J ⃗ π .

Agent state for POMDPs. An agent-state is model-free recursively updateable function of the history of observations and actions. In particular, let Z denote agent-state space. Then, the agent state process { z t } t ≥ 0 , z t ∈ Z , starts with an initial value z 0 , and is then recursively computed as z t +1 = ϕ ( z t , y t +1 , a t ) for a pre-specified agent-state update function ϕ .

We use π = ( π 1 , π 2 , . . . ) to denote an agent-state based policy, 2 i.e., a policy where the action at time t is given by a t ∼ π t ( z t ) . An agent-state based policy is said to be stationary if for all t and t ′ , we have π t ( a | z ) = π t ′ ( a | z ) for all ( z, a ) ∈ Z × A . An agent-state based policy is said to be periodic with period L if for all t and t ′ such that t ≡ t ′ (mod L ) , we have π t ( a | z ) = π t ′ ( a | z ) for all ( z, a ) ∈ Z × A .

2.2 PASQL: Periodic agent-state based Q-learning algorithm for POMDPs

We now present a periodic variant of agent-state based Q-learning, which we abbreviate as PASQL. PASQL is an online, off-policy learning algorithm in which the agent acts according to a behavior policy µ = ( µ 1 , µ 2 , . . . ) which is a periodic (stochastic) agent-state based policy µ with period L .

Let J t K := ( t mod L ) and L := { 0 , 1 , . . . , L -1 } . Let ( z 1 , a 1 , R 1 , z 2 , a 2 , R 2 , . . . ) be a sample path of agent-state, action, and reward observed by the agent. In PASQL, the agent maintains an L -tuple of Q-functions ( Q 0 t , Q 1 t , . . . , Q L -1 t ) , t ≥ 1 . The ℓ -th component, ℓ ∈ L , is updated at time steps when t = ℓ . In particular, we can write the update as

̸

where the learning rate sequence { ( α 0 t , . . . , α L -1 t ) } t ≥ 1 is chosen such that α ℓ t ( z, a ) = 0 if ( ℓ, z, a ) = ( J t K , z t , a t ) and satisfies Assm. 1. PASQL differs from ASQL in two aspects: (i) The behavior policy µ is periodic. (ii) The update of the Q-function is periodic. When L = 1 , PASQL collapses to ASQL.

The standard convergence analysis of Q-learning for MDPs shows that the Q-function convergences to the unique solution of the MDP dynamic program (DP). The key challenge in characterizing the convergence of PASQL is that the agent state { Z t } t ≥ 1 does not satisfy the Markov property. Therefore, a DP to find the best agent-state based policy does not exist. So, we cannot use the standard analysis to characterize the convergence of PASQL. In Sec. 2.3, we provide a complete characterization of the convergence of PASQL.

The quality of the converged solution depends on the expressiveness of the agent state. For example, if the agent state is not expressive (e.g., agent state is always constant), then even if PASQL converges to a limit, the limit will be far from optimal. Therefore, it is important to quantify the degree of sub-optimality of the converged limit. We do so in Sec. 2.4.

2.3 Establishing the convergence of tabular PASQL

To characterize the convergence of tabular PASQL, we impose two assumptions which are standard for analysis of RL algorithms [JSJ94; BT96]. The first assumption is on the learning rates.

̸

Assumption 1 For all ( ℓ, z, a ) , the learning rates { α ℓ t ( z, a ) } t ≥ 1 are measurable with respect to the sigma-algebra generated by ( z 1: t , a 1: t ) and satisfy α ℓ t ( z, a ) = 0 if ( ℓ, z, a ) = ( J t K , z t , a t ) . Moreover, ∑ t ≥ 1 α ℓ t ( z, a ) = ∞ and ∑ t ≥ 1 ( α ℓ t ( z, a )) 2 < ∞ , almost surely.

The second assumption is on the behavior policy µ . We first state an immediate property.

Lemma 1 For any behavior policy µ , the process { ( S t , Z t ) } t ≥ 1 is Markov. Therefore, the processes { ( S t , Z t , A t ) } t ≥ 1 and { ( S t , Y t , Z t , A t ) } t ≥ 1 are also Markov.

Assumption 2 The behavior policy µ is such that the Markov chain { ( S t , Y t , Z t , A t ) } t ≥ 1 is timeperiodic 3 with period L and converges to a cyclic limiting distribution ( ζ 0 µ , . . . , ζ L -1 µ ) , where ∑ ( s,y ) ζ ℓ µ ( s, y, z, a ) > 0 for all ( ℓ, z, a ) (i.e., all ( ℓ, z, a ) are visited infinitely often).

For the ease of notation, we will continue to use ζ ℓ µ to denote the marginal and conditional distributions w.r.t. ζ ℓ µ . In particular, for marginals we use ζ ℓ µ ( y, z, a ) to denote ∑ s ∈ S ζ ℓ µ ( s, y, z, a ) and so on; for conditionals, we use ζ ℓ µ ( s | z, a ) to denote ζ ℓ µ ( s, z, a ) /ζ ℓ µ ( z, a ) and so on. Note that ζ ℓ µ ( s, z, y, a ) = ζ ℓ µ ( s, z ) µ ( a | z ) P ( y | s, a ) . Thus, we have that ζ ℓ µ ( s | z, a ) = ζ ℓ µ ( s | z ) .

Theorem 1 Under Assms. 1 and 2, the process { ( Q 0 t , . . . , Q L -1 t ) } t ≥ 1 converges to a limit ( Q 0 µ , . . . , Q L -1 µ ) a.s., where the limit is the unique fixed point of the DP for a periodic MDP: 4

where the periodic rewards ( r 0 µ , . . . , r L -1 µ ) and dynamics ( P 0 µ , . . . , P L -1 µ ) are given by

See App. E for proof. Some salient features of the result are as follows:

In contrast to Q-learning for MDPs, the limiting value Q ℓ µ depends on the behavioral policy µ . This dependence arises because the agent state Z t is not an information state and thus is not policy-independent. See [Wit75] for a discussion on policy independence of information states.

Wecan recover some existing results in the literature as special cases of Thm. 1. If we take L = 1 , Thm. 1 recovers the convergence result for ASQL obtained in [Sey+23, Thm. 2]. In addition, if the agent state is a sliding window memory, Thm. 1 recovers the convergence result obtained in [KY22, Thm. 4.1]. Note that the results of Thm. 1 for these special cases is more general because the previous results were derived under a restrictive assumption on the learning rates.

The policy learned by PASQL is the periodic policy π µ = ( π 0 µ , . . . , π L -1 µ ) given by

Since PASQL learns a periodic policy, it circumvents the limitation of ASQL described in the introduction. Thm. 1 addresses the main challenge in the convergence analysis of PASQL: the nonMarkovian dynamics of { Z t } t ≥ 1 . A natural follow-up question is: How good is the learnt policy (PASQL-policy) compared to the optimal? We address this in the next section.

2.4 Characterizing the optimality-gap of the converged limit

History-dependent policies and their value functions. Let h t = ( y 1: t , a 1: t -1 ) denote the history of observations and actions until time t . and let σ t : h t ↦→ z t denotes the map from histories to agent-states obtained by unrolling the memory update function ϕ , i.e., σ 1 ( h 1 ) = ϕ ( z 0 , y 1 , a 0 ) , where z 0 is the initial agent state, a 0 is a dummy action used to initialize the process, σ 2 ( h 2 ) = ϕ ( σ 1 ( h 1 ) , y 2 , a 1 ) , etc.

For any history dependent policy ⃗ π = ( ⃗ π 1 , ⃗ π 2 , · · · ) , where ⃗ π t : h t ↦→ a t , let V ⃗ π t ( h t ) := E ⃗ π [∑ ∞ τ = t γ τ R τ ∣ ∣ h t ] denote the value function of policy ⃗ π starting from history h t at time t . Let V ⋆ t ( h t ) := sup ⃗ π V ⃗ π t ( h t ) denote the optimal value function, where the supermum is over all history dependent policies. In Thm. 1, we have shown that PASQL converges to a limit. Let ⃗ π µ = ( ⃗ π µ, 1 , ⃗ π µ, 2 , . . . ) denote the history dependent policy corresponding to the periodic policy ( π 0 µ , . . . , π L -1 µ ) given by (PASQL-policy), i.e., ⃗ π µ,t ( h t ) := π J t K ( σ t ( h t )) , In this section, we present a bound on the sub-optimality gap V ⋆ t ( h t ) -V ⃗ π µ t ( h t ) .

Integral probability metrics. Let F be a convex and balanced 5 subset of (measureable) real-valued functions on S . The integral probability metric (IPM) w.r.t. F , denoted by d F , is defined as follows: any probability measures ξ 1 and ξ 2 on S , we have d F ( ξ 1 , ξ 2 ) := sup f ∈ F ∣ ∣ ∫ fdξ 1 -∫ fdξ 2 ∣ ∣ . Moreover, for any real-valued function f on S , define ρ F := inf { ρ > 0: f/ρ ∈ F } to be the Minkowski functional w.r.t. F . Note that if for every positive ρ , f/ρ ̸∈ F , then ρ F ( f ) = ∞ .

Many commonly used metrics on probability spaces are IPMs. For example, (i) Total variation distance for which F = { span( f ) ≤ 1 } , where span( f ) = max f -min f is the span seminorm of f . In this case, ρ F ( f ) = span( f ) . (ii) Wasserstein distance for which F = { Lip( f ) ≤ 1 } , where Lip( f ) is the Lipschitz constant of f . In this case, ρ F ( f ) = Lip( f ) . Other examples include Kantorovich metric, bounded Lipschitz metric, and maximum mean discrepancy. See [M¨ ul97; Sub+22] for more details.

Sub-optimality gap. Let T ( t, ℓ ) := { τ ≥ t : τ = ℓ } . Furthermore, for any ℓ ∈ L and t , define

Then, we have the following sub-optimality gap for ⃗ π µ .

See App. F for proof. The salient features of the sub-optimality gap of Thm. 2 are as follows.

We can recover some existing results as special cases of Thm. 2. When we take L = 1 , Thm. 2 recovers the sub-optimality gap for ASQL obtained in [Sey+23, Thm. 3]. In addition, when the agent state is a sliding window memory, Thm. 2 is similar to the sub-optimality gap obtained in [KY22, Thm. 4.1]. Note that the results of Thm. 2 for these special cases is more general because the previous results were derived under a restrictive assumption on the learning rates.

The sub-optimality gap in Thm. 2 is on the sub-optimality w.r.t. the optimal history-dependent policy rather than the optimal non-stationary agent-state policy. Thus, it inherently depends on the quality of the agent state. Consequently, even if L →∞ , the sub-optimality gap does not go to zero.

It is not easy to characterize the sensitivity of the bound to the period L . In particular, increasing L means changing behavioral policy µ , and therefore changing the converged limit ( ζ 0 µ , . . . , ζ L -1 µ ) , which impacts the right hand side of (3) in a complicated way. So, it is not necessarily the case that increasing L reduces the sub-optimality gap. This is not surprising, as we have seen earlier in Ex. 2 presented in the introduction that even the performance of periodic agent-state based policies is not monotone in L .

3 Numerical experiments

In this section, we present a numerical example to highlight the salient features of our results. We use the following POMDP model.

Example 1 Consider a POMDP with S = { 0 , 1 , . . . , 5 } , A = { 0 , 1 } , Y = { 0 , 1 } and γ = 0 . 9 . The dynamics are as shown in Fig. 2. The observation is 0 in states { 0 , 1 , 2 } which are shaded white and is 1 in states { 3 , 4 , 5 } which of +0 . 5 ; others give no reward.

0

1

2

3

4

5

1

0

.

5

0

.

5

0

.

5

0

.

5

1

0

.

5

0

.

5

0

.

5

0

.

5

0

1

2

3

4

5

1

0

.

5

0

.

5

0

.

5

0

.

5

1

0

.

5

0

.

5

0

.

5

0

.

5

Figure 2: The model for Ex. 1, where states which have the same color give the same observation; the green edges give a reward of +1 and blue edges give a reward of +0 . 5 .

are shaded gray. The transitions shown in green give a reward of +1 ; those in in blue give a reward

We consider a family of models, denoted by M ( p ) , p ∈ [0 , 1] , which are similar to Ex. 1 except the controlled state transition matrix is pI +(1 -p ) P , where P is the controlled state transition matrix of Ex. 1 shown in Fig. 2. In the results reported below, we use p = 0 . 01 . The hyperparameters for the experiments are provided in App. H.

Convergence of PASQL with L = 2 . We assume that the agent state Z t = Y t and take period L = 2 . We consider three behavioral policies: µ k = ( µ 0 k , µ 1 k ) , k ∈ K := { 1 , 2 , 3 } , where µ ℓ k : { 0 , 1 } →

𝜇1

𝜇2

𝜇3

3.50

3.50

3.50

3.50

3.25

3.00

369

3.25

3.00

3.25

3.00

3.25

2.75

2.75

2.75

2.50

2.50

2.50

2.50

3.50

3.25

0

2.50×10

Q"(1, 0)

5.00×10

7.50×10°

1.00×106

0

2.50×10

Q"(1,1)

5.00×10

7.50x10

1.00×1(

3.50

3.25

0

2.50×10°

Q;(1, 0)

5.00×10

7.50×10°

1.00×106

350

2.50x10

Q;(1,1)

5.00×10

7.50×10

1.00x1(

3.00

3.00

3.00

3.00

2.75

2.75

2.75

2.75

2.50

2.50

2.50

2.50

0

2.50×10

5.00×10

7.50×10°

1.00×10

0

2.50×10

5.00×10

7.50×10°

1.00×1(

0

2.50×10

5.00×10

7.50x10

1.00×106

0

2.50x10

5.00×10

7.50x10

1.00×1(

3.25

3.50

Q(0, 0)

3.50

3.25

Q*(0,1)

3.50

3.25

[

Q(0,0)

3.50

3.25

Q′(0,1)

3.00

3.00

3.00

3.00

2.75

2.75

2.75

2.75

2.50

2.50

2.50

2.50

2.50x10

Q*(1, 0)

5.00x10

7.50x10°

1.00×10

2.50×10

Q(1,1)

5.00×10°

7.50×10°

1.00×1(

0

2.50x10°

(0'T);0

5.00x10°

7.50×10°

1.00×106

2.50x10

Q;(1,1)

5.00x10°

7.50×10°

1.00×1(

3.50

3.25

C

3.50

3.25

3.25

3.50

3.50

3.25

3.00

3.00

3.00

3.00

2.75

2.75

2.75

2.75

2.50

2.50

2.50

2.50

2.50x10

5.00×10°

7.50×10°

1.00×10

0

2.50×10

5.00×10°

7.50×10°

1.00×1(

0

2.50x10°

5.00×10

7.50×10

1.00x10

2.50×10

5.00x10

7.50×10°

1.00×1(

(0°0)°0

Q”(0,1)

(0'0)0

Q;(0,1)

3.50

3.25

323

3.50

325

3.50

325

3.50

3.00

2.75

2.75

2.75

2.75

2.50

2.50

2.50

2.50

3.50

3.25

2.50x10

Q(1, 0)

5.00×10

7.50×10

1.00×10

3.50

3.25

0

2.50×10

Q"(1, 1)

5.00×10

7.50×10°

1.00×1(

3.29

3.50

0

2.50×10°

(0'1);0

5.00×10

7.50×10°

1.00×10

325

3.50

2.50×10

Q(1,1)

5.00×10

7.50×10°

1.00×1(

3.00

3.00

2.75

2.75

2.75

2.75

2.50

2.50

2.50

2.50

0

2.50×10

5.00x10°

7.50×10

1.00x10

0

2.50×10

5.00×10°

7.50×10

1.00×1(

2.50x10°

5.00×10°

7.50×10°

1.00x10

0

2.50×10°

5.00×10°

7.50×10°

1.00×1(

∆( { 0 , 1 } ) , ℓ ∈ { 0 , 1 } . The policy µ k is completely characterized by four numbers which we write in matrix form as: [ µ 0 k (0 | 0) , µ 1 k (0 | 0); µ 0 k (0 | 1) , µ 1 k (0 | 1)] . With this notation, the three policies are given by µ 1 := [0 . 2 , 0 . 8; 0 . 8 , 0 . 2] , µ 2 := [0 . 5 , 0 . 5; 0 . 5 , 0 . 5] , µ 3 := [0 . 8 , 0 . 2; 0 . 2 , 0 . 8] .

For each behavioral policy µ k , k ∈ K , run PASQL for 25 random seeds. The median + interquantile range of the iterates { Q ℓ t ( z, a ) } t ≥ 1 as well as the theoretical limits Q µ k ( z, a ) (computed using Thm. 1) are shown in Fig. 3. The salient features of these results are as follows:

PASQL converges close to the theoretical limit predicted by Thm. 1.

As highlighted earlier, the limiting value Q ℓ µ k depends on the behavioral policy µ k .

When the aperiodic behavior policy µ 2 is used, the Markov chain { ( S t , Y t , Z t , A t ) } t ≥ 1 is aperiodic, and therefore the limiting distribution ζ ℓ µ 2 and the corresponding Q-functions Q ℓ µ 2 do not depend on ℓ . This highlights the fact that we have to choose a periodic behavioral policy to converge to a non-stationary policy (PASQL-policy).

Comparison of converged policies. Finally, we compute the periodic greedy policy π µ k = ( π 0 µ k , π 1 µ k ) given by (PASQL-policy), k ∈ K , and compute its performance J π µ k via policy evaluation on the product space S × Z (see App. G). We also do a brute force search over all L = 2 periodic deterministic agent-state policies to compute the optimal performance J ⋆ 2 over all such policies. The results, displayed in Table 1, illustrate the following:

The greedy policy π µ k depends on the behavioral policy. This is not surprising given the fact that the limiting value Q ℓ µ k depends on µ k .

The policy π µ 1 achieves the optimal performance, whereas the policies π µ 2 and π µ 3 do not perform well. This highlights the importance of starting with a good behavioral policy. See Sec. 5 for a discussion on variants such as ϵ -greedy.

Advantage of learning periodic policies. As stated in the introduction, the main motivation of PASQL is that it allows us to learn non-stationary policies. To see why this is useful, we run ASQL (which is effectively PASQL with L = 1 ). We again consider three behavioral policies: ¯ µ k , k ∈ K := { 1 , 2 , 3 } , where ¯ µ k : { 0 , 1 } → ∆( { 0 , 1 } ) , where (using similar notation as for L = 2 case) ¯ µ 1 := [0 . 2; 0 . 8] , ¯ µ 2 := [0 . 5; 0 . 5] , ¯ µ 3 := [0 . 8; 0 . 2] .

For each behavioral policy ¯ µ k , k ∈ K , run ASQL for 25 random seeds. The results are shown in App. A.1. The performance of the greedy policies π ¯ µ k and the performance of the best period

L = 1 deterministic agent-state-based policy computed via brute force is shown in Table 2. The key implications are as follows:

As was the case for PASQL, the greedy policy π ¯ µ k depends on the behavioral policy. As mentioned earlier, this is a fundamental consequence of the fact that the agent state is not an information state. Adding (or removing) periodicity does not change this feature.

The best performance of ASQL is worse than the best performance of PASQL. This highlights the potential benefits of using periodicity. However, at the same time, if a bad behavioral policy is chosen (e.g., policy µ 3 ), the performance of PASQL can be worse than that of ASQL for a nominal policy (e.g., policy ¯ µ 2 ). This highlights that periodicity is not a magic bullet and some care is needed to choose a good behavioral policy. Understanding what makes a good periodic behavioral policy is an unexplored area that needs investigation.

4 Related work

Policy search for agent state policies. There is a rich literature on planning with agent state-based policies that build on the policy evaluation formula presented in App. G. See [KWW22] for review. These approaches rely on the system model and cannot be used in the RL setting.

State abstractions for POMDPs are related to agent-state based policies. Some frameworks for state abstractions in POMDPs include predictive state representations (PSR) [RGT04; BSG11; HFP14; KJS15b; KJS15a; JKS16], approximate bisimulation [CPP09; Cas+21], and approximate information states (AIS) [Sub+22] (which is used in our proof of Thm. 2). Although there are various RL algorithms based on such state abstractions, the key difference is that all these frameworks focus on stationary policies in the infinite horizon setting. Our key insight that non-stationary/periodic policies improve performance is also applicable to these frameworks.

ASQL for POMDPs. As stated earlier, ASQL may be viewed as the special case of PASQL when L = 1 . The convergence of the simplest version of ASQL was established in [SJJ94] for Z t = Y t under the assumption that the actions are chosen i.i.d. (and do not depend on z t ). In [PP02] it was established that Q 0 µ is the fixed point of (ASQL), but convergence of { Q t } t ≥ 1 to Q 0 µ was not established. The convergence of ASQL when the agent state is a finite window memory was established in [KY22]. These results were generalized to general agent-state models in [Sey+23]. The regret of an optimistic variant of ASQL was presented in [DVZ22]. However, all of these papers focus on stationary policies.

Our analysis is similar to the analysis of [KY22; Sey+23] with two key differences. First, their convergence results were derived under the assumption that the learning rates are the reciprocal of visitation counts. We relax this assumption to the standard learning rate conditions of Assm. 1 using ideas from stochastic approximation. Second, their analysis is restricted to stationary policies. We generalize the analysis to periodic policies using ideas from time-periodic Markov chains.

Q-learning for non-Markovian environments. As highlighted earlier, a key challenge in understanding the convergence of PASQL is that the agent-state is not Markovian. The same conceptual difficulty arises in the analysis of Q-learning for non-Markovian environments [MH+18; Cha+24; DY24]. Consequently, our analysis has stylistic similarities with the analysis in [MH+18; Cha+24; DY24] but the technical assumptions and the modeling details are different. And more importantly, they restrict attention to stationary policies. Given our results, it may be worthwhile to explore if periodic policies can help in non-Markovian environments as well.

Continual learning and non-stationary MDPs. Non-stationarity is an important consideration in continual learning (see [Abe+24] and references therein). However, in these settings, the environment is non-stationary. Our setting is different: the environment is stationary, but non-stationary policies help because the agent state is not Markov.

Hierarchical learning. The options framework [Pre00; SPS99; Die00; BHP17] is a hierarchical approach that learns temporal abstractions in MDPs and POMDPs. Due to temporal abstraction, the policy learned by the options framework is non-stationary. The same is true for other hierarchical learning approaches proposed in [WS97; CSL21; Vez+17]. In principle, PASQL could be considered as a form of temporal abstraction where time is split into trajectories of length L and then a policy of length L is learned. However, the theoretical analysis for options is mostly restricted to MDP setting

and the convergence guarantees for options in POMDPs are weaker [Ste+18; Qia+18; LVC18]. Nonetheless, the algorithmic tools developed for options might be useful for PASQL as well.

Double Q-learning. The update equation of PASQL are structurally similar to the update equations used in double Q-learning [Has10; VGS16]. However, the motivation and settings are different: the motivation for Double Q-learning is to reduce overestimation bias in off-policy learning in MDPs, while the motivation for PASQL is to induce non-stationarity while learning in POMDPs. Therefore, the analysis of the two algorithms is very different. More importantly, the end goals differ: double Q-learning learns a stationary policy while PASQL learns a periodic policy.

Use of non-stationary/periodic policies in MDPs is investigated in [SL12; LS15; Ber13] in the context of approximate dynamic programming (ADP). Their main result was to show that using non-stationary or periodic policies can improve the approximation error in ADP. Although these results use periodic policies, the setting of ADP in MDPs is very different from ours.

5 Discussion

Deterministic vs. stochastic policies. In this work, we restricted attention to periodic deterministic policies. In principle, we could have also considered periodic stochastic policies. For stationary policies (i.e., when period is one), stochastic policies can outperform deterministic policies [SJJ94] as illustrated by Ex. 3 in App. A.3. However, we do not consider stochastic policies in this work because we are interested in understanding Q-learning with agent-state and Q-learning results in a deterministic policy. There are two options to obtain stochastic policies: using regularization [GSP19], which changes the objective function; or using policy gradient algorithms [Sut+99; BB01], which are a different class of algorithms than Q-learning.

However, as illustrated in the motivating Ex. 2 presented in the introduction, non-stationary policies can do better than stationary stochastic policies as well. So, adding non-stationarity via periodicity remains an interesting research direction when learning stochastic policies as well.

PASQLis a special case of ASQL with state augmentation. In principle, PASQL could be considered as a special case of ASQL with an augmented agent state ¯ Z t = ( Z t , J t K ) . However, the convergence analysis of ASQL in [KY22; Sey+23] does not imply the convergence of PASQL because the results of [KY22; Sey+23] are derived under the assumption that Markov chain { ( S t , Y t , Z t , A t ) } t ≥ 1 is irreducible and aperiodic, while we assume that the Markov chain is periodic . Due to our weaker assumption, we are able to establish convergence of PASQL to time-varying periodic policies.

Non-stationary policies vs. memory augmentation. Nonstationarity is a fundamentally different concept than memory augmentation. As an illustration, consider the T-shaped grid world (first considered in [Bak01]) shown in Fig. 4, which has a corridor of length 2 n . In App. A.4, we show that for this example, a stationary policy which uses a sliding window of past m observations and actions as the agent state needs a memory of at least m > 2 n to reach the goal state. In contrast, a periodic policy with period L = 3 can reach the goal state for every n . This example shows that periodicity is a different concept from memory augmentation and highlights the fact that mechanisms other than memory augmentation can achieve optimal behavior.

2𝑛

cells

g1

s

1

2

3

⋯

⋯

⋯

2𝑛

t

g2

The analysis of this paper is applicable to general memory augmented policies, so we do not need to choose between memory augmentation and periodicity. Our main message is that once the agent's memory is fixed based on practical considerations, adding periodicity could improve performance.

Choice of the period L . If the agent state Z t is a good approximation to the belief state, then ASQL (or, equivalently, PASQL with L = 1 ) would converge to an approximately optimal policy. So, using PASQL a period L > 1 is useful when the agent state is not a good approximation of the belief state.

As shown by Ex. 2 in the introduction, the performance of the best periodic policy does not increase monotonically with the period L . However, if we consider periods in the set { n ! : n ∈ N } , then the performance increases monotonically. However, PASQL does not necessarily converge to the best periodic policy. The quality of the converged policy (PASQL-policy) depends on the behavior

policy µ . The difficulty of finding a good behavioral policy increases with L . In addition, increasing the period increases the memory required to store the tuple ( Q 0 , . . . , Q L ) and the number of samples needed to converge (because each component is updated only once every L samples). Therefore, the choice of the period L should be treated as a hyperparameter that needs to be tuned.

Choice of the behavioral policy. The behavioral policy impacts the converged limit of PASQL, and consequently it impacts the periodic greedy policy that is learned. As we pointed out in the discussion after Thm. 1, this dependence is a fundamental consequence of using an agent state that is not Markov and cannot be avoided. Therefore, it is important to understand how to choose behavioral policies that lead to convergence to good policies.

Generalization to other variants. Our analysis is restricted to tabular off-policy Q-learning where a fixed behavioral policy is followed. Our proof fundamentally depends on the fact that the behavioral policy induces a cyclic limiting distribution on the periodic Markov chain { ( S t , Y t , Z t , A t ) } t ≥ 1 . Such a condition is not satisfied in variants such as ϵ -greedy Q-learning and SARSA. Generalizing the technical proof to cover these more practical algorithms (including function approximation) is an important future direction.

Acknowledgments

The work of AS and AM was supported in part by a grant from Google's Institutional Research Program in collaboration with Mila. The numerical experiments were enabled in part by support provided by Calcul Qu´ ebec and Compute Canada.

References

[Abe+24] David Abel, Andr´ e Barreto, Benjamin Van Roy, Doina Precup, Hado P van Hasselt, and Satinder Singh. 'A definition of continual reinforcement learning'. In: Advances in Neural Information Processing Systems 36 (2024).

[ ˚ Ast65] K.J ˚ Astr¨ om. 'Optimal Control of Markov Processes with Incomplete State Information'. In: Journal of Mathematical Analysis and Applications 10.1 (Feb. 1965), pp. 174-205. ISSN: 0022247X.

[BHP17] Pierre-Luc Bacon, Jean Harb, and Doina Precup. 'The option-critic architecture'. In: Proceedings of the AAAI conference on artificial intelligence . Vol. 31. 1. 2017.

[Bak01] Bram Bakker. 'Reinforcement Learning with Long Short-Term Memory'. In: Advances in Neural Information Processing Systems . Ed. by T. Dietterich, S. Becker, and Z. Ghahramani. Vol. 14. MIT Press, 2001. URL: https://proceedings.neurips. cc/paper_files/paper/2001/file/a38b16173474ba8b1a95bcbc30d3b8a5Paper.pdf .

[BB01] Jonathan Baxter and Peter L Bartlett. 'Infinite-horizon policy-gradient estimation'. In: Journal of Artificial Intelligence Research 15 (2001), pp. 319-350.

[BMP12] Albert Benveniste, Michel M´ etivier, and Pierre Priouret. Adaptive algorithms and stochastic approximations . Vol. 22. Springer Science & Business Media, 2012.

[BT96] Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynamic programming . Athena Scientific, 1996.

[Ber13] Dimitri P Bertsekas. Abstract Dynamic Programming . Athena Scientific, 2013.

[BP24] Shalabh Bhatnagar and L.A. Prashanth. Personal communication. 2024.

[BSG11] Byron Boots, Sajid M Siddiqi, and Geoffrey J Gordon. 'Closing the learning-planning loop with predictive state representations'. In: The International Journal of Robotics Research 30.7 (2011), pp. 954-966.

[Bor08] Vivek S Borkar. Stochastic approximation: a dynamical systems viewpoint . Hindustan Book Agency, 2008.

[CLZ97] Anthony Cassandra, Michael L. Littman, and Nevin L. Zhang. 'Incremental pruning: A simple, fast, exact method for partially observable Markov decision processes'. In: Uncertainty in Artificial Intelligence . 1997.

[CKL94] Anthony R Cassandra, Leslie Pack Kaelbling, and Michael L Littman. 'Acting optimally in partially observable stochastic domains'. In: AAAI Conference on Artificial Intelligence . Vol. 94. 1994, pp. 1023-1028.

[Cas98] Anthony Rocco Cassandra. 'Exact and approximate algorithms for partially observable Markov decision processes'. PhD thesis. Brown University, 1998.

[Cas+21] Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. 'Mico: Improved representations via sampling-based state similarity for Markov decision processes'. In: Advances in Neural Information Processing Systems 34 (2021), pp. 3011330126.

[CPP09] Pablo Samuel Castro, Prakash Panangaden, and Doina Precup. 'Equivalence Relations in Fully and Partially Observable Markov Decision Processes'. In: International Joint Conference on Artificial Intelligence . 2009, pp. 1653-1658.

[Cha+24] Siddharth Chandak, Pratik Shah, Vivek S Borkar, and Parth Dodhia. 'Reinforcement learning in non-Markovian environments'. In: Systems & Control Letters 185 (2024), p. 105751.

[CSL21] Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. 'Goal-conditioned reinforcement learning with imagined subgoals'. In: International Conference on Machine Learning . PMLR. 2021, pp. 1430-1440.

[Cha07] Joseph Chang. Stochastic Processes . Unpublished. Availale at http://www.stat. yale.edu/ ~ pollard/Courses/251.spring2013/Handouts/Chang-notes.pdf . 2007.

[DY24] Ali Devran Kera and Serdar Y¨ uksel. 'Q-Learning for Stochastic Control under General Information Structures and Non-Markovian Environments'. In: Transactions on Machine Learning Research (2024).

[Die00] Thomas G Dietterich. 'Hierarchical reinforcement learning with the MAXQ value function decomposition'. In: Journal of Artificial Intelligence Research 13 (2000), pp. 227-303.

[DRZ22] Shi Dong, Benjamin Van Roy, and Zhengyuan Zhou. 'Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent States'. In: Journal of Machine Learning Research 23.255 (2022), pp. 1-54.

[DVZ22] Shi Dong, Benjamin Van Roy, and Zhengyuan Zhou. 'Simple agent, complex environment: Efficient reinforcement learning with agent states'. In: Journal of Machine Learning Research 23.255 (2022), pp. 1-54.

[Dur19] Rick Durrett. Probability: Theory and Examples . Cambridge University Press, Apr. 2019. ISBN: 9781108473682. DOI: 10.1017/9781108591034 .

[GSP19] Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. 'A theory of regularized markov decision processes'. In: International Conference on Machine Learning . PMLR. 2019, pp. 2160-2169.

[Gru+18] Audrunas Gruslys, R´ emi Munos, Ivo Danihelka, Marc G. Bellemare, and Alex Graves. 'The Reactor: A Sample-Efficient Actor-Critic Architecture'. In: Proceedings of the International Conference on Learning Representations (ICLR) . 2018. URL: https: //arxiv.org/abs/1704.04651 .

[Haf+20] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. 'Dream to Control: Learning Behaviors by Latent Imagination'. In: International Conference on Learning Representations . 2020.

[Haf+21] Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. 'Mastering Atari with Discrete World Models'. In: International Conference on Learning Representations . 2021.

[HFP14] William Hamilton, Mahdi Milani Fard, and Joelle Pineau. 'Efficient learning and planning with compressed predictive states'. In: The Journal of Machine Learning Research 15.1 (2014), pp. 3395-3439.

[Han98] Eric A. Hansen. 'Solving POMDPs by searching in policy space'. In: Uncertainty in Artificial Intelligence . Madison, Wisconsin, 1998, pp. 211-219. ISBN: 155860555X.

[Has10] Hado Hasselt. 'Double Q-learning'. In: Advances in neural information processing systems 23 (2010).

[HS15] Matthew Hausknecht and Peter Stone. 'Deep recurrent Q-learning for partially observable MDPs'. In: AAAI Fall Symposium Series . 2015.

[Hau97] Milos Hauskrecht. 'Planning and control in stochastic domains with imperfect information'. PhD thesis. Massachusetts Institute of Technology, 1997.

[Hau00] Milos Hauskrecht. 'Value-function approximations for partially observable Markov decision processes'. In: Journal of artificial intelligence research 13 (2000), pp. 3394.

[JSJ94] Tommi Jaakkola, Satinder Singh, and Michael Jordan. 'Reinforcement Learning Algorithm for Partially Observable Markov Decision Problems'. In: Advances in Neural Information Processing Systems . Vol. 7. MIT Press, 1994, pp. 345-352.

[JKS16] Nan Jiang, Alex Kulesza, and Satinder P Singh. 'Improving Predictive State Representations via Gradient Descent.' In: AAAI Conference on Artificial Intelligence . 2016, pp. 1709-1715.

[Kap+19] Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. 'Recurrent Experience Replay in Distributed Reinforcement Learning'. In: International Conference on Learning Representations . 2019.

[KY22] Ali Devran Kara and Serdar Y¨ uksel. 'Convergence of Finite Memory Q Learning for POMDPs and Near Optimality of Learned Policies Under Filter Stability'. In: Mathematics of Operations Research (Nov. 2022). ISSN: 1526-5471. DOI: 10.1287/moor. 2022.1331 .

[KWW22] Mykel J Kochenderfer, Tim A Wheeler, and Kyle H Wray. Algorithms for decision making . MIT press, 2022.

[KJS15a] Alex Kulesza, Nan Jiang, and Satinder Singh. 'Low-rank spectral learning with weighted loss functions'. In: Artificial Intelligence and Statistics . 2015, pp. 517-525.

[KJS15b] Alex Kulesza, Nan Jiang, and Satinder P Singh. 'Spectral Learning of Predictive State Representations with Insufficient Statistics.' In: AAAI Conference on Artificial Intelligence . 2015, pp. 2715-2721.

[KY97] Harold J. Kushner and G. George Yin. Stochastic Approximation Algorithms and Applications . Springer New York, 1997. DOI: 10.1007/978-1-4899-2696-8 .

[LVC18] Tuyen P Le, Ngo Anh Vien, and TaeChoong Chung. 'A deep hierarchical reinforcement learning algorithm in partially observable Markov decision processes'. In: Ieee Access 6 (2018), pp. 49089-49102.

[LS15] Boris Lesner and Bruno Scherrer. 'Non-Stationary Approximate Modified Policy Iteration'. In: International Conference on Machine Learning . Vol. 37. Proceedings of Machine Learning Research. Lille, France: PMLR, July 2015, pp. 1567-1575.

[Lit96] Michael Lederman Littman. 'Algorithms for sequential decision-making'. PhD thesis. Brown University, 1996.

[Lu+23] Xiuyuan Lu, Benjamin Van Roy, Vikranth Dwaracherla, Morteza Ibrahimi, Ian Osband, Zheng Wen, et al. 'Reinforcement learning, bit by bit'. In: Foundations and Trends in Machine Learning 16.6 (2023), pp. 733-865.

[MH+18] Sultan Javed Majeed, Marcus Hutter, et al. 'On Q-learning Convergence for NonMarkov Decision Processes.' In: IJCAI . Vol. 18. 2018, pp. 2546-2552.

[Mni+13] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 'Playing atari with deep reinforcement learning'. In: arXiv preprint arXiv:1312.5602 (2013).

[M¨ ul97] Alfred M¨ uller. 'Integral probability metrics and their generating classes of functions'. In: Advances in Applied Probability 29.2 (1997), pp. 429-443.

[Nor98] James R Norris. Markov chains . Cambridge University Press, 1998.

[PP02] Theodore J. Perkins and Mark D. Pendrith. 'On the Existence of Fixed Points for QLearning and Sarsa in Partially Observable Domains'. In: International Conference on Machine Learning . San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2002, pp. 490-497.

[PGT+03] Joelle Pineau, Geoff Gordon, Sebastian Thrun, et al. 'Point-based value iteration: An anytime algorithm for POMDPs'. In: International Joint Conference on Artificial Intelligence . Vol. 3. 2003, pp. 1025-1032.

[Pla77] Loren Kerry Platzman. 'Finite Memory Estimation and Control of Finite Probabilistic Systems.' PhD thesis. Massachusetts Institute of Technology, 1977.

[PB24] L.A. Prashanth and Shalabh Bhatnagar. Gradient-based algorithms for zeroth-order optimization . Now publishers, 2024. URL: http : / / www . cse . iitm . ac . in / ~ prashla/bookstuff/GBSO_book.pdf .

[Pre00] Doina Precup. Temporal abstraction in reinforcement learning . University of Massachusetts Amherst, 2000.

[Qia+18] Zhiqian Qiao, Katharina Muelling, John Dolan, Praveen Palanisamy, and Priyantha Mudalige. 'Pomdp and hierarchical options mdp with continuous actions for autonomous driving at intersections'. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC) . IEEE. 2018, pp. 2377-2382.

[Rii65] Jens Ove Riis. 'Discounted Markov Programming in a Periodic Process'. In: Operations Research 13.6 (Dec. 1965), pp. 920-929. ISSN: 1526-5463. DOI: 10.1287/ opre.13.6.920 .

[RM51] Herbert Robbins and Sutton Monro. 'A Stochastic Approximation Method'. In: The Annals of Mathematical Statistics 22.3 (1951), pp. 400-407.

[RGT04] Matthew Rosencrantz, Geoff Gordon, and Sebastian Thrun. 'Learning low dimensional predictive representations'. In: International Conference on Machine Learning . 2004.

[Sch16] Bruno Scherrer. On Periodic Markov Decision Processes . European Workshop on Reinforcement Learning. Dec. 2016. URL: https://ewrl.files.wordpress.com/ 2016/12/scherrer.pdf .

[SL12] Bruno Scherrer and Boris Lesner. 'On the use of non-stationary policies for stationary infinite-horizon Markov decision processes'. In: Advances in Neural Information Processing Systems 25 (2012).

[Sey+23] Erfan SeyedSalehi, Nima Akbarzadeh, Amit Sinha, and Aditya Mahajan. 'Approximate information state based convergence analysis of recurrent Q-learning'. In: European conference on reinforcement learning . 2023. URL: https://arxiv.org/abs/ 2306.05991 .

[Sil+16] David Silver et al. 'Mastering the game of Go with deep neural networks and tree search'. In: Nature 529 (2016), pp. 484-489.

[SJJ94] Satinder P Singh, Tommi Jaakkola, and Michael I Jordan. 'Learning without stateestimation in partially observable Markovian decision processes'. In: Machine Learning . Elsevier, 1994, pp. 284-292.

[SS73] Richard D. Smallwood and Edward J. Sondik. 'The Optimal Control of Partially Observable Markov Processes over a Finite Horizon'. In: Operations Research 21.5 (Oct. 1973), pp. 1071-1088. DOI: 10.1287/opre.21.5.1071 .

[SS04] Trey Smith and Reid Simmons. 'Heuristic search value iteration for POMDPs'. In: Conference on Uncertainty in Artificial Intelligence . Banff, Canada, 2004, pp. 520527.

[SV05] Matthijs TJ Spaan and Nikos Vlassis. 'Perseus: Randomized point-based value iteration for POMDPs'. In: Journal of Artificial Intelligence Research 24 (2005), pp. 195220.

[Ste+18] Denis Steckelmacher, Diederik Roijers, Anna Harutyunyan, Peter Vrancx, H´ el` ene Plisnier, and Ann Now´ e. 'Reinforcement learning in POMDPs with memoryless options and option-observation initiation sets'. In: Proceedings of the AAAI conference on artificial intelligence . Vol. 32. 1. 2018.

[Sub+22] Jayakumar Subramanian, Amit Sinha, Raihan Seraj, and Aditya Mahajan. 'Approximate information state for approximate planning and reinforcement learning in partially observed systems'. In: Journal of Machine Learning Research 23.12 (2022), pp. 1-83.

[Sut+99] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 'Policy gradient methods for reinforcement learning with function approximation'. In: Advances in Neural Information Processing Systems . Vol. 12. 1999.

[SPS99] Richard S Sutton, Doina Precup, and Satinder Singh. 'Between MDPs and semiMDPs: A framework for temporal abstraction in reinforcement learning'. In: Artificial intelligence 112.1-2 (1999), pp. 181-211.

[SB08] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . MIT Press, 2008.

[VGS16] Hado Van Hasselt, Arthur Guez, and David Silver. 'Deep reinforcement learning with double q-learning'. In: Proceedings of the AAAI conference on artificial intelligence . Vol. 30. 1. 2016.

[Vez+17] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. 'Feudal networks for hierarchical reinforcement learning'. In: International conference on machine learning . PMLR. 2017, pp. 3540-3549.

[WS94] Chelsea C White III and William T Scherer. 'Finite-memory suboptimal design for partially observed Markov decision processes'. In: Operations Research 42.3 (1994), pp. 439-455.

[WS97] Marco Wiering and J¨ urgen Schmidhuber. 'HQ-learning'. In: Adaptive behavior 6.2 (1997), pp. 219-246.

[Wie+07] Daan Wierstra, Alexander Foerster, Jan Peters, and Juergen Schmidhuber. 'Solving deep memory POMDPs with recurrent policy gradients'. In: International Conference on Artificial Neural Networks (ICANN) . Springer. 2007, pp. 697-706.

[Wie+10] Daan Wierstra, Alexander F¨ orster, Jan Peters, and J¨ urgen Schmidhuber. 'Recurrent policy gradients'. In: Logic Journal of the IGPL 18.5 (2010), pp. 620-634.

[Wit75] Hans S Witsenhausen. 'On policy independence of conditional expectations'. In: Information and Control 28.1 (1975), pp. 65-75.

[Zha09] H. Zhang. 'Partially Observable Markov Decision Processes: A Geometric Technique and Analysis'. In: Operations Research (2009).

Contents of Appendix

A Illustrative examples

A.1 Ex. 1: Learning curves for ASQL

For each behavioral policy ¯ µ k , k ∈ K , we run PASQL for 25 random seeds. The median + interquantile range of the iterates { Q t ( z, a ) } t ≥ 1 as well as the theoretical limits Q ¯ µ k ( z, a ) (computed as per Thm. 1 for L = 1 ) are shown in Fig. 5. These curves show that the result of Thm. 1 is valid for the stationary case ( L = 1 ) as well.

̅

𝜇 1

𝜇 2

𝜇 3

2.50×10

5.00×10

7.50×10°

1.00×106

2.50×10

5.00x10

7.50×10

1.00×1(

C

Q”(1, 0)

Q"(1,1)

2.50×10

5.00×10

7.50×10

1.00×10

2.50×10

5.00×10^

7.50×10

1.00×1(

Q*(0, 0)

Q°(0,1)

0

2.50×10

5.00×10

7.50×10

1.00×106

0

2.50×10

5.00×10

7.50×10

1.00×1(

Q(1, 0)

Q*(1,1)

0

2.50×10°

5.00×10

7.50×10

1.00x106

0

2.50×10

5.00×10

7.50×10

1.00×1(

Q°(0,0)

Q°(0,1)

0

2.50×10°

5.00×10

7.50×10°

1.00×106

0

2.50×10

5.00×10

7.50×10°

1.00×1(

(0')0

Q*(1,1)

0

2.50×10

5.00×10°

7.50×10°

1.00×106

0

2.50x10°

5.00×10

7.50×10°

1.00×1(

A.2 Ex. 2: non-stationary policies can outperform stationary policies

Example 2 Consider a POMDP with S = Z > 0 , A = { 0 , 1 } , and Y = { 0 , 1 } . The system starts in an initial state s 1 = 1 and has deterministic dynamics. To describe the dynamics and the reward function, we define D 0 := { n ( n +1) / 2+1 : n ∈ Z ≥ 0 } , D 1 = N \ D 0 , and D = D 0 ×{ 0 }∪ D 1 ×{ 1 } ⊂ S × A . Then, the dynamics, observations, and rewards are given by

Thus, the state is incremented if the agent takes action 0 when the state is in D 0 and takes action 1 when the state is in D 1 . Taking these actions yield a reward of +1 . Not taking such an action results in a reward of -1 and resets the state to 1 . The agent does not observe the state, but only observes whether the state is odd or even. A graphical representation of the model is shown in Fig. 1.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

⋅

⋅

⋅

⋅

⋅ ⋅

Figure 1: Graphical representation of Ex. 2. The cells indicate the state of the environment. Cells with the same background color have the same observation. The cells with a thick red boundary correspond to elements of the set D 0 := { n ( n +1) / 2 + 1 : n ∈ N } , where the action 0 gives a reward of +1 and moves the state to the right, while the action 1 gives a reward of -1 and resets the state to 1 . The cells with a thin black boundary correspond to elements of the set D 1 = N \ D 0 , where the action 1 gives the reward of +1 and moves the state to the right while the action 0 gives a reward of -1 and resets the state to 1 . Discount factor γ = 0 . 9 .

For policy class Π BD (the class of all belief-based deterministic policies), since the system starts in a known initial state and the dynamics are deterministic, the agent can compute the current state

̅

̅

(thus, the belief is a delta function on the current state). Thus, the agent can always choose the correct action depending on whether the state is in D 0 and D 1 . Hence J ⋆ BD = 1 / (1 -γ ) , which is the highest possible reward.

For policy class Π SD (the class of all agent-state based deterministic policies), there are four possible deterministic policies. For odd observations, the agent may take action 0 and 1 . Similarly, for even observations, the agent may take action 0 or 1 . Note that the system starts in state 1 , which is in D 0 . Therefore, if the agent chooses action 1 when the observation is odd, it receives a reward of -1 and stays at state 1 . Therefore, the discounted total reward is -1 / (1 -γ ) , which is the least possible value. Therefore, any policy that chooses 1 on odd observations cannot be optimal. Therefore, the optimal (deterministic) action on odd observations is to pick action 0 . Thus, there are two policies that we need to evaluate.

If the agent chooses action 0 at both odd and even observations, the state cycles between 1 → 2 → 3 → 1 → 2 → 3 · · · with the reward sequence (+1 , +1 , -1 , +1 , +1 , -1 , . . . ) . Thus, the cumulative total reward of this policy is (1 + γ -γ 2 ) / (1 -γ 3 ) .

If the agent chooses action 0 at odd observations and action 1 at even observations, the state cycles between 1 → 2 → 1 → 2 · · · with the reward sequence (+1 , -1 , +1 , -1 , . . . ) . Thus, the cumulative total reward of this policy is 1 / (1 + γ ) .

It is easy to verify that for γ ∈ (0 , 1) , 1 / (1 + γ ) < (1 + γ -γ 2 ) / (1 -γ 3 ) . Thus,

We also consider policy class Π SS : the class of all stationary stochastic agent-state based policies. For policy class Π SS , the policy is characterized by two numbers ( p 0 , p 1 ) ∈ [0 , 1] 2 , where p y denotes the probability of choosing action 1 when the observation is y , y ∈ { 0 , 1 } . We compute the approximately optimal policy by doing a brute force search over ( p 0 , p 1 ) by discretizing them two decimal places and for each choice, running a Monte Carlo simulation of length 1 , 000 and averaging it over 100 random seeds. We find that there is negligible difference between the performance of stochastic and deterministic policies.

Finally, we consider policy class Π L , which is the class of periodic deterministic agent-state based policies. A policy π ∈ Π L is characterized by two vectors p 0 , p 1 ∈ { 0 , 1 } L , where p y,ℓ denotes the action chosen when t mod L = ℓ and the observation is y . We do an exhaustive search over all deterministic policies of length L , L ∈ { 1 , . . . , 10 } to compute the numbers shown in the main text.

A.3 Ex. 3: stochastic policies can outperform deterministic policies

When the agent state is not an information state, the optimal stochastic stationary policy will perform better than (or equal to) the optimal deterministic stationary policy as observed in [SJJ94]. Here is an example to illustrate this for a simple toy POMDP.

0

1

2

0

.

5

0

.

5

1

1

0

1

2

0

.

5

0

.

5

0

.

5

0

.

5

1

(b) Dynamics under action

1

Example 3 Consider a POMDP with S = { 0 , 1 , 2 } , A = { 0 , 1 } and Y = { 0 } . The system starts at an initial state s 1 = 0 and the dynamics under the two actions are shown in Fig. 6. The agent does not observe the state, i.e., Y t ≡ 0 . The rewards under action 0 are r ( · , 0) = [ -1 , 0 , 2] and the rewards under action 1 are r ( s, 1) = -0 . 5 , for all s ∈ S .

We consider agent state Z t = Y t . Let Π SS denote the of all stationary stochastic policies and Π SD denote the class of of all stationary deterministic policic A policy π ∈ Π SS is parameterized by a single parameter p ∈ [0 , 1] , which indicates the probability of choosing action 1 . Wedenote such a policy by π p . Note that p ∈ { 0 , 1 } , π p ∈ Π SD . Let ( P a , r a ) denote the probability transition matrix and reward func-

tion when a ∈ A is chosen and let ( P p , r p ) = (1 -p )( P 0 , r 0 ) + p ( P 1 , r 1 ) . Then, the performance of policy π p is given by J π p = [(1 -γP p ) -1 r p ] 0 . The performance for all p ∈ [0 , 1] for γ = 0 . 9 is shown in Fig. 7, which shows that the best performance is achieved by the stochastic policy π p with p ≈ 0 . 39 .

Thus, stochastic policies can outperform deterministic policies.

A.4 Ex. 4: conceptual difference between state-augmentation and periodic policies

2𝑛

cells

g1

s

1

2

3

⋯

⋯

⋯

2𝑛

t

g2

Example 4 Consider a T-shaped grid world showed in Fig. 4 with state space P × G , where P = { S , 1 , 2 , . . . , 2 n, T } is the position of the agent and G = { G 1 , G 2 } is the location of the goal. The observation space is Y = { 0 , 1 , 2 , 3 } . The observation is a deterministic function of the state and is given as follows:

At state ( S , G i ) , i ∈ { 1 , 2 } , the observation is i and reveals the location of the goal state to the agent.

At states { 1 , . . . , 2 n } × G , the observation is 0 , so the agent cannot distinguish between these states.

At states { T } × G , the observation is 3 , so the agent knows when it reaches the T state.

The action space depends on the current state: actions { LEFT , RIGHT , STAY } are available when the agent is at { S , 1 , . . . , 2 n } and actions { UP , DOWN } are available at position T.

The agent receives a reward of +1 if it reaching the goal state and -1 if it reaches the wrong goal state state (i.e., reaches G 2 when the goal state is G 1 ). The discount factor γ = 1 .

We consider two classes of policies:

(i) Π SD ( m ) : Stationary policies with agent state equal to a sliding window of the last m observations and actions.

(ii) Π L : Periodic policies with agent state equal to the last observation and periodic L .

It is easy to see that as long as the window length m ≤ 2 n , any policy in Π SD ( m ) yields an average return of 0 ; for window lengths m> 2 n , the agent can remember the first observation, and therefore it is possible to construct a policy that yields a return of +1 .

We now consider a deterministic periodic policy with period L = 3 given as follows: 6 π = ( π 0 , π 1 , π 2 ) where π ℓ : Y → A . We denote each π ℓ as a column vector, where the y -th component indicates the action π ℓ ( y ) , where - means that the choice of the action for that observation is

irrelevant for performance. The policy is given by

It is easy to verify if the system starts in state (0 , G 1 ) , then by following policy ( π 0 , π 1 , π 2 ) , the agent reaches state G 1 at time 3 n + 3 . Moreover, when the system starts in state (0 , G 2 ) , then by following the policy ( π 0 , π 1 , π 2 ) , the agent reaches G 2 at time 3 n + 4 . Thus, in both cases, the policy ( π 0 , π 1 , π 2 ) yields the maximum reward of +1 .

B Periodic Markov chains

In most of the standard reference material on Markov chains, it is assumed that the Markov chain is aperiodic and irreducible. In our analysis, we need to work with periodic Markov chains. In this appendix, we review some of the basic properties of Markov chains and then derive some fundamental results for periodic Markov chains.

Let S be a finite set. A stochastic process { S t } t ≥ 0 , S t ∈ S , is called a Markov chain if it satisfies the Markov property : for any t ∈ Z ≥ 0 and s 1: t +1 ∈ S t +1 , we have

If is often convenient to assume that S = { 1 , . . . , n } . We can define an n × n transition probability matrix P t given by [ P t ] ij = P ( S t +1 = j | S t = i ) . Then, all the probabilistic properties of the Markov chain is described by the transition matrices ( P 0 , P 1 , . . . ) .

In particular, suppose the Markov chain starts at the initial PMF (probability mass function) ξ 0 and let ξ t denote the PMF at time t . We will view ξ t as a n -dimensional row vector. Then, Eq. (4) implies ξ t +1 = ξ t P t and, therefore,

B.1 Time-homogeneous Markov chains and their properties

AMarkov chain is said to be time-homogeneous if the transition matrix P t is the same for all time t . In this section, we state some standard results for time-homogeneous Markov chains [Nor98].

B.1.1 Classification of states

The states of a time-homogeneous Markov chain can be classified as follows.

We say that a state j is accessible from i (abbreviated as i ⇝ j ) if there is exists an m ∈ Z ≥ 0 (which may depend on i and j ) such that [ P m ] ij > 0 . The fact that [ P m ] ij > 0 implies that there exists an ordered sequence of states ( i 0 , . . . , i m ) such that i 0 = i and i m = j such that P i k i k +1 > 0 ; thus, there is a path of positive probability from state i to state j .

Accessibility is an transitive relationship, i.e., if i ⇝ j and j ⇝ k implies that i ⇝ k .

Two distinct states i and j are said to communicate (abbreviated to i ↭ j ) if i is accessible from j (i.e., j ⇝ i ) and j is accessible from i ( i ⇝ j ). Alternatively, we say that i and j communicate if there exist m,m ′ ∈ Z ≥ 0 such that [ P m ] ij > 0 and [ P m ′ ] ji > 0 .

Communication is an equivalence relationship, i.e., it is reflexive ( i ↭ i ), symmetric ( i ↭ j if and only if j ↭ i ), and transitive ( i ↭ j and j ↭ k implies i ↭ k ).

The states in a finite-state Markov chain can be partitioned into two sets: recurrent states and transient states . A state is recurrent if it is accessible from all states that are from it (i.e., i is recurrent if i ⇝ j implies that j ⇝ i ). States that are not recurrent are transient .

It can be shown that a state i is recurrent if and only if

States i and j are said to belong to the same communicating class if i and j communicate. Communicating classes form a partition the state space. Within a communicating class, all states are of the same type, i.e., either all states are recurrent (in which case the class is called a recurrent class) or all states are transient (in which case the class is called a transient class).

AMarkov chain with a single communicating class (thus, all states communicate with each other and are, therefore, recurrent) is called irreducible .

The period of a state i , denoted by d ( i ) , is defined as

If the period is 1 , the state is aperiodic , and if the period is 2 or more, the state is periodic . It can be shown that all states in the same class have the same period.

A Markov chain is aperiodic, if all states are aperiodic. A simple sufficient (but not necessary) condition for an irreducible Markov chain to be aperiodic is that there exists a state i such that P ii > 0 . In general, for a finite and aperiodic Markov chain, there exists a positive integer T such that

B.1.2 Limit behavior of Markov chains

We now state some special distributions for a time-homogeneous Markov chain.

A PMF ζ on S is called a stationary distribution if ζ = ζP . Thus, if a (time-homogeneous) Markov chain starts in a stationary distribution, it stays in a stationary distribution.

A finite irreducible Markov chain has a unique stationary distribution. Moreover, when the Markov chain is also aperiodic, the stationary distribution is given by ζ ( j ) = 1 /m j , where m j is the expected return time to state j .

A PMF ζ on S is called a limiting distribution if

A finite irreducible Markov chain has a limiting distribution if and only if it is aperiodic. Therefore, for an aperiodic Markov chain, the limiting distribution is the same as the stationary distribution.

Theorem 3 (Strong law of large numbers for Markov chains, Theorem 5.6.1 of [Dur19])

Suppose { S t } t ≥ 1 is an irreducible Markov chain that starts in state i ∈ S . Then,

Therefore, for any function h : S → R ,

If, in addition, the Markov chain { S t } t ≥ 1 is aperiodic, and has a limiting distribution ζ , then we have that

B.2 Time-varying with periodic transition matrix

In this section, we consider time-varying Markov chains where the transition matrices ( P 0 , P 1 , . . . ) are periodic with period L . Let J t K = ( t mod L ) and L = { 0 , . . . , L -1 } . Then, the transition matrix P t is the same as P J t K . Thus, the system dynamics are completely described by the transition matrices { P ℓ } ℓ ∈ L . With a slight abuse of notation, we will call such a Markov chain as L -periodic Markov chain . We will show later that the notion of time-periodicity that we are considering is equivalent to the notion of state-periodicity for time-homogeneous Markov chains defined earlier.

B.3 Constructing an equivalent time-homogeneous Markov chain

Since the Markov chain is not time-homogeneous, the classification and results of the previous section are not directly applicable. There are two ways to construct a time-homogeneous Markov chain: using state augmentation or viewing the process after every L steps.

B.3.1 Method 1: State augmentation

The original time-varying Markov chain { S t } t ≥ 0 is equivalent to the time-homogeneous Markov chain { ( S t , J t K ) } t ≥ 0 defined on S × L with transition matrix ¯ P given by

Example 5 Consider a 2 -periodic Markov chain with state space S = { 1 , 2 } and transition matrices

The time-periodic Markov chain of Ex. 5 may be viewed as a time-homogeneous Markov chain with state space { 1 , 2 } × { 0 , 1 } and transition matrix

where 0 denotes the all zero matrix and I denotes the identity matrix (both of size 2 × 2 ). Note that the time-homogeneous Markov chain is periodic.

Define the following:

L block diagonal matrices Λ 0 , . . . , Λ L -1 ∈ R nL × nL as follows:

A permutation matrix Π ∈ { 0 , 1 } nL × nL as follows

where each block is n × n .

The permutation matrix Π satisfies the following properties (which can be verified by direct algebra):

(P1) ΠΠ ⊺ = I and therefore Π -1 = Π ⊺ .

(P2) Π L = I .

(P3) Λ ℓ Π = ΠΛ J ℓ +1 K , ℓ ∈ L .

In general, the transition matrix of the Markov chain { ( S t , J t K ) } t ≥ 0 is

B.3.2 Method 2: Viewing the process every L steps

The original Markov chain viewed every L -steps, i.e., the process { S kL + ℓ } k ≥ 0 , ℓ ∈ L , is a timehomogeneous Markov chain with transition probability matrix P ℓ given by

that is,

B.3.3 Relationship between the two constructions

The two constructions are related as follows.

Proposition 1 We have that ¯ P L = blkdiag( P 0 , . . . , P L ) .

PROOF From (P3), we get that ¯ P = ΠΛ 1 . Therefore,

Similarly

Continuing this way, we get

where the last equality follows from (P2). The result then follows from the definitions of Λ ℓ and P ℓ , ℓ ∈ L . ✷

B.4 Limiting behavior of periodic Markov chain

In the subsequent discussion, we consider the following assumptions.

Assumption 3 Every {P ℓ } , ℓ ∈ L , is irreducible and aperiodic

Suppose Assm. 3 holds. Define ζ ℓ to be the unique stationary distribution for Markov chain P ℓ , ℓ ∈ L , i.e., ζ ℓ is the unique PMF that satisfies ζ ℓ = ζ ℓ P ℓ .

Proposition 2 The PMFs { ζ ℓ } ℓ ∈ L satisfy

PROOF We prove the result for ℓ = 0 . The analysis is the same for general ℓ . By assumption, we have that

Let ¯ ζ 1 := ζ 0 P 0 . Then, we have

Thus ¯ ζ 1 is a stationary distribution. Since P 1 is irreducible, the stationary distribution is unique, hence ¯ ζ 1 must equal ζ 1 . ✷

We can verify this result for Ex. 5. For this model, we have

Thus,

And we can verify that ζ 0 P 0 = ζ 1 and ζ 1 P 1 = ζ 0 .

Proposition 3 Under Assm. 3, the limiting distribution of the Markov chain { S t } t ≥ 0 is cyclic. In particular, for any initial distribution ξ 0 ,

Furthermore,

Consequently, for any function h : S → R ,

PROOF The results follow from standard results for the time-homogeneous Markov chain { S kL + ℓ } k ≥ 0 . ✷

PROOF (ALTERNATIVE) We present an alternative proof that uses the state augmented Markov chain ¯ P . We first prove that under Assm. 3, the chain ¯ P is irreducible periodic with period L .

The proof of irreducibility relies on two observations.

Fix an ℓ ∈ L and consider i, j ∈ S . Since P ℓ is irreducible, we have that there exists a positive integer m (depending on i , j , and ℓ ) such that [ P ℓ m ] ij > 0 . Note that Prop. 1 implies that [ ¯ P mL ] ( i,ℓ ) , ( j,ℓ ) = [ P ℓ ] ij > 0 . Therefore, in the Markov chain ¯ P , states ( i, ℓ ) ⇝ ( j, ℓ ) . Since i and j were arbitrary, all states S ×{ ℓ } belong to the same communicating class.

Now consider two ℓ, ℓ ′ ∈ L . Suppose we start at some state ( i, ℓ ) ∈ S ×{ ℓ } , then in [ ℓ ′ -ℓ ] steps, we will reach some state ( j, ℓ ′ ) ∈ S ×{ ℓ ′ } . Thus, ( j, ℓ ′ ) is accessible from ( i, ℓ ) . But, we have already argued that all states in S × { ℓ } belong to the same communicating class, therefore all states in S ×{ ℓ ′ } are accessible from all states in S ×{ ℓ } . By interchanging the roles of ℓ and ℓ ′ , we have that all states in S ×{ ℓ } are accessible from all starts in S ×{ ℓ ′ } . Therefore, the states S ×{ ℓ } and S ×{ ℓ ′ } belong to the same communicating class. Since ℓ and ℓ ′ were arbitrary, we have that all states of ¯ P belong to the same communicating class. Hence, ¯ P is irreducible.

We now show that ¯ P is periodic. First observe that the Markov chain starting in the set S ×{ ℓ } does not return to the same set for the first L -1 steps. Thus, [ ¯ P t ] ( i,ℓ ) , ( i,ℓ ) = 0 for t ∈ { 1 , 2 , . . . , L -1 } . Therefore, the only possible values of t for which [ ¯ P t ] ( i,ℓ ) , ( i,ℓ ) > 0 are those that are multiples of L . Hence, for any ( i, ℓ ) ∈ S × L ,

Moreover, since P ℓ is aperiodic, gcd { k ∈ Z ≥ 1 : [ P k ℓ ] ii > 0 } = 1 . Substituting in (9), we get that d ( i, ℓ ) = L for all ( i, ℓ ) . Thus, all states have a period of L .

Now, from Prop. 1, we know that ¯ P L = blkdiag( P 0 , . . . , P L -1 ) . Therefore

Consequently, if we start with an initial distribution ¯ ξ 0 such that ¯ ξ 0 ( S ×{ 0 } ) = 1 , then,

where the 0 vectors are of size n . Consequently, Prop. 2 implies that

where ζ ℓ is the ℓ -th place. This completes the proof of (7).

Now consider the function ¯ h : S × L → R defined as ¯ h ( s, ℓ ′ ) = h ( s ) 1 { ℓ ′ = ℓ } . Then, by taking T = KL , we have

where the last equation uses (5) from Thm. 3. Now, (8) follows from observing that mean return time to state ( s, ℓ ) in Markov chain ¯ P is L times the mean-return time to state s in Markov chain P ℓ , which equals 1 / [ ζ ℓ ] s since P ℓ is irreducible and aperiodic. ✷

C Periodic Markov decision processes

Periodic MDPs are a special class time non-stationary MDPs where the dynamics and rewards are periodic. In particular, let M be a time-varying MDP with state space S , action space A , and dynamics and reward at time t given by P t : S × A → ∆( S ) and r t : S × A → R .

As before, we use J t K to denote t mod L and L to denote { 0 , . . . , L -1 } . The MDP M is periodic with period L if there exist ( P ℓ , r ℓ ) , ℓ ∈ L such that for all t :

Periodic MDPs were first considered in [Rii65]. Periodic MDPs may be viewed as stationary MDPs by considering the augmented state ( S t , J t K ) . By this equivalence, it can be shown that there is no loss of optimality in restricting attention to periodic policies. In particular, let ( V 0 , . . . , V L -1 ) denote the fixed point of the following system of equations

Define π ℓ ⋆ ( s ) to be the arg-max or the right hand side of (10). Then the time-varying policy π = ( π 1 , π 2 , . . . ) given by π t = π J t K ⋆ is optimal.

See [Sch16] for a discussion of how to modify standard MDP algorithms to solve periodic dynamic program (10).

D Stochastic Approximation with Markov noise

We now state a generalization of Thm. 3 to stochastic approximation style iterations.

Theorem 4 Let { S t } t ≥ 1 , S , be an irreducible and aperiodic finite Markov chain with unique limiting distribution ζ . Let F t denote the natural filtration w.r.t. { S t } t ≥ 1 and { α t } t ≥ 1 be a non-negative real-valued process adapted to {F t } that satisfies

Let { M t +1 } t ≥ 1 be a square-integrable margingale difference sequence w.r.t. {F t } t ≥ 1 such that E [ M 2 t +1 | F t ] ≤ K (1 + ∥ X t ∥ 2 ) for some constant K . Consider the iterative process { X t } t ≥ 1 , where X 1 is arbitrary and for t ≥ 1 , we have

Then, the sequence { X t } t ≥ 1 converges almost surely to limit. In particular,

Eq. (12) is similar to standard stochastic approximation iteration [RM51; KY97; Bor08], which the 'noise sequence' h ( S t ) is assumed to be a martingale difference sequence. The setting considered above is sometimes referred to as stochastic approximation with Markov noise. In fact, more general version of this result where the noise sequence is allowed to depend on the state X t are typically established in the literature [BMP12; Bor08; KY97; PB24]. For the sake of completeness, we will show that Thm. 4 is a special case of these more-general results.

Before presenting the proof, we point out that Thm. 4 is a generalization of Thm. 3, Eq. (6). In particular, suppose the learning rates are α t = 1 / (1 + t ) . Then, simple algebra shows that

Then, (6) of Thm. 3 implies that the limit is given by the right had side of (13). Therefore, Thm. 4 is a generalization of Thm. 3 to general learning rates which satisfy (11).

PROOF To establish the result, we will show that the iteration { X t } t ≥ 1 satisfies the assumptions for the convergence of stochastic approximation with (state dependent) Markov noise and stochastic recursive inclusions given in [PB24, Theorem 2.7]. The proof is due to [BP24]. In particular, we can rewrite (12) as

where g ( x, s ) = -x + h ( s ) . Moreover, for ease of notation, define ¯ h = ∑ s ∈ S h ( s ) ζ ( s ) . Then, we have

g ( x, s ) is Lipschtiz continuous in the first argument, so A2.14 of [PB24] holds.

From (6), the ergodic occupation measure of { h ( S t ) } t ≥ 1 is { ¯ h } , which is compact and convex. So, A2.15 of [PB24] is satisfied.

The conditions on the martingale noise sequence { M t } t ≥ 1 imply that A2.16 of [PB24] holds.

Eq. (11) is equivalent to A2.17 of [PB24].

To check A2.18 of [PB24], for any measure ν on S , define

Also define

Let ˜ h ∞ ( x, ν ) = lim c →∞ h c ( x, ν ) = x . Thus, the differential inclusion in A2.18(ii) is actually an ODE

which has origin as the unique global asymptotically stable equilibrium point. Thus, A2.18 of [PB24] is satisfied.

Therefore, all assumptions of Theorem 2.7 of [PB24] are satisfied. Therefore, by that result, the iterates { X t } t ≥ 1 converge to solution of the ODE (note that the differential inclusion in Theorem 2.7 of [PB24] is an ODE in our setting)

Note that x = ¯ h is the unique asymptotically stable attractor of the ODE (14). Therefore, Theorem 2.7 of [PB24] implies (13). ✷

Thm. 4 also implies the following generalization of Prop. 3.

̸

Proposition 4 Suppose { S t } t ≥ 1 is a time-periodic Markov chain with period L that satisfies Assm. 3 with the unique limiting distribution { ζ ℓ } ℓ ∈ L . Let {F t } t ≥ 1 denote the natural filtration w.r.t. { S t } t ≥ 1 and { α ℓ t } t ≥ 1 , ℓ ∈ L , be non-negative real-valued processes adapted to {F t } t ≥ 1 such that α ℓ t = 0 when ℓ = J t K and

Let { M t +1 } t ≥ 1 be a square-integrable margingale difference sequence w.r.t. {F t } t ≥ 1 such that E [ M 2 t +1 | F t ] ≤ K (1 + ∥ X t ∥ 2 ) for some constant K . Fix any ℓ ∈ L , Consider the iterative process { X ℓ k } k ≥ 1 , where X 1 is arbitrary and for k ≥ 1 , we have

Then, the sequence { X ℓ t } t ≥ 1 converges almost surely to the following limit

PROOF Note that the learning rates used here can be viewed as the learning rates of L separated stochastic iterations on a common timescale t . Each separate stochastic iteration ℓ ∈ L is actually only updated once every L steps on the timescale t . Because of the condition α ℓ t = 0 when ℓ = J t K , each update is followed by L -1 'pseudo'-updates where the learning rate is 0 . Therefore, each X ℓ is updated only once every L steps on timescale t .

̸

The result then follows immediately from Thm. 4 by considering the process { S t } t ≥ 1 every L steps for each ℓ ∈ L . ✷

E Thm. 1: Convergence of periodic Q-learning

The high-level idea of the proof is similar to [KY22] for ASQL when the agent state is a finite window of past observations and action. The key observation of [KY22] is the following: Consider an iterative process X t +1 = (1 -α t ) X t + α t U t with the learning rates α t = 1 / (1 + t ) . Then, X t +1 = ( X 0 + ∑ t τ =1 U t ) / (1 + t ) . Then, if the process { U t } t ≥ 1 has an ergodic limit (e.g., when { U t } t ≥ 1 is a function of a Markov chain, see Thm. 3), the process { X t } t ≥ 1 converges to the ergodic limit of { U t } t ≥ 1 . We follow a similar idea but with the following changes:

Instead of assuming 'averaging' learning rates (i.e., reciprocal of the number of visits), we allow for general learning rates of Assm. 1.

We account for the fact that that the 'noise' is periodic.

The rest of the analysis then follows along the standard argument of convergence of Qlearning [JSJ94; KY22; DY24].

Define the error function ∆ ℓ t +1 := Q ℓ t +1 -Q ℓ µ , for all ℓ ∈ L . To prove Thm. 1, it suffices to prove that ∥ ∆ ℓ t ∥ → 0 for all ℓ ∈ L , where ∥·∥ is the supremum-norm. The proof proceeds in three steps.

E.1 Step 1: State splitting of the error function

Define V ℓ t ( z ) := max a ∈ A Q ℓ t ( z, a ) and V ℓ µ ( z ) := max a ∈ A Q ℓ µ ( z, a ) , for all ℓ ∈ L , z ∈ Z . We can combine (PASQL), (1), and (2) as follows

where

̸

Note that we have added extra indicator functions in the U ℓ,i t ( z, a ) terms, i ∈ { 0 , 1 } . This does not change the value of α ℓ t ( z, a ) U ℓ,i t ( z, a ) because the learning rates have the property that α ℓ t ( z, a ) = 0 if ( ℓ, z, a ) = ( t , z t , a t ) (see Assm. 1).

J K For each ℓ ∈ L , Eq. (16) may be viewed as a linear system with state ∆ ℓ t +1 and three inputs U ℓ, 0 t , U ℓ, 1 t and U ℓ, 2 t . We exploit the linearity of the system and split the state into three components: ∆ ℓ t +1 = X ℓ, 0 t +1 + X ℓ, 1 t +1 + X ℓ, 2 t +1 , where the three components evolve as follows:

Linearity implies that (16) is equivalent to (17). We will now separately show that ∥ X ℓ, 0 t ∥ → 0 , ∥ X ℓ, 1 t ∥ → 0 and ∥ X ℓ, 2 t ∥ → 0 .

E.2 Step 2: Convergence of component X ℓ, 0 t

Fix ( ℓ, z ◦ , a ◦ ) ∈ L × Z × A and define

Then the process { X ℓ, 0 t ( z ◦ , a ◦ ) } t ≥ 1 is given by the stochastic iteration

which is of the form (15). The process { ( S t , Z t , A t ) } t ≥ 1 is a periodic Markov chain and the learning rates { α ℓ t ( z ◦ , a ◦ ) } t ≥ 1 satisfy the conditions of Prop. 4 due to Assm. 1. Therefore, Prop. 4 implies

that { X ℓ, 0 t ( z ◦ , a ◦ ) } t ≥ 1 converges a.s. to the following limit

Hence, for all ( ℓ, z ◦ , a ◦ ) , the process { X ℓ, 0 t ( z ◦ , a ◦ ) } t ≥ 1 converges to zero almost surely.

E.3 Step 3: Convergence of component X ℓ, 1 t

Let W t denote the tuple ( S t , Z t , A t , S t +1 , Z t +1 , A t +1 ) . Note that { W t } t ≥ 1 is also a periodic Markov chain and converges to a cyclic limiting distribution ¯ ζ ℓ µ , where

We use ¯ ζ µ ℓ ( s, z, a, S , Z , A ) to denote the marginalization over the 'future states' and a similar notation for other marginalizations. Note that ¯ ζ µ ℓ ( s, z, a, S , Z , A ) = ζ ℓ µ ( s, z, a ) .

Fix ( ℓ, z ◦ , a ◦ ) ∈ L × Z × A and define

Then the process { X ℓ, 1 t ( z, a ) } t ≥ 1 is given by the stochastic iteration

which is of the form (15). As argued earlier, the process { W t } t ≥ 1 is a periodic Markov chain. Due to Assm. 1, the learning rate α ℓ t ( z ◦ , a ◦ ) is measurable with respect to the sigma-algebra generated by ( Z 1: t , A 1: t ) and is therefore also measurable with respect to the sigma-algebra generated by W 1: t . Combining this with Prop. 4 implies that the learning rates { α ℓ t ( z ◦ , a ◦ ) } t ≥ 1 satisfy the conditions of Prop. 4. Therefore, Prop. 4 implies that { X ℓ, 1 t ( z ◦ , a ◦ ) } t ≥ 1 converges a.s. to the following limit

where the last step follows from the fact that ¯ ζ ℓ µ ( S , z ◦ , a ◦ , S , Z , A ) = ζ ℓ µ ( z ◦ , a ◦ ) and ¯ ζ ℓ µ ( S , z ◦ , a ◦ , S , z ′ , A ) = ζ ℓ µ ( z ◦ , a ◦ ) P ℓ µ ( z ′ | z ◦ , a ◦ ) .

E.4 Step 4: Convergence of component X ℓ, 2 t

The remaining analysis is similar to corresponding step in the standard convergence proof of Qlearning and its variations [JSJ94; KY22; DY24]. In this section, we use ∥·∥ to denote the supremum norm, i.e., ∥·∥ ∞ .

In the previous step, we have shown that ∥ X ℓ,i t ∥ → 0 a.s., for i ∈ { 0 , 1 } . Thus, we have that ∥ X ℓ, 0 t + X ℓ, 1 t ∥ → 0 a.s. Arbitrarily fix an ϵ > 0 . Therefore, there exists a set Ω 1 of measure one and a constant T ( ω, ϵ ) such that for ω ∈ Ω 1 , all t > T ( ω, ϵ ) , and ( ℓ, z, a ) ∈ L × Z × A , we have

Suppose for some t > T ( ω, ϵ ) , max ℓ ∈ L ∥ X ℓ, 2 t ∥ > Cϵ . Then, for ( z, a ) ∈ Z × A ,

where ( a ) follows from the fact that an upper bound is obtained by maximizing over all realizations of Z t +1 , ( b ) follows from (18), ( c ) follows from the fact that max ℓ ∈ L ∥ X ℓ, 2 t ∥ > Cϵ , ( d ) follows from (19). Thus, for any t > T ( ω, ϵ ) and max ℓ ∈ L ∥ X ℓ, 2 t ∥ > Cϵ , we have

Hence, when max ℓ ∈ L ∥ X ℓ, 2 t ∥ > Cϵ , it decreases monotonically with time. Hence, there are two possibilities: either (i) max ℓ ∈ L ∥ X ℓ, 2 t ∥ always remains above Cϵ ; or (ii) it goes below Cϵ at some stage. We consider these two possibilities separately.

E.4.1 Possibility (i): max ℓ ∈ L ∥ X ℓ, 2 t ∥ always remains above Cϵ

We will show that max ℓ ∈ L ∥ X ℓ, 2 t ∥ cannot remain above Cϵ forever. We first start with a basic result for random iterations. This is a self-contained result, so we reuse some of the variables used in the rest of the paper.

Lemma 2 Let { X t } t ≥ 1 , { Y t } t ≥ 1 , and { α t } t ≥ 1 be non-negative sequences adapted to a filtration {F t } t ≥ 1 that satisfy the following:

Then, the sequence { X t } t ≥ 1 converges to zero almost surely and the sequence { Y t } t ≥ 1 converges to c almost surely.

Now pick a constant C such that where c is a constant. Suppose

PROOF The iteration (21a) implies that

Condition (22) implies that the term in the square brackets converges to zero. Therefore, X t → 0 . Observe that the iteration (21b) can be rewritten as

which is of the form (21a). Therefore, Y t -c → 0 .

✷

We will now prove that max ℓ ∈ L ∥ X ℓ, 2 t ∥ cannot remain above Cϵ forever. The proof is by contradiction. Suppose max ℓ ∈ L ∥ X ℓ, 2 t ∥ remains above Cϵ forever. As argued earlier, this implies that max ℓ ∈ L ∥ X ℓ, 2 t ∥ , t ≥ T ( ω, ϵ ) , is a strictly decreasing sequence, so it must be bounded from above. Let B (0) be such that max ℓ ∈ L ∥ X ℓ, 2 t ∥ ≤ B (0) for all t ≥ T ( ω, ϵ ) . Eq. (20b) implies that ∥ U ℓ, 2 t ∥ < κB (0) . Then, we have that

which implies that max ℓ ∈ L ∥ X ℓ, 2 t ∥ ≤ ∥ M ℓ, (0) t ∥ , where { M ℓ, (0) t } t ≥ T ( ω,ϵ ) is a sequence given by

Lem. 2 implies that M ℓ, (0) t ( z, a ) → κB (0) and hence ∥ M ℓ, (0) t ∥ → κB (0) . Now pick an arbitrary ¯ ϵ ∈ (0 , (1 -κ ) Cϵ ) . Thus, there exists a time T (1) = T (1) ( ω, ϵ, ¯ ϵ ) such that for all t > T (1) , ∥ M ℓ, (0) t ∥ ≤ B (1) := κB (0) + ¯ ϵ . Since max ℓ ∈ L ∥ X ℓ, 2 t ∥ is bounded by ∥ M ℓ, (0) t ∥ , this implies that for all t > T (1) , max ℓ ∈ L ∥ X ℓ, 2 t ∥ ≤ B (1) and, by (20b), ∥ U ℓ, 2 t ∥ ≤ κB (1) . By repeating the above argument, there exists a time T (2) such that for all t ≥ T (2) ,

and so on. By (19), κ < 1 and ¯ ϵ is chosen to be less than Cϵ . So eventually, B ( m ) := κ m B (0) + κ m -1 ¯ ϵ + · · · +¯ ϵ must get below Cϵ for some m , contradicting the assumption that max ℓ ∈ L ∥ X ℓ, 2 t ∥ remains above Cϵ forever.

E.4.2 Possibility (ii): max ℓ ∈ L ∥ X ℓ, 2 t ∥ goes below Cϵ at some stage

Suppose that there is some t > T ( ω, ϵ ) such that max ℓ ∈ L ∥ X ℓ, 2 t ∥ < Cϵ . Then (20a) implies that

where the last inequality uses (19). Therefore,

where the last inequality uses the fact that both ∥ U ℓ, 2 t ∥ and max ℓ ∈ L ∥ X ℓ, 2 t +1 ∥ are both below Cϵ . Thus, we have that

Hence, once max ℓ ∈ L ∥ X ℓ, 2 t +1 ∥ goes below Cϵ , it stays there.

E.4.3 Implication

We have show that for sufficiently large t > T ( ω, ϵ ) , max ℓ ∈ L X ℓ, 2 t ( z, a ) < Cϵ . Since ϵ is arbitrary, this means that for all realizations ω ∈ Ω 1 , max ℓ ∈ L ∥ X ℓ, 2 t ∥ → 0 . Thus,

E.5 Putting everything together

Recall that we defined ∆ ℓ t = Q ℓ t -Q µ and in Step 1 , we split ∆ ℓ t = X ℓ, 0 t + X ℓ, 1 t + X ℓ, 2 t . Steps 2 and 3 together show that ∥ X ℓ, 0 t + X ℓ, 1 t ∥ → 0 , a.s. and Step 3 (23) shows us that max ℓ ∈ L ∥ X ℓ, 2 t ∥ → 0 , a.s. Thus, by the triangle inequality,

which establishes that Q ℓ t → Q µ , a.s.

F Thm. 2: Sub-optimality gap

The high-level idea of proving Thm. 2 is as follows. Thm. 1 shows that PASQL converges to a cyclic limit, which is the solution to a periodic MDP. Thus, the question of characterizing the suboptimality gap is equivalent to the following. Given a PODMP P , let M be a periodic agent-state based model that approximates the reward and the dynamics of P (in the sense of an approximate information state, as defined in [Sub+22]). Let ˆ π ⋆ be the optimal policy of model M . What is the sub-optimality gap when ˆ π ⋆ is used in the original POMDP P ?

To answer such questions, a general framework of approximate information states was developed in [Sub+22] for both finite and infinite horizon models. However, we cannot directly used the results of [Sub+22] because the infinite horizon results there were restricted to stationary policies, while we are interested in the sub-optimality gap of periodic policies.

Nonetheless, Thm. 2 can be proved by building on the existing results of [Sub+22]. In particular, we start by looking at finite horizon model rather than infinite horizon model. Then, as per [Sub+22, Definition 7], the agent state process may be viewed as an approximate information state with approximation errors { ( ε t , δ t ) } t ≥ 1 , where

Let V ⃗ π t,T ( h t ) = E ⃗ π [∑ T τ = t γ τ -1 R τ | h t ] denote the value function of policy ⃗ π for the finite horizon model starting at history h t at time t . Let V ⋆ t,T ( h t ) := sup ⃗ π V ⃗ π t,T ( h t ) denote the optimal value function, where the optimization is over all history dependent policies. Moreover, let ˆ V t,T ( z t ) denote the optimal value function for the periodic MDP model constructed in Thm. 1. Let ⃗ π µ denote the history-based policy defined in Sec. 2.4.

Then, from [Sub+22, Theorem 9] we have

where we set ˆ V T +1 ,T ( z ) ≡ 0 for convenience.

The following hold when we let T →∞ .

Since R t is uniformly bounded, V ⋆ t,T ( h t ) → V ⋆ t ( h t ) as T →∞ .

By the same argument, V ⃗ π µ t,T ( h t ) → V ⃗ π µ t ( h t ) as T →∞ .

By standard results for periodic MDPs (see App. C), ˆ V t,T → V J t K µ as T →∞ .

By definition, ε t ≤ ε J t K t and δ t ≤ δ J t K t .

Therefore, by taking T →∞ in (24), we get

The result then follows from observing that for τ ∈ T ( t, ℓ ) , ϵ ℓ t and δ ℓ t are non-decreasing sequences.

G Policy evaluation of an agent-state based policy

The performance of any agent-state based policy can be evaluated via a slight generalization of 'cross-product MDP' method originally presented in [Pla77]. This method has been rediscovered in slightly different forms multiple times [Lit96; Cas98; Hau97; Han98].

The key intuition is Lem. 1. Thus, for any agent-state based policy, { ( S t , Z t ) } t ≥ 1 is a Markov chain. The only difference in our setting is that the Markov chain is time-periodic. Thus, for any periodic agent-state based policy ( π 0 , . . . , π L -1 ) , we can identify the periodic rewards (¯ r 0 , . . . , ¯ r L -1 ) and periodic dynamics ( ¯ P 0 , . . . , ¯ P L -1 ) (which depend on π but we are not carrying that dependence in our notation) as follows:

We can then evaluate the performance of this time-periodic Markov chain via performance evaluation formulas for periodic MDPs (App. C). In particular, define

to be the L -step cumulative rewards and dynamics for the time-periodic Markov chain. Then define

Thus, ˜ V ( s, z ) gives the performance of periodic policy π when starting at initial state ( s, z ) . If the initial state is stochastic, we can average over the initial distribution.

H Reproducibility information

The hyperparameters for the numerical experiments presented in Sec. 3 are shown in Table 3. The experiments were run on a computer cluster by running jobs that requested 2 -CPU nodes with < 8 GB memory. Each seed typically took less than 10 minutes to execute.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made have been proved theoretically in detail in the appendix. Illustrative examples have also been shown to provide more clarity on the ideas involved.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper includes the limitations in the discussion section (Sec. 5).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the theoretical results in an organized manner with numbered and crossreferenced equations, theorems, lemmas, assumptions etc. We have considered the proofs in rigorous detail and explain all the details in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The information needed to reproduce the main experimental results of the paper are disclosed in further detail in App. G and App. H.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We intend to make the code open access after the review process is complete.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The main details are provided in the paper. Additional details are provided in the appendix (app. H).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The example mentioned in the main text is run with 25 random seeds and the median and interquantile range are plotted.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The relevant information about compute resources is provided in the appendix (App. H).

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with every aspect of the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The research conducted in this paper is foundational research that focuses on developing theoretical results and as such, does not directly have a societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer:[NA]

Justification: Since the paper is based on foundational research, it does not pose any direct risks that require safeguarding.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposes a new RL algorithm for POMDPs. The standard approach to convert the POMDP in a belief MDP is not possible in the RL setting, when there is no knowledge about the system model. The alternative is to use an agent-state which is a function of the observation history. Standard RL algorithms can be adapted to the agent-state to learn policies for POMDPs. However, these standard algorithms lead to stationary policies which might not be optimal, as the agent-state may not satisfy the Markov property.
As a consequence, the authors propose a special form of nonstationary policies that may be better suited for non-Markovian agent-states.

STRENGTHS:
1. The paper has an interesting problem setting and describes well the limitations of stationary policies applied to non-Markovian agent-states.

2. The authors provide an extensive discussion of the related work.

WEAKNESSES:
1. The proposed algorithm leads to a very limited class of policies. Having periodic policies might only help, because several policies mixed might lead to a better strategy than having only one deterministic policy. Apart from that, is there an inductive bias to repeat the policies?

2. Another weakness is, that the paper mainly discusses deterministic policies. It would be more interesting to increase the set of policies to stochastic policies. It could be that a single stochastic policy has the same effect on mixing the policies and therefore the periodic policy has no advantage.

3. The numerical experiments consider only small examples.

QUESTIONS:
Have you considered other non-stationary approaches?

LIMITATIONS:
The authors addressed the limitations adequately.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a type of non-stationary policy for POMDPs that is periodic. The authors argue that typical agent states in partially observable RL do not satisfy the Markov property and illustrate why introducing non-stationarity can improve the optimal policy within the policy class (vs considering only stationary policies), even if this policy cannot achieve global optimality across all history-dependent policies. The claims are supported by several well chosen examples, rigorous theory, and numerical experiments.\n\nI'm happy to strongly recommend acceptance.\n\nSTRENGTHS:\n**Originality**: To my knowledge, the proposal of periodic policies for POMDPs is novel (though I am unfamiliar with theoretical papers in this area). The authors do a good job of presenting the relevant prior literature and what contributions in the work at hand are novel. Since the work generalizes stationary policies (using a period of $L=1$), I appreciated the remarks on how previous theoretical results can be obtained using the new theorems in this work.\n\n**Quality**: The approach is rigorously supported by theory, and some numerical experiments. While I did not verify all the proofs, the theorems seem correct. Throughout the paper, the discussion presented by the authors was extremely insightful.\n\n**Clarity**: The paper is well written and the ideas are clearly communicated. While the writing can be technically dense, this should be expected of a work of this nature. I particularly commend the authors for the example in Figure 1, since it immediately conveys a number of important features of this work in a concise manner. \n\n**Significance**: While it is not surprising that performance in POMDPs can be improved by leveraging non-stationarity (in the absence of belief states), it can be challenging to implement non-stationarity due to the infinite parameter requirement the authors highlight. This work is an interesting proposal of how this can be done in principle with only finite parameters. While it may still not necessarily be practical, I think the ideas are nonetheless valuable.\n\nWEAKNESSES:\n**Originality**: No weaknesses that are not already apparent. \n\n**Quality**: My only major concern is with the numerical experiments. First, the agent models used are rather weak (the agent only considers the current observation when deciding an action). Perhaps some stronger choices that would be more convincing are frame stacking or an approximation of the belief state with added noise.\n\nThe following is a list of limitations. These are mostly stated clearly in the paper and I don't think they constitute reasons to reject.\n- Choosing an appropriate period $L$ is difficult. While it is shown that choosing $L = n!$ monotonically improves, large values of $L$ have large memory requirements. \n- For the above reason, I'm not sure this particular algorithm will see practical use. \n- The choice of behavioural policy is critical.\n- The framing of the problem is limited to deterministic Q-learning. It does not consider stochastic or policy gradient methods. \n\n**Clarity**: A couple of minor issues: \n- In the introductory example, explicitly writing the values of $J^*_{BD}$, $J^*_{SD}$ under $\\gamma = 0.9$ would aid in the comparison  with $J^*_L$. \n- In the numerical examples, is there an explanation for why $u_1$ is a better choice than $u_2$ and $u_3$? \n\n**Significance**: See prior comments on practicality.\n\nQUESTIONS:\n(As mentioned above:)\n\n1. Could you please comment on the relevance of the insights towards larger POMDPs, like those common in deep RL?\n2. In the numerical examples, is there an explanation for why $u_1$ is a better choice than $u_2$ and $u_3$? \n3. I can see why most agent states that aren't the belief state will not satisfy the Markov property. Could you elaborate on some other cases to help clarify this (in the paper)? For example, when the agent state is uninformative (e.g. constant), or when the agent state is an RNN encoding of the history, do these satisfy the Markov property?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the problem of learning a policy in a partially observable environment in a model-free way. The authors address this problem by proposing to learn a non-stationary periodic policy that can outperform stationary ones. This aspect is motivated by the fact that the policy can be constructed either using the last observation or a fixed set of last observations as a state, which in general do not satisfy the Markov property.  \nAn approach based on learning periodic policies using Q-learning is presented and convergence guarantees for this type of policies are presented under some assumptions. \nFinally, a numerical experiment validates the claims and the benefits of using the devised approach.\n\nSTRENGTHS:\nThe writing of the paper is clear. The authors consider the approach of learning periodic policies and are able to extend the convergence guarantees that hold under the stationary case also to periodic policies.\n\nWEAKNESSES:\nI will highlight here some weaknesses of the presented approach.  \n\nThe authors claim in lines 71-73 that \"a non-stationary deterministic agent-state based policy performs better than stationary deterministic agent-state based policies\". This claim is reported here using only the results reported based on example 2, thus lacking a more formal characterization of the statement.\nAs a second note, if we formally consider the set of non-stationary policies as containing the set of stationary ones, the results of the claim are always true since the set of non-stationary policies is more expressive and thus more powerful than the second set.\n\nTheorem 1 addresses the main challenge of the convergence analysis of PASQL that is the non-markovianity of the agent state. However the same challenge is already faced, under stationary policies (for ASQL), in [1]. Since the same result was also proved in [1], this reduces the contribution of the presented statement which seems to extend it to the set of periodic policies.\n\nIt is not clear what in general the benefits of using periodic policies are and, even though the authors show some examples where they are effective, it is in general not easy to choose the right value of the period $L$. Nonetheless, as stated by the authors, an increase in $L$ does not necessarily reduce the sub-optimality of the policy. Its increase however leads to an increase in the complexity of finding a good estimate of the Q-function for each $l \\in L$. Furthermore, no indication is given on the choice of the behavioral policies that lead to the convergence to good policies. \n\n\n[1] Erfan SeyedSalehi, Nima Akbarzadeh, Amit Sinha, and Aditya Mahajan. “Approximate information state based convergence analysis of recurrent Q-learning”. In: European conference on reinforcement learning. 2023.\n\nQUESTIONS:\nIt is not clear to me the point (ii) in line 80. In particular, what does it mean that \"the set of all periodic deterministic agent-state based policies of period L is not monotonically increasing\"? It is also not clear to me why periodic policies with period specified as in line 83 have monotonically increasing performance.\n\nIt is not clear why the authors state that the main challenge in this setting is the absence of the Markov property of the agent state $[Z_t]_{t\\ge 1}$, however Lemma 1 defines the process (including the agent state $\\{Z_t\\}$) as a Markov chain. Another questions is: how is Lemma 1 derived? Is it an assumption?\n\nThe authors claim that previous results hold under a restrictive assumption on the learning rates but they do not state what these assumptions are. Since this aspect is claimed as an improvement over previous work, it would be useful to describe the previous assumptions.  \n\nThe convergence guarantees of PASQL hold under the assumption that the Markov chain is periodic. How difficult is this assumption in practice? Would in this case the period of the optimal policy coincide with the periodicity of the chain?  \nFurthermore, in section 5, the authors claim that, differently from previous results which assume the irreducibility and the aperiodicity of the Markov chain, the current work assumes that the chain is periodic which is \"a weaker assumption\". I do not get why this assumption is weaker since it appears that the irreduciblity and aperiodicity assumption should also hold for each $l \\in L$ as appears in Assumption 3 in the appendix. Could the authors clarify this aspect?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces PASQL (Periodic Agent-State Based Q-Learning), a novel reinforcement learning approach tailored for Partially Observable Markov Decision Processes (POMDPs). Traditional methods often rely on transforming POMDPs into fully observable MDPs by employing belief states. However, belief states require system model knowledge, limiting their practicality in model-free settings like RL. PASQL addresses this by learning non-stationary, periodic policies using agent states—compressed representations of history—which do not assume the Markov property. This method leverages periodic Markov chains and stochastic approximation theories to prove that PASQL converges to a cyclic limit and can outperform traditional stationary policies in terms of approximation error.\n\nSTRENGTHS:\n1. PASQL creatively combines concepts from periodic Markov processes and Q-learning to handle non-Markovian dynamics, a significant departure from traditional RL methods.\n2. The paper provides a rigorous theoretical framework, including convergence proofs and bounds on sub-optimality, which substantiates the claims made about PASQL's efficacy.\n3. Through numerical experiments, the paper effectively demonstrates that PASQL can outperform standard Q-learning in POMDP settings, offering practical relevance to the theoretical claims.\n4. The paper offers a comprehensive comparison with existing methods, highlighting PASQL’s advantages in handling agent states that do not satisfy the Markov property.\n\nWEAKNESSES:\n1. The introduction of periodic policies increases complexity and may require more computational resources, which could limit practical applications.\n2. The current study focuses on tabular representations, which might not scale well to high-dimensional problems or those requiring function approximation.\n\nQUESTIONS:\n1. How does PASQL scale with the dimensionality of the state and action spaces in practical scenarios?\n2. On what basis were the specific periods chosen for the policies in the experiments, and how sensitive is PASQL's performance to these choices?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}