SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pasadena, CA, USA
Emile Anand
eanand@caltech.edu
Yisong Yue
yyue@caltech.edu
James A. Preiss
japreiss@caltech.edu
Yingying Li
Urbana, IL, USA
yl101@illinois.edu
We study online adaptive policy selection in systems with time-varying costs and dynamics. We develop the Gradient-based Adaptive Policy Selection (GAPS) algorithm together with a general analytical framework for online policy selection via online optimization. Under our proposed notion of contractive policy classes, we show that GAPS approximates the behavior of an ideal online gradient descent algorithm on the policy parameters while requiring less information and computation. When convexity holds, our algorithm is the first to achieve optimal policy regret. When convexity does not hold, we provide the first local regret bound for online policy selection. Our numerical experiments show that GAPS can adapt to changing environments more quickly than existing benchmarks.
We study the problem of online adaptive policy selection for nonlinear time-varying discrete-time dynamical systems. The dynamics are given by  $x_{t + 1} = g_t(x_t,u_t)$ , where  $x_{t}$  is the state and  $u_{t}$  is the control input at time  $t$ . The policy class is a time-varying mapping  $\pi_t$  from the state  $x_{t}$  and a policy parameter  $\theta_{t}$  to a control input  $u_{t}$ . At every time step  $t$ , the online policy incurs a stage cost  $c_{t} = f_{t}(x_{t},u_{t})$  that depends on the current state and control input. The goal of policy selection is to pick the parameter  $\theta_{t}$  online to minimize the total stage costs over a finite horizon  $T$ .
Online adaptive policy selection and general online control have received significant attention recently [1-8] because many control tasks require running the policy on a single trajectory, as opposed to restarting the episode to evaluate a different policy from the same initial state. Adaptivity is also important when the dynamics and cost functions are time-varying. For example, in robotics, time-varying dynamics arise when we control an aircraft under changing wind conditions [9].
†This work is supported by NSF Grants CNS-2146814, CPS-2136197, CNS-2106403, NGSDI-2105648, CCF-1918865, and Gift from Latitude AI, with additional support for Yiheng Lin provided by Amazon AI4Science Fellowship and PIMCO Graduate Fellowship in Data Science.
In this paper, we are interested in developing a unified framework that can leverage a broad suite of theoretical results from online optimization and efficiently translate them to online policy selection, where efficiency includes both preserving the tightness of the guarantees and computational considerations. A central issue is that, in online policy selection, the stage cost  $c_{t}$  depends on all previously selected parameters  $(\theta_0,\dots ,\theta_{t - 1})$  via the state  $x_{t}$ . Many prior works along this direction have addressed this issue by finite-memory reductions. This approach led to the first regret bound on online policy selection, but the bounds are not tight, the computational cost can be large, and the dynamics and policy classes studied are restrictive [136-8].
Contributions. We propose and analyze the algorithm Gradient-based Adaptive Policy Selection (GAPS, Algorithm) to address three limitations of existing results on online policy selection. First, under the assumption that  $c_t$  is a convex function of  $(\theta_0, \dots, \theta_t)$ , prior work left a  $\log T$  regret gap between OCO and online policy selection. We close this gap by showing that GAPS achieves the optimal regret of  $O(\sqrt{T})$  (Theorem 3.3). Second, many previous approaches require oracle access to the dynamics/costs and expensive resimulation from imaginary previous states. In contrast, GAPS only requires partial derivatives of the dynamics and costs along the visited trajectory, and computes  $O(\log T)$  matrix multiplications at each step. Third, the application of existing regret analysis frameworks is limited to specific policy classes and systems because they require  $c_t$  to be convex in  $(\theta_0, \dots, \theta_t)$ . We address this limitation by showing the first local regret bound for online policy selection when the convexity does not hold. Specifically, GAPS achieves the local regret of  $O(\sqrt{(1 + V)T})$ , where  $V$  is a measure of how much  $(g_t, f_t, \pi_t)$  changes over the entire horizon.
To derive these performance guarantees, we develop a novel proof framework based on a general exponentially decaying, or "contractive", perturbation property (Definition 2.6) on the policy-induced closed-loop dynamics. This generalizes a key property of disturbance-action controllers [e.g. 118] and includes other important policy classes such as model predictive control (MPC) [e.g. 11] and linear feedback controllers [e.g. 12]. Under this property, we prove an approximation error bound (Theorem 3.2), which shows that GAPS can mimic the update of an ideal online gradient descent (OGD) algorithm 13 that has oracle knowledge of how the current policy parameter  $\theta_{t}$  would have performed if used exclusively over the whole trajectory. This error bound bridges online policy selection and online optimization, which means regret guarantees on OGD for online optimization can be transferred to GAPS for online policy selection.
In numerical experiments, we demonstrate that GAPS can adapt faster than an existing follow-the-leader-type baseline in MPC with imperfect disturbance predictions, and outperforms a strong optimal control baseline in a nonlinear system with non-i.i.d. disturbances. The source code for all experiments is published at https://www.github.com/jpreiss/adaptive_policy_selection.
Related Work. Our work is related to online control and adaptive-learning-based control [2, 14-19], especially online control with adversarial disturbances and regret guarantees [1, 6-8, 20, 21]. For example, there is a rich literature on policy regret bounds for time-invariant dynamics [1, 6, 8, 15, 20, 22]. There is also a growing interest in algorithms for time-varying systems with small adaptive regret [7, 21], dynamic regret [23-25], and competitive ratio [26-29]. Many prior works study a specific policy class called disturbance-action controller (DAC) [1, 3, 6-8]. When applied to linear dynamics  $g_{t}$  with convex cost functions  $f_{t}$ , DAC renders the stage cost  $c_{t}$  a convex function in past policy parameters  $(\theta_0,\dots ,\theta_t)$ . Our work contributes to the literature by proposing a general contractive perturbation property that includes DAC as a special case, and showing local regret bounds that do not require  $c_{t}$  to be convex in  $(\theta_0,\ldots ,\theta_t)$ . A recent work also handles nonconvex  $c_{t}$ , but it studies an episodic setting and requires  $c_{t}$  to be "nearly convex", which holds under its policy class [30].
In addition to online control, this work is also related to online learning/optimization [13, 31, 32], especially online optimization with memory and/or switching costs, where the cost at each time step depends on past decisions. Specifically, our online adaptive policy selection problem is related to online optimization with memory [29, 33-39]. Our analysis for GAPS provides insight on how to handle indefinite memory when the impact of a past decision decays exponentially with time.
Our contractive perturbation property and the analytical framework based on this property are closely related to prior works on discrete-time incremental stability and contraction theory in nonlinear systems [40-46], as well as works that leverage such properties to derive guarantees for (online) controllers [47-49]. In complicated systems, it may be hard to design policies that provably satisfy
1 A concurrent work [10] also closes this gap in a less general setting. See Section 3.2 for a discussion.
Figure 1: Diagram of the causal relationships between states, policy parameters, control inputs, and costs.
these properties. This motivates some recent works to study neural-based approaches that can learn a controller together with its certificate for contraction properties simultaneously [50, 51]. Our work contributes to this field by showing that, when the system satisfies the contractive perturbation property, one can leverage this property to bridge online policy selection with online optimization.
Notation. We use  $[t_1:t_2]$  to denote the sequence  $(t_{1},\ldots ,t_{2})$ $a_{t_1:t_2}$  to denote  $(a_{t_1},a_{t_1 + 1},\dots ,a_{t_2})$  for  $t_1\leq t_2$  , and  $a_{\times \tau}$  for  $(a,\ldots ,a)$  with  $a$  repeated  $\tau \geq 0$  times. We define  $q(x,Q) = x^{\top}Qx$  Symbols 1 and 0 denote the all-one and all-zero vectors/matrices respectively, with dimension implied by context. The Euclidean ball with center 0 and radius  $R$  in  $\mathbb{R}^n$  is denoted by  $B_{n}(0,R)$  We let  $\| \cdot \|$  denote the (induced) Euclidean norm for vectors (matrices). The diameter of a set  $\Theta$  is  $\mathrm{diam}(\Theta)\coloneqq \sup_{x,y\in \Theta}\| x - y\|$  . The projection onto the set  $\Theta$  is  $\Pi_{\Theta}(x) = \arg \min_{y\in \Theta}\| y - x\|$
We consider online policy selection on a single trajectory. The setting is a discrete-time dynamical system with state  $x_{t} \in \mathbb{R}^{n}$  for time index  $t \in \mathcal{T} \coloneqq [0: T - 1]$ . At time step  $t \in \mathcal{T}$ , the policy picks a control action  $u_{t} \in \mathbb{R}^{m}$ , and the next state and the incurred cost are given by:
$$
\text {D y n a m i c s :} x _ {t + 1} = g _ {t} \left(x _ {t}, u _ {t}\right), \quad \text {C o s t :} c _ {t} := f _ {t} \left(x _ {t}, u _ {t}\right),
$$
respectively, where  $g_{t}(\cdot ,\cdot)$  is a time-varying dynamics function and  $f_{t}(\cdot ,\cdot)$  is a time-varying stage cost. The goal is to minimize the total cost  $\sum_{t = 0}^{T - 1}c_t$ .
We consider parameterized time-varying policies of the form of  $u_{t} = \pi_{t}(x_{t},\theta_{t})$ , where  $x_{t}$  is the current state at time step  $t$  and  $\theta_t \in \Theta$  is the current policy parameter.  $\Theta$  is a closed convex subset of  $\mathbb{R}^d$ . We assume the dynamics, cost, and policy functions  $\{g_t,f_t,\pi_t\}_{t\in \mathcal{T}}$  are oblivious, meaning they are fixed before the game begins. The online policy selection algorithm optimizes the total cost by selecting  $\theta_{t}$  sequentially. We illustrate how the policy parameter sequence  $\theta_{0:T - 1}$  affects the trajectory  $\{x_{t},u_{t}\}_{t\in \mathcal{T}}$  and per-step costs  $c_{0:T - 1}$  in Figure 1. The online algorithm has access to the partial derivatives of the dynamics  $f_{t}$  and cost  $g_{t}$  along the visited trajectory, but does not have oracle access to the  $f_{t},g_{t}$  for arbitrary states and actions.
We provide two motivating examples for our setting. Appendix H contains more details and a third example. The first example is MPC with confidence coefficients, a generalization of [38].
Example 2.1 (MPC with Confidence Coefficients). Consider a linear time-varying (LTV) system  $g_{t}(x_{t},u_{t}) = A_{t}x_{t} + B_{t}u_{t} + w_{t}$ , with time-varying costs  $f_{t}(x_{t},u_{t}) = q(x_{t},Q_{t}) + q(u_{t},R_{t})$ . At time  $t$ , the policy observes  $\{A_{t:t+k-1},B_{t:t+k-1},Q_{t:t+k-1},R_{t:t+k-1},w_{t:t+k-1|t}\}$ , where  $w_{\tau|t}$  is a (noisy) prediction of the future disturbance  $w_{\tau}$ . Then,  $\pi_{t}(x_{t},\theta_{t})$  commits the first entry of
$$
\underset {u _ {t: t + k - 1 \mid t}} {\arg \min } \sum_ {\tau = t} ^ {t + k - 1} f _ {\tau} \left(x _ {\tau \mid t}, u _ {\tau \mid t}\right) + q \left(x _ {t + k \mid t}, \tilde {Q}\right) \tag {1}
$$
$$
\text {s . t .} x _ {t \mid t} = x _ {t}, \quad x _ {\tau + 1 \mid t} = A _ {\tau} x _ {\tau \mid t} + B _ {\tau} u _ {\tau \mid t} + \lambda_ {t} ^ {[ \tau - t ]} w _ {\tau \mid t}: t \leq \tau <   t + k,
$$
where  $\theta_t = (\lambda_t^{[0]},\lambda_t^{[1]},\dots,\lambda_t^{[k - 1]}),\Theta = [0,1]^k$  and  $\tilde{Q}$  is a fixed positive-definite matrix. Intuitively,  $\lambda_t^{[i]}$  represents our level of confidence in the disturbance prediction  $i$  steps into the future at time step  $t$ , with entry 1 being fully confident and 0 being not confident at all.
The second example studies a nonlinear control model motivated by [12, 47].
Example 2.2 (Linear Feedback Control in Nonlinear Systems). Consider a time-varying nonlinear control problem with dynamics  $g_{t}(x_{t},u_{t}) = Ax_{t} + Bu_{t} + \delta_{t}(x_{t},u_{t})$  and costs  $f_{t}(x_{t},u_{t}) = q(x_{t},Q) +$
$q(u_{t},R)$ . Here, the nonlinear residual  $\delta_t$  comes from linearization and is assumed to be sufficiently small and Lipschitz. Inspired by [12], we construct an online policy based on the optimal controller  $u_{t} = -\bar{K} x_{t}$  for the linear-quadratic regulator LQR(A,B,Q,R). Specifically, we let  $\pi_t(x_t,\theta_t) = -K(\theta_t)x_t$  where  $K$  is a mapping from  $\Theta$  to  $\mathbb{R}^{n\times m}$  such that  $\| K(\theta_t) - \bar{K}\|$  is uniformly bounded.
2.1 Policy Class and Performance Metrics
In our setting, the state  $x_{t}$  at time  $t$  is uniquely determined by the combination of 1) a state  $x_{\tau}$  at a previous time  $\tau < t$ , and 2) the parameter sequence  $\theta_{\tau:t-1}$ . Similarly, the cost at time  $t$  is uniquely determined by  $x_{\tau}$  and  $\theta_{\tau:t}$ . Since we use these properties often, we introduce the following notation.
Definition 2.3 (Multi-Step Dynamics and Cost). The multi-step dynamics  $g_{t|\tau}$  between two time steps  $\tau \leq t$  specifies the state  $x_{t}$  as a function of the previous state  $x_{\tau}$  and previous policy parameters  $\theta_{\tau:t-1}$ . It is defined recursively, with the base case  $g_{\tau|\tau}(x_{\tau}) \coloneqq x_{\tau}$  and the recursive case
$$
g _ {t + 1 | \tau} (x _ {\tau}, \theta_ {\tau : t}) = g _ {t} \left(z _ {t}, \pi_ {t} \left(z _ {t}, \theta_ {t}\right)\right), \forall t \geq \tau ,
$$
in which  $z_{t} \coloneqq g_{t|\tau}(x_{\tau}, \theta_{\tau:t-1})$ . The multi-step cost  $f_{t|\tau}$  specifies the cost  $c_{t}$  as function of  $x_{\tau}$  and  $\theta_{\tau:t}$ . It is defined as  $f_{t|\tau}(x_{\tau}, \theta_{\tau:t}) \coloneqq f_{t}(z_{t}, \pi_{t}(z_{t}, \theta_{t}))$ .
In this paper, we frequently compare the trajectory of our algorithm against the trajectory that would arise from applying a fixed parameter  $\theta$  since time step 0, which we denote as  $\hat{x}_t(\theta) \coloneqq g_{t|0}(x_0, \theta_{\times t})$  and  $\hat{u}_t(\theta) \coloneqq \pi_t(\hat{x}_t(\theta), \theta)$ . A related concept that is heavily used is the surrogate cost  $F_t$ , which maps a single policy parameter to a real number.
Definition 2.4 (Surrogate Cost). The surrogate cost function is defined as  $F_{t}(\theta) \coloneqq f_{t}(\hat{x}_{t}(\theta), \hat{u}_{t}(\theta))$ .
Figure 1 shows the overall causal structure, from which these concepts follow.
To measure the performance of an online algorithm, we adopt the objective of adaptive policy regret, which has been used by [7, 52]. It is a stronger benchmark than the static policy regret [1, 6] and is more suited to time-varying environments. We use  $\{x_{t},u_{t},\theta_{t}\}_{t\in \mathcal{T}}$  to denote the trajectory of the online algorithm throughout the paper. The adaptive policy regret  $R^A (T)$  is defined as the maximum difference between the cost of the online policy and the cost of the optimal fixed-parameter policy over any sub-interval of the whole horizon  $\mathcal{T}$ , i.e.,
$$
R ^ {A} (T) := \max  _ {I = \left[ t _ {1}: t _ {2} \right] \subseteq \mathcal {T}} \left(\sum_ {t \in I} f _ {t} \left(x _ {t}, u _ {t}\right) - \inf  _ {\theta \in \Theta} \sum_ {t \in I} F _ {t} (\theta)\right). \tag {2}
$$
In contrast, the (static) policy regret defined in [11, 6] restricts the time interval  $I$  to be the whole horizon  $\mathcal{T}$ . Thus, a bound on adaptive regret is strictly stronger than the same bound on static regret. Adaptive regret is particularly useful in time-varying environments like Examples [2.1] and [2.2] because an online algorithm must adapt quickly to compete against a comparator policy parameter that can change indefinitely with every time interval [32]. Section 10.2].
In the general case when surrogate costs  $F_{0:T-1}$  are nonconvex, it is difficult (if not impossible) for online algorithms to achieve meaningful guarantees on classic regret metrics like  $R^{A}(T)$  or static policy regret because they do not have oracle optimization solvers or even the exact knowledge of the surrogate costs. Therefore, we introduce the metric of local regret, which bounds the sum of squared gradient norms over the whole horizon:
$$
R ^ {L} (T) := \sum_ {t = 0} ^ {T - 1} \| \nabla F _ {t} \left(\theta_ {t}\right) \| ^ {2}. \tag {3}
$$
Similar metrics have been adopted by previous works on online nonconvex optimization [53]. Intuitively,  $R^L (T)$  measures how well the online agent chases the (changing) stationary point of the surrogate cost sequence  $F_{0:T - 1}$ . Since the surrogate cost functions are changing over time, the bound on  $R^L (T)$  will depend on how much the system  $\{g_t,f_t,\pi_t\}_{t\in \mathcal{T}}$  changes over the whole horizon  $\mathcal{T}$ . We defer the details to Section 3.3.
2.2 Contractive Perturbation and Stability
In this section, we introduce two key properties needed for our sub-linear regret guarantees in adaptive online policy selection. We define both with respect to trajectories generated by "slowly" time-varying parameters, which are easier to analyze than arbitrary parameter sequences.
${}^{2}{z}_{t}$  is an auxiliary variable to denote the state at  $t$  under initial state  ${x}_{\tau }$  and parameters  ${\theta }_{\tau  : t}$  .
Definition 2.5. We denote the set of policy parameter sequences with  $\varepsilon$ -constrained step size by
$$
S _ {\varepsilon} \left(t _ {1}: t _ {2}\right) := \left\{\theta_ {t _ {1}: t _ {2}} \in \Theta^ {t _ {2} - t _ {1} + 1} \mid \| \theta_ {\tau + 1} - \theta_ {\tau} \| \leq \varepsilon , \forall \tau \in \left[ t _ {1}: t _ {2} - 1 \right] \right\}.
$$
The first property we require is an exponentially decaying, or "contractive", perturbation property of the closed-loop dynamics of the system with the policy class. We now formalize this property.
Definition 2.6 ( $\varepsilon$ -Time-varying Contractive Perturbation). The  $\varepsilon$ -time-varying contractive perturbation property holds for  $R_{C} > 0, C > 0, \rho \in (0,1)$ , and  $\varepsilon \geq 0$  if, for any  $\theta_{\tau : t-1} \in S_{\varepsilon}(\tau : t-1)$ ,
$$
\left\| g _ {t | \tau} \left(x _ {\tau}, \theta_ {\tau : t - 1}\right) - g _ {t | \tau} \left(x _ {\tau} ^ {\prime}, \theta_ {\tau : t - 1}\right) \right\| \leq C \rho^ {t - \tau} \| x _ {\tau} - x _ {\tau} ^ {\prime} \|
$$
holds for arbitrary  $x_{\tau}, x_{\tau}^{\prime} \in B_{n}(0, R_{C})$  and time steps  $\tau \leq t$ .
Intuitively,  $\varepsilon$ -time-varying contractive perturbation requires two trajectories starting from different states (in a bounded ball) to converge towards each other if they adopt the same slowly time-varying policy parameter sequence. We call the special case of  $\varepsilon = 0$  time-invariant contractive perturbation, meaning the policy parameter is fixed. Although it may be difficult to verify the time-varying property directly since it allows the policy parameters to change, we show in Lemma [2.8] that time-invariant contractive perturbation implies that the time-varying version also holds for some small  $\varepsilon > 0$ .
The time-invariant contractive perturbation property is closely related to discrete-time incremental stability [e.g. 45] and contraction theory [e.g. 46], which have been studied in control theory. While some specific policies including DAC and MPC satisfy  $\varepsilon$ -time-varying contractive perturbation globally in linear systems, in other cases it is hard to verify. Our property is local and thus is easier to establish for broader applications in nonlinear systems (e.g., Example 2.2).
Besides contractive perturbation, another important property we need is the stability of the policy class, which requires  $\pi_{0:T - 1}$  can stabilize the system starting from the zero state as long as the policy parameter varies slowly. This property is stated formally below:
Definition 2.7 ( $\varepsilon$ -Time-varying Stability). The  $\varepsilon$ -time-varying stability property holds for  $R_S > 0$  and  $\varepsilon \geq 0$  if, for any  $\theta_{\tau : t-1} \in S_\varepsilon(\tau : t-1)$ ,  $\left\| g_{t|\tau}(0, \theta_{\tau : t-1}) \right\| \leq R_S$  holds for any time steps  $t \geq \tau$ .
Intuitively,  $\varepsilon$ -time-varying stability guarantees that the policy class  $\pi_{0:T-1}$  can achieve stability if the policy parameters  $\theta_{0:T-1}$  vary slowly. Similarly to contractive perturbation, one only needs to verify time-invariant stability (i.e.,  $\varepsilon = 0$  and the policy parameter is fixed) to claim time-varying stability holds for some strictly positive  $\varepsilon$  (see Lemma 2.8). The reason we still use the time-varying contractive perturbation and stability in our assumptions is that they hold for  $\varepsilon = +\infty$  in some cases, including DAC and MPC with confidence coefficients. Applying Lemma 2.8 for those systems will lead to a small, overly pessimistic  $\varepsilon$ .
2.3 Key Assumptions
We make two assumptions about the online policy selection problem to achieve regret guarantees.
Assumption 2.1. The dynamics  $g_{0:T-1}$ , policies  $\pi_{0:T-1}$ , and costs  $f_{0:T-1}$  are differentiable at every time step and satisfy that, for any convex compact sets  $\mathcal{X} \subseteq \mathbb{R}^n$ ,  $\mathcal{U} \subseteq \mathcal{R}^m$ , one can find Lipschitzness/smoothness constants (can depend on  $\mathcal{X}$  and  $\mathcal{U}$ ) such that:
1. The dynamics  $g_{t}(x,u)$  is  $(L_{q,x},L_{q,u})$ -Lipschitz and  $(\ell_{q,x},\ell_{q,u})$ -smooth in  $(x,u)$  on  $\mathcal{X} \times \mathcal{U}$ .
2. The policy function  $\pi_t(x,\theta)$  is  $(L_{\pi ,x},L_{\pi ,\theta})$  -Lipschitz and  $(\ell_{\pi ,x},\ell_{\pi ,\theta})$  -smooth in  $(x,\theta)$  on  $\mathcal{X}\times \Theta$
3. The stage cost function  $f_{t}(x,u)$  is  $(L_f, L_f)$ -Lipschitz and  $(\ell_{f,x}, \ell_{f,u})$ -smooth in  $(x,u)$  on  $\mathcal{X} \times \mathcal{U}$ .
Assumption [2.1] is general because we only require the Lipschitzness/smoothness of  $g_{t}$  and  $f_{t}$  to hold for bounded states/actions within  $\mathcal{X}$  and  $\mathcal{U}$ , where the coefficients may depend on  $\mathcal{X}$  and  $\mathcal{U}$ . Similar assumptions are common in the literature of online control/optimization [25, 29, 47].
Our second assumption is on the contractive perturbation and the stability of the closed-loop dynamics induced by a slowly time-varying policy parameter sequence.
Assumption 2.2. Let  $\mathcal{G}$  denote the set of all possible dynamics/policy sequences  $\{g_t, \pi_t\}_{t \in \mathcal{T}}$  the environment/policy class may provide. For a fixed  $\varepsilon \in \mathbb{R}_{\geq 0}$ , the  $\varepsilon$ -time-varying contractive perturbation (Definition 2.6) holds with  $(R_C, C, \rho)$  for any sequence in  $\mathcal{G}$ . The  $\varepsilon$ -time-varying stability
3This property is standard in online control and is satisfied by DAC [1] [3] [6-8] as well as Examples 2.1 & 2.2
(Definition 2.7) holds with  $R_S < R_C$  for any sequence in  $\mathcal{G}$ . We assume that the initial state satisfies  $\| x_0 \| < (R_C - R_S) / C$ . Further, we assume that if  $\{g, \pi\}$  is the dynamics/policy at an intermediate time step of a sequence in  $\mathcal{G}$ , then the time-invariant sequence  $\{g, \pi\}_{\times T}$  is also in  $\mathcal{G}$ .