SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CSAIL
The problem of piecewise affine (PWA) regression and planning is of foundational importance to the study of online learning, control, and robotics, where it provides a theoretically and empirically tractable setting to study systems undergoing sharp changes in the dynamics. Unfortunately, due to the discontinuities that arise when crossing into different "pieces," learning in general sequential settings is impossible and practical algorithms are forced to resort to heuristic approaches. This paper builds on the recently developed smoothed online learning framework and provides the first algorithms for prediction and simulation in PWA systems whose regret is polynomial in all relevant problem parameters under a weak smoothness assumption; moreover, our algorithms are efficient in the number of calls to an optimization oracle. We further apply our results to the problems of one-step prediction and multi-step simulation regret in piecewise affine dynamical systems, where the learner is tasked with simulating trajectories and regret is measured in terms of the Wasserstein distance between simulated and true data. Along the way, we develop several technical tools of more general interest.
A central problem in the fields of online learning, control, and robotics is how to efficiently plan through piecewise-affine (PWA) systems. Such systems are described by a finite set of disjoint regions in state-space, within which the dynamics are affine. In this paper, we consider the related problem of learning to make predictions in PWA systems when the dynamics are unknown. While recent years have seen considerable progress around our understanding of linear control systems, the vast majority of dynamics encountered in practical settings involve nonlinearities. Learning and predicting in nonlinear systems is typically far more challenging because, unlike their linear counterparts, the dynamics in a local region of a nonlinear system do not determine its global dynamics.
PWA systems allow for discontinuities across the separate regions ("pieces"), and are thus a simplified way of modeling rich phenomena that arise in numerous robotic manipulation and locomotion tasks [41, 64, 40], such as modeling dynamics involving contact. In addition, deep neural networks with ReLU activation [24] are PWA systems, providing further motivation for their study. Already, there is a computational challenge simply in optimizing an objective over these systems, which is the subject of much of the previous literature. Here, we take a statistical perspective, assuming that we have access to effective heuristic algorithms for this optimization task, as is common in online learning [29, 9, 28]. Uniformly accurate learning of the dynamics across all pieces is typically impossible
because some regions of space require onerous exploration to locate; thus, we instead consider an iterative prediction setting, where at each time  $t$ , the learner predicts the subsequent system state. From this, we extend to prediction of entire trajectories: over episodes  $t = 1,\dots ,T$ , a learner suggests a policy  $\pi_t$ , and the learner attempts to learn the dynamics so as to minimize prediction error along the trajectory  $\pi_{\tau}$  induces. This is motivated by iterative learning control, where our notion of error is an upper bound on the discrepancy between the planner's estimate of policy performance and the ground truth.
Our iterative formulation of the learning problem is equivalent to the regret metric favored by the online learning community, and which has seen application to online/adaptive control and filtering problems in recent work. Critically, regret does not require uniform identification of system parameters, which is typically impossible. The key pathology is that policies can oscillate across the boundary of two regions, accruing significant prediction error due to sharp discontinuities of the dynamics between said regions, a problem well-known to the online learning community [36]. Moreover, this pathology is not merely a theoretical artifact: discontinuities and non-smoothness pose significant challenges to planning and modeling contact-rich dynamics such as those encountered in robotic manipulation [58, 59, 46].
Our solution takes inspiration from, and establishes a connection between, two rather different fields. Recent work in the robotics and control communities has relied on randomized smoothing to improve planning across discontinuous and non-smooth dynamics [58, 59, 35, 44]. Additionally, the online learning community has studied smoothed online learning [49, 27, 28, 9, 7, 6], which circumvents the threshold-effect pathologies described above. We show that, if the dynamics and the control inputs are subject to randomized smoothing, low regret becomes achievable. We note that the randomized smoothing approach is in some sense canonical in mitigating the aforementioned pathology of policies that oscillate across the boundaries of two regions; smoothing prohibits this pathology by ensuring that the system is generally far from these boundaries. More importantly, our proposed no-regret algorithm is efficient in terms of the number of calls to an optimization oracle, a popular notion of computational efficiency in the online learning community [29, 34, 9, 28]. In our setting, the optimization oracle required finds the best-fit PWA system to a batch of given data. Though this problem is intractable [5], there is a rich literature of popular heuristics [23, 45, 19]. Unlike those works, we examine the statistical challenge of generalization to novel data given such an oracle. We remark that in many practical cases, our smoothness assumption comes for free, in the sense that the Gaussian noise used to smooth gradients in Suh et al. [58] is already sufficient for our results to hold.
Contributions. A formal description of our setting is deferred to Section 2. Informally, we consider the problem of prediction in a PWA system over a horizon of  $T$  steps, where the regions are determined by intersections of halfspaces; we obtain prediction in PWA dynamical systems as a special case. We aim to achieve sublinear-in-  $T$  excess square-loss error (regret) of  $O\left(T^{1 - \alpha}\right)$  with respect to the optimal predictor which knows the system dynamics, where  $\alpha >0$  is constant and the prefactor on the regret is at most polynomial in all of the problem parameters. Our result is derived from a general guarantee for online piecewise affine regression, which subsumes the online PWA identification setting. We show that, when the dynamics and the control inputs are subject to randomized smoothing satisfying the  $\sigma_{\mathrm{dir}}$ -directional smoothness condition introduced by [7], our regret bound is polynomial in  $\sigma_{\mathrm{dir}}$ , dimension, and other natural problem parameters. While the exact dependence on horizon may be far from optimal, we emphasize that our algorithm is the first to achieve regret even polynomial in all parameters in the difficult setting of PWA systems. Moreover, our work provides the first regret bound with polynomial dependence in dimension for oracle-efficient smoothed online learning in the sense of Hazan and Koren [29] of a noncontinuous function class without a realizability assumption and allowing for process noise. As a further application, we adapt our algorithm to make predictions, at each time  $t$ , of the trajectory comprising the next  $H$  steps in the evolution of the dynamical system. We bound the simulation regret, the cumulative error in Wasserstein distance between the distribution of our proposed trajectory and that of the actual trajectory. Assuming a Lyapunov condition, we demonstrate that our modified algorithm achieves simulation regret  $O\left(\mathrm{poly}(H)\cdot T^{1 - \alpha}\right)$ , which allows for efficient simulation.
Key challenges. As noted above, vanishing regret in our setting is impossible without directional smoothness due to discontinuity in the regressors between different regions. The key challenge is to leverage smoothness to mitigate the effect of these discontinuities. Note that, if the learner were to observe the region in addition to the state, regression would be easy by decomposing the problem into
$K$  separate affine regression instances; the difficulty is in minimizing this approach without explicit labels indicating which points are in which regions. We remark that were we to care only about low regret without regard to efficiency, a simple discretization scheme of size exponential in dimension coupled with a standard learning from experts algorithm [12] can achieve low regret; realizing an oracle-efficient algorithm is significantly more challenging.
We adopt a natural, epoch-based approach by calling the optimization oracle between epochs, estimating the underlying system, and using this estimate to assign points to regions in the subsequent epoch. This introduces three new challenges, each of which requires substantial technical effort to overcome. First, we need to control the performance of our oracle on the regression task, which we term parameter recovery below. Second, we must enforce consistency between regions learned in different epochs, so that we can predict the correct regions for subsequent covariates; doing this necessitates an intricate analysis of how the estimated parameters evolve between epochs. Finally, we need to modify the output of our oracle to enforce low regret with respect to the classification problem of correctly predicting which mode we are in on as yet unseen data.
Our techniques. Our algorithm proceeds in epochs, at the end of each of which we compute a best-fitting piecewise affine approximation to available data by calling the Empirical Risk Minimization (ERM) oracle. The analysis of this best-fit shows that we recover the parameters and decision boundaries of the associated regions ("pieces") frequently visited during that epoch. By the pigeonhole principle, this ultimately covers all regions visited a non-negligible number of times  $t$  over the horizon  $T$ . Here, we leverage a careful analysis based on two modern martingale-least squares techniques: the self-normalized concentration bound [2] for establishing recovery in a norm induced by within-region empirical covariance matrices, and the block-martingale small-ball method [54] to lower-bound the smallest eigenvalues of empirical covariances with high probability. In addition, we introduce and bound a new complexity parameter, the disagreement cover, in order to make our statements uniform over the set of possible decision boundaries.
After the estimation phase, each epoch uses this estimate to make predictions on new data as it arrives. The key challenge is classifying the region to which each newly observed data point belongs. Without smoothing, this can be particularly treacherous, as points on or near to the boundary of regions can easily be misclassified and the correct predictions are discontinuous. To leverage smoothing, we propose a reduction to online convex optimization on the hinge loss with lazy updates and show, through a careful decomposition of the loss, that our classification error can be controlled in terms of the assumed directional smoothness. Combining this bound on the classification error with the parameter recovery described in the preceding paragraph yields our sublinear regret guarantee.
We instantiate one-step prediction as a special case of the more general online linear regression problem, showing that the approach described above immediately applies to the first problem under consideration. Finally, in order to simulate trajectories, we use the learned parameters in the epoch to predict the next  $H$  steps of the evolution. We then use the fact that our main algorithm recovers the parameters defining the PWA system to show that simulation regret is bounded, which solves the second main problem and allows for simulation in such systems.
System Identification. System-identification refers to estimating the governing equations of a dynamical system. The special case of linear dynamical systems is classical [37, 38, 16], and more recent work has provided finite-sample statistical analyses in many regimes of interest [54, 15, 43, 61, 63]. Estimation for non-linear systems has proven challenging, and has been studied most in the setting where the dynamics involve a known and smooth nonlinearity [39, 51, 20], or where only the observation model is nonlinear [14, 42]. [52] study Markov jump systems, where the system dynamics alternate between one of a finite number of linear systems ("modes"), with switches between modes governed by a Markov chain. PWA systems are thus an attractive intermediate step between the simple linear setting and the intractable general case. To our knowledge, ours is the first work to tackle piecewise affine dynamics in full generality, where the system mode is determined by the system state.
Despite its worst-case computational intractability [5], there is a rich literature of heuristic algorithms for computing the best-fit PWA system to data; see the surveys [23] and [45], which compare various approaches, notably the clustering method due to [19]. These references focus strictly on
the algorithmic facets of computing a best-fit given an existing data set. We instead abstract the computation of a best-fit into an ERM oracle in order to focus on the statistical considerations.
Online Learning and Smoothing. The study of online learning is now classical, with a small sample of early works including Littlestone [36], Freund and Schapire [22, 21]. More recently, a general theory of online learning matching that which we have in the statistical learning setting was built up in Rakhlin et al. [49], Rakhlin and Sridharan [47], Rakhlin et al. [50], Block et al. [8], with Rakhlin and Sridharan [48] characterizing the minimax regret of online regression. Due to the robustness and generality of this adversarial model, many problems that are trivial to learn in the statistical setting become impossible in the adversarial regime. Thus, some work [49, 27, 28, 9, 6] has sought to constrain the adversary to be smooth, in the sense that the adversary is unable to concentrate its actions on worst-case points. In addition, Block and Simchowitz [7], which introduces the notion of directional smoothness to control the regret in a realizable, online classification problem; realizability, here, means that there is no process noise in the sense that there exists some piecewise affine function that perfectly predicts all targets given the covariates. We also note that the algorithm in that paper is highly tailored to this realizable setting and thus is not at all robust to the noise considered in the present paper. Of particular note is the concurrent work of Block et al. [10], which solves a similar problem with a stronger oracle. Note that in that paper, the oracle assumption is strong enough to break the lower bound of Hazan and Koren [29], while in this work we use the much weaker oracle assumption found in Hazan and Koren [29], Haghtalab et al. [28], Block et al. [9]. This distinction is important because even in the standard (fully adversarial) online learning setting, there is a computational separation between these two oracle models in general, as per Agarwal et al. [4]; thus the fact that this separation does not apply to PWA systems given smoothness may be of independent interest.
Online Learning for Control. In addition to the recent advances in finite-sample system identification, a vast body of work has studied control tasks from the perspective of regret [1, 3, 56, 53, 13]. While we do not consider a control objective in this work, we share with these works the performance metric of regret. Our work is more similar in spirit to online prediction in control settings [31, 62], in that we also consider the task of next-step predictions under a system subject to control inputs.
Smoothing in RL and Control. One interpretation of the smooth online learning setting is that well-conditioned random noise is injected into the adversary's decisions. It is well known that such noise injection can be viewed as a form of regularization [17], and recent recent work in the robotics literature has shown that randomized smoothing improves the landscape in various robotic planning and control tasks [58, 59, 35, 44]. More broadly, randomization has been popular for computation of zeroth-order estimates of gradients, notably via the acclaimed policy-gradient computation [66].
In this section, we formally describe the problem setting of online PWA regression. We suppose there is a ground truth classifier  $g_{\star}:\mathbb{R}^{d}\to [K]$  and matrices  $\{\Theta_i^\star \mid i\in [K]$  and  $\Theta_i^\star \in \mathbb{R}^{m\times (d + 1)}\}$  such that
$$
\mathbf {y} _ {t} = \boldsymbol {\Theta} _ {i _ {t}} ^ {\star} \cdot \bar {\mathbf {x}} _ {t} + \mathbf {e} _ {t} + \boldsymbol {\delta} _ {t}, \quad i _ {t} = g _ {\star} (\bar {\mathbf {x}} _ {t}), \quad \bar {\mathbf {x}} _ {t} = [ \mathbf {x} _ {t} ^ {\top} | 1 ] ^ {\top}, \quad | | \boldsymbol {\delta} _ {t} | | \leq \varepsilon_ {\mathrm {c r p}}, \tag {2.1}
$$
where,  $\mathbf{x}_t\in \mathbb{R}^d$  are covariates,  $\mathbf{y}_t\in \mathbb{R}^m$  are responses,  $\mathbf{e}_t\in \mathbb{R}^m$  are zero-mean noise vectors in  $\mathbb{R}^m$ ,  $\delta_t$  are (small) non-stochastic corruptions with norm at most  $\varepsilon_{\mathrm{crp}}\ll 1$ , and  $i_t\in [K]$  are the regression modes, which depend on the covariates  $\mathbf{x}_t$ . The extension to  $\bar{\mathbf{x}}_t$  accommodates affine regressors. We suppose that the learner has access to pairs  $(\mathbf{x}_t,\mathbf{y}_t)$  but, critically, does not know the regression modes  $i_t$ . We do however suppose that  $g_{\star}$  is an affine classifier; that is  $g^{\star}\in \mathcal{G} = \{\mathbf{x}\mapsto \arg \max_{1\leq i\leq K}\langle \mathbf{w}_i,\mathbf{x}\rangle +b_i\}$ , where  $\mathbf{w}_i\in S^{d - 1}$  is the unit sphere and  $b_{i}\in [-B,B]$ . It can be shown that many natural physical systems are indeed modeled as PWA with affine boundaries [40], and the closed loop dynamics for model-predictive control (MPC) in these cases is also a PWA system [11]. We will assume throughout access to an empirical risk minimization oracle, ERMORACLE, which satisfies the following guarantee:
Assumption 1 (ERMORACLE guarantee). Given data  $(\bar{\mathbf{x}}_{1:s},\mathbf{y}_{1:s})$ , ERMORACLE returns  $\left\{\widehat{\Theta}_i|i\in [K]\right\}$ ,  $\widehat{g}$  satisfying
$$
\sum_ {t = 1} ^ {s} \left| \left| \widehat {\boldsymbol {\Theta}} _ {\widehat {g} (\mathbf {x} _ {t})} \bar {\mathbf {x}} _ {t} - \mathbf {y} _ {t} \right| \right| ^ {2} - \inf  _ {g \in \mathcal {G}, \{\boldsymbol {\Theta} _ {i} | i \in [ K ] \}} \sum_ {t = 1} ^ {s} \left| \left| \boldsymbol {\Theta} _ {g (\mathbf {x} _ {t})} \bar {\mathbf {x}} _ {t} - \mathbf {y} _ {t} \right| \right| ^ {2} \leq \varepsilon_ {\text {o r a c}}. \tag {2.2}
$$
While such an oracle is not computationally efficient in the worst case [5], there are many popular heuristics used in practical situations [19, 23]. The learner aims to construct an algorithm, efficient with respect to the above oracle, that at each time step  $t$  produces  $\widehat{\mathbf{y}}_t$  such that
$$
\operatorname {R e g} _ {T} := \sum_ {t = 1} ^ {T} \| \widehat {\mathbf {y}} _ {t} - \boldsymbol {\Theta} _ {g _ {\star} (\bar {\mathbf {x}} _ {t})} ^ {\star} \bar {\mathbf {x}} _ {t} \| ^ {2} = o (T). \tag {2.3}
$$
This notion of regret is standard in the online learning community and, as we shall see, immediately leads to bounds on prediction error.
Directional Smoothness. As previously mentioned, without further restriction, sublinear regret is infeasible, as formalized in the following proposition:
Proposition 1. In the above setting, there exists an adversary with  $m = d = 1$ ,  $K = 2$ , that chooses  $\Theta^{\star}$  and  $g_{\star}$ , as well as  $\mathbf{x}_1, \ldots, \mathbf{x}_T$  such that any learner experiences  $\mathbb{E}\left[\mathrm{Reg}_T\right] \geq \frac{T}{2}$ .
Proposition 1 follows from a construction of a system where there is a discontinuity across a linear boundary, across which the states  $\mathbf{x}_t$  oscillate. The bound is then a consequence of the classical fact that online classification with linear thresholds suffers linear regret; see Appendix B.1 for a formal proof, included for the sake of completeness. Crucially, a discontinuity in the dynamics necessitates an  $\Omega(1)$  contribution to regret each time the decision boundary is incorrectly determined. To avoid this pathology, we suppose that the contexts are smooth; however, because standard smoothness [26] leads to regret that is exponential in dimension [7], we instead consider a related condition, "directional smoothness."
Definition 2 (Definition 52 from Block and Simchowitz [7]). Given a scalar  $\sigma_{\mathrm{dir}} > 0$ , a random vector  $\mathbf{x}$  is called  $\sigma_{\mathrm{dir}}$ -directionally smooth if for all  $\mathbf{w} \in S^{d-1} := \{w \in \mathbb{R}^d : \|w\| = 1\}$ ,  $c \in \mathbb{R}$ , and  $\delta > 0$ , it holds that  $\mathbb{P}(|\langle \mathbf{w}, \mathbf{x} \rangle - c| \leq \delta) \leq \delta / \sigma_{\mathrm{dir}}$ .
As an example, for any vector  $\mathbf{z}_t$ , if  $\mathbf{w}_t \sim \mathcal{N}(0, \sigma_{\mathrm{dir}}^2 \mathbf{I})$ , then  $\mathbf{z}_t + \mathbf{w}_t$  is  $\frac{\sigma_{\mathrm{dir}}}{\sqrt{2\pi}}$ -directionally smooth; see Appendix C.1 for a proof and examples of directional smoothness for other noise distributions. Crucially, directional smoothness is dimension-independent, in contradistinction to standard smoothness, where we would only have  $\Theta(\sigma_{\mathrm{dir}}^d)$ -smoothness (in the standard sense) in the previous example. We remark that directional smoothness is a weak condition that holds in many natural settings. Indeed, whenever noise is injected into the dynamics, directional smoothness will come for free. Such noise injection can occur when there is uncertainty in position, as is common in robotic applications; in this case, we can interpret this uncertainty itself as stochastic, which was the original motivation for smoothed analysis of algorithms [57]. Further, note that while directional smoothness ensures the system spends little time close to a boundary, due to the discreteness in time, there is no restriction on the number of mode switches, which can be  $\Theta(T)$  in general.
We defer further discussion of the assumption to Appendix C.1. We require our smoothness condition to hold conditionally:
Assumption 2. Let  $\mathcal{F}_t$  denote the filtration generated by  $\mathbf{x}_1, \ldots, \mathbf{x}_t$ . For all times  $t$ ,  $\mathbf{x}_t$  conditional on  $\mathcal{F}_{t-1}$  is  $\sigma_{\mathrm{dir}}$ -directionally smooth.
Further Assumptions. Next, we make two standard assumption for sequential regression, one controlling the tail of the noise and the other the magnitude of the parameters.
Assumption 3. Let  $\mathcal{F}_t^y$  denote the filtration generated by  $\mathcal{F}_t$  and  $\mathbf{e}_1, \ldots, \mathbf{e}_{t-1}$  (equivalently, by  $\mathcal{F}_t$  and  $\mathbf{y}_t, \ldots, \mathbf{y}_{t-1}$ ). For all  $t$ , it holds that  $\mathbf{e}_t$  is  $\nu^2$ -sub-Gaussian conditioned on  $\mathcal{F}_t^y$ ; in particular,  $\mathbf{e}_t$  is conditionally zero mean.
Assumption 4. We suppose that for all  $1 \leq t \leq T$ ,  $\|\bar{\mathbf{x}}_t\| \leq B$  and, furthermore,  $\|\Theta_i^*\|_{\mathrm{F}} \leq R$  for all  $i$ . We will further assume that ERMORACLE always returns  $\widehat{\Theta}_i$  such that  $\|\widehat{\Theta}_i\|_{\mathrm{F}} \leq R$ .
For a comparison between directional and standard smoothness, consult [7].