Title: Smoothed Online Learning for Prediction in Piecewise Affine Systems
Venue: NeurIPS 2023
Accepted: yes
Keywords: Smoothed Online Learning

Piecewise Affine Prediction

Learning Dynamics
Abstract: The problem of piecewise affine (PWA) regression and planning is of foundational importance to the study of online learning, control, and robotics, where it provides a theoretically and empirically tractable setting to study systems undergoing sharp changes in the dynamics.  Unfortunately, due to the discontinuities that arise when crossing into different ``pieces,'' learning in general sequential settings is impossible and practical algorithms are forced to resort to heuristic approaches.  This paper builds on the recently developed smoothed online learning framework and provides the first algorithms for prediction and simulation in PWA systems whose regret is polynomial in all relevant problem parameters under a weak smoothness assumption; moreover, our algorithms are efficient in the number of calls to an optimization oracle.  We further apply our results to the problems of one-step prediction and multi-step simulation regret in piecewise affine dynamical systems, where the learner is tasked with simulating trajectories and regret is measured in terms of the Wasserstein distance between simulated and true data.  Along the way, we develop several technical tools of more general interest.

Smoothed Online Learning for Prediction in Piecewise Affine Systems

Abstract

The problem of piecewise affine (PWA) regression and planning is of foundational importance to the study of online learning, control, and robotics, where it provides a theoretically and empirically tractable setting to study systems undergoing sharp changes in the dynamics. Unfortunately, due to the discontinuities that arise when crossing into different 'pieces,' learning in general sequential settings is impossible and practical algorithms are forced to resort to heuristic approaches. This paper builds on the recently developed smoothed online learning framework and provides the first algorithms for prediction and simulation in PWA systems whose regret is polynomial in all relevant problem parameters under a weak smoothness assumption; moreover, our algorithms are efficient in the number of calls to an optimization oracle. We further apply our results to the problems of one-step prediction and multi-step simulation regret in piecewise affine dynamical systems, where the learner is tasked with simulating trajectories and regret is measured in terms of the Wasserstein distance between simulated and true data. Along the way, we develop several technical tools of more general interest.

1 Introduction

A central problem in the fields of online learning, control, and robotics is how to efficiently plan through piecewise-affine (PWA) systems. Such systems are described by a finite set of disjoint regions in state-space, within which the dynamics are affine. In this paper, we consider the related problem of learning to make predictions in PWA systems when the dynamics are unknown. While recent years have seen considerable progress around our understanding of linear control systems, the vast majority of dynamics encountered in practical settings involve nonlinearities. Learning and predicting in nonlinear systems is typically far more challenging because, unlike their linear counterparts, the dynamics in a local region of a nonlinear system do not determine its global dynamics.

PWAsystems allow for discontinuities across the separate regions ('pieces'), and are thus a simplified way of modeling rich phenomena that arise in numerous robotic manipulation and locomotion tasks [41, 64, 40], such as modeling dynamics involving contact. In addition, deep neural networks with ReLU activation [24] are PWA systems, providing further motiviation for their study. Already, there is a computational challenge simply in optimizing an objective over these systems, which is the subject of much of the previous literature. Here, we take a statistical perspective, assuming that we have access to effective heuristic algorithms for this optimization task, as is common in online learning [29, 9, 28]. Uniformly accurate learning of the dynamics across all pieces is typically impossible

because some regions of space require onerous exploration to locate; thus, we instead consider an iterative prediction setting, where at each time t , the learner predicts the subsequent system state. From this, we extend to prediction of entire trajectories: over episodes t = 1 , . . . , T , a learner suggests a policy π t , and the learner attempts to learn the dynamics so as to minimize prediction error along the trajectory π τ induces. This is motivated by iterative learning control, where our notion of error is an upper bound on the discrepancy between the planner's estimate of policy performance and the ground truth.

Our iterative formulation of the learning problem is equivalent to the regret metric favored by the online learning community, and which has seen application to online/adaptive control and filtering problems in recent work. Critically, regret does not require uniform identification of system parameters, which is typically impossible. The key pathology is that policies can oscillate across the boundary of two regions, accruing significant prediction error due to sharp discontinuities of the dynamics between said regions, a problem well-known to the online learning community [36]. Moreover, this pathology is not merely a theoretical artifact: discontinuities and non-smoothness pose significant challenges to planning and modeling contact-rich dynamics such as those encountered in robotic manipulation [58, 59, 46].

Our solution takes inspiration from, and establishes a connection between, two rather different fields. Recent work in the robotics and control communities has relied on randomized smoothing to improve planning across discontinuous and non-smooth dynamics [58, 59, 35, 44]. Additionally, the online learning community has studied smoothed online learning [49, 27, 28, 9, 7, 6], which circumvents the threshold-effect pathologies described above. We show that, if the dynamics and the control inputs are subject to randomized smoothing, low regret becomes achievable. We note that the randomized smoothing approach is in some sense canonical in mitigating the aforementioned pathology of policies that oscillate across the boundaries of two regions; smoothing prohibits this pathology by ensuring that the system is generally far from these boundaries. More importantly, our proposed no-regret algorithm is efficient in terms of the number of calls to an optimization oracle, a popular notion of computational efficiency in the online learning community [29, 34, 9, 28]. In our setting, the optimization oracle required finds the best-fit PWA system to a batch of given data. Though this problem is intractable [5], there is a rich literature of popular heuristics [23, 45, 19]. Unlike those works, we examine the statistical challenge of generalization to novel data given such an oracle. We remark that in many practical cases, our smoothness assumption comes for free, in the sense that the Gaussian noise used to smooth gradients in Suh et al. [58] is already sufficient for our results to hold.

Contributions. A formal description of our setting is deferred to Section 2. Informally, we consider the problem of prediction in a PWA system over a horizon of T steps, where the regions are determined by intersections of halfspaces; we obtain prediction in PWA dynamical systems as a special case. We aim to achieve sublinear-inT excess square-loss error ( regret ) of O ( T 1 -α ) with respect to the optimal predictor which knows the system dynamics, where α > 0 is constant and the prefactor on the regret is at most polynomial in all of the problem parameters. Our result is derived from a general guarantee for online piecewise affine regression, which subsumes the online PWA identification setting. We show that, when the dynamics and the control inputs are subject to randomized smoothing satisfying the σ dir -directional smoothness condition introduced by [7], our regret bound is polynomial in σ dir , dimension, and other natural problem parameters. While the exact dependence on horizon may be far from optimal, we emphasize that our algorithm is the first to achieve regret even polynomial in all parameters in the difficult setting of PWA systems. Moreover, our work provides the first regret bound with polynomial dependence in dimension for oracle-efficient smoothed online learning in the sense of Hazan and Koren [29] of a noncontinuous function class without a realizability assumption and allowing for process noise. As a further application, we adapt our algorithm to make predictions, at each time t , of the trajectory comprising the next H steps in the evolution of the dynamical system. We bound the simulation regret , the cumulative error in Wasserstein distance between the distribution of our proposed trajectory and that of the actual trajectory. Assuming a Lyapunov condition, we demonstrate that our modified algorithm achieves simulation regret O ( poly ( H ) · T 1 -α ) , which allows for efficient simulation.

Key challenges. As noted above, vanishing regret in our setting is impossible without directional smoothness due to discontinuity in the regressors between different regions. The key challenge is to leverage smoothness to mitigate the effect of these discontinuties. Note that, if the learner were to observe the region in addition to the state, regression would be easy by decomposing the problem into

K separate affine regression instances; the difficulty is in minimicking this approach without explicit labels indicating which points are in which regions. We remark that were we to care only about low regret without regard to efficiency, a simple discretization scheme of size exponential in dimension coupled with a standard learning from experts algorithm [12] can achieve low regret; realizing an oracle-efficient algorithm is significantly more challenging.

We adopt a natural, epoch-based approach by calling the optimization oracle between epochs, estimating the underlying system, and using this estimate to assign points to regions in the subsequent epoch. This introduces three new challenges, each of which requires substantial technical effort to overcome. First, we need to control the performance of our oracle on the regression task, which we term parameter recovery below. Second, we must enforce consistency between regions learned in different epochs, so that we can predict the correct regions for subsequent covariates; doing this necessitates an intricate analysis of how the estimated parameters evolve between epochs. Finally, we need to modify the output of our oracle to enforce low regret with respect to the classification problem of correctly predicting which mode we are in on as yet unseen data.

Our techniques. Our algorithm proceeds in epochs, at the end of each of which we compute a bestfitting piecewise affine approximation to available data by calling the Empirical Risk Minimization (ERM) oracle. The analysis of this best-fit shows that we recover the parameters and decision boundaries of the associated regions ('pieces') frequently visited during that epoch. By the pigeonhole principle, this ultimately covers all regions visited a non-neglible number of times t over the horizon T . Here, we leverage a careful analysis based on two modern martingale-least squares techniques: the self-normalized concentration bound [2] for establishing recovery in a norm induced by within-region empirical covariance matrices, and the block-martingale small-ball method [54] to lower-bound the smallest eigenvalues of empirical covariances with high probability. In addition, we introduce and bound a new complexity parameter, the disagreement cover , in order to make our statements uniform over the set of possible decision boundaries.

After the estimation phase, each epoch uses this estimate to make predictions on new data as it arrives. The key challenge is classifying the region to which each newly observed data point belongs. Without smoothing, this can be particularly treacherous, as points on or near to the boundary of regions can easily be misclassified and the correct predictions are discontinous. To leverage smoothing, we propose a reduction to online convex optimization on the hinge loss with lazy updates and show, through a careful decomposition of the loss, that our classification error can be controlled in terms of the assumed directional smoothness. Combining this bound on the classification error with the parameter recovery described in the preceding paragraph yields our sublinear regret guarantee.

We instantiate one-step prediction as a special case of the more general online linear regression problem, showing that the approach described above immediately applies to the first problem under consideration. Finally, in order to simulate trajectories, we use the learned parameters in the epoch to predict the next H steps of the evolution. We then use the fact that our main algorithm recovers the parameters defining the PWA system to show that simulation regret is bounded, which solves the second main problem and allows for simulation in such systems.

1.1 Related Work

System Identification. System-identification refers to estimating the governing equations of a dynamical system. The special case of linear dynamical systems is classical [37, 38, 16], and more recent work has provided finite-sample statistical analyses in many regimes of interest [54, 15, 43, 61, 63]. Estimation for non-linear systems has proven challenging, and has been studied most in the setting where the dynamics involve a known and smooth nonlinearity [39, 51, 20], or where only the observation model is nonlinear [14, 42]. [52] study Markov jump systems , where the system dynamics alternate between one of a finite number of linear systems ('modes'), with switches between modes governed by a Markov chain. PWA systems are thus an attractive intermediate step between the simple linear setting and the intractable general case. To our knowledge, ours is the first work to tackle piecewise affine dynamics in full generality, where the system mode is determined by the system state.

Despite its worst-case computational intractability [5], there is a rich literature of heuristic algorithms for computing the best-fit PWA system to data; see the surveys [23] and [45], which compare various approaches, notably the clustering method due to [19]. These references focus strictly on

the algorithmic facets of computing a best-fit given an existing data set. We instead abstract the computation of a best-fit into an ERM oracle in order to focus on the statistical considerations.

Online Learning and Smoothing. The study of online learning is now classical, with a small sample of early works including Littlestone [36], Freund and Schapire [22, 21]. More recently, a general theory of online learning matching that which we have in the statistical learning setting was built up in Rakhlin et al. [49], Rakhlin and Sridharan [47], Rakhlin et al. [50], Block et al. [8], with Rakhlin and Sridharan [48] characterizing the minimax regret of online regression. Due to the robustness and generality of this adversarial model, many problems that are trivial to learn in the statistical setting become impossible in the adversarial regime. Thus, some work [49, 27, 28, 9, 6] has sought to constrain the adversary to be smooth , in the sense that the adversary is unable to concentrate its actions on worst-case points. In addition, Block and Simchowitz [7], which introduces the notion of directional smoothness to control the regret in a realizable, online classification problem; realizability, here, means that there is no process noise in the sense that there exists some piecewise affine function that perfectly predicts all targets given the covariates. We also note that the algorithm in that paper is highly tailored to this realizable setting and thus is not at all robust to the noise considered in the present paper. Of particular note is the concurrent work of Block et al. [10], which solves a similar problem with a stronger oracle. Note that in that paper, the oracle assumption is strong enough to break the lower bound of Hazan and Koren [29], while in this work we use the much weaker oracle assumption found in Hazan and Koren [29], Haghtalab et al. [28], Block et al. [9]. This distinction is important because even in the standard (fully adversarial) online learning setting, there is a computational separation between these two oracle models in general, as per Agarwal et al. [4]; thus the fact that this separation does not apply to PWA systems given smoothness may be of independent interest.

Online Learning for Control. In addition to the recent advances in finite-sample system identification, a vast body of work has studied control tasks from the perspective of regret [1, 3, 56, 53, 13]. While we do not consider a control objective in this work, we share with these works the performance metric of regret. Our work is more similar in spirit to online prediction in control settings [31, 62], in that we also consider the task of next-step predictions under a system subject to control inputs.

Smoothing in RL and Control. One interpretation of the smooth online learning setting is that well-conditioned random noise is injected into the adversary's decisions. It is well known that such noise injection can be viewed as a form of regularization [17], and recent recent work in the robotics literature has shown that randomized smoothing improves the landscape in various robotic planning and control tasks [58, 59, 35, 44]. More broadly, randomization has been popular for computation of zeroth-order estimates of gradients, notably via the acclaimed policy-gradient computation [66].

2 Setting

In this section, we formally describe the problem setting of online PWA regression. We suppose there is a ground truth classifier g /star : R d → [ K ] and matrices { Θ /star i | i ∈ [ K ] and Θ /star i ∈ R m × ( d +1) } such that

where, x t ∈ R d are covariates, y t ∈ R m are responses, e t ∈ R m are zero-mean noise vectors in R m , δ t are (small) non-stochastic corruptions with norm at most ε crp /lessmuch 1 , and i t ∈ [ K ] are the regression modes, which depend on the covariates x t . The extension to ¯ x t accomodates affine regressors. We suppose that the learner has access to pairs ( x t , y t ) but, critically, does not know the regression modes i t . We do however suppose that g /star is an affine classifer ; that is g /star ∈ G = { x ↦→ arg max 1 ≤ i ≤ K 〈 w i , x 〉 + b i } , where w i ∈ S d -1 is the unit sphere and b i ∈ [ -B,B ] . It can be shown that many natural physical systems are indeed modeled as PWA with affine boundaries [40], and the closed loop dynamics for model-predictive control (MPC) in these cases is also a PWA system [11]. We will assume throughout access to an empirical risk minimization oracle, ERMORACLE, which satisfies the following guarantee:

Assumption 1 (ERMORACLE guarantee) . Given data (¯ x 1: s , y 1: s ) , ERMORACLE returns { ̂ Θ i | i ∈ [ K ] } , ̂ g satisfying

While such an oracle is not computationally efficient in the worst case [5], there are many popular heuristics used in practical situations [19, 23]. The learner aims to construct an algorithm, efficient with respect to the above oracle, that at each time step t produces y t such that

̂ This notion of regret is standard in the online learning community and, as we shall see, immediately leads to bounds on prediction error.

Directional Smoothness. As previously mentioned, without further restriction, sublinear regret is infeasible, as formalized in the following proposition:

Proposition 1. In the above setting, there exists an adversary with m = d = 1 , K=2, that chooses Θ /star and g /star , as well as x 1 , . . . , x T such that any learner experiences E [Reg T ] ≥ T 2 .

Proposition 1 follows from a construction of a system where there is a discontinuity across a linear boundary, across which the states x t oscillate. The bound is then a consequence of the classical fact that online classification with linear thresholds suffers linear regret; see Appendix B.1 for a formal proof, included for the sake of completeness. Crucially, a discontinuity in the dynamics necessitates an Ω (1) contribution to regret each time the decision boundary is incorrectly determined. To avoid this pathology, we suppose that the contexts are smooth; however, because standard smoothness [26] leads to regret that is exponential in dimension [7], we instead consider a related condition, 'directional smoothness.' 1

Definition 2 (Definition 52 from Block and Simchowitz [7]) . Given a scalar σ dir > 0 , a random vector x is called σ dir -directionally smooth if for all w ∈ S d -1 := { w ∈ R d : ‖ w ‖ = 1 } , c ∈ R , and δ > 0 , it holds that P ( | 〈 w , x 〉 -c | ≤ δ ) ≤ δ / σ dir .

As an example, for any vector z t , if w t ∼ N (0 , σ 2 dir I ) , then z t + w t is σ dir √ 2 π -directionally smooth; see Appendix C.1 for a proof and examples of directional smoothness for other noise distributions. Crucially, directional smoothness is dimension-independent , in contradistinction to standard smoothness, where we would only have Θ ( σ d dir ) -smoothness (in the standard sense) in the previous example. We remark that directional smoothness is a weak condition that holds in many natural settings. Indeed, whenever noise is injected into the dynamics, directional smoothness will come for free. Such noise injection can occur when there is uncertainty in position, as is common in robotic applications; in this case, we can interpret this uncertainty itself as stochastic, which was the original motivation for smoothed analysis of algorithms [57]. Further, note that while directional smoothness ensures the system spends little time close to a boundary, due to the discreteness in time, there is no restriction on the number of mode switches, which can be Θ ( T ) in general.

We defer further discussion of the assumption to Appendix C.1. We require our smoothness condition to hold conditionally:

Assumption 2. Let F t denote the filtration generated by x 1 , . . . , x t . For all times t , x t conditional on F t -1 is σ dir -directionally smooth.

Further Assumptions. Next, we make two standard assumption for sequential regression, one controlling the tail of the noise and the other the magnitude of the parameters.

Assumption 3. Let F y t denote the filtration generated by F t and e 1 , . . . , e t -1 (equivalently, by F t and y t , . . . , y t -1 ). For all t , it holds that e t is ν 2 -sub-Gaussian conditioned on F y t ; in particular, e t is conditionally zero mean.

Assumption 4. We suppose that for all 1 ≤ t ≤ T , || ¯ x t || ≤ B and, furthermore, ‖ Θ /star i ‖ F ≤ R for all i . We will further assume that ERMORACLE always returns ̂ Θ i such that ‖ ̂ Θ i ‖ F ≤ R . 1 For a comparison between directional and standard smoothness, consult [7].

Algorithm 1 Main Algorithm

Initialize Epoch length E , Classifiers w 0 1: K = ( 0 , . . . , 0 ) , margin parameter γ > 0 , learning rate

1: η > 0 2: for τ = 1 , 2 , . . . , T /E do ( % iterate over epochs ) 3: ( ̂ Θ τ ,i ) 1 ≤ i ≤ K , ̂ g τ ← ERMORACLE (¯ x 1: τ E , y 1: τ E ) 4: ̂ g τ ← Reorder ( ̂ g τ , ( ̂ Θ τ ,i ) 1 ≤ i ≤ K , ( ̂ Θ τ -1 ,i ) 1 ≤ i ≤ K ) 5: w τ E 1: K ← OGD ( w ( τ -1) E 1: K , ¯ x ( τ -1) E : τ E ; ̂ g τ , γ , η ) 6: Let ˜ g τ (¯ x ) = arg max 1 ≤ i ≤ K 〈 w τ E i , ¯ x 〉 denote classifier induced by w τ E 1: K . 7: for t = τ E,. . . , ( τ +1) E -1 do 8: Receive ¯ x t , and predict ˆ y t = ̂ Θ τ , ˜ g τ (¯ x t ) · ¯ x t 9: Receive y t

Finally, we assume that the true affine parameters are well-separated:

Assumption 5. There is some ∆ sep > 0 such that for all 1 ≤ i < j ≤ K , it holds that ‖ Θ /star i -Θ /star j ‖ F ≥ ∆ sep .

Note that Assumption 5 is, in a sense, generic, as explained in Remark 4 We defer the formal setting of one-step prediction and simulation regret to Appendix G.1 and section 3.1.

3 Algorithm and Guarantees

We propose Algorithm 1, an oracle-efficient protocol with provably sublinear regret and polynomial dependence on all problem parameters. Algorithm 1 runs in epochs: at the beginning of epoch τ , the learner calls ERMORACLE on the past data to produce a linear classifier ̂ g τ and estimates of the regression matrices in each mode, ̂ Θ τ ,i . A major challenge is identifying the same affine regions between epochs. To this end, the learner permutes the labels of ̂ g τ to preserve consistency in labels across epochs, as explained informally in Section 4.2 and in greater detail in Appendix E.

The learner then runs online gradient descent (Algorithm 2) on the hinge loss to produce a modified classifier ˜ g τ ; finally, throughout the epoch, the learner uses ̂ Θ τ ,i and ˜ g τ to predict ̂ y t before repeating the process in the next epoch. The reason for using a secondary algorithm to transform ̂ g τ into ˜ g τ is to enforce stability of the predictor across epochs, which is necessary to bound the regret. Again, we suppose that Assumption 1 holds throughout.

Theorem 3 (Regret Bound) . Suppose we are in the setting of online PWA regression (2.1) and Assumptions 2-5 hold. Then, if the parameters E, γ , η are set as in (F.6) , it holds that with probability at least 1 -δ , Reg T ≤ poly (par , log(1 / δ )) · T 35 / 36 + ε orac · T + K 2 · ∑ T t =1 || δ t || , for par := max { d, m, σ -1 dir , ∆ -1 sep , B, R, K, ν } , where the polynomial dependence is given in (F.7) .

Remark 1. While Theorem 3 requires that we are correctly setting the parameters of the algorithm, aggregation algorithms allow us to tune these parameters in an online fashion (see, e.g. [30]).

Remark 2. The misspecification errors δ t can be indicators corresponding to the ¯ x t being in small, rarely visited regions; thus, we could run an underspecified ERMORACLE , with K ′ /lessmuch K modes and recover a regret bound depending only on frequently visited modes.

When ε crp = ε orac = 0 , we obtain exactly the polynomial, no-regret rates promised in the introduction. Along the way to proving our regret bound, we establish that ERMORACLE correctly recovers the linear parameters within frequently visited modes.

̂ Above we have given the result with the assumption that ε crp = ε orac = 0 for the sake of presentation; in the formal statement, the general case is stated. Theorem 4 implies that directional smoothness

Theorem 4 (Parameter Recovery) . Suppose that Assumptions 2-4 hold and let { ̂ Θ i | i ∈ [ K ] } , ̂ g denote the output of ERMORACLE (¯ x 1: T , y 1: T ) and I ij ( ̂ g ) denote the set of times t such that ̂ g (¯ x t ) = i and g /star (¯ x t ) = j . Then with probability at least 1 -δ , for all 1 ≤ i, j ≤ K such that | I ij ( ̂ g ) | ≥ poly ( par , log ( 1 δ )) · T 1 -Ω (1) it holds that ‖ ̂ Θ i -Θ /star j ‖ 2 F is bounded by √ T | I ij ( g ) | · poly ( par , log ( 1 δ )) .

is sufficient to guarantee parameter recovery at a parametric rate, in marked distinction to most adversarial learning regimes. Before providing more detail on the algorithm and its analysis in Section 4, we discuss an application to H -step prediction. An application to one-step prediction with learner-provided controls can be found in Appendix G.1

3.1 Guarantees for Simulation Regret

We now describe an application of our results to H -step prediction; more formal description is detailed in Appendix G.3. The learning process occurs in episodes t = 1 , 2 , . . . , T , consisting of steps h = 1 , 2 , . . . , H . In each episode, an external planner provides the learner a policy π t . For simplicity, we assume that π t = ( u t, 1 , . . . , u t,H ) is open loop (state-independent) and stochastic, minimicking situations such as model predictive control; extensions are mentioned in Appendix G.6. With each episode, the true dynamics are given by

We assume that the process noise e t,h are independently drawn from a known distribution D , and that z t, 1 is sampled from an arbitrary σ dir -smooth distribution. At the start of episode t , the learner then produces estimates ˆ A t , ˆ B t , ˆ g t and simulates the plan π t using the plug in estimate of the dynamics, i.e., ˆ z t ; h +1 = ˆ A t, ˆ i t,h ˆ z t,h + ˆ B t, ˆ i t,h u t,h + ˆ m t, ˆ i t,h ˆ e t,h , ˆ i t,h = ̂ g t (ˆ z t,h , ˆ u t,h ) , where ˆ e t,h i . i . d . ∼ D , and ˆ u t,h are an i.i.d. draw from the stochastic, open-loop policy π t . Because the simulated and true processes use a different noise sequence, we consider the following notion of error, which measures the distance betwen the two distributions induced by the dynamics, as opposed to the regret, which measure the distance between the realizations thereof. We choose the Wasserstein metric W 2 , as it upper bounds the difference in expected value of any Lipschitz reward function between the true and simulated trajectories (see Appendix G.3 for a formal definition and more explanation).

Definition 5 (Simulation Regret) . Let W 2 ( · , · ) denote the L 2 -Wasserstein distance, define the concatenations x t,h = ( z t,h , u t,h ) and ˆ x t,h = (ˆ z t,h , ˆ u t,h ) , and let ¯ F t be the filtration generated by { x s,h } 1 ≤ s ≤ t, 1 ≤ h ≤ H . We define the T -episode, H -step simulation regret as SimReg T,H := ∑ T t =1 W 2 ( x t, 1: H , ˆ x t, 1: H | ¯ F t -1 ) 2 .

Our goal is to achieve SimReg T,H /lessorsimilar poly( H ) · T 1 -Ω (1) , but polynomial-inH dependence may be challenging for arbitrary open-loop policies and unstable dynamics of the pieces. Thus, in the interest of simplicity, we decouple the problems of linear stability from the challenge of error compounding due to discontinuity of the dynamics by adopting the following strong assumption.

Assumption 6. There exists a known , positive definite Lyapunov matrix P ∈ R d z × d z that satisfies ( A /star i ) /latticetop P ( A /star ) i /precedesequal P for all modes i ∈ [ K ] .

Using a minor modification of Algorithm 1, detailed in Algorithm 4, we show that we can get vanishing simulation regret, summarized in the following result:

Theorem 6. Suppose that we are in the setting of (3.1) and that Assumptions 2-5 and 6 hold. If we run a variant of Algorithm 1 (see Algorithm 4 in Appendix G.4), then with probability at least 1 -δ , it holds that SimReg T,H ≤ poly (par , H ) · T 35 / 36 .

The proof of Theorem 6 with the exact polynomial dependence on the parameters can be found in Appendix G.3 and rests on a lemma showing that under Assumption 6, simulation regret at a given time t can be bounded as poly ( H ) multiplied by the maximum one-step prediction error for times t, t +1 , . . . , t + H , which is bounded in Theorem 3. We provide an exact recovery guarantee in the case H = 1 and without requiring Assumption 6 in Appendix G.1.

4 Analysis

In this section, we present a sketch of the proof of Theorem 15. There are two primary sources of regret: that which is due to poorly estimating the linear parameters within a mode and that which is due to misclassification of the mode. We analyze each source separately , beginning with the parameter recovery result of Theorem 4.

Algorithm 2 Online Gradient Descent (Single Epoch)

Initialize Data ¯ x 1: E , clasifiers w 0 1: K , margin parameter γ > 0 , learning rate η > 0 , classifier g : X [ K ]

4.1 Proof Sketch of Theorem 4

We break the proof into three parts: first, we show that the regret of ̂ g and the ̂ Θ i , when restricted to times t ∈ I ij ( ̂ g ) , is small; second, we relate the prediction error Q ij ( ̂ g ) = ∑ t ∈ I ij ( ̂ g ) ‖ ( ̂ Θ i -Θ /star j )¯ x t ‖ 2 . to the regret on I ij ( ̂ g ) ; third, demonstrate that Σ ij ( ̂ g ) = ∑ t ∈ I ij ( ̂ g ) ¯ x t ¯ x T t has minimal eigenvalue bounded below by some constant with high probability. If all three of these claims hold, then, noting that ‖ ̂ Θ i -Θ /star j ‖ 2 F ≤ || Σ ij ( g ) -1 || op · ∑ t ∈ I ij ( g ) || ( ̂ Θ i -Θ /star j )¯ x t || 2 ,

̂ We now turn to our three claims. For the first, proved in Lemma D.15, we introduce the excess risk R ij ( g, Θ ) = ∑ t ∈ I ij ( g ) || Θ i ¯ x t -y t || 2 -|| e t + δ t || 2 , for each pair ( i, j ) , and note that the cumulative empirical excess risk ∑ i,j R ij ( ̂ g, ̂ Θ ) of predicting using ̂ g and ̂ Θ returned by the ERM oracle is bounded by ε orac because R ij ( g /star , Θ /star ) = 0 due to Equation (2.1). Thus, showing that R ij ( ̂ g, ̂ Θ ) is small can be done by showing that for no i ′ , j ′ is R i ′ j ′ ( ̂ g, ̂ Θ ) /lessmuch 0 . This can be accomplished by a concentration argument for a single g, Θ 0 , coupled with a covering argument using our notion of disagreement covers to boost the result to one uniform in G .

̂ ̂ we conclude. Each of these claims requires significant technical effort in its own right. We introduce disagreement covers , a stronger analogue of ε -nets adapted to our problem and allowing us to turn statements proven for a single g ∈ G into ones uniform over G . The first step is to bound the size of disagreement covers for the class G of interest; we then prove each of the three claims for some fixed g ∈ G and lift the result to apply to the data-dependent g . This is done in Appendix D.1.

For the second claim, i.e., that Q ij ( ̂ g ) is controlled by R ij ( ̂ g, ̂ Θ ) , we provide a proof in Lemma D.14. For a fixed g ∈ G , we decompose R ij ( g, ̂ Θ ) into Q ij ( g ) and an error term. We then use a generalization of the self-normalized martingale inequality from Abbasi-Yadkori et al. [2] to control the error in terms of Q ij ( g ) , providing a self-bounded inequality. Finally, we rearrange this inequality and apply a union bound on a disagreement cover to boost the result to one uniform in G and Θ .

To prove the third claim, that Σ ij ( ̂ g ) has singular values uniformly bounded from below, we split our argument into two parts. First, in Appendix D.2, we assume the following sliced-small ball condition on the data, i.e., for some ζ , ρ > 0 , it holds that P ( 〈 ¯ x t , w 〉 2 ≥ c · ζ 2 | F t , t ∈ I ij ( ̂ g )) ≥ c · ρ . Given the above condition, we establish a high probability lower bound on Σ ij ( g ) for fixed g using a self-normalized martingale inequality, using analysis that may be of independent interest. We then again apply a union bound on a disagreement cover to lift this result to be uniform in G . Finally, in Appendix D.3, we provide bounds on ζ , ρ using directional smoothness and Markov's inequality.

4.2 Mode Prediction

In this section, we address the second source of error, mode misclassification. The primary challenge is that directional smoothness, as opposed to independent data, is insufficiently strong to guarantee that ̂ g τ generalizes well to unseen data in epoch τ +1 . We take inspiration from online learning and stabilize our algorithm across epochs by modifying the classifier ̂ g τ . There are two challenges in ensuring good performance of our online classifier. First, the dynamics described in (2.1) are only identifiable up to a permutation of the labels. Thus, in order to enforce stability across epochs, we need to enforce consistency of labelling accross epochs. This task is made more difficult by the fact that different modes may be combined or split up across epochs due to the black box nature of ERMORACLE. We solve this by appealing to a subroutine, Reorder , described in Algorithm 3 in Appendix E, which combines modes that are sufficiently large and have sufficiently similar parameters and then relabels the modes so that similar nominal clusters persist accross epochs.

The second challenge is enforcing stability of our online classifier. We do this by introducing a surrogate loss ˜ /lscript γ ,t, ̂ g , the multi-class hinge loss with parameter γ > 0 2 . Formally, let w 1: K = ( w 1 , . . . , w K ) for w i ∈ B d -1 , and for g ∈ G , define the (1 / γ ) -Lipschitz and convex surrogate loss:

/negationslash

In OGD (see Algorithm 2), we modify the output of ERMORACLE to construct a classifier ˜ g τ by running lazy online gradient descent on this surrogate loss, using the reordered labels given by the output of ERMORACLE evaluated on the previous epoch. By invoking OGD, we show that our mode classifier is sufficiently stable to ensure low regret, as stated informally in the following result.

Theorem 7 (Mode Prediction, Informal Statement of Theorem 13) . Suppose that Assumptions 2-5 hold and let E mode t be the event that the learner, invoking Algorithm 1 with correctly tuned parameters, misclassifies the mode of ¯ x t . Then, with probability at least 1 -δ , it holds that ∑ T t =1 I { E mode t } ≤ poly ( par , log ( 1 δ )) · T 1 -Ω (1) .

/negationslash

/negationslash

/negationslash

Proof Sketch. We begin by addressing the lack of identifiability of the modes. For the purposes of analysis, for each epoch τ , we introduce the function π τ : [ K ] → [ K ] such that π τ ( i ) = argmin 1 ≤ j ≤ K ‖ ̂ Θ τ ,i -Θ /star j ‖ F . In words, π τ ( i ) is the mode j according to the ground truth whose parameters are closest to those estimated by ERMORACLE. We let E mode t denote the event of misclassifying the mode, i.e. the event that π τ ( ˜ g τ (¯ x t )) = g /star (¯ x t ) . We can then decompose the misclassification as E mode t ⊂ { π τ +1 ( ˜ g τ (¯ x t )) = g /star (¯ x t ) } ∪ { π τ ( ˜ g τ (¯ x t )) = π τ +1 ( ˜ g τ (¯ x t )) } . We call the first event look-ahead classification error and the second permutation disagreement error. To bound the first of these, we further decompose the look-ahead classification error event: { π τ +1 ( ˜ g τ (¯ x t )) = g /star (¯ x t ) } ⊂ { π τ +1 ( ̂ g τ +1 (¯ x t )) = g /star (¯ x t ) } ∪ { ˜ g τ (¯ x t ) = ̂ g τ +1 (¯ x t ) } . Controlling the first of these events is proved as a corollary of the bound on the permutation disagreement error and we defer its discussion for the sake of simplicity. To bound the probability of the second event, upper bound the indicator loss by ˜ /lscript γ ,t, ̂ g τ +1 and apply standard online learning techniques to show that regret of OGD on the (1 / γ ) -Lipschitz and convex surrogate loss is small. We then use smoothness to show that the optimal comparator with respect to the surrogate loss does not experience large regret with respect to the indicator loss, which in turn has a comparator experiencing zero loss due to the well-specified nature of g /star . Thus, putting everything together, this provides bound on the look-ahead classification error, with full details presented in Appendix E.1.

/negationslash

/negationslash

/negationslash

/negationslash

Bounding the permutation disagreement error is what necessitates the gap assumption. We show that if there are sufficiently many data points that ̂ g τ assigns to a given mode, with the threshold defined in terms of ∆ sep and other parameters of the problem, then with high probability the cluster becomes stable in the sense that ̂ Θ τ ,i ≈ ̂ Θ τ +1 ,i . This result is proved using the fact that if | I i,j ( ̂ g τ ) | is large enough, then the parameter recovery results from the previous section tell us that ‖ ̂ Θ i -̂ Θ j ‖ F is small. Using the triangle inequality and the fact that ‖ Θ /star j -Θ /star j ′ ‖ F > ∆ sep for j ′ = j ensures that j is unique in satisfying ‖ ̂ Θ τ ,i -Θ /star j ‖ F /lessmuch 1 . A similar argument applies to epoch τ +1 , and thus we can identify ̂ Θ τ ,i with some ̂ Θ τ +1 ,i ′ by these matrices being sufficiently close in Frobenius norm. Algorithm 3 takes advantage of exactly this property and permutes the labels across epochs in order to maintain consistency and control the permutation disagreement error; full details can be found in Appendix E.2. Combining this argument with the bound on the look-ahead classification error suffices to control the online classification component of the regret. /squaresolid

5 Discussion

We have given an efficient online learning algorithm for prediction in piecewise affine (PWA) systems. Our results are the first foray into the study of such systems, and a number of outstanding questions remain: Does directional smoothness facilliate low regret for planning and control, in addition to for simulation and prediction? If the systems are piecewise-affine but forced to be continuous on their boundaries, is there an oracle efficient algorithm which suffers low regret without assuming directional

smoothness? Can learning in piecewise affine systems be obtained under partial observation of the system state? Can the dependence of regret on horizon be improved? We hope that our work initiates investigation of the above as the field continues to expand its understanding beyond the linear regime.

Acknowledgments and Disclosure of Funding

AB acknowledges support from the National Science Foundation Graduate Research Fellowship under Grant No. 1122374 as well as support from ONR under grant N00014-20-1-2336 and DOE under grant DE-SC0022199. MS acknowledges support from Amazon.com Services LLC grant; PO# 2D-06310236. We also acknowledge Zak Mhammedi, Terry H.J. Suh and Tao Pang for their helpful comments.

References

Y. Abbasi-Yadkori and C. Szepesvári. Regret bounds for the adaptive control of linear quadratic systems. In Proceedings of the 24th Annual Conference on Learning Theory , pages 1-26. JMLR Workshop and Conference Proceedings, 2011.

Y. Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

N. Agarwal, B. Bullins, E. Hazan, S. Kakade, and K. Singh. Online control with adversarial disturbances. In International Conference on Machine Learning , pages 111-119. PMLR, 2019.

N. Agarwal, A. Gonen, and E. Hazan. Learning in non-convex games with an optimization oracle. In Conference on Learning Theory , pages 18-29. PMLR, 2019.

E. Amaldi and M. Mattavelli. The min pfs problem and piecewise linear model estimation. Discrete Applied Mathematics , 118(1-2):115-143, 2002.

A. Block and Y. Polyanskiy. The sample complexity of approximate rejection sampling with applications to smoothed online learning. arXiv preprint arXiv:2302.04658 , 2023.

A. Block and M. Simchowitz. Efficient and near-optimal smoothed online learning for generalized linear functions. arXiv preprint arXiv:2205.13056 , 2022.

A. Block, Y. Dagan, and A. Rakhlin. Majorizing measures, sequential complexities, and online learning. In Conference on Learning Theory , pages 587-590. PMLR, 2021.

A. Block, Y. Dagan, N. Golowich, and A. Rakhlin. Smoothed online learning is as easy as statistical learning. arXiv preprint arXiv:2202.04690 , 2022.

A. Block, A. Rakhlin, and M. Simchowitz. Oracle-efficient smoothed online learning for piecewise continuous decision making. arXiv preprint arXiv:2302.05430 , 2023.

F. Borrelli, A. Bemporad, and M. Morari. Predictive control for linear and hybrid systems . Cambridge University Press, 2017.

N. Cesa-Bianchi and G. Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

A. Cohen, A. Hasidim, T. Koren, N. Lazic, Y. Mansour, and K. Talwar. Online linear quadratic control. In International Conference on Machine Learning , pages 1029-1038. PMLR, 2018.

S. Dean and B. Recht. Certainty equivalent perception-based control. In Learning for Dynamics and Control , pages 399-411. PMLR, 2021.

S. Dean, H. Mania, N. Matni, B. Recht, and S. Tu. On the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics , 20(4):633-679, 2020.

M. Deistler, K. Peternell, and W. Scherrer. Consistency and relative efficiency of subspace methods. Automatica , 31(12):1865-1875, 1995.

J. C. Duchi, P. L. Bartlett, and M. J. Wainwright. Randomized smoothing for stochastic optimization. SIAM Journal on Optimization , 22(2):674-701, 2012.

H. Federer. Geometric measure theory . Springer, 2014.

G. Ferrari-Trecate, M. Muselli, D. Liberati, and M. Morari. A clustering technique for the identification of piecewise affine systems. Automatica , 39(2):205-217, 2003.

D. Foster, T. Sarkar, and A. Rakhlin. Learning nonlinear dynamical systems from a single trajectory. In Learning for Dynamics and Control , pages 851-861. PMLR, 2020.

Y. Freund and R. E. Schapire. Adaptive game playing using multiplicative weights. Games and Economic Behavior , 29(1-2):79-103, 1999.

Y. Freund and R. E. Schapire. Large margin classification using the perceptron algorithm. Machine learning , 37(3):277-296, 1999.

A. Garulli, S. Paoletti, and A. Vicino. A survey on switched and piecewise affine system identification. IFAC Proceedings Volumes , 45(16):344-355, 2012.

I. Goodfellow, Y. Bengio, and A. Courville. Deep learning . MIT press, 2016.

V. Guruswami and P. Raghavendra. Hardness of learning halfspaces with noise. SIAM Journal on Computing , 39(2):742-765, 2009.

N. Haghtalab, T. Roughgarden, and A. Shetty. Smoothed analysis of online and differentially private learning. arXiv preprint arXiv:2006.10129 , 2020.

N. Haghtalab, T. Roughgarden, and A. Shetty. Smoothed analysis with adaptive adversaries. arXiv preprint arXiv:2102.08446 , 2021.

N. Haghtalab, Y. Han, A. Shetty, and K. Yang. Oracle-efficient online learning for beyond worst-case adversaries. arXiv preprint arXiv:2202.08549 , 2022.

E. Hazan and T. Koren. The computational power of optimization in online learning. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 128141, 2016.

E. Hazan and C. Seshadhri. Efficient learning algorithms for changing environments. In Proceedings of the 26th annual international conference on machine learning , pages 393-400, 2009.

E. Hazan, K. Singh, and C. Zhang. Learning linear dynamical systems via spectral filtering. Advances in Neural Information Processing Systems , 30, 2017.

E. Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

C. N. Jones and M. Morari. Approximate explicit mpc using bilevel optimization. In 2009 European control conference (ECC) , pages 2396-2401. IEEE, 2009.

A. Kalai and S. Vempala. Efficient algorithms for online decision problems. Journal of Computer and System Sciences , 71(3):291-307, 2005.

Q. L. Lidec, L. Montaut, C. Schmid, I. Laptev, and J. Carpentier. Leveraging randomized smoothing for optimal control of nonsmooth dynamical systems. arXiv preprint arXiv:2203.03986 , 2022.

N. Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine learning , 2(4):285-318, 1988.

L. Ljung. System identification: Theory for the users," prentice hall, new jersey, 1999.

L. Ljung and B. Wahlberg. Asymptotic properties of the least-squares method for estimating transfer functions and disturbance spectra. Advances in Applied Probability , 24(2):412-440, 1992.

H. Mania, M. I. Jordan, and B. Recht. Active learning for nonlinear system identification with guarantees. arXiv preprint arXiv:2006.10277 , 2020.

T. Marcucci and R. Tedrake. Mixed-integer formulations for optimal control of piecewiseaffine systems. In Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control , pages 230-239, 2019.

M. T. Mason. Mechanics of robotic manipulation . MIT press, 2001.

Z. Mhammedi, D. J. Foster, M. Simchowitz, D. Misra, W. Sun, A. Krishnamurthy, A. Rakhlin, and J. Langford. Learning the linear quadratic regulator from nonlinear observations. Advances in Neural Information Processing Systems , 33:14532-14543, 2020.

S. Oymak and N. Ozay. Non-asymptotic identification of lti systems from a single trajectory. In 2019 American control conference (ACC) , pages 5655-5661. IEEE, 2019.

T. Pang, H. Suh, L. Yang, and R. Tedrake. Global planning for contact-rich manipulation via local smoothing of quasi-dynamic contact models. arXiv preprint arXiv:2206.10787 , 2022.

S. Paoletti, A. L. Juloski, G. Ferrari-Trecate, and R. Vidal. Identification of hybrid systems a tutorial. European journal of control , 13(2-3):242-260, 2007.

M. Posa, C. Cantu, and R. Tedrake. A direct method for trajectory optimization of rigid bodies through contact. The International Journal of Robotics Research , 33(1):69-81, 2014.

A. Rakhlin and K. Sridharan. Online learning with predictable sequences. In Conference on Learning Theory , pages 993-1019. PMLR, 2013.

A. Rakhlin and K. Sridharan. Online non-parametric regression. In Conference on Learning Theory , pages 1232-1264. PMLR, 2014.

A. Rakhlin, K. Sridharan, and A. Tewari. Online learning: Stochastic, constrained, and smoothed adversaries. Advances in neural information processing systems , 24:1764-1772, 2011.

A. Rakhlin, K. Sridharan, and A. Tewari. Sequential complexities and uniform martingale laws of large numbers. Probability Theory and Related Fields , 161(1):111-153, 2015.

Y. Sattar and S. Oymak. Non-asymptotic and accurate learning of nonlinear dynamical systems. Journal of Machine Learning Research , 23(140):1-49, 2022.

Y. Sattar, Z. Du, D. A. Tarzanagh, L. Balzano, N. Ozay, and S. Oymak. Identification and adaptive control of markov jump systems: Sample complexity and regret bounds. arXiv preprint arXiv:2111.07018 , 2021.

M. Simchowitz and D. Foster. Naive exploration is optimal for online lqr. In International Conference on Machine Learning , pages 8937-8948. PMLR, 2020.

M. Simchowitz, H. Mania, S. Tu, M. I. Jordan, and B. Recht. Learning without mixing: Towards a sharp analysis of linear system identification. In Conference On Learning Theory , pages 439-473. PMLR, 2018.

M. Simchowitz, R. Boczar, and B. Recht. Learning linear dynamical systems with semiparametric least squares. In Conference on Learning Theory , pages 2714-2802. PMLR, 2019.

M. Simchowitz, K. Singh, and E. Hazan. Improper learning for non-stochastic control. In Conference on Learning Theory , pages 3320-3436. PMLR, 2020.

D. A. Spielman and S.-H. Teng. Nearly-linear time algorithms for graph partitioning, graph sparsification, and solving linear systems. In Proceedings of the thirty-sixth annual ACM symposium on Theory of computing , pages 81-90, 2004.

H. J. Suh, M. Simchowitz, K. Zhang, and R. Tedrake. Do differentiable simulators give better policy gradients? In International Conference on Machine Learning , pages 20668-20696. PMLR, 2022.

H. J. T. Suh, T. Pang, and R. Tedrake. Bundled gradients through contact via randomized smoothing. IEEE Robotics and Automation Letters , 7(2):4000-4007, 2022.

R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999.

A. Tsiamis and G. J. Pappas. Finite sample analysis of stochastic system identification. In 2019 IEEE 58th Conference on Decision and Control (CDC) , pages 3648-3654. IEEE, 2019.

A. Tsiamis, N. Matni, and G. Pappas. Sample complexity of kalman filtering for unknown systems. In Learning for Dynamics and Control , pages 435-444. PMLR, 2020.

A. Tsiamis, I. Ziemann, N. Matni, and G. J. Pappas. Statistical learning theory for control: A finite sample perspective. arXiv preprint arXiv:2209.05423 , 2022.

A. J. Van Der Schaft and H. Schumacher. An introduction to hybrid dynamical systems , volume 251. springer, 2007.

G. Williams, A. Aldrich, and E. Theodorou. Model predictive path integral control using covariance variable importance sampling. arXiv preprint arXiv:1509.01149 , 2015.

R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3):229-256, 1992.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider a smoothed nonlinear regression problem with time-series data in which the hypothesis class is decomposed into the product of (i)  affine classes across polyhedral components and (ii) an indicator/classifier \"choosing\" the component. They first show that parameter recovery for the affine component of the class is possible at a slow rate and then proceed to control a notion of simulation regret for the prediction error at a (marginally) sublinear rate. They establish a few more results on similar lines along the way.\n\nSTRENGTHS:\nOverall, this is an interesting theory paper contributing to the growing field of learning in dynamical/control systems and I recommend it to be accepted contingent on a few of my minor concerns being addressed (after which I will gladly raise my score to 7 or higher). It is relatively original in terms of formulation and the quality of writing is good. The technical details also appear sound to me, although I did not check them all beyond quick \"sanity checks\". Details:\n\n* The directional smoothness concept taken from [7] and its use here are interesting. Overcoming the curse of dimensionality in learning nonlinear systems is a nontrivial matter and this is an interesting contribution applying this concept.\n\n* The problem is well-chosen and relatively well-motivated in terms of control applications (MPC). The claim to being foundational is perhaps a bit over the top but otherwise I quite liked the formulation.\n\n* The paper is generally quite well-written and it is easy to follow the main developments.\n\nWEAKNESSES:\nOverall, as I state above, I quite liked the paper and found the techniques/results to be interesting. However, it does feel a little unfinished for the reasons I outline below:\n\n* The regret bound is quite weak (which is basically just $o(T)$). While I appreciate that this perhaps is more to be seen as a \"proof of concept\" I cannot help but wonder whether some more effort should have been put into proving a more palatable rate before submission.\n\nPerhaps my main concern is the following: \n\n* I also wonder whether the parameter recovery bound could have been improved, which as it stands, appears to be at a slow rate. I did not really appreciate this being passed of as a \"parametric rate\" (line 270). The authors prove in Theorem 4 that \n$$error^2_i \\lesssim \\sqrt{T} / (\\textnormal{number of visits to i}) \\leq 1/\\sqrt{\\textnormal{number of visits to i}}$$  \nwhere I have omitted dimensional and complexity factors. Normally, the error in square norm for parameter recovery occurs at a fast rate (i.e. removing the square-root in the RHS above). If I squint I can also see how this could be obtained by controlling the $L^2$ prediction error of the hypothesis class as in [A] below. Heuristically I would expect by lower isomorphism in $L^2$:\n$$ error^2_i (L^2) \\times \\textnormal{occupancy rate}_i \\lesssim error^2(L^2) \\lesssim  error^2(\\textnormal{empirical }L^2) \\lesssim \\textnormal{self-normalized term}.$$\nRe-arranging this should in principle give a fast rate for parameter recovery. I am sure the predictor $\\hat g$ complicates matters, but the additional complexity should be controllable in terms of the number of regions. While it would be appreciated, let me also say that I do not expect the authors to  entirely \"fix\" this for the final submission but I think it at least deserves comment.\n\n* While on the topic of [A], this reference should be added to the related work section on nonlinear system identification for (stochastically) stable systems. The ideas therein may also be useful for answering parts of the discussion in Section 5.\n\n[A] Ziemann, Ingvar, and Stephen Tu. \"Learning with little mixing.\" Advances in Neural Information Processing Systems, 2022.\n\n\n**minor:**\n\n* overflow on line 723-724. \n\n*\n\nQUESTIONS:\n* Is assumption 4 really consistent with the use of the self-normalized inequality of de la Pena et al/Abbasi-Yadkori et al? It looks like this restricts the ERM to search over bounded regions in Euclidean space. \n\n* relating to the above weakness about parametric rates: is Lemma D.13 really anywhere near sharp? Why is there a factor $\\sqrt{T}$ on the RHS? Assuming realizability ($\\delta=0$) I would expect this to be  $\\tilde O(\\textnormal{complexity hyp class})$. Could this perhaps be removed by using an offset style of analysis as in [B] (which is for iid data, but see also [A] above for the dynamic (martingale) case, \nand see also your own reference [48])?\n\n* Is your claim that you have provided an \"efficient algorithm\" (line 399) really motivated without further nuance? Certainly, no claim to statistical efficiency can be made based on your analysis. See also my comments about your rates above.\n\n[B] Liang, Tengyuan, Alexander Rakhlin, and Karthik Sridharan. \"Learning with square loss: Localization through offset rademacher complexity.\" Conference on Learning Theory. PMLR, 2015.\n\n* How reasonable is your assumption 6? It is not obvious to me how to construct/verify this condition.\n\nLIMITATIONS:\nSee above caveats about rates and assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops the theory to show that the online smoothed learning of piecewise affine model (PWA), under mild assumption, can achieve sublinear regret in dynamics prediction, and also the regret bound is polynomial in system parameters.\n\nSTRENGTHS:\nI think the results of this paper are of great importance both to robotics community, where the online learning of PWA models can be used to represent non-smooth physical contact systems, and to online learning community, where some new results, e.g., the sublinear regret is polynomial instead of exponential in system dimension, is developed. \n\nMost importantly, the main algorithm is easy to implement and seems really promising to apply to the real problem.\n\nWEAKNESSES:\nTo further improve the paper, I have the following comments:\n- How does Definition 1 of directionally smooth imply that  \"directional smoothness ensures the system spends little time close to a boundary\" ?\n- It is not sure if Assumption 2 is mild, given there are jumps/discontinuities in PWA.\n- line 216, what's the meaning of \"$\\Theta(\\sigma_{dir}^{d})$-smoothness\"?\n- In Algorithm 1, the re-labeling step (Algorithm 3)  needs to check each pair of labels between the new and old partitions. How is the Algorithm scalable with large number of partitions?\n- Line 312, \"a sketch of the proof of Theorem 15\", I didn't see any Theorem 15 in the main text.\n- Typo in Equ. (3.1), $z_t$ should be $z_{t,h}$\n- Typo in the text right below Euq. (3.1), $\\hat{m}_{t,\\hat{i}_{t,h}}\\hat{e}_{t,h}$ should be $\\hat{m}_{t,\\hat{i}_{t,h}}+\\hat{e}_{t,h}$\n- Since the paper is partially motivated by the recent progress in random smoothing in physical contact systems, the authors may need to discuss the promise or limitation of the proposed algorithm when it is applied to that domain.\n\nQUESTIONS:\nPlease see my questions in the above section.\n\nLIMITATIONS:\nPlease see my questions in the above section.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper presents an algorithm for learning policies for piecewise affine systems, with both parameter-accuracy and regret bound guarantees, by using a kind of randomized smoothing of the boundaries of the affine regions. It identifies an anticoncentration property ("directional smoothness") that was previously proposed, along with some more standard assumptions on the noise distribution and separation of parameters in the affine regions that enable efficient identification; the problem was known to be inherently hard in the absence of such assumptions.

STRENGTHS:
The problem tackled in this work is natural and was known to be quite challenging. Although the basic idea of randomized smoothing to avoid the accumulation of error around the boundaries of affine regions is quite clear, actually executing this plan required quite a bit of technical sophistication. The paper itself gives a decent sketch of the approach and where these difficulties lay.

WEAKNESSES:
The only weakness is that the bounds, while meeting the requirements of being polynomial, sublinear, etc. are a bit extravagant. I don't know if this is a practical algorithm, and I hope this work isn't the last word on this problem.

QUESTIONS:
What avenues do you see for potentially improving the algorithm or its analysis? What would you conjecture the correct order of dependence on the time horizon to be?

LIMITATIONS:
I think this is fine. Although it isn't clear whether or not the algorithm is practical, the work is positioned as being a first work to meet basic theoretical desiderata, and the assumptions used seem to be well motivated.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of prediction in piecewise affine systems over a finite horizon. The main results are a linear regret bound and a sublinear regret bound under the proposed metric of simulation regret. To obtain the regret bounds, the authors made many assumptions and assume the access to an empirical risk minimization oracle.\n\nSTRENGTHS:\nThere is a good motivation to study piecewise-affine systems as an intermediate step between linear and nonlinear systems. This paper also presents many theoretical results.\n\nWEAKNESSES:\nIt is hard for me to follow the paper from the problem setting of piece-wise affine regression. There are too many mathematical expressions without an appropriate amount of context and motivating examples. And many key definitions and explanations are deferred to the appendix.\n\nThe setting is confusing because some assumptions are not stated formally as assumptions. For example, in line 185, the authors assume $g_*$ to be an affine classifier, but they did not explain the implication or motivation of this assumption. It seems that taking argmax only contains a small subset of piece-wise affine systems. Besides, it is also unclear where the non-stochastic corruption $\\delta_t$ comes from and if it is oblivious.\n\nI also have some concern about whether the assumptions made to obtain the regret bound are too strong, especially for Assumption 1. I conjecture that the upper bound $\\varepsilon_{orac}$ on the right hand side will increase with the horizon $s$ in practice if we want to solve this problem efficiently with unknown parameters. Assumption 6 also seems very strong and I feel it is almost equivalent to knowing the exact dynamics under each mode.\n\nQUESTIONS:\nPlease see my comments in the previous section.\n\nLIMITATIONS:\nThe authors discussed many future directions at the end of the paper, and I did not see any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}