Title: Performative Control for Linear Dynamical Systems
Venue: NeurIPS 2024
Accepted: yes
Keywords: Performative prediction; Policy-dependent Dynamics; Disturbance-action control policy; Stochastic control
Abstract: We introduce the framework of performative control, where the policy chosen by the controller affects the underlying dynamics of the control system. This results in a sequence of policy-dependent system state data with policy-dependent temporal correlations. Following the recent literature on performative prediction \cite{perdomo2020performative}, we introduce the concept of a performatively stable control (PSC) solution. We first propose a sufficient condition for the performative control problem to admit a unique PSC solution with a problem-specific structure of distributional sensitivity propagation and aggregation. We further analyze the impacts of system stability on the existence of the PSC solution. Specifically, for {almost surely strongly stable} policy-dependent dynamics, the PSC solution exists if the sum of the distributional sensitivities is small enough. However, for almost surely unstable policy-dependent dynamics, the existence of the PSC solution will necessitate a temporally backward decaying of the distributional sensitivities. We finally provide a repeated stochastic gradient descent scheme that converges to the PSC solution and analyze its non-asymptotic convergence rate. Numerical results validate our theoretical analysis.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the new framework of performative control based on the performantive prediction concept.  The performative stable control (PSC) problem is formulated and its solvability conditions are derived. This paper also presents the algorithm of finding the PSC solution and addresses the convergence analysis.\n\nSTRENGTHS:\nThe concept of performative control is nicely motivated in Introduction.  The control policy, design method, and convergence analysis are stated mathematically rigorously.\n\nWEAKNESSES:\nThe disturbance-action policy defined in Definition 1 implicitly assumes that the disturbance w is measurable.  This assumption is practically severe and limits the application of the control policy.\n\nThe results of this paper rely heavily on the performative prediction results from the work [22].  Under many practically severe assumptions, the authors provide a new control policy and convergence analysis.\n\nQUESTIONS:\nThe assumption that the disturbance w is available for determining the control action u, is practically severe.  Could the authors relax the assumption?\n\nLemma 4, the convergence condition, is technically correct.  However, the reviewer cannot find any interpretation of the condition (11).  Could you give some remarks on the interpretation?\n \nThe reviewer cannot agree the statement of Introduction, \"Most prior studies on control of linear dynamical systems rely on the key assumption that the system state transition model is static\".  The pioneering works on state-space-based theory by R. E. Kalman, published 1960, handle time-varying system, meaning the transition is time-dependent.  Could you give more surveys on classical control problems?\n\nLIMITATIONS:\nThe limitations of the proposed control policy such as measurable disturbance and perfect modeling on the state transition are not adequately stated in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce the problem of performative linear control whereby one aims to control the evolution of a linear dynamical system whose dynamics are influenced by the choice of control policy. Apart from defining the problem, they define the concept of a performatively stable controller, extending the definition of performative stability introduced by Perdomo et al for the supervised learning setting. Given their model, their main technical results are results showing existence and uniqueness of a performatively stable controller under specific conditions, and a convergence result that illustrates how repeated stochastic gradient descent will converge to stability in this new setting at a O(1/T) rate.\n\nSTRENGTHS:\nThe idea of connecting performative prediction and linear control is very exciting, and to the best of my knowledge novel. I really enjoyed this idea that the very choice of control policy might actively influence the underlying system dynamics, and hence we should start to think about new models that explicitly account for this feedback. The technical results seem solid, although I wasn’t able to go through the proofs line by line. I also enjoyed the insights regarding how sensitivities need to be smaller at the beginning of the time horizon versus at the end.\n\nWEAKNESSES:\nWhile the motivation regarding how control policies might influence the dynamics is exciting, I find that it is quite underdeveloped in the paper. The main illustrative example the authors have is this stock market example. The intuition behind how control policies can affect the dynamics is plausible to me, but I would have liked to see it written out in detail in the main body. The primary exposition of this example is deferred to the appendix and I felt that it lacked clarity. \n\nThis underdeveloped motivation then makes the formal model itself hard to interpret and justify. I didn’t quite understand why the policy only affects the state matrix A, why all policy induce disturbances that are zero-mean, or why the disturbances between time steps are uncorrelated for the same policy. It would be helpful if the authors spent more time explaining the model and how it maps onto the high level ideas explained in the introduction. \n\nLastly, I think the paper suffers from a lack of clarity in the technical writing. Several quantities are referenced without being previously defined and there are some claims in the paper than are unsubstantiated (specifically, the idea that performative stability implies performative optimality in this new control model). Please see below for a full list of comments and questions.\n\nQUESTIONS:\nCould the authors motivate why they assume that A_t is performative but B is time invariant?\n\nWhy are the performative disturbances for A_t assumed to all be zero mean? \n\nIn L171, what is \\tilde{A?}? Is this just A? Or A - BK?\n\nIn Lemma 1, is K assumed to be strongly stable? If so, could you please make that explicit?\n\nIn L182: the authors claim that the PSC is approximately performative optimal, why is that?  In L236 they cite [22], but the result in [22] is shown in a different model with different assumptions\n\nL205: there is a mistaken reference, the authors reference A1-6, but only A4-6 are about the costs\n\nWhy do the authors refer to the condition that the operator norms of the matrices are <1 as almost surely stable? I usually understand stability in linear control as the condition that the spectral radius (not the operator norm) is smaller than 1. Assuming that the operator norm is smaller than 1 is a strictly stronger condition.\n\nTheorem 1 is not very readable. Please write this out as a lemma in the appendix and state the result formally with the step sizes plugged in so that the reader can appreciate the final convergence rate (stated formally).\n\nWhat is reference 13 it doesn’t seem to have an author? Reference 18 was published in a conference.\n\nLIMITATIONS:\nThese issues have been discussed to the extent necessary.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work presents a new approach for linear dynamical systems and highlights how control policies can directly affect the system’s dynamics thereby resulting in policy-dependent changes in system states. It also outlines specific conditions under which a stable control strategy can be achieved and introduces a method based on repeated stochastic gradient descent to reach this solution. The authors support their theoretical claims with relevant empirical results.\n\nSTRENGTHS:\n- Work addresses gap in traditional control theory via introduction of performative control\n- Provides theoretical foundation with sufficient conditions for performative stable control solutions\n- Paper provides application of the framework via numerical examples\n- Provides insights into system stability under different conditions\n- Development of custom stochastic gradient descent (rsgd) tailored to this new pscs control framework\n\nWEAKNESSES:\n- The concepts and algorithms introduced may be complex for practical implementation without extensive customization\n- Reliance on numerical examples vs extensive empirical data might limit how the framework will perform on real-world conditions\n\nQUESTIONS:\n- How is policy-depended temporal correlation in this work different from existing performative prediction works? this needs to be covered to highlight novelty\n- As per A4, A5 and A6, the cost function is assumed to be strongly convex, smooth and have bounded gradients respectively. These are common assumptions in optimization, can we relax this for cost functions in this control setting?\n- The RSGD Algorithm 1 has a projection step Proj_M - what metric is used for this projection? How is $M_n$ updated if the calculated M_n+1 falls outside of M? Also, is it reasonable to expect access to the true $A_t$, $w_t$ matrices in practice to compute the gradient eqn (17)?\n- The projection step assumes projecting into feasible set M is computionally tractable. How is it justified based on M structure?\n- RSGD is relatively straightforward, why it over other potential optimization methods for finding the psc solution?\n- the RSGD convergence rate is analyzed but the work will benefit from an emperical results comparing the actual convergence speed & efficiency of rsgd over other candidate algo.\n- Algo 1 assumes access to true gradients of the cost function, in practice they need to be estimated. So, impact of noisy gradient estimates on the convergance properties should be discussed\n\nLIMITATIONS:\n- Assumes the per stage cost function is strongly convex, smooth, and bounded. These assumptions may not hold for all types of control costs, limiting the applicability of the results.\n- The implementation of the proposed repeated stochastic gradient descent and other algorithmic strategies may involve high computational costs, particularly as system dimensions and data volumes grow.\n- The validation of the framework heavily relies on numerical simulations rather than empirical testing with real-world data.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the framework of performative control in the context of linear dynamical systems, where the control policy chosen by the controller impacts the underlying system dynamics. This interaction results in policy-dependent system states with temporal correlations. Inspired by the concept of performative prediction, the authors define a performatively stable control (PSC) solution and provide conditions for its existence and uniqueness. Specifically, they analyze how system stability affects the existence of the PSC solution, showing that for almost surely stable dynamics, the PSC solution exists if the sum of distributional sensitivities is sufficiently small. Conversely, for almost surely unstable dynamics, the existence of the PSC solution requires a backward decay of these sensitivities. The paper also introduces a repeated stochastic gradient descent (RSGD) algorithm that converges to the PSC solution, with theoretical analysis of its non-asymptotic convergence rate. Numerical experiments validate the theoretical findings, demonstrating the practical applicability of the proposed framework in scenarios such as stock investment risk minimization.\n\nSTRENGTHS:\n- The paper offers a thorough theoretical examination of the conditions required for the existence and uniqueness of performatively stable control solutions in the linear dynamical system.\n\nWEAKNESSES:\nI feel I does not get the punchline of the paper. Many of the convergence results appear standard: if your problem is convex and sufficiently smooth ($\\epsilon$-sensitivity), gradient descent is the obvious approach. This observation is well exploited since the original performative prediction paper. \n\nMoreover, despite the existing MDP literature typically considering finite action and state spaces, it is not clear to me that the linear model presents a serious challenge. \n\nFinally, I would like to add I believe the model presented in the paper has potential to enhance our understanding of general performative prediction. Albeit this is not the primary intention of the paper.  \nIn the most abstract sense, the distribution map $\\mathcal{D}$ depends on the controller's action $u$ (or policy $M$). I wonder if we can interpret this paper's model as having a distribution map that includes a linear term $Bu_t$ along with a small perturbation $A_t(M)x_t$. In this context, we might say this paper generalizes the original performative prediction framework, which requires the distribution map $\\mathcal{D}$ to be insensitive to the controller's actions, by allowing for any linear shift represented by $B$.\n\nQUESTIONS:\nCan you highlight the challenge the is unique to linear dynamical system compared to MDP with finite numbers of states and actions?\n\nLIMITATIONS:\nThere is not particular limitations for the paper to address.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}