PID Control-Based Self-Healing to Improve the Robustness of Large Language Models
Authors: Zhuotong Chen
Zihu Wang
Yifan Yang
Qianxiao Li
Zheng Zhang
Venue: Accepted by TMLR ()

Abstract: Despite the effectiveness of deep neural networks in numerous natural language processing applications, recent findings have exposed the vulnerability of these language models when minor perturbations are introduced. While appearing semantically indistinguishable to humans, these perturbations can significantly reduce the performance of well-trained language models, raising concerns about the reliability of deploying them in safe-critical situations. In this work, we construct a computationally efficient self-healing process to correct undesired model behavior during online inference when perturbations are applied to input data. This is formulated as a trajectory optimization problem in which the internal states of the neural network layers are automatically corrected using a PID (Proportional-Integral-Derivative) control mechanism. The P controller targets immediate state adjustments, while the I and D controllers consider past states and future dynamical trends, respectively. We leverage the geometrical properties of the training data to design effective linear PID controllers. This approach reduces the computational cost to that of using just the P controller, instead of the full PID control. Further, we introduce an analytical method for approximating the optimal control solutions, enhancing the real-time inference capabilities of this controlled system. Moreover, we conduct a theoretical error analysis of the analytic solution in a simplified setting. The proposed PID control-based self-healing is a low-cost framework that improves the robustness of pre-trained large language models, whether standard or robustly trained, against a wide range of perturbations.

Published in Transactions on Machine Learning Research (03/2024)
PID Control-Based Self-Healing to Improve the Robustness of Large Language Models
Zhuotong Chen
ztchen@ucsb.edt edu
Deportment of Electrical and Computer Engineering, University of California, Sonta Barbar
Zihu Wang
aibu wwng edu
Department of Electrical and Computer Engineering, University of California, Sonta Barbara
Yifan Yang
vifenyyng@uucsb.co edu
Department of Computer Sciences, University of California, Sonta Barbara
Qianxiao Li
Gianziao@nus.edu
Department of Mathematics, National University of Singapore, Singapore Institute of High Performance Computing, A STAR,, Singapore
Zheng Zhang
zhengahang ucsb.edu
Department of Electrical and Computer Engineering, Universily of California, Senta Barbara
Reviewed on OpenReview: https: openreview. net/ perm/ id= Fa4muBOXIU
Abstract
Despite the effectiveness of deep neural networks in numerous natural language processing applications, recent findings have exposed the vulnerability of these language models when minor perturbations are introduced. While appearing semantically indistinguishable to humans, these perturbations can significantly reduce the performance of well-trained language models, raising concerns about the reliability of deploying them in safe-critical situations. In this work, construct computationally efficient self-healing process to correct undesired model behavior during online inference when perturbations are applied to input data. This is formulated as a trajectory optimization problem in which the internal states the ncural network layors are automatically corrected using a PID (Proportional Prooportional--nterra-----eee satteerrl--erivatieee control mechanism. The controller largets immediate state adjustments, while the I and D controllers consider past states and future dynamical trends, respectively. We leverage the geometrical propertics of the training data. to design effective linear PID controllers. This approach reduces the computational cost to that of using just the P controller, instead of the full PID control. Further, we introduce analytical method for approximating the optimal control solutions, enhancing the real-time inference capabilities of this controlled system. Morcover, we conduct a thcorctical crror analysis of the analytic solution a simplified selting. The proposed PID rol-babe self-healing is a low-cost framework that improves the robustness of pre-trained large language models, whether standard or robustly trained, against wide range of perturba- tions. A detailed implementation can be found in https:://gtttttt http://hhhhb.ccmm/hoooooooggg/eg Seleeed-Seleeeee to
1
Introduction
The growth of data and advanccments in computing power have heralded new era in the field natural language processing (NLP), significantly shaped by the advent of deep neural networks. One of the most influential innovations in this domain is the transformer architecture (Vaswani et. al. 2017), which is the

---

Published in Transactions on Machine Learning Research (03/2024)
fundamental block of many successful large language models (LLMs) (Brown et al. 2020 This architecture has become the state-of-the-art. in various NLP tasks, including sentiment analysis (Vinodhini & Chan- drasekaran 2012), ; text summarization (Nenkova & McKeown 2012), and speech recognition (Hannun et al. 2014), among others.
However, many deep neural networks are vulnerable malicious perturbations (Morris et al. 2020 While appearing semantically indistinguishable to humans, these perturbations can significantly degrade the per- formance of pre-trained LLMs. This vulnerability raises concerns about the reliability of deploying them in safety-critical situations, such as in clinical decision support systems, where LLMs serve critical role in assisting healthcare professionals with patient care insights (Huang et al. 2019.. In response to this chal- lenge, there has been significant progress in developing algorithms to enhance model robustness against such perturbations (Yoo & Qi 2021 Zhu et al., 2019 Wicker et al., 2021). Predominantly, these methods are rooted in the foundation of adversarial training (Madry et al.] 2018), a method where pre-trained LLMss are fino-tuned (or trained from random initialization) to overcome the effects of specific adversarial perturba- tions. This is achieved by adjusting either the entire set of model parameters or a significant portion thereof (Hu et al. 2021). Despite its effectiveness, this approach raises three critical concerns. Firstly, adjusting model parameters using adversarial examples requires substantial computational resources (Zhang et al. 2019 Due to the discrete input space of NLP tasks, searching for an adversarial example generally involves solving a combinatorial optimization problem (Bernhard & Vygen 2008), which suffers from an exponential growth in the number of feasible solutions as the size of the problem increases. Secondly, there exists a po- tential trade-off where improved adversarial robustness may lead to compromised performance on standard, natural datasets (He et al. 2021). Thirdly, and more problematically, adversarial training is less effective against unforescen adversarial perturbations (Tramer & Bonch 2019). This limitation becomes particularly noticeable when deploying LLMs in practice, where anticipating the potential adversarial attacks in advance is nearly impossible.
In this paper, we investigate the concept of a self-healing process as a cost-cffective method to improve the robustness of pre-trained LLMs against a range of perturbations. The most well-known self-healing mechanism is probably the human immune system: B cells and T cells can work together identify and kill many external attackers (e.g., bacteria) to maintain the health of the human body (Rajapakse & Groudine
2011). In the context of machine learning, self-healing refers to the ability of model to automatically identify and correct issues that may arise during its operation (Wang et al. 2021 Chen et al. 2022 To achieve this, we consider a pre-trained LLM (typically composition of transformation blocks) as a discretization of the contimous dynamical system EE 2017), this allows us to formulate the robustness issuc of LLMs as a trajectory optimization problem (Hehn & D D'Andrea 2015). Our approach involves designing PID (Proportional-Inteer Interaalnnnnneaaa-evvvvviio controllers at hidden layers of pre-trained LLM. PID controller continuously calculates an error value as the difference between a desired reference and a measured process variable and applies correction control signal based proportional, integral, and derivative terms. More specifically, let the error value be the difference between a desired reference and the current state. Tf the error is large, the output of the P controller will be proportionately large, thereby making significant adjustment and helping the controller respond quickly to errors. The I controller determines the present control oulput based on the integration of past errors, which ensures that even small errois are corrected over time. The D controller generates control signals basod on the derivative of the error dynamics. The combination of P. I, and D controllers quantifies undesired model behavior from present errors, past accumulated errors, and future error trends, and generates control signals to correct the errors. Figure 1 illustrates the proposed PTD control-base based self-healing framework. Given a T-layer LLM, time-dependent PID controllers (represented as P., It, and D.) generate a feedback control based on the state x (to simplify the demonstration, only x; is considered as the input. for both I and D controllers). These leedback controls aim to remove the undesirable effects caused by input perturbations.
The methodology of constructing a self- helf-healing process to improve the robustness of deep neural networks was initially introduced in Chen et al.) (2020) and its subsequent work Chen et al. (2022). Tt leveraged closed-loop control method to detect and correct potential errors applied to input data. This method belongs to a special case of P control in the proposed PID control framework, in which only the errors from the present states are corrected. However, the effectiveness of using only proportional controllers is
2

---

Published in Transactions on Machine Learning Research (03/2024)
X
X
XT-1
Blocko
Block
Block
Po(x
P P2(
P Px-1
1x(xo)
(x2
I-1 (x
Do(X
D (1
D -x-(X
Figure 1: The structures of feed-forward deep neural network (highlighted in blue) and the proposed PID control method (highlighted in red).
limited, as it merely addresses the error at each time step, neglecting the overall error dynamics (we provide numorical evidence to show this in Section 3.4 . Additionally, a major limitation of the closod-loop control
method adapted in Chen et al. 2020 2022 is its computational inefficiency. The optimal control solution
requires simulating the Hamiltonian dynamics over several iterations during online inference (Pontryagin 1987), which involves both forward and backward propagation of a deep neural network (Chen al.) 2020) (we discuss the details about simulating the Hamiltonian dynamics in Section 2.1 This inefficiency renders the self-healing framework impractical for deployment. in large-scale LLMs, which may contain millions
even billions of parameters (ennonn & Toutanova 2019 Liu et al.) 2019 Brown et al. 2020). To address the
challenges associated with the previously mentioned adversarial training and existing closed-control method, this study presents three contributions:
We introduce novel PID control framework to realize the self-healing capability to improve the robustness of LLMs during online inference. The proposed framework generalizes the conventional robustness improvement methods that predominantly focus on proportional errors. We demonstrate that employing all P. I, and D controllers can be as computationally efficient as single control schemes, achieved through special controller design.
We approximate the layer-wise transformations in the pre-trained LLM as linear orthogonal trans- formations and derive an analytical solution for generating PID control solutions. This analytical method yields a closed-form expression for the optimal solution, enhancing the speed of online in- ference. This acceleration especially beneficial when integrating the self-healing framework into I.L.ss While these approximations might not align with practical scenarios, our method exhibits superior robustness improvement in variety of numerical experiments.
We derive a comprehensive error analysis of the controlled system, highlighting the robustness im- provement of LLMs through PID control solutions. This analysis provides insight into the effective- ness of PID control in improving the robustness of LLMs in simplified settings, thereby contributing to the understanding of controlled systems.
1.1 Background on PID Control
Here we provide background knowledge on PID control, which is an essential building block of our proposed framework. A PID (Proportional, Integral, Derivative) controller is feedback- control system exten- sively utilized in industrial settings and numerous other domains where continuous adjustment is essential. Specifically, given continuous dynamic system, typically described as an ordinary differential equation,
+ ((x,, u.), Xo E Rd
where &, represents the derivative with respect time, while Rd x Rm Rdd defines function or vector field. The term Xg specifies the initial state. Moreover, u, E Rm denotes the control input exerted on the dynamic system (e.g. in this work, the control input is applied linearly to the state x.). The PID control
3

---

Published in Transactions on Machine Learning Research (03/2024)
continuously computes error term, C,, by calculating the difference between target reference r, and the actual state variable Xt,
ct
|r xx
The construction for the target reference generally depends on the application. In this work, the target reference constructed by sequence of embedding manifolds (see Section 2.1 and the process variable represents the hidden state of deep neural network during forward propagation.
Then, a PID controller applies a control u, based on proportional, integral, and derivative terms to correct the measured error. In the continuous case,
Kper + Ki I'" e(T)dr + Ka de, dt
where Kp, K,, and Ka; all non-negative, denote the coefficients for the proportional, integral, and derivative terms respectively. In the PID control design,
The output from proportional control directly correlates with the current. value of the error, et- Thus, a larger error will yield a proportionally larger control output, adjusted by the gain factor Ky. However, employing only proportional control leads to persistent difference between the desired larget and the actual process variable, since the generation of the proportional response necessitates the presence of an error.
The output from integral control considers the accumulation of past error values over time. This means that when a residual error remains after proportional control applied, the integral control works to correct this residual error by leveraging the historical total error. This will result in the proportional effect diminishing as the error decreases, but this is compensated for by the growing integral effect.
The output from derivative control provides estimate of the trend of the error, using its current rate of change as basis. Tt effectively seeks to reduce the effect of the error by exerting control influence generated by the rate of error change. Hence, the more rapid the crror's progression, the more intense the applied correction.
Although PID controller has three control terms, some applications need only one or two terms to provide appropriate control. This is achieved by setting the unused parameters to zero and is called PI, PD, P. or I controller in the absence of the other control actions. PD controllers are fairly common in applications where integral action would be sensitive to measurement noise, but the derivative term is often needed for the system to reach its target reference.
2 The PID Control-Based Self-Healing Framework for Large Language Models
In this work, we use the concept of "sell-heallng" to describe the capability of LLM to automatically correct errors that may arise. This idea has been studied in the domain of integrated circuits, primarily addressing errors aftributable to variations in nano-scale fabrication processes. More specifically, the internal dynamics of an electronic circuit network can be understood through the lens of ordinary differential equations (Ho eet al. 1975). This approach conceptualizes the circuit network in terms of state variables that track nodal voltages and branch currents as they change over time. Moreover, it's possible to create a self-healing system within the circuit network, cnabling it continuuusy monitor and optimize its performance throughout the lifetime of operation (Tang et al. 2012 Lee al.) 2012). There is a growing body of research, including works by E 2017 Haber & Ruthotto 2017 Li et al. 2018b), and others, demonstrating the connection between dynamical systems and deep neural networks. In study, we interpret pre-trained T T-layer LLM as form of discrete dynamical system,
X++1 F((x, m())),,.,,, VI 0,1,
(1)

---

Published in Transactions on Machine Learning Research (03/2024)
where Fi(,, 0t) E + Rd represents transformer block parametrized by 0,, R + R is a feedback controller that maps the current state X, to a control action. We aim to construct feedback controllers के : { 17 to ensure that the controlled states (x, + <x(x))) yield the desired output when perturbations are applied to input data. This can be formulated as a trajectory optimization problem,
min minn ( [J(x,, v. m] == min E (x0.))-D
v(xT.V)
Σ (((x))/=o: Te, Fe. s.t. equation 1
(2)
where initial states and labels (Xo, y) are sampled from the underlying data distribution D. The terminal loss ((xr, v) evaluates the discrepancy between the terminal state and pre-defined destination set. In machine learning applications, this measures the consistency between the terminal state xT (or its transformation) with the true label (c.g., cross-cntropy loss). However, this is not feasible in general machine learning ap- plications as the true label remains unknown during inference. More specifically, during online inference, given an initial condition Xu, its label y cannot be accessed to optimize the states since this quantity is unknown. Consequently, the terminal loss negated by setting it zero. In these cases, the optimal con- trollers ( t== minimize the cumulative running losses L C((x=)'s )s TT, fi), which assess the state trajectory and control using certain measurement function ft-
In Section 2.1 we construct the rumning loss, which forms a crucial part of the objective function defined in equation 2 This objective fonction needs to solve a complex optimization problem for each initial state during online inference, which presents a challenge to computational efficiency. To address this, the subsequent Scction 2.2 presents more officient algorithm for solving the objective function under specific assumptions. Furthermore, Section 2.3 presents comprehensive theoretical error analysis for the proposed algorithm. Lastly, Section 2.4 provides additional details the implementation of constructing PID controls.
2.1 PID Control Design via Embedding Manifolds
In analyzing an LLM through the lens of discrete dynamical systems, we observe that its state trajectory, governed by the composition of transformations, forms lower-dinensional structure embedded in the ambi- ent state space, also known as the "manifold hypothesis" (Fefferman et al. 2016) (empirical evidence is shown in Table 7) This can be conceptualized as a sequence of embedding manifolds. We consider a r-dimensional smooth manifold embodded in Rd as fx: : f(x) = 0}, where IR + Rddrr) is a surjective mapping that can be used to measure the distance between a state x, to the embedding manifold. For instance, [|](x)]] = if x belongs to the embedding manifold, and [x) > if x is outside the embedding manifold.
At each time step t (c.g., the tth hidden states), construct three embedding manifolds with three distinct surjective functions / Rd - RRd-r), Sf Rd + R(d-"), and SP : Rd - (dd-r), that represent the embedding manifolds of the state, the integration of past states, and the derivative of the state, respectively. In a discrete sctting, integration corresponds to the accumulation of past states, while the derivative is approximated by the difference between two successive states. Under this setting, fl denotes the embedded manifold of past states, and fP represents the embedded manifold derived from the difference between two consecutive states. Given these embedding functions, we propose the following running loss to evaluate the controlled state at time step t,
2 _1lf (x: + T. (x:))1++++ + (x:: Tx(x)) + Σ xl)
1
(( Te (f.. fl. (P))
+ 2 |s (x: + mx(x)) -x--xxx-111 x1 11 (x):)
(3)
where the layer-dependent regularization term e prevents using large controls. The running loss consists of three components, cach asscssing the error in the controlled state through distinct embedding functions: proportional, integration, and derivative. In this construction of running loss, the optimal controller results in a controlled state (x; + ne (xx)) which is expected to closely align with the state embedding manifold, evaluated by f.P. Additionally, the controller must ensure that past controlled states remain close to the embedding manifold of integrated states, as evaluated by f/, and the state's derivative should similarly align closely with the manifold of the state's derivative embedding, as quantified by SP.
5

---

Published in Transactions on Machine Learning Research (03/2024)
The objective function defined in equation 2 and the associated running loss detailed in equation can be solved via the dynamical programming principle (Bellman 1952). However, this method faces exponential complexity in terms of the dimension of the state. To overcome this "curse of dimensionality', we can reinterpret the optimal control problem through Pontryagin's 's Maximum Principle and approximate using the method of successive approximation (Chen et al., 2020). To begin with, we define the Hamiltonian (t, {x=)==o> Pc+1, U,) as
Ht, {x Pc+1, 01 u.) = P Fi + u, 0.) ((xx,)/- u. (fe Se. (.)))))
where u, = (Xx) is control solution. Pontryagin's maximum principle consists of a two-point boundary value problem,
x Vtll(t, {x,)/=o P-+1, Oc, ut),
(xo, y) ~ D,
(4)
P = V, (t, (x.) Pr+1, O2 u.),
pr 0,
(5)
plus maximization condition of the Hamiltonian.
(t,, (x Po+1,, 0c, u,) 2 (t,, (x.))=o: P++1, uz), V and Vt E.
(6)
To obtain numerical solution, one consider iterating through the forward dynamic equation to obtain all states (x) the backward dynamic equation 5 to compute the adjoint states {Pr) and maximizing the Hamiltonian defined in equation 0 with current states and adjoint states via gradient ascent. This iterative process is continued until convergence. Given an initial condition Xo. Pontryagin's 's Maximum Principle characterizes the optimal feedback control (x,)) with an open-loop control ur, this open-loop control necessitates both forward and backward propagation through a pro-trained deep neural network over several iterations during online inference.
However, implementing the above Pontryagin's Maximum Principle is generally infeasible for LLMs. In the subsequent section, we construct an analytic solution with certain relaxation assumptions.
2.2 An Analytic Solution for Fast Inference
In this section, we develop an analytic solution for solving the objective function defined in equation 2 under certain assumptions. These assumptions are summarized in the following,
Assumption Both embedding manifold and layer-wise transformation are simplified linear functions. In this case, the layer-wise transformation, denoted Fi(-), is linearized through matrix 0, E Raxd A smooth embedding manifold is represented by linear embedding subspace. This linear embedding subspace is defined by set of basis vectors, which are captured by the column space of a matrix V E Raxr, corresponding to an r-dimensional embedding subspace.
Assumption 2: Both embedding manifold and layer-wise transformation are orthogonal. In this case, layer-wise transformations are represented by orthogonal matrices, satisfying 0, 0, = 0,0, =I Additionally, the basis vectors V.. V and V are considered to be mutually orthogonal,
(v Tv 0, (VDTVP- =0, (V)) VV = 0.
Based on these assumptions, the computational costs of the proposed control algorithm are similar to per- forming forward propagation with the original model. This implies that the PID control approach introduces negligible computational cost. The negative impact of these assumptions are discussed in Section 3.4
An analytic solution under Assumption 1. In the special linear case, for the linear embedding sub- spaces linked to the state, state integration, and state derivative, we define the basis as VP, Vi, and vP. respectively. Consequently, the embedding manifolds, represented by the surjective functions f(" ft. and f., are orthogonal projections Q", Q and Q," where
Qf -- v!(VP)T, - v((V),, -I=I-VP(VP]
The following proposition solves the objective function defined in equation under linearity assumptions.

---

Published in Transactions on Machine Learning Research (03/2024)
Proposition 1. Consider the following objective function,
min E (D [J y, 7. == min EE (xo.u)>D ((xT, y)
Σ
((1x)) Te. (Q Q. Q.
1=
s.t. X 0c(x; (x(()))).
(7)
the optimal value function, parametrized as V(x,) x; p,x, satisfies the Riccati equation:
(Q: 1 Q, 0, P.+10 (Q: + 20, P P.+10.) (Q: + 20, P + + P.+10)).
P: = 2
(8)
The optimal control solution is given by
xx,) = - (Q (Q + P. ((Q))))))..)) + 20 P.+10.)x..
(9)
where Q = Of + Q + o.
We provide an outline of the proof, with detailed derivation available Appendix g The optimal value function Vxx,) of the optimal control problem defined in equation 7 satisfies the Bellman optimality equation,
V(x.) [((x))== TT, (ff Q QP)) + ((x+1), s.b. Xr+1 0.(x, + (x:)).
In the linear case, the optimal value function is parametrized by quadratic function, expressed as V(x,) dV
x; Pix. By setting the derivative (x:( to zero, we arrive at the optimal control solution, as detailed in equation [ Furthermore, the Riccati equation in equation emerges from substituting this optimal control solution into the Bellman optimality equation.
Remark 2. As derived in equation D using combination of P. and D controllers incurs the same compulational cost using single lype of control scheme. This is due to the linearily of the control process, where the orthogonal projections onto the state embedding, state integration embedding, and state derivative embeddings can be effectively merged. This results in a projection onto the intersecting subspace of the three linear embedding subspaces.
Starting with a pre-trained LLM, the layer-wise transformations can be linearized to form a linear dynamical system. From this, the parameters the optimal value function P. are computed using the discrete dynam- ical system outlined in equation g Subsequently, the optimal foedback control solution (x,)) is constructed from equation own set of complexities. Moving forward, propose an analytic solution that does not rely on linearizing the base model, under additional orthogonality assumptions.
g Although this method is feasible, the linearization of series of transformer layers poses its
An analytic solution under Assumption 2. We further consider the scenario where both embedding manifold and layer-wise transformation are orthogonal. As result, the linear combination of orthogonal projections, represented as Q Qf"+ +f 0f++Q Qf,, forms an orthogonal projection itself. With these conditions in place, can then establish analytic formulation for the optimal control solution, detailed in the following proposition.
Proposition 3. When the layer-wise transformations are represented as orthogonal matrices, and the basis of state embedding, state integration embedding, and state derivative embeddings are mutually orthogonal, the optimal feedback control, denoted as Ti (x(x),, can be computed as follows:
0
Te (xx)) = Ve
0
V Xb
0 0
0
1+A111
0
I+he+ite
where the time-varying parameter X is governed by backward diffference equation X c(1+de+1) terminal condition specified as \r 0.
1+Amito with the

---

Published in Transactions on Machine Learning Research (03/2024)
A brief overview of the proof is presented here, while more detailed derivation be found in Appendix The condition for the embedding subspaces to be orthogonal guarantees that the linear combination represented by Q Qf + Q + Q forms an orthogonal projection. Moreover, the orthogonality in layer- wise transformations simplifies the Riccati equation. This simplification leads to a recursive approach to formulating control regularization.
When c = O, it holds that A, for every i, and the optimal feedback control corresponds the orthogonal projection onto the orthogonal complement of the linear subspace. On the other hand, for e 0, the approach yields time-varying regularization in control across different layers. This analytical solution, which assumes linear orthogonality, is independent of the underlying model. Therefore, the time-variant control regularization C can be pre-calculated prior to the inference process.
2.3 Theoretical Error Analysis
Under both assumptions 1 and 2 defined in Section are formulated as follows:
2.2 the controlled dynamics, under some perturbations,
X = 0,(x, + ne. (x.) X0 = Xo + z,
where z represents an arbitrary perturbation decomposable into two mutually orthogonal components z!l 3 zz' z!!, aligning within the data embedding subspace, and z orthogonal to the data manifold. We represent the state trajectory in the absence of input perturbation and without any applied control Xc+1 = (x:), the following theorem quantifies the error as |X, - x1112. which evaluates the difference between the perturbed state after control correction and the original state from unperturbed input data.
Theorem 4. For any time step & 2 1, assuming that each Og is an orthogonal matriz, we have the following error computation:
t-1 II
||x
x1113
a 1z 12 + ++ 12
0
where on time- varying parameter defined in relation to the control regularization c, and de are auziliary variables, as follows:
c(1 A4t1).
04 1 A411 +
AT =0 0, AT-1 = I+c'
A 1+ Ae41
The detailed derivation is provided in Appendix g This computation rigorously demonstrates that pertur- bations, specifically those spanning the orthogonal complement denoted by z,, exhibit a decay phenomenon during the process of forward propagation. Furthermore, scenarios where control parameters are subject regularization constraints, when c 0, our analysis reveals nuanced insights. We establish that the opti- mal control solution, which is derived by considering the intricate interplay among different transformation layers, adheres to these constraints while optimizing performance, which captures the complex dynamics between layers.
Theorem outlines how errors in state computations at any given time step are influenced by input per- turbations represented by z, despite these perturbations existing within the continuous domain Rd, this setting fits real-world adversarial attacks on LLMs, which involve modifying discrete elements, such as tokens or characters, in the input text. The act of modifying a word or substring through an adversarial attack leads to discrepancy between the embedding sequences of the original and modified input tokens, manifest- ing the perturbation vector within the input embedding space. Specifically, the embedding manifolds, derived from unperturbed training data, capture the structure of this data in a nower-dimensional subspace. Adversarial examples, meanwhile, are designed to be semantically similar to the original input yet. induce a marked divergence in the embedding space during the model's forward propagation. Under these circum- stances, the difference between the embodding sequences of the original input and the adversarial example can be quantified and adjusted within the PID control framework. We provide empirical error computation of Theorem 4 in Section 3.4
8

---

Published in Transactions on Machine Learning Research (03/2024)
Algorithm Tucker Decomposition.
Input: An L-way tensor X. Output: Core tensor 9, orthogonal basis V V2 V for i = to do
X = Reshape (X, i), // Reshape the tensor along the ith mode.
U', S V SVD (X'), // Perform singular value decomposition on the reshaped tensor.
Save the singular vectors V as the orthogonal basis.
end for g X, // Initialize the tensor core with the 1-way tensor X.
for 1 to do
g g x, V Multiply the core tensor by the 1th orthogonal basis.
end for
2.4 Additional Details for Constructing PID Control
Here, we provide details on implementing the proposed PID control method. We begin with constructing the linear embedding basis VP. V/, and VP from training dataset. In NLP tasks, the hidden states are generally represented as 2-dimensional matrix (sequence of embedding vectors), X₃ Rixd where / denotes the temporal length. Given N pieces of data sampled from the data distribution D (N training data), we can concatenate the hidden states as 3-way tensor XG E Rxxixd, and apply Tucker decomposition (De Lathauwer et al.) 2000b Kolda & Bader 2009 (known as high-order singular value decomposition) to generate linear embedding basis along both temporal and state embedding dimensions.
Tucker Decomposition is an extension of the traditional singular value decomposition to higher-order tensors. Mathematically, Tucker decomposition represents an I-way tensor X = = x1 V X2 V X3 X v where g is the core tensor, which governs the interaction between different modes, V are orthogonal bases corresponding to the principal components in each tensor mode, Xn is the mode-n tensor product. The mode-n product of a lensor A E R4ix pllxxxxla xla with a matrix U E IR Jxla is defined as
B == A bn
aig..in...dd Ujint
where the (i1, ia)-ttt elements of A and B are denoted as Qirig-naa and bi respectively.
An implementation of Tucker decomposition is detailed in Algorithm [ Along each of the I modes, the concatenated high-dimensional states X' are reshaped along the ith dimension, which used to compute the orthogonal basis from singular value docomposition. The core tensor is computed by multiplying the states X' with each of the I basis along each mode. The low-rank reconstruction of concatenated states X can be obtained by g X1 V X2 V X3 x, V
Given pre-trained LLM (naively trained or robustly trained), we collect the concatenated states from training data, which results in a set of 3-way tensors {x)) Then Tucker decomposition is applied at every X. (refer Algorithm [ Extending this to integral and derivative controls is straightforward, as one can substitute the concatenated states X by either the summation of past states X' Ei-p N, or the subtraction of two consequential states X; X Using the linear embedding bases V... V and vP obtained from Tucker decomposition, the construction of the feedback controller is achieved by adhering to the methodology outlined in Proposition 3
3 Numerical Experiments
In this section, we first discuss experimental setup in Section 3.1 In Sections 3.2 and 3.3 we assess the performance of the proposed PID control framework against various bascline methods across multiple NLP tasks. Subsequently, in Section 3.4 an ablation study is conducted, providing exploratory justification for the proposed approach.
9

---

Published in Transactions on Machine Learning Research (03/2024)
3.1 Experimental Setup
Evaluation methods: We consider both adversarial allack algorithms (e.g. A2T, PSO, TexiBugger, TextFooler), applied on the SNLI (Bowman et al.) 2015), MNLI datasets (Williams et al. 2018 and adver- sarial datasets (e.g. ANLI) to evaluate the robustness of the proposed PID control and basclines.
A2T (Attacking to Training (Yoo & Qi 2021 utilizes cost-cffective gradient-basod technique to rank the significance of words. This approach encompasses the iterative replacement of each word with synonyms sourced from counter-fiten -ittted word embeddings.
PSO Zang al. 2020 exploits a population of interacting individuals iteratively search for the optimal solution in the specific space.
2019] finds important words by computing the Jacobian matrix of the model and then chooses an optimal perturbation from the generated perturbations.
TextBugger (Li et al.
TextFooler (Jin et al. 2020) is the state-of-the-art word-level adversarial attack method to gener- ate adversarial examples. This technique identifies the important words for the target model and subsequently prioritizes their replacement with the most semantically similar and grammatically correct words. This process continues until there is a discernible shift in the model's prediction.
Adversarial NLI (ANLI) NNe et al. 2020 is large-scale NLI benchmark, This dataset was curated through an iterative process that incorporates both human and model inputs in an adversarial loop, largeting specific models for allack. The ANLI dataset is particularly polent as an adversarial lool, demonstrating a significant capability to diminish the accuracy of pre- pre-trained models.
Baseline methods: This study examines two baseline methods focused on adversarial training. The Naive adversarial training (AT), as proposed by Yoo & Qi [2021] employs the A2T attack for its adversarial training process. FreeLB, introduced by Zhu et al. (0019) implements adversarial training in language models during the tinettunig stage, aiming to enhance both generalization and robustness. It is noteworthy that the PID control method, in contrast to these adversarial training-basod approaches, offers distinct perspective on enhancing model robustness without knowing the attack type in advance. It can be applied to models that have undergone adversarial training further improve their robustness.
We fine-tune four baseline models using LoRA (Hu et al. 2021 namely distilbert (Sanh et al. 2019
BERT-large (Kenton & Toutanova 2019 RoBERTaBase and RoBERTaLarge (Liu et al. 2019).
Using a pre-trained model (e.g., BERT), we select training data that this model can accurately predict. Next, we simulate forward propagation using the pre-trained model on this specific set of training data, which generates a collection 3-dimensional tensors, denoted as {x)) Following this, we employ Algorithm [ each tensor determine the basis for a linear embedding subspace (see Section 2.4p. The dimension of this subspace is chosen based the criterion that it must account for 99% of the total variance observed (this is done by accumulating the singular values). Finally, the optimal solution outlined in Proposition 3 is implemented to generate a tine-dependent control regularization paramcter.
PID control implementation details:
Threat model: In this work, we consider word/token level adversarial attacks that manipulate discrete tokens or characters in the input text. We verify this from both empirical and thcorctical perspectives. the numerical experiments, we consider a range of adversarial attacks, including A2T, PSO, TextBugger, and TextFooler. These adversarial attacks aim cause misclassifiaation by modifying the tokens of the input string while maintaining the same semantic meaning.
3.2 Robustness Against Adversarial Examples
Here, we empirically validate the robustness improvement of employing the proposed PID control on pre- trained LLMs. In Figure 2 (a), a radar plot is presented illustrate the comparative performance between the baseline and controlled models, utilizing the DistilBERT architecture and evaluated on the SNLI dataset.
10

---

Published in Transactions on Machine Learning Research (03/2024)
Baseline
Controlled
A2T
A2T
A2T
A2TT
PSQ
PSO
P50
PSO
20 4g 6o 80
20 206060 60
4h 60 80
20% 40 60 80
None
None
None
None
TextBugger
TextBugger
TextBugger
TextBugger
TextFooler
Textfooler
TextFooler
TextFooler
(a): Distilbert (SNLI) (b): RoBERTa (SNLI)
(c): Distilbert (MNLI) (d): RoBERTa (MNLI)
Figure 2: (a) and (b) are radar plots that summarize Distilbert and RoBERTaLarge in Table for SNLI dataset, respectively. (c) and (d) are radar plots that summarize Distilbert and RoBERTaLarge in Table 9 for MNLI datasct, respectively.
This demonstrates that the employment of PID control significantly improves model robustness against all four distinet types of perturbations, with a negligible impact on performance with unaltered data (denoted as None). Shifting to a different model architecture, Figure 2 (b) reveals that applying the proposed PID control approach to the RoBERTa model yiclds analogous enhancements in robustness. For more challenging scenar- ios, Figures 2 (c) and (d) detail the performance of the MNLI dataset. In these plots, both the DistilBERT examined. These figures showcase that, despite the increased complexity of the MNLI dataset, the PID control method consistently maintains robustness improvements. The increased complexity of the MNLI dataset poses additional challenges in creating embedding subspaces, making it more difficult to accurately represent state, state integration, and state derivatives with linear embeddings. The plots distinctly highlight that the controlled models exhibit increased resistance to a broader spectrum of linguistic perturbations and complexities, without significant trade-olls in overall accuracy. This underlines the efficacy of PID control enhancing model robustness across different architectures and datasets.
and RoBERTa a architectures
More detailed comparisons of performance between bascline and controlled models the SNLI and MNLI datasets are provided in Tables 8 and g in Appendix 10 When a method's 's accuracy surpasses others by more than 1%, it's highlighted in red. It is evident that the proposed PID control method significantly enhances the robustness of both standard and robustly trained LLMs. The enhancement is more pronounced in standard trained models, which are generally more vulnerable to adversarial attacks. On average, the PID control method yields an improvement of nearly 10% in standard models and about 5% in robustly trained models, including both AT and FreeLB training.
In addition, we present the numerical results of OPT-1.3B. OPT-1.3B is decoder-based large language model that contains 1.3 billion model parameters. For the proposed PID control, we follow the same P- D control implementation against all four types of adversarial attacks. Specifically, on the SNLI dataset, the average improvement is over 20% compared with the base model. On more challenging MNLI dataset, with only a 2.5% accuracy drop on the unperturbed testing dataset. The improvement reaches 21 % against the TextBugger attack, and 11% both A2T and PSO attacks.
as done in all numerical experiments from the paper. that the controlled OPT-1.3B model consistently improves the robustness performance
proppottinallddervatie
Table 1: Measurement on SNLI dataset: baseline model / controlled model
SNLI Dataset
None
A2T
PSO
TextBugger
Text 16.64 / 44.70
OPT
91.24
88.69
49.15
63.28
48.00 / 60.06
17.57 / 41.79
MNLI Datasct
None
A2T
PSO
BextBugger 24.12 / 45.97
TextFooler 21.68 / 49.13
OPT
86.89
84.27
54.47
65.87
45.14
59.08
11

---

Published in Transactions on Machine Learning Research (03/2024)
Table 2: Measurement on ANLI dataset: baseline model / controlled model
rl
r2
r3
RoBERTaLarge (Dev)
72.60
72.65
50.99
52.33
40.99
43.31
RoBERTaLarge (Test
72.79
72.60
48.19
49.39
40.66
42.41
3.3 Robustness Against Adversarial Dataset
In this study, we assess the effectiveness of the PID control approach in an adversarial Natural Language Inference (NLI) task. The ANLI dataset is created through an iterative process involving both humans and models, aimed at improving natural language understanding. Initially, humman annotators create examples that challenge the current pest-performing models. These difficult examples, intended to reveal more weak- nesses, are then incorporated into the training set to enhance the models. This cycle of identifying and addressing weaknesses is repeated across several rounds, each producing an increasingly complex adversarial dataset (ANLI consists of three rounds of development and test datasets). Unlike the evaluation using ad- versarial examples described in Section 3.2 the ANLI dataset is pre-constructed by human annotators. In contrast, adversarial examples from Section 3.2 are created in relation to the specific characteristics of the underlying classifier.
The evaluation with the ANLI dataset encompasses both baseline and controlled models, utilizing the de- velopment and test. datasets. The results obtained from the ANLI dataset outlined in Table ANLI involves three progressively challenging rounds. The baseline model shows decline in performance with increasing difficulty from round to round 3. Conversely, the PID control demonstrates a more pronounced improvement in performance as the challenge increases. Specifically, the proposed control method leads to 1.0783% in the mean of performance improvement, and a 95% confidence interval of 0.0564% to 2.1004%
3.4
Ablation Study
This section provides exploratory justifications for the proposed PTD control framework.
Justification of the selection of a P-D control scheme. We begin with comparative analysis of various control schemes, emphasizing the benefits of implementing multiple controllers over the single use of Proportional (P) control as previously explored in (Chen et al. 2020). Table 3 showcases a comparison of the robustness performance across Proportional (P), Proportional Interraa (P-I), Proportional iroportional-Derivattvee Derivative
(P-D), and ProportionallIntege training methodologies. is evident that the P-D control scheme significantly surpasses the others in most scenarios, underscoring the efficacy of the proposed PID control framework, which expands upon the limited capability of earlier control (closed-loop control) methods. The mean of employing the Proportional- Derivative (P-D) control over the Proportional (P) control is 2.35%, with 95% confidence interval of 1.677% to 3.0121%. This validates the choice of P-D control.
(P-I-D) control schemes within different model architectures and
The reason why P-D outperforms P-1-D is mainly due to noise sensitivity and hyperparameter tuning.
Noise sensitivity: The integral term has the potential to aggregate errors across multiple hidden layers, incorporating noise inherent. in the embedding manifolds, as well as the distributional shift between the training and testing datascts. scenarios where substantial noises are presented in cach hidden layer, the integral component, dependent. on the embedding manifold of accumulated past. states, may lead to instability during model inference. Conversely, a Proportional- Derivvttve (PD) controller, lacking the integral component, tends to exhibit. improved performance under such noisy conditions by not accumulating this noise.
Hyperparameter tuning: In the realm of traditional PID control design, selecting the appropriate control gains, denoted as Ky; K2, and K2, for proportional, integral, and derivative controls respectively, presents a notable challenge. These gains are crucial for achieving a balance among the different types of controls. Typically, the calibration of these gains is empirically based, with the aim of optimizing the performance of
12

---

Published in Transactions on Machine Learning Research (03/2024)
Table 3: Measurement on SNLI dataset: P P-I P-D P-I-D
Distilbert
Standard training
Adversarial training
FreeLB
A2T
60.11
58.24/ / 62.31
61.30
72.09 / 71.12
71.81
/ 71.97
59.63
57.90
62.95
61.88
PSO
53.39
52.67
54.96
53.96
55.80
54.72
57.87
/ 56.35
54.40
53.91
56.86
/ 55.89
TuggBu
37.15
37.15
/ 40.26
39.54
38.91
38.98
41.64
/ 40.98
32.79
32.24
37.80
36.21
TextFooler
36.81
34.84
/ 41.73
38.98
40.15
/ 38.21
43.81 41.12
32.49
31.22
39.64
37.39
RoBERTaBase
Standard training
Adversarial training
FreeLB
A2T
61.78
60.81 / 64.11 61.94
76.63 / 75.82 77.08 / 76.28
65.93 / 65.04/
68.85 / 66.59
PSO
53.34
52.94 / 54.40/ / 53.35
55.49
54.76
56.45 / 54.99
53.64 / 52.99
55.24 / 53.55
Tegger
40.46
39.00
/ 43.20
/ 40.53
41.71
/ 40.22
43.35
/ 41.33
39.72
38.24
/ 42.75 / 40.17
Fexteooler
33.19
32.10
37.35
33.83
34.48 / 32.47
39.39 / 35.29
30.75
29.77
36.81
32.21
BERT- large
Standard training
Adversarial training
FreeLB
A2T
75.75 / 75.68 / 75.54 / 75.60
86.13 / 86.03 85.76 / 85.92
78.16
78.16 / 78.21
78.04
PSO
67.72
67.69
67.55
67.60
70.21 / 70.26 70.38 / 70.25
65.49
65.46
/ 65.56
65.46
TextBugger
64.59
64.53
64.41
64.36
69.74 / 69.89
69.55 / 69.62
59.28
59.35
59.29
59.27
Foole
58.48
58.25
58.27
58.12
65.43 / 65.25/ 65.27 / 65.10
55.34
55.27
55.26
55.14
RoBERTaLarge
Standard training
Adversarial training
FreeLB
A2T
65.10/ / 64.89/ / 64.95 64.38
81.91 / 81.72 81.62 / 81.80
70.38 / 70.40
71.30 / 70.51
PSO
55.83
55.04 / 56.70 55.31
57.99 / 57.29 59.71 / 58.18
56.23
55.60
57.20 / /56.18
TextBugger
44.61
/ 42.20/ / 42.43/ / 41 11.29
45.00/
/ 43.54/
44.74
/ 43.53
44.52
/ 43.11 / 44.42 / 43.21
TextFooler
36.63
35.52 / 37
35.39
39.64 / 37.06/
42.44 / 39.87
37.56
/ 35.97
(38.59 / /36.71
PID control. Our method follows a similar strategy, determining the gains through experimentation with training data. Given that our hyperparameter searching space only contains and 0.5 for each control gain, this results the values K₂ = 0.5, K2 0.5, and K2 = more principled method would entail adjusting these hyper-parameters through numerical optimization, treating these control gains as adjustable variables. The development of more sophisticated strategy for fine-tuning the control gains will be studied for future exploration.
Discussion on the linearity and orthogonality assumptions. Here we discuss the negative impact of violating the assumptions made derive the analytic solution. Through empirical evaluations, we highlight how the main assumptions have increasingly adverse effects, especially when the embedding manifolds fail to accurately capture the complex, high-dimensional states. More specifically, applying regularization control solutions can mitigate these inaccuracies. However, the precision of the embedding manifolds decreases, a greater degree of regularization is required, thereby complicating the optimal control problems. The increased complexity in the optimal control problem makes the negative impact of violating the main assumptions more significant.
Our validation approach involves a performance comparison between the proposed analytic solution and the implementation of Pontryagin's Maximum Principle, an iterative solver that operates without the need for additional assumptions. Pontryagin's Maximum Principle provides the necessary conditions for optimal control solution, typically offering a robust approximation of such solutions. We further claborate this comparison by creating linear embedding subspaces with varying thresholds for accumulated variances, specifically aiming to capture 99%, 95% 90%, and 85% of the variances in the underlying states. As the variance threshold is lowered, the accuracy of these embedding subspaces decreases, thus posing greater challenges in solving optimal control problems. The performance comparison, detailed in Table 4 includes three I.LMs across five evaluation tasks. These tasks include standard scenario with no perturbation
13

---

Published in Transactions on Machine Learning Research (03/2024)
Table 4: Performance Comparison (Analytic Solution / PMP)
Distilbert
Base
0.99%
0.95%
0.9%
0.85%
None
87.23
85.88
/ 86.52
68.92
80.21
34.24
54.06
34.28
46.75
A2T
53.89
61.75
/ 60.87
57.93
62.88
34.10
44.17
34.28
42.01
PSO
49.84
54.33
/ 52.80
56.21
58.22
34.13
46.27
34.28
42.55
Bugger
24.73
40.35
36.69
43.89
42.50
36.24
39.02
34.28
42.20
Tex(Fooler
24.69
40.28
36.13
49.05
46.69
34.37
39.56
34.28
40.80
RoBERTaBasc
Base
0.99%
0.95%
0.9%
0.85%
None
90.87
90.10 / 90.59
85.09 / 89.63
64.60
85.82
40.02
76.71
A2T
38.36
63.82
/ 62.19
65.12 / 66.44
51.19
66.86
37.56
60.41
PSO
51.44
54.36 / 52.98
59.97 / 56.75
52.55
59.18
37.91
59.07
TextBugger
35.90
43.03
/ 40.53
46.74 / 46.41
38.72
47.36
34.84
42.43
TextFooler
27.03
37.18 / 33.47
47.16 / 42.79
41.40
46.60
35.76
45.49
RoBERTa
Base
0.99%
0.95%
0.9%
0.85%
None
92.39
91.98 / 92.11
86.50 / 91.68
66.40
90.53
44.54
86.52
A2T
59.40
64.64 / 63.15
67.17
65.03
54.67
64.99
41.54
61.94
PSO
52.15
56.62
/ 54.35
62.14
/ 55.51
55.13
58.41
41.15
58.85
TextBugger
33.72
42.39
39.18
47.48
41.59
41.30
43.77
35.49
40.32
TextFooler
26.43
36.92
/ 32.27
48.79
37.14
47.09
41.43
40.47
41.97
and four adversarial attacks: A2T, PSO, TextBugger, and TextFooler. The results reveal that while the performance difference between the analytic solution and Pontryagin's Maximum Principle is negligible at higher accuracy levels (e.g., 99% variance), the scenario changes significantly at lower accuracies (e.g.. 90% and 85% variances). In these instances, employing Pontryagin's Maximum Principle, which operates independently of simplifying assumptions, yields noticeably better control solutions.
Computational wall time comparison. Here we present a detailed discussion of the computation over- head of the proposed PID control method. Specifically, we compare the computational wall time between the base model without any controls applied, the proposed analytic solution, and Pontryagin's 's maximum princi- ple employed in the previous closed-loop control approach. As shown in Table 5 across all four models, the computational wall time between the base model and the proposed analytic solution is comparable, the ana- lylic solution only adds small amount wall time during inference. However, solving the PMP significantly adds the computational wall time of the base model.
Table 5: Computational Wall Time
Wall Time (s) of 10, 000 Test Samples (averaged over 5 experiments)
Distilbert
RoBERTaBase
RoBERTaLarge
OPT
Base model
6.3751
11.7756
36.0178
123.5379
Controlled model
6.4221
11.8620
36.5051
124.1090
PMP
62.2667
81.2795
263. 7920
757.0649
Error computation of the main theorem. We provide the details of the error computation outlined in Theorem 4 Our objective is to demonstrate that the accuracy the error computation specified in Theorem diminishes with the addition of more layers to the language model. This decrease in accuracy is due to the assumptions of lincarity and orthogonality. According to these assumptions, the transformations applied each layer of language model merely rotate the hidden state without altering its magnitude. However,
14

---

Published in Transactions on Machine Learning Research (03/2024)
the model incorporates more of these layer-wise transformations, the accuracy of these assumptions starts to decrease.
Table 6 presents the calculation of the absolute difference between the actual error and the error estimate as per Theorem !
It is evident that with all types of adversarial perturbations (A2T, PSO, TextBugger, and TextFoolery, the increase the number of layers within the language model (with 6 layers representing Distilbert, 12 layers symbolizing RoBERRaaaae, and 24 layers signifying RoBERIaaLage) leads to rise in the absolute error. This indicates decline in the precision of the error estimation.
Table Error Comparison (difference between Theorem and true error)
Layers
12 Layers
24 Layers
A2T
3.2189
4.3062
5.1566
PSO
1.9156
2.6087
3.3047
TextBuuger 3.2189
4.3062
5.1566
TextFooler
3.1348
4.2894
5.2915
Justification on the effectiveness of the PID control framework. Here we provide evidence that the PID control framework can improve model robustness against adversarial examples. As detailed in Theorem the working principle of the PID control framework is based on the two facts:
There exists embedding structure in the state at every layer. 7 the existence of lower- dimensional embedding subspaces. With the OPT-1.3 B LLM (only the first 6 layers are shown), the dimensions of proportional, integral, and derivative embedding subspaccs are presented. As can be seen, the dimension of proportional embedding is around 350 on average in 2048 dimensional space, integral and derivative embedding subspaces also show low dimensions compared with the full space.
The sequence of states from adversarially perturbed input deviates from the true embedding struc- tures. Table presents the error (measured in 2-norm) detected by the combination of P, and D embedding subspaces. As be seen, the perturbation aims to amplify the error propagated into deeper layers, and the embedding subspaces can effectively detect these errors at all layers.
Table 7: Ranks and Embedding Errors
Ranks: OPT-1.3B (first 6 layers)
Layer 1
Layer 2
Layer 3
Layer 4
Layer 5
Layer 6
Proportional
191
2048
148
2048
224 / 2048
337
2048
451
2048
549 / 2048
Integral
191
2048
181
2048
180 / 2048
212
2048
253
2048
298 / 2048
Derivative
191 2048
355
2048
1001
2048
1352
2048
1494
2048
1535
2048
Embedding error:
OPT-1.3B
(first 6 layers)
OPT
1.8237
9.5877
6.9810
6.3207
7.5278
16.6526
4
Related Works
We delve into the existing body of literature surrounding robustness issues in NLP tasks (Section 4.1 Moreover, we explore the realm of machine learning from an optimal control perspective, emphasizing its relevance and applicability to the task at hand (Section 4.2h.
4.1 Robustness in NLP
In recent years, variety of approaches have been developed for generating effective adversarial attacks in the context of NLP. Traditionally, text attacks produced through direct modifications to sentences at the
15

---

Published in Transactions on Machine Learning Research (03/2024)
character level, word level, sentence level, combination of these Ren et al. (0019) Li et al. (0118 Xu
et al. (2021). Studies such as those by Liu et al. (2020b); Belohlávek et al. (2018); Zhu et al. (2019) explore methods for generating attacks within the text embedding space. An alternative approach by Wallace et al.
(2019) involves prepending the same tokens to all texts for attacks. Moving away from adding perturbations
to samples, La Malla& & Kwiatkowska (0022 Wang el. al. (2020) inlegrate generative models for the creation
of attacks.
In response to adversarial attacks, most defense strategies primarily rely on the principles of adversarial training Morris Morris et al. (2020) Jiang al. (2019); : Wu et al. (2022). Given that adversarial training is inherently
resource-ineensive both in terms of computation and time, studies such as Zhu et al. (2019) Zhang et al.
(2019) 5 Shafahi et al. (2019) have proposed methods to expedite the training process. However, despite the
efficacy of adversarial training, there's evidence suggesting that fine-tuning model parameters can diminish performance on clean data He et al. (2021). Additionally, it has been observed that adversarial training often produces suboptimal results when faced with novel, unforeseon perturbations Tramer & Bonch (2019). Adversarial training is analogous to open-loop control, as it leverages set of fixed controls (e-g- model parameters) for any new, unobserved data. In contrast, the proposed PID control approach dynamically focuses on inputs using feedback controls.
4.2 Deep Learning with Optimal Control
Recent studies have increasingly focused on elucidating the intricate connections between dynamical systems and deep learning, highlighting the fundamental interrelations that exist between them (E 2017 Haber &
Ruthotto 2017 Li et al. 2018b). These studies have established a robust theoretical framework, offering novel perspective to comprehend deep learning methodologies through the lens of optimal control theory [Liu & Thcodorou| 2019). The work of Li et al. (2018b) and Li & Hao (0018) have successfully bridged the gap between the classical back-propagation algorithm and optimal control theory. This intersection notably highlights how Pontryagin's 's Maximum Principle (Kirk 1970), cornerstone of control theory, aligns closely with gradient-based training methods in neural networks.
In advancing this line of inquiry, E et al. (2018) has developed the theoretical foundations for interpreting deep learning within the optimal control franework. These insights have laid the groundwork for subsequent research that applies optimal control principles to address key challenges in deep learning. A notable example of this is the work by (Liu et al. 2020a), which introduced sophisticated high-order optimization strategies rooted in differential dynamic programming. This methodology has been instrumental in enhancing the training process's convergence rates and stability. Moreover, the closed-loop control framework has been proposed to improve the model's robustness against adversarial attacks (Chen et al. 2020 2022) and fairness issues Chen et al. 2023
5 Discussion and Future Works
The wider perspective of the proposed method on the trustworthy ML: The presented PID control approach generalizes previous closed-loop control approaches with additional integral and deriva- tive controllers. This development. leads to more flexible control schemes, derivative controllers are more effective when the underlying states change rapidly, integral controllers play more significant roles when lower-dimensional embedding structures can be constructed in accumulated states. Such flexibility in control design broadens the applicability of the control framework across variety of trustworthy ML applications.
This work paves the way for the development of robust large language models. Presently, many large lan- guage models face challenges related to trustworthiness, including biases against minority groups in natural language generation tasks. In principle, by constructing state embedding manifolds that capture desired model behaviors, the PID control framework can be employed to adjust any unwanted bohaviors in the model. This idea is similar to prompt engineering techniques used to modify input strings for achieving specific outcomes from models. These avenues will be explored further in future research.
16

---

Published in Transactions on Machine Learning Research (03/2024)
How this complements the research on adversarial attacks? The PID control framework leads to new method to generate adversarial attacks. In the current work, the aim is to improve model robustness by minimizing the objective function defined in Equation 2. On the contrary, maximizing this loss w.r.t, some input perturbation is equivalent to generating adversarial examples. This can be an optimal control-basod adversarial attack algorithm.
Can the same method be used for vision problems? This method is applicable to computer vision problems. Typically, in deep convolutional neural networks, both the input and hidden states lie in extremely high-dimensional spaces, where the embedding manifolds for these states tend to exist. This assumption aligns with the 'manifold hypothesis." which is based on the characteristics of real-world image data, and is further supported by empirical evidence as demonstrated in the studies by Chen et. al. (2020) and Chen et al. (2022). Once the embedding manifolds for both imput and hidden states are constructed, it becomes possible to formulate the optimal control objoctive function as outlined in Equations and 3. By aiming to minimize this objective function, there is potential to significantly improve the robustness of the model.
How does this complement the existing trustworthy ML literature? The current body of re- search on trustworthy machine learning predominantly emphasizes adversarial training, which leads to two significant challenges. Firstly, the process of modifying model parameters with adversarial examples de- mands extensive computational resources. In the context of natural language processing tasks, identifying an adversarial example typically entails solving a combinatorial optimization problem, which suffers from an exponential growth in the number of feasible solutions as the size of the problem increases. Secondly, adversarial training's efficacy diminishes in the face of unexpected adversarial attacks. This shortcoming is especially evident in the real-world application of large language models, where predicting potential ad- versarial attacks beforehand is unfeasible. The suggested PID control framework is designed to overcome these challenges by offering two key advantages: 1) It does not significantly increase the inference time when compared to the base model, and 2) It leverages the embedding structure of unperturbed states, making it robust against various unforeseen adversarial allacks.
Extension to other trustworthy issues. The proposed PID) framework focuses on improving the ro- bustness of pre-trained models through set of linear embedding subspaces. These subspaces effectively encapsulate the embedding structure of the underlying states. This framework may be broadened to tackle various trustworthy concerns in machine learning, including issues related to fairness (Chen et al., 2023) To achieve this, it is necessary to develop embedding subspaces that are capable of capturing embedding structures that are invariant against demographic information. Our next objective is to demonstrate how the PID control framework can be adapted to manage and resolve fairness-related challenges.
Analytic solution on the nonlinear dynamics. In our approach, we determine the optimal control solution using analytical method outlined in Proposition 3 This analytical method is based on the assumption that the layer transformation lincarization in the pre-trained model is orthogonal. Conscquently, this leads to a Lime-varying control regularization across various layers, which is independent. of the pre- trained model. This method has yielded satisfactory empirical outcomes. Nonetheless, there is a need for a more precise analytical solution that accounts for the intrinsic aspects of the underlying model. To achieve a more refined analytical solution for the optimal control issue, our next step involves lincarizing the non nonelinear layer transformations in the pre-trained model and subsequently utilizing the Riccati equation to generate the optimal control solution.
6 Conclusion
Our study has introduced novel PID control framework to improve neural network robustness against (unforeseen) input perturbations, outperforming traditional adversarial training methods. This approach maintains computational efficiency, enhances robustness in large language models, and allows for rapid online inference. Our comprchensive error analysis has confirmed the framework's effectiveness in simulated environments, contributing significantly to neural network security and robustness, and paving the way for more reliable NLP models in critical applications.
17

---

Published in Transactions on Machine Learning Research (03/2024)
7
Acknowledgements
Z. Chen and Z. Zhang supported by the NSF grant #2107321 under the CCF division. Q. Li is supported by the National Research Foundalion, Singapore, under the NRF fellowship (project No. NRF-NRFF13- 2021-0005).
References
Richard Bellman. On the theory of dynamic programming. Proceedings of the National Academy of Sciences of the United States of America, 38(8):716, 1952.
Petr Bělohlávek, Ondřej Plátek, Zdeněk Zabokrtský, and Milan Straka. Using adversarial examples in natural language processing. In Proceedings of the Fleventh International Conference Language Resources and Evaluation (LREC 2018), 2018.
Korte Bernhard and Jens Vygen. Combinatorial optimization: Theory and algorithms. Springer, Third Edition, 2005., 2008.
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632 642, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901, 2020.
Zhuotong Chen, Qianxiao Li, and Zheng Zhang. Towards robust neural networks via close-loop control. In International Conference on Learning Representations, 2020.
Zhuotong Chen, Qianxiao Li, and Zheng Zhang. Self healing robust neural networks via closed-loop control. The Journal of Machine Learning Research, 23()):14399-1488) 2022.
Zhuotong Chen, Qianxiao Li, and Zheng Zhang. Asymptotically fair participation in machine learning models: an optimal control perspective. arXiv preprint ar rrXii:2311.10222 2023.
Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. On the best rank-1 and rank-(ri, r2, ... .r) approximation of higher-order tensors. SIAM journal on Matrix Analysis and Applications, 21(4):1324 1342, 2000a.
Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. A multilinear singular value decomposition. SIAM journal Matrir Analysis and Applications, 21 21():11233-12 1278, 2000b.
Weinan E. A proposal on machine learning via dynamical systems. Communications in Mathematics and Slatistics, 5(1):1-11, 2017.
Weinan E, Jiequn Han, and Qianxiao Li. A mean-field optimal control formulation of deep learning. Research in the Mathematical Sciences, 1(6):1-44) 41, 2018.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4):983-1049, 2016.
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34(1): 014004, 2017.
Awni Hannun, Carl Caso, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjoev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speoch: Scaling up and-to-end speech recognition. arXiv preprint are: 142.:1122.500 2014.
18

---

Published in Transactions on Machine Learning Research (03/2024)
Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng. Analyzing the forgetting problem in pretrain-finetuingg of open-domain dialogue response models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1121-1133, 2021.
Markus Hehn and Raffacllo D' D'Andrea. Real-time trajectory generation for quadrocopters. IEEE Transactions on Robolics, 31 (4):8877 892, 2015.
Chung- Wen Ho, Albert Ruchli, and Pierce Brennan. The modified nodal approach to network analysis. IEEE Transactions on circuits and systems, 22(6))504-504, 1975.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Represen- tations, 2021.
Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv: 05344, 2019.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient finc-tuning for pre-trained natural language models through principled regularized optimiza- tion. arXic preprint arXiw:1911.03459 2019.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 8018-8025, 2020.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pro-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pp. 4171-44 4186, 2019.
Donald E Kirk. Optimal control theory: an introduction. Springer, 1970.
Tamara G Kolda and Brett W Bader, Tensor decompositions and applications. SIAM review, 51(3):455-500) 500, 2009.
Emanuele La Malfa and Marta Kwiatkowska. The king is naked: on the notion of robustness for natural language processing. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 36, pp. 11047-11057, 2022.
Jangjoon Lee, Srikar Bhagavatula, Swarup Bhunia, Kaushik Roy, and Byunghoo Jung. Self-healing design in deep scaled emos technologies. Journal of Circuits, Systems, and Computers, 21(06))1240011, 2012.
Li, S Ji, T Du, B Li, and T Wang. Textbugger: Generating adversarial text against real-world applications. In 26th Annual Network and Distributed System Security Symposium, 2019.
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. Textbugger: Generating adversarial text against real-world applications. arXiv preprint arx 05271, 2018a.
Qianxiao Li and Shuji Hao. An optimal control approach to deep learning and applications to discrete-woight neural networks. In International Conference on Machine Learning, pp. 2985 2994. PMLR, 2018.
Qianxiao Li, Long Chen, Cheng Tai, and Weinan E. Maximum principle based algorithms for deep learn- ing. Journal of Machine Learning Research, 18(165):1-22 29, 2018b. URL http://jmar..or//ppprrrtttt 17-653. html
Guan-Horng Liu and L'vangelos A Theodorou. Deep learning theory review: An optimal control and dy- namical systems perspective. arXiv preprint XrXiv:198.. 10920, 2019.
Guan-Horng Liu, Tianrong Chen, and Evangelos A Theodorou. Differential dynamic programming neural optimizer. XiXio preprint or orXi:22002.0880 2020a.
19

---

Published in Transactions on Machine Learning Research (03/2024)
Hui Liu, Yongzheng Zhang, Yipeng Wang, Zheng Lin, and Yige Chen. Joint character-level word embedding and adversarial stability training to defend adversarial text. In Proceedings of the AAAT conference on artificial intelligence, volume 34, pp. 8384 8391, 2020b.
Yinhan Läu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint ArXio:1907. 1 11692, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: framework for adversarial attacks, data augmentation, and adversarial training in nlp. arXiu preprint arXiv:2005.050) 05900, 2020.
Ani Nenkova and Kathleen McKeown. A survey of text summarization techniques. Mining text data, pp. 43 76, 2012.
Yixin Nic, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kicla. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeling of the Association for Computational Linguistics, pp. 4885-4901, 2020.
Lev Semenovich Pontryagin. Mathematical theory of optimal processes. CRC press, 1987.
Indika Rajapakse and Mark Groudine. On emerging nuclear order. Journal of Cell Biology, 192(5):711-721, 2011.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 1085 1097, 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smnlleer, faster, cheaper and lighter. ar preprint Xrxiv:1910. 111:1910.01108, 2019.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studor, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advances in Neurul Information Processing Systems, 32, 2019.
Adrian Tang, Frank Hsiao, David Murphy, I-Ning Ku, Jenny Liu, Sandeep D'Souza, Ning-Y Yi Wang, Hao Wu, Yen-Hsiang Wang, Mandy Tang, et al. low-overhead self-healing embedded system for ensuring high yield and long-term sustainability 60ghz 4gb/s /s radio-on-a-chip. 2012 IEEE International Solid-State Circuits Conference, pp. 316 318.. IEEE, 2012.
Florian Tramer and Dan Boneh. Adversarial training and robustness for multiple perturbations. Advances in neural information processing systems, 32, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
G Vinodhini and RM Chandrasekaran. Sentiment analysis and opinion mining: survey, International Journal, 2(6):282-292) 2012.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nlp. arXio preprint arXiv:1908.077 07125, 2019.
Ren Wang, Tianqi Chen, Stephen Lindsly, Cooper Stansbury, Alnawaz Rehemtulla, Indika Rajapakse, and Alfred Hero. RAILS: A robust adversarial immune-inspired learning system. arXiv preprint arXiv:2107.028.4 2021.
20

---

Published in Transactions on Machine Learning Research (03/2024)
Tianlu Wang, Xuezhi Wang. Yao Qin, Ben Packer, Kang Li, Jilin Chen, Alex Beutel, and Ed Chi. Cat- gen: Improving robustness in nlp models via controlled adversarial text generation. or preprinl arXiv:2010.0233 2020.
Matthew Wicker, Luca Laurenti, Andrea Patane, Zhuotong Chen, Zheng Zhang, and Marta Kwiatkowska. Bayesian inference with certifiable adversarial robustness. In International Conference Artificial Intel- ligence and Slatistics, pp. 2431 2439. PMLR, 2021.
Adina Williams, Nikita Nangia, and Samuel Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapler of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers),
pp. 1112 1122, 2018.
Hongqiu Wu, Yongxiang Liu, Hanwen Shi, Min Zhang, et al. Toward adversarial training on contextualized language representation. In The Eleventh International Conference on Learning Represenlations, 2022.
Ying Xu, Xu Zhong, Antonio Jimeno Yepes, and Jey Han Lau. Grey-box adversarial attack and defence for sentiment classification. arXiv preprint 1151: 2021.
Jin Yong Yoo and Yanjun Qi. Towards improving adversarial training of nlp models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 945-956, 2021.
Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. Word-level textual adversarial attacking combinatorial optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6066 6080, 2020.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Ac- celerating adversarial training via maximal principle. Advances in Neural Information Processing Systems, 32, 2019.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. In International Conference on Learning Representations, 2019.
21

---

Published in Transactions on Machine Learning Research (03/2024)
8 Appendix A
In this section, we claborate on the derivation of the analytic solution presented in Propositions and Let O, represent the thh linear transformation, and Ti Rd + Rd denote the PID controller. The controlled dynamical system can be expressed as:
X
(x(x, + me (x,)),
where the control action is added to the current state. Recall the running loss defined equation
C((x))'-or ne, S(.s)
1 11 + mx(x)) 12+ (x+ + (xi) + Σ xx) 122 211e (xt + +x()) - xx ==
Gt 2
(x,))
==0
consider the surjective mappings ff f and JP orthogonal projections. Let Qf Q and of be the orthogonal projections associated with ft ft, and fP, respectively, assuming a uniformly bounded state space with maxxex x12 S B, the running loss can be bounded as follows,
f((x,) n. (Q. Q.
2 110 (x)))) Q (x+ + (x,)
+
QP (x, + (xx))-xxxx- xx-1)11 + || (x,) x, 11
==0
(xe 2 11 (x: + (xx of (xx + +x
of Σ xx)
) 11 Q (x+ +x((x))
1 Q x-111
$ 10 (xi + (x(x))) Q (x, + +xx))))++ (x, (x:)))1
1 x
x
2 |
/ (x, (x,x)) of
(x,, (x, + n. (x,))) Q| (xx 1
m (xx))11
+
2 |T (x, )11
TB
(10)
2
where T represents the maximum number of layers of the neural network, and B is the uniform upper bound for the state space.
Let Q: Qf + Q + +.P, the following Lomma derives the analytic solution for the PID control ny (xx).
Proposition 5. Consider the following objective function,
min (xo.)))>D [[(x,, v. == min E (xx.w))D T(XT. u) Σ Te (. QQ,, 22 Q.7)
s.t. X 0c(x: - (x()))).
(7)
the optimal value function, paramelrized as V(xi) = x, Pixe, salisfies the Riccali equalion:
P. = Q. + 0, P.+ O. 1 (Q. + 20, P (.+10.)((00 (Qr + 20, Pr+10, + aal) - + 20, P.+
(8)
The optimal control solution is given by
(x(xx) = - + I+ 2 P.+10.) 20, P.+10.)x1.
(9)
where Q Of + Q/ QP.
22

---

Published in Transactions on Machine Learning Research (03/2024)
Proof. In the objective function defined in equation 7 the terminal loss $(xr, y) quantifies the discrepancy between the terminal state and the true label y- However, in general machine learning applications, the true label remains unknown during online inference, leading to the terminal loss being set to zero. Recall equation 10 the running loss is defined as
(((x))-00 Tes (Q., 0 QP))
= 1 10 1f (xt + ( (xx)) - (10 (x + Σ x1) 11Q Q/((x (x: + (x())) xx-1) + Tr (xx)
Consequently, the optimal value function Vxx,) satisfies
1 V(x.) min _ Q x, + (x (Q) x, Q₂ (x:)) + マ 1 + (x.)) (x)))
1
m
+ 2 O/ x, Q (x,)) x Qp प. xx )) + (x()) (x(xx) + (xx+1)),
s.t. X++1 = 0, (x + (x{)).
Taking the derivative of the right-hand side with respect to (xt) yields
dV,)
dx
(V(x++1)
Q!x: + + (x)
Q x + Q m (x1) + cTT(xx)
(x,();)
di (LTr(x1) (xx)
dx
Q²x; + 0/
Q (xx) + cTTe - -20, P P1+1X111:
= =QQ. + Q
x + Qf + Q/ Q |z (x) + cm,, (x,) + 20, P P.+10.x, 20, Pe+ 1e (x1);
= =Q,x, + Q my (x,) + cm (x,) + 20, P.+ 0,x, + 20, P 0,,10,,((x,))
where Q = =f + Qf Q..
Setting the derivative
(x4) dar, (x,) to results in the optimal control (x,) (as shown in equation
(x+) -(Q, + c I+ 20 P.+ 10+ + 20, P.+10,)x..
Parametrizing the value function V (xx)) as x P:x, and considering the optimal control solution equation can convert the expression of the value function follows,
x, P.x.
1
min 2
(x))) (0/x, + Q(m((x)) + (x)((x)) 1QPx, x, Qx [Qxx Q (x+))
QQ x,'',=-3
2 Tx,xx) (x(x,) + x P.+1X
1 x xT of + _ + 20. P 0 x マタ ( (x.)) T( (Qf + Qf + Qf +c + 20, Pt+10.)777x
xx Q/ Q? 20, P 0 (x1),
xx T (Q; + 20. P 0x)x, 2 (x) T (Q: + cl + 20. P.+10,) 0,) + x (Q, + 20. P.+
where (x:) is the optimal control solution leading to the minimum, Q: = Qf + Q + Q. For the second term in the above, recall the optimal control solution m. (x:) from equation
2 (TT (xx)) (T)))((Q) (Q, + 20, P 1(+10,)
- - -I-++-222 -0 P.+10 -( + 20, P P.+10.)x.) (Q: + 20,
x- (Q: + 20, P.+10.)
23

---

1 0 Published in Transactions on Machine Learning Research (03/2024)
P.-10,)-1 the above uses the fact that (Qr P(-10))))) is symmetric. Therefore,
x Pix.
T (Q: 20, P.+10,)x:
2 x (Q: +0, P(+10)))) + (Q. + +2, ((+10))
x,
2x (Q: (Q, 20, P(+10,)xx P,,
+ 2 (Q) Q + (1)
which results in the algebraic Riccati equation
1 1
1 -0, + 20, Pct10:) + 20, P++10++c))+ + +20 P.+10,).
P. - Q 0, 2
Pe+10 - 2
0
In analysis, focus specific scenario where each linear transformation 0, is both orthogonal and full-rank. This implics that the lincar transformations satisfy the condition 0, 0, 0,0, = I for all t in the considered range.
Recall that Q: = =f-" + + QP, where
Qf =I (VV), -I-I-V -I-
are orthogonal projections corresponding linear embedding subspaces state, state integration, and state derivative. For simplicity, assume that the basis VP. V and VP mutually orthogonal to each other, meaning that
vV = (VP) 0, (V)) =.
Based on this assumption, the combination of three orthogonal projections Qt is an orthogonal projection,
0
0
Q V?
(VP)T + V
(V))T T. + vP
(VP)T,
0
1
O
1
0 1
0
O
0
0
V
0
0
VT.
0
0 1
where V, represents the basis for the intersection of VP, Vl, and VP.
Lemma 6. Consider T -laaye neural network characterized by orthogonal linear transformations. The solution to the algebraic Riccati equation, as delineated in equation is given by
0
0
0
0
0
0
P V
V.T
(11)
0
O
0
0
A
O
0
O
X
where the parameter d is governed by backward differrnce equation X c(1+de+1) specified as \r 0.
1t with the initial condition
24

---

Published in Transactions on Machine Learning Research (03/2024)
Proof. The proof proceeds by induction t. Recall the algebraic Riccati equation condition Pr = 0, the equation for I = T - is
Given the terminal
1 -Q1 -
1 Q (
P
+
cl)
'Q--1
2
0
0
0
0
1
2
V
T-1
0
V T-11
0
0
0
Suppose it is true for ++ 1, such that,
0
0
0
0
0
0
1
P41 V41
2
0
VT
6+1
A++1
0
0
A441
Given that 0, 0.07 I, 0, V V., in which case, Q2 and 0, Pit10, contain the same basis V..
Recall the algebraic Riccati equation
1 Qu + + P
1
P 2
P
P4t10))T 20, P4t10 + +cl)= (Q,
2
+ 20,
+20 P P2t10,),
0
0
0
V
V
V
V
2
0
0
1 Az+1
(1 A )2 + Ae41 + +c)
0
V.
V..
2
0
0
c(1+A++1)
1+ce1ttc
Recall the optimal control solution equation 9 and Lemma G we reach the following analytic formulation.
Proposition 7. When the layer-wise transformations
represented as orthogonal matrices, and the basis of state embedding, state integration embedding, and state derivative embeddings are mutually orthogonal, the optimal feedback control, denoted as Te (x(x),, can be computed as follows:
Te (xx) =
-
0
V X
0 0
I+Ar41
0
0
1+
where the time-varying parameter X is governed by backward diffeerence equation x c(1+Ae+1) terminal condition specified as Ar 0.
1+Amito with the
25

---

Published in Transactions on Machine Learning Research (03/2024)
9 Appendix B
Recall the optimal control formulation in Proposition g
define control gain matrix K,
0
0
K, = -
V
0
Ic
Let 0g represent the 8th linear transformation, and æ Rd - Rd be the closed-loop controller. We denote the unperturbed state at time as Xt, and the controlled state with perturbation z applied at the initial condition as X
x = 0,(x, + (x))), =
The difference between the controlled system applied with perturbation at the initial condition and the uncontrolled system without perturbation is shown
Xe+1 Xe+1 0,(x: + Te(x))) x(),
(%e,t KiX,, xt),
0, K,)x, 0,x, + O,K,
-. K,)(X, Xt),
(12)
where 0,K,x, = since X₁ is in the null space of the control gain matrix K,.
Lemma 8. For t 2 O, have
I K- = - I+ (1 ar) Pts
where P. = Vi(V.) at 14Ac11t0
Proof. Recall equation 12
[I K,) can be expressed as
l 0
0
0 1
I- K- = V
VT,
I+ 1+
where the first r diagonal elements are 1, and the last (d - r) diagonal elements are I+A+41+e* By denoting the projection of first r columns as V and last (d - r) columns as V it can be further shown
I K, = Vi(V:) T + 1+ A4+1 (VI(
P az (PPr)).
= = + (1 - -00) Pt..
where 0; I+A
In the presented formulation, the input state space, denoted Z, is partitioned into direct sum comprising two orthogonal subspaces. This decomposition is expressed as Z = Z @ 2+, where Z represents the linear embedding subspace, encapsulating the input data. This is characterized by the condition Xo E Z for all pairs (x, y) sampled from the distribution D. Concurrently, - defines the orthogonal complement of zll. Extending this notion, the time-dependent state space at any given timestep is represented as Z, 2!! 7
26

---

Published in Transactions on Machine Learning Research (03/2024)
Lemma 9. For t 2 0, let Pž be defined as follows,
P Pt: P (((t)) 0 1- P,0---1 8,
Then
1. P. is a projection.
2. Pt is projection onlo 2 i.e. range(Pi) 2
3. If all O, are orthogomal, then P Vt, where Po is the orthogonal projection onto 2..
Proof-
1. We prove it by induction for each t. For 0, P Pt, which projection by its definition. Suppose it is true for such that P P.P. (P is projection if P P2), then for (s+1). 1),
(P,t1}== = (0 108-1 P,O P-O
0 0 0
(Pi) 2
0 0- p..0pPp..... P.O
p
We prove it by induction on for each t. For 0, P Pt. which the orthogonal projection
onto Z Suppose that it is true for such that P is projection onto z!! then for (s + 1),
p 0 1--0-PP:0---- which implies
range(P!tt) -rangee range(00To__1
= =1070-1x:x0 2 =z!-_ -
0, is orthogonal,
ps+1 -0,1 P:O,-s-1
=0,- P,000
= -P
pat1 is orthogonal projection onto range Z!-.-1- Therefore, PT is orthogonal projection onto z||, orthogonal projection onto the same range is unique, P.. Po. Vt.
The following Lemma uses the concept of oblique projection to show recursive relationship to project any (th state space Eq. (12 back to the input data space.
Lemma 10. Define for 0 ossst,
G = at It (1 Pr
Then, Eq.
12 can be written as
x4 x1 -
GX) xo). 12 1.
27

---

Published in Transactions on Machine Learning Research (03/2024)
Proof. We prove it by induction on t. For t 1, by the definition of G2 and transformation from Lemma
X1 = Ob(T Ko) - -x),; = 00(an I (1 - -00 (x) - xn),
Eq.
12
(x0 - xo).
Suppose that is true for (xt - xt), by Lemma g we have
X++1 Xx+1 0 (x) - xt);
=0((0 I - ar) (x) xt),
Lemma 8
(0-1000-1 ... 0o) (G ... G))(X) - xo).
(13)
Recall the definitions of P (s+1) 0, -0---1 P:0:-a-1 and G = at I (1-aa) P;
G 04 I+ (1 ar) p(++1)
at I (1 ar) 0 P:0:-8-1,
0 (a: I+ I+ a.) P;) 0-
0 - G;O,-.-1
which results the equality for the oblique projections. Furthermore,
0 0 (a+1) G;O,-...
Applying the above to Eq.
13 results
(2)x0 -x):
Ke+1 X
= (00. G.) (20 -xo):
(0000000000(00000000-00000000:10 (%)
(No -x0).
- xx).
Lemma 11. Let
F, = G (t-2)) 1-2
G 121.
Then,
F. Il a. I+ (1 a=0
Il a.) Po-
==0
Proof. We prove it by induction on Recall the definition of G := ar I+ (1 ar) P. When t= 1,
F = G ap II (1-ap) Po-
Suppose that it is true for t such that
F.
II ==0
a,
I + (1
a.) Po.
8 ==0
28

---

Published in Transactions on Machine Learning Research (03/2024)
for (t + 1),
F F+t = F F,F,,
(a I+ (1 a4) F.
-a; I+ (1 ar) P
a. I+ (1 II
ax) Po).
==0
==0
[-]
1-1
II 02 I+ a
II
as) Po (1 04)
0s P + (1 oz)
II a.) P.Po
Recall Lemma
if all 0 is orthogonal, then P = Po. and P.P. Po. Hence,
1-1
t-1 II a.) Po.
F6-1
0,
I+ a1
II
a.)
Po
+
(1 ())
II
a2 Po + (1 a1)
==0
==0
==0
s=1
t-1
t-]
0g3
1+
II
0
a. + 1 04
0g
+
Po.
8=0
8=0
8=0
==0
8=0
8=0
1+
Po
==0
s=0
Theorem 4. For any time step t 2 assuming that each O, is orthogonal matriz, we have the following error computation:
| X
II a ||2-
1
where a is time-varying parameter defined in relation to the control regularization and X are auriliary variables, as follows:
c(1 + Aet1) AT =0 AT-1 = Itc d
1+ A411 cc
1 Ae41
Proof. The input perturbation Ng - Xy can be decomposed as z zll + +z+, where z E 2! and at Z and zll and x- are vectors such that
zll z- 0 almost surely.
z have uncorrelated components.
z Zll, and x-z z-
Since the layer transformations O, are orthogonal matrices for all 4, recall the dynamical system Eq. and Lemma 10
12
| - xx = |[0([I K,)0 K,-1) On Ko)
= 11(0,-10-2---- 00) (G 2))
-I(G ....1...G... G))
(14)
29

---

Published in Transactions on Machine Learning Research (03/2024)
For the term [|
Gz)z1 recall Lemma 11
[[G
2
t-1
0g 1+ (1
II ag Po
2
x=0
[-1
t-1
0g Z + (1
II (1.) z 112
==0
==0
o,) ) | + (1 II 02)) ) || 11 + I
(1
(z) T z!!
==0
==0
8=0
==0
a2)²² (| + 11=
II ag )
||z||
2(
(1- II az) + )z)²z
5=0
t-1
(-1
[-1
[-1
||z
12
II
0 +1
0.)
)
o1)
ass)) -||2| 12
1-1
|
1/2
1
Recall the error computation in Eq.
14
[-]
X
X, 112
II
0
||
11
||2
10
Appendix C
The tables referenced as S and provide comprehensive numerical data for the SNLI and MNLI datasets, respectively. The results in Table reveal that the PD control mechanism significantly improves the ro- bustness of both the standard and robustly trained baseline models against every type of adversarial attack. Specifically, the application of PD control to the standardly trained Distilbert model boosts its accuracy by 15% and 16% in facing TextBugger and TextFoller attacks, respectively, while incurring a minimal 1% ac- curacy reduction on the original, unaltered dataset. Moreover, the robustly trained Distilbert model, which includes adversarial training (AT), benefits from the addition of PD control by showing a 12% accuracy in- crease when confronted with both TextBugger and TextFoller attacks. The comparative analysis of bascline models and the proposed control framework for the MNLI dataset, as detailed in Table 0 demonstrates that implementing the PD controller enhances the standard trained Distilbert's resistance to perturbations by an average of nearly 10%. However, this improvement somewhat diminished in models trained for robustness, with an average enhancement of 5% against all types of perturbations.
11 Appendix D
A optimal control framework for robust deep neural networks. We start with a description of the dynamical system approach to machine learning. In the dynamical system framework, we consider the input E X the initial condition of system of difference equations,
x = Fx(x; + ne (x:), 0t).
where F, represents a time-varying difference equation, 0, are model parameters of Ft, ne Rd Rd is a feedback controller that maps the current state x, to control action. The goal of optimal control is to
30

---

Published in Transactions on Machine Learning Research (03/2024)
Table Measurement on SNLI dataset: baseline model
/onntrllled model
Standard models
None
A2T
PSO
TextBugger
Fooler
Distilbert
87.24 / 86.05
53.89
/ 62.31
49.84 / 54.96
24.73
/ 40.26
24.69 / 41.73
RoBEITTaBase
90.87 / 90.64
58.36
/ 64.11
51.44
/ 54.40
35.90 / 43.20
27.03 / 37.35
BERT-large
90.36 / 89.75
74.18
75.54
66.84 / 67.55
64.13
/ 64.41
56.37
/ 58.27
RoBERTaLarge
92.39 / 92.05
59.40
64.95
52.15
/ 56.70 70
33.72 / 42.43
26.43 / 37.29
Robust models (trained with AT)
None
A2T
PSO
TegtBu
Foote
Distilbert
86.74/ / 85.81
71.78 / 71.81
52.85 / 57.87
29.63 / 41.64
31.59 / 43.81
RoBERTaBasc
90.65 / 89.87
76.28
/ 77.08
53.85
/ 56.45
35.43 / 43.35
29.64 / 39.39
BERT-large
90.29 / 90.33
86.02 / 85.76
69.23 / 70.38
69.17 / 69.55
63.78
/ 65.27
RoBERTaLarge
92.10 / 91.62
81.11 / 81.62
55.28 / 59.71
34.15 / 44.74
28.74 / 42.44
Robust models (trained with FreeLB)
None
A2T
PSO
TextBugger
Text
Distilbert
85.68 / 84.50
57.75 / 62.95
52.53 / 56.86
26.68 / 37.80
25.47 / 39.64
RoBERTaBase
91.31 / 90.67
64.23
/ 68.85
52.22
/ 55.24
34.08 / 42.75
24.80 / 36.81
BaRT-large
90.81 / 90.72
77.64
/ 78.21
64.72
/ 65.56
58.21
/ 59.29
53.31
/ 56.26
RoBERTaLarge
92.37
92.26
67.53
/ 71.30
53.37
57.20
34.64 / 44.42
27.55 / 38.59
Table 9: Measurement on MNLI dataset: baseline model / controlled model
Standard models
None
A2T
PSO
Text
Text
Distilbert
79.39 / 76.98
59.43 / 64.61
51.81 / 59.49
36.02 / 47.34
38.78 / 50.62
RoBERTaBase
86.66 / 85.84
59.60 / 63.39
49.77
/ 53.05
34.76 / 40.68
31.43
/ 40.22
BERT-large
84.92 / 84.79 79
77.38 / 77.96
69.71 / 70.41
65.11
/ 65.54
64.80
/ 65.91
RoBERTaLarge
89.71 / 89.40
62.85 / 67.93
51.19 / 56.77
37.18 / 45.11
32.81 / 43.27
Robust models (trained with AT)
None
A2T
PSO
Text
TextFooler
Distilbert
79.70 / 76.50
66.52 / 67.71
57.34 / 62.90
40.22 / 50.00
45.62 / 54.93
RoBERTaBase
86.55 / 85.54
64.52
/ 66.70
53.41 / 56.78
35.61
/ 40.88
34.27 / 43.41
BERT- large
84.90 / 85.02
81.37
81.69
73.05 / 74.05
68.27
68.91
71.69 / 72.80
RoBERTaLarge
90.10 / 89.51
76.94 / 78.36
59.34 / 64.44
41.21
48.34
40.69
49.33
Robust models (trained with FreeLB)
None
A2T
PSO
TextBuuger
ToxtFooler
Distilbert
78.76 / 75.33
64.10 / 66.25
58.03 / 62.94
38.87 / 49.50
43.58 / 52.88
RoBERTaBase
86.10 / 85.59
61.60 / 65.31
51.69
/ 54.93
36.06 / 42.42
33.27
/ 42.21
BERT-large
85.32 / 85.62
79.34
/ 79.64
72.25
/ 72.68
65.90 / 66.58
67.44
/ 68.49
RoBERTaLarge
90.18 / 89.81
67.28
71.61
53.27
58.04
36.40 / 44.91
32.83
43.84
design these leedback controllers (m.)Tt) such that some objectives are satisfied. as the following objective function,
This can be represented
T-1
min (xx. [x y, 7)] ] := min E B( ((xT.v) +
Σ C((xa) ) Tes se)
1=0
s.t. X F((x; + Te(xx), Ot).
31

---

Published in Transactions on Machine Learning Research (03/2024)
where ( and (() represent terminal and rumning losses, respectively. In general, objectives in the real world can be structured as terminal and running loss functions. Therefore, optimizing such an objective function aligns with achieving real-world goal. Take the development of autonomous vehicles as an example. This involves guiding the vehicle to a destination along a specific route. This challenge can be approachod as an optimal control problem, where reaching the destination is expressed as a terminal loss function, and the deviation from the planned route is captured through a running loss function.
The essential task of supervised learning is to approximate some function
F: X + + where F= Fr-1 o... Fi Fo,
which maps inputs in X' E Rd (e.g. images, natural language sequences) to labels in y (categories, numerical predictions). The objective of developing robust deep neural networks can be formulated within optimal control framework. Here, the aim is to minimize the discrepancy between the model's predictions and the actual labels through a terminal loss function, ((xr,y), by implementing feedback controllers. However, this ideal scenario is challenged by the practical limitation that true labels are unavailable during the model's inference phase. As a consequence, our focus shifts towards developing robust model predictions through the development of running loss functions. In this work, we introduce running loss function that assesses the state of control at cach timestep t,
C((xo)-=o: Te, (. (t. = 111s (x: + n. (xx)))1 5 (x: + <x(x)) + x2
] 11? (x: + (xx:)) xx-1) + (x::)
This rumning loss function calculates loss by measuring the difference between the controlled state and certain embedding manifolds. These manifolds capture the structural, integrative, and derivative aspects of state embeddings. Ideally, the states from unperturbed input samples should align with these embedding manifolds. Thus, when perturbations are introduced to the input, this running loss function assesses the quality of the state. Minimizing this running loss helps in adjusting the states correct the effects of such perturbations. By defining both terminal and running loss functions, solving this optimal control problem is equivalent generating leedback controllers (m:)/=o such that the controlled state trajectory of perturbed input performs similarly to the unperturbed counterpart.
32


Reviews:
[{"review_id": "h0oBAqck0R", "reviewer": "Reviewer_sddg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The approach is computationally efficient and avoids expensive on-line optimization while also avoiding compromising on accuracy. Lots of theoretical analysis.\n\nHowever, while it proposes a promising approach for an artificial embedding perturbation problem, I don't understand how it's applicable for any real world scenario for enhancing NLP robustness. The proposed approach assumptions seem disconnected from any practical adversarial attacks and the it's overall hard to figure out what exactly is being measured in the reported experimental results.\nTo go into slightly more detail:\n- The method assumes perturbations and threats exist directly in the continuous embedding space of neural networks. However, most real-world adversarial attacks on LLMs would be expected to manipulate discrete tokens or characters in the input text? This calls into question the practical utility of making models robust to embedding perturbations. If the adversary has access to manipulate the network activations, there might be bigger security issues with the deployment. The threat model seems quite artificial.\n- It's unclear that any intentionally crafted input level perturbations would truly manifest as embedding noise that could be corrected via PID control theory.\n- Embeddings themselves don't have any ground truth labels, so it's unclear what \"undesirable\" perturbation even means in this context.\n\nOther issues are the obviously big simplifying assumption about linear orthogonal transformations that enables the performance claims. I don't understand when they would practically hold.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a PID control framework to improve the robustness of LLMs against input _embeddings_ perturbations. The key ideas are noticing the parallels between neural network depth and discrete dynamical systems, allowing formulation of robustness as a trajopt problem.\nThis allows designing PID controller for each layer to monitor errors between desired & actual internal states. Linear embedding manifolds for the PID states are constructed by leveraging the geometric properties of training data. The analytical solution is derived under a big assumption of linear orthogonality of transformations.\nThe approach is evaluated on NLI tasks using adversarial attacks to demonstrate improved robustness.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Given the paper's motivations, would be good to clarify how the paper's assumptions connect to real world threat models.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jAuwUomffD", "reviewer": "Reviewer_TdCP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n- Interesting idea, which extends the use of PID control introduced by Chen et al (2020) and Chen et al (2022)\n\n- The method appears to address the important questions of robustness to the community\n\n- I checked the derivations and they look correct to me. \n\nCons:\n\nClarity and experimental evidence needs to be improved (see below), due to the experimental evidence issues I answer 'no' on the claims and evidence question below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose correction mechanism using PID (Proportional-Integral-Derivative) control for robustness of pre-trained large language models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Comments:\n\n- The following few sentences are not entirely clear: “The methodology of constructing a self-healing process to improve the robustness of deep neural networks was initially introduced in Chen et al. (2020) and its subsequent work Chen et al. (2022). It leveraged a closed-loop control method to detect and correct potential errors applied to input data. This method belongs to a special case of P control in the proposed PID control framework, in which only the errors from the present states are corrected.” “It leveraged”… refers to Chen et al, I would understand? \n\n- Crucially, Table 1 and Table 2 should indicate why these results are statistically significant (in many cases, the difference is only a few percentage points, so seeing a confidence interval would be crucial to understanding whether these results are significantly different from the baseline). The claims such as “In models trained using standard methods, the implementation of P-D control shows an average performance improvement of 5% over the P control scheme. In robustly trained models, this improvement is nearly 3%.“ should be backed up by some confidence intervals. Furthermore, it seems from Table 2 that the results do not show the consistent winner method.\n\n- The ablation study excludes certain parts of the PID controller. It is, however, unclear whether P-baseline exactly corresponds to Chen et al (2020) or not. This is especially important given that at the end of page 2, the authors discuss computational inefficiency of  the optimal control solution by Chen et al (2020, 2022) which implies that they are not just the P-ablation of the current method. Besides, it would be great to see if one can compare it with both Chen et al (2020) and Chen et al (2022)\n\n- It is important to discuss the wider perspective of the proposed method on the trustworthy ML and how it complements the research on adversarial attacks. Can the same method be used for vision problems, for example? How does it complement the existing trustworthy ML literature? Perhaps, adding more information into the Related Work section could help. \n\n- It would be also good to see if it is possible to empirically calculate the error in Section 2.3 and demonstrate it through forward propagation?", "broader_impact_concerns": "No broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kO3tS8eaO2", "reviewer": "Reviewer_Tx49", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The work aims to solve a difficult and important problem.\n* The empirical results of the best-performing proposed approach are quite good.\n* The theoretical derivation of the proposed solution is elegant.\n* The paper is well-written and easy to read\n\n\nWeaknesses:\n* The title states that the work focuses on large language models, but the authors only test their methods on a few relatively small variants of BERT. These models were released at least 3 years ago and have at most a few hundred million parameters. In most cases, I would be okay with testing on smaller models, but here I'd like to argue that LLMs have undergone drastic changes in the last few years and there are hints that certain interesting behaviors only appear when we reach a few billion parameters. I realize that such experiments would be very expensive, so if these are infeasible, I would like to ask the authors to tone down the claims in the paper and clearly state that the latest large models are out of scope.\n    * Additionally, I would like the authors to discuss whether the method is applicable in an autoregressive, decoder-only setting since most of the recent LLMs are decoder-only. \n* The novelty is limited, as that approach operates in the framework introduced by [1, 2]. Since this is TMLR, novelty is not exactly required, but I think the differences should be more clearly highlighted in the text. Ideally, I would like to see an empirical comparison -- I understand that the P-only variant is the closest to the previous models, but the impact of linearization is not well understood. For example, I would appreciate a comparison of the PMP-based solution with the linearization-based solution on a simpler dataset, where PMP is actually applicable. \n* In the end, PID seems to work worse than just PD. This seems to be inconsistent with the premise of the paper since most experiments are run on the whole PID. Can you provide a discussion on this point?\n* I would appreciate a discussion of the computation overhead of the proposed method, e.g., how much slower in practice is BERT with PID compared to regular BERT?\n\n\nDisclaimer: I haven't carefully checked the proofs in the Appendix.\n\n[1] Chen, Zhuotong, Qianxiao Li, and Zheng Zhang. \"Towards robust neural networks via close-loop control.\" arXiv preprint arXiv:2102.01862 (2021). \\\n[2] Chen, Zhuotong, Qianxiao Li, and Zheng Zhang. \"Self-healing robust neural networks via closed-loop control.\" The Journal of Machine Learning Research 23.1 (2022): 14329-14382.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider the problem of improving the robustness of large language models. They do this by introducing three controllers (P, I, and D) that fix the trajectory of activations of the network during inference so that it is closer to the manifold of activations observed during the training. Due to the computational complexity in this domain, they simplify the problem by assuming that the embedding manifold and layer-wise transformations are orthogonal. They show that in this case, one can efficiently find a solution. The authors then show that the method works well empirically using several BERT variants.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The requested changes are closely related to the points mentioned in the weaknesses section:\n* Please either run experiments on larger models or clearly state that the paper does not consider the current large (1B+ parameters) language models.\n* Please discuss the differences wrt. to Chen et al's work. A discussion in the text would be good, but an empirical comparison would be ideal, in particular, to understand the trade-offs of linearization.\n* Please discuss why in your opinion PD works better and why PID is used for most of the experiments instead.\n* Please discuss the computation overhead of the method, preferably empirically (wall time).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While reviewers raised some concerns in their initial reviews, some of which I shared based on a first read of the paper, these concerns were addressed with the updated manuscript (added experiments, expanded writing, clarifications) and authors' responses. All reviewers have recommended a 'leaning accept' and I agree that all major open issues were addressed by the authors and the paper is ready for publication in TMLR.

Major issues raised by the reviewers were:
* Strong theoretical assumptions for the closed-form solution, which will not hold in practice. Authors discussed this and presented an empirical analysis of how much impact the violation of these assumptions has in practice.
* Nature of robustness and undesired embeddings and their relation to practical adversarial attacks not entirely clear. Discussed by authors in their responses.
* Questions around the difference to closely related previous work. Clarified by authors in their responses.
* Relevance to trustworthy ML and removing bias seem overstated. Addressed hypothetically by authors, but the usefulness of the method to remove model bias in practice is not shown in the paper (and I personally would consider that beyond the scope of this work).
* Concerns about size of "large" language models (orig. publication had models with hundreds of millions of parameters) and the models being somewhat outdated. Authors added larger models (~1B parameters) during rebuttal. The reviewer that raised this concerns thinks that the concern still somewhat stands and suggests removing the term "large language models" from the title. I personally agree somewhat, but have no strong reason to believe that the method would behave qualitatively very differently at larger model scale. The term "large" will likely not stand the test of time either way, and as long as the paper (early on) clarifies the scale of models used, I personally think this is fine.
Decision: Accept (poster)