Stochastic Bridges as Effective Regularizers for Parameter-Efficient Tuning
Year: 2023

Abstract:
Parameter-efficient tuning methods (PETs) have achieved promising results in tuning large pre-trained language models (PLMs). By formalizing frozen PLMs and additional tunable parameters as systems and controls respectively, PETs can be theoretically grounded to optimal control and further viewed as optimizing the terminal cost and running cost in the optimal control literature. Despite the elegance of this theoretical grounding, in practice, existing PETs often ignore the running cost and only optimize the terminal cost, i.e., focus on optimizing the loss function of the output state, regardless of the running cost that depends on the intermediate states. Since it is nontrivial to directly model the intermediate states and design a running cost function, we propose to use latent stochastic bridges to regularize the intermediate states and use the regularization as the running cost of PETs. As the first work to propose regularized PETs that use stochastic bridges as the regularizers (running costs) for the intermediate states, we show the effectiveness and generality of this regularization across different tasks, PLMs and PETs. In view of the great potential and capacity, we believe more sophisticated regularizers can be designed for PETs and better performance can be achieved in the future. The code is released at https://github.com/thunlp/ stochastic-bridge-pet/tree/main.

heading: 1 Introduction
text: Recent years have witnessed the dramatic growth of pre-trained language models (PLMs) in various fields (Devlin et al., 2019; Dosovitskiy et al., 2021). As the size of PLMs continues to increase, the number of parameters has now even reached hundreds of billions (Brown et al., 2020; Smith et al., 2022), making fine-tuning the whole PLM both computationally impractical and environmentally unfriendly. In view of this, a variety of
∗Corresponding author.
Parameter-Efficient Tuning methods (PETs) are proposed (Houlsby et al., 2019; Hu et al., 2022; Zaken et al., 2022; Lester et al., 2021). By only tuning a small number of additional parameters, PETs can be comparable to full-parameter fine-tuning.
Despite the success of PETs, their underlying mechanism remains an open problem. Recently, several works have proposed to interpret PETs with optimal control theory. Yang and Liu (2022) first show that the optimization in Prefix Tuning (Li and Liang, 2021) (a typical method of PETs) can be considered as the search for optimal control variables in the context of optimal control, i.e., the trainable prefixes can be seen as the control variables that drive the PLM (the system) to the desired output. Ding et al. (2022) further show that the optimal control perspective can be applied to almost all PETs. The optimization of PETs’ parameters can be seen as minimizing the two cost functions in the optimal control literature: (1) terminal cost LT , which measures the quality of the terminal state, and (2) running cost LR, which measures the feasibility of the controlled intermediate states and the control variables. Although LT can well correspond to the loss function of the model output, LR is only vaguely described as the regularizers on the parameters of PETs (control variables) in Yang and Liu (2022) and Ding et al. (2022), ignoring the dependency of LR on the intermediate states.
In this work, we show that designing a running cost to regularize intermediate states not only makes the optimal control perspective of PETs more theoretically sound but also empirically leads to better PETs. We begin by assuming that in PLMs, the intermediate hidden states for generating different tokens in a sentence have different dynamics (or trajectories), and the dynamics can be approximated with stochastic processes in a latent space. Specifically, we first freeze the PLM and learn a mapping from the original hidden state space of the PLM to a latent space. In the latent
10400
space, the dynamics of the intermediate hidden states for generating different target tokens can be approximated with different target-specific diffusion bridges. The obtained mapping can then be plugged to the model to regularize the intermediate hidden states when training PETs. Besides, since a diffusion bridge is (1) a Markov process and (2) a solution to a stochastic differential equation (SDE), we correspondingly propose two methods to learn the mapping: (1) fitting the Markov transition probability density function (PDF) and (2) fitting the SDE directly. The two methods act as a tradeoff between efficiency and effectiveness: the first method incurs only negligible computational cost and has satisfactory results, while the second one is slower but yields better regularizers.
We conduct experiments on different PLMs of different sizes, and the experimental results on GLUE (Wang et al., 2019) under both full-set and few-shot settings demonstrate the effectiveness of our proposal across four different PETs. Further analyses show that the learned regularizer helps pull apart the hidden states of different label words. We also observe that when we project the intermediate hidden states of PETs without our regularizer into our latent space, the better the PETs perform, the closer the latent states are to our latent bridges. This spontaneous approaching behavior may indicate that stochastic-bridge-like latent dynamics naturally exist in well-trained PETs.
In summary, our work has the following contributions: (1) Guided by the perspective of optimal control for PETs, we design latent stochastic bridge regularizers on the intermediate states during the training of PETs. (2) We propose two methods to construct the latent space according to the two representations of stochastic bridges, offering a trade-off between efficiency and effectiveness. (3) Our regularizers are shown to be effective and general across different PLMs, different PETs, and different tasks. (4) We show that well-trained PETs without any regularization spontaneously exhibit stochastic-bridge-like latent dynamics.

heading: 2 Background

heading: 2.1 Definition and Mathematical Notations
text: Consider using a L-layer PLM with the vocabulary V to handle a text-to-text task D. For each sample (x, y) ∈ D, y ∈ V is the output token and x ∈ VN
is the input token sequence 1, where N is the length of x. With x as the input, each layer of the PLM will output a sequence of hidden states, and we denote the hidden states of the i-th PLM layer as h(i) = {h(i)j }Nj=1 ∈ Rd×N . We denote the position where the model outputs the target y as o, i.e., the model should predict y with the hidden states h(L)o .

heading: 2.2 Optimal Control Perspective of PETs
text: Conventionally, adapting the PLM to D requires full-parameter fine-tuning, which is given as:
min ∆θ
Ex,y∼D [ L ( h(L)o , y ) +R ( ∆θ )] ,
h(i) =
{ h(i−1) + G(i)θ+∆θ ( h(i−1) ) , i = 1, . . . , L,
Embed(x), i = 0, (1)
where θ is the parameters, ∆θ is the full-parameter update, L is the loss function, R is the regularization function, G(i)θ+∆θ is the i-th layer forward propagation with updated parameters, Embed transforms the input tokens into embeddings.
As |θ| continues to increase, full-parameter finetuning becomes impractical, and various PETs are proposed to mitigate this problem. Let ϕ = {ϕ(i)}Li=0 be PETs’ parameters. Ding et al. (2022) give a unified view of PETs from the perspective of optimal control, and Eq. 1 can be re-written as
min ϕ
Ex,y∼D [ LT ( h(L)o , y ) + L∑
i=0
LR ( ϕ(i) )] ,
h(i) =
{ h(i−1) + G̃(i)θ ( h(i−1), ϕ(i) ) , i = 1, . . . , L,[
ϕ(0); Embed(x) ] , i = 0,
(2)
where G̃(i)θ represents the i-th layer forward propagation intervened by PETs, [·; ·] is the concatenation operation,LT is the terminal cost andLR is the running cost. Since |ϕ| ≪ |θ|, PETs can greatly reduce the tuning cost (more details in Appx. A). Typically, LT corresponds to the prediction loss, and LR can be seen as the regularizer on PETs’ parameters ϕ. However, in the optimal control literature, LR depends on not only the control variables ϕ, but also the controlled intermediate states {h(i)o }Li=1. In this paper, we show that additionally introducing dependence on {h(i)o }Li=1 for LR makes the optimal control perspective of PETs more theoretically sound, and empirically leads to better PETs.
1Here we assume y ∈ V since a sample where y ∈ VM can be decomposed to M samples, The i-th sample is ([x;y<i], yi) for auto-regressive language modeling or ([x;y−i], yi) for auto-encoding language modeling.

heading: 2.3 Diffusion Bridges
text: A diffusion process X = (Xt)t∈[0,T ] is a continuous-time Markov process. For any ta < tb, the diffusion process is equipped with a transition Probability Density Function (PDF) p(tb, b|ta, a), which gives the probability density of reaching b at time tb given the history of reaching a at time ta. A diffusion process is also the solution to an Itô SDE dX̃t = µ(t, X̃t)dt+ σ(t, X̃t)dBt, where Bt is a standard Brownian motion, µ(·, ·) is called drift function and σ(·, ·) is called diffusion function.
A diffusion bridge XT ;α,β is a diffusion process conditioning on the path observations of the two endpoints (0, α) and (T, β), i.e., XT ;α,β0 = α and XT ;α,βT = β. For simplicity, we assume α=0 in this work, and omit the superscript α. We consider two typical diffusion bridges, the Brownian bridge and the Ornstein-Uhlenbeck bridge (OU bridge). We present here the properties of the Brownian bridge and leave the properties of OU bridge to Appx. B.
Proposition 2.1 (Properties of Brownian Bridge). A Brownian bridge XT ;β with XT ;β0 = 0 and XT ;βT = β is the solution to the following SDE:
dX̃t = (β − X̃t)/(T − t) dt+ dBt, X̃0 = 0. (3)
The transition PDF from XT ;β0 = 0 to X T ;β tb
= b is given as
pT ;β(tb, b|0, 0) = 1√ 2πtb(T−tb)
exp [ − (b−(tb/T )β)2
2tb(T−tb)
] . (4)
Diffusion bridges and SDEs are battle-tested tools to model the stochastic dynamics of complex systems in engineering (Sobczyk, 2013), finance (Wang and Sloan, 2011), etc. Considering
the dynamics of PLMs’ hidden states are necessarily complex, diffusion bridges and SDEs serve as ideal tools for us to model the dynamics.

heading: 3 Latent Stochastic Bridges Regularizer

heading: 3.1 The Overall Framework
text: Building latent dynamics in the latent space. Since directly regularizing the intermediate states and constructing the running cost are non-trivial, we introduce a projection from the intermediate state space to a latent space, and leverage diffusion bridges as regularizers to construct the running cost. Specifically, we define a r-dimensional latent space U ⊆ Rr(r < d) and a learnable mapping gγ : Rd × Rd → U, where γ denotes the parameters. gγ projects the hidden state h (i) o and its context state h̄(i) into the latent space U at each layer of the PLM. Since h(i)o is contextualized while latent bridges are not, introducing the dependency on h̄(i) can inform gγ about the context at the i-th layer and allow gγ to decontextualize the hidden states. We simply take the averaged states at the i-th layer h̄(i) = 1N ∑N j=1 h (i) j as the context. We define the latent states with discrete time as
uD(gγ , {h(i)o }Li=0) = {ti+1, gγ(h(i)o , h̄(i))}Li=0, ti+1 = (i+ 1)/(L+ 2), (5)
where ti+1 is the normalized layer index. We include the 0-th layer (input layer) because some PETs (e.g., prompt tuning) act on the 0-th layer. We use t0 = 0, tL+2 = 1 represent the two endpoints. By using natural cubic spline knotted at {h(i)o }Li=0 to interpolate over L = [−1, L + 1], we further give a continuous representation of the
states in the latent space U as
uC(gγ , {h(x)o }x∈L) = {tx+1, gγ(h(x)o , h̄(x))}x∈L, tx+1 = (x+ 1)/(L+ 2) ∈ [0, 1]. (6)
Learning the mapping from hidden state space to latent space. Since adapting PLMs to downstream tasks can be seen as transferring the knowledge obtained from pre-training tasks to downstream tasks, we argue that the latent dynamics of intermediate hidden states for generating the same token y should be similar in both the pre-training and downstream tasks. Therefore, we train the mapping gγ on the corpus that is used to pre-train the backbone PLM2, and then apply the learned mapping to downstream tasks to encourage the latent dynamics to be similar to that in pre-training.
Specifically, we assume that the states to generate the token y in the latent space U form a trajectory that is a path sampled from X1;βy with high probability, where X1;βy is the diffusion bridge describing the latent dynamics to generate y, and βy is the tail endpoint of the diffusion bridge. More details of X1;βy will be discussed in Sec. 3.2.
On the corpus where the PLM is pre-trained, we fix the PLM and use its hidden states {h(i)o }Li=1 to learn gγ by maximizing the goodness of approximation for latent states u under the bridge X1;βy :
γ ← argmax γ′
[ goodness ( u(gγ′ , {h(·)o }), X1;βy )] , (7)
where u can be uD (Eq. 5) or uC (Eq. 6) depending on the fitting method, goodness(·, ·) is also a function depends on the choice of the fitting method, measuring how likely u is a sample trajectory of X1;βy . In Sec. 3.3, we will define this function alongside the fitting methods. Regularizing PETs with latent dynamics. After learning gγ with Eq. 7, we freeze γ and use the goodness function as the running cost in Eq. 2 for PETs on downstream tasks. The objective becomes
L = LT (h(L)o , y) + α · goodness ( u(gγ , {h(·)o }), X1;βy ) ,
(8)
where the second term is the running cost and α is a hyper-parameter controlling the regularization intensity. By optimizing Eq. 8, PETs learn to predict y correctly and keep the latent states at the position o conform to the diffusion bridge X1;βy . Note that introducing gγ as the regularizer does not increase the number of trainable parameters for PETs during the training stage since γ is fixed, and since we
2A small portion (0.1%) of the pre-training corpus is sufficient, see Appx. F
only use the pre-training corpus, no extra information of downstream tasks is leaked. Moreover, the regularizer only helps in training better PETs and does not intervene the inference.

heading: 3.2 Determining Endpoints for Bridges
text: An intuitive approach to determine the endpoints for the diffusion bridges for each target token is to optimize the endpoints together with the mapping gγ . However, optimizing endpoints and gγ jointly may admit a trivial solution: endpoints are both 0 ∈ Rr and gγ always outputs 0. Since 0 is always a point in the sample path of such a degenerated diffusion bridge, the value of goodness function can be meaninglessly high. Although sophisticated constraints can be imposed here, as the first work that uses diffusion bridges as regularizers, we simply pre-determine the endpoints and keep them fixed. We leave introducing constraints as future work.
Specifically, we apply principal component analysis (PCA) to the output token embedding matrix V ∈ R|V|×d of the PLM, obtaining a rdimensional embedding matrix, and re-normalize each row to have a norm η. Let the resulting embedding matrix be β ∈ R|V|×r. We then use 0 ∈ Rr as the heads for all the bridges, and β as the tails of the diffusion bridges, i.e., the r-dimensional embedding of y in β is used as βy in X1;βy . The intuition for using β as the tails is that the trajectories of the intermediate states for similar target tokens should be close. In V , similar tokens are close, and β obtained by PCA can well preserve the token similarity after reducing dimensions.

heading: 3.3 Fitting the Mapping gγ
text: We use the Brownian bridge to illustrate the fitting of gγ . It can be analogous to OU bridge easily. Method 1: Approximating the Transition PDF. Generalizing Eq. 4 to high dimension, we can derive the transition PDF from (0,0) to (ti+1, gγ(h (i) o , h̄(i))) for X1;βy :
p1;βy (ti+1, gγ(h (i) o , h̄ (i)) | 0,0)
∝ exp(∥gγ(h (i) o , h̄ (i))− ti+1βy∥2 2ti+1(1− ti+1) ), (i = 0, . . . , L),
where ti has the same definition as that in uD (Eq. 5). To make gγ approximate the transition PDF, we maximize the sum of log-probability of uD under the Brownian bridge X1;βy :
goodness = L∑
i=0
log [ p1;βy (ti+1, gγ(h (i) o , h̄ (i)) | 0,0) ] + C,
(9)
where C is a constant. Here, gγ can be seen as a mapping from the hidden state space to the latent space by predicting the expectation of the Brownian bridge X1;βy at {ti+1}Li=0. Method 2: Approximating the SDE. Since the Brownian bridge is the solution to the SDE in Eq. 3, we let gγ approximate the SDE. Solving the SDE requires continuous latent states, while we only have L+ 1 discrete observations, we thus use the continuous representation uC introduced in Eq. 6. Generalizing Eq. 3 to high dimension, the SDE approximated by gγ can be defined as:
dZt = gγ(h (x) o , h̄ (x), t)dt+ dBt, x = (L+ 2)t− 1, (10) where x is the same as that in Eq. 6, B : [0, 1] → Rr is a standard r-dimensional Brownian motion. Here, we additionally introduce the dependence on t for gγ , since time information is shown to be important in previous neural differential equation works (Zhang et al., 2020; Dupont et al., 2019). Following Li et al. (2020), when two SDEs share the same diffusion function, the KL divergence between the probability measures induced by the two SDEs is finite. Since the diffusion function σ ≡ I for Eq. 10 and the multi-dimensional generalization of Eq. 3, the KL divergence between the probability measures µY of Eq. 10 and µX of generalized Eq. 3 can be estimated by:
DKL(µX ||µY ) = EZ [ ∫ T
0
1 2 ∥u(t, γ)∥22
] ,
u(t, γ) = σ−1 ( gγ(h (x) o , h̄ (x), t)− µ(t, Zt) )
= gγ(h (x) o , h̄ (x), t)− (βy − Zt)/(1− t),
where µ(·, ·) is the drift function of the predetermined Brownian bridge X1;βy . We use the KL divergence as the goodness function to optimize the mapping gγ . Here, gγ can be seen as a mapping from the hidden state space to the latent state space by approximating the drift vector field of the underlying Brownian bridge X1;βy .

heading: 4 Experiments
text: To verify the effectiveness and generality of the regularizers built on stochastic bridges, we conduct experiments on (1) different PLMs: BERTlarge (340M) (Devlin et al., 2019) and Debertaxlarge (750M) (He et al., 2021); (2) different PETs: Prompt tuning, LoRA, BitFit and Adapter; (3) different diffusion bridges: Brownian bridge and OU bridge. We show that the regularizers effectively improve the performance on GLUE (Wang et al., 2019) under both full-set and few-shot settings.

heading: 4.1 Experimental Setups
text: Datasets. Since both BERTlarge and Debertaxlarge use Wikipedia and BookCorpus (Zhu et al., 2015) for pre-training, we thus use these two corpora to train gγ . We report F1 for MRPC and QQP; Matthews correlation for CoLA; and accuracy for other tasks. We report the average performance and the standard deviation on the development set over 3 different runs. We append [MASK] to each sequence, and require the PLM to output the label word at [MASK] (e.g., negative or positive for SST2). We exclude STS-B for it is a regression task. Models and PETs. We use the checkpoint released by Shoeybi et al. (2019) for BERTlarge, and the official v1 checkpoint for Debertaxlarge. We use a simple three-layer MLP to build gγ . For Prompt tuning, we use a soft prompt of length 20, and append it to the end of each sequence. For LoRA, we apply it to the query and value of attention modules. For Adapter, we apply it to the output of attention and feed-forward modules. For BitFit, we tune all the bias terms in linear layers and layer normalization modules. Hereafter, we use PDF regularizer to refer to using gγ fitted by approximating the transition PDF, and SDE regularizer to refer to using gγ fitted by approximating the SDE, vanilla x to refer to the PET x without using regularizers. Few-shot Experiments. We randomly sample 2× k examples from the original training set Dtrain for each class. The sampling is performed 5 times with different seeds to form 5 training sets and development sets {D̃(i)train, D̃ (i) dev}5i=1 with each being k-shot. Each time we train PETs on D̃(i)train, we select the best model on D̃(i)dev, and report its performance on the original development set Ddev. Hyper-parameters. Hyper-parameters are listed in Appx. E. We focus on the difference in performance between vanilla PETs and regularized PETs. Therefore, we set the hyper-parameters to common values from previous works and do not perform much hyper-parameter search. But we ensure the hyper-parameters for vanilla PETs and regularized PETs are the same for a fair comparison.

heading: 4.2 Full-set Results
text: The experimental results for BERTlarge and Debertaxlarge are reported in Tab. 1 and Appx. C respectively. Due to space limitation, see Tab. 6 for the complete results including OU bridge regularizers. The first line of each block in the table is the performance of vanilla PETs, and the rest of the
lines are the performances of the regularized PETs. In general, both Brownian and OU bridges, and both PDF and SDE regularizers are able to improve the performance of PETs, showing the effectiveness of our proposed regularizers. Particularly, for Prompt tuning, the SDE regularizer with both diffusion bridges yield an average performance improvement of more than 2%. We assume that it is because Prompt tuning has far less trainable parameters than other PETs, and it only acts at the input layer, which is far from the supervision signals of the terminal cost LT . Therefore, when provided with the regularization on the hidden states, the prompts receive more guidance and eventually reaching a better local optimal.
Overall, the two diffusion bridges in our experiments do not show much difference. As for the two fitting methods, SDE regularizer is generally more effective, especially for Prompt tuning where the number of trainable parameters is restricted. However, we also observe that SDE regularizer is about 3 times slower than PDF regularizer, which brings the trade-off between performance and efficiency. One can expect a better performance by leveraging more sophisticated underlying stochastic bridges, exploring more reasonable endpoints for bridges and designing better mapping gγ . As the first work using latent stochastic bridges as regularizers, we mainly consider the most simple cases and aim to show the potential of the approach.

heading: 4.3 Few-shot Results
text: In Tab. 1, the improvements are more substantial on small datasets MRPC, CoLA and RTE. This is probably because in large datasets, the abundant data has provided enough information to train high
quality PETs; while in small datasets, the data is insufficient and the regularizer can offer additional supervision. To validate this, we conduct the experiments under the few-shot setting on GLUE.
The 16-shot results are shown in Tab. 2, and the results for the OU bridge, results for 4-, 8- and 32-shot and results for Debertaxlarge are placed in Appx. C. For all PETs, the SDE regularizer yields an improvement of more than 3%. Particularly, the SDE regularizer on LoRA brings an improvement of 5.2%. Also, there is now a substantial boost on what was originally a richresource dataset, such as MNLI, QQP and QNLI. The PDF regularizer also gives modest improvements. Though slightly inferior to the SDE regularizer, it is still satisfying, considering that the PDF regularizer brings such a performance improvement with little computational cost introduced. We additionally observe that the improvement is more significant on Debertaxlarge in Tab. 8, demonstrating the potential of our regularizers on larger models.

heading: 5 Analyses
text: To better understand the role played by our regularizers, we analyze the hidden states of the PETs with and without regularizers. We choose Prompt tuning as a representative. By varying the hyper-parameter α in Eq. 8, we show that as the regularization intensity gets stronger, the clusters of hidden states corresponding to different labels become more distinguishable. Also, we show that the hidden states of vanilla PETs spontaneously approach the latent bridges in the latent space without knowing the bridges, indicating that there may exist intrinsically diffusion-bridge-like latent dynamics for PETs.

heading: 5.1 Distances between Labels are Widen
text: We use the different prompts obtained with or without regularizers on the full-set GLUE, and record the intermediate hidden states {h(i)[MASK]}Li=1. We vary the regularization intensity by adjusting the coefficient α in Eq. 8 to inspect the impact of the regularization intensity on the hidden states. Note that when α = 0, it degenerates to the vanilla PET.
We randomly sample 100 samples for each label in MNLI, use UMAP (McInnes et al., 2018) to reduce the dimension of the last layer’s hidden states of Prompt tuning and plot them in Fig. 2. It shows clearly that for both regularizers, as the regularization intensity gets stronger, the hidden states of the last layer become more distinguishable among labels. By looking at the axes of these plots, we find that the distances between the clusters generally increase when the regularization intensity is increased. We also notice that the SDE regularizer better helps separate the hidden states of the last layer by substantially enlarging the distance between the centroids of different labels, which could be one of the reasons why the SDE regularizer has better effectiveness in almost all experiments.
We also calculate the Pearson’s correlation between the α and the average distance between the centroids of different clusters. The results are shown in Tab. h. On all the datasets, the α has a positive correlation to the average centroid distance, and on most of the datasets, the correlations are significant (p-value <.05). This indicates that as the regularization intensity gets stronger, the centroids of different label clusters become more distant, which is a desired effect because the regularizer encourages the hidden states for different labels to conform to different latent bridges.

heading: 5.2 Hidden States Spontaneously Approach the Latent Bridges
text: An interesting phenomenon we observe is that the vanilla PETs’ intermediate hidden states spontaneously approach our latent bridges when they are projected by our mapping gγ . That is, applying our mapping gγ to the hidden states of vanilla PETs, we find that when the performance of vanilla PETs becomes better, the average distance from gγ({h(·)o , h̄(·)}) to our latent bridge gets closer. Here, similar to Wang et al. (2022), we define the distance from gγ(h (·) o , h̄(·)) to its corresponding latent bridge Xy using Eq. 9 without the constant. Note that the vanilla PETs have no access to gγ and the latent bridges during the training process, and gγ also has no access to the PETs during its fitting.
We show the above phenomenon by conducting analyses in few-shot scenarios with PDF regularizer, and reporting in Tab. 3 the correlation between (1) the number of shots and the average distance from latent hidden states to latent bridges (2) the performance and the average distance from latent hidden states to latent bridges. We report Kendall’s rank correlation for (1), and Pearson’s correlation for (2). See Appx. D for the detailed setup.
From Tab. 3, the number of shots has a negative correlation to the distance, and the correlation is significant on 4 out of 6 datasets. This indicates that as the amount of available data increases for vanilla PETs, its intermediate hidden states in latent space spontaneously approach latent bridges even without knowing the mapping gγ and the bridges. Additionally, the results in Tab. 3 show the negative correlation between the performance of vanilla PETs and the distance to the latent bridges, and it is significant on 3 out of 6 datasets.
Altogether, the two findings on correlation show that as the PETs’ performance improve, their intermediate hidden states projected by gγ spontaneously approach our bridges in the latent space. This implies that there exists intrinsically diffusionbridge-like latent dynamics for PETs, and also justifies our use of diffusion bridges as regularizers.

heading: 6 Related Works
text: Recent years have witnessed the success of PLMs (Raffel et al., 2020; Brown et al., 2020). However, as the sizes of PLMs continue to grow, it becomes increasingly impractical to perform finetuning on downstream tasks. Many efforts have been devoted to PETs, aiming to tune only a few parameters rather than the whole PLM on downstream tasks. For example, Prompt tuning (Lester et al., 2021) prepends tunable embeddings to the
input, Adapter (Houlsby et al., 2019) inserts small modules into each layer, BitFit (Zaken et al., 2022) tunes only the bias terms, and LoRA (Hu et al., 2022) decomposes the weight updates into lowrank matrices. In this paper, based on the theoretical grounding of PETs on optimal control (Yang and Liu, 2022; Ding et al., 2022), we propose stochastic bridges as the regularizer on intermediate hidden states and introduce regularized PETs.
Our work also closely relates to continuous-time neural differential equations (NDEs) (Chen et al., 2018; Rubanova et al., 2019; Li et al., 2019; Kidger et al., 2021). Continuous-time NDEs model the dynamics of the hidden states with ODEs or SDEs parameterized by neural networks. Inspired by these works, we use SDEs to represent the latent dynamics of PETs in the latent space. Our work differs from them in that we focus on using neural SDEs as regularizers for intermediate hidden states, rather than feature extractors. We also notice that Wang et al. (2022) explore the use of Brownian bridge in regularizing the model dynamics across time. Our work differs from theirs in that we regularize the dynamics of intermediate hidden states across model layers. We additionally show that other diffusion bridges can be easily applied as the regularizer. As far as we know, we are the first to show the diffusion-bridge-like dynamics for hidden states across PLM layers and use diffusion bridges as regularizers on intermediate hidden states.

heading: 7 Conclusion
text: Starting from the optimal control perspective of PETs, we notice that the existing PETs lack a running cost that regularizes the intermediate hidden states. We thus propose to use stochastic bridges in a latent space as the regularizers for PETs. Experimental results on different models, tasks and PETs show that the proposed regularizers effectively improve the PETs’ performance. Our analyses further show that the hidden states of the vanilla PETs spontaneously approach our diffusion bridges, indicating that there may exists intrinsically diffusionbridge-like dynamics in PETs. As the first work using stochastic bridges as regularizers, we show its effectiveness and generality. We believe it will be a promising direction.

heading: 8 Limitations
text: Introducing the regularizers inevitably incurs additional computational cost in the training of PETs. To show their impact on the training speed, we plot the time-performance curves for both PDF and SDE regularizers on full-set GLUE in Figures 5, 6, 7 and 8.
On different PETs, the regularized PETs with PDF regularizer has similar running time to the vanilla PETs. On the two large datasets, QQP and MNLI, regularized PETs with SDE regularizer take about 2 to 3 times longer to achieve the best performance than vanilla PETs. However, on medium-sized (QNLI, SST-2) and small datasets (CoLA, MRPC, RTE), the time to achieve the best results with SDE regularizer is comparable to vanilla PETs.
Overall, the PDF regularizer can effectively improve the performance of PETs without introducing much computational cost. In scenarios where there is relatively more focus on the inference performance of PETs and less concern about the slightly longer training time, or when the dataset is small, SDE regularizer should be a good choice.
Our method does not introduce additional risk to the original risks of PETs.

heading: Acknowledgement
text: This work is supported by the National Natural Science Foundation of China (No. 62236004, No. 62236011), Institute Guo Qiang at Tsinghua University.

heading: A Background for Parameter-Efficient Tuning Methods
text: The large number of parameters in the PLMs makes fine-tuning impractical, therefore different PETs are proposed to mitigate the problem. The current PETs can be categorized into three groups: addition-based, specification-based and reparameterization-based (Ding et al., 2022). To verify the generality of our method, we include one or two PETs from each category in this work, and we give a brief review to these PETs.
Prompt Tuning is an addition-based PET. It prepends or appends trainable virtual tokens P ∈ Rm×d to each sequence x ∈ Rn×d to form a new input sequence [P ;x] or [x;P ] ∈ R(n+m)×d, where n,m are length of original sequence and virtual tokens respectively, d is the embedding dimension. The virtual tokens P can be either continuous (Lester et al., 2021) or be restricted to be embeddings of discrete tokens in vocabulary (Gao et al., 2021).
Adapter (Houlsby et al., 2019) is an additionbased PET. It inserts two-layer MLPs after the attention module and feed-forward module at each layer. Denote h ∈ Rd as the input of Adapter, r
as the intermediate dimension of Adapter’s MLP, Wd ∈ Rr×d,Wu ∈ Rd×r as the down-projection and up-projection of Adapter, and σ as the activation function. Then the computation of Adapter can be formulated as
h←Wuσ(Wdx) + h
BitFit (Zaken et al., 2022) is a specificationbased PET. It specifies the bias terms in layer normalization modules and linear transformation modules as trainable.
LoRA (Hu et al., 2022) is a reparameterizationbased PET. It assumes that when training the model, the updates ∆W for model’s pre-trained parameters W ∈ Rd×k are low-rank, and thus reparameterize the ∆W of each matrix in attention module with a low-rank decomposition ∆W = BA, where B ∈ Rd×r,A ∈ Rr×k. For a forward pass h = Wx, the computation of LoRA can be written as
h = (W +∆W )x = Wx+BAx

heading: B Properties for Ornstein-Uhlenbeck Bridge
text: Proposition B.1 (Properties of Ornstein-Uhlenbeck Bridge). A Ornstein-Uhlenbeck XT ;β pinned at XT ;β0 = 0 and X T ;β T = β is the solution to the following SDE:
dX̃t = qµtdt+ σdBt,
µt = − coth [q(T − t)] X̃t + β sinh [q(T − t)] ,
X̃0 = 0,
(11)
where q is the diffusion coefficient and σ is the diffusion for the OU process. The transition proba-
bility density function reads as:
pT ;β(t, y | 0, 0) = 1√ 2πσ(s, t) exp
{ − (y − µ(t)) 2
2σ(s, t)
} ,
µ(t) = sinh(q(T − t)) sinh(q(T − s))β
σ(s, t) = σ2
q sinh(q(T − t)) sinh(q(t− s)) sinh(q(T − s)) .
(12)

heading: C Other Results for GLUE Experiments
text: In this section, we present the complete results including OU bridge regularizer for Tab. 1 and Tab. 2. We also report the results for Debertaxlarge, and the results on few-shot GLUE for both BERTlarge and Debertaxlarge under 4-, 8-, 16-, and 32-shot. We observe that BERTlarge cannot give reasonable answers on CoLA and the Matthews correlations are around 0 for all the PETs and all the shots we have experienced with. However, the situation gets better for the larger model Debertaxlarge. Therefore, we exclude CoLA for BERTlarge and keep it for Debertaxlarge. We only select the Brownian bridge
as the representative in this section, since the Brownian bridge and Ornstein-Uhlenbeck bridge have no significant difference in Tab. 1.
In Tab. 6 and Tab. 5, we report the performance of OU bridge regularizers. The experimental setups are the same as Tab. 1 and Tab. 2 respectively. The performances between OU bridge and Brownian bridge do not have a significant difference. In Tab. 7, we report the performance of Debertaxlarge on full GLUE datasets. On all four PETs, the SDE regularizer outperforms the PDF regularizer, this is consistent with the results we see in Tab. 1. The results for 4-, 8-, and 32-shot for BERTlarge and Debertaxlarge are plotted respectively in Fig. 3 and Fig. 4. For simplicity, we only plot the average performance for each PET. We report the results for 16-shot experiments for Debertaxlarge in Tab. 8. The setup for the experiment is almost the same as the experiment in Sec. 4.3, and the hyper-parameters are listed in Appx. E. The SDE regularizer outperforms the PDF regularizer on most of the PETs except Prompt tuning. We no-
tice that the SDE regularizer helps Debertaxlarge substantially on CoLA for most of the PETs, indicating the SDE regularizer can effectively provide useful guidance when the data is scarce and the task is hard.

heading: D Calculation of Correlation in Sec. 5
text: In this section, we elaborate on how we calculate the correlations reported in Tab. 3.
D.1 Correlation between Number of Shots and Distance to Bridge
Definition D.1 (Tie). A pair of observation {(xi, yi), (xj , yj)} is defined as tied if xi = xj or yi = yj .
Since we generate the few-shot datasets using 5 random seeds for each shot, each PET has 5 results for each shot. This results in observations with ties, e.g., the two observations for distances on the first seed and second seed for 8-shot {(8, d1), (8, d2)} are tied. To calculate the correlation for data with ties, the Tau-b of Kendall’s rank correlation is more suitable than Pearson’s correlation. We therefore
report the Kendall’s rank correlation for the correlation between the number of shots and the hidden states distance to the latent bridges.
D.2 Correlation between Performance and Distance to Bridge
We mix all the few-shot results for different shots and different seeds to form observations of performance and the hidden states distances to bridges, and then calculate the Pearson’s correlation.

heading: E Hyper-parameters
text: E.1 Training gγ
we use simple 3-layer MLP for gγ in all of our experiments. For PDF regularizer, the output dimensions of each layer are 1024, 256, 128, and for SDE regularizer, the output dimensions of each layer are 1024, 256, 32. We observe that the running time increases noticeably with the final output dimension for SDE regularizer, we thus choose a smaller one for SDE regularizer. The hyper-parameters for training gγ are listed in Tab. 4.
E.2 Training PETs on full-set GLUE
We run all the experiments for 50k steps, and evaluate on the development set every 1k steps. For BERTlarge, we use 32 as the batch size while for Debertaxlarge, we use 16 as the batch size. We choose learning rate 1e-3 for Prompt tuning for both PLMs, and 1e-4 for other PETs for both PLMs. We use 0.01 weight decay, 1.0 maximum gradient norm and no learning rate warm-up for all the experiments. We search the best regularization strength α in {0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0} for PDF regularizer, and in {0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.2, 0.5, 1.0} for SDE regularizer. The best α for PDF regularizer are listed in Tab. 9, and best α for SDE regularizer are listed in Tab. 10.
E.3 Training PETs on few-shot GLUE
We run all the experiments for 1k steps, and evaluate on the development set every 50 steps. For all the shots for both regularizers and both models, we use a batch size of 2. Other hyper-parameters are kept the same as in the experiments on full-set GLUE.

heading: F Performance of Regularizers Trained with Tiny Corpus
text: Although we use the pre-training corpus to train our mapping gγ , the training is actually fast and dataefficient. We show that when using only 10,000 documents in the pre-training corpus (about 0.1% of the corpus), the obtained regularizers still perform great and are comparable to the regularizers trained on the whole pre-training corpus. We train the mapping gγ for 5,000 iterations with 128 batch size. On a single NVIDIA A100 GPU, the training can be done in 1 hour for PDF regularizer, and 3 hours for SDE regularizer. The cost of training our regularizer is quite small compared to the resources required for pre-training.
We conduct the same experiments as Sec. 4.2 and Sec. 4.3 with the regularizers trained with tiny corpus. The results are presented in Tab. 11 and Tab. 12 respectively.
On full-set GLUE, the PDF regularizer performs even better on three out of four PETs, and although its performance is affected on Adapter, it still outperforms the vanilla Adapter. The SDE regularizer is slightly affected on three out of four PETs, but
it still brings substantial improvements on all the PETs.
On few-shot GLUE, the impact of the shrinkage of the corpus is relatively obvious. But overall, the regularizers still performs great on all the PETs. The drop in performances are relatively small com-
pared to the boost they bring to vanilla PETs.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 8
3 A2. Did you discuss any potential risks of your work? Section 8
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 4, 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4 and Appendix
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave introducing constraints as future work. Specifically, we apply principal component analysis (PCA) to the output token embedding matrix V ∈ R|V|×d of the PLM, obtaining a rdimensional embedding matrix, and re-normalize each row to have a norm η. Let the resulting embedding matrix be β ∈ R|V|×r. We then use 0 ∈ Rr as the heads for all the bridges, and β as the tails of the diffusion bridges, i. e. , the r-dimensional embedding of y in β is used as βy in X1;βy. The intuition for using β as the tails is that the trajectories of the intermediate states for similar target tokens should be close. In V , similar tokens are close, and β obtained by PCA can well preserve the token similarity after reducing dimensions