FaLA: Fast Linear Adaptation for Replacing Backbone Models on Edge Devices
Authors: Shuo Huang
Lizhen Qu
Xingliang YUAN
Chunyang Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: Foundation Models
Personalization
Efficient Parameter Tuning
On device

Abstract: In this work, we study the language model backbone replacement problem for personalized downstream tasks in a non-stationary on-device scenario. In real world, company may periodically update the knowledge and architectures of backbones to keep the competitive in the market, meanwhile, to accommodate the users' own preference, models are personalized to fit users' own distribution locally.  Traditional full model tuning or transfer learning for such replacements often incur considerable local device training costs and necessitate extensive backpropagation within deep transformer layers. Addressing this issue, we propose a novel, lightweight tuning method for personalized NLP classification tasks post-backbone replacement. Our approach leverages a personalized matrix calculated from documents corresponding to users' old and new backbones. This matrix facilitates top-layer parameter tuning, drastically reducing backpropagation computation. To further mitigate training costs associated with matrix linear optimization, we employ correlation clustering to curate a few examples from personalized cluster sets for individuals. Our method achieves over 1000 times computation reduction in Flops for backpropagation and brings the user-specific initialization for personal matrix yielding significant performance boost compared with popular transfer learning methods.

FaLA: Fast Linear Adaptation for Replacing Backbone Models on Edge Devices
Shuo Huang, Lizhen Qu', Xingliang Yuan, Chunyang Chen
Faculty of Information and Technology, Monash University (shuo.huang,, lizhen.qu, xingliang.yuai chunyang.chen @ @monash.edu
Abstract
Due the fast growing popularity of large lan- guage models (LLMs), novel pre-trained models are released almost every week. However, the com- puting power edge devices is limited. How can edge devices benefit from the most recent advances of pre-trained backbone models without incurring regular and high training costs in local devices? Fig. illustrates when and how the replacement of backbone models may occur. For example, an early stage, an on-device NLP model, consisting of Glove embeddings (Goldberg and Levy, 2014) and the task-specific layers, was fine-tuned using the private data on a local device. Thus, the model may well remember personal information and adapt it- self well into the local distribution. When new pre-trained model is released, such as BERT, the user is eager to the new backbone model but does not want fine-tune the whole model again on the on-device data after the replacement.
this work, we study the language model backbone replacement problem for personal- ized downstream tasks in a non-stationary on- device scenario. In real world, company may periodically update the knowledge and archi- lectures of backbones to keep the competitive in the market, meanwhile, accommodate the users' own preference, models are personalized fit users' own distribution locally. Tradi- tional full model tuning transfer learning for such replacements often incur considerable lo- cal device training costs and necessitate exten- sive backpropagation within deep transformer layers. Addressing this issue, we propose a novel, lightweight tuning method for person- alized NLP classification tasks post-backbone replacement. Our approach leverages person- alized matrix calculated from documents cor- responding to users' old and new backbones. This matrix facilitates top-layer parameter tun- ing, drastically reducing backpropagation putation. To further mitigate training costs as- sociated with matrix linear optimization, employ correlation clustering to curate a few ex- amples from personalized cluster sets indi- 1000 times
Parameter-efficient tuning (Zhuang al., 2023) and continual learning (De Lange et al., 2021) are the closest rescarch areas to address this problem. The current parameter-efficien tuning techniques need to update only small proportion of model pa- rameters during training, but they still need to run expensive backpropagation throughout the whole models. Continual learning algorithms focus on mitigating catastrophic forgetting, while learning new tasks, but they do not consider changing back- bone models with low costs.
viduals. Our method achieves computation reduction in Flops for backpropa- gation brings the user-specific initialization for personal matrix yielding significant perfor- mance boost compared with popular transfer learning methods.
1 Introduction
Therefore, we propose novel task: efficient per- sonalized model tuning after replacing backbone models and provide the first solution, Fast Linear Adaptation(FaLA), that avoids backpropagation through the backbone models after replacement. Herein, we keep the logits of the old model the private training data and apply forward propaga- tion of the new backbone model obtain hidden representations for the task-specific layers. The new hidden representations are used to learn a per- sonal matrix that is regularized by the logits of the
Current Natural Language Processing (NLP) mod- els, including the ones developed for edge de- vices (Vucetic et al., 2022). heavily rely on pre- trained backbone models, such as BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019). Re- cent studies show that it is important to personalize models on edge devices because of the privacy con- cerns and the distribution discrepancies between lo- cal data and public training data (Yan al., 2022).
"Coreesponding author.

---

Global
Global
Task-Specific Layer
Task-Specific Layer
Global data
Global Backbone
Replace
Backbone(Clioee)
GloVe
BERT
Model Deployment
Model Deployment
Global
Task Specific
Private data tuning
Personal
Personal
Predidction Score
Finetuned Task-Specifff
Layer
Private data
Collect output layer and encoder output
Task-Specitic Layer
BERT
Personal
Collect CC center
Matrix
Backbone(GioVo)
Optimisation
Insert personal
matriix
Collect cluster center via Comelation Clustering(C))
Figure The replacement scenario and workflow of FaLA
old model. The corresponding learning problem becomes a simple linear regression problem that is feasible for resource limited devices. In addi- tion, we show that the tuning speed can be further improved by applying correlation clustering (Saha and Subramanian, 2019) and sample representa- tive training examples from the clusters. Through extensive experiments, we show that
techniques (Flek, 2020) offer personal-oriented pre- dictions under user-specific contexts.
As the data used for personalization often asso- ciates with sensitive information like gender, age, or user history behaviors, one compelling approach is to conduct personalization locally especially un- der the regulations like GDPR (Voigt and Von dem Bussche, 2017). Wang (Wang et al., 2019) evalu- ated on-device personalization under federated learning setting where personal data is preserved on local devices. Their experiment on virtual key- board next-word prediction for smartphones indi- cated that a single user can benefit from personal- ization. P-meta (Qu et al., 2022) proposed a mem- ory and computation efficient method for model adaptation by identifying certain fraction of pa- rameters that ensure rapid generalization and dy- namically updating parameters facilitate unseen tasks. Zhang (Zhang et al., 2021) relaxed the con- straint of model homogeneity and aggregated the soft prediction of heterogeneous local devices us- ing knowledge group coefficient matrix to keep the global model updated with personalized context. However, these works did not extend to language tasks, which heavily rely pretrained representa- tion from the backbone model, and all assume that the personalized model static in terms of model architecture and globally learned knowledge. our work, we aim to investigate the preservation of personal information with model replacement
Our method significantly outperforms the competitive baseline that only fine tunes the parameters of task specific layers, in terms of Fl tasks: News Recommendation (Wu et al., 2022) and Hate Speech Detection (Kan- clerz et al., 2022).
Our method consumes only 5.45e+9 FLOPS on News Recommendation which is sev- eral magnitudes lower than that of Adapter (1.40e+12) (He et al., 2021) and LoRA (1.35e+12) (Hu et al., 2021) using the same backbone models.
2 Related Work
On-device Personalization Personalization refers the incorporation of individual characteristics and user-dependent contextual information into generally trained models. Contrary to the assump- tion that globally trained models interpret texts with the same meaning for everyone, personalization

---

while minimizing the fine-tuning cost.
computations of these methods during backprop- agation are conditioned on the depth of the trans- former blocks with adapters, yielding more float- ing point operations per second (FLOPS) ) when training on hardware and also requiring memory to store the gradient. To address these drawbacks, our method only adds trainable parameters on the top layer and reduces gradient flow within the last prediction layer, which significantly reduce the training cost in personalized fine-tuning.
Personalized Neural News Recommendation News Recommendation is essential to mitigate in- formation overload for users (Wu et al., 2022). It is widely employed in large news platforms like Google and Microsoft. The system generally con- sists of user modeling and news modeling. Using learned representations of users and news, the sys- tem predicts whether the news candidate will be clicked or not. In most works, the click decision modeled by a similarity score between the encoded user's news history and the encoded news repre- sentation (Wu et al., 2021). Welch (Welch et al., 2022) used similar past user behaviors to learn dynamic representation for new users. Interactive matching system (Qi al., 2021) encoded user behaviors in the similarity semanties and knowl- edge entities using a knowledge graph. We address the more challenging and dynamic scenario where the backbone is replaced and introduce user-level non-independent and identically distributed data into our work.
Methodology
An edge device model provider regularly devises and trains new models on public or company- owned data. When deploying new models to edge devices, the goal is to minimize local tuning costs to achieve superior performance than the previous ones, while preserving privacy. In the following. the models trained on the public data is referred to as global models, while the models deployed local devices are called local models.
Personalized Hate Speech Detection Hate Speech Detection is highly subjective tasks which leverages user-annotated data to perform supervised learning. Each person can have different considerations when labeling if a document is hateful (Paun et al., 2018; Röttger et al., 2022). Recent works identified the inherent diversity of hate speech data (Kanclerz et al.. 2022) and empirically validated the user-specific perception based on personalized embedding and tailored classifiers. Since we focus on language- based classification tasks that inherently have user-specific distributions, we take personalized hate speech classification as our target task.
3.1 Tasks
We conduct our experiments on two highly per- sonalized tasks, News Recommendation (NR) and Hate Speech Detection (HS).
News Recommendation Our general framework is inspired by similarity matching (Wu et al., 2021) which utilizes the language model as shared backbone to encode news candidate and user his- tory representations. For each user Ug history behaviors are a set of news that they read de- noted as L [4,]2,... A candidate news set, prepared to make predictions about user pref- erence, is denoted as R [ri. Tm,), where m is the number of news candidates spanned by the user. A BERT based news encoder, along with an attention-based aggregation method, is then employed generate representations for the news. News candidate representation encoded as Ve = News (rewsEncoderL... and user representa- tion as u = User Encoder N ewws "ousEnncdrr' (L)) L)). In first training phrase, the model is fine-tuned with globally collected user information. Given user behavior list and candidate news, compute the inner product of two representations, =uTu. the prediction probability. We further incorporate negative sampling into the model training to cal- culate the posterior click probability for positive samples (Okura et al., 2017).
Efficient Parameter Tuning In the context of large language models, the training cost of fully tuning model to new domain from scratch is substantial. To preserve old knowledge and reduce fine-tuning efforts, methods, such as Adapter (He et al., 20 Prefix-Tuning (Li and Liang, 2021) and LoRA (Hu et al., 2021), need to train only a small fraction of model parameters. These meth- ods demonstrate improved performance in model fine-tuning compared to simply fine-tuning the top layer for downstream tasks, but they also encounter certain drawbacks. They often suffer from insta- bility since their parameters need to be randomly initialized and require more training epuchs to con- verge (Chen et al., 2022). In addition, the gradient

---

Hate Speech Detection We consider the clas- sification of Hate Speech (HS) detection, where we use aggregated comments denoted as R = [r,, r2..., mm] Each is a text sequence for a docu- ment. The review representation is calculated using = N NewssEncder((()) (Kanclerz et al., 2022), and this representation is passed through clas- sification layer to calculate the prediction score 0 w Uc-
with the new one (.) and adaptively learns a new (.),,-) by using the logits derived from the previous model. The backbone model is usu- ally referred to as the text encoder. The func- tion (.) essentially maps encoder outputs h to task-specifie labels. For news recommenda- tion, fk.,, (h) = arg maxy po = y/b) with E {0,1}. Herein, h = (u, v). where is the user representation and denotes the represen- tation of the current news. For given news c, p(Y = y/hh azz)) and & = u Vg where the diagonal matrix A& captures user preference from the local distribution. Then 2, is referred to as the logit of the news c. The rationale behind this is that the model intends to recommend the news similar to what user read in the past. Hate speech detection is also binary classification problem. The logit 2e w Age. where he denotes the output from the backbone text encoder for a text e and w is a weight vector shared between users to characterize the importance of each feature. The di- agonal matrix Ag encodes user-specific hate speech patterns.
3.2 Task Formulation
Formally, given text x; with X; X, we aim to build a model to predict the classification label y E y. where y denotes the label space. A model is a composite function m(x) = f o bro (x), where (x)( is referred to as the backbone model and f(z) estimates labels of x based on the outputs of bea (.). At time to. a global model Stp (x) fg bto (x) is trained a public labeled dataset Do {(x;, 2 and its distribution is de- noted by (Y])). The globally trained model is deployed on a set of local devices. On each of such local devices k; there is a local dataset Dr = [(x,, د and Xx characterizes the corresponding local distribution. On each de- vice k, a local model (x) = fk bry (x) is cre- ated by fine tuning the global model on the local dataset without tuning the parameters of (-). Fur- thermore, the distributions are different from each other such that p& (Y A ()])) if + j, in- cluding the distribution of the public dataset. As a result, the optimal function f-(-) + f.) j * k. At time 4. there is a significant update of the global model due to changes of the public dataset or introduction of new techniques. There- fore, the updated global model can use a totally different architecture than previous models, long as the outputs of its backbone model are vector sequences. When deploying the new global model (x,(x) = fg.: o by, (x) to local devices, we aim to minimize the use of computing resources and obtain a new model my.,, (x) = Sk.t, o be, (x) with optimal performance on each local device. Herein, the parameters of the backbone model (.) are frozen during deployment. Due to privacy con- cerns, the local data on each local device is not used to train global models collaboratively.
Instead of using all instances from a local dataset, we build fine-tuning dataset D/ by sampling the most representative examples from local distribu- tion by applying correlation clustering. The details of correlation clustering are described in the fol- lowing subscction.
During deployment, our key hypothesis is that the logits are similar before and after replacing the backbone model. Therefore, we save the log- its & of the model before replacement for all in- stances in the fine-tuning dataset D in order compare them with the logits from the new model.
As A diag(O) (0) is a diagonal matrix for both ap- plications, we can reformulate the learning problem after replacement as a linear regression problem by setting 2; = (Te; for all instances in D/-- where e, = Ui, Vi.t. for news recommendation and e; = Wi.f, hag, for hate speech detection. Be- cause the backbone model is frozen, we need to make only one forward propagation for each in- stance in Dí obtain the corresponding hidden representations. As result, the adaptation process after replacement a ridge regression problem be- low.
3.3 Personalized Model Tuning and Matrix Replacement
j(0)= = ID| Σ(2
2)2 + Sllll
(1)
At time t; when a local device receives a new global model, it replaces its current backbone
The learned diagonal matrix should capture user-

---

specific information for respective applications.
4 Experiment
4.1 Datasets
Correlation Clustering To further reduce the number of document embeddings required to com- pute the personal matrix, we employ correlation clustering to cluster the documents read by the user. In our assumption, each user maintains their own list of documents and reading preferences, which means the clusters gathered for each individual can vary. By using correlation clustering, which does not need to specify the number of clusters, we obtain a varying number of clusters for each user. To cope with limited computing resources for a large document list, we choose the Kwikbucks method (Silwal et al., 2023), a query-efficienn clus- tering method, as clustering algorithm. It first picks pivots from uniformly sampled subset of vertices from the graph and then gradually merges the neighboring nodes based on strong and weak signal pairwise relationships. In our constructed graph, we follow two rules to label the positive edge. Firstly, if the document pair have the same label for the user, for example, the user clicked on both of the documents or clicked on neither of them, we add a positive edge to this document pair, shown in 2, where and j are the indices of documents inside the user's document set.
Our experiments utilizes two distinct datasets perform the tasks of News Recommendation and Hate Speech Detection.
For News Recommendation, we conduct exper- iments using the Mind dataset (Wu et al., 2020) from Microsoft. This dataset consists of news articles and a corresponding log that tracks user behavior over a six six-week period. Each log entry includes the click history of individual users, times- tamped to provide temporal context. To facilitate personalization,
focus on the most active users in the dataset, selecting five for our study. These selected users have between 60 and 107 records cach, indicating substantial interaction with the platform. We split the data for these users into train- ing and testing subsets, with 70:30 ratio. We use the small-size training dataset as our global train- ing set. For Hate Speech Detection, the Wikipedia Detox Aggression dataset (Wulezyn et al., 2017), which contains 116k texts labeled by more than 4k annotators. The aggression score the original dataset ranges from -3 to As our focus is on the dynamic personal distribution the tasks, we follow the setting from (Kanclerz et al., 2022) that emphasizes the significance of performing penalization. We convert the aggres- sion scores to binary system: 0 for non-aggressive content and for aggressive content. Similar our approach in the News Recommendation task, we identify the top five most active annotators, each with review counts ranging from 1382 to 1475, to create a personalized user set. All examples from picked users are not included in the global training datasets.
leverage
1, - -1 otherwise
Yn; = Yd,
E(d;, djj) -
Intuitively, a user who clicks on certain news may be more likely to click on documents with similar content. We also compute pairwise similarity of all documents that the user has read and mark the top k similar documents for certain document with positive edge, shown in 3.
4.2 Experiment Settings
To simulate the training with updates between the cloud and local models, we divide the training pro- cess into three parts: Global Training, Personal Training, and Backbone Replacement Training.
1,
if Cos(di, dz) e TopK D)
Eldi, dz)
1, otherwise
Global Training Phase: In this phase, we use GloVe with 300 dimensions, BERT base size with 768 dimensions, and RoBERTa base model with 768 dimensions to initialize our backbones respec- tively. For each backbone model, train models with globally collected examples and perform eval- uations on personalized data. This establishes the baseline for individual users with different back- bones. During training, we set the learning rate to
(3)
Together with the above two conditions, we con- struct strong signal matrix with hard labels and
cosine similarity label with all pairwise similari- ties of the document for each user, respectively. Thus, each user holds a cluster set that specifically belongs to them, denoted as CU.- We then chouse examples from each cluster for linear optimization.

---

le-5 and randomly pick 4 news articles as nega- tive samples for News Recommendation, For hate speech, we use the same settings as above.
4.4 Evaluation Metrics
4.4.1 Model Effectiveness
Personal Training Phase: In the personal train- ing phase, we further train the personal model for individual users and test it with the corresponding test set. We use a personalized GloVe backbone old fine-tuned backbone, which is ready for backbone replacement. Additionally, we con- duct full parameter model tuning for BERT and RoBERTa as baseline, and perform FaLA for each user. To emphasize the significance of per- sonalization, we repeatedly test all users across all personalized models to observe performance changes. As we are only concerned about the per- formance of penalization with replacement, we do not split personal data over more time intervals, which could be studied further.
We compare the performance of state-of-the- art News Recommendation systems (Wu et al., 2021, 2019) with our proposed method on per- sonalized users. Following previous work on News Recommendation metrics (An et al., 2019; Wu et al., 2019), we use AUC as an evaluation met- ric. For Hate Speech Detection, and F-1 score (Kanclerz et al., 2022) as our metrics to illustrate the effectiveness of the personalized classification model.
accuracy
4.4.2 Model Efficiency
To evaluate computational efficiency, we record the FLOPS (Floating Point Operations Per Sec- ond) for each personal training setting. FLOPS is measure of scientific computation capability of device. Language models containing large num- ber of parameters use FLOPS as an indicator of their computation cost. A model that achieves sim- ilar performance with fewer FLOPS is generally considered more efficient. We employ the Tensor- Flow profiler (Abadi et al., 2015) to record forward and backward computation of training batch on NVIDIA 3090 to illustrate the computational differences among the tested methods.
Backbone Replacement Training: For the per- sonalized model, we replace the outdated back- bone with a new one. We switch the personal- ized backbone to a globally trained new backbone, specifically, from GloVe to BERT and GloVe to RoBERTa We insert the personalized computed diagonal matrix into the top layer of the new global model for the corresponding users. Then, we carry out the same procedure in personal training but only tune the parameters within the personal ma- trix.
4.5 Implementation Details
4.3 Baseline
Overall, our experiments are carried out on an NVIDIA 3090 24G GPU over multiple rounds. We use hyperparameters suggested from previous work to establish the baseline. In addition, we mainly tuned three hyperparameters during the training process: model learning rate 7), À for L2 regularization, top similarity examples used construct strong signal relation group, and reg- ularization selection for linear optimization. As these three parameters have a significant impact on the downstream task, we use a grid search to obtain the optimal values for these terms while keeping the other hyperparameters constant. For News Recommendation. we use the top 10 similar examples, 1e-3 learning rate for linear optimiza- tion and le-5 for backbone tuning, and set À to le-1. For Hate Speech Detection, we use le-3 for linear optimization, le-4 for backbone tuning, and set the L2 regularization term to lc-4. For the impact different selections, we further discuss this in
We consider five basclines to perform personalized model tuning for each user, depicted in Table 2.
Global Model (Global): It directly applies the new global backbone model local devices.
Full parameter tuning (Full): It fine-tunes all parameters of the new global model locally.
Top layer fine tuning (Top): This method fine- tunes only the parameters of the task-specific lay- ers, which is widely used for most on-device mod- els (Xu et al.. 2018).
Adapter: Adapter (He et al., 2021) used inside the BERT model is popular in transfer learning for language models to reduce the training parameters inside the backbone.
LoRA: LoRA (Hu al., 2021) constructs low rank decomposition for BERT 's high dimensional weights inside the language model, claiming the state-of-the-an performance for model fine tuning.

---

Table I: Measured FLOPS of one training batch for News Recommendation
ing personalized training. For the 5 most fre- quent users from News Recommendation and Hate Speech Detection, we recorded their average performance and standard deviation in terms of AUC for News Recommendation, Accuracy and F- for Hate Speech Detection We firstly observed the impact of personalization in our old backbone GloVe as shown in table 3, the personalization improves the performance of GloVe based models when encountering new users, indicating the per- sonal preference learned by the system. As shown in Table 2, fully parameter-fine- sistently outperform the globally trained models. Moreover, for each personalized model, we tested it across the data of all users to show the performance differences for the tuned model using a heat map as shown in Figure 2. Each row in the heat maps represents the performance difference of all users tested by one users' personalized model. The distin- guished prediction for each user further illustrates the inherent distribution difference, enhancing the necessity of performing personalization.
BERT
RoDERTa
Adapter Method
Torward
Backward
Forward
wackward
1.566e+122
27100e=12
2.7127e-12
54200e+
Adapter
1.3941e412
14000c=12
2.7883e-12
2,8000+12
LaRA
1.3633+112
L3500e+12
2.7226e-12
2,6900++12
13353c+12
5.4500e=09
1.090c+10
Heatmap of User Prediction
models con-
6:003
600
41M7
0.903
-03
0.001
6am
6.41
6.40
63811
631
6008
Figure 2: Performance difference for individuals
Personalized Tuning For personalized fine tun- ing, as Table 2 depicts, if we only tune the task specific layers without additional parameters added in, the user domain adaption ability is poor for all metrics. Adapter failed to adapt user do- main with limited examples. From the table 2, it is evident that our proposed method outperforms the other techniques in terms of AUC and F-1 score. Notably, the performance improvement is not marginal but significant, reinforcing the effi- cacy of our approach. Specifically, compared to the Adapter, our method shows an improvement of approximately 6.46%,7.8. 7.81 %, and against the LoRA, the enhancement is about 2.8%,2.26% for BERT andRoBERTa in News Recommendation re- spectively. In Hate Speech Detection FaL.A con- sistently outperforms all baselines least over 6% in F-1 for both backbones. These significant im- provements underscore the superiority of our ap- proach.
4.6
Experiment Results
4.6.1
Measured FLOPS
As presented Table 1, the computational ef- ficiency the forward pass per batch does not show considerable escalation, owing to the rela- tive insignificance of the added parameters com- pared to the full parameter size tuning of the base models. For backpropagation, full parameter tun- ing yields almost 2 times the FLOPS compared forward computation. Considering the Adapter and LoRA methods, the computational cost nearly mirrors that of the forward pass. This can be at- tributed to the backpropagation of the gradient into the transformer blocks equipped with adapter mod- ules. Since our method is only applied to the top layer and involves only one layer gradient com- putation, this leads striking improvement in computational efficiency during backpropagation training, making our approach over 100 times more efficient than both the Adapter and LoRA methods.
This underlines the superiority of our method in terms of computational efficiency.
Ablation Study
4.6.2 General Performance
To further explore and validate our methods, we conduct ablation study focusing on diverse factors throughout our personalized matrix computation process.
Personal Training The experiments compar- ing globally trained models to personalized mod- els demonstrate the effectiveness of employ-

---

Table 2: Overall Performance
Tasks
Model
Metrics
Global
Full
Adapter
LoRA
Top
FaL.A
NR
BERT
AUC
57.45 + 3.05
63.07 + 10.25
61.01 13.11
64.67 11.27
61.39 + 12.83
67.47 13.65
NR
RoBERTa
AUC
60.11 # 1.27
60.50 # 11.99
57.81 # 9.25
60.98 = 17.28
63.38 + 12.50
65.64 = 5.26
HS
BERT
ACC
43.97 23.14
87.02 = 3.62
66.57
16.12
86,61 5.03
64.85 = 15.15
85.98 = 5.67
HS
BERT
25.70
44.00 +: 11.69
18.20 +: 8.82
41.55
18,46
22.94 :: 10.98
50.94 +: 6.49
HS
RoBERTa
ACC
24.84
27.26
87.74 4.81
57.88
31.48
87.54
4.30
54.06 + 33.30
86.17 + 6.83
HS
RoBERTa
F1
17.97 10.21
40.83 1: 21.21
16.88 + 15.77
39.21 d: 23.51
10.40
9.10
48.42 11.23
Table Performance Global and Personalized GloVe
increases the training loss and demands extra train- ing effort for both tasks. Our empirical findings show that the optimal K for K-means for BERT based both tasks 50 and 10 for RoBERTa based tasks. Surprisingly, for BERT based models, FaLA with K examples showcased the best performance, surpassing the FaLA for News Recommendation. With the collected cluster number from FaLA be- ing less than 50, randomly sampled the 50 - K examples from the remaining examples to construct the training examples for linear optimization. This resulted superior personal initialization. For the RoBERTa based model, which limited 10 exam- ples for training, our method exhibited competitive results for our method in 10 examples compared to K-means. And FaLA demonstrated comparable performance in all cases with respect to the full rank matrix initialization.
Task
Metrie
Global
Personal
NR
AUC
45.42 +. 8.11
53.14 7.29
HS
ACC
49.41 = 33.52
83.86 6.90
HS
F-1
32.11 + 22.46
37.52 + 20.21
5.1 Validating Personalized Matrix Tuning
In our design, we added personalized matrix when updating the model, we would like to answer two main questions: the initialization of a personal- ized matrix essential? If so, how does the quality of our clustering compare to conventional cluster method? To derive results, we considered the fol- lowing scenarios:
Random Initialization The matrix param- eters are initialized randomly to examine whether the matrix inserted into the top layer is effective.
5.2 Evaluation of Adaptation Objectives
Full Rank Example Optimization The ma- trix is trained full rank with multitude of examples gathered from the users' log, equiv- alent to the dimension of the matrix.
Choices of Linear Objectives We explored the influence diverse linear objective functions on the system's performance. Each combination of ob- jectives denotes a unique learning approach, which can significantly impact the results. The prediction of our system is logits that can be further converted to probabilities using the sigmoid function. As indicated in Table 5, individually eliminated each objective function to observe the subsequent performance changes. The objectives evaluated included Mean Square Error(MSE) and Cross En- tropy(CE) both with and without L2 regularization. We found that combining these two terms led to no noticeable performance difference, suggesting that this objective combination may not be crucial our model's performance. However, when we solely applied MSE optimization, it resulted in the best performance for both tasks, underscoring the effectiveness of logits mapping for clustered examples.
K-Means We implement k-means in our ex- ample selection process as it is fast and pop- ular clustering method.
Our method with K examples To justify the performance with K-means, we compiled samples from the top clusters using FaLA, and tuned the model with the same quantity of examples as K-means.
FaLA The baseline performance for FaLA
As per the results shown in Table 4, we observed that directly applying the matrix to the top layer does not sufficiently impact training in most cases. The method's 's performance is the least effective as

---

Table 4: Matrix Initialization Methods
Task
Model
Metric
Random Initial
Full Rank
K-Means
F
FaLA
NR
BERT
AUC AUC
57.45%
67. 17.48%
66.29%
69.16%
67.47%
NR
RoBERTa
60. 11111%
62.76%
59.66%
61.94%
65.64 %
HS
BERT
ACC
82.71%
82.26%
84.35%
83.95%
85.98%
HS
BERT
F1
42.43 %
48. %
41.30%
42.93%
50.18 %
HS
RoBERTa
ACC
84.24%
79.14%
84.86%
85.94%
86.17 %
HS
RoBERTa
F1
46.25%
51.55%
43.42%
46.85%
48.42%
Table Selection of Objective Term
training scenario that the service providers want minimize the computational cost of updating the backbone of the on-device models deployed their client devices while preserving privacy of the local data. Specifically, we reformulate the com- plex model replacement problem as a simple Ridge regression problem. Our approach notably outper- forms established baseline techniques in terms of both accuracy and computational efficiency. This work identifies research problems regarding backhone replacement. For future work, may move beyond classification tasks broader appli- cations with diverse backbones.
TASK
Model
Lambda
CE
MSE L2
MSE L2 CE
NR
BERT
0.0001
62.37%
67.47%
65.16%
NR
BERT
0,001
67.74%
65.34%
63.31%
NR
BERT
0.01
67.26%
62.91%
64.35%
BERT
0.1
67.31%
69.05%
65.27%
68.65%
64.61%
HS
BERT
0.0001
84.14%
86.33%
83.58
IS
BERT
0.001
84.57%
83.54%
84.08%
HS
BERT
0.01
85,59
81 62.
81. .14%%%%
HS
BERT
0.1
82.09%
85.41%
82.84%
HS
BERT
0
84.60%
84.644%
84.90%
Hyperparamenter Sensitivity We also engaged a sensitivity analysis of the system's perfor- mance in relation to hyperparameters, particularly focusing on the L2 regularization term. This term is pivotal in controlling the model's com- plexity and preventing overfitting. Given that method necessitates only a handful of ex- amples to optimize high-dimensional parameters, we intended to calibrate the weight decay fac- for avert overfitting during the learning pro- cess. We observed noticeable trend whereby an incremental increase in the L2 term from lower to higher values corresponded with an enhance- ment in the model's accuracy for Cross Entropy (CE) when evaluated individually. Meanwhile, the Mean Square Error (MSE) displayed optimal per- formance when the regularization strength 0.1 for News Recommendation and 0.0001 for Hate Speech Detection. This underscores the sig- nificance of carefully selecting this hyperparameter to ensure both stability and accuracy in our model's predictions.
Limitations
The limitations of our work are three-fold:
Firstly, our experiments primarily conducted subset of datasets. The performance, effi- ciency, and efficacy of parameter tuning may vary across different types of datasets. In our experi- ments, replaced backbone models are BERT and RoBERTa. More comprehensive experiments with more backbones may make the empirical results more solid.
Secondly, hyperparameter search is still time- consuming for each user. More efficient methods may be applied to reduce computational cost.
Thirdly, in our experiments, we apply one for- ward pass to each user example and store the corre- sponding representations in disk to build clusters. way of trading off the storage
This is regarded cost with the computational time. Further reduction of the storage cost remains an open question.
To sum up. work investigates practical scenario regarding efficient model deployment for edge devices. The aforementioned limitations still pose potential areas future research.
6 Conclusion
this paper, we introduce FaLA, the fast linear adaptation method for backbone model replace- ment. We consider a novel and realistic dynamic

---

References
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro. Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, lan Goodfellow, Andrew Harp. Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefow- icz, Lukasz Kaiser, Manjunath Kudlur, Josh Leven- berg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Nly Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas. Oriol Vinyals, Pete Warden, Mar- tin Wattenberg, Martin Wicke, Yuan Yu, and Xiao- qiang Zheng. 2015. TensorFlow: Large-scale chine learning on heterogeneous systems. Software available from tensorflow.org.
Kamil Kanclerz, Marcin Gruza, Konrad Karanowski, Julita Bielaniewiez, Piotr Milkowski, Jan Kocoñ, and Przemyslaw Kazienko. 2022. What if ground truth is subjective? personalized deep neural hate speech detection. In Proceedings of the 1st Workshop on Per- spectivist Approaches to NLP@ LREC2022, pages 37-45.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages 4582- 4597.
Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie, 2019. Neural news recom- representa-
mendation with long- and short-term tions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 336-345, Florence, Italy. Association for Computa- tional Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach, arXiv e-prints, page arXiv: 1692.
Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. 2022. Revisiting parameter- efficient tuning: Are we really there yet? arXiv preprint 222.:2200..0766..
Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017. Embedding news recom- mendation for millions of users. In Proceedings of the 23rd ACM SIGKDD international conference knowledge discovery and data mining, pages 1933- 1942.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021. continual learning sur- vey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelli- gence, 44(7):3366-333 3385.
Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018. Comparing bayesian models of annotation. Transac- rions of the Association for Computational Linguis- tics, 6:571-585.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint
Tao Qi, Fangzhao Wu, Chuhan Wu, and Yongfeng Huang. 2021. Personalized news recommendation with knowledge-awar interactive matching. In Pro- ceedings of the 44th International ACM SIGIR Con- ference on Research and Development in Information Retrieval, SIGIR '21. page 61-70, New York, NY. USA. Association for Computing Machinery.
Lucie Flek. 2020. Returning the N to NLP: Towards contextually personalized classification models. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7828- 7838, Online. Association for Computational Lin- guistics.
Zhongnan Qu, Zimu Zhou, Yongxin Tong, and Lothar Thiele. 2022. p-meta: Towards on-device deep model adaptation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing, pages 1441-1451.
Yoav Goldberg and Omer Levy. 2014. word2vec explained: deriving mikolov et al.'s negative- sampling word-embedding method. arXiv preprint arXiv:1402.3722.
Paul Röttger. Bertie Vidgen. Dirk Hovy, and Janet Pier- rehumbert. 2022. Two contrasting data annotation paradigms for subjective nip tasks. Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 175-190.
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding. Liying Cheng, Jiawei Low, Lidong Bing. and Luo Si. 2021. On the effectiveness of adapter-based tuning for pretrained language model adaptation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 1 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 2208- 2222.
Barna Saha and Sanjay Subramanian. 2019. Correla- tion clustering with same-cluster queries bounded by optimal cost. arXiv preprint arXiv: 1908.00 04976.

---

Sandeep Silwal, Sara Ahmadian, Andrew Nystrom. Andrew McCallum, Deepak Ramachandran, and Seyed Mehran Kazemi, 2023. Kwikbucks: Cor- relation clustering with cheap-weak and expensive- strong signals. In The Eleventh International Confer- ence on Learning Representations.
Yikai Yan, Chaoyue Niu, Renjie Gu, Fan Wu, Shaojie Tang, Lifeng Hua, Chengfei Lyu, and Guihai Chen. 2022. On-device learning for model personaliza- tion with large-scale cloud-coordinated domain adap- tion. In Proceedings of the 28th ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining, pages 2180-2190.
Paul Voigt and Axel Von dem Bussche. 2017. The eu general data protection regulation (gdpr). A Prac- tical Guide, 1st Ed., Cham: Springer International Publishing, 1063152676::0-55555
Jie Zhang, Song Guo, Xiaosong Ma. Haozhao Wang. Wenchao Xu, and Feijie Wu. 2021. Parameterized knowledge transfer for personalized federated learn- ing. In Advances in Neural Information Processing Syxtems, volume 34, pages 10092-10104. Curran As- sociates, Inc.
Danilo Vucetic, Mohammadreza Tayaranian, Maryam Ziaeefard, James J Clark, Breut H Meyer, and War- ren Gross. 2022. Efficient fine-tuning of bert mod- els on the edge. In 2022 IEEE International Sympo- sium Circuits and Systems (ISCAS), pages 1838- 1842. IEEF
Bohan Zhuang. Jing Liu, Zizheng Pan, Haoyu He, Yue- tian Weng, and Chunhua Shen. 2023. A survey on efficient training of transformers. arXiv preprint urXiv:230..01107
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise Beaufays, and Daniel Ramage. 2019. Federated evaluation of on-device personaliza- tion. arXiv preprint arXiv: 1910.10252.
A
Appendix
A.1
User Clsuter States
Charles Welch, Chenxi Gu, Jonathan Kummerfeld, Verónica Pérez-Rosas, and Rada Mihalcea. 2022. Leveraging similar users for personalized language modeling with limited data. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages
We record the number of optimal k similarity for all models and tasks shown in table 7. From the table, we can see that for News Recommendation task with small value of topk to construct the graph. there is fewer postitive edge constucted therefore re- sulting more clusters. For Hate Speech Detection. the optimal value is relatively high, the number of cluster center is less.
Chuhan Wu. Fangzhao Wu, Mingxiao An, Jianqiang Huang. Yongfeng Huang, and Xing Xie. 2019. Npa: neural news recommendation with personalized at- tention. Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2576-2584.
A.2 Heat Map for Hate Speech Detection
We present the heat map of hate speech detection to show the performance of trained user-specific classifiers. As depicted in Figure 3, each classifier act in a distinguished manner demonstrating the existence of user specific distributions.
Chuhan Wu, Fangzhao Wu, Yongfeng Huang, and Xing Xie. 2022. Personalized news recommenda- tion: Methods and challenges. ACM Trans. Inf. Syst.
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2021. Empowering news recommendation with pre-trained language models. SIGIR '21, page 1652-1656, New York, NY, USA. Association for Computing Machinery.
A.3 Training without Global Data
We also explored the setting with directly fine tun- ing the model with user's data respectively. As illustrated in Table 6, solely relying on fine-tuning without global training led to models that were inadequately personalized and exhibited consider- able performance variation. For instance, while our method achieved an AUC of 65.64% for news rec- ommendation, the result for a model that was only line-tuned stood at 52.44%. This significant perfor- mance discrepancy was consistent across various settings.
Fangzhao Wu, Ying Qiao, Wu, Tao Qi, Jianxun Lian. Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. 2020. MIND: A large-scale datasct for news recommenda- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3597-3606, Online. Association for Computational Linguistics.
Chen, Chuhan
Ellery Wulczyn, Nithum Thain, and Lueas Dixon. 2017. Wikipedia Talk Labels: Aggression.
The rationale behind this discrepancy is that task- specific global training embeds a broader knowl- edge base regarding the tasks into the model. Sub- sequent personalized training can then adjust this generalized knowledge to cater to individual user
Mengwei Xu, Feng Qian, Qiaozhu Mei, Kang Huang. and Xuanzhe Liu. 2018. Deeptype: On-device deep learning for input personalization service with min- imal privacy concern. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol, 2(4).

---

Table 6: Fine Tuned Model with Glo
Task
Metric
BERT
RoBERTa
BERT -FaLA
RoBERTa -FaLA
NR
AUC
55.86%1 4.05%
52.44% 12.48%
65.64% 67.47%+13.65%% 48.42% 11.23%
HS
F-1
40.93% + 12.41%
4 24.76%
50.94%:16.49%
Table 7: User Clusters
TASK
MODEL
Topk Similarity
User
User 2
User 3
User 4
User
NR
BERT
10
42
100
112
429
90
NR
RoBERTa
10
42
100
68
418
104
HS
BERT
250
15
8
24
26
HS
RoBERTa
250
21
12
20
20
Heatmap
414X
8422
64:08
L10E
SC:M
Figure 3: Performance difference for individuals in Hate Speech Detection
distributions. However, in the absence of substan- tial dataset for specific user, accurately model- ing such user-specific distribution becomes im- mensely challenging. This is compounded by the fact that local devices have constraints in terms of data storage capacity and require periodic updates.


Reviews:
[{"review_id": "O0toIn7fLY", "reviewer": "Reviewer_MvrZ", "summary": "", "questions": "Question A: Do I understand it right that in line 207 the input to the NewsEncoder is a *set* of news L, not a single news article? Furthermore, is it also NewsEncoder in the equation on line 221 (subsection Hate speech detection)?\n\nQuestion B: Could you please explain the equation 5 a little better? (where does it come from)", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe sincerely appreciate your thorough review and detailed feedback on our submission. Your insights have been invaluable in helping us improve the quality and clarity of our paper. Below are our responses to each of your comments:\n\nReasons To Accept and Contributions:\n\nWe are pleased that you recognize the value of our new approach and its potential implications in the field. We also acknowledge and appreciate your commendation on our \"Limitations\" section.\n\nQuestions For The Authors:\n\nQuestion A: Do I understand it right that in line 207 the input to the NewsEncoder is a set of news L, not a single news article? Furthermore, is it also NewsEncoder in the equation on line 221?\n\nYou're right about line 207; the input to the NewsEncoder is indeed a set of news \n$L$, not a single news article. We apologize for any confusion caused and will clarify this in the revised manuscript. Also, regarding line 221 in the Hate speech detection subsection, you are correct. We use the same denotation as in this task the comments are same encoded with news encoder like previous task into numeric space. We will ensure clarity in the equation's representation.\nQuestion B: Equation\nIn the equation 5, it can be viewed as prediction layer of a classifier or similarity calculation equation.\nWe would like to make our prediction which is the summation of $\\theta X_i$ calculated by Hadamard product $\\odot$ to be adjusted equal to the prediction of old model which is $\\hat{Yt}_{i-1}$ using a parameter vector $\\theta$. This parameter vector is the diagonal elements of previously mentioned matrix from this Equation 2. Equation 5 is reformulated from equation 2 to adapt to a form of linear optimization.\n We understand that its presentation might be confusing, and will enhance the description in the revised version to offer a clearer understanding by explicitly explaining each element and operations in detail.\n\n We believe these revisions have improved the readability and presentation of our paper. Once again, thank you for your constructive feedback. We hope the revised version addresses your concerns and meets the standards of EMNLP.\n\nWarm regards.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Reasons To Accept:\n* New approach to change a backbone model on device with low cost.\n* The approach is validated on two NLP tasks: News recommendation and Hate speech detection.\n* Extensive experiments with many models and datasets.\n* Ablation study.\n* Thorough \"Limitations\" section which, evidently, highlights some limitations of the current research but also, probably, shows some directions of future research in the field, and overall is a very good example of how to approach the writing of the \"Limitations\" section. ", "reasons_to_reject": "I see no solid reason to reject the paper.", "questions_to_authors": "Question A: Do I understand it right that in line 207 the input to the NewsEncoder is a *set* of news L, not a single news article? Furthermore, is it also NewsEncoder in the equation on line 221 (subsection Hate speech detection)?\n\nQuestion B: Could you please explain the equation 5 a little better? (where does it come from)", "paper_topic_and_main_contributions": "The paper is about how to replace a backbone network on a user device with minimal computational costs. The question is important because simple download-and-replace is not enough. The model have to be adjusted somehow to better solve personalized downstream tasks.\n\nContributions:\n* New approach to tune the model on the user device after the replacing of the backbone model. The approach outperforms the baselines on News recommendation and Hate speech detection tasks in both quality and computational efficiency (FLOPS).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Line 35 (and many other places): Missing space between text and reference. For example, \"for edge devices(Vucetic et al ...)\"\n* Line 57 (and many other places): Excess space before dot or comma. For example, \"such as BERT , ...\"\n* Line 97: duplicate \"consumes only\"\n* Lines 27, 164: \"Flops, FLOPS\". Different symbols for the same notion.\n* Lines 232, 311: \"L2, l2\". Also, different symbols for the same things.\n* Line 256: Dot instead of comma.\n* Line 262 (eq 2). Should be \"\\diag\" (mathrm font), not just \"diag\" (italic font).\n* Line 289 (eq 3). Duplicate \"=\" sign.\n* Line 296 (eq 4). Not quite understand the clause with Cos \\in TopK.\n* Line 307 (eq 5). Probably wrong indices for \"X\" letter.\n* Line 318 (eq 6). The top vertical space between text and equation is bigger than the bottom space.\n* Line 343: Wrong formatting of negative number \"-3\" (this is not a minus sign).\n* Line 431: FLOPS (Floating point operations per second). I guess it is too late for such an explanation :)\n* Line 494: \"F-1\" splits on two lines.\n* Table 3: not all best values are in bold (there are two lines with no bold values at all).\n* Table 4: The \"%\" sign in table rows can be omitted. Just name the column, eg. \"Random Initial, %\" and so on.\n* Table 5: In my opinion, it would be better to span the values in Task column on several lines (so as to do without x5 repeating).\n* Line 635: \"Specifically, we preserve the predicted logits from old logits ...\" The sentence probably needs revising.\n* Line 648: \"As our methods can efficiently bring personal initialization and bring efficient parameter tuning.\" Also, probably needs revising.\n* Line 650: \"folders\" ?\n* Table 6: Strange formatting: it takes two text columns, and is placed on a separate page.\n* Figures 2 and 3: small font size.\n* Line 854: Typo in \"clusters\"\n"}}, {"review_id": "ViCcOOpNBN", "reviewer": "Reviewer_GxQw", "summary": "", "questions": "- Is the backbone of the model frozen when producing the first personalized local model at time $P_{t_0}$ ? \n- Did you explore any experimentation without training the global model on all user's data? Is there a clear separation between the data using the global training phase and the personal training phase?\n- Why do you only select the top 5 users? Does this mean the global model is trained on the data for just those 5 users?\n- It's surprising that Full would perform worse. Why is this? When you say it finetunes all parameters of the new model, I'm assuming this is for each user correct?\n- Line 497 - 500 You mentioned this shows applicability to new unseen users. In which experimentation is this explored?\n- What is the performance gap across time gaps? How did FaLA perform at timestep 0 compared to timestep 1?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for the detailed review and constructive feedback on our submission. We sincerely appreciate the time and effort you spent in reviewing our paper. We have addressed each of your concerns and made appropriate revisions to improve the clarity and quality of the paper. Below, we provide a point-by-point response to your comments.\n\nWe would like to respond your review for reasons to reject and questions respectively.\n1. Diagramming needs work:\n\n\nResponse: \n\nWe acknowledge your concern regarding Figure 1. Based on your feedback, we have revised the figure to provide a clearer representation of the system workflow and the problem at hand. We have the following adjustment to our digram:\n\nWe have updated the figure to more clearly delineate the system's workflow and the central issue we're addressing.\nTo elucidate the model replacement process, we've highlighted the shift from GloVe to BERT more distinctly within the revised figure.\nOur approach is now visually integrated into the workflow to offer a more comprehensive view.\nAdditionally, we have incorporated steps that detail how correlation clustering is utilized for data collection and how the personal linear matrix is optimized.\nSubsequently, we illustrate the fine-tuning of the personal matrix to align with the updated, personalized backbone model.\nTo ensure clarity, added annotations and explanations accompany each step, making the process more intuitive for readers.\n\nWe trust that these revisions more accurately address your comments and enhance the figure's interpretability.\n\n\n\n2. Experimentation is unclear:\n\nResponse: \n\nWe apologize for any confusion caused. we are describing the phase in which the model backbone is substituted, and the local device begins personalizing the newly integrated on-device model. Our training processes are in three distinct stages, capturing the transformations both preceding and succeeding the backbone replacement. The primary objective of this stage is to fine-tune the new model locally, leveraging private data.\n\nCentral to our discourse is the acceleration of the fine-tuning process, harnessing pre-existing information to minimize associated costs. Our whole methodology is operational during the \"backbone replacement training\" phase. To elucidate, we have incorporated a new workflow. This schematic underscores the chronology and mechanisms of our process, highlighting the steps at which we undertake example clustering, linear optimization, and subsequent model training. We believe this visual aid enhances clarity and provides a more intuitive understanding of our approach.\n\nWe hope this explanation offers a more coherent insight into our methodology and addresses the queries raised.\n\n3. Applicability to generative tasks:\n\nResponse: \n\nWe appreciate your feedback. As stated both in our abstract and throughout the paper, our proposed methodology specifically targets NLP classification tasks. The approach we have developed centers around integrating a personalized matrix directly into the model's classification layer. Given that generative tasks possess a fundamentally different problem formulation and output architecture compared to ours, they were not within the purview of our study.\n\nIn our revised version, we will articulate this more clearly in the \"limitations\" section to ensure comprehensive coverage of the topic's scope and ensure there's no ambiguity regarding our focus.\n\nWe trust this amendment provides clarity and addresses the nuances of our research.\n\nQuestions For The Authors:\n\nQ: Is the backbone of the model frozen when producing the first personalized local model at time $P_{t_0}$?\n\nResponse: No, we use $P_{t_0}$ to denote the stage the local model personalized using arbitrary method. In our example, full personalized training as we have not introduced the lightweight fine tuning yet. However in real-world case, it can be learned with different methods as long as the personalized model can perform better than generalized global model.\n\n\nQ: Did you explore any experimentation without training the global model on all user's data?\n\nResponse: Yes we explored that this setting with directly fine tuning the model with user's data respectively. We report our result here for only fine tuning the BERT and RoBERTa in news recommendation, the AUC is 55.86/%$\\pm$ 4.05/% and 52.44/% $\\pm$ 12.48/% respectively. For hate speech detection, the F-1 score is 40.93\\% $\\pm$ \\%12.41 and 42.51\\%$\\pm$ \\%24.76.\n\nFrom above result, solely relying on fine-tuning without global training led to models that were inadequately personalized and exhibited considerable performance variation. For instance, while our method achieved an AUC of 65.64\\% for news recommendation, the result for a model that was only fine-tuned stood at 52.44\\%. This significant performance discrepancy was consistent across various settings.\nThe rationale behind this discrepancy is that task-specific global training embeds a broader knowledge base regarding the tasks into the model. Subsequent personalized training can then adjust this generalized knowledge to cater to individual user distributions. However, in the absence of a substantial dataset for a specific user, accurately modeling such a user-specific distribution becomes immensely challenging. This is compounded by the fact that local devices have constraints in terms of data storage capacity and require periodic updates.\n\n\n\n\nQ: Why do you only select the top 5 users? Does this mean the global model is trained on the data for just those 5 users?\n\n\n\nResponse:\nFor the news recommendation task, our initial decision to select the top 5 users from the dataset was primarily influenced by the dataset's limitations. The dataset becomes exceedingly sparse when including users with fewer data points, which compromises the quality of personalization. Insufficient data may fail to capture nuances specific to individual user distributions. To effectively execute personalized training, it's important that we consider data robust enough to model these user-specific tendencies and preferences.\n\nRegarding your second question: We've ensured a distinct partitioning of data for the global training phase and the personalized training phase. Upon identifying our target users, in this case, the 5 selected users, we meticulously extracted their data from the main dataset, double-checking to prevent any overlap with the global training data set. This guarantees that the data pertaining to these users is isolated and does not intersect with the global training pool. This protocol was consistently followed for both the news recommendation and hate speech detection tasks.\n\nQ: It's surprising that Full would perform worse. Why is this?\nWhen you say it finetunes all parameters of the new model, I'm assuming this is for each user correct?\n\nResponse: \nIn the full model parameter tuning as the global model is trained on global distribution, for personalized tuning, as training error drops, the test error increases. The data collected is non-i.i.d and imbalance in amount compared with global training data set. In news recommendation, the most frequent user only hold 103 records for training and testing. It is much less than the data used for global training which is over 150k records. This can cause the overfitting for limited data with respect to large model.\n\nQ: Line 497 - 500: In which experimentation is the applicability to new unseen users explored?\n\nResponse: We apologize for the oversight. As our users for personalization is drawn from global dataset and any records of them will not be seen on global training stage. we refer our ability of fast adapt to new users as the applicability to new unseen users. These unseen users can accumulate their behaviors in a short period then our model can use these information to adapt model when replacing the backbone.\n\nQ: What is the performance gap across time gaps? How did FaLA perform at timestep 0 compared to timestep 1?\n\nResponse: Here we only consider the scenario of adapting one gap. Since the previous backbone can be personalization in arbitrary ways as along as it contains the information of user-specific distribution, for a new update, we only need to hold the prediction score from previous global model. Here for timestep 0 we consider it as time before we apply FaLa to update personalized model but the local model has also shown the personalized performance if trained using other personalized methods. \n\nWe will add the specification of this question into our paper to avoid this confusion again.\n\n\n\n\n\nOnce again, thank you for your feedback. We believe that these revisions substantially improve our paper, making it clearer and more informative for the readers. We hope the revised version addresses your concerns and meets the criteria for publication at EMNLP.\n\nWarm regards.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The problem scenario is interesting and has real-world implications. Keeping a clear separation between cloud and edge is vital for the efficiency and privacy of ML models on low-powered devices.\n- Initial experimentation results show that FaLA is more computationally efficient and can outperform comparable PEFT methods.", "reasons_to_reject": "- Diagramming needs work, Figure 1 does a poor job of providing the user with any understanding of the system workflow or the problem at hand. While interesting overall, the paper does seem rushed.\n- Experimentation is unclear: limited information is provided on the main focus of the paper which is \"Backbone Replacement Training\".\n- Applicability of the method to generative tasks is not explored in experimentation. As such I would advise the authors to revise the wording to make it explicit that this approach is geared toward encoder-based classification tasks.", "questions_to_authors": "- Is the backbone of the model frozen when producing the first personalized local model at time $P_{t_0}$ ? \n- Did you explore any experimentation without training the global model on all user's data? Is there a clear separation between the data using the global training phase and the personal training phase?\n- Why do you only select the top 5 users? Does this mean the global model is trained on the data for just those 5 users?\n- It's surprising that Full would perform worse. Why is this? When you say it finetunes all parameters of the new model, I'm assuming this is for each user correct?\n- Line 497 - 500 You mentioned this shows applicability to new unseen users. In which experimentation is this explored?\n- What is the performance gap across time gaps? How did FaLA perform at timestep 0 compared to timestep 1?", "paper_topic_and_main_contributions": "This paper focuses on the problem of language model backbone replacement for personalized training in the cloud edge scenario. In real applications, user data is stored on the local device alongside the task-specific parameters trained on that data whereas the main base/backbone model is stored within the cloud. This is particularly important for the personalized use case due to privacy concerns, the efficiency of model training as well as the distribution discrepancies between user data and global data. Over time this base backbone model is updated, resulting in the need to retrain all local models across users. This paper proposes the task of efficiently personalizing the model after the backbone is replaced. They introduce Fast Linear Adaption (FaLA) where they preserve old logits and compute a new personalized matrix based on the hidden representations of the new model. They showcase how this method is computationally more efficient compared to existing parameter-efficient fine-tuning methods.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iaHucFJiDq", "reviewer": "Reviewer_y97i", "summary": "", "questions": "refer to reasons to reject", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for taking the time to provide a thorough review of our submission. We value your feedback and have taken each comment seriously. We are making significant revisions to address the concerns raised.\n\n1. Writing and presentation quality, diagram quality:\n\nResponse: We acknowledge the feedback regarding the clarity and presentation of our work. To address this:\n\nWe've revisited the problem definition and provided a rigorous mathematical formulation as the following. \n\nwe aim to build a model to predict the classification label $y_i \\in \\mathcal{Y}$, where $\\mathcal{Y}$ denotes the label space. A model is a composite function $m(\\mathbf{x}) = f \\circ b_{t_0}(\\mathbf{x})$, where $b_{t_0}(\\mathbf{x})$ is referred to as the backbone model and $f(\\mathbf{z})$ estimates labels of $\\mathbf{x}$ based on the outputs of $b_{t_0}(\\cdot)$. At time $t_0$, a global model $g_{t_0}(\\mathbf{x}) = f_g \\circ b_{t_0}(\\mathbf{x})$ is trained on a public labeled dataset $\\mathcal{D}_g = \\{(\\mathbf{x}_i, y_i)\\}_i^N, i=1$ and its distribution is denoted by $p_g(Y|X)$.  The globally trained model is deployed on a set of local devices. On each of such local devices $k$, there is a local dataset $\\mathcal{D}_k = \\{(\\mathbf{x}_j, y_j)\\}_j^{N_k}, j=1$ and $p_k(Y|X)$ characterizes the corresponding local distribution.  On each device $k$, a local model $m(\\mathbf{x}) = f \\circ {b_t}_0(\\mathbf{x})$, is created by fine tuning the global model on the local dataset without tuning the parameters of ${b_t}_0(\\cdot)$. Furthermore, the distributions are different from each other such that $p_k(Y|X) \\neq p_j(Y|X)$ if $k \\neq j$, including the distribution of the public dataset.   As a result, the optimal function $f_j(\\cdot) \\neq f_k(\\cdot)$ if $j \\neq k$. At time $t_i$, there is a significant update of the global model due to changes of the public dataset or introduction of new techniques. Therefore, the updated global model can use a totally different architecture than previous models, as long as the outputs of its backbone model are vector sequences. When deploying the new global model to local devices, we aim to minimize the use of computing resources and obtain a new modelwith optimal performance on each local device. Herein, the parameters of the backbone model ${b_t}_i(\\cdot)$ are frozen during deployment. Due to privacy concerns, the local data on each local device is not used to train global models collaboratively.\n\n\nAt time $t_i$, when a local device receives a new global model, it replaces its current backbone with the new one $b_{t_i}(\\cdot)$ and adaptively learns a new $f_{k,t_i}(\\cdot)$ by using the logits derived from the previous model. The backbone model is usually referred to as the text encoder. The function $f_{k,t_i}(\\cdot)$ essentially maps encoder outputs $\\mathbf{h}$ to task-specific labels. For news recommendation, $f_{k,t_i}(\\mathbf{h}) = \\arg\\max_y \\log p(Y = y | \\mathbf{h})$ with $y \\in \\{0, 1\\}$. Herein,  $\\mathbf{h} = (\\mathbf{u}, \\mathbf{v})$, where $\\mathbf{u}$ is the user representation and $\\mathbf{v}$ denotes the representation of the current news. For a given news $c$, $p(Y = y | \\mathbf{h}) = \\sigma (z_c)$ and $z_c = \\mathbf{u}^{\\text{T}} \\Lambda_k \\mathbf{v}_c$, where the diagonal matrix $\\Lambda_k$ captures user preference from the local distribution. Then $z_c$ is referred to as the logit of the news $c$. The rationale behind this is that the model intends to recommend the news similar to what a user read in the past. Hate speech detection is also a binary classification problem. The logit $z_c = \\mathbf{w_t}_i^{\\text{T}} \\Lambda_k \\mathbf{h}_c$, where $\\mathbf{h}_c$ denotes the output from the backbone text encoder for a text $c$ and $\\mathbf{w}$ is a weight vector shared between users to characterize the importance of each feature. The diagonal matrix $\\Lambda_k$ encodes user-specific hate speech patterns.\n\n\nInstead of using all instances from a local dataset, we build a fine-tuning dataset $D_k^f$ by sampling the most representative examples from a local distribution by applying correlation clustering. The details of correlation clustering are described in Line 263 - 303.\n\n\nDuring deployment, our key hypothesis is that the logits are similar before and after replacing the backbone model. Therefore, we save the logits $\\hat{z}_i$ of the model before replacement for all instances in the fine-tuning dataset $D_k^f$, in order to compare them with the logits from the new model. As $\\Lambda_k = \\text{diag}(\\theta)$ is a diagonal matrix for both applications, we can reformulate the learning problem after replacement as a linear regression problem by setting $z_i = \\theta^{\\text{T}}\\mathbf{e}_i$ for all instances in $\\mathcal{D}^f_k$. Because the backbone model is frozen, we need to make only one forward propagation for each instance in $\\mathcal{D}^f_k$ to obtain the corresponding hidden representations.\nThe learned diagonal matrix should capture user-specific information for respective applications. Because the backbone model is frozen, we need to make only one forward propagation for each instance in $\\mathcal{D}^f_k$ to obtain the corresponding hidden representations. As a result, the adaptation process after replacement is a ridge regression problem below.\n\n$$\nJ(\\theta) = \\frac{1}{|\\mathcal{D}_k^f|} \\sum_{i} (z_i - \\hat{z}_i)^2 + \\lambda \\| \\theta \\|_2^2\n$$\n\nThank you for your feedback regarding Figure 1. In response to your comments, we have updated the figure to offer a more transparent representation of the system workflow and the associated problem. Specifically:\n\nWe've detailed the model replacement process by using a more concrete example and illustrating the transition from GloVe to BERT explicitly. The workflow now incorporates a visual representation of our method. We've added additional steps highlighting how correlation clustering is employed to gather data and how the optimization of the personal linear matrix takes place. Subsequent to these steps, the process of personal matrix fine-tuning has been visualized, demonstrating how it facilitates adaptation to the personalized new backbone model. We've enriched the figure with supplementary documentation and descriptions to ensure every phase is straightforward and intuitive for the readers.\n\n\nWe hope these revisions address your concerns and make the figure more informative and clear.\n\n\n2.memory report and flops for correlation clustering?\n\n\n\n\nModel Memory: We implement our method using TensorFlow, thus the GPU memory for models is pre-allocated before training and inference. The allocation of GPU memory was 2900 MB in total. %Discerning a direct difference in memory usage becomes challenging due to this preloading mechanism. However we can estimate the memory usage for each stage given the existing condition mathematically. \nBecause we cannot direct observe actual GPU memory usage at each stage, we estimate it based on the expected consumption. For bert-base-model with 768 dimensions with token length of 768 with batch size of 150 examples\nWe can calculate the activation memory for forward pass as $Forward Pass Memory=150×25×768×4×12 \\approx 1.1G$ and for backward pass we need extra memory to store the gradient and weight. For full parameter tuning it would be $Backward Pass Memory (Full Model)≈Forward Pass Memory \\approx 1.1G$. If we only tune last layer in gradient checkpoint it would be $Backward Pass Memory (Top Layer)=150×25×768×4\n\\approx 92M$. There will be overall around 400M for static parameters of models. \n\nSo, for both forward and full backward pass, you'd need roughly 2.2 GB + 440 MB = ~2.64 GB. For forward pass and top layer backprop, you'd need roughly 1.1 GB + 92 MB + 440 MB = ~1.63 GB. As this is case of training on service, with much less batch size of personal training on deivce for example 3 examples, the computation can be largely reduced and fit to on-device scinerio.\n\n\n\nFlops for the Correlation Clustering Method (Kwikbucks): Present techniques for monitoring Flops are predominantly oriented towards deep learning methods, making a direct measurement for our correlation clustering method impractical. Nevertheless, we can offer insights into the execution time of the correlation clustering algorithm. Each clustering iteration is completed in approximately 2 seconds. This duration is notably shorter than what's needed for either a forward pass or a backward pass. \nWe trust that this elucidation aptly addresses your concerns and presents the information more succinctly.\n\n\n\n3. Related work on federated learning:\n\nResponse:\n\nFor the version we submitted, we opted to omit this section due to space constraints and for the following reasons:\n\nFederated learning aims to collaboratively learn models in a distributed manner. However, our problem setting does not consider learning a model collaboratively by using the data from local devices. Instead, we consider only the deployment of a globally trained model to local devices without using any model updates or local data from them. Moreover, we consider the adaptation process by replacing one component of the local models, not all of them, while most federated learning applications consider shared model architectures between servers and local devices. Our setting is also different from the FL works that us smaller versions of models on local devices. We do not consider model pruning or model compression but only want to accelerate the deployment process using as few computing resources as possible.\n\nPersonalized parameter-efficient tuning strategies, such as the adapter mechanism or partial parameter updates as mentioned by scholars[2,4,5], are not directly applicable to our method, because they assume the same or similar architecture for the backbone models. Their methods will fail if the dimension of the backbone model outputs changes vastly. For instance, transitioning from an old model with a dimension of 512 to a new one with 768. The dimension mismatch prevents from a direct integration of the old model's upper layers into the new backbone model.\n\n\n\nAs in our first version of paper, we do include the personalized federated learning in our related work the original paper is attached here:\n\n\"Wang[1] evaluated on-device personalization under a federated learning setting where personal data is preserved on local devices. Their experiment on virtual keyboard next-word prediction for smartphones indicated that a single user can benefit from personalization.\nP-meta[2] proposed a memory and computation efficient method for model adaptation by identifying a certain fraction of parameters that ensure rapid generalization and dynamically updating parameters to facilitate unseen tasks. \nZhang[3] relaxed the constraint of model homogeneity and aggregated the soft prediction of heterogeneous local devices using a knowledge group coefficient matrix to keep the global model updated with personalized context.\nHowever, these works did not extend to language tasks, which heavily rely on pretrained representation from the backbone model, and all assume that the personalized model is static in terms of model architecture and globally learned knowledge. In our work, we aim to investigate the preservation of personal information with model replacement while minimizing the fine-tuning cost.\" \nHowever, these studies didn't venture into language-centric tasks that are heavily dependent on pretrained representations from primary models. They also operated under the assumption that the personalized model remains constant in terms of architecture and globally acquired knowledge.\"\n\n\nConsidering the feedback, we're contemplating the addition of an in-depth section on parameter-efficient personalized federated learning, either in the related work or appendix. This will help to distinctly highlight the differences and underscore our research objective.\n\nWe hope this revision provides clearer insights into our decision-making process and the distinctive aspects of our research.\nreference:\n\n[1]Wang, K., Mathews, R., Kiddon, C., Eichner, H., Beaufays, F., & Ramage, D. (2019). Federated evaluation of on-device personalization. arXiv preprint arXiv:1910.10252.\n\n[2]Qu, Z., Zhou, Z., Tong, Y., & Thiele, L. (2022). p-Meta: Towards On-device Deep Model Adaptation. In 28th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2022) (pp. 1441-1451). Association for Computing Machinery.\n\n[3]Zhang, J., Guo, S., Ma, X., Wang, H., Xu, W., & Wu, F. (2021). Parameterized Knowledge Transfer for Personalized Federated Learning. Advances in Neural Information Processing Systems, 34, 10092-10104. \n\n[4] Pillutla, K., Malik, K., Mohamed, A. R., Rabbat, M., Sanjabi, M., & Xiao, L. (2022, June). Federated learning with partial model personalization. In International Conference on Machine Learning (pp. 17716-17758). PMLR.\n\n[5] Xie, C., Huang, D. A., Chu, W., Xu, D., Xiao, C., Li, B., & Anandkumar, A. (2023). PerAda: Parameter-Efficient and Generalizable Federated Learning Personalization with Guarantees. arXiv preprint arXiv:2302.06637.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The topic of this paper studied is important and interesting. \n2. The proposed method is lightweight and can be easily implemented on-device, which is important for real-world applications.\n3. The method achieves significant performance improvements over popular transfer learning methods.", "reasons_to_reject": "1. The writing and presentation quality are not good enough to be accepted in EMNLP. First, the problem definition is not clear. Please strictly define the problem to be solved in mathematical terms, including the data in global and local respectively and the goal of the paper. Please summarize the whole training flow. Please add more description about Figure 1. It is very difficult for readers to guess the meaning of the components in the figure.\n2. The paper proposes methods for on-device applications. Therefore, it is necessary to report the memory used for each method. And for back-propagation computation reduction, it is better to show the actual running time of each method. Just reporting FLOPs is not convincing enough. Besides, is the clustering step included in FLOPs computation?\n3. The paper lacks related work about federated learning. The parameter-efficient personalized federated learning is very similar to the setting in this paper. However, the paper does not show related work about parameter-efficient personalized federated learning. And the paper should add one baseline [1] or [2].\n\n[1] Pillutla, Krishna, et al. \"Federated learning with partial model personalization.\" International Conference on Machine Learning. PMLR, 2022.\n[2] Xie, Chulin, et al. \"PerAda: Parameter-Efficient and Generalizable Federated Learning Personalization with Guarantees.\" arXiv preprint arXiv:2302.06637 (2023).", "questions_to_authors": "refer to reasons to reject", "paper_topic_and_main_contributions": "The paper studies the problem of language model backbone replacement for personalized downstream tasks in a non-stationary on-device scenario and proposes a lightweight tuning method to solve it. The proposed method achieves over 1000x computation reduction in Flops for back-propagation and brings significant performance boost compared with popular transfer learning methods.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "refer to reasons to reject", "Typos_Grammar_Style_and_Presentation_Improvements": "004: company->companies"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is commended for addressing an interesting and important problem of language model backbone replacement for personalized downstream tasks in an on-device, non-stationary scenario. The clear separation between cloud and edge is acknowledged as crucial for the efficiency and privacy of ML models on low-powered devices, making this a relevant and timely topic. Reviewers appreciate that the proposed method achieves over 1000x computation reduction in FLOPs for back-propagation while outperforming comparable transfer learning methods. Additionally, the paper is recognized for conducting extensive experiments with various models and datasets, as well as including an ablation study. 

However, reviewers raise concerns about the clarity and presentation of the paper. They note that the problem definition is not clear, and the paper lacks a clear, concise summary of the training procedure. There are concerns about the lack of certain experimental details. Reviewers suggest reporting memory usage for each method and providing actual running times in addition to FLOPs computation. I believe the inclusion of these details would strengthen the paper's experimental evidence.

Overall, the paper addresses an important and interesting problem with significant computational and performance improvements, but it needs to address concerns related to clarity, presentation and experimental details. Addressing these issues would enhance the paper's quality.
Decision: Accept-Findings