Private and Efficient Meta-Learning with Low Rank and Sparse decomposition
Authors: Soumyabrata Pal
Prateek Varshney
Gagan Madan
Abhradeep Guha Thakurta
Gaurav Aggarwal
Pradeep Shenoy
Gaurav Srivastava
Prateek Jain
Venue: Submitted to ICLR 2023 (2023)
Keywords: Meta-learning
Low-rank
Sparse
Privacy

Abstract: Meta-learning is critical for a variety of practical ML systems -- like personalized recommendations systems -- that are  required to generalize to new tasks despite a small number of task-specific training points. Existing meta-learning techniques use two complementary approaches of either learning a low-dimensional representation of points for all tasks, or task-specific fine-tuning of a global model trained using all the tasks. In this work, we propose a novel meta-learning framework that combines both the techniques to enable handling of a large number of data-starved tasks. Our framework models network weights as a sum of low-rank  and sparse matrices. This allows us to capture information from multiple domains together in the low-rank part while still allowing task specific personalization using the sparse part.  We instantiate and study the framework in the linear setting, where the problem reduces to that of estimating the sum of a rank-$r$ and a $k$-column sparse matrix using a small number of linear measurements. We propose an alternating minimization method with hard thresholding -- AMHT-LRS -- to learn the low-rank and sparse part effectively and efficiently. For the realizable, Gaussian data setting, we show that AMHT-LRS indeed solves the problem efficiently with nearly optimal samples.  We extend  AMHT-LRS  to ensure that it preserves privacy of each individual user in the dataset, while still ensuring strong  generalization with nearly optimal number of samples. Finally, on multiple datasets, we demonstrate that the framework allows personalized models to obtain superior performance in the data-scarce regime.

Under review as a conference paper at ICLR 2023
PRIVATE AND EFFICIENT META-LEARNING WITH Low RANK AND SPARSE DECOMPOSITION
Anonymous authors Paper under double-blind review
ABSTRACT
Meta-learning is critical for a variety of practical ML systems - like personalized recommendations systems that are required generalize to new tasks despite small number of task-specific training points. Existing meta-learning techniques use complementary approaches of either learning low-dimensional representation of points for all tasks, or task-specific fine-tuning of a global model trained using all the tasks. In this work, we propose novel meta-learning framework that combines both the techniques enable handling of large number of data-starved tasks. Our framework models network weights as a sum of low-rank and sparse matrices. This allows us to capture information from multiple domains together in the low-rank part while still allowing task specific personalization using the sparse part. We instantiate and study the framework in the linear setting, where the problem reduces to that of estimating the sum of a rank-r and a k-column sparse matrix using a small number of linear measurements. We propose an alternating minimization method with hard thresholding - AMHT--LRS------- to learn the low-rank and sparse part effectively and efficiently. For the realizable, Gaussian data setting, show that AMHT-LRS indeed solves the problem efficiently with nearly optimal samples. We extend AMHT-LRS ensure that it preserves privacy of each individual user the dataset, while still ensuring strong generalization with nearly optimal number of samples. Finally, on multiple datasets, we demonstrate that the framework allows personalized models to obtain superior performance in the data-scarce regime.
INTRODUCTION
Typical real world settings like multi user/enterprise personalization have long tail of tasks with a small amount of training data. Meta-learning addresses the problem by learning "learner" that extracts key infornattion/eerrssnttiion from large number of training tasks, and can be applied to new tasks despite limited number of task specific training data points.
Most existing meta-learning approaches can be categorized as: 1) Neighborhood Models: these methods learn global model, which is then "fine-tuned" to specific tasks (Guo et al., 2020 Howard & Ruder 2018 Zaken et al., 202T). ). 2) Representation Learning: these methods learn a low-dimensional representation of points which can be used to train task- linear learners (Javed & White 2019: Raghu et al. 2019) Lee et al.) 2019 Bertinetto et al., 2018 Hu et al., [021] In particular, task-specific fine-tune -unnttnnng has demonstrated exceptional results across many natural language tasks (Devlin et al.) 2018 Liu et al. 2019 Yang et al. 2019 Lan et al., 2019). However, such fine-tuned models update all parameters,
each fine-tuned total parameter footprint is same as the original model. This implies that fine-tuning large models -like say standard BERT model (Devlin et al. 2018) with about 110M parameters for thousands or millions of tasks would be quite challenging even from storage point of view. One potential approach to handle the large number of parameters is fine-tune only the last layer. but empirical findings suggest that such solutions can be significantly less accurate than fine-tuning the entire model (Chen et al.) [020aa] Salman et al. (020). Moreover, representation learning based approaches apply strict restrictions like each task's 's parameters have to be in low-dimensional subspace which tend to affect performance in general (Sec.
In this work, we propose and study the LRS framework that combines both the above mentioned complementary approaches. That is, LRS restricts the model parameters (4) for the 7th task as
( = 11 W B(), where first term denotes applying low-dimensional linear operator

---

Under review as a conference paper at ICLR 2023
W(', while B is restricted to be sparse. That is the first term is based representing parameters of each task in a low dimensional subspace while the second term allows task siss-specffic fine-tuning but only of a few parameters. Note that methods that only allows fine-tuning of batch-norm statistics, or of the final few layers can also be thought of as "sparse" fine-tuning but with a fixed set of parameters. In contrast, allow the tunable parameters be selected from any part of the network.
The framework allows collaboration among different tasks so as to learn accurate model despite lack of data per task. Similarly, presence of sparse part allows task-specific personalization of arbitrary but a small set of weights. Finally. the framework allows tasks/users to either contribute to a central model using privacy preserving bill-board models (see Section 2.2 or also allows flexibility where certain tasks only learns their parameters locally and do not contribute to the central model.
While the framework applies more generally, to make the exposition easy follow, we instantiate in the case of linear models. In particular, suppose the goal is to learn linear model for the 2th task that is parameterized by 010), i.e., expected prediction for the data point x E Rd is given by (x, 0(3).
g(0) is modeled as g(+) == U'w+(s) + b*(() for all tasks 1 S i S 1, where U E RRdx (where r < d) a tall orthonormal matrix that captures task representation and is shared across all the tasks.
Note that estimation of the low-rank and sparse part is similar to the robust-PCA (Nttrapllli et al.) 2014) that is widely studied in the structured matrix estimation literature. However, in that setting, the matrices are fully observed and the goal is to separate out the low-rank and the sparse part. In comparison, in the framework proposed in this work, we only get few linear measurements of the underlying matrix due to which the estimation problem is significantly more challenging, We address this challenge of estimating U along with (+),, b(()) (for each task) using a simple alternating minimization style iterative technique. Our method AMHT-LRS- alternatingly estimates the global parameters U as well as the task-specific parameters (+) and b-(1) independently for each task. To ensure sparsity of b we use an iterative hard-thresholding style estimator (Jain et al. 2014).
In general, even estimating U is NP-hard problem (Thekumparampil et al. 2021). One of the main contributions of the paper is novel analysis that shows that AMHT-LRS indeed efficiently converges to the optimal solution in the realizable setting assuming the data is generated from Gaussian distribution. Formally, consider f, d-dimensional linear regression tasks (indexed by i E [t]
with m samples {(x))) 3)
(6) (,) )))) being provided each of them such that
(1) y x ( Lz) and
(xx U''' + +b+()) + 2/0 for all i [t], E [m].
(1)
where 2000 mid N NNO, o?). Below, we state our main result informally in the noiseless setting (a 0):
Theorem (Informal, Noiseless setting), Suppose we are given m t samples from t linear regression tasks of dimension d as equation [ Goal is to learn a new regression task's 's parameters using m samples, iee, learn the shared rank-r parameter matrix U along with task-specific w .b.. Then, AMHT-LRS with total recover all the parameters exactly and in time nearly linear in m tt
((kdr*) samples and m = -(mmxxks samples per task can
That is, AMHT-LRS is able to learn the underlying model exactly as long as the total number of tasks is large enough, and per task samples scale only linearly in sparsity of b and cubically in the rank of U Assuming k < d, additional parameter overhead per task is small, allowing efficient deployment of such models in production. Finally, using the billboard model of (c, 8) differential privacy (DP) (Jain et al.) 2021 Chien et al. [2021; Kearns et al., (2014), we can extend AMHT-LRS to preserve privacy of each individual. Furthermore, for similar sample complexity as in the above theorem albeit with slightly worse dependence on we can guarantee strong generalization error up to standard error term due to privacy.
Summary of our Contributions:
We propose a theoretical framework for combining the meta-learning approaches of representation learning and neighborhood model. Our model non-trivially generalizes guarantees of Thekumpara mpil et al. (2021): Tripuraneni et al. (2021) ; Boursier et al. (2022) to the setting where the parameter matrix allows low rank plus sparse decomposition.
We propose an efficient method AMHT-LRS for the above problem and provide rigorous total sample complexity and par-task sample complexity bounds that nearly optimal (Theorem []]
2

---

Under review as a conference paper at ICLR 2023
We provide DP variant of AMHT-LRS that guarantees user level privacy. At a high-level show that under (c, 6D-DP, one can obtain a generalization error as non non-private version but with an additional eπror due to privacy budget (see Theorem (3).
We demonstrate experiments on synthetic data and Movielens dataset using linear models (Sec. and toy neural nets (Appendix A]; apart from showing the advantage of our framework, they also show limitations of only using representation learning or a single model among other baselines.
Technical Challenges (Linear Models): Denote (W*)*= [[
w w*()) E R and B [b+(1) b+ E Rdxt; hence the matrix of optimal regressors is given by ( +B"
B There several technically novel steps that are required for recovering the model parameters in ( that combine both the representation and neighborhood models. In the general setting, even the task- specific r-dimensional regression coefficients (w heett are unknown and therefore, we are faced with the additional challenge of learning [w*(0) hee(s] jointly along with the shared representation matrix U and the task-specific sparse parameter vectors (b*(t) Jee[s) Note that in the AM framework proposed in Thekumparanpil et al. (2021), the authors consider only the representation model and not the neighborhood model (which is simpler case of LRS setting). Similarly, in Netrapalli et al. (0014), the authors design an AM algorithm for the problem of reconstructing the low rank and sparse components of matrix if the matrix provided as an input. However, in our setting, we only observe linear measurements of the individual columns of the parameter matrix. Therefore, informally speaking, our analysis is faced with the key challenge of combining both sets of complementary techniques in Thekumparamp et al. (2021); Netrapalli et al. (2014). This leads to the analysis of several crucial steps in each iteration: I) We track the incoherence of several intermediate matrices corresponding to the latest estimates W( U(() of W U 2) We also track the L2,ss norm of the matrix (I - U*(U*) T)U()) to make progress on learning B*, In particular, the second step is the most technically involved component of our analysis.
Organization: In Sec. 2 we introduce the general low rank+sparse (LRS) framework. In Sec. 2.1 we provide theoretical guarantees for the canonical linear model in the LRS framework and provide differentially private version in Sec. 2.2 We analyze a special setting in Appendix B as warm-up (Rmk. 3 while detailed proofs are delegated to Appendix [ In Sec. and Appendix A provide experimental results on synthetic and real datasets. In Appendix E we discuss how to obtain a good initialization for our methods in the realizable setting via Method of Moments.
2 LRS FRAMEWORK FOR MECX--LEARNIN//PPROOVAAZZZO
[m]] to denotes the set m.). For a matrix A, A; denotes 7th row of A. For a vector x, z; denotes 7th element of x. We sometimes use x, to denote an indexed vector; in this case Tj denotes the 7th element of x,. 11-11, denotes euclidean norm of vector and the operator norm of a matrix. 11=11 I1-11, will denote the lov and by norms of a vector respectively. |1-112..e |1-I1p will be used denote the L2,00 and Frobenius norm of a matrix respectively. For sparse vector v e Rd we define the support supp(v) S [d] to be set of indices such that v * 0 for all i E supp(v) and 2 0 otherwise. We use I to denote the identity matrix. Ö(.) notation subsumes logarithmic factors.
Notations:
(6) E IRm denotes the matrix of covariates for the ,th task such that X = (x Similarly, write (+), z 1 to denote the task-specific response vector and noise vector respectively.
Let x E Rdd be the input point, and let ÿ = f(x; +) be the predicted label using a DNN f with parameters θ Let there be tasks/domains. Then, the goal is to personalize O to ( for each task i (t) such that a) (i) does not over-fit despite a small number of data-points labelled as task i, b) (o 1115 i < t) can be stored and inferred fast even for large f, say for t 2 1M, c) the framework allows enough flexibility ensure that different tasks/domains//ssrss can contribute their data at different levels of privacy risks. Our method LRS attempts to address all the three requirements using simple towsrank-eparre approach: we model each (+) as ()) ;= 21 ( + B(O, where 2/ (i) denotes linear operation on (i) in r-dimensional basis specified by U (can be very large for e.g. standard BERT with ~ 110M parameters), and ( is k-sparse matrix. That is, represent ()) as a combination of a small number of parameter matrices represented by Ut along with a sparse set of weights that can be fine-tuned arbitrarily for given task.
3

---

Under review as a conference paper at ICLR 2023
Note that due to low-dimensional and sparse representation, LRS should require relatively small number of points per task. In the next section, we formally prove this claim for a simple linear setting with Gaussian data. Furthermore, for each task the additional number of parameters is relatively small - assuming r and k are small - which implies that memory cost can be controlled. The latency cost also remains same as the baseline model, assuming we can explicitly compute ( on the fly. Finally, the model allows tasks//omains/users to contribute data to learn 24 with differential privacy (see Section 2.2). but it also admits domains/tasks who are not inclined to share data, and can just fine-tune their model privately by learning ( and ( in isolation.
Comparison with Hu et al. (202T): LORA (Low Rank Adaptation of Large Language Models) was proposed by Hu et al. (2021) for meta-learning with large number of tasks at scale. Although the authors demonstrate promising experimental results, LORA only allows a central model (in low dimensional manifold) and does not incorporate sparse fine-tuning. Hence, LORA becomes ineffective when the output dimension is small (say 1). The said limitation of LORA has been demonstrated in detailed experiments on the MovieeLens 1M dataset (a standard recommendation dataset) in Sec. Moreover, the low rank fine-tuning (as proposed in LORA) is limited when output dimension is small. Finally, LORA does not have any theoretical guarantees even in simple settings.
Comparison with representation learning style models: A recent line of work (Thekumparampil iet al. 2021 Du et al. 2020 Tripuraneni et al. 2021 Boursier et al. 2022 Jain et al. (202]] proposes a similar model LRS but specific to representation learning. These papers only consider a low-rank representation of the task parameters but no sparse fine-tuning Moreover, the proposed algorithms in these papers have not been explored at scale and mostly been applied vanilla linear models (without considering privacy constraints extension to more complex models). Even from a theoretical point of view, these methods do not apply in our case due to the additional non-convex sparsity constraint.
Comparison with Prompt- and Batch-norm Fine-tuning: Another popular approach for personalization is to use prompt-based batch-norm based fine-tuning (Wang et al. 2022 Liu et al. 2021 Lester et al. 2021): this usually involves a task-based feature embedding concatenated with the covaritte. Note that in linear model, such approach will only lead to additional scalar bias which can be easily modeled in our framework; thus our framework is richer and more expressive with smaller number of parameters. We have compared against such techniques in our experiments and demonstrated their limitations in both real and synthetic datasets (see Sec.
pedell-pesoonalization is key application of meta-learning, where we wish to have a personalized model for each user i.e. each user represents a task. Due to sensitivity of user-data, we would want to preserve privacy of each user for which we use user-level (e, 6)- as the privacy notion (see Definition Tp. In this setting, each user i E [t] holds set of data samples DD0 fx و S m). Furthermore, users interact via a central algorithm that maintains the common representation matrix lI which is guaranteed to be DP w.r.t. all the data samples of any single user. The central algorithm publishes the current to all the users (a.k.a. on a billboard) and obtains further updates from the users. It has been shown in prior works (Jain et al. 2021 Chien et al. 202T] Thakkar et al., 2019) that such a billboard mechanism allows for significantly more accurate privacy preserving methods while ensuring user-level privacy. In particular, it allows learning of 24 effcctively, while each user can keep a part of the model which is personal to them, e.g..
Private Meta-learning:
the (+)) B (s in our context. See (Jain al., 2021 Section 3) for more details about billboard model in the personalization setting. Traditionally, such model of private computation is typically called the billboard model of DP, which in turn is a subclass of joint DP (Kearns et al. 2014).
Definition 1. Differential Privacy Dwork et al. (2006b Bun & Steinke (2016) A randomized algorithu A is (e. 8) diffferentiall private for any pair of data sets D and D' that differ in one user (i.e.. D&D' = 1), and for all S in the output range of A. we have
Pr(A(D) E S) e .PiAA(D) +
where probability is over the randonness of A. Similarly, an algorithm A is p-zero Concentrated DP (zCDP) if Da (A(D))IA(D') I ap,, where Da is the Rényi divergence of order a.
In Definition I when define the notion of neighborhood, we define it w.r.t. the addition (removal) of a single user (i.c.. additional removal of all the data samples D, for any user i E [t]). In the literature Dwork & Roth 2014), the definition is referred to user-level DP.
4

---

Under review as a conference paper at ICLR 2023
2.1 LINEAR LRS: ALGORITHM AND ANALYSIS
In this section, we describe our LRS framework for the linear setting, provide an efficient algorithm for parameter estimation, and provide rigorous analysis under realizable setting with Gaussian data. We then extend our framework, algorithm and analysis to allow user-level differential privacy. Consider the linear LRS model introduced in Section I where we have d-dimensional linear (8)
regression tasks (indexed by i E [t]) with m samples {(x) Mj } being provided to cach of
them such that the tt sample for the jth task (xx" yy v/ is generated independently according to eq. I So the problem reduces to that of designing statistically and computationally efficient algorithms to estimate the common representation learning parameter U* as well as task-specific parameters (w hejj) heja), (b+(9) be(t)- The ERM for this model assuming squared loss is given by:
(LRS)
minimize L(U, W,
Σ
Σ
Mj
(x)
Uw ( + )b))))²
iE(t] [m(m
s.L. UU--I
||6(0
S k Vi
[t] and BBllo SQViE E [d].
(2)
where U E Rdxr W (w() (2)
w( ("IT E R stores the task-specific coefficients, and B [b()) b b(t] ddxt stores the task-specific sparse vectors for fine-tuning. Note that
LRS is non-convex due to: a) bilinearity of U, W, b) non-convexity of to norm constraint.
We propose AMHT-LRS that handles the non-convexity in the objective and the constrained set by carefully combining alternating minimization for U, w and b with hard thresholding to ensure sparsity of b. Let HT Red R Rd be hard thresholding function that takes vector B and a parameter A as input and returns a vector v' E Rd such that of = U; if |us| > and 0 otherwise. Let (+((-1), (w(i.--1) Jeet and {b (b(,)-1)))s Jseja be the latest iterates at the beginning of the et iteration. First, for each task E [t], given estimates (e-1-1) w(1--1), can update (t.e)
by solving the following problem:
argmin
X ( (U+ + +b) y(0 such that |bbll S k.
(3)
While the problem is non-convex, reduces to iterative hard thresholding. In particular, we use Algorithm for the 7th task where in each iteration, we run a gradient descent step on the parameter vector estimate b (of by()) and subsequently apply HT(:; A) function where A 0 is set appropriately. Next, given estimates U+((-1), b(0,), we update w((,2) by solving the following task-specific optimization problem
still apply a projected gradient descent algorithm which
argmin (i) (U+(E-)) + b(:0)) y ||| for each i E [t].
(4)
Subsequently, given the updated estimates of the task-specific parameters {w (10 helt and (b0.0) heen, we update (+) (estimate of shared representation matrix) using:
Σ | (Uw(())))) +b0.0) - y(6)
argminuER4xx KE(A)
followed by a QR decomposition of the solution. Note that the above two problems 4 and (5) can be solved using standard least squares regression methods. Finally, we must ensure independence of the estimates (which are random variables themselves) from the data that used a particular update. We can ensure such independence by using a fresh batch of samples in every iteration.
Analysis: As prior works, we are interested the few-shot learning regime when there are only a few samples per task. From information theoretic viewpoint, we expect the number of samples per task to scale linearly with the sparsity & and rank r and logarithmically with the dimension d. On the other hand, U has dr parameters and therefore, it is expected that the total number of samples across all tasks scales linearly with dr which implies we would want the number of tasks t to scale linearly with dimension d. Note that if the sparse vectors (b+()) 1 have the same support (or a high overlap between the supports), then the model parameters might not be uniquely identifiable. This is because, in that case, the matrix B can be represented as a low-rank matrix. To establish identifiability of U and sparse vectors by make the following assumption:
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 AMHT-LRS
Require: Data {(x) (6)
E R)} for all i [t], column sparsity & of B, S B, max; [ b lleo S y(0), Parameter e 0.
E , (6)
A U*)
F
1: for l = 1,2,... do
2: Set (()) = Q flog
3:
1,2,.... do for b(i() t OptimizeSparseVector/
4:
,())), ( ( Vk , O( (B, ((111), T= T(0) for suitable constants c4, C3 > 0.
w((l-)
w(1.0) (XX (X -x(0))) x(O((.)))
-1
5:
6:
end for
7: Set A Eiely)
x vec vec (A vvec V)) and U+(() QR(UIO)
xj (x (x) and V Eiery(X(0)) (5(0-
( (wli.0)). Compute U =
8: y00) (cg)*=leB cB for suitable constant Cy < 1.
9: end for
10: Return w(1) U+(1) and (b(.0) Je(0--
Assumption (A1). Consider the matrix B E TRaxt whose ith column is the vector (0 Then each row of B is Ç-sparse i.e. | S for all [d], and each column is k-sparse.
Note that the orthonormal matrix U cannot have extremely sparse columns otherwise it would be information theoretically impossible to separate columns of U* from b*. Moreover, similar to Tripuraneni et al. (202I), we need to ensure that each task contributes to learning the underlying representation U These properties can be ensured by the standard incoherence assumptions Tripuraneni al. (202 Collins et al. (2021): Netrapalli et al. (2014) and therefore, have
Assumption 2 (A2). Let >i and X be the largest and smallest eigenvalues of the task diversity matrix (r/t)(W")TWw W E Rexr, We assume that W !R and the representation matrix U* Rdd
are iicohheerr i.e. || S / and 110*112,0x V
Theorem 1. Consider the LRS problem equation 2 with linear regression tasks and samples obtained by equation 7 Let model parameters satisfy assumptions A1, A2. Let the row sparsity of B* satisfy Ç 0 ((((2(11))
X and let k = =0 d. (2) Suppose Algorithm / is initialized with
U+(0) such that (I- U ( -
and || =O(Vurr/ /r////, and is
run L = (() iterations. Then, with high probability, the outputs (+(L),((e) (b( Jiej satisfy:
Ō(L)sS Viexz" ((64-b-(9))
I O(1)sS
, [t],
(6)
VI
where S
red mr
+
y3 md;
+ v
provided the total number of samples satisfies:
2 02p3
m = δ ((k +
mt = 56 dp (1(4))(()))
+ +L+
For new task, modified AMHT-LRS (Alg. in Appendix D
has the following generalization bound:
C(U, L(U*, w", w²bb")===-"""" = =
m ))))
Note that the per-task sample complexity of our method roughly scales as m 3r+3+ k). which is information theoretically optimal in is and is roughly 7 factor larger. Total sample complexity scales as mt = kdr4, which is roughly kr multiplicative factor larger than the information theoretic bound. Note that typically and & are considered to be small, the additional factors small, but

---

Under review as a conference paper at ICLR 2023
leave further investigation into obtaining tighter bounds for future work. Finally, the generalization error scales as (r + k) m which is nearly optimal. Note that, ignoring meta- learnng,,, and directly optimizing the single-task error would lead to significantly larger enror of o o2d/m.
Remark (Runtime and Memory), The run nun-tim of Algorithm 7 is dominated by the update for U+()) For each iteration L, Step has time complexity of ([dr)² + (mnt)dr))); however in practice, a gradient descent step for the update of U()) can bring down the time complexity to O(mldr). Moreover, the memory usage of Algorithm [ ((dr)) + lr²)).
Remark (Initializattonn. Note that Algorithm I has local convergence properties as described in Theorem In practice, typically we use random initialization for U+(0) However, similar to the representation learning framework in TTriurrannn et al. (2021), 1y, we can use the Method of Moments to obtain good initialization. See Appendix for more details.
Remark 3 (Special Settings). the setting where for each task, we just need to learn a single central
model for all tasks and sparse fine-tune the weights for each task i.e. w=()) 1 for all e [t] is fixed, AMHT-LRS obtains global convergence guarantees (Theorem 4 in Appendix B). Moreover, if the central model U also frozen, then the task-based sparse fine-tuning reduces to standard compressed sensing. In the realizable setting, our framework recovers the standard generalization error of a² kt/m in compressed sensing (Jain & Kar / Chapter 7].
Remark 4 (Sample complexity comparison). Note that the theoretical guarantees in representation models studied in Tripuraneni et al. (2027); Thekunparampil et al.) (202T); Collins el al. (202T) cannot capture the sparse fine-tuning in each task (with potentially arbitrary magnitude). However, since our framework combines neighborhood and representation models, our sample complexity guarantees are sub-optimal by only a factor r² (generalization error is optimal) when restricted to the special case of representation model (Thekumparampil et al. 2027)
2.2 PRIVATE LINEAR LRS: PRIVACY PRESERVING META-IFFARNING
In this section, we provide DP variant of Algorithm in the billboard model. We obtain DP for the computation of each ( by perturbing the covariance matrix A and the linear term V in the algo- rithm with Gaussian noise to ensure that the contribution of any single user is protected. We start by introducing the function clip R x R- + R that takes as input a scalar x, parameter p and returns clip(x:, p)
level
Algorithm 2 OPTIMIZE SPARSE VECTOR
Require: Data (X,y y) E R x Rn where we min- imize ||y X(v* + b*)ll2 such that |lb*| < k. Estimate v (of v*) and initialization b (of b*). It-
erations T, parameters a, a, > O and suitable
constants c > 0,0 c 1/2.. for j 1.2..... do c b m (x00)((X!))
1:
2:
+ -(()
3:
A + c1 (++
8
and t HT(c.
min {1, { We can extend the defini- tion of clip to vectors and matrices by using clip(v, p) = v min 1,, [vlla for a vee-
vk)
4:
+ 2017 + 2(a
vk
5: end for
6: Return vector b.
tor v and clip(A, = A - min (1, } for matrix A. In order ensure that Algorithm I is private, for input parameters A1, Ag, As, Aw, we first clip the covariates and re-
sponses: for all i [4,j E [m], will have xf( clip (xx") A.)A Ds clip (1)9) A2)
(x) TTb clip ((x ( As) and ((.)) +- clip (w(f.)), Aw) Now, we can modify Line
7 in Algorithm I as follows (let L be the number of iterations of Alg. D
A = mt
(i.f)
(w(i)))
1 (x ()) + +NO))
(7)
Σ
11)
y
(x) ()) ( (wi(TO)) + +N
(8)
ml
JE(m]
7

---

Under review as a conference paper at ICLR 2023
louniig bud hongt Lorven pl/
teawing
Poing Lorving 1f
676
Sperity
Speriity
Spariayy 96
(a) Overall RMSE
(b) RMSE for data-starved tasks
(c) RMSE for data surplus tasks
Figure 1: Decrease in RMSE MovicLens data for AMHT-LRS algorithm increase in fine-tunable
parameters. Note that AMHT-LRS outperforms other baselines for both data-starved and data-surplus tasks.
O, each entry of N is independently generated from N (0, m2 A1 A L. opp); similarly, each entry of N(2) is independently generated from N (0.m². (A2 Ag. -L. -L. We are ready to state our main result:
where, for some DD
Theorem 2. Algorithm (with modifications mentioned in equation 7 Znd equation & satisfies OD set the noise multiplier ODP 2 (log((111//)) + Furthermore, if log(1/8), then
2 - zCDP and correspondingly satisfies (8,5)-dij)) --ddifserrltiall privacy in the billboard model, when
ODP 2 E (olg((1/)) suffices to ensure (), diiffferrett privacy.
Next, we characterize the generalization properties of modified AMHT-LRS:
Theorem 3. Consider the LRS problem equation 2 with all parameters m,t,5 obeying the bounds
(ra) Vlog(1/0)+e p²)) Suppose we run AMHT-LRS Õ(1) iterations with A1 Ö(vd), A2
stated in Theorem 1 and furthermore,
(Step 7 in Alg. replaced with 7 and for
O(/μ²*; + (max; [bx(0]12) (12)), A3 = Ô
X
1" x
A (()))))), Then, with high probubility.
generalization error for a new task satisfies:
drog +
w), C(U", w", w*, - -
(nx2 +r r² +r2d))
2212
where S = (11 (
/rid rdd ml
+
mA:
r
+ +V/ 1)
n= //
ō 01-1111022000 (1
/ x2
maxxelt
lbb00/12
Vm
ODP)
and K- =1 AF maxiet!
[lb
V
Note that the modified AMHT-LRS ensures (e, (e) privacy without any assumptions. However Thm. 3 still has good generalization properties; moreover, the per-task sample complexity guarantee m still only needs to scale polylogarithmically with the dimension d. In other words, our algorithm can ensure good generalization along with privacy in data-starved settings as long as the number of tasks is large- scales as d3/2 /c. Similarly, generalization error for a new task has two terms: the first has standard dependence on noise o² and the second has scaling of d (et)) which is standard private linear regression and private meta-learning (Smith et al. 2017 Jain el al. 2021). Detailed proofs of our main results namely Theorems
delegated to Appendix
EMPIRICAL RESULTS
In this section, we conduct an empirical study of AMHT-LRS with the following two goals: a) demonstrate that personalization with AMHT-LRS indeed improves accuracy for tasks with a small number of points, b) for a fixed budget of parameters, AMHT-LRS is significantly more accurate than existing baselines. For simplicity, we fix the model class to be linear and consider the following baselines: 1) Single Model (Ucentrat): learns single model for all tasks, 2) Full Fine-tuning (Uinav) separate model for each task aka standard fine-tuning, 3) Representation Learning or Rep. Learning (uw): only low rank model (Thekumparampil et al. 2021) and 4) Prompt Learning T ||e): Modified covariate by concatenation with a task-embedding vector. Note that the models considered in Hu et al. (202T); Chua et al. (202T): Denevi et al. (2018) all reduce to Full fine-tuning models (that have high memory footprint) in the experimental settings that consider. Also, the
8

---

Under review as a conference paper at ICLR 2023
above approaches are not just restricted to linear models and can be extended to complex model classes such as Neural Networks (see Appendix for extension to 3 layer Neural Net architectures).
We conduct experiments two datasets: Synthetic dataset: here, for each task i E [t], we generate m = 100 samples {(x)) :yj
(6) )ij[m where ( ~ (ox, y) = (x) _u _==(()))
b+()). We select d = 150, set k; Ç. the column and row sparsity level of (b*()) Jeete] to be 10 and 5, respectively. We sample u² uniformly from the unit sphere; non-zero elements of (b*(9) Je(() and (+) are sampled i.i.d. from (0, 1) with the indices of zeros selected randomly.
The Mov 1M dataset comprises of 1M ratings of 6K users for 4K movies. Each user is associated with some demographic data namely gender, age group, and occupation in the MovicLens dataset. We partition the users into 241 disjoint clusters where each cluster represents a unique combination of the demographic data. Each user group thus represents "task" in the language of our paper. We partition the data into training and validation in the following way: for each task, we randomly choose 20% movies rated by at least one user from that task and put all ratings made by users from that task for the chosen movie into the validation set. The remaining ratings belong to the training set. Based on the ratings in the training set, we fit a matrix of rank 50 onto the ratings matrix and obtain a 50 dimensional embedding of each movie. Thus we ensure that there is no data leakage while creating the embeddings. For each task, the samples consist of (movie embedding, average rating) tuples; the response is the average rating of the movie given by users in that task. The number of samples per task varies from 22 to 3070 clearly many clusters are data starved. We use the training data to learn the different models (with some hyper-parameter tuning) mentioned earlier and use them predict the ratings in the validation data.
b. Moviel Data:
Empirical Observations on Synthetic Data: Figure shows that not only having a single model can lead to poor performance, but a fully fine-tuned model per task can also be highly inaccurate as scarcity of data per task can leading to over- over-fitting, Finally, low-rank representation learning as well as prompt tuning based techniques do not perform well due lack of modeling power. In contrast, our method is able to exactly recover the underlying parameters as also predicted by Theorem and provides orders of magnitude better RMSE.
Empirical Observations on MovieLens:
Simulsted Data Comparisons
The overall average validation RMSE for AMHT-LRS and the different baselines that we consider is shown in Fig. Ta against percentage of fine-tunable parameters used by the model. With respect to the single model as reference, in the linear rank-1 case, the representation learn- ing and the prompt learning based baselines have and 50 additional parameters per task respectively; they are unable to personalize well. In contrast, with only 10%(= 5) additional parameters per task, AMHT-LRS has smaller RMSE than fully fine-tuned model, which re- quire 241x more parameters. However, for data-starved clusters/tasks (samples < 100), we observe that fully fine-tuning approach start to overfit. In contrast, our method outperforms other baselines for both data-starved and data-surplus tasks.
Segle Modil libural
Procreneg (hoa)
Loaring
Trampt Leaning tofeeet
4PM' USSioN a))
30*
Number tasks
Figure 2: Decrease in RMSE on Synthetic data for AMHT-LRS on increase in fine-tunable parameters
4 CONCLUSIONS
We presented a powerful theoretical framework to study meta-learning, and develop novel algorithms. In particular, our framework combines representation learning and neighborhood fine-tuning based approaches for meta-learning. We proposed AMHT-LRS method, that combines alternative mini- mization - popular in representation learning - with hard thresholding based methods. We rigorously proved that AMHT-LRS is statistically and computationally efficient, and is able to generalize to new tasks with only O(r + k) samples, where r is the representation learning dimension and & is the number of fine-tune -uninnnnn weights. Finally, we extended our result to ensure that privacy of each task is preserved despite sharing information across tasks. Extending our framework to non non-rea realizablle setting and adversarial settings are critical future directions.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Luca Bertinetto, Joao Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differen- tiable closed-form solvers. arXiv preprint arXiv:1805.00 08136, 2018.
Etienne Boursier, Mikhail Konobeev, and Nicolas Flammarion. Trace norm regularization for multi-task learning with scarce data. arXiv preprint arXiv:2202.06774, 2022.
Mark Bun and Thomas Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography Conference, pp. 635-658. Springer, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020a.
Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, and Yuling Yan. Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex optimization. SIAM journal on optimization, 30(4):3098-311 2020b.
Steve Chien, Prateck Jain, Walid Krichene, Steffen Rendle, Shuang Song, Abhradeep Thakurta, and Li Zhang. Private alternating least squares: Practical private matrix completion with tighter rates. In International Conference on Machine Learning, 2021.
Kurtland Chua, Qi Lei, and Jason D Lee. How fine-tuning allows for effective meta-learning. Advances in Neural Information Processing Systems, 34:8871-8884, 2021.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared represen- tations for personalized federated learning. In International Conference on Machine Learning, pp. PMLR, 2021.
Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Learning to learn around common mean. Advances in Neural Information Processing Systems, 31, 2018.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.0880 2018.
Simon Du, Wei Hu, Sham Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002..09434, 2020.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3 9(3-4):211 4):211-407, 2014.
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, llya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology- EUROCRYPT, pp. 486-503, 2006a.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proc. the Third Conf. on Theory of Cryptography (TCC), pp. 265-284, 2006b. URL http: //0: dow..og.....oo..oooooo 61.1
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruming. arXiv preprint urXiv:20/2.07463, 2020.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv: 1801006161, 2018
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
1.Xi:22006.09685. 2021.
Prateck Jain and Purushottam Kar. Non-convex optimization for machine learning, arXiv preprint 172.7777 2017.
Prateek Jain, Ambuj Tewari, and Purushottam Kar. On iterative hard thresholding methods for high-dimensional m-estimation. Advances in neural information processing systems, 27, 2014.
10

---

Under review as a conference paper at ICLR 2023
Prateek Jain, John Rush, Adam Smith, Shuang Song, and Abhradeep Guha Thakurta. Differentially private model personalization. In Advances in Neural Information Processing Systems, pp. 29723 29735. Curran Associates, Inc., 2021.
Khurram Javed and Martha White. Meta-learning representations for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Michael Kearns, Mallesh Pai, Aaron Roth, and Jonathan Ullman. Mechanism design in large games: Incentives and privacy. In Proceedings of the 5th conference on Innovations in theoretical computer science, 2014.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1909. / /942, 2019.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10657-10665, 2019.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint (et::2104.089)) 2021.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhenghao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.33886 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. / 1692, 2019.
Ilya Mironov. Rényí differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263-275. IEEE, 2017.
Pranceth Netrapalli, Niranjan UN. Sujay Sanghavi, Animashree Anandkumar, and Prateck Jain. Non-convex robust pea. Advances in Neural Information Processing Systems, 27, 2014.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909. 09099999999999 157, 2019.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversari- ally robust imagenet models transfer better? Advances Neural Information Processing Systems, 33:3533-3545, 2020.
Adam Smith, Abhradeep Thakurta, and Jalaj Upadhyay. Is interaction necessary for distributed private learning? 2017 IEEE Symposium on Security and Privacy (SP), pp. 58-77. IEEE, 2017.
Om Thakkar, Galen Andrew, and H Brendan McMahan. Differentially private learning with adaptive clipping, arXiv preprint
2019.
Kiran K Thekumparampii, Prateek Jain, Praneeth Netrapalli, and Sewoong Oh. Statistically and computationally efficient linear meta-representatiio learning. Advances in Neural Information Processing Systems, 34, 2021.
Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear representations. In International Conference Machine Learning, pp. 14434-10443 PMLR, 2021.
Prateek Varshney, Abhradeep Thakurta, and Prateck Jain. (nearly) optimal private linear regression for sub-gaussian data via adaptive clipping. In Po-Ling Loh and Maxim Raginsky (eds.), Proceedings of Thirty Fifth Conference Learning Theory, volume 178 of Proceedings of Machine Learning Research, pp. 1126-1166. PMLR, 02-05 Jul 2022. URL preas/vil2 www./////00 http////wwwahey22...m html
www.oceediing..... ing. mlr
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy. and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 139-149, 2022.
11

---

Under review as a conference paper at ICLR 2023
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficien fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199 2021.
12


Reviews:
[{"review_id": "In2onE1L_F", "reviewer": "Reviewer_aEd6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "When facing an ML problem with multiple tasks, and few datapoints per task, meta learning is used as a mean to provide good results simultaneously on all tasks. Typical approaches to meta learning include 1) fine-tuning a large model for each task at hand, and 2) learning a low dimensional representation to be shared by all tasks and used for predictions. The first approach provides good results in practice, however, it is not always feasible due to memory and computational constraints. In contrast, the second approach is efficient, however, provides lower performance. \n\nThe authors of this paper propose a new solution that aims to combine the goods of both worlds. In particular, the authors propose learning 1) a shared low dimensional representation, and 2) a task specific fine-tuning of this low dimensional representation To this end, the authors employ a technique based on low rank and sparse decomposition of the original large weights. \n\nThe authors then analyze theoretically the proposed method, called AMHT-LRS, in terms of computational bounds, when considering a Gaussian setup and a linear projection scheme. Furthermore, they propose a small tweak to the proposed method in order to satisfy differential privacy for each of the tasks.\n\nThe authors finally empirically verify their results in a toy setup, and on a real world dataset, showing the advantages of their proposed method. ", "summary_of_the_review": "The paper provides a new approach for meta learning that combines two existing approaches. The proposed approach is intuitive and makes sense, and the authors study their method theoretically. The authors also validate their method using a toy experiment and a literature dataset. The authors discuss theoretically the privacy implications of their method, however, it is unclear whether this aspect has been validated empirically.", "strength_and_weaknesses": "Strengths:\n- the problem addressed in the paper is of direct real world implications, where companies have access to user data from multiple services, and need to learn a good model for all of them.\n- the authors considered the privacy implications of their scheme and proposed a modification to solve it.\n- the authors analyzed theoretically their approach, and provided computational bounds for their method.\n- the results achieved by the proposed methods are promising and validate the intuition behind the method.\n\nWeaknesses:\n- the authors evaluate the proposed method, but do not indicate whether the employed method is differentially private or not.\n- the authors do not provide actual values for the DP parameters for their proposed method.\n- the authors test their method in one real-world dataset (MovieLens). While this step shows some validation of the results, I think more evaluation with real datasets is beneficial (e.g. meta-dataset)\n- the authors do not provide runtime values for their proposed method.\n- the evaluation lacks intuition about the choice of the model's parameters, e.g. low dimensional embedding size.\n- I am slightly confused about the results. In the synthetic experiment, the proposed method is much better than all methods by a big margin. This however is not the case in the MovieLens dataset, where the proposed method achieves on average similar results to full fine-tuning. An intuition about the discrepancy source would be nice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2fs9s7AGsyF", "reviewer": "Reviewer_wf3D", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies meta-learning approaches for representation learning in a linear setting with a low-rank and a sparse assumption. The authors proposed an alternating minimization method to solve the resulting problem. The sample complexity bound is also provided. ", "summary_of_the_review": "This paper studies meta-learning approaches for representation learning in a linear setting with a low-rank and a sparse assumption. The motivation for adding such a low-rank and a sparse assumption is unclear. Comparisons with the existing work [1] are not clear/enough.", "strength_and_weaknesses": "Strengths:\n\n1: The theoretical analysis is rigorous and solid.\n\n2: The studied problem is interesting and could have a high impact on the community.\n\nWeaknesses:\n\n1: The motivation/intuition for adding such a low-rank and a sparse assumption is unclear. \n\n2: Comparisons with [1] are not clear. What are the contributions/advantages beyond [1]?\n\n[1] Differentially private model personalization. NeurIPS 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0J16et3k3ou", "reviewer": "Reviewer_6iVb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies a linear meta-learning model comprising task-specific parameters: a low-rank weight and a sparse bias. The authors propose an alternating optimization algorithm for it. The generalization bound is also derived. The empirical performance is evaluated on Movielens 1M and synthetic datasets. ", "summary_of_the_review": "This paper proposes a simple linear model with a theoretically-guaranteed algorithm. However, the empirical evaluation is limited and whether the model is really applicable to real-world tasks remains mostly unknown.", "strength_and_weaknesses": "Strengths\n- Rigorous theoretical analysis is provided.\n\nWeaknesses\n- The paper is densely written and hard to follow.\n- The model looks too simple to solve complex real-world data.\n- The empirical evaluation is limited. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject