Conditional Meta-Learning of Linear Representations
Authors: Giulia Denevi
Massimiliano Pontil
Carlo Ciliberto
Venue: NeurIPS 2022 Accept (2022)
Keywords: Conditional Meta-Learning
Linear Representation Learning
Statistical Learning Theory
Online Learning

Abstract: Standard meta-learning for representation learning aims to find a common representation to be shared across multiple tasks. The effectiveness of these methods is often limited when the nuances of the tasks’ distribution cannot be captured by a single representation. In this work we overcome this issue by inferring a conditioning function, mapping the tasks’ side information (such as the tasks’ training dataset itself) into a representation tailored to the task at hand. We study environments in which our conditional strategy outperforms standard meta-learning, such as those in which tasks can be organized in separate clusters according to the representation they share. We then propose a meta-algorithm capable of leveraging this advantage in practice. In the unconditional setting, our method yields a new estimator enjoying faster learning rates and requiring less hyper-parameters to tune than current state-of-the-art methods. Our results are supported by preliminary experiments.

Conditional Meta-Learning of Linear Representations
Giulia Denevi"
Leonardo Labs (Italy)
giulia. iallia.dn denevi. exteleonardo.com com
Massimiliano Pontil
Istituto Italiano di Tecnologia (Italy) & University College of London (UK) massimiliano. pontil@iit.it
Carlo Ciliberto
University College of London (UK) & Istituto Italiano di Tecnologia (Italy)
c.ciliberto@ucl. ac.uk
Abstract
Standard meta-learning for representation learning aims to find common represen- tation to be shared across multiple tasks. The effectiveness of these methods is often limited when the nuances the tasks' distribution cannot be captured by single representation. In this work we overcome this issue by inferring a conditioning function, mapping the tasks' side information (such as the tasks' training dataset itself) into a representation tailored to the task at hand. We study environments in which our conditional strategy outperforms standard meta-learning, such those which tasks can be organized separate clusters according to the representation they share. We then propose meta-algorithm capable of leveraging this advantage practice. the unconditional setting, our method yields new estimator enjoy- ing faster learning rates and requiring less syper-parameters to tune than current late-of-the-an methods. Our results supported by preliminary experiments.
1 Introduction
Learning a shared representation among a class of machine learning problems is a well-established
approach used both in multi-task learning [3. 20. 11] and meta-learnin [18, 15, 35, 24, 30, 9, 7]. The idea behind this methodology is consider two nested problem: at within-task level an empirical risk minimization is performed on each task, using inputs transformed by the current representation, on the outer-task (meta-) level, such representation is updated taking into account the errors of the within-task algorithm on previous tasks.
Such technique was shown to be advantageous in contrast to solving each task independently when the tasks share low dimensional representation, see e.g. [27, 25, 15, 24, 35, 5, 22, 9]. However, in real world applications we often deal with heterogeneous classes of learning tasks, which may overall be only loosely related. Consequently, the tasks' commonalities may not be captured well by a single representation shared among all the tasks. This is for instance the case in which the tasks can be organized in different groups (clusters), where only tasks belonging the same cluster share the same now-dimensional representation.
order to overcome this issue, previous authors developed non-convex methods (or convex relax- ations) attempting at clustering the tasks, e.g. [4, 26, 2, 20, 28, 40, 38, 31]. In this work, we follow
Work done while the first author was with Istituto Italiano Tecnologia (Italy).
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

the recent literature on heterogeneous meta-learning [37, 36, 32, 21, 10, 39, 14, 7] and propose a so-called conditional meta-learning approach for meta-learning representation. Our algorithm learns a conditioning function mapping available tasks' side information into a linear representation that is tuned to that task at hand. Our approach borrows from [14], where the authors proposed a conditional meta-learning approach for fine tuning and biased regularization. In those cases however, the tasks' target vectors are assumed to be all close to a common bias vector rather than sharing the same low-dimensional linear representation, as instead explored in this work. As we explain in the following, working with the representation setting requires a significant contribution with respect to the bias one in order give new formulation of the problem, to consider different meta-objective and different interpretation of the results. In addition, the representation setting is known to be a more relevant and effective framework in many scenarios in comparison to the bias one (see e.g. [28)). this work, we propose for the first time an online conditional method for linear representation learning with strong theoretical guarantees. In particular, we show that the method is advantageous over standard (unconditional) representation learning methods used in meta-learning when the environment of observed tasks is heterogeneous.
Contributions and organization. The contributions of this work are the following. First, in Sec. 2, we design conditional meta-learning approach to infer a linear representation that is tuned to the task at hand. Second, in Sec. 3, we formally characterize cireumstances under which our conditional framework brings advantage with respect the standard unconditional approach. In particular, we argue that this is the case when the tasks are organized in different clusters according to the support pattern linear representation their target vectors" share. Third, in Sec. 4, we design a convex meta-algorithm providing a comparable gain as the number of the tasks it observes increases. In the unconditional setting, the proposed method is able to recover faster rates and it requires to tune one less perr-aareeter with respect the state-of-the present numerical experiments supporting our theoretical claims. We conclude our work in Sec. 6 and posipone the missing proofs to the supplementary material.
unconditional methods. Finally, in Sec. 5, we
2 Conditional representation learning
In this section introduce our conditional meta-learning setting for representation learning. Then, we proceed to identify the differences with respect to (with respect to) the standard unconditional counterpart. We begin our overview by first introducing the class of inner learning algorithms we use in this work.
Within-task algorithms. We consider the standard linear supervised learning setting over = X with X' C Rd and y C R input and output spaces, respectively. We denote by P(z) the set of probability distributions (tasks) over Z. For any task μ P(()) and a given loss function
l: R x R + R, we aim at finding weight vector W/ E IRd minimizing the expected risk
min WERU
(w)
(w) = E( ((1x, w))
(1)
where, (, represents the Euclidean product in RRd, In practice, fs is only partially observed through dataset Z = (Ir2, v1); ~ n², namely, a collection of n identically independently distributed (i.i.d.) points sampled from μ.. Thus, the goal becomes to use learning algorithm in order to estimate a candidate weight vector with a small expected risk converging to the ideal R2, (w,,) as the sample size n grows. Specifically, in this work we will consider as candidate estimators, the family of regularized empirical risk minimizers for linear feature learning [3]. Formally, denoting by D = UneN 2" the space of all datasets on Z, for given 0 θ in θ $4. the set of positive definite d x d matrices, we will consider the following learning algorithms A(0,-) D Rd.
Rz, = 1 Σ E((,,, w), v.) + (w, Oww)
A(0, Z) = argmin @
Rz,
(2)
where Ran(@) denotes the range of 6. Here Øt denotes the pseudoinverse
Throughout this work
we will denote by Rz(-) = 1/n Σm= (((2;; (.3))) vk) the empirical risk associated to Z. Here, plays the role of linear feature representation that learned during the meta- meta-learning process (see [3]).

---

Remark 1 (Within-tash regularization parameter). Differently to previous work, see e.g. [15], we do not impose any constraints on the trace of (e.g. Tr(()) S 1). This allows us absorb the regularization parameter À typically used to control X (w. H 3 in 8. As we will discuss later, this choice reduces the number of hyper-parameter to tune and it allows to enjoy faster learning rates. Remark (Online variant of Eq. (2)). Paying additional negligible logarithmic factors, our analysis and results extend also the setting in which the minimizer in Eq. (2) is replaced by a pre-conditioned variant of online gradient descent on Rz with starting point We = 0 and appropriate step size:
A(0, 2)
1 TI
Σ
Wi:::
W+++1 w1
0p:
Pr = 8,34 + 0 WF,
81 E (ed((;y))) wx)). (3)
Unconditional Meta-Learning. The standard unconditional meta-learning setting assumes there exist a meta-distribution p E P(M) - also called environment in [6] - over a family M S P(()) of distributions (tasks) μl and it aims at selecting inner algorithm in the family above that is well suited solve tasks μ sampled from p. This target can be reformulated as finding a linear representation 0p E θ such that the corresponding algorithm A(Op. -) minimizes the transfer risk
(4) In practice, this stochastic problem is usually tackled by iteratively sampling a task μ ~ p and a corresponding dataset Z ~ ,t,,, and, then, performing step of stochastic gradient descent on an empirical approximation of Eq. (4) computed from Z. This approach has proven effective for instance when the tasks of the environment share a simple common linear representation, see e.g. [18, 5, 22, 15, 16, 12, 17, 9]. However, when a single linear representation is not sufficient for the entire environment of tasks (c.g. multi-clusters), this homogeneous approach is expected to fail. In order overcome this limitation, some recent works have adopted the following conditional approach to the problem, see e.g. [37, 36, 32, 21, 10, 39, 14].
min EO), AEe
Ep) Eucup Ez-pn R2 (A(0,
Conditional Meta-learning. Analogously to [14], we assume that any task μ ~ is provided of additional side information E S. In such case, we consider the environment p distribution p E S) over the set M of tasks and the set S of possible side information. Moreover, as usual, we assume / to decompose in p(-{s)ps(-) and p( p(-1)))m(-)) (-) the conditional and marginal distributions with respect S and M. For instance, we observe that the side information 8 could contain descriptive features of the associated task, for example attributes in collaborative filtering [1]. or additional information about the users in recommendation systems [19]). Moreover s could be formed by a portion of the dataset sampled from l (see [37, 14)). Conditional meta-learning leverages this additional side information in order adapt (or condition) the linear representation 0 e è on the associated task at hand, by learning a linear www.rr-rrreeettionvaaaaae function 7 solving
the problem
min ET),, TET
(5)
over the space T of measurable functions T O. Notice that we retrieve the unconditional meta-learning problem in Eq. (4) if we restrict Eq. (5) to the set of functions T = {t [(+)=
=
0, 8 )), mapping all the side information into the same constant linear representation.
In the next section, we will investigate the theoretical advantages of adopting such conditional perspective and, then, we will introduce a convex meta-algorithm to tackle Eq. (5).
3 The advantage of conditional representation learning
order to characterize the behavior of the optimal solution of Eq. (5) and to investigate the potential advantage of conditional meta-learning, we analyze the generalization properties of a given con- ditioning function т. Formally, we compare the error (T(r with respect to the optimal minimum risk
Eg = Epop (w,))
Wpp argmin (w). wERd
(6)
In order to do this, we first need to introduce the following standard assumptions used also in previous literature. Throughout this work we will denote by- .T the standard transposition operation.
Assumption 1. Let be a convex and L-Lipschitz loss function in the first argument. Additionally, there exist R > 0 such that |0]|| < R for any X.
3

---

Theorem (Excess risk with generic conditioning function T). Asm. hold. For any ~ Ps, intro- duce the conditional covariance matrices (s)) ( and C(s) [ Bumpp(ls) Bann, xx,, where, Th denotes the inputs' marginal distribution of the task p.. Let T E *T such that Ran(W (8)) S Ran(T(s)) for any ps and let A(T(s), be the associated inner algorithm from Eq. (2). Then,
Ep(T) - Eanps Tr W(()))
+ 2L Tr(r(s)C(())
(7)
Proof. For any (1,, 8) ~ consider the decomposition Ep(T) - EE,- = =B(motpp Bas + Cu.ll, with p.. Ez (,,(A((ss) Z)) (2((((()), 2))
C,, Ezzzzm (2((A((()), 2)) Rp(ww))
B,. is the generalization error of the inner algorithm A(T(s), -) on the task μr. Hence, applying stability arguments (see Prop. 6 in App. A),
write B,,s I 2T) )n- Regarding the term Cu.s. for any conditioning function T such that w/ E [Ran(T(s). we can write
where, the incquality exploits the definition of the algorithm in Eq. (2) minimum of the regularized empirical risk. The desired statement follows by combining the two bounds above and rewriting B(p.))>p = Essps Bu--(-ls)-
C,.s S En R2.r(s) (vew) - (w,(w,) = 2 (T(("TT()))
Thm. 1 suggests that the conditioning function Tр minimizing the right hand side of Eq. (7) is a good candidate solve the meta-learning problem. The following result explores this question by showing that such a minimizer admits a closed form solution. The proof is reported in App. B. In the following, will denote by ||- || and ||. the Frobenius and trace norm of matrix, respectively. Proposition 2 (Best conditioning function in hindsight). The conditioning func- tion minimizer and the minimum of the bound presented in Thm. 1 over the set (r E T (s)) S RRa((T()),, Ps-almost surely), are respectively
Tp(s) = (2L)-1 n/122 (((3))))))) Ep(TT) Eg S 2T.E.-ps (I))
1/2
(8)
We observe that, in comparison [14], the numerator term in the bound above describes different kind of tasks' similarity assumption: the conditional variance term ((p.s)~plluu E Eu--ptlss) ll? present in [14] is now substituted by the trace norm term Earps [[(s) ((2))) above. Ad-
ditionally, the bound above allows us quantify the benefits of adopting the conditional feature learning strategy.
Conditional unconditional Meta-Learning, Applying Prop. 2 to Tronst, obtain the optimal (constant) meta-parameter and the corresponding excess risk bound for unconditional meta-learning
T = 0p (2L) n²/² (C/ w ///3)//22///
(,,) - E s 21 / C /2
with unconditional covariance matrices W, "p and Cp Ex xx We observe that in the previous literature [13, 15J the authors restricted the unconditional problem over the smaller class of linear representation Ô E S Ran(W' (W,) C. Ran(@), Tr(0) < 1 and they considered the best unconditional representation, the matrix minimizing only part of the previous bound, namely,
argmin Tr(O = w (T//((((()?/))" Acê
(10)
On the other hand, the unconditional oracle we introduce above in Eq. (9) allows us to recover a tighter bound which is able to recover the best performance between independent task learning (ITL) and the oracle considered in previous literature [15]. Indeed, by exploiting the duality between the trace norm || ||. and the operator norm Eq. (9) by the quantity
Iloo of a matrix, we upper bound the right-side-term in
min |||W///1.1|C/
/ 1/2
|| FJ
1/2

---

namely, the minimum between the bound for independent task learning and the bound for uncon- ditional oracle obtained by previous authors. Notice that the unconditional quantity in Eq. (9) is always bigger than the conditional quantity in Eq. (8), since Eq. (9) coincides with the minimum over a smaller class of function. In order to quantify the gap between these two quantities namely, the advantage using the conditional approach with respect to the unconditional one we have to compare the term || h ///211 I|. with the term Bg-ps || C(s))/2 (s))/// (s)
We report below a setting that can be considered illustrative for many real-world scenarios in which such gap in performance is significant.
Example 1 (Clusters). Let S R29 be the side information space, for some integer q > O. Let P be such that the side information marginal distribution ps is given by a uniform mixture of m uniform distributions. More precisely, let ps
(8) Σ Pis.. with Pgg (B(a,, 1/2)) the uniform distribution on the ball of radius 1/2 centered at E S, characterizing the cluster i. For a given side information s, a task y ~ p(-ls) is sampled such that: 1) its inputs' marginal Th is a distribution with constant covariance matrix C(s) = 1Euwp(-le) 1e Pa
C. for some C E S4, 2) Wp is sampled from distribution with conditional covariance matrix W W(s) Bumpptlo)) Wop with ((s) such that (C2/2W (C)/2 (3)C4/2)(C4/2W (c)))///WWWW)) (p) 0 if + p. Then, Es-ps ((s))/2(()))))
[WP/ 1/2 C// Il.
The inequality above tells us that, in the setting of Ex. 1, the conditional approach gains a /m factor in comparison to the unconditional approach. Therefore, the larger the number of clusters is, the more pronounced the advantage of conditional approach with respect to the unconditional one will be. The vm gain factor follows from the fact that the weight vectors Wp sampled from the different clusters share disjoint supports (they share orthogonal representations). This allows us to rewrite the overall clusters weight vectors' covariance as the average of the intra clusters weight vectors' covariances. The /m term comes from this rewriting and the quadratic behavior of the covariance matrix. We refer to App. C for more details and the deduction. We also observe that particular case of the setting above could be that one in which q = 1 and the side information are noisy observations of the index of the cluster the tasks belong to. In our experiments, in Sec. 5, we consider more interesting and realistic variant of the setting above, in which we will use as task's side information a training dataset sampled from that task. In the next section, we introduce a convex meta-algorithm mimicking this advantage also in practice.
4 Conditional representation Meta-Learning algorithm
To tackle conditional meta-learming in practice consider parametrization where the conditioning functions that are modeled with respect to given feature map 4 S - I2 (with E N) on the side information space. In other words, we consider 7 : S $4.
7(-) (M)()) M+ +C,
(11)
for some tensor M E Rxxdxk (p E N) and matrix E S4. By construction, the above parametriza- tion guarantees us to learn functions taking values in the set of positive semi-definite matrices. However, directly addressing the meta-learning problem poses two issues: first, dealing with tensorial structures might become computationally challenging in practice and second, such parametrization is quadratic in M and would lead to non-convex optimization functional in practice. To tackle this issue, the following results shows that we can rewrite the conditioning function in the form of Eq, (11) by using matrix in 84k This will allows us to implement our method working with matrices in $4; instead of tensors in xxx Throughout this work, we will denote by & the Kronecker product. Proposition 3 (Matricial formulation of Tas (s)). Let be as in Eq. (11). Then.
-(s) (Ia ((s)") MM (Ia @ ((+)) +C, where Ia is the identity in ddd and Нa the matrix in dk defined by the entries
(12)
(H (1 -1) (M i, h), (t. 1.j=l...,, d, h, = 1, 1....,k.
k.
The arguments above motivate to consider the following set of conditioning functions: To = { r() = 2 x() H (())) + C such that H e sdh C E &
(13)
5

---

To highlight the dependency of a function E To with respect to its parameter H and C, we will denote TH.C- Evidently, To contains the space of all unconditional estimators Tronst, We consider T& equipped with the canonical norm |TH (H, C)Il2 | + ||C||| where, recall, || || denotes the Frobenius norm. The following two standard assumptions will allow us to design and analyse our method.
function Tp belongs to Ta. namely there exist Hp E 8 and Cp E st, Assumption 2. The functtion such that Tpp.) = TH,,C,(-) = (Ia & (()) () H & (()) + Cp.
Assumption 3. There exists K > such that [+)[ S K for any E.
Asm. 2, known as well-specified setting assumption (see e.g. [33]), is a standard assumption in learning theory and it allows us to restrict the conditional meta-learning problem in Eq. (5) to Ta rather than to the entire space T of measurable functions. Asm. ensures that the meta-objective Lipschitz (see below).
The convex surrogate problem. We start from observing that, exploiting the generalization properties of within-lask algorithm (see Prop. 6 in App. A), we can write the following
2L
X
(T( I ((,e)) Ez-ap F2(T(8)),
Fz = R2,0 2))
Tr
T2
where X E Ruxd is the matrix with the inputs vectors (x;) as rows. The inequality above suggests us to introduce the surrogate problem
min FET
Ep(T), Ep(r) = =Ein Ezznnn F2(((s)).
(14)
We stress that the surrogate problem we take here is different from the one considered in previous work [12, 15, 14, 9], where the authors considered as meta-objective only a part of the function above, namely, ((ss)>> Ez~z" (A(((s)) 2)) As we will see in the following, such a choice is more appropriate for the problem hand, since, differently from the meta-objective used previous literature, it will allow us to develop conditional meta-learning method that is theoretically grounded also for linear representation learning.
Exploiting Asm. the surrogate problem Eq. (14) can be restricted to the class of linear functions Ta in Eq. (13) and it can be rewritten more explicitly as min H&S4k Ces4
E(H,s)<p Ezzna L(H, C, Z), C(H, = F2(THH.(s)). (15)
=2
In the following proposition we outline some useful properties of the meta-loss C(., s, Z) introduced above (such as convexity) supporting its choice as surrogate meta-loss.
Proposition (Properties of the surrogate meta-loss L). For any Z E D and 8 S, the function C(.,',, 8, Z) is convex and one of its subgradients is given, for any H E s and C S4, by
s,, 44
(C),,C,,,)(H)) = (Ia 2 4
(s)) ( 8 (())),
where
4 -= A -n. www Wwm.coooincttst..
2
n2
Moreover, by Asm. I and Asm. 3,
|V O) $ $1+ +2n 1)
The proof of Prop. is reported in App. D.2. It follows from combining results from [15] with the composition of the linear parametrization of the functions TH,C Tpp
The conditional Meta-Learning estimator. The meta-learning strategy we propose consists in applying Stochastic Gradient Descent (SGD) on the surrogate problem Eq. (15). Such meta- algorithm is implemented in Alg. 1: we assume to observe a sequence of i.i.d. pairs (Z,, st) of training datasets and side information, and at each iteration we update the conditional parameters (H,, C.) by performing a step of constant size y > O in the direction of V..,,' 8t, (H,, C.) and a projection step on g Sq. Finally, we output the conditioning function TH e parametrized by (H, C), the average across all the iterates (H,, Ct) The theorem below analyzes the generalization properties of such conditioning function.

---

Algorithm Meta-algorithm, SGD on Eq. (15)
Input Y 0 meta-step size, Ho 34 Co 8 Initialization II, 1o E $ C Co 84
For -=1
Receive (Ht, 8e) ~ p and Ze pai
Let 6, = (Ia @ (s,)) Hr Ia @ ((s)) + C2 and compute wa. A(O,. Z,) by Eq. (2)
Compute ,,, (H, C.) as in Prop. with wa,
Update (He+1, C++1) = proje ( Ct) - -VL(: 2,) Ct))
T
T
Return π Σ 11, C
T Σ C.
Theorem 5 (Excess risk bound for the conditioning function returned by Alg. 1). Let Asm. and Asm. 3 hold. For any o recall the conditional covariance matrices W (s) and C(s) introduced in Thm. 1. Let THCC be fixed function in Ta such that Ran(W(s)) C. Can(TH,o(s) for any s ~ ps- Let H and C be the outputs of Alg. I applied to sequence (Z,, st) of i.d. pairs sampled from P with an appropriate meta-step size 7, Then, expectation with respect to the sampling of (Ze, St)E=1
B Eg S Bares (((H),(s))W(ss) 2L' Tr(TH,c(S)C(s))
2 (1 (L)K))
n Ho. C Co)lle
+ G 2 1
VT
The detailed proof is reported in App. D.4. Exploiting the fact that, for any T E T, Ep(T) S (T( and adding =Ep(TH,c), we can write the following
Proof (Sketch).
Ez (T Ep S A(TH.C) + B(TH,c) A(TH,C) = Bz ( - (TH..)
(16)
B(TH,C) = E,(TH,C)- -
The term A(Tr,c) can be controlled according to the convergence properties of the meta-algorithm in Alg. 1 as described in Prop. 12. Regarding the term B(TH.C), exploiting the definition of the within- task algorithm in Eq. (2) as minimum, for any T E T such that (an(Bpppp(l)) W. S Ran(r(s)) for any Ps. can rewrite
B(T) B(na)rp ([T()))),)]]
2
The desired statement then derives from combining the two parts above and optimizing with respect 7
Remark 3 (Online variant of Eq. (2)). Similarly to the bias regularization framework in [14], for the online inner family Rem. 2, we approximate meta-subgraddent Prop. 4 by replacing the batch minimizer A(TH.c(s), Z) in Eg. (2) with the last iterate of the online algorithm in Eq. (3).
Proposed vs. optimal conditioning function. Specializing the bound in Thm. to the best conditioning function Tp in Prop. 2, thanks to Asm. 2, we get, up to contants, the following bound for estimator,
E W ())))))))))))))))))))))))))))))) 11/2 + I111,- 1l. C, - Co)ll 1/2
From such a bound, we can state that our proposed meta-algorithm achieves comparable performance the best conditioning function Tp in hindsight, when the number of observed tasks is sufficiently large. Moreover, recalling the unconditional oracle i, in Eq. (10) used in previous literature, regarding the second term vanishing with T, we observe that our conditional meta- meta-learning approach incurs a cost of (H, - Ho. C. - Coll as opposed to the cost of ||0 6 - associated to state-of-the-an unconditional learning approaches (see [15, 5, 22, 9J). Thus, our conditional approach presents a faster convergence rate with respect to than such unconditional methods, but a complexity term that expected to be larger due to the larger complexity of the class of functions

---

ITL
ITL
Test Error
0.16
uncond.
Test Error
0.18
uncond.
cond.
cond.
0.17
0.14
0.16
0.12
100
200
300
400
500
0
100
200
300
400
500
Training Tasks
Training Tasks
Figure 1: Mean test error (5 generations) on synthetic data, 2 (Left) and 6 (Right) clusters.
are working with. Such a faster rate with respect to T is essentially due to our formulation of the problem on the entire set of sesitive-emmidffnnie matrices (with no trace constraints). This in fact allows us to incorporate the within-lask regularization parameter directly in the linear representation and gain a VT order that was lost in previous literature when tuning with respect the parameter A. At the same time, this allows us to develop also method requiring to tune just one hyper-parameter. while previous unconditional approaches requires tune two hyper-parameters.
Comparison unconditional Meta-Learning. Specializing Thm. the best unconditional estimator TH,C = Op we introduced in Eq. (9), the bound for our estimator becomes, up to constants,
W // / 1/2 + 110, Coll 1/2
From the bound above, we can conclude that the conditional approach provides, at least, the same guarantees its unconditional counterpart. Moreover, stress again that the bound above presents a faster rate with respect to T in comparison the state-of-the-at th--hh-rrt art unconditional methods.
5 Experiments
We now present preliminary experiments in which we compare the proposed conditional meta- learning approach in Alg. (cond.) with the unconditional counterpart (uncond.) and solving the tasks independently (ITL,, namely, running the inner algorithm separately across the tasks with the constant linear representation 0 = Ia E 3 We considered regression problems and we evaluated the errors by É the absolute loss. We implemented the online variant of the within-task algorithm introduced in Eq. (3). The
was chosen by (meta- ccress validation on separate Tar Twa and Tie respectively meta-train, -validation and -test sets. Each task is provided with training dataset Zu of fhr points and a test dataset Zre of the points used to evaluate the performance of the within-tasks algorithm. In App. E we report the details of this process in our experiments.
Synthetic clusters. We considered two variants of the setting described in Ex. I with side information corresponding to the training datasets Zur associated each task. both settings, we sampled Tiot = 900 tasks from uniform mixture of m clusters. For each task μi, generated the target vector W/, Rd with
20 as W, = P(ja) duz. where, ju {1, .m) denotes the cluster from which the task μ was sampled and with the components of ūū E Idd(110 sampled from the Gaussian distribution (00, 1) and then ū. normalized to have unit norm, with P(jx) Iddxd/(10) a matrix with orthonormal columns. We then generated the corresponding dataset (2i4, ys (t)
Thte with That 80 according to the linear equation y = (z, W.) + <, with x sampled uniformly on the unit sphere in Rd and sampled from Gaussian distribution, ~ GO,, 0.1). In this setting, the operator norm of the inputs' covariance matrix is small (equal 1/d) and the weight vectors' covariance matrix of each single cluster is low-rank (its rank is d/(10) = 2). We implemented our conditional method using the feature map & D -> 2d defined by (Z) = Rer d(zi), with d(z;) = vec (3;. 1). where, for any matrix A = [01, a2] E d with columns d1 a2 E Rd. vec(A) (2),42) Rdd Fig. I, we report the results we got on an environment of tasks generated as above with m = 2 (Left) and m = 6 (Right) clusters, respectively. As we can see, when the clusters are two, the unconditional approach outperforms ITL (as predicted from previous literature), but the unconditional method in turn outperformed by our conditional counterpart. When the number of clusters raises six, the performance of unconditional meta-learning degrades the same performance of ITL.,

---

uncond.
Test Erro
cond.
Test Error
uncond.
ITL
cond.
3
20
40
60
80
100
0
50
100
150
200
Training Tasks
Training Tasks
ITL
Test Error
10
uncond.
cond.
8
100
150
200
250
Training
Tasks
Figure 2: Mean test error (5 splits) on Lenk (Top-Left), Movielens-100k (Top-Right), Jester-1 (Bottom) dataset.
while conditional meta-learning outperforms both methods. Summarizing, the more the heterogeneity of the environment (number of clusters) is significant, the more the conditional approach brings advantage with respect to the unconditional one. This is in line with Ex.
Real datasets. tested the performance of methods also on the regression problem on the computer survey data from [23] (see also [28]). Tiot 180 people (lasks) rated the likelihood of purchasing one of noot 20 computers. The input represents d = 13 computers' characteristics and the label is rate in {O..... 10). In this case, we used as side information the training datapoints = (2)) and the feature map + D + Rd++ defined by (Z) = wzz with wz the solution Tikhonov regularization with the squared loss, namely, the vector satisfying (XT X + la+ 1) XTy, y, where, E (dd++)xx is the matrix obtained by adding to the matrix X E RRnxd one column of ones the end. Fig. (Top-Left) shows that also in this case, the unconditional approach outperforms ITL, but the performance of its conditional counterpart much better.
We also tested the performance of the methods on the Moviclens- 100k and Jester-1 real-world datasets, containing ratings of users (tasks) movies and jokes (points), respectively. Recommendation system settings with d items can be interpreted within the meta-learning setting by considering each data point (x, y) to have input movic or a joke) and y E IR the corresponding score, see c.g. [12] for more details. We restricted the 20 most voted movies/jokes (as consequence, by formulation, d = 20). original dataset to the voted least movies/jokes, which led total of Teo 400/ 400/450 /50 tasks We guaranteed each
I to be the one-hot encoding of the current item be rated (c.g.
(i.e. users). In both cases, we used as side information the training datapoints Z = (a.) For the Movielens- 100k dataset in Fig. 1. For the Jester-1 dataset, let M and m denote the maximum and minimum rating value that can be assigned to joke. We adopted the feature map 4 D + z = (Ira, bi) we have (Z) (vec(((2)) 1), where vec denotes the vectorization operator (i.e. mapping a matrix the vector concatenating all its columns) and $ - dxxx2 is such that (2) cos(( (2)), sin((()))) C (E 22) with a(Z) = Σm 2,
used the same feature map described for the synthetic clusters experiments
22d+1 such that, for any dataset
M- and @ denoting [
the Hadamard (entry-wise) product broad-casted across both columns. The rationale behind this feature map is to represent as similar vectors those users with similar scores for the same movies. In particular, each item-score pair observed in training is represented the angle depending on the score attributed to that item (the vector corresponds to zero if that movie not observed at the training time). As it can be noticed in Fig. (Top Right and Bottom), the
unitary
in R2,,, with

---

ITL
ITLL
Test Error
0.16
cond.
Test Error
0.18
cond.
cond. B
cond. B
0.17
0.14
0.16
100
200
300
400
500
0
100
200
300
400
500
Training Tasks
Training Tasks
Figure 3: Mean test emor (5 generations) of our method and [14] on 2 (Left) and 6 (Right) clusters.
proposed approach performs significantly better than ITL and its unconditional counterpart also on these two benchmarks. This suggests that groups of users might rely each similar features (but different from those of other groups) to rate an item in the dataset (respectively movie or joke).
Comparison with [14]. We conclude the experimental section by comparing the performance of our method with the conditional meta-learning approach for biased regularization proposed in [14], In that case, the tasks' target vectors are assumed to be all close to a common bias vector rather than sharing the same low-dimensional linear representation, setting is known to be a more relevant and effective framework in many scenarios comparison to the bias one (see e.g. [28]). This is confirmed in Fig. 3 where our conditional representation learning method ('cond.') significantly outperforms the one [14] ('cond. B') in the synthetic settings used in Fig. 1, since the tasks' similarity leveraged by [14] is not appropriate in these cases.
instead in our method. The representation
6 Conclusion
We proposed conditional meta-learning approach aiming at learning function mapping task's side information into linear representation that well suited for the task at hand. We theoretically and experimentally showed that the proposed conditional approach is advantageous with respect to the standard unconditional counterpart when the observed tasks share heterogencous linear representa- tions. As a consequence of our analysis we also developed new unconditional meta-leamingg variant requiring tuning less pyper-parameters and relying on faster rates with respect state-of-the art unconditional approaches. We identify two future directions addressing the limitations our method. A first question left opened is how to design suitable feature map & when the tasks' training data used as side information. Following [32, 37], we adopted mean embedding representation. However, given the importance played by such feature map in Thm. 5, it will be worth investigating better alternatives. Secondly, will be valuable to investigate how predict non-linear conditioning functions (similarly to e.g. [7. 16, 32]) and use less expensive algorithms to update the positive matrices, such the Frank- Wolfe algorithm used in [9] for unconditional settings. In this last case, according to our analysis, applying standard convergence rates for Frank-Wolfe algorithm, we expect the meta-learning algorithm based on Frank- Wolfe iteration incur a slower rate of order T- 1/4 (instead of 1/2 for the SGD method proposed here) in Thm. paying the computational benefits terms statistical performance.
Acknowledgments and Disclosure of Funding
This work was supported in part by SAP SE and EPSRC Grant N. EP/P009069/1. C.C. acknowledges the support of the Royal Society (grant SPREM RGS' RI 201 149) and Amazon.com Inc. (Amazon Research Award - ARA). G.D. acknowledges Leonardo SpA for funding her participation to the conference.
RI

---

References
[1] Jacob Abemntthy, Francis Bach, Theodoros Evgeniou, and Jean-Philippe Vert. new approach to collaborative filtering: Operator estimation with spectral regularization. Journal of Machine Learning Research, 10(Mar):803-$5) 826, 2009.
Andreas Argyriou, Stéphan Clémengon, and Ruocong Zhang. Learning the graph of relations among multiple tasks. 2013.
[3] Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Convex multi-task feature learning. Machine Learning, 73(3):243-772, 2008.
[4] Andreas Argyriou, Andreas Maurer, and Massimiliano Pontil. An algorithm for transfer learning in a heterogeneous environment. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 71-85. Springer, 2008.
[5] Maria-Florina Balcan, Mikhail Khodak, and Ameet Talwalkar. Provable guarantees for gradient- based meta-learning. In International Conference on Machine Learning, pages 424-433, 2019.
[6] Jonathan Baxter. A model of inductive bias learning. Artif. Intell. Res., 12(149-198):3, 2000.
Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. arXiv preprint arXiv:1805.00 08005.8136. 36, 2018.
[8] Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of machine learning research, 2(Mar):499-526 2002.
Brian Bullins, Elad Hazan, Adam Kalai, and Roi Livni. Generalize across tasks: Efficient algorithms for linear representation learning. In Algorithmic Learning Theory, pages 235-246, 2019.
[10] T Tony Cai, Tengyuan Liang, and Alexander Rakhlin. Weighted message passing and minimum energy flow for heterogeneous stochastic block models with side information. Journal of Machine Learning Research, 21(11))1-34, 2020.
[11] Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
[12] Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Learning-to-Leant stochastic gradient descent with biased regularization. In International Conference on Machine Learning, pages 1566-1575, 2019.
[13] Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Incremental learning- to-learn with statistical guarantees. In Proc. 34th Conference on Uncertainty in Artificial Intelligence (UAI), 2018.
[14] Giulia Denevi, Massimiliano Pontil, and Carlo Ciliberto. The advantage of conditional meta- learning for biased regularization and fine tuning. Advances in Neural Information Processing Systems, 33, 2020.
[15] Giulia Denevi, Dimitris Stamos, Carlo Ciliberto, and Massimiliano Pontil, Online-within-onlin meta-learning, In Advances in Neural Information Processing Systems, pages 13089-13099 2019.
[16] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR, 2017.
[17] Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. International Conference on Learning Representations, 2018.
[18] Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference Machine Learning, pages 1920-1930 2019.

---

[19] F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions interactive intelligent systems (tiis), 5(4):1-19, 2015.
[20] Laurent Jacob, Jean-philippe Vert, and Francis R Bach. Clustered multi-task learning: A convex formulation. In Advances in neural information processing systems, pages 745-752, 2009.
[21] Ghassen Jerfel, Erin Grant, Tom Griffiths, and Katherine A Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. In Advances in Neural Information Processing Systems, pages 91 19-91390 2019.
[22] Mikhail Khodak, Maria-Florina F Balcan, and Ameet S Talwalkar. Adaptive gradient-based meta-learning methods. In Advances Neural Information Processing Systems, pages 5915- 5926, 2019.
[23] Peter J Lenk, Wayne S DeSarbo, Paul E Green, and Martin R Young. Hierarchical bayes conjoint analysis: Recovery of partworth heterogeneity from reduced experimental designs. Marketing Science, 15(2):173-11, 1996.
[24] Andreas Maurer. Transfer bounds for linear feature learning. Machine Learning, 75(3):327-350. 2009.
[25] Andreas Maurer, Massi Pontil, and Bernardino Romera-Paredee Sparse coding for multitask and transfer learning. In International Conference Machine Learning, 2013.
[26]
Andreas Maurer and Massimiliano Pontil. Transfer learning in a heterogeneous environment. In 2012 3rd International Workshop on Cognitive Information Processing (CIP), pages 1-6. IEEE, 2012.
[27] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. The Journal of Machine Learning Research, 17(1):2853-2884, 2016.
[28] Andrew M McDonald, Massimiliano Pontil, and Dimitris Stamos. New perspectives on k-
support and cluster norms. Journal of Machine Learning Research, 17(155):1-38, 2016.
[29] Charles A Micchelli, Jean M Morales, and Massimiliano Pontil. Regularizers for structured sparsity. Advances in Computational Mathematics, 38(3)) 489, 2013.
[30] Anastasia Pentina and Christoph Lampert. A PAC-Bayesian bound for lifelong learning. In International Conference on Machine Learning, pages 991-999, 2014.
[31] Bernardino Comera-Paredoe Hane Aung, Nadia Multilinear multitask learning. In International Conference Machine Learning, pages 1444-1452. 1452. PMLR, 2013.
and Massimiliano Pontil.
[32] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2018.
[33] Shai Shalev-Shwartz and Shai Ben-David Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014.
[34] Ingo Steinwart and Andreas Christmann. Support vector machines. Springer Science & Business Media, 2008
[35] Nilesh Tripuraneni, Chi Jin, and Michael I Jordan. Provable meta-learning of linear representa- tions. arXiv preprint arXiv:2002.1 1684, 2020.
[36] Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim. Multimodal model-agnostic meta- learning via task-aware modulation. In Advances in Neural Information Processing Systems, pages 1-12, 2019.
[37] Ruohan Wang, Yiannis Demiris, and Carlo Ciliberto. structured prediction approach for conditional meta-learning. Advances in Neural Information Processing Systems, 2020.
12

---

[38] Kishan Wimalawarne, Masashi Sugiyama, and Ryota Tomioka. Multitask learning meets tensor factorization: task imputation via convex optimization. In NIPS, pages 2825-2833. Citeseer, 2014.
[39] Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning. arXiv preprint arXiv:1905.0530/ 2019.
[40] Jiayu Zhou, Jianhui Chen, and Jieping Ye. Clustered multi-task learning via alternating structure optimization, Advances in neural information processing systems, 2011:702, 201
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See the abstract and Lines 44 - 54.
(b) Did you describe the limitations of your work? [Yes] See Sec. 6.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our work does not imply any negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Asm. 1, Asm. 2, Asm. 3.
(b) Did you include complete proofs of all theoretical results? [Yes] See See. 3, Sec. and Supplementary material.
you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL.) [Yes] See Sec. 5, App. E and Supplementary material.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? Yes] See Sec. App. and Supplementary material.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See See. 5, App. and Supplementary material.
(d) Did you include the total amount of compute and the type of resources used (e-g., type of GPUs, internal cluster, or cloud provider)? [Yes] See App. E and Supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] In Fig. we compared with the method in [14] and we cited that work.
(b) Did you mention the license of the assets? [Yes] See Fig. 3.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We attached our code in the Supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you't rr using/curating? [N/A] We used benchmark data and we cited the original source where one can find all the details.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] The data we are using does not contain personally identifiable information or offensive content.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We did not use crowdsourcing conducted research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] We did not use crowdsourcing or conducted research with human subjects.

---

(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] We did not crowdsourcing conducted research with human subjects.


Reviews:
[{"review_id": "mpfiqeyyTEX", "reviewer": "Reviewer_4JRE", "summary": "This paper addresses representation learning in meta learning setting when tasks' distribution cannot be captured by a single cluster and representation. It proposes to group tasks into different clusters where  tasks in the same cluster share the same representation, and to map task side information to a task specific linear representation via conditional function. The paper provides theoretical and experimental demonstration of the advantages of the proposed approach over standard meta-learning approach.\n", "questions": "What's the differences and advantages of the proposed method comparing with [14] which is also a conditional meta learning method?\n", "limitations": "The authors addressed the limitations explicitly, and did not discuss potential negative societal impact of the work.\n", "rating": 7, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper proposes a conditional meta-learning approach to deal with representation learning among hyterogenous tasks.\n\n+ It derives a new unconditional meta-learning variant that can have faster learning rate and requires less hyper-parameter tuning.\n\nWeaknesses:\n- The algorithm for updating the positive matrices is computationally expensive.\n\n- Currently the conditional function is linear, it cannot deal with non-linear mapping from task side information to a representation.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WnMRE-JcVi6", "reviewer": "Reviewer_5r8e", "summary": "Meta-learning, but additionally using “side information” to resolve contexts among a heterogeneous collection of tasks, in order to induce better task-specific representations. The side information is captured through a linear representation, which is then relaxed suitably for tractable optimization. As a special case, the formulation also reduces one hyper-parameter choice in the unconditional case. The paper then demonstrates results using experiments with a synthetic dataset (of clusters), and a few real-world datasets.", "questions": "- (Question) I wonder whether it might be possible to add to the text an explanation intuitively motivating the sqrt{m} factor in Example 1.\n\n- (Suggestion) It would be helpful to contrast Example 1 with what would happen if the method from ref. [14] were to be applied (to better elucidate the significance of this new approach).\n\n- (Suggestion) It might be helpful for readers if the paper could interpret the changed modeling assumptions (and notion of task similarity), and elaborate on the general nature of problems (incl. example scenarios) where the approach in this paper could lead to substantial improvements over the method presented in ref. [14].\n", "limitations": "Yes, the authors seem to have adequately addressed the limitations (as avenues for future work) and potential social impact (this work is abstracted to substantial generality beyond any specific application).", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper follows somewhat closely the presentation in ref. [14], building on it with what might be interpreted as a modeling improvement that also corresponds to a different notion of task similarity. The claims are augmented with what appears to be a rigorous characterization. Though I have not carefully verified the math, the results seem reasonable & understandable. My main concern regards the significance of the new result, given what has previously been elucidated in ref. [14].\n\n[14]: The Advantage of Conditional Meta-Learning for Biased Regularization and Fine Tuning, NeurIPS 2020\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OaRP01M6riY", "reviewer": "Reviewer_1CMy", "summary": "This paper presents a principled algorithm for performing meta-learning conditioned on available side information in the linear setting. The paper theoretically justifies that conditional meta-learning can produce superior convergence rates over unconditioned counter-part in the particular problem framing described. \nThey conduct experiments on both synthetic and real data settings to show the efficacy of their results ", "questions": "Relevant related work missing :\nCan the authors comment about the connection of their approach to the following - (when considered in the linear setting) \n\n[1] Task Similarity Aware Meta-Learning: https://proceedings.mlr.press/v161/zhou21a/zhou21a.pdf\n\n\n**Nit**\n* Figure 1 - the caption should be (left) and (right) - no ?", "limitations": "The authors rightly identify multiple limitations of their work and suggest to tackle them in future work", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. This paper tackles an important problem of conditioned meta-learning when side information is provided. They prove faster convergence rates for a  flavor of the conditional setting.\n2. The paper is well written and intuitively structured. The chosen notation for the proofs make them a bit easier to go through.\n3. Experiments, though simple, drive the point home about the relevance of the setting explored.\n\n\n**Weaknesses**\n1. The paper presents a core assumption (2), that the optimal function $\\tau_{\\rho}$ lies in the given space but provide no intuition or at least rough justification for why this could be the case. Though the experiments suggest that this does not prevent good performance over baselines, it is unclear that this is not a quirk of the setting they test in.\n2. k is introduced in section 4 as the dimension of the output feature map given side information but subsequently it is not discussed how it is set in experiments - did I miss this ?\n3. The experiments seem a bit limited - but this seems defensible in light of the extensive theory provided. Eg - an exploration of a wider range of feature mapping functions for the side information would be a good ablation to perform.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IIS5uaKzA5G", "reviewer": "Reviewer_oWcc", "summary": "This paper introduces a new representation learning method designed for working with a diverse set of heterogeneous tasks. In contrast to prior works that learn task vectors (via a conditional mapping that takes in side information) that are assumed to be close to a common bias vector, this work directly learns linear representations for the task at hand. They provide theoretical analyses for this approach and validate the method on small datasets.", "questions": "- The proposed method appears to be limited to learned linear representations that will be adapted to the tasks at hand, so it's a bit tricky for me to see how applicable/scalable this approach is to more complex, real-world datasets. The paper also compared their approach to very weak baselines (e.g. the unconditional meta-learning formulation). Would the authors comment on this point? ", "limitations": "The authors addressed a few limitations of their approach in the conclusion, but did not provide any details on potential negative societal impacts of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper provides rigorous theoretical analyses regarding their approach (e.g. excess risk, conditions for optimality, relationship to existing approaches, etc.).\n\nWeaknesses:\n- The empirical evaluations are quite weak -- I've elaborated a bit more on this in the \"Questions\" section.\n- This is a very minor point, but there are various places throughout the paper where the writing could be cleaned up to make the presentation more clear (e.g. \"w.r.t. the bias one in l. 36).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept