Title: No title found

Abstract: No abstract found

:



Figures and Tables:

Peer review (iclr2020):
Summary: The MAML meta-learning objective is where are test and train sets for task , is the regular training loss and is the parameter of e.g. a neural network model. By minimizing it is possible to find a good univeral initialization , i.e. one that generalizes well between tasks. The authors propose a modification of the above update in the form of where is a learned preconditioner for the gradients. The specific model they propose for is a low rank model that also depends on a parameter which is set as the gradient of the previous gradient step updated with the proposed preconditioner with set to 0. The authors propose that their preconditioner helps with the problem of noise in meta learning and also reduces the number of steps required in the inner loop. They claim to validate those propositions by presenting experimental evidence. I propose to reject the paper. The proposed model seems somewhat arbitrary and is not well-motivated through a priori theoretical reasoning. Many of the design decisions are not explained well. For example the dependence on the context vector is not well formalized. The claims about the adaptivity gained through the GECCO model and the noise are not well motivated. Furthermore, there are in my view various inaccuracies (see detailed comments). Details: - Abstract: The claim that gradient steps required for more samples in a task become more costly makes sense if we think full-batch. But in that case there is no additional noise. So the first two sentences already seem vague without providing a proper reference for what is truly being claimed here. - Abstract: *We show that...*, perhaps specify in what way you show this. - Page 1: are are - Page 1: needs reference / reasoning for increase in noise with higher-order solvers - Page 1: what does higher-order solvers are *required* mean? when are they required? - Page 1: Hessian is not a higher-order solvers, it is a second derivative, perhaps you mean (quasi-)Newton here? - Page 1: *so-called gradient steps* perhaps _formally_ define *gradient step* early, if you refer to it so often (i.e. give MAML definition already in intro) - Page 2: In this paper, ... here you are just repeating what you already just explained. - Page 2: Our method converges fast and ... , in what context? / under what assumptions? Say the type of experiments this holds for. - Page 2: ...few-shot scenario, etc. , what do you mean by etc.? - Page 2: ... GECCO learns the update functions of the gradients, how do you define update functions? Perhaps better to say learns a preconditioner for the gradient step? - Page 3: , perhaps say something about stochastic assumptions here, is this iid? - Page 3: (nitpick) functionality of the model of interest, why functionality? is just the model / function, no? - Page 3: define / explain universal initialization - Throughout the whole paper: Many uses of articles where they do not belong. Perhaps ask a native speaker to proof read. - Page 4: immediately clarify that are the parameters of your preconditioner - Page 4: in Remark 2: Adam etc. are adaptive methods whose motivation comes from approximating the covariance matrix of the stochastic gradients and NOT approximating a Hessian. They are really quite different (especially when away from the optimum). The adaptivity in Adam and so on is then based on the square root of the (diagonal) covariance approximation. In quasi-Newton / natural gradient methods you would use the inverse of the Hessian / Fisher. Considering that your paper proposes a way to model something very similar to a preconditioner it would be perhaps be good if you could properly put your model into the context of ways to low-rank approximate covariance / Fisher / Hessian matrices. - Page 4: Give an intuition for what is supposed to be if you are going to use it. - Page 5: Eq. equation 3 - Page 5: Why do you need if you can just set ? How is that an update? - Page 5: GECCO uses a context vector..., you have just been talking about this for three paragraphs, why are you saying this again? - Page 5: Algorithm 1, line 7, shouldn*t it be instead of ?