Role of two learning rates in convergence of model-agnostic meta-learning
Model-agnostic meta-learning is known as a powerful meta-learning method.However, MAML is notorious for being hard to train because of the existence of two learning rates.Therefore, in this paper, we derive the conditions that inner learning rate and meta-learning rate must satisfy for MAML to converge to minima with some simplifications.We find that the upper bound of depends on, in contrast to the case of using the normal gradient descent method.Moreover, we show that the threshold of increases as approaches its own upper bound.This result is verified by experiments on various few-shot tasks and architectures; specifically, we perform sinusoid regression and classification of Omniglot and MiniImagenet datasets with a multilayer perceptron and a convolutional neural network.Based on this outcome, we present a guideline for determining the learning rates: first, search for the largest possible; next, tune based on the chosen value of.
We analyzed the role of two learning rates in model-agnostic meta-learning in convergence.The authors tackled the optimization instability problem in MAML by investigating the two learning rates.This paper studies a method to help tune the two learning rates used in the MAML training algorithm.