Before revision:  How to effectively train multilingual models has not been well studied. Using multilingual machine translation as a testbed, we study optimization challenges from loss landscape and parameter plasticity perspectives. We found that imbalanced trainingdata poses task interference between high and low resource languages , characterized by nearly orthogonal gradients for major parameters and the optimization trajectory being mostly dominated by high resource .
After revision:  How to effectively train multilingual models has not been well understood. In this paper, we show that a common situation in multilingual training, data imbalance among languages, poses optimization tension between high resource and low resource languages , characterized by nearly orthogonal gradients for major parameters and the optimization trajectory being mostly dominated by high resource .