Order Matters in the Presence of Dataset Imbalance for Multilingual
  Learning
Authors: name: Dami Choi; name: Derrick Xin; name: Hamid Dadkhahi; name: Justin Gilmer; name: Ankush Garg; name: Orhan Firat; name: Chih-Kuan Yeh; name: Andrew M. Dai; name: Behrooz Ghorbani
Published: 2023-12-11 05:46:57

Abstract: In this paper, we empirically study the optimization dynamics of multi-task
learning, particularly focusing on those that govern a collection of tasks with
significant data imbalance. We present a simple yet effective method of
pre-training on high-resource tasks, followed by fine-tuning on a mixture of
high/low-resource tasks. We provide a thorough empirical study and analysis of
this method's benefits showing that it achieves consistent improvements
relative to the performance trade-off profile of standard static weighting. We
analyze under what data regimes this method is applicable and show its
improvements empirically in neural machine translation (NMT) and multi-lingual
language modeling.