Before revision:  Sequential fine-tuning and multi-task learning are two methods for sharing information, but suffer from problems such as catastrophic URLetting and difficulties in balancing multiple tasks. Additionally, multi-task learning requires simultaneous access to data used for each of the tasks, which does not allow for easy extensions to new tasks on the fly. We propose a new architecture as well as a two-stage learning algorithm that allows us to effectively share knowledge from multiple tasks while avoiding these crucial problems.
After revision:  Sequential fine-tuning and multi-task learning are two methods for sharing information, but suffer from problems such as catastrophic URLetting and difficulties in dataset balancing. To address these shortcomings, we propose AdapterFusion, a new two stage learning algorithm that allows us to effectively share knowledge from multiple tasks while avoiding these crucial problems.