Rep-MTL: Unleashing the Power of Representation-level Task Saliency for
  Multi-Task Learning
Authors: name: Zedong Wang, name: Siyuan Li, name: Dan Xu

Despite the promise of Multi-Task Learning in leveraging complementary
knowledge across tasks, existing multi-task optimization (MTO) techniques
remain fixated on resolving conflicts via optimizer-centric loss scaling and
gradient manipulation strategies, yet fail to deliver consistent gains. In this
paper, we argue that the shared representation space, where task interactions
naturally occur, offers rich information and potential for operations
complementary to existing optimizers, especially for facilitating the
inter-task complementarity, which is rarely explored in MTO. This intuition
leads to Rep-MTL, which exploits the representation-level task saliency to
quantify interactions between task-specific optimization and shared
representation learning. By steering these saliencies through entropy-based
penalization and sample-wise cross-task alignment, Rep-MTL aims to mitigate
negative transfer by maintaining the effective training of individual tasks
instead pure conflict-solving, while explicitly promoting complementary
information sharing. Experiments are conducted on four challenging MTL
benchmarks covering both task-shift and domain-shift scenarios. The results
show that Rep-MTL, even paired with the basic equal weighting policy, achieves
competitive performance gains with favorable efficiency. Beyond standard
performance metrics, Power Law exponent analysis demonstrates Rep-MTL's
efficacy in balancing task-specific learning and cross-task sharing. The
project page is available at HERE.