Task-agnostic distillation is an approach to knowledge distillation that aims to compress pre-trained language models without specifying tasks. This allows the distilled model to perform transfer learning and adapt to any sentence-level downstream task, making it more versatile and efficient.