Before revision:  Then we use the pseudo-label for the task-specific loss and masked language modelloss to pre-train. The experiment shows that using the fine-tuned model's predictions for pseudo-labeled pre-training offers further gains in the downstream task.
After revision:  Then we use the pseudo-label for the task-specific training on the large amounts of user-generated data. We consider this task-specific training step on pseudo labels as a pre-training offers further gains in the downstream task.