Before revision:  then, we pre-train the transformer-based model with those attentions in a self-supervised fashion and use them as feature extractors on downstream tasks, including phoneme classification and speaker classification.
After revision:  then, we pre-train the transformer-based model with those attention algorithms in a self-supervised fashion and use them as feature extractors on downstream tasks, including phoneme classification and speaker classification.