Before revision:  In particular, we find that such pretraining enables FPT to generalize in zero-shot to these modalities, matching the performance of a transformer fully trained on these tasks.
After revision:  Additionally, we perform an analysis of the architecture, comparing the performance of a transformer fully trained on these tasks.