Before revision:  These comprise 24 fine-tuned transformer layers from pre-trained RoBERTa and 24 relation-aware transformer layers trained from scratch.
After revision:  These comprise 24 fine-tuned transformer layers from pre-trained RoBERTa and 24 relation-aware layers trained from scratch.