Before revision:  Firstly, the BiLSTM encoder is replaced by the attention-based transformer structure , which is more capable of learning sequential patterns in longer texts.
After revision:  Firstly, the BiLSTM encoder is replaced by the attention-based transformer layers , which is more capable of learning sequential patterns in longer texts.