Question: On top of BERT does the RNN layer work better or the transformer layer?
Answer: Transformer over BERT (ToBERT)
Alternative answer: The transformer layer