Before revision:  We  also present a three-dimensional visualization of the interaction between attention and protein structure.
After revision:  We find this behavior to be consistent across three Transformer architectures (BERT, ALBERT, XLNet) and two distinct protein datasets. We also present a three-dimensional visualization of the interaction between attention and protein structure.