Before revision:  We evaluate these memory augmented Transformers on machine translation task and demonstrate that memory size positively correlates with the model performance . Attention patterns over the memory suggest that it improves the model's ability to process a global context .
After revision:  We evaluate these memory augmented Transformers on machine translation task and demonstrate that memory size positively correlates with the model performance for machine translation and language modelling tasks. Augmentation of pre-trained masked language model with memory tokens shows mixed results for tasks from GLUE benchmark. Visualization of attention patterns over the memory suggest that it improves the model's ability to process a global context .