SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(a) Embedding 15, run 1
Figure 9: Second-layer attention patterns of two-layer Transformers with a minimal first layer: (a), (b) are based on embedding 15 with different random seeds. (c), (d) are based on embedding 17 and 16. Different embedding functions lead to diverse attention patterns, most of which are not stack-like.
Figure 10: Relationship Between Balance Violation and Length Generalization. Accuracy from Transformers with minimal first layer with embedding 15, using both standard training and contrastive regularization (Equation (18)). We again observe that contrastive regularization helps reduce the balance violation and improve the length generalization performance.
(a)  $seed = 0$
Figure 11: Third-layer attention. The test sentence is fixed and the attention patterns learned by different 3-layer models with the same architectures on the same dataset show large variation visually.
(b) seed  $= 1$
(c) seed = 2
(d) seed = 3