Title: Empirically Testing Expressivity Bounds for Neural Network Architectures
Primary area: learning theory
Keywords: transformers, rnns, lstms, chomsky hierarchy, star-free languages, language recognition

Abstract: Understanding neural networks' algorithmic capabilities is key to predicting their performance on real-world tasks, and formal language recognition offers a rigorous framework for evaluating what algorithms these models can employ. To this end, a growing body of theoretical work has sought to characterize the classes of formal language that various neural network architectures---particularly transformers---can recognize.
These mathematical results generally rely on simplified versions of the transformer network that differ from those used in practice,
so empirically testing the hypothesized expressivity bounds remains crucial. Some work has already evaluated neural networks as recognizers of hand-picked languages, but in order to test broad claims about language recognition bounds, one should test on an appropriately broad distribution of languages and see how well the hypotheses and experiments align. This paper is an attempt to do just that, using efficient algorithms for randomly sampling formal languages from certain language classes and generating labeled datasets to be used for training and evaluating neural networks on those languages. We do so for the classes of regular languages, context-free languages, partially-ordered star-free languages, and star-free languages of varying dot-depth. Along the way, we develop a novel algorithm for efficiently sampling a string of a desired length from a probabilistic context-free grammar. We experimentally test the language recognition performance of transformers, simple RNNs, and LSTMs on these classes and find that transformers can recognize only a subset of star-free languages, underperforming theoretical predictions.
Surprisingly, all architectures surpass theoretical expectations on context-free languages. Our code is publicly available.

TL;DR: We test transformers, RNNs, and LSTMs on a large number of languages at different levels of the Chomsky hierarchy to verify theoretical expressivity bounds.