Title: Transformers are Inherently Succinct
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords: Transformers, complexity, expressivity, automata, LTL

Abstract: We propose succinctness as a measure of expressive power
    of a transformer in describing a concept. To this end, we prove that 
    transformers are highly expressive in that they can represent formal 
    languages substantially more succinctly than standard representations of 
    formal languages like finite automata and Linear Temporal Logic (LTL) 
    formulas. As a by-product of this expressivity, verifying 
    even simple properties of transformers is shown to be provably intractable (i.e. EXPSPACE-complete).