Title: Limit Analysis for Symbolic Multi-step Reasoning Tasks with Information Propagation Rules Based on Transformers
Primary area: learning theory
Keywords: multi-step reasoning, parallel reasoning, large language models, interpretability, buffer mechanism, model capacity

Abstract: Transformers have ability to perform reasoning tasks, however the intrinsic mechanism remains widely open. In this paper we propose a set of information propagation rules based on Transformers and utilize symbolic reasoning tasks to theoretically analyze the limit reasoning steps. 
We show that the number of reasoning steps has an upper bound of $O(3^{L-1})$ and a lower bound of $O(2^{L-1})$ for a model with $L$ attention layers.