Kati8/ICLR_2023_2025 huggingface dataset:
authors: Ashok Vardhan Makkuva
authors: Marco Bondaschi
authors: Adway Girish
authors: Alliot Nagle
authors: Martin Jaggi
authors: Hyeji Kim
authors: Michael Gastpar
reviews: 
Comments: This paper introduces a new framework for analyzing transformers through the lens of Markov chains, which is a novel and theoretically rich approach. This framework enhances our understanding of how transformers handle sequential data, particularly first-order Markov processes. Specifically, the theoretical characterization of the loss landscape for single-layer transformers, identifying the presence of both global minima (bigram) and bad local minima (unigram), is a good contribution. This helps explain the empirical observations and provides a solid foundation for future work.
Strengths: + The theoretical results are novel and informative.

+ The experiments are well-designed, providing solid support of the considered insights.
Weaknesses: I found no explicit assumptions in the paper, which may imply that few assumptions are utilized in the theoretical analysis. Thus, I suggest that the authors list the assumptions that were used.

The set of words under consideration is too toyish, which is the primary concern.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper introduces a framework for learning a first-order Markov chain using a single-layer transformer. Within this framework, the authors characterize the network’s loss landscape, identifying the presence of both global and bad local minima.
Strengths: 1. Examining the limitations of a single-layer transformer in learning a simple Markov kernel is an intriguing aspect of this work.

2. The authors characterize the roles of Markov switching probabilities and weight-tying in contributing to the presence of local optima.

3. Extensive experimental results are provided to support the theoretical insights
Weaknesses: The main limitation of this paper is that the current results may not fully address their central question: systematically characterizing the learning gap of a single-layer transformer. The authors show that a bad local minimum exists when $p+q > 1$, but they do not rule out the possibility that only global optima exist when $p+q < 1$. Given that the results are primarily existence arguments, this does not entirely support the empirical observation that the transformer converges to a bad minimum in the $p+q > 1$ regime while learning the global minimum in the $p+q < 1$ regime. This implicit bias would benefit from additional theoretical support, such as insights from learning dynamics.  Moreover, the data model is overly simplified, which raises expectations for a more rigorous analysis that fully characterizes the solution learned by transformers rather than relying on existence arguments alone.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper establishes a general and novel framework for the theoretical and empirical study of the attention mechanism on Markov data. They theoretically show that for one layer of Transformers, if the data comes from a Markov chain with binary states and transition probability given by P = (1-p, p \\ q, 1-q), the global minima of the next-token cross-entropy loss can recover the optimal risk. They theoretically show that there exist bad local minima with weight tying if p+q > 1, and without weight tying, local minima become saddle point. Empirically, they validate the theory and show those saddle point can be avoided if without weight tying. They also present experimental results on multi-state Markov chain and mention some open problems about deep Transformers.

I really like this paper. I think the framework is novel, interesting, and worth investigating. The theoretical and empirical results are solid, and the presentation is particularly clear. Although this is very initial research, they propose some future directions about why single-layer Transformers can sometimes be trapped in bad local minima when trained on the Markov data, and why deep Transformers can behave well and avoid local minima or saddle point. I think we should encourage this style of theoretical research since there are too much theoretical research with complex setups and heavy math, that explains nothing at all.
Strengths: 1. The theoretical results are solid and can be validated via experiments.

2. They establish a novel framework to investigate the attention mechanism on Markov data.

3. The presentation is particularly clear and comfortable.
Weaknesses: Current, I do not see any big issues.
Recommendation: 10
Reviewer confidence: 3
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: The authors study a curious case of transformers via Markov Chains, in which one-layer transformer struggles to find the global minimum in the Markov task. They further provide theoretical and empirical studies, showing that weight tying can lead to bad local minima when Markovian switching is greater than one. They also extend the results to multi-state Markov Chains. Experiments in multi-layer transformers further show that they can reach the global minimum, which is an intriguing future direction.
Strengths: 1. The problem setup is interesting and concise.
2. The presentation is clear. 
3. The analysis is solid, containing both theory and extensive experiments.
Weaknesses: The attention structure is missing here. Specifically, the Markov chain requires no attention structure, nor does theoretical analysis include the attention layer. It is more like "MLPs with Markov" instead of "attention with Markov".
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4