reviews: 
Comments: This paper explores how and why properties of the data distribution control a transformer's propensity to use in-context vs. in-weights learning strategies.  The authors design a synthetic, parameterized family of tasks that expose dependencies of learning strategies on the data distribution that have been identified in prior work.  They then analyze the behavior of models trained on these tasks to understand how the dynamics of learning give rise to different strategies over time.  One of the key findings is that models learn a shortcut strategy that enables better-than-chance (but suboptimal) performance -- choosing an answer among one the labels presented in-context, without regard for the inputs presented in context.  In a stripped-down model that maintains the qualitative behavior of the full transformer parameterization, the authors show how this shortcut strategy facilitates (but is not necessary for) learning of a true ICL strategy via an analysis of the loss landscape.
Strengths: This paper tackles a timely and interesting topic, and contains several insights and useful contributions.

-- The synthetic task family introduced is a clean and intuitive way of exposing dependencies of learning strategies on pretraining data distributions.  Showing that key phenomena identified in (Chan et al., 2022) can be replicated in this setting is a useful contribution.

-- The characterization of the initial slow learning phase as driven by an increase in context-label accuracy is interesting

-- The idea that a strategy that results in good context-label accuracy can facilitate (despite not being necessary for) learning of a true ICL strategy is very interesting, and some evidence is provided for this idea

-- The evidence provided that the emergence of induction heads is strongly linked with the development of the ICL strategy, while not entirely novel, is nice to see
Weaknesses: I have the following concerns about this paper.  Many of them involve claims that I feel are made too strongly in the paper relative to the level of evidence provided.

-- The paper illustrates a set of phenomena in figure 2, and promises a mechanistic understanding of these phenomena.  But the mechanistic analysis provided later in the paper does not speak to most of the phenomenology -- for instance, the dependence of the ICL/IWL tradeoff on B, epsilon, K, and alpha.  In fact, the mechanistic analysis focuses on the p_C > 0 case, which is different from the p_c = 0 regime that gives rise to all the tradeoffs observed in Figure 2.  Thus, the connection between pages 1-4 of the paper and the rest is not entirely clear.

-- -- The following sentence, while intuitively reasonable, is written as a key strong claim and as far as I can tell is not really justified with evidence: "Therefore, the relative rates at which the network acquires ICL and IWL control the fraction of loss explained by each mechanism after convergence."

-- The paper makes strong causal claims based only on correlational evidence.  For instance, "Induction head formation drives the abrupt transition during ICL."  As far as I can tell no evidence is given for this claim, other than the (very suggestive, I agree!) fact that they coincide in time.

-- The paper makes strong claims about the three-parameter model proving or ruling out certain hypotheses.  An example is the sentence "This rules out hypothesis V as only the factors corresponding to the progress measures (a) through (d) have been included in the minimal model."  In my opinion, such claims are much too strong.  The three-parameter model is ultimately a different model from the original transformer architecture being used!  While the analysis of its behavior is suggestive of the learning strategies used by the original architecture, it is not conclusive.  The strength of the claims should be adjusted accordingly.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 2 fair
Clarity: 2 fair