Title: Understanding Scaling Laws in Deep Neural Networks via Feature Learning Dynamics
Primary area: learning theory
Keywords: Scaling laws; Feature learning dynamics; Infinite-width and infinite-depth limit; Residual networks; Stochastic differential equations; Neural Tangent Kernel (NTK); Maximal update parameterization (μP).

Abstract: The remarkable success of deep neural networks is often attributed to empirical *scaling laws*, which predict consistent performance gains as model size, data, and compute increase. However, practical challenges such as training instability and diminishing returns highlight a fundamental gap in our theoretical understanding of these principles. The Neural Tangent Kernel (NTK) framework falls short in explaining these phenomena, as it cannot capture the rich feature learning (FL) that underlies modern AI breakthroughs (e.g., LoRA, CLIP, and in-context learning). By contrast, the maximal update parameterization ($\mu$P) preserves active FL at scale, but existing work has primarily analyzed scaling rules rather than training dynamics. This work bridges this gap by introducing a theoretical framework for the training dynamics of ResNets in the joint infinite-width and infinite-depth limit. We prove that in this regime, the training trajectory converges to a coupled forward–backward stochastic dynamics system, termed *Neural Feature Dynamics (NFD)*. NFD explains scaling success when its convergence conditions are satisfied and failure when these conditions break down. Our analysis further shows that diminishing returns emerge naturally as the model's dynamics approach the NFD limit and its effective capacity saturates. Experiments on ResNets validate these predictions, while additional empirical studies with Transformers and the Adam optimizer demonstrate broader applicability. This framework positions NFD as a principled foundation for future research on training dynamics, stability, and generalization in the FL regime.

TL;DR: We study why neural scaling laws succeed and fail by proving that deep ResNets trained with SGD converge to a coupled forward–backward SDE system in the joint infinite-width and infinite-depth limit.