Physics in Next-token Prediction
Authors: name: Hongjun An; name: Yiliang Song; name: Xuelong Li
Published: 2024-11-01 15:26:15

Abstract: We discovered the underlying physics in Next-token Prediction (NTP). We
identified the law of information conservation within NTP and proposed the
First Law of Information Capacity (IC-1), demonstrating that the essence of
intelligence emergence in auto-regressive models is fundamentally a process of
information transfer. We also introduced Landauer's Principle into NTP,
formulating the Second Law of Information Capacity (IC-2), which establishes
the relationship between auto-regressive model training and energy consumption.
Additionally, we presented several corollaries, which hold practical
significance for production practices. Finally, we validated the compatibility
and complementarity of our findings with existing theories.