What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models
Authors: Keyon Vafa
Peter G. Chang
Ashesh Rambachan
Sendhil Mullainathan
Venue: ICML 2025 poster (2025)
Keywords: world models
foundation models
inductive bias
large language models
Primary area: deep_learning->foundation_models
TL;DR: We develop inductive bias probes to test whether foundation models have inductive biases toward specific world models.

Abstract: Foundation models are premised on the idea that sequence prediction can uncover deeper domain understanding, much like how Kepler's predictions of planetary motion later led to the discovery of Newtonian mechanics. However, evaluating whether these models truly capture deeper structure remains a challenge. We develop a technique for evaluating foundation models that examines how they adapt to synthetic datasets generated from some postulated world model. Our technique measures whether the foundation model's inductive bias aligns with the world model, and so we refer to it as an inductive bias probe. Across multiple domains, we find that foundation models can excel at their training tasks yet fail to develop inductive biases towards the underlying world model when adapted to new tasks. We particularly find that foundation models trained on orbital trajectories consistently fail to apply Newtonian mechanics when adapted to new physics tasks. Further analysis reveals that these models behave as if they develop task-specific heuristics that fail to generalize.

What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models
Keyon Vafa" | Peter G. Chang?
Ashesh Rambachan Sendhil Mullainathan'
2
Abstract
gerich, 2004). This path from predicting sequences to understanding the deeper mechanisms that underlie them is not unique to physics. In biology. animal breeders noticed patterns in the traits of offspring long before their predictive insights inspired Mendel to develop theory of genetics.
Foundation models are premised on the idea that sequence prediction can uncover deeper
domain understanding, much like how Kepler's predictions of planetary motion later led to the discovery of Newtonian mechanics. However, evaluating whether these models truly capture deeper structure remains a challenge. We develop a technique for evaluating foundation models that examines how they adapt to synthetic datasets generated from some postulated world model. Our technique measures whether the foundation model's inductive bias aligns with the world model, and so we refer to it as an inductive bias probe. Across multiple domains, we find that foundation models can excel at their training tasks yet fail to develop inductive biases towards the underlying world model when adapted tasks. We particularly find that foundation models trained on orbital trajectories consistently fail to apply Newtonian mechanics when adapted to new physics tasks. Further analysis reveals that these models behave as they develop task-specific heuristics that fail to generalize.
How would we know if foundation models have also made the leap from making accurate predictions to developing reliable world models? This paper develops a framework for answering this question. Specifically, we create procedure that, when given a foundation model and world model, tests whether the foundation model has learned that world model. We call this technique an inductive bias probe, and built on a simple insight: the implicit world model of a foundation model is revealed by how it extrapolates from small amount of information. This is inspired by how scientists use world models to make inferences from small amounts of data. Similarly, the inductive bias of a foundation model reveals its world model.
We first demonstrate this procedure using an example from physics. Specifically, we aim replicate Kepler's and New- ton's 's experiments, albeit replacing the physicist with foun- dation model of orbital mechanics. Much like Kepler, the model is able to predict orbital trajectories, even for solar systems it has not seen.
What would mean for this foundation model's inductive bias to be toward Newtonian mechanics? We demonstrate one tangible way test this: we fine-tune the foundation model on small dataset where the output is exactly the force vector (a comnersonn of Newtonian mechanics) at each point in the trajectory. If the foundation model's world model is toward Newtonian mechanics, it should have an inductive bias towards these force vectors. In contrast, Fig- ure 1 shows that the model produces poor force vectors. More extremely, when we perform this exercise at larger scale across many solar systems, the laws of gravity uses generalize bear no resemblance to Newton's 's law (Table l).
1. Introduction
The promise of foundation models relies on a central pre- sumption: that learning to predict sequences can uncover deeper truths, or optimistically, even world model. While this idea is new in one sense, it is old in another. Hundreds of years ago, astronomers like Kepler discovered geometric patterns that could pinpoint the future locations of planets in the night sky. Newton would later expand on this progress to develop Newtonian mechanies, fundamental laws that could not only predict the movement of planets but also explain physical properties across the universe (Koestler, 1959; Gin-
We further apply inductive bias probes in other domains with a known world model: lattice problems and Othello games (Liu et al., 2022; Hazineh et al., 2023; Nanda et al., 2023b; Vafa et al., 2024). Across these domains, we find that neural sequence models have weak inductive biases
'aavardd University PMIT. Correspondence to: Keyon Vafa <kvafa@ hdd>..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
Earth (true)
Earth (predicted)
Ea
Sun
Tru fxce
Earth
Sul
Prodiced fares
True force law (Newton)
Mi m2
F
Recovered force law (transformer)
F x sin
+ 1.45
sin (r - 0.24)
Mercury
Venus
Mars
Jupiter
Predided fance
Toce
Prodcted Exce
Tran Sanee
Prediceed Sron
Tree face
Preckded Syoe
Saturn
Uranus
Neptune
Preditedd lance
Figure 1. Each pair of panels illustrates the trajectory of planet in the solar system and its gravitational force vectors, comparing the true Newtonian forces (left) to the predicted forces (right) from transformer foundation model pretrained on orbital sequences and fine-tuned to predict forces. While the model excels at generating accurate predictions of planetary trajectories, it does not have an inductive bias toward true Newtonian mechanies; moreover, its force predictions recover nonsensical force law, as revealed by symbolic regression.
toward the given world models. We also highlight a practical implication: models that perform better on inductive bias probes have better performance when they'r re fine-tuned to perform new tasks that rely on the underlying world model.
note some output. A dataset D = ((21,,1),..... (In, } is a collection of n input-output pairs. A task f: X y is mapping between inputs and outputs.
Foundation models:
foundation model is learning al- gorithm which, when given dataset D, returns prediction function mp X y that relates the input to the outputs. Foundation models can take many forms; for example, mD could be some pre-trained model that is fine-tuned on the dataset D, or it can be an LLM that is supplied in-context.
Taken together, our results provide direction for under- standing the deficiencies of foundation models: if model's inductive bias isn't 't toward a known model of reality, what is toward? We explore this question by examining whether these foundation models have alternative inductive biases. Our analysis reveals that these models instead behave as if they develop task-speeific heuristics that fail to generalize. For physics, rather than learning law, the foundation model applies different, seemingly nonsensical laws depending on the task it's being applied to. In lattice and Othello, models have an inductive bias toward the set of legal next-tokens (e.g. a board's legal next moves) rather than the world model itself.
World model: A postulated world model summarized by state space & and a mapping Φ: X' & that associates each input with some state o(x) E Φ. A dataset D is consistent with the world model if for each (z, v) E D, the output is deterministic function of the state, = g()(x)) for some g y
universal physical
2.1. Comparing foundation models to world models
2. Framework
There is a challenge in defining what it means for foun- dation model to recover a world model: foundation models and world models operate in different spaces. A foundation model outputs a new predictive model from data, whereas world model describes state structure implicit in data.
In this section, we lay out our framework for evaluating whether a foundation model has learned a postulated world model. We develop an inductive bias probe, which is a procedure that evaluates the foundation model's behavior as adapts to new tasks.
One approach would be to mechanistically probe the foun- dation model, e.g. by comparing its weight-level representa- tions to the postulated states in the world model. However,
Data and tasks. Let X denote input and y de-
2

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
understanding the internal mechanisms of large models is challenging (Olah, 2022) and even then may not reflect how a model actually behaves on new data (Casper et al., 2023). Another approach is to study the model's behavior statically, on a single task (Toshniwal et al., 2022; Vafa et al., 2024), but this doesn't capture how foundation models are used in the real world: as tools for new tasks.
At high level, the probe repeatedly applies the foundation model to synthetic datasets consistent with the postulated world model and each time evaluates its predictions on held-out inputs. If the foundation model's inductive bias is towards the postulated world model, its extrapolations should have two properties. First, the foundation model's predictions should respect state; if two inputs æ, x' map to the same state (0(x) = o(2')) the foundation model should have the same predicted outputs (mp (2) mp when applied across synthetic datasets. If not, it means that the foundation model fits functions that do not belong to the world model. Second, the foundation model's predictions should distinguish state: if two inputs x; x' map to different states (1b(x) A d(x²)), the foundational model should typi- cally have different predicted outputs (mp(x) # mn ) across synthetic datasets. If not, then the foundation model does not fit functions that fully cover the world model's s allowable functions.
We take a different approach, motivated by the no-free- lunch theorem (Wolpert, 1996). Loosely speaking, the oo-free-lunch theorem states that no learning algorithm can perform better than another one on average if any function could have generated the data it is applied to. Given limited data, learning algorithms must extrapolate to unseen inputs, and if any underlying function possible, any such extrap- olation must be equally good or bad. This means that every learning algorithm is better for some collection of possible functions- - those functions that it tends to learn when ex- trapolating from limited data. The functions that a learning algorithm tends to learn represent its inductive bias.
These properties can be measured using two metrics. Let (y,, y') indicate whether y = y'. We specify a sampling distribution over consistent datasets D PD and a sam- pling distribution over inputs X)) ~ Px x Px. The foundation model's inductive bias towards respecting state (R-IB) is
The idea of inductive bias offers a connection between foun- dation models and world models. A world model a re- striction on the possible functions from inputs to outputs: only those that obey its state structure are possible. Conse- quently, foundation model that has learned a postulated world model should have an inductive bias towards functions that obey the world model's state structure. For example, physicists may train a foundation model on sequences of planetary orbits. Since planetary orbits obey Newtonian mechanics, they might hope the model has an inductive bias toward functions of Newtonian mechanics (e.g- predicting the force vector between two planets).
Ex Xx, (((mm/(()))........mm....)) (X)) ( = o(x)). (1)
R-IB measures the similarity between the model's 's extrap- olations on inputs in the same state under the postulated world model: higher R-IB indicates more similar predic- tions for the same states. The foundation model's inductive bias towards distinguishing state (D-IB) is
We develop an inductive bias probe for testing whether a foundation model's inductive bias matches the postulated world model's state structure. The inductive bias probe re- peatedly applies foundation model to synthetic datasets consistent with the world model and studies the extrapo- lated functions together (Figure 9). In each such simulation, we do not calculate the "accuracy" of the resulting extrap- olations since there is no one accurate function; multiple ways to extrapolate may be allowed by the true world model. Instead, we evaluate whether the extrapolations resemble those that are allowed by the true world model.
1 E (m
I ((X.) + (Xx)),
D-IB measures whether inputs that belong to different states under the postulated world model nonetheless receive consis- tently similar predictions by the foundation model: higher D- IB indicates more dissimilar predictions for different states. Figure 10 illustrates both metrics.
Together, R-IB and D-IB provide contrasting perspectives foundation model's implicit world model, analogous to precision and recall in binary classification. For example, while it is trivial for a foundation model to achieve high R- IB by making the same prediction on every input, its D-IB will suffer. Both metrics are needed to contrast a foundation model's inductive bias with the postulated world model.
2.2. Special case: finite state space and binary outputs.
To provide more intuition for the inductive bias probe, we first consider the special case of a binary output ) = (0, 1} and a postulated world model with finite state space 4. The two metrics we introduce in this setting special cases of the general inductive bias probe defined in the next section.
In this sense, the inductive bias probe captures behavior of foundation model that is not captured by standard probe tests (Nanda et al., 2023b), which measures how well a sim- ple predictive model (e.g., a linear model) can predict state from a foundation model's intermediate representation. By contrast, the inductive bias probe directly analyzes how the foundation model behaves when adapted to synthetic tasks
The inductive bias probe evaluates whether a foundation model's 's inductive bias is towards a postulated world model.
3

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
from the postulated world model. When there are many distinct state mappings that are predictable from a founda- tion model's internal representation, standard probes can- not distinguish which is actually being used by the model. Moreover, the standard probe is sensitive to how state is mechanistically represented by the chosen world model. For example, Nanda et al. (2023b) find that different representa- tions of the Othello game board (one based on the standard board and another that inverts the board based on whose turn it is) lead to different results by standard probes. By contrast, because inductive bias probes only depend on state equality, they are insensitive to equivalent representations.
values of extrapolative predictability indicate higher lev- els of predictability. If a foundation model behaves as if it extrapolates based on the postulated world model, the extrapolative predictability should be larger for inputs with more similar states.
Oracle foundation model. As a calibration, we calculate the extrapolative predictability for an "oracle" foundation model that is given access to the true state space and admissible functions G. When applied to consistent dataset D, the oracle foundation model returns
D = arg min 9Eg Toj E (g(((x;)); ui). (4)
To implement the inductive bias probe, a practitioner must supply a sampling distribution over consistent datasets Pn and a sampling distribution over inputs Px. In our ex- periments with finite state space and binary outputs (see Section 4), we sample consistent datasets by assigning each unique state the output 0 or 1 uniformly at random.
(The loss function used here need not be the same as the loss function used to calculate extrapolative predictability.) The oracle extrapolative predictability is
(x), = min ED- (eoo:(((((m(()))))) (z))))) (5) NEW
2.3. Inductive bias probe
Inductive bias towards the world model. The inductive bias probe compares the foundation model's extrapolative predictability to that of the oracle. Specifically, the foun- dation model's s inductive bias towards the world model is defined as, for any 0 SS53,
We now describe the inductive bias probe allowing for gen- eral outputs, state spaces, and tasks. For example, for se- quences of two planets orbiting one another, the states could correspond to their relative positions, relative velocities, and the masses of each planet under Newtonian mechan- ics. We further introduce a collection of admissible func- tions state G that govern the relationship between the state space and the output under the world model with each g E G: t + y' For example, in some settings, we may expect the output to vary smoothly with the state, in which case G could be the collection of K K-Lipschitz functions. A dataset is now consistent with the world model if for each (x., v) E D. y = g(((z)) for some g E G.
IB (s, 3) Ex,x, [X,,, Xx)
X,) s 3].
(6)
We calculate this quantity over a grid of values so < $1 .... Sm. visualizing how IB(s, 8) varies over the grid. The foundation model's inductive bias towards the world model can be interpreted like a calibration curve: if the foundation model behaves like the oracle when applied to many small datasets, then (B 3) should lie on the 45- degree line in this visualization (as illustrated in Figure 2).
Given a sampling distribution over consistent datasets Pn and sampling distribution over inputs Px, the inductive bias probe repeatedly applies the foundation model to sam- pled datasets, and then evaluates its predictions on held- out inputs. It measures how predictable the foundation model's predicted outputs for one input are from those of another input across many synthetic datasets. The intuition is unchanged: inputs in "similar" states should be more predictable from one another than inputs from "different" states. We next formalize this property.
R-IB and D-IB are special cases of Equation 6. Consider the case in which the output is binary, 4 is finite, and G is the col- lection of all mappings. Provided PD places positive proba- bility on all possible consistent datasets and H is limited to the identity function, there are only two possible values for the oracle's extrapolative predictability, which occur when (z2)) = o(x,)) and when Φ(x:) # p(2,). Consequently, the foundation model's inductive bias towards the world model reduces to R-IB in the former case (Equation 1) and D-IB (up to sign change) in the latter case (Equation 2).
Extrapolative predictability. We further specify a family of predictors H with h E H such that h y + y and a loss function over outputs f ) x y > R... We define the extrapolative predictability between two inputs as
3. Orbital Mechanics
We illustrate these ideas by testing whether a transformer trained predict the locations of planets in motion has recovered Newtonian mechanics.' We first train model to predict the location of planets across solar systems.
[(x,, = minn NER
m) (3)
which measures how predictable the foundation model's predicted outputs for one input are from the other. Higher
OOur code is available at https://github.com keyonva deyonvafa/induuse indunndiiiii ive ve-bi biss- aasppoose.
4

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
Lincar amde
MLP oracle
initial relative velocities) to target the shape of orbits ob- served in known exoplanets (Kipping. 2013). We simulate each planet's trajectory around the sun using Newton's 's laws of motion; because planet masses are much smaller than the sun's, uun's, planet interactions are minimal, so we omit them.
Oracie incuclivv biss
Orazia industivv biess
to
Matches oracle
Matchee orade
To convert orbits into sequences, we record (a, y) coordi- nates of each planet and the sun across regular intervals, and interleave all the positions into a single sequence with 1,000 observations. This means that each sequence denotes a different solar system. We randomly sample half of the sequences to use 6-month time intervals between observations and use 1-week time intervals for the other half, using a special token at the beginning of the sequence to indicate the interval length. For example, in a solar system with K planets, the first timestep encodes the interval length, the next K observations are the (z, v) coordinates for each planet at the first point in time, and the next K are the coordinates for each planet the appropriate timestep later, etc. (We also considered using fixed-length intervals and found similar results.) We use a training set of 10M sequences and 20B tokens.
Transfomer inductiva blas
Transfomar inductive blas
Figure 2. Inductive bias probe performance (Equation 6) for a trans- former pretrained on orbital trajectories. A 45-degree line would indicate perfect inductive bias toward an oracle that extrapolates based on the Newtonian state vector.
Despite the model's ability to accurately predict the future trajectories of planets, the inductive bias probe reveals that it has low inductive bias toward Newtonian mechanics. This is corroborated by the fact that when the model is fine-tuned to predict planet's force vector - a cornersonne of Newtonian mechanies - its predictions imply nonsensical law of gravitation. We find that the model has recovered piecemeal heuristics rather than a compact world model; it recovers different laws of gravitation on different data slices is applied to.
We train a 109M parameter transformer (Vaswani al., 2017) to predict the next token of each sequence in the train- ing set. We experimented between using a) continuous coor- dinates (and MSE loss) and b) discretized coordinates (with cross-entropy loss), finding the latter worked better. We dis- cretize each position vector of each body in the solar system by creating 7K bins per coordinate (zz, y), where the coordi- nates spans from -50 to 50 AU. We train for 25 epochs using 8 H100 GPUs. See Appendix A for more training details.
Background. For centuries, astronomers and physicists have worked predicting the orbits of planets around the sun. A groundbreaking model was offered by the astronomer Johannes Kepler in the 17th century. His model was based on geometric patterns: for example, that the orbit of each planet followed an ellipse with the sun at one of its foci. While the model could predict orbits with a near-perfect level of precision, it couldn't 't explain why the planets obeyed these geometric orbits or be applied to new problems beyond predicting trajectories.
We evaluate model predictions on held-out data. The model makes good predictions: its R² is above 0.9999, and it sig- nificantly outperforms baseline models that always predict the most recent position or the per-orbit mean (Table 8). can also generate long orbits with a high degree of accuracy.
Later, Isaac Newton expanded on this model using new laws of motion, now known as Newtonian mechanics. These laws involved computing properties of the set of planets in motion, such as their relative velocities and masses. Using these properties, he could derive Kepler's earlier laws for orbital trajectories, but also go beyond, understanding and formalizing other concepts like force and gravity.
Has the model recovered Newtonian mechanics? The transformer's s predictions reflect very good sequence model. But has it recovered Newtonian mechanics? To test this, we note that Newtonian mechanics dictate that each observation in a sequence of orbits is governed by a state vector consisting of the masses, relative velocities, and relative positions each planet. Given the current state of a trajectory, the next position of an orbit is deterministic. This is our world model; if foundation model's 's inductive bias depends on Newtonian mechanics, it must be extrapolating based on this state vector.
From Kepler to Newton, scientists were able to move beyond good predictive models of sequences to a deeper understand- ing of them. In this section, we test whether a transformer that can predict sequences of orbital trajectories is merely good sequence model, or whether it has also made the transition to providing a world model.
We use the inductive bias probe described in Section 2 to assess the model's inductive biases. We create 100 synthetic datasets where the outputs are linear functions of the state of the sequence. We then fine-tune the transformer by training it to predict these functions. We measure
Data and pre-training. We first simulate a dataset of sequences, where each sequence describes planets in motion around a sun. To do this, we randomly sample initial condi- tions (e.g. the masses and positions of the planets and their
5

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
122
et al., 2020). When the symbolic regression is applied to the transformer's predictions, the physical law is nonsensical (Figure 1). In contrast, an oracle trained on the true state predicts the force vectors well and symbolic regression recovers the true physical law (Figure in Appendix C). See Appendix C for implementation details and Appendix D for a similar experiment with LLMs.
Ground-truth law
F
r²
Galaxy 1
F (it)(
1145)
7
Galaxy 2
F x
Estimated laws
Galaxy 3
F oc cos cos(sin( sin( (im( (.:t)))
Galaxy 4
F x sin(r + 8509.2 +
Galaxy 5 F x cod(ros(e"
How can a model perform so well at predicting orbit loca- tions without having inductive biases towards the laws of physics that govern them? We study this question by apply- ing the fine-tuned model's force predictions to five different sets of randomly sampled galaxies (each consisting of many solar systems). We then perform a symbolic regression on the force magnitude for each sample. The symbolic regres- sion finds a different implied law of gravitation for each sample (Table 1). In contrast, the oracle trained true state recovers the same (correct) law for each galaxy. These results show that rather than building single universal law, the transformer extrapolates as if it constructs different laws for each sample.
Table 1. Force equations recovered via symbolic regression of transformer pretrained on orbital data and fine-tuned to different galaxy samples. The model recovers different equations for each sample, never recovering the true law.
the model's extrapolative predictability across inputs (Equation 3) by considering H consist of the identity and the loss function E to be MSE. We evaluate Equation 6 by comparing the model to an oracle that extrapolates based on state directly (we consider both linear models and 2-layer neural networks for the oracle, finding similar results). The inductive bias toward simple functions of Newtonian state poor; see Figure? for a visualization, In other words, the model's inductive bias is not toward Newtonian state; when it has to extrapolate, it makes similar predictions for orbits with very different states and different predictions for orbits with very similar states. For implementation details and ablations, see Appendix B.1.
4. Other Applications
We now apply the inductive bias probe to evaluate the extent to which foundation models obey known world models in other domains. Evaluating world models requires studying domains where there's a state structure and ground-truth state is known. We study two such types of datasets: lattice problems and the board game Othello.
To understand the degree to which the model fails to apply Newtonian mechanics, we test its ability to predict spe- cific quantities derived from Newtonian mechanics. Specifi- cally, we consider each planet's force vector, simple trans- formation of state given by Newton's 's law of gravitation: F F=C -G and the sun to their masses m1, m2 and their squared dis- tance ||r|| (in the direction e, of its relative position). The force vector can be computed for each observation in se- quence; force is a simple transformation of state, so the predictions of a model that has recovered Newtonian me- chanics should obey this law.
Lattice. One common type of structure to assess models against is spatial structure, or lattices (Vafa et al., 2024; Liu et al., 2022). We study lattice setting that simulates an agent moving along a line segment with a finite number of positions. There is a true state space consisting of $ states: + = (1, 2, S}. The language x consists of sequences with three tokens: Σ- = {L, -, R). The initial state of the sequence is 1. For a token σ = R, the state increases by 1, while the state decreases by for
||<|| er, which relates the force F between planet
L and stays the same for = When the state is 1, the state is at the boundary, so α = L is not a valid token; similarly, when the state is S, a = R is not a valid token. All tokens are valid for all other states. We randomly generate sequences of length 100 over the language by sampling a move uniformly at random over the set of valid moves for each timestep. We consider different versions of the lattice problem, varying the number of states from 2 to 5. We consider sequences taken from a training set containing 10M tokens, along with 100k hold-out tokens.
We test this by creating a sequence-tor to-suueee-- dataset where each input is trajectory and each output is the force vector F on the planet implied by the state of the orbit. We first fine-tune the pretrained transformer to predict the force vector on orbits from our solar system, providing 1% of the true forces as training data. Figure shows these force predictions are poor. To assess how close the model to recovering Newton's 's law of gravitation, we further fine-tune it to predict the force magnitude solar systems. We then perform a symbolic regression (us- ing the PySR software (Cranmer, 2023)) of the predicted force magnitudes on the true values of m, m2, and r. A symbolic regression is a method to search for a symbolic ex- pression that optimizes a regression-like like objective (Cranmer
larger dataset of 10K
Othello. We also study the board game Othello, a common testbed for evaluating the world models of sequence models (Li et al., 2023; Nanda al., 2023b; Hazineh et al., 2023; Vafa et al., 2024). The game consists of two players taking
6

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
Lattice (5 States)
Othello
Pre-training
R-IB (1)
D-IB (t)
R-IB (t)
D-IB (1)
RNN (Elman, 1990)
Untrained
0.346 (0.026)
0.749 (0.027)
0.228 (0.016)
0,990 (0.002)
NTP trained
0.574 (0.026)
0.803 (0.032)
0.632 (0.023)
0.797 (0.023)
LSTM
Untrained
0.456 (0.028)
0.718 (0.031)
0.438 (0.030)
0.681 (0.031)
(Hochreiter, 1997)
NTP trained
0.782 (0.021)
0.921 (0.030)
0.563 (0.030)
0.610 (0.034)
Transformer
Untrained
0.268 (0.027)
0.742 (0.028)
0.708 (0.022)
0.843 (0.021)
(Vaswani et al., 2017)
NTP trained
0.483 (0.031)
0.677 (0.034)
0.703 (0.025)
0.624 (0.033)
Mamba (Gu & Dao, 2023)
Untrained
0.260 (0.026)
0.771 (0.027)
0.303 (0.016)
0.929 (0.009)
NTP trained
0.571 (0.023)
0.866 (0.029)
0.682 (0.021)
0.728 (0.027)
Mamba-2
Untrained
0.244 (0.026)
0.785 (0.026)
0.468 (0.019)
0.896 (0.016)
(Dao & Gu, 2024)
NTP trained
0.617 (0.021)
0.864 (0.029)
0.653 (0.022)
0.694 (0.029)
Table The inductive bias towards respecting state (R-IB) and inductive bias towards distinguishing state (D-IB) metries (1 perfect performance, 0 is equivalent to noninformative model). "NTP-traines represents model pre- pre-trained on next-token prediction, while "untrained" refers to a model trained on the same synthetic tasks, initialized from scratch.
R-IB as function of states
D-IB a function of states
Toshniwal et al. (2022), we measure the fraction of a model's top predictions that are legal in the underlying state. Table 7 in Appendix G shows the results. All models do very well across all datasets, e.g. every model's top prediction is legal == 90% of the time for Othello and legal 100% of the time for a lattice problem with five states.
1.0
0.8
0.8
0.6
RNN
0.6
RNN
LSTM
LSTM
Mamba
Mambaa
0.4
Mambb-?
Memba-2
Inductive bias probe results. We measure each model's inductive bias using the procedure from Section 2 to assess the inductive bias of these models. The procedure involves fine-tuning each model to small datasets of randomly gen- erated outputs and assessing whether the model's inductive bias as measured by its extrapolations - obeys state structure. We use the discrete version of the procedure for both models.
Transfomer
Transfomer
2
Number of states
Number states
Figure 3. Inductive bias probe results (R-IB and D-IB) for the lattice problem as a function of the underlying number of states, A different model pre-trained on data consistent with each number of states and its inductive bias for that state structure is recorded using the metrics in Section
The results for the lattice problem are depicted in Figure 3. While models have high inductive biases when the number of states small, as the number of states increases, the in- ductive biases drop off. Notably, the transformer model con- sistently does worse than the other models, all of which have architectures based on recurrent or state-space models. The results for Othello are depicted in Table 2. Here, all models perform worse than on the lattice problems, indicating poor inductive bias. Despite generating legal moves nearly 100% of the time when pretrained to play Othello, these models don't 't use the board as inductive bias on new tasks.
turns placing tiles on an 8x8 board. Each game of Othello is tokenized into a sequence of at most 60 moves, where each token indicates which of the 60 squares the most recent tile was placed on (the middle four tiles are always occupied). The true state space & corresponds to all 8x8 boards and the mapping & converts game sequences into states. We con- sider game sequences taken from training set containing 20M games, along with 3.8M hold-out games.
We study the properties for five classes of pre- trained sequence models: RNNs (Elman, 1990), LSTMs (Hochreiter, 1997), transformers (Vaswani et al., 2017), Mamba (Gu& & Dao, 2023), and Mamba-2 (Dao & Gu, 2024). We train each model using next-token prediction for each domain. By way of comparison, we also compare these pretrained models to untrained models that fine-tune from a random initialization. See Appendix A for details.
Models.
To understand the implications of these results, we study how different models transfer to new functions of state (the board). Specifically, we take the Othello dataset and con- struct new sequence-to-sent to-sequence datasets. The input se- quence for each dataset is the original game transcript, and we consider three different output transformations that are functions of state. In "Majority Tiles", each element of the output is I or 0 indicating where there are more black or white tiles in the board implied by the sequence so far. In "Board Balance",, each element of the output sequence
All pre-trate preitrdined models perform well at next-token predic- tion, generating outputs that appear to obey state. Following
7

---

What Has Foundation Model Found?
Using Inductive Bias to Probe for World Models
True board
Predicted board
Mamba
Mamba-2
1.00
Accuracy
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
0
2500
5000 7500
10000
2500
5000
7500
10000
Transfer steps
Transfer steps
Set va moves match At east one move predicted board valid
Legal next move
X Incorrect tile prediction
All moves in predicted board are valid
Full board matches
Figure 4. On the left, a true Othello board implied by a sequence, and on the right, the predicted board from a model fine-tuned to predict boards. Although the prediction has errors, the of predicted next tokens exactly matches the true board. On the right, metrics about board reconstruction during fine-tuning, Consistently, even as Mamba models struggle to recover full boards, they recover them well enough such that the sets valid next moves match those in the true boards.
indicates whether black has more pieces the top half of the board in the bottom half of the board. Finally, in "Edge Balance", the output measures whether black has more pieces along the edge squares of the board. Each of these functions is deterministic function of state (the board), so foundation models that have inductive bias toward state should be better at transfer. The results are depicted in Table The last row shows the (unsigned) correlation for each metric and the ratio, I-D-IB that summarizes the induc- tive bias measures in Table 2. There strong correlation across all metrics; models that do better inductive bias metrics transfer better to these functions of state.
frequently matches the set of legal moves from the true board. Rather than recovering the full board, the foundation model often recovering "enough of"" the board calculate legal next moves.
To quantify this hypothesis generally, we modify the in- ductive bias probe to test whether a model's inductive bias is toward next-token partitions of state. Recall that D-IB measures how similar extrapolations for two points with different states are one from another. If a model is extrap- olating based which next-tokens are legal, sequences in different states that happen to have the same legal next tokens will have more similar predictions than sequences in different states that have different legal next tokens.
What are the inductive biases? These results show that models can perform well at predicting token sequences with- out appearing to learn the underlying world model. This raises the question: If foundation model's inductive bias isn't toward given world model, what is toward?
Specifically, let q denote the next-token coarsening of the state space such that g(x) NextTokens(@((z))
((x') if and where
only if NextTokens(o(x:) NextTokens(s) is the set of valid next tokens for state s. We decompose D-IB into two quantities. First, de- fine Same(X, X,) as the event that o(X,) + o(X,) but ((x,) = (. We then define,
Here, we consider one hypothesis motivated by the next- token pretraining objective: that when foundation models are applied to new tasks, they group together sequences with distinct states for which the set of legal next tokens are nevertheless equivalent. For example, in the board game Othello, two distinct boards can have the same of al- lowable next moves. Therefore, model's inductive bias might be toward boards with the same sets of allowable next moves rather than the true board itself.
D-IB, == = 1 - 1 [I (i)(x(()) mD(X))) Same Same(X,, (X,,X,) X,)]
which measures how predictable the extrapolations for in- puts associated with different states that have the same legal next tokens are. Similarly, define Diff(X;, X,) as the event that ((X.) + ((X,) and ((.)) + ((x)). Analogously,
To first demonstrate this concept with Othello, we fine-tune a foundation model originally pretrained to perform next- token prediction on IM games to now predict the true board of each sequence. We record metrics when we fine-tune: 1) whether the predicted board exactly matches the true board, and 2) whether the set of valid moves in the predicted board matches the set of valid moves in the true board. The results are depicted in Figure 4: surprisingly, even when the predicted board is incorrect, the set of legal moves
D-IB 4 = 1 - E (m X.), mx(x)) Diff(X,, X,)],
which measures how predictable the extrapolations for in- puts associated with different states that have different legal next tokens are. If distinct-state inputs with the same legal next tokens are more predictable than distinct-state inputs with different legal next tokens (i.e., D-IB,- g= < D-IB,y)) then it suggests the model extrapolates based on the next- token partition rather than the true board state.
8

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
We compute these refined metrics for lattice and Othello. Each has a natural definition of legal next moves (cor- responding to boundaries and game rules). The results are depicted in Table 9. For all models, the gap between D- and of is statistically significant, suggesting that models are grouping together distinct states with the same sets of legal next tokens.
knowledge constitutes a robust world model.
The methodology in this paper is rooted in the observation that generative models can reach the same generated outputs in different ways. This is related to the Rashomon effect (also referred to as model multiplicity) for predictive models, where there can exist many different models that achieve similar performance on a predictive task (Breiman, 2001; Marx et al., 2020; D' Amour et al., 2022; Black et al., 2022). The literature on causal representation learning suggests that models should learn representations corresponding to the true causal mechanisms to generalize robustly to new tasks (Schölkopf et al., 2021; Bengio et al., 2019). Our method is based on a similar motivation, as it studies the properties of a foundation model not by its generations for one task but rather its inductive bias. A model that has learned a true world model should find new tasks "informationally close" (Achille et al., 2021) and adapt easily.
5. Related Work
This paper studies whether predictive models form world models (LeCun, 2022). One strand of world model research studies whether the outputs of a fixed model accord with a known world model by studying the fixed model's outputs (Vafa et al., 2024). For example, one way that Toshniwal et al. (2022) and Li et al. (2023) study world models is by assessing whether a model trained on sequential game data always predicts legal moves in the underlying game. The question we study is a different yet related question: rather than studying the world model properties of a fixed model, we study what it means to test if a learning algorithm- - a foundation model- - has a world model embodied in it.
Recent work developing foundation models in scientific do- mains such as protein folding, gene regulation, and molecu- lar chemistry (Chowdhury et al., 2022; Benegas et al., 2023; Boiko et al., 2023; Jablonka et al., 2024) use predictive mod- els stepping stones toward uncovering deeper principles. Our orbital mechanics example relates specifically to the large body of work studying AI and physics (Hao et al., 2022; Wu & Tegmark, 2019). It is most closely related to works studying whether Al models can uncover physical laws (Chen et al., 2022: Belyshev et al., 2024; Kansky et al., 2017; Udrescu & Tegmark, 2020; Iten et al., 2020). Most closely to us, Lemos al. (2023) demonstrate that Newton's 's gravitational law can indeed be recovered from a graph neu- ral network trained on orbital data by modifying the model architecture to explicitly impose Newton's 's laws of motion as inductive biases. We find that a transformer without New- ton's inductive biases does not recover the gravitational law, but imposing domain-speciffe inductive biases is a promis- ing approach to improving these models. We adopt general tools from this literature such using symbolic regres- sions for interpretability - to study the inductive biases of a algorithms (Liu & Tegmark, 2021; Wu & Tegmark, 2019).
Another strand of the literature assesses whether model's parametric representations encode world models (Abdou et al., 2021; Patel & Pavlick, 2022; Gurnee & Tegmark, 2023; Nanda et al., 2023a). For example, common method uses probes or sparse autoencoders (SAFs) (Cunningham et al., 2023; Trenton Bricken et al., 2023) to assess whether an intermediate representation used by a neural network is predictive of state (Hewitt & Liang, 2019; Li et al., 2021; Abdou et al., 2021; Jin & Rinard, 2023; Li et al., 2023; Spies et al., 2024; Karvonen, 2024). However, there are open questions about the reliability of probes (Belinkov, 2022), such as appropriate function complexity (Alain & Bengio, 2018; Cao et al., 2021; Li et al., 2023). Our method sidesteps these issues by asking how a model learns, rather than what's 's encoded in its fixed representations. Closely related to us, jylin04 et al. (2024) and Nikankin et al. (2024) find that a GPT model trained on Othello and math tasks, respectively, performs internal computations corresponding to "bags of heuristics" rather than a coherent world model. While our procedures differ in aim, these findings support our analysis of the Othello model relying on heuristics, rather than state, its inductive bias (McCoy et al., 2019).
6. Conclusion
The promise of foundation models that sequence predic- tion can uncover deeper understanding of underlying mech- anisms. We develop framework for evaluating whether foundation model has learned a postulated world model by measuring its inductive biases when transferring to new tasks. Our empirical results reveal that while many sequence models excel at next-token prediction tasks, they often have limited inductive bias toward genuine world models. Rather than learning coherent world models, we find that these models may be relying on coarsened state representations or non-parsimonious representations.
Complementary to the internal focus of mechanistic inter- pretability, other research uses behavioral probes to study a model's 's ability to synthesize knowledge, methodology closer to our own. For instance, recent work demonstrates that LLMs can infer and internalize latent knowledge from disparate information seen during training, and then ap- ply this inferred knowledge downstream tasks (Berglund et al., 2023; Treutlein et al., 2024). Our inductive bias probe provides framework for testing whether such emergent
9

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
Acknowledgments
Boiko, D. A., MacKnight, R., Kline, B., and Gomes, G. Au- tonomous chemical research with large language models. Nature, 624(7992):57--578 2023.
Keyon Vafa is supported by the Harvard Data Science Ini- tiative. Peter Chang is supported by the NSF CSGrad4US
Fellowship.
Breiman, L.
Statistical modeling: The two cultures (with
comments and a rejoinder by the author). Statistical science, 16(3):199- 2-11 2001.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Cao, S., Sanh, V., and Rush, A. M. Low-complexity probing via finding subnetworks. arXiv preprint arXiv:2104.03514, 2021.
Casper, S., Bu, T., Li, Y., Li, J., Zhang, K., Hariharan, K., and Hadfield- Menell,, D. Red teaming deep neural networks with feature synthesis tools. Advances in Neural Information Processing Systems, 36:80470-80516, 2023.
References
Abdou, M., Kulmizev, A., Hershcovich, D., Frank, S., Pavlick, E., and Søgaard, A. Can language models encode perceptual structure without grounding? A case study in color. arXiv preprint awwiv:2209.0611111111 2021.
Chen, B., Huang, K., Raghupathi, S., Chandratreya, I., Du, Q., and Lipson, H. Automated discovery of fundamental variables hidden in experimental data. Nature Computa- tional Science, 2(7):433-444 2022.
Achille, A., Paolini, G., Mbeng, G., and Soatto, The information complexity learning tasks, their structure and their distance. Information and Inference: Journal of the IMA, 10(1):51-72, 2021.
Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., Sorger, K., and AlQuraishi, M. Single-sequence protein structure prediction using lan- guage model and deep learning. Nature Biotechnology. 40(1 40(11):1617-1623 2022.
Alain, G. and Bengio, Y. Understanding intermediate lay- ers using linear classifier probes. 2018. arXiv preprint arXiv:1610.0164. 2018.
Belinkov, Y. Probing classifiers: Promises, shortcomings, and advances. Computational Lingusstics, 48( (8)1)2207-,,, 2022.
Cranmer, M. Interpretable machine learning for science with pysr and symbolicregression. jl. arXiv preprint arXiv:2305.015553 2023.
Belyshev, A., Kovrigin, A., and Ustyuzhanin, A. Beyond dynamics: Learning to discover conservation principles. Machine Learning: Science and Technology, 5(2)0025055, 2024.
Cranmer, M., Sanchez Gonzalez, A., Battaglia, P., Xu, R., Cranmer, K., Spergel, D., and Ho, S. Discovering symbolic models from deep learning with inductive bi- ases. Neural Information Processing Systems, 33:17429- 17442, 2020.
Benegas, G., Batra, S. S., and Song, Y. S. DNA language models are powerful predictors of genome-wiee variant effects. Proceedings of the National Academy of Sciences, (00(44)x2311219111 2023.
Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly inter- pretable features in language models. arXiv preprint 2rX.v:2309.08600, 2023.
Bengio, Y., Deleu, T., Rahaman, N., Ke, R., Lachapelle, S., Bilaniuk, O., Goyal, A., and Pal, C. A teeta-transfer objective for learning to disentangle causal mechanisms. arXiv preprint arXiv: /901. 10912, 2019.
D' Amour, A., Heller, K., Moldovan, D., Adlam, B., Ali- panahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 23(226):1-61, 2022.
Berglund, L., Stickland, A. C., Balesni, M., Kaufmann, M., Tong, M., Korbak, T., Kokotajlo, D., and Evans, O. Taken out of context: On measuring situational awareness in llms. arXiv preprint arXiv:2309.00663 2023.
Dao, T. and Gu, A. Transformers SSMs: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21061 2024.
Black, E., Raghavan, M., and Barocas, S. Model multiplic- ity: Opportunities, concerns, and solutions. In ACM Con- ference on Fairness, Accountability, and Transparency, pp. 850 063, 2022.
Elman, J. L. Finding structure in time. Cognitive science, 14(2):179-211, 1990.
10

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
Gingerich, O. The book nobody read: Chasing the revolu- fions of Nicolaus Copernicus. Bloomsbury Publishing USA, 2004.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988) 2014.
Kipping, D. M. Parametrizing the exoplanet eccentricity distribution with the beta distribution. Monthly Notices of the Royal Astronomical Society: Letters, 434(1): L55, 2013.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00772, 2023.
Koestler, A. The Sleepwalkers: A History of Man 's Chang- ing Vision of the Universe. Hutchinson, London, 1959. First edition.
Gurnee, W. and Tegmark, M. Language models represent space and time. arXiv preprint arXiv:2310.02207, 2023.
Hao, Z., Liu, S., Zhang. Y., Ying, C.,, Feng. Y., Su, H., and informed machine learning: A survey
LeCun, Y. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. OpenReview, 62(1):1-62, 2022.
Zhu, J. on problems, methods and applications. arXiv preprint arXiv:2211.0806/ 2022.
Lemos, Jeffrey, N., Cranmer, M., Ho, S., and Battaglia, P. Rediscovering orbital mechanics with machine learning. Machine Learning: Science and Technology, 4(4):045002, 2023.
Hazineh, D. S., Zhang, Z., and Chiu, J. Linear latent world models in simple transformers: A case study on Othello- GPT. arXiv preprint arXiv:2310.07582, 2023.
Hewitt, and Liang, Designing and interpreting probes with control tasks. arXiv preprint arXivv190..03368, 2019.
Li, B. Z., Nye, M., and Andreas, J. Implicit representations of meaning in neural language models. arXiv preprint arXiv: WViv:2106.0073:: 2021.
Hochreiter, S. Long short-term memory. Neural Computa- tion MIT-Press, 1997.
Li, K., Hopkins, A. K., Bau, D., Viégas, F,, Pfister, H., and Wattenberg, M. Emergent world representations: Exploring a sequence model trained on a synthetic task. In International Conference on Learning Representations, 2023.
Iten, R., Metger, T., Wilming, H., del Rio, L., and Renner, R. Discovering Physical Concepts with Neural Networks. Physical Review Letters, 124(1):010500 January 2020. doi: 10. 1..11000yyseeeeett...m 124.01000..2.000050..
Liu, B., Ash, J. T., Goel, S., Krishnamurthy, A., and Zhang, C. Transformers learn shortcuts to automata. arXiv preprint arXiv:2210.1077 2022.
Jablonka, K. M., Schwaller, P., Ortega- Guegreroo A., and Smit, B. Leveraging large language models for predictive chemistry. Nature Machine Intelligence, pp. 1-9, 2024.
Liu, Z. and Tegmark, M. Ai poincare: Machine learning conservation laws from trajectories. arXiv:201 1.04698, 2021.
Jin, C. and Rinard, M. Evidence of meaning in lan- guage models trained on programs. arXiv preprint arXiv:2305.1 11169, 2023.
Marx, C., Calmon, F., and Ustun, B. Predictive multiplicity in classification. In International Conference on Machine Learning, pp. 6765-67777 66744, PMLR, 2020.
jylin04, JackS, Karvonen, A. and Rager, C. Othellogpt learned bag of heuristics, jul 2024. URL https://www. www.wesswrong.
McCoy, R. T., Paylick, E., and Linzen, T. Right for the wrong reasons: Diagnosing syntactic heuristics natural language inference. In Proceedings the Association for Computational Linguistics, 2019.
wwwllloopt-eeerneee a -- bag- of heurisstttsss-11
Posted on LessWrong.
Nanda, N., Chan, L., Lieberum, T., Smith, J., and Stein- hardt, J. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217 2023a.
Kansky, K., Silver, T., Mély, D. A., Eldawy, M., Lázaro- Gredilla, M., Lou, X., Dorfman, N., Sidor, S., Phoenix, S., and George, D. Schema Networks: Zero-shot transfer with a cenerative causal model of intuitive physics. In International Conference Machine Learning, pp. 1809- 1818. PMLR, July 2017.
Nanda, N., Lee, A., and Wattenberg. M. Emergent linear rep- resentations in world models of self self-super sequence models. arXiv preprint 2rxiv:2309.00941, 2023b.
2023b.
Nikankin, Y., Reusch, A., Mueller, A., and Belinkov, Y. Arithmetic without algorithms: Language models solve math with a bag of heuristics. arXiv preprint arXiv:2410.11222..22 2024.
Karvonen, A. Emergent world models and latent vari- able estimation in chess-playynn language models. arXiv preprint arxi::2403. 15498, 2024.

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
Olah, C. Mechanistic Interpretability, Variables, and the Im-
portance of Interpretable Bases. -tths//www..raanffomerr-- June 2022.
Patel, and Pavlick, Mapping language models to grounded conceptual spaces. In International Conference Learning Representations, 2022.
Schölkopf. B., Locatello, F., Bauer, S., Ke, N. R., Kalch- brenner, N., Goyal, A., and Bengio, Y. Toward causal representation learning. Proceedings of the IEEE, 109(5): 612-634, 2021.
Spies, A. F., Edwards, W., Ivanitskiy, M. I., Skapars, A., Räuker, T., Inoue, K., Russo, A., and Shanahan, M. Trans- formers use causal world models in maze-solving tasks.
arXiv preprint arXiv:2412.11800 2024.
Toshniwal, S., Wiseman, S., Livescu, K., and Gimpel, K. Chess as testbed for language model state tracking. Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 11385-11393, 2022.
Trenton Bricken, Adly Templeton, B. C. et al. To- wards Monosemanticity: Decomposing Language Models With Dictionary Learning. https:/transformmr- circuits. Srtuiiipppb20022/mmmmmmmiiiiieee htw.am/mmiietttttttei/ne....iiee 2023.
Treutlein, J., Choi, D., Betleeteyyy. J., Marks, S., Anil, C., Grosse, R., and Evans, O. Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data, December 2024.
Udreseu, S.-M. and Tegmark, M. Al Feynman: A physics- inspired method for symbolic regression. Science Ad- vances, 6(16)) April 2020. doi: 10. 10.1126/sc.adv. aay2631.
Vafa, K., Chen, J. Y., Kleinberg. J., Mullainathan, S., and Rambachan, A. Evaluating the world model implicit in a generative model. Neural Information Processing Systems, 2024.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, t.,, and Polosukhin, I. Atten- tion is all you need. In Neural Information Processing Systems, 2017.
Wolpert, D. H. The lack of a priori distinctions between learning algorithms. Neural Computation, 8(7) 1390. 10 1996. ISSN 0899-76677. doi: 10.1162/neco. 1996.8.7.1341. URL https: httl:s///doo.eee org/10..000.. http:/:///o../o//..//// 1162/ neco 1996.999999 8 1341.
Wu, T. and Tegmark, M. Toward an artificial intelligence physicist for unsupervised learning. Physical Review E, 100(3):033311, 2019.
12

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
A. Model and Training Details
We use the following specifications for each model:
RNN (Elman, 1990): For Othello, We use 6 uni-directional RNN layers with 768 embedding dimensions. For the lattice experiments, the architecture is the same except we use only 2 layers because it optimizes to better in-sample and out-of-sample loss.
LSTM (Hochreiter, 1997): We use the same specification as for the RNN, except we use LSTM layers.
Transformer (Vaswani et al., 2017): We use a transformer decoder architecture, with 12 layers, 12 attention heads, and 768 embedding dimensions.
Mamba (Gu & Dao, 2023): We first encode inputs with 768-dimension embedding layer. We then pass inputs through 24 Mamba layers (analogous to 12 layers in transformer due to how Mamba layers are defined). We use 768 embedding dimensions, 16 for the SSM state expansion factor, 2 for the block expansion factor, and 4 for the convolutional width.
Mamba-2 (Dao & Gu, 2024): We use the same architecture as for Mamba except the mixer in each block is a Mamba-2 module. We use the same specifications as well: 768 embedding dimensions, an SSM state expansion factor of 16, block expansion factor of 2, and a convolutional width of 4.
We use Adam (Kingma& & Ba, 2014) to optimize each model. We use a learning rate of 6e-4 and decay the learning rate with 2000 warmup iterations. We use weight decay of 0.1 and gradient clipping at for each model. When we pre-train models on next-token prediction, we include a head to predict next tokens (tying its parameter weights to the initial embedding layer parameters).
For physics dataset generation, of planets from Uni 2.... 10]), the eccentricity from a Beta(a 0.867, 3 = 3.03) following Kipping (2013), the semi-major axis from Unif(0.3, 42), in astronomical units (AU), the mass of each planet from Log LogUniform(1007, 10-3), the mass of the star from Unif(0.55, 5). These distributions ensure that our solar system is within the training distribution of the model. order to generate sequences, we randomly generate initial conditions and solve Kepler's equation to obtain each trajectory.
the following sampling strategy. For each solar system, sample the number
B. Metric Implementation Details
B.1. Physies
To compute the empirical approximations of Equation 6, we follow the following procedure. First, we create 100 datasets of 100 examples, Dı,..., D100- For each dataset D;, we sample 100 sequences uniformly at random among the set of data points and consider their corresponding sequences of state-vectors. First, we randomly sample 50 matrices of dimension (6 x 1) from standard Gaussian. We consider the linear projection of each state-vector using each of the 50 matrices, and choose the one that maximizes the Spearman correlation between pairwise Fuclidian distances in the 6D state space and the projected 1D space. We randomly sample projected point from each sequence, leading D, of size 100. We then fine-tune model separately for each dataset, resulting in 100 fine-tuned models vi((;; Di).... nn(: D D1o0). We then calculate the associated prediction functions across all inputs x, from the same hold-out dataset, resulting in new datasets of the form {(r,, nn;;;; D,)),... ((r1, rñ;;;;;; D1
To compute the metrics, we first randomly sample 2,000 examples from all inputs, Tki Thion compute the pairwise Euclidean distance among the Oracle (a linear map or a 2 layer MLP with 5 nodes in each hidden layer) predictions on the inputs, and divide the range of predictions into 20 equally-spaced bins. For all the points that lie in each bin, compute the mean pairwise Euclidiean distance among the model predictions. The resulting figure is shown in Figure 2.
Modified setup. To further validate our findings and test the robustness of our inductive bias probe, we conducted additional experiments with modified training configurations and evaluation protocols.
The pretraining data for the orbital simulations in Section is constructed to resemble our universe: each sampled galaxy consists of one sun and up to 10 planets, where the mass of the sun much larger than that of the planets. In this setting, the
13

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
Linear oracle
MLP oracle
Oracle inductive bias
Oracle inductive bias
2
Matches oracle
Matches oracle
Transformer inductive blas
Transformer inductive blas
Figure 5. Modified inductive bias probe performance for transformer pretrained on two-body systems. For these modified metrics, when the model extrapolates it doesn't extrapolate to brand new sequences but rather sequences have been partially observed during training.
sun's movements are negligible. and interactions between planets are also minimal and ignored for computational reasons. Still, it's possible that more restrictive setting where there are only two masses in each solar system, each with similar masses- - would result in better performance.
To test this, we pretrained transformer on 10 million two-body systems, over epochs, the center-of- reference frame, with both masses sampled from a uniform distribution with range le-4 and 1e-2 solar masses, and other parameters unchanged from Section 3. We found nearly identical the ones in the main text:.
We also considered simpler version of the inductive bias metries. Instead of using the extrapolations for out-of-sample sequences, we used the same trajectories for training and extrapolation. Specifically, we fine-tuned the model on two randomly observed output points per sequence, and then extrapolated the model on the rest of the trajectory. This would make it easier for the model to capture constant terms (e.g. masses) that do not change during the trajectory.
Figure shows the results for the model trained on two-body data and evaluated using the modified inductive bias probe. The model still exhibits poor inductive bias toward Newtonian mechanies, with points clustered away from the 45-degree oracle line. Symbolic regression force magnitude predictions yiclds the nonsensical equation F m, exp(r)+ failing recover Newton's law.
B.2. Lattice and Othello
To compute the empirical approximations of Equation and Equation 2, we follow the following procedure. First, we create 100 datasets of 100 examples, DI...., D100. For each dataset, we sample sequences uniformly at random among the of data points and sample outputs from Bernouli(0..5) distribution. In construction make sure that any sequences with the same state mapped to the same output variable. We then fine-tune model separately for each dataset, resulting in 100 fine-tuned models rn::; Di).... (:; D D1o0). We then calculate the associated prediction functions across all inputs T; from the same hold-out dataset, resulting in new datasets of the form {(T,, fn (I;; Di)).... {(z,, m (x;;; Dioo)
To compute the metrics, we first randomly sample 2,000 examples from all inputs, Ik. Fheno then measure the average predictive loss for all pairs (Ik,, 2k,) with the same state, ((F,,) = (((x,) (R-IB):
R-IB Dew (eijsttee [m(z;;; D) = m(x;; D)]
and the average predictive loss for all pairs (aa IR2)) with different states, ((xx)) + ((xx;) (D-IB):
D-IB = Diest E [BLs: [m[x;; D) m may: Dj]
(8)
We rescale them so that the value of 0 corresponds to perfect accuracy and corresponds to random guessing (large values for both indicate the model exhibits stronger inductive bias towards the state).
For the lattice example, we use a state space consisting of k states: & = (1.2....,4 k}. taken from 1,000 random sequences of valid moves, each of length 100, for total of 100,000 sub-sequences of moves. Our
The inputs x, for extrapolation are
14

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
procedure for Othello follows the same steps as for the lattice example, except the state is a 64-dimensional board instead of single categorical variable.
Note that for Othello, if we randomly sample sequences from game transcripts. it is exceedingly likely that we end up with a dataset in which two sequences lead to the same state if and only if they are permutations of one another. This implies that a non-sophisticated model that detects unique permutations of sequences would appear to have high inductive bias towards the state. To prevent this, we first construct all valid Othello game openings of depth 10, randomly choose a board that appears many times in this dataset, then use all possible valid permutations of any sequence of moves that leads that board as our input dataset. Note that since all sequences will be permutations of one another, the non-sophisticated model would no longer be able to distinguish different states. We end up with input dataset of 210 Othello openings, each of length 10, for a total of 2,100 subsequence of moves.
C. Force Prediction Implementation Details
Here we describe more implementation details for the force prediction experiments.
Force vector prediction. To create Figure we fine-tuned the transformer to predict force vectors in two-body gravitational systems. We keep foree vectors continuous, and normalize the foree vectors in each sequence the maximum force vector in each sequence is unit length. We specifically fine-tune the model on the 8 sequences consisting of the trajectories in our solar system, randomly using 1% of the observations in each sequence as labeled force vector data for the model. We fine-tune the model to minimize MSE for 10,000 steps. We consider learning rate grid between 1e-6 and 5e-4, finding that 2e-4 has the best validation loss. We keep the checkpoint with the lowest held-out loss. The model is then extrapolated to make predictions across the remainder of the points in each sequence.
For comparison, we perform the same procedure for an oracle model that predicts force vectors based on the true state matrices. Specifically, using the same sampling procedure, the oracle fits k-nearest neighbor model with A based on Euclidean distance with the true state. We then use this model to predict force vectors for the remainder of the points in the solar system. The oracle predictions are depicted in Figure 6. These results show that it is feasible for model to make accurate predictions if it is extrapolating based on the correct world model.
Force magnitude prediction and symbolic regression. We use a symbolic regression to assess how close the recovered force equation is to the true law. To simplify, we use the force magnitude rather than the full vector for these experiments (the vector is always in the direction of the sun). Here, we don't normalize the force magnitudes per solar system in order to preserve the force magnitude's 's dependence on the sun's mass.
We start by creating training set that includes 9K two-body problems sampled using the sampling strategy in Appendix A. We create a test set of 1K sequences of two-body problems. We additionally ensure that the model is always extrapolating to sequences where has seen partial information by adding two randomly sampled timestep observations of each test set sequence to the training set. Because F x m. m2/rr² this means that the only factor changing within the sequence the ,² term. Additionally, instead of imputing predictions on the full test set, we select the 5,000 timesteps across the 1,000 sequences that have the most similar states to states in the training set (using Euclidean distance). This ensures that the model is extrapolating to states that are similar to the ones it is trained on.
We fine-tune the transformer on the training set for 10,000 steps with a batch size of 64, keeping the checkpoint with the lowest held-out MSE. We impute the model's predictions on 1,000 randomly sampled points from the test set. We fit a symbolic regression to these predictions using the PySR library (Cranmer, 2023). Specifically we constrain our search space to have a max size of 20 and we consider two binary operators (addition and multiplication) along with 4 unary operators (sine, cosine, exponentiation, and inverse). We use loss function that applies 0 penalty if the model is within le-8 of the magnitude and otherwise penalizes based on the absolute distance. We choose the model with the best score across three random restarts of 100 iterations each. We perform this symbolic regression procedure five times, each time randomly sampling 1,000 different points from the test set to correspond to a different galaxy. The symbolic regression returns different equations for each sample, never recovering the true law (Table 1).
To make sure this procedure is feasible when a model is extrapolating based on true state, we also consider an oracle model that is given true state. Specifically, we use the same data and fit a k-nearest neighbor model with k: 2 based on Euclidean distance to the true state. We then use this model to predict the same held-out points as above and fit symbolic regressions in the same manner. In contrast to the transformer results in Table 1, we find that this procedure recovers the true gravitational
15

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
Oracle model
Earth (true)
Earth (predicted)
True force law (Newton)
Earh
Pred Gidd
F M1 M22
Recovered law (oracle)
M m2
F
Mercury
Venus
Mars
Jupiter
Satum
Uranus
Neptune
Figure 6. Each pair of panels illustrates the trajectory of planet in the solar system and its gravitational force vectors, comparing the true Newtonian forces (left) the predicted forces from an oracle model that predicts force vectors based on the true state matrices. symbolic regression recovers the true gravitational law from its predictions.
Ground-truti law
F ,2
03
F x m2
Estimated laws
Claude Sonnet 4 Gemini 2.5 Pro
m2-00.50
F mi
Table Force equations recovered via symbolic regression LLMs predicting force magnitudes.
law for all five sampled galaxies.
D. LLM Physics Experiments
Throughout this paper, we train foundation models on domain- domain-specific data. Here, we consider large language models (LLMs) foundation models for physics. While LLMs aren't 't trained the same domain-specific trajectories we use, they trained on large quantities of text that contain information about physics and orbital trajectories.
We consider three advanced reasoning models: o3 (from OpenAl), Claude 4 Sonnet (from Anthropic), and Gemini 2.5 Pro (from Google). Fine-tuning these models is infeasible because they're proprietary and running the full inductive bias probe is expensive because it involves applying the model to many new datasets. Instead, we run a small-scale experiment assessing each model's ability to predict the force magnitude of orbital trajectories. Rather than fine-tuning models, we provide them with information in-context, and study their extrapolation behavior. Specifically, we sample 5 random solar systems with 450 observations each. For each solar system, we provide each LLM with prompt that describes the structure of the data, also including the true force magnitudes for 10 randomly selected observations. We instruct the LLM to predict the outputs for the remaining data points (we do not provide any information in the prompt indicating that the outputs correspond to forces). See Figure 8 for an example of the prompt.
We collect the magnitude inferences for each solar system (2,250 observations per LLM). Figure shows the predicted force magnitudes for each solar system for each model. Most of the results are poor, which is further corroborated by symbolic
16

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
Model: o3
Solar system!
Solar system
Solar system 3
Solar system
Solar system
3.0e-08
3.2e06
Tiue
2:1e06
Trum
Trun
True
1.9e-00
True
3.1e-06
4.5u-08
8.0e
1.9w066
3.0e-00 00-00
4.0u-06
1.9e-88
2.9e-05
Timeelop
Timestep
Timesllp
Timestop
Timeslop
Model: Claude Sonnet 4
Solar system
Bolar system 2
Solar sysilem 3
Solar system
Soler system
Force Maqniuue
3.0e-06
5.0e-00
2.1e08
T
4.5e-08
80a008
1.9w-66
1.0e-06
2.1e-08
0.0e-00
1.9e:06
2.0w008
1.5e-05
1.5e
Timestop
Timestep
Timestop
Timestep
Timestop
Model: Gemini 2.5 Pro
system
Solar system
Solar system
Solar system
system
Magniude
LLM
LLM
5.0e00
LLM
LLM
LLM
1.505
3 1e-00
3.0a-06
1.1006
4.0c 05
1.0005
2.0e-00
2.9-006
2.0e-06
200
400
200
400
200
400
200
400
200
400
Timestop
Timestep
Timestop
Timestep
Timestep
Figure 7. Comparing LLM magnitude predictions to the true magnitude across timesteps for 5 randomly sampled solar systems. Fach LLM is provided the full trajectory and random 2% sample of force magnitudes, and is prompted to impute the remaining outcomes.
regressions (Table 3). Interestingly, the symbolic regressions are simpler than the ones found for the domain-specinf foundation models. However, this may be due to differences in experimental setup (e.g. using fewer solar systems for the LLM due to cost concerns).
17

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
LLM Prompt
You are physics expert. You are given sequence of coordinates and outcomes. The coordinates are the positions of planet in a 2-body solar system. The planet is orbiting the The is at the origin.
Here is a sequence of observations. some of them are unknown. Your job is to predict the outcomes for the unknown timesteps.
Timestep: 0, Coordinates: (-26.08, -6.98)
Outcome:
Unk
Timestep: 1, Coordinates: (-26.08,
-6.99),
Outcome:
Unk
Timestep: 2, Coordinates: (-26.06,
-7.01),
Outcome: 2. 7.907672751462087ee-11
Timestep: 3, Coordinates: (-26.06,
-7.04),
Outcome:
Unk
Timestep: Coordinates: (-26.05,
-7.02),
Outcome:
Unk
Timestep: 5, Coordinates: (-26.04,
-7.081,
Outcome: 2. 90934006677451366-00
Timestep: 6, Coordinates: (-26.04,
-7.09),
Outcome:
Unk
Timestep: 7, Coordinates: (-26.02,
-7.12),
Outcome:
Unk
Timestep: 8, coordinates: (-26.02,
-7.14),
Outcome: Unk
Timestep: Coordinates: (-26.01, -7.16),
Outcome: Unk
Timestep: 449, Coordinates: (-20.28, 115.66) Outcome:
Unk
reason all you' d like, but your answer should end with "ANSWER: " followed by the predicted outcomes for all the timesteps, even the unknown ones. You sheuld structure your predictions as a dict, where each key is a timestep and each value is the prediction, You should make predictions for all of the timesteps, even the ones that are known.
Here an example the output format: ANSWER: {
0: :.0e-9,
1: 1.0e-8,
1.0e-8,
449: 1. 1.0e-8,
Figure 8, Example prompt used in the LLM physics experiments.
E. Inductive Bias Ablations
On the Othello dataset, we perform ablation of the IB metries on the number of fine-tuning iterations (Table 4), keeping the number of fine-tuning examples fixed to 100, and the number of fine-tuning examples (Table 5), keeping the number of fine-tuning iterations fixed to 100.
iltrations
10
50
500
R-IR (f)
D-TB
R-IB (1)
D-IR (f)
R-IB
D-IB (1)
R-IR (t)
THIB
RNN
0.339 (0.020)
0,631 (0.00)
(0.00(0.022)
0.756 (0.025)
0.632 (0.023)
0,397 (0.023)
0.550 (0.027)
0.868 (0.019)
1.TTM
0.805 (0.020)
0,510 (0.037)
0.576 (0.029)
0.605 (0.034)
0.563 (0.000)
(0.004)
0.553 (0.030)
0.615 (0.034)
Trunsormer
0.775 (0.022)
0.585 (00022)
(0.024)
0.619 (0.033)
0.703 (0.055
0.624 (0.033)
0.714 (0.024)
0.629 (0.033)
0.775 (0.019)
(0.025)
0.698 (0.021)
0.707 (0.028)
(0.021)
0.728 (0.027)
0.683 (0.021)
0.710 (2.028)
(0.22))
(I,031)
(.653(0.022)
(0.029)
(0.02))
(0:022)
(0.022)
(0.029)
Table 4, Results for ablating the number of iterations of fine-tuning.
F. Additional Transfer Results
Table 6 shows the full transfer learning results described in Section
18

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
Vesamples
10
90
100
500
R-IB (1)
D-IB (1)
R-IB (1)
D-IB (1)
R-IB (1)
D-IB (1)
R-IB (1)
D-IB (1)
0,815 (0.024)
0.354 (0,088
(0.024)
0.695 (0.033)
0,632 (0.023)
(0.023)
0.475 (0/022)
(0.010)
1STM
0.750 (0.30))
0.374
(.625(00:02))
0.543 (0.87)
0.563 (0,030
(0,054)
0.483 (0/021)
0,322
Transformer
0.562 (0.019)
0,363 (0,038)
0,721 (0.022)
0,610 (0.032)
0,700 (0.02))
0.624 (0,03))
0.578 (0.021)
0.853 (0.08))
Mamba
0.211 (0,020)
0.456 (0.039)
0.666 (0.023)
0,763 (0.026)
0.682 (0.021)
0,728 (0.027)
0.654 (0.018)
0.864 (0.04))
Mamba-2
0,548 (0.018)
0.453 (0,09)
0,704 (0.023)
0.684 (0.030)
0.653 (0.022)
0.694 (0.0299
0.644 (0.024)
0,886 (0.015)
Table 5. Results for ablating the number of examples used for fine-tuning.
Majority Tiles
Board Balance
Edge Balance
Pretraining
NLL (4)
ACC (1)
NLL (4)
ACC (t)
NLL (4)
ACC (f)
RNN
Untrained
0.492 (0.004)
0.755 (0.003)
0.405 (0.005)
0,806 (0.003)
0.462 (0.002)
0.816 (0.002)
NTP trained
0.431 (0.004)
0.792 (0.002)
0.302 (0.004)
0.856 (0.002)
0.080 (0.002)
0,964 (0.001)
LSTM
Untrained
0.436 (0.004)
0.786 (0.003)
0.305 (0.004)
0,864 (0.002)
0.105 (0.002)
0,953 (0.001)
NTP trained
0.232 (0.004)
0,901 (0.002)
0.164 (0.003)
0,927 (0.001)
0.041 (0.002)
0,982 (0.001)
Transformer
Untrained
0.497 (0.004)
0.754 (0.003)
0.340 (0.005)
0,855 (0.002)
0.075 (0.002)
0,967 (0.001)
NTP trained
0.100 (0.002)
0.956 (0.001)
0.086 (0.002)
0,965 (0.001)
0.013 (0.001)
0,996 (0.000)
Mamba
Untrained
0.377 (0.004)
0.816 (0.002)
0.246 (0.004)
0.888 (0.002)
0.099 (0.002)
0.952 (0.001)
NTP trained
0.149 (0.003)
0.937 (0.002)
0.158 (0.003)
0.931 (0.002)
0.027 (0.001)
0,989 (0.001)
Mamba-2
Untrained
0.379 (0.004)
0.821 (0.002)
0.258 (0.004)
0,891 (0.002)
0.068 (0.001)
0,969 (0.001)
NTP trained
0.069 (0.002)
0.970 (0.001)
0.059 (0.002)
0.976 (0.001)
0.012 (0.002)
0.995 (0.001)
IB Correlation
0.462
0.477
0.610
0.653
0.970
0.960
Table 6. Results showing transfer performance across new functions of state. "NLL" represents negative log- liliieiiioodd (lower is better). and "ACC" represents accuracy (higher is better). "IB Correlation" measures the (unsigned) correlation between each column of results Transfer learning results are correlated to the inductive bias metrics; models with
the ratios of the inductive hias metrics in Table 2, R-18 INIB
low inductive bias perform worse at transfer.
G. Next Token Performance
Table 7 shows results for the next-token test (Toshniwal et al., 2022; Li et al., 2023) for the pre-trained models on the lattice and Othello models. It measures the share of top model predictions that true for the underlying state. All models learn good next token predictions that appear to obey state.
Table 8 shows results for physics. Across 200 held-out trajectories, we autoregressively generate the model's predicted trajectory given the first 50 steps. Then, we compute the MSE of the predicted trajectory, 1, 5, 10 steps from the 50th step. We include the MSE of a baseline that always predicts the most recent timestep.
H. What are models using to extrapolate?
Here we describe how compute the decomposition of D-IB into D-IB,= and D-IB For lattice, coarsen the state-space by defining a mapping from the ground- truth state-space (of size N = 5) to pseudo-state-space of size 3. The
mapping is defined as {1} + 12,2,,... - N + 2', (N) +
Lattice
Othello
RNN
1.00
0.992
LSTM
1.00
0.996
Transformer
1.00
0.999
Mamba
1.00
0.999
Mamba-2
1.00
0.999
Table 7. Results for the next token test (Toshniwal et al., 2022; Li et al., 2023) for models pre-trained on next-token prediction.
19

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
# steps out
5
100
Per-orbit mean
(7.53 0.59)
-10-2
(5.53 0.58)
10
(1.39 0.08) 10
Previous position
(1.16 + 0.21) -10-
(1.37 + 0.38) 10-4
: 10-2
Transformer
(1.90 + 0.45) 10 -00
(1.56 + 0.45) 10
(3.74 + 3.37) 10
Table Orbit trajectory prediction performance (MSE) for models pre-trained on next-token prediction. Each column shows prediction accuracy when forecasting planetary positions 5, or 100 time steps ahead from position 500 in the sequence. We compare the transformer model to two simple baselines (one that always predicts a planet's 's position at the previous timestep, and another that uses the per-orbit mean). All results are evaluated held-out test trajectories.
Lattice
Othello
D-IBg= g=
D-IB
D-IB
D-IByy
RNN
0.740 (0.042)
0.844 (0.034)
0.521 (0.031)
0.798 (0.023)
LSTM
0.873 (0.051)
0.952 (0.034)
0.519 (0.035)
0.610 (0.034)
Transformer
0.626 (0.037)
0.710 (0.037)
0.458 (0.033)
0.625 (0.033)
Mamba
0.764 (0.040)
0.933 (0.035)
0.485 (0.030)
0.729 (0.027)
Mamba-2
0.778 (0.042)
0.920 (0.033)
0.553 (0.032)
0.694 (0.029)
Table 9. Metrics for assessing whether model's inductive bias is toward its legal next-token partition. Low values of D-IB,, and high values D-IB suggest that failures to differentiate state are driven by the models having an inductive bias toward the legal next-token partition.
For Othello, coarsen the state-space by defining a mapping from board the set of legal next moves possible from the state. Notice that this mapping is many-to-one: the pair of boards in Figure 4 demonstrate, there can be many boards that share the same set of legal next moves.
Then, we measure the expected extrapolative predictability of a random pair of sequences that have different states but the same pseudo-state (D-IB,=) and a random pair of sequences that have both different states and also different pseudo-states (D-IB as defined in Section 4.
The results are shown in Table 9. Note that across all models, D-IB, is smaller than D-IB In other words, among sequences with different states, extrapolations sequences that share the same legal next tokens more predictable from each other than those on sequences that do not.
I. Additional Figures
Figure shows an overview of the inductive bias probe. Figure 10 illustrates the inductive bias probe in the special case where the given world model has finite state space.
20

---

What Has Foundation Model Found? Using Inductive Bias to Probe for World Models
Inductive bias probe
Inputs
Step 1
Step
P
?
fa
World model: functions that obey implicit state space data
Foundation model: leams functions from data
foundation model synthetic datasets and extract learned functions
Compare learned functions to the given world model
Figure An inductive bias probe measures whether foundation model has an inductive hias toward given world model. The probe involves repeatedly fitting foundation model to small, synthetic datasets and comparing the functions it learns to the functions in the given world model.
Example: Finite state space
True world model
Low R-IB:
Low D-IB:
Learned functions don't respect state
Learned functions don't
lipues met map zame
Viues thet map stiie
Inpurs rhat mup sarte
distiinuissh state
h
4
f
6
B
4
8
f
Figure An illustration the inductive bias prohe when the given world model has a state space. Each row represents function and each column represents input 2;, with inputs belonging to the same state grouped together. The shading illustrates each function's value the corresponding input. foundation model low R-IB (middle) it learns functions that divide states, while foundation model has low D-IB (right) if it learns function that merge states.
21


Reviews:
[{"review_id": "tBfatOvBUz", "reviewer": "Reviewer_Genp", "summary": "This paper investigates whether foundation models trained via next-token prediction learn \"world models\". The paper uses synthetic tasks to measure whether such models are able to generalize from their training tasks to other tasks drawn from the same distribution. The evaluation is performed using two metrics to quantify how much the model learns inductive biases that _respect_ and _distinguish_ ground-truth states, according to an expert-defined history-to-state mapping. The paper includes experiments in predicting orbital mechanics, Othello board states, and discrete position on a number line, and presents evidence that despite good performance at next-token prediction, models appear not to use the expert-defined state representations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review of our paper. We respond to your comments and describe new results below; to summarize the new results, we've added:\n- New experiments showing robustness to predictor in R-IB and D-IB calculations\n- A more intuitive explanation of our framework\n- Clarification and new analysis of multiple representations in Othello\n\n**We hope our comments have addressed your concerns. If not, please let us know if you have any more questions we can address in the follow-up.**\n\n> _Can you provide a more intuitive explanation of the inductive bias definitions? Can you provide a toy example to go along with the Section 2: Implementation?_\n\nWe've revised our paper to make this more intuitive. Broadly: \n- A foundation model's predictions are consistent if two properties hold: \n  1. When two inputs map to the same state in the world model, do they have the same predictions for each synthetic task? This is measured by R-IB.\n  2. When two inputs have the same predictions for each synthetic task, do they belong to the same state? This is measured by D-IB.\n- So R-IB and D-IB are different perspectives on measuring consistency, analogous to type-I and type-II errors in classification. \n\nHere's an implementation example for the lattice problem:\n- X is a sequence of directions (\"L\", \"R\", \"stay\")\n- 5 states (1-5)\n- phi(X) maps movement in X starting at state 1\n  - So phi(\"R\", \"R\", \"L\")=2 (two states right, one state left)\n- We create random synthetic datasets (X, Y) such that phi(x) = phi(x') implies y = y' for each (x,y), (x',y') pair.\n- We fit a foundation model (e.g. by fine-tuning) on a very small subset of each dataset and make predictions for the held-out data points. \n- R-IB: do two held-out points with the same state always have the same predictions across synthetic datasets?\n- D-IB: do two held-out points with different states have unpredictable extrapolations across synthetic datasets?\n\n> _My impression is that if the approach fails, it could be that the predictor model is underpowered, rather than due to a lack of world model_\n\nWe can think of two ways to interpret this question: one is about the predictor that's part of the foundation model; the other is about the predictor used to calculate R-IB and D-IB. \n\nFor the first interpretation: We're interested in understanding a foundation model's extrapolative properties in small data regimes (e.g. few shot learning) since this is how they're typically used. Thus, our metrics measure consistency with the world model, not predictor accuracy, so they wouldn't penalize an underpowered predictor.\n\nFor the second interpretation: R-IB and D-IB are based on a predictor that only uses a single binary feature. Therefore the Bayes optimal solution is nonparametrically identifiable. [This table varies the number of data points used to form R-IB and D-IB](https://imgur.com/a/ppaDzrx); empirically we find little sensitivity, because the predictor converges to the optimal solution quickly. \n\n> _I wonder how useful this method can be when there [are] multiple acceptable state representations. It would be helpful for the paper to analyze... what happens in Othello under the two representations described above._\n\nThis is an important question. Our framework depends on the state mapping, not how it's represented. The only important quantities for D-IB and R-IB are whether two inputs have the same or different state. This means the **metrics aren't sensitive to different ways state can be represented**, so the two Othello representations have identical metrics. In comparison, sensitivity to representation makes other metrics (e.g. probes) more fragile. \n\nHowever we could still use different representations to help _analyze_ poor IB metrics. D-IB measures how similar the predictions for input pairs with distinct states are. We consider two distance metrics for distinct-state pairs: standard (\"cell is black\") vs. relative (\"cell has my color\") Hamming distances. We find:\n- Corr(D-IB, standard distance) = 0.094\n- Corr(D-IB, relative distance) = 0.001\n\nInterestingly this indicates states with similar standard representations are more easily confused.\n\n> _I believe... this paper assumes the generalization is still in-distribution._\n\nThis is a good point. It's crucial to also test out-of-distribution (OOD). While our submission was unclear, we do test on OOD sequences -- the only restriction on synthetic tasks is that they be consistent with the world model, not the original sampling distribution. We've changed the writing to make this clear. \n\n> _It is highly unlikely that the model will see the same state twice for two different trajectories [in Othello]._\n\nThis is absolutely right. Going back to your earlier point, it shows the importance of testing distribution shifts. In practice, we sample inputs in Othello so that we have enough same-state examples (3,889) to make accurate estimates. Appendix B has more information but we'll highlight it in the main text.", "claims_and_evidence": "Claims:\n\n1. \"Framework for testing foundation models by analyzing how they adapt to synthetic tasks that share mechanisms with their underlying training domain.\"\n\n    - Supported. The framework itself appears sound. The orbital mechanics dataset samples random initial conditions (masses, positions, and relative velocities), and shows that the model can predict the trajectories without correctly predicting force vectors. I believe this demonstrates that the framework can be applied for the intended purpose.\n- Measuring the model's ability to predict sequences it was not trained on feels like an appropriate way to test generalization, and true generalization [seems to require](https://arxiv.org/abs/2402.10877) a (causal) world model. That said, I believe the linked paper makes that claim with respect to out-of-distribution data, whereas this paper assumes the generalization is still in-distribution.\n\n2. \"Metrics that measure a model's inductive bias toward known models of reality.\"\n\n    - I had trouble evaluating this. I found the inductive bias definitions for respecting and distinguishing state to be a bit hard to follow. In Section 2: Implementation, a toy example would have been helpful. My impression is that if the approach fails, it could be that the predictor model is underpowered, rather than due to a lack of world model.\n    - The defined metrics appear to rely heavily on whether two inputs share the same underlying state. It would be helpful to quantify what fraction of the dataset contains different inputs from the same state. I'm concerned that if there aren't many such examples, the metrics might behave poorly. For example, in Othello there are a combinatoric number of states, and after a certain number of steps, it is highly unlikely that the model will see the same state twice for two different trajectories.\n\n3. \"Models can excel at their training tasks yet fail to develop inductive biases toward the true mechanisms when being adapted to new tasks.\"\n\n    - Mostly supported: I already mentioned the orbital mechanics experiments, and the Othello experiments also showed that the model struggles to predict the correct board state but successfully predicts the set of next legal moves.", "methods_and_evaluation_criteria": "The definition of a \"world model\" as a mapping from histories to a particular ground-truth state representation feels problematic for answering the ultimate question of whether foundation models contain world models. There are many valid and useful state representations, and if the foundation model fails to match a particular expert-defined representation, it does not mean the foundation model lacks a world model altogether.\n\nFor example, [this project](https://www.neelnanda.io/mechanistic-interpretability/othello) showed that when evaluating whether Othello-GPT had a linear representation of the board state, it matters whether the representation is of the form \"this cell is black\" vs. \"this cell has my color\", since the model plays games as both colors.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "It is an extremely important question whether foundation models learn world models. Foundation models are the dominant paradigm in AI right now, and much of our confidence in them is predicated on the assumption that their performance at sequence prediction implies that they have learned a world model.\n\nThis paper doesn't address the question fully. Instead it addresses the question of whether a particular foundation model has learned a particular world model. A satisfying answer to this narrower question would still be a useful step towards answering the broader question.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "I wonder how useful this method can be when there multiple acceptable state representations. It would be helpful for the paper to analyze, for example, what happens in Othello under the two representations described above.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Can you provide a more intuitive explanation of the inductive bias definitions?\n\nCan you provide a toy example to go along with the Section 2: Implementation?\n\n\n=====================\n\nPOST-REBUTTAL UPDATE\n\n=====================\n\nI found the comments by reviewer 86Gp and the subsequent response very helpful in understanding the paper's main claim. I agree that explanation could have been much clearer, and I'm happy the authors are taking steps to improve it.\n\nIn the rebuttal \"Lattice problem\" example, I am still unclear on what exactly $y$ is in the pair $(x,y)$. I suppose it's some predicted quantity that depends on the encoded state $\\phi(x)$? I would encourage the authors to make this example even more concrete by saying what specific thing is being predicted, how it depends on the state, and maybe even provide some example predictions from which we can see what the D-IB and R-IB metrics would report.\n\nI thank the authors for providing results for the two different Othello representations. Unfortunately I'm not sure what you mean by \"poor IB metrics\", and I don't understand what the takeaway is here. In regards to my original question of whether the method is sensitive to the choice of representation, it sounds like the answer is no. But then why are there two different correlation numbers for the different distance metrics? I would encourage the authors to take a bit more time to explain this.\n\nOverall, I think the weakest aspect of the paper is the clarity of the presentation, and I urge the authors to do whatever they can to clarify the points that the other reviewers and I were confused about.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SHaW7d4HDh", "reviewer": "Reviewer_kBZN", "summary": "In this paper, the authors focus on the problem of understanding the generalization capabilities of foundational models. For example, can a foundational model truly develop an inductive bias towards Newtonian mechanics or the rules of a board game? The authors test this question, developing a framework for testing world models in addition to proposing metrics such as Inductive Bias towards Respecting State (R-IB) and inductive bias towards distinguishing state (D-IB). The test hypothesis in a number of domains - specifically orbital mechanics, the board game Othello, and Lattices. They find that while common world models such as transformers can excel at specific tasks, they fail to learn inductive biases and rely on data-specific, piecemeal heuristics.\n\n## update after rebuttal\nIn line with the other reviewers' comments, I keep my assessment of accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review of our paper. We're glad that you appreciated our paper and findings, and that our results gave you \"a deeper understanding [of] why foundation models fail to generalize\". \n\n> _The proposed metrics (R-IB and D-IB), while needed for a comprehensive evaluation, are somewhat abstract and sometimes difficult to understand in the context of the various domains (Orbital physics, Othello). It would be helpful to ground the concepts for these metrics with domain-specific examples when describing them in section 2._\n\nThis is a great point. We've revised our paper to make the framework clearer, and we've added more domain-specific examples. \n\nSpecifically: \n- A foundation model's predictions are consistent if two properties hold: \n  1. When two inputs map to the same state in the world model, do they have the same predictions for each synthetic task? This is measured by R-IB.\n  2. When two inputs have the same predictions for each synthetic task, do they belong to the same state? This is measured by D-IB.\n- So R-IB and D-IB are different perspectives on measuring consistency, analogous to type-I and type-II errors in classification. \n\nHere's an implementation example for the lattice problem:\n- X is a sequence of directions (\"L\", \"R\", \"stay\")\n- 5 states (1-5)\n- phi(X) maps movement in X starting at state 1\n  - So phi(\"R\", \"R\", \"L\")=2 (two states right, one state left)\n- We create random synthetic datasets (X, Y) such that phi(x) = phi(x') implies y = y' for each (x,y), (x',y') pair.\n- We fit a foundation model (e.g. by fine-tuning) on a very small subset of each dataset and make predictions for the held-out data points. \n- R-IB: do two held-out points with the same state always have the same predictions across synthetic datasets?\n- D-IB: do two held-out points with different states have unpredictable extrapolations across synthetic datasets?", "claims_and_evidence": "Yes. The authors clearly show quantitatively and qualitatively that world models struggle with learning general principles from data.", "methods_and_evaluation_criteria": "Yes. Even though the domains used for evaluation are very simple (two-world body physics, 2d-board game), they allow for a deeper understanding why foundational models fail to generalize. The qualitative failure cases (such as the Figure 1 and 2) make the strongest cases for the paper. The quantitative metrics such as R-IB and D-IB also support the argument even if they are somewhat abstracted away from the domains.", "theoretical_claims": "The paper contains no proofs.", "experimental_designs_or_analyses": "Experimental designs are sound even if the datasets used are simplistic and metrics abstract.", "relation_to_broader_scientific_literature": "This paper contests the idea that foundational models are able to learn inductive biases in various domains and shows specifically how and why they fail. One of the domains (Othello) has been used in prior literature.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The proposed metrics (R-IB and D-IB), while needed for a comprehensive evaluation, are somewhat abstract and sometimes difficult to understand in the context of the various domains (Orbital physics, Othello). It would be helpful to ground the concepts for these metrics with domain-specific examples when describing them in section 2.", "other_comments_or_suggestions": "None.", "questions_for_authors": "I have no questions for the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OguTYHoOcK", "reviewer": "Reviewer_86Gp", "summary": "This paper proposes methods for evaluating if the predictions made by foundation models following fine-tuning on new tasks are compatible with a reference world model.  The authors introduce two metrics; inductive bias towards respecting state and inductive bias towards distinguishing state, which aim to measure how well the foundation model's extrapolations align with a reference world model. They demonstrate their approach in the domains of classical mechanics (orbital bodies), lattice problems, and the game Othello, and give evidence that foundation models trained on next-token prediction fail to transfer to new tasks in a way that is consistent with the underlying world model, but instead rely on task-specific heuristics..\n\n\n## update after rebuttal\n\nFollowing the authors rebuttal, and proposed clarifications of the paper, I am now learning towards acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful and insightful review. \n\n> _I will increase my score if the Framework section is sufficiently improved, and if the language is toned down a bit re: the results showing that the foundation model has learned a world model (rather than, to what degree its predictions are consistent with a given world model)._\n\nWe really like the way you've described the framework and we've modified our paper to match (see below). \n\nWe also agree with your language change: it's overstating to say “has learned a world model”; given our test, it's much more accurate to say “consistent with a given world model”, which is the language we'll use going forward. \n\nYour comments were very constructive and have improved the paper. **We believe we've addressed your comments below; if we haven't, please let us know if you have any further questions.** \n\n\n> _Question 1: ...My first question is if this accurately captures what they are proposing..._\n\nWe've rewritten the framework to incorporate your suggestions. We don't have space to include the full section here but the outline is:\n\n- This paper is about measuring the degree to which a foundation model's predictions are consistent with a given world model.\n- Note a mechanistic interpretation is difficult to measure. \n- Instead we examine model predictions on synthetic tasks that obey the world model\n- Consistency requires two properties: \n  1. R-IB: When two inputs map to the same state in the world model, do they have the same predictions for each task? \n  2. D-IB: When two inputs have the same predictions for each task, do they belong to the same state?\n- R-IB and D-IB are different perspectives on consistency, analogous to type-I/type-II errors. \n\n> _Question 2: It could be that the foundation model has learned the correct world model but its predictions cannot be described by prediction = f(state = WM(input))_\n\nGreat question. This is why we have _two_ metrics: \n- If a model predicts f(state, input) != f(state, input') for same-state inputs, R-IB is low\n- If a model predicts f(state, input) = f(state', input') for different-state inputs, D-IB is low\n\nThese metrics can be used independently (e.g. focusing only on R-IB). But there are many applications where we’d care if a model has extraneous information. For example, if a model carries extraneous information its predictions on small data may be subject to spurious correlations. Empirically we find that low D-IB scores suggest that models base their predictions not on true state but on next-token partitions (see Figure 2 and Table 8), hurting small-data performance.  \n\n> _Question 3: When you say about D-IB “If a learning algorithm’s effective foundation is based on the world model, it should now extrapolate in predictable ways between inputs values in different state”, what precisely do you mean?_\n\nYou've helped us identify an unfortunate typo. The sentence should read \"...it should **not** extrapolate in predictable ways...\". So if the model learns according to not only the effective foundation but other inputs, D-IB is lower. We've corrected this.\n\n> _Question 4: Physical laws involve two kinds of object; kinematics (states) and dynamics..._\n\nWe want to make sure we're interpreting your question correctly:\n- kinematics: mapping from inputs X to state space (i.e. given by phi)\n- dynamics: how the state spaces evolve.\n\nWe agree the distinction is interesting. Our framework only addresses kinematics (e.g. state, not the laws that may govern dynamics on top of state). Therefore R-IB and D-IB tell us about kinematics. But the two are related; knowing kinematics often allows recovering dynamics from enough sequences.\n\nWe focus on states because even in physics there's value in studying kinematics independently from dynamics. Although orbital trajectories (dynamics) are fixed, many interesting functions depend on state (e.g. energy, angular momentum). Addressing dynamics explicitly is valuable future work.\n\n> _[Does the result that] models are learning task-specific heuristics [come only] from these symbolic regressions?_\n\nSymbolic regressions are one tool for showing that models learn task-specific heuristics (we've added more experimental details in the appendix), and it's a common method for validating physics models [1, 2]. But it's not the only tool; our appendix has additional experiments with other methods (Table 8) in non-physics domains. We've moved this to the main text.\n\nWe've also added added new physics experiments to Table 8:\n- D-IB for distinct states with same next tokens: 0.709\n- D-IB for distinct states with different next tokens: 0.764  \n\nSo poor D-IB can be partially attributed to models basing their predictions not on state but on next-token partitions. Figure 2 illustrates this point for Othello.\n\n[1]  Udrescu et al. \"AI Feynman: a Physics-Inspired Method for Symbolic Regression.\" (2020).  \n[2] Cranmer et al. \"Discovering Symbolic Models from Deep Learning with Inductive Biases.\" (2020).", "claims_and_evidence": "The authors do present \"a framework for testing foundation models by analyzing how they adapt to synthetic tasks that share mechanisms with their training domain\". They give quite convincing evidence that the models they evaluate are not generalizing in a way that respects the underlying statespace. \n\nOne of the key claims—that foundation models learn heuristics rather than fundamental world models—seems to come mostly from symbolic regression on the predictions of a model, rather than applying the metrics that are the main results of the paper. It is not clear how robust this claim is from this single, and quite small, experiment.", "methods_and_evaluation_criteria": "The metric R-IB seems well motivated to me, though I struggled to understand the metric D-IB (see questions below)", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes, pre-training and fine-tuning settings are well-described. Baseline models (RNNs, LSTMs, Transformers, Mamba) are compared, and the novel metrics (R-IB, D-IB) are computed over multiple training seeds. Symbolic regression is used to validate whether models recover correct physical laws, which I am less certain about. The experiment is fairly small, and its not immediately clear how robust the result is, and if its really coming from the fact that the models are learning the wrong physical laws.", "relation_to_broader_scientific_literature": "There is a good discussion of related work as far as I can see.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Overall I think the proposed method is compelling, though Im not sure it quite does what the authors claim and establishes that a world model has been learned. The main weakness to my mind is the formalism section, which I found very unclear and had to read multiple times before realising (I think) what the authors are proposing. See notes below.", "other_comments_or_suggestions": "I will increase my score if the Framework section is sufficiently improved, and if the language is toned down a bit re: the results showing that the foundation model has learned a world model (rather than, to what degree its predictions are consistent with a given world model).", "questions_for_authors": "Question 1\n\nMy current understanding of the formalism the authors propose is this, and my first question is if this accurately captures what they are proposing.\n\n1. The authors define consistency in the following way. Assume a foundation model makes predictions using a world model prediction = f(state = WM(input)). Assume the foundation model predicts in this way, and that the WM is re-used across tasks (e.g. when we fine-tune the foundation model on some new task). Then, if two inputs map to the same state in the world model, then they must give the same predictions across all tasks, where the task is to prediction some function that depends only on the state (e.g. not any additional information in the input that is not captured in the state, such as how the state was reached). \n2. They then invert this relationship, saying that if two inputs are treated the same across all tasks, then they are compatible with being the same states in a world model. This does not imply they are the same states in the world model, or even that there is a world model, just that the foundation models predictions are compatible with this being true. \n3. Then, if we have a reference world model (such as the statespace of a dynamical system, or the board state of a game), we can map inputs (e.g. trajectories) to states on the `correct’ world model. Then, for the foundational model to be compatible with this correct world model, the predictions must be the same for all inputs x that correspond to the same state on the correct world model. If this is not true, then the world model implicit in the foundation model (if there is one) is to some extend incompatible with the correct or reference world model. \n\nIf this is correct, then the introduction of the framework could have been a lot clearer and easier to understand. It took me multiple reads to get to this understanding, that I am still not confident on. But (if the above is accurate), the core ideas are not very complex, and will be better motivated by a simplified introduction. \n\nQuestion 2\n\nIt could be that the foundation model has learned the correct world model but its predictions cannot be described by prediction = f(state = WM(input)). For example, if prediction = f(state = WM(input), input), e.g. the prediction is not just based on the world state but some spurious information from the input (such as having a bias towards certain predictions depending on the input length). This would give a false negative (there is an accurate world model, but a poor process of drawing inferences from that model). The model prediction = f(state = WM(input), input) even makes sense if assume the world model is encoded in the intermediate layers of the transformer, and the action of the residual stream. Does your result get around this in some way? Its fine if not, but it would be nice to A) have all the assumptions on how the foundation model generates predictions in a single paragraph, B) some examples of possible ways foundation models could generate predictions (as described here) that violate the assumptions (including false positives, where there is no world model, but the foundation model passes the eval).\n\nQuestion 3\n\nthe motivation and intuition for D-IB is not clear when it is introduced. When you say about D-IB “ If a learning algorithm’s effective foundation is based on the world model, it should now extrapolate in predictable ways between inputs values in different state”, what precisely do you mean here by \"in predictable ways\"? What should we expect D-IB to be if there is a world model, and why? Are you saying something like, if the predictions are based on the (true) world model, then the correlation between predictions that map to the same state should be invariant between different synthetic tasks? Or that the correlation should be invariant when we vary the inputs x_i, x_j but they still map to fixed (but different) states? \n\nQuestion 4\n\n\nYou talk about foundation models learning the correct underlying physical laws of a system, but focus purely primarily on the state space. Physical laws involve two kinds of object; kinematics (states) and dynamics, and the dynamics are typically the most important thing as they allow for future states to be predicted. For example in RL, world models capture both of these fundamental objects. In the newtonian mechanics example, the most interesting part in my opinion was the symbolic regression on the actual predictions, precisely because you are seeing if the model is learning the correct dynamics or not. Can you explain what R-IB and D-IB are telling you here, in simple terms? And am I right in thinking that the main claim, that the models are learning task-specific heuristics, comes from these symbolic regressions? If so, it feels like this should be a larger experiment, with a lot more details on experimental design.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel framework and metrics to test if foundation models learn underlying world models using synthetic tasks. Reviewers appreciate the significant and timely research question, the valuable novel framework and metrics, and the use of well-chosen synthetic tasks. However, initial concerns were raised regarding the clarity of the metrics/framework and the robustness of some analyses. Overall, the paper is recommended for acceptance, as the authors' rebuttal addressed concerns, confirming the work's valuable contribution.
Decision: Accept (poster)