Can Test-Time Scaling Improve World Foundation Model?
Authors: Wenyan Cong
Hanqing Zhu
Peihao Wang
Bangya Liu
Dejia Xu
Kevin Wang
David Z. Pan
Yan Wang
Zhiwen Fan
Zhangyang Wang
Venue: COLM 2025 (2025)
Keywords: world foundation model
test time scaling
autoregressive video generation
evaluation tookit
TL;DR: Test time scaling could work for world foundation models and shows a clear scaling law. Smaller model with test time scaling is even better than x2 larger pretrained model. A extensible tookit will be opensourced for evaluating performance of WFM.

Abstract: World foundation models, which simulate the physical world by predicting future states from current observations and inputs, have become central to many applications in physical intelligence, including autonomous driving and robotics. However, these models require substantial computational resources for pretraining and are further constrained by available data during post-training. 
As such, scaling computation at test time emerges as both a critical and practical alternative to traditional model enlargement or re-training.
In this work, we introduce **SWIFT**, a test-time scaling framework tailored for WFMs. SWIFT integrates our extensible WFM evaluation toolkit with process-level inference strategies, including fast tokenization, probability-based Top-K pruning, and efficient beam search.
Empirical results on the COSMOS model demonstrate that test-time scaling exists even in a compute-optimal way. Our findings reveal that test-time scaling laws hold for WFMs and that SWIFT provides a scalable and effective pathway for improving WFM inference without retraining or increasing model size. Project page: [https://scalingwfm.github.io/](https://scalingwfm.github.io/).

Published as a conference paper at COLM 2025
Can Test-Time Scaling Improve World Foundation Model?
Wenyan Cong", Hanqing Zhu Peihao Wang', Bangya Liu Dejia Xu', Kevin Wang', David Z. Pan', Yan Wang", Zhiwen Fan Zhangyang Wang'
University of Texas at Austin, NNIIDA,, University of Wisconsin- Madissn,, *Texas A&M University
"Equal contribution. Correspondence to at atlaswang@utexas.ced
Abstract
World foundation models, which simulate the physical world by predicting future states from current observations and inputs, have become central to many applications in physical intelligence, including autonomous driving and robotics. However, these models require substantial computational resources for pretraining and are further constrained by available data during post-training. As such, scaling computation at test time emerges as both a critical and practical alternative to traditional model enlargement or re-training. In this work, we introduce SWIFT, test-time scaling frame- work tailored for WFMs. SWIFT integrates our extensible WFM evaluation toolkit with process-level inference strategies, including fast tokenization, probability-based Top-K pruning, and efficient beam search. Empirical re- sults on the COSMOS model demonstrate that test-time scaling exists even in compute-optimal way. Our findings reveal that test-time scaling laws hold for WFMs and that SWIFT provides a scalable and effective pathway for improving WFM inference without retraining or increasing model size. Project page: https:/ tttps:///calingw/ingihuus
1 Introduction
World foundation models (WFMs) aim to simulate the dynamics of the physical world by predicting future states based on current observations and inputs. They play a central role in physical intelligence, particularly as a means of generating synthetic data, supporting a wide range of downstream tasks such as autonomous driving, robotics, and embodied planning for scalable simulation, analysis, and agent training.
Despite their promise, training WFMs at scale present significant challenges. Pretraining demands massive computational resources, especially since WFMs operate on video inputs. For example, the most recent COSMOS Agarwal et al. (2025) is trained on tens of millions of video hours using thousands of high-end GPUs over several months. Even post-training, model scaling is often constrained by data availability and diminishing returns from scaling laws. These limitations highlight critical need for test-time computation scaling- strategy to improve performance by allocating additional compute during inference, without enlarging or retraining the base model.
Motivated by the success of test-time scaling in large language models and diffusion-based vision-language models Snell et al. (2024); Ma al. (2025), we, for the first time, explore the test-time scaling for world foundation models.
However, several key challenges make this far from a straightforward borrow-and-tes scenario. 0): We need to have a dedicated evaluation testbed for WFMs. Existing video generation benchmarks typically emphasize aesthetic quality or semantic alignment, which do not align well with the goals of physical realism and consistency that world modeling requires. Moreover, a plausible and extensible cvaluation toolkit is needed to assess WFM outputs across diverse downstream applications. In contrast, traditional RL-based reward models are often rigid and difficult to generalize across tasks or domains. @ A practical and effective test-time scaling strategy musi be tailored for WFMs. Unlike LLMs, WFMs commonly rely

---

Published as a conference paper at COLM 2025
Unsatisfactory Simulation
Cosmos
Text Prompt (optional):
the xailo degices a pins coca-lela truck driviing ewy cestered rood StrestUubts, tvere sidoeel with and the scere eppeers to be snturber or send-
Autoregressive World Modeling
Testbed
MPN
Search Algorithn
Newerd/Metrics
SWIFT
?
Boosted Physical World Simulation
Figure 1: The SWIFT pipeline for studying test-time scaling (TTS) in world foundation models (WFMs). We use autoregressive world model COSMOS as base, which initially produces unrealistic world simulation (top panel). By applying our TTS method, the simulation is significantly enhanced and becomes more physically plausible (bottom panel).
diffusion-based decoders to produce high-quality outputs, which are inherently slow. This makes it impractical to directly apply advanced test-time strategies like chain-of-thought or tree-of-thought which involve checking intermediate steps.
To address these challenges, we introduce SWIFT, test-time scaling framework specifically designed for WFMs, as illustrated in Figure 1.
Our contribution as follows:
WFM Evaluation Toolkit:
We propose the first evaluation toolkit specifically designed for world foundation models. The toolkit is modular, extensible, and supports a range of rule-based metrics, enabling multi-aspect evaluation across diverse downstream tasks.
WFM Test-Time Scaling Framework: We introduce SWIFT, the first test-time scaling framework tailored for WFMs. SWIFT enables efficient process-level test-time search by incorporating a fast tokenizer for accelerated decision-making, probability-base Top-K pruning to mitigate overconfidence, and a beam search algorithm to maintain performance-efficiencyy trade-offs.
Empirical Findings: We conduct the first empirical study of test-time scaling for WFMs using COSMOS as base model. Our findings reveal that: (1) A test-time scaling law exists for "FFMs-surprisiggy even under compute-optimal conditions. Small models enhanced with test-time scaling can match or surpass the performance significantly larger models, given the same inference-time compute budget. (2) Our proposed frame- work, SWIFT, further boosts performance as the number of generated samples increases, and does so efficiently through carefully designed anference-time strategies.
2 Motivation: Why Test- Time Scaling For World Foundation Models
Test-time scaling has shown promise for leveraging a model's full capacity, where allocating more computation at inference can yield substantial performance gains, even exceeding the benefits of simply increasing model size Snell et al. (2024).
World Foundation Models (WFMss simulate real-world dynamics and bridge the physical and digital domains, playing a key role in generating domain-specific synthetic data for applications like autonomous driving and robotics. To investigate whether WFMs can similarly benefit from test-time scaling, we highlight two key motivating factors:
① Training larger WFM is extremely costly in both compute and data! Unlike LLMs' training, primarily involving text, WFMs process massive amounts of video data, requiring immense resources. For instance, training the COSMOS autoregressive video model
2

---

Published as a conference paper at COLM 2025
prompt
ampoypalatts aa shaive shoul sanai pocked with muder sistte thaghaa vilees
fante
belsttycreze destanee otppinn prisutuct sonart
party dourty gapningg
Unsatisfied 3d consistancy
Unsatistied temporal consistancy
Improved consistency with scaling
Improved temporsi consistency with teat-lime scaling
Figure 2: Videos of world foundation model without (top) and with (bottom) TTS, where TTS improves 3D consistency (left) and temporal consistency (right) the generated videos.
20M hours of footage demanded 10K NVIDIA H100 GPUs over months even for 13B model. Moreover, scaling laws suggest that bigger models demand more data. Collecting such large-scale, domain-specific video is substantially hard. This creates a chicken-and-eye problem: models intended to generate synthetic video as digital twins require high-quality data-yet such data often only becomes available after capable WFM has been developed.
@ Larger WFM inference is nearly as costly running multiple smaller models. Even after training, inference for WFMs remains a bottleneck. Generating long, high-resolution videos with an autoregressive model is computationally expensive because each step re- quires a full transformer pass. This process results in slow generation speeds and high memory usage, making large-scale deployment challenging. Under these constraints, test- time scaling-whitn invests extra computation resources during inference anference-becooes becomes an attractive alternative to simply building bigger models. For example, running 12B model requires approximately three times the FLOPs of a 4B model, meaning we can afford to run the 4B model two to three times instead.
Our Aim: Enhance WFMs at Test Time without Enlarging or Retraining Them.
3
WFM Evaluation Toolkit
World Foundation Models have recently evolved to incorporate video generation means of building digital twins-synthetic video representations that simulate the physical world. This capability enables powerful applications in simulation, analysis, and agent training across domains like autonomous driving and robotics.
Despite this progress, there is still standardized evaluator tailored to the unique goals of WFMs. ① General video generation benchmarks (e.g., VBench Huang et al. (2024), VideoScore He et al. (2024)) focus on aesthetic or text-conditioned outputs, which are misaligned with the physical realism and dynamical consistency central to world modeling, @ Task-specific benchmarks (e.g., ACT-Bench Arai et al. (2024)) measure downstream control performance but overlook key video generation aspects such as temporal coherence.
To address this gap, we introduce the first general evaluation toolkit for world foundation models, specifically designed to assess their capabilities across diverse domains. Unlike task-specific or generic video metrics, our toolkit defines a modular, extensible framework that supports domain-specific evaluation while remaining broadly applicable.
Our evaluation toolkit now includes the following key metrics:
3D Consistency: Assesses the geometric coherence of the generated scene (left in Figure 2), critical for tasks like autonomous driving. We use the 3D foundation model CUT3R Wang et al. (2025), which reconstructs 3D structure from video in feed-forward manner, enabling efficient evaluation.
Evaluates whether generated video sequences maintain coherent temporal dynamics-pprservvin smooth background transitions and object permanence across frames, which is essential for tasks such as robotics and planning (right in Figure 2). We measure this using CLIP and DINO similarity scores.
Temporal Consistency:
3

---

Published as a conference paper at COLM 2025
Spatial Relationship Awareness: Measures whether spatial relationships between enti- ties- especialyy human-environmen interactions are physically plausible. For instance, in factory simulations, assess adherence to left-right and top-bottom relationships implied by prompts, inspired by Huang et al. (2024).
Perceptual Quality: Assesses the visual fidelity and aesthetic appeal of generated content. Since WFMs must model the world vividly and convincingly, color, harmony, and artistic quality. We use the LAION aesthetic predictor to evaluate each frame individually. Low-level distortions (e.g., noise or blur) are not penalized, as they can reflect natural properties of real-world sensors.
focus on richness of
Text-to- Video Alignment: Measures semantic alignment between the generated video and its associated text prompt. We employ both CLIPScore (frame-level cosine similarity between prompt and visual features) and X-CLIPScore (video-level alignment using cross-modal embeddings).
As noted in the COSMOS work, evaluation of WFMs is inherently challenging due to their generality and complexity. While our proposed toolkit is not exhaustive, is intentionally modular and extendable, allowing new metrics and tasks to be incorporated as the field matures. To support further research and reproducibility, the toolkit will be open-sourced.
Thoroughly evaluating WFMs across different domains is challenging, as generating thou- sands of videos demands substantial time and GPU resources. Most existing world modeling work Hu et al. (2023) has focused on autonomous driving, domain where generating realistic and diverse data partipautllllarly for rare corner cases-is both critical and difficult. In this work, we also adopt autonomous driving as our primary testbed for studying test-time scaling strategies, which aligns with the COSMOS model's target application scenario.
4 SWIFT: The First Test-Time Scaling For World Foundation Models
Built upon our evaluation toolkit, we propose SWIFT, the first test-time scaling framework for WFMs, as illustrated in Figure 1, to address the following timely and important questions:
QΦ Can test-time scaling improve WFMs even in compute-optimal way? Beyond verifying test-time scaling, we also aim to explore if smaller model, with test-time scaling under fixed compute budget, can match, or even outperform, larger model.
Q3 How can we design an effective and practical test-time scaling strategy tailored to WFM video generation? Rather than directly adapting test-time scaling techniques from LLMs, we seek to design strategies that address the unique challenges of WFMs- suchh as sequential autoregressive video generation and the high cost of diffusion-based decoding.
4.1 Scaling via Test-Time Verifiers: Framework
To demonstrate the effectiveness of test-time scaling for world foundation models, we focus on its autoregressive video generation paradigm. Diffusion-based video generation has been studied extensively for its test-time scaling behaviors (e.g., Ma et al. (2025)), the test-time scaling properties of autoregressive video models remain underexplored. This paradigm is especially relevant, as it offers a unified approach to multimodal generation-recentyy exemplified by GPT-4o's image capabilities (which is believed be the underlying recipe).
We first formalize the generation process in autoregressive WFMs. Let Pe denote a pre- trained WFM with parameters O This model processes an input consisting of video chunk Vo, and prompt to generate video represented as V {v, .2, , where V consists of N-step response. Each step response d; contains frames, instead of single frame, for efficiency and temporal consistency, generated in an autoregressive manner:
i = Po(v; c, 01, (/.1.
(1)
Formulate the video generation as a Markov Decision Process (MDP): We draw on prior work that casts sequential decision-making tasks into an MDP framework. Since WFM generates each step of a video in an autoregressive manner- conditioning on both the input and all previously generated frames its generation naturally maps to an MDP represented
4

---

Published as a conference paper at COLM 2025
by the tuple A,, R). Each state E S corresponds to a partially generated video segment, beginning from an initial state so that includes input frames and any textual conditions. The reward function R(s,a), or called verifier Snell et al. (2024), evaluates the quality or alignment of the generated content, while the action space A dictates how to refine the model's ss outputs given a search algorithm.
4.2 Scaling via Test-Time Verifiers: Verifier R. Design
In the context of test-time scaling, verifiers (rewards) play a central role in assessing whether generated candidate is desired. Broadly, two categories of reward design exist: (1) siference-based rewards aim to simulate human preferences by leveraging real-world feedback. But collecting large-scale human annotations labor-intensive, prompting recent efforts to train reward models as human judgments proxies Guo et al. (2025b); (2) Rule- based rewards rely on feature-based metrics. Since they do not incorporate external biases, they are generally more objective and free from inductive bias.
Our Design Choice: Rule-Based Rewards for Robustness and Extensibility: To determine the more suitable reward formulation for autoregres- sive WFMs, we conduct preliminary analysis us- ing best-of-N sampling strategy. For each prompt, we generate N candidate videos and evaluate them using both reward types. For preference-basee re- wards, we adopt VideoScore He et al. (2024), which provides holistic assessments across multiple aspects (e.g, visual quality, temporal consistency). Given our unconditioned video generation task in autonomous driving using COSMOS-4B, we retain only the rele- vant dimensions- visual quality and temporal consis- tency-and exclude those tied to conditioning signals. For rule-based rewards, we select established metrics corresponding to these same two aspects (e.g., aesthetic quality for visual quality and object permanence score for consistency).
Geometric Mean
0.33
023-
0:5
0.55
baod CDSMCS-48
0.09
bacel te CDSMCS-
Best-of-N Selection
Figure 3: Rule-based vs. preference-
based rewards.
As shown in Figure 3, rule-based rewards consistently outperform preference-based ones in both stability and alignment with qualitative inspection. This corroborates findings of DeepSeek rl Guo et al. (2025a), which also reported that rule-based evaluation is more robust and less prone to reward hacking. In addition to their reliability, rule-based rewards are easier to extend. Our WFM coaluation toolkit offers modular design that allows new metrics to be seamlessly integrated into the verification pipeline, making it scalable for future evaluation needs.
4.3 Scaling via Test-Time Verifiers: Action A Design
In LLMs, two common strategies are used to improve output quality: (1) modifying the input prompt to alter the proposal distribution, and (2) sampling multiple candidate completions and selecting or refining the best one. However, directly adapting strategy (1) to video gen- eration is challenging. First, integrating textual feedback for "reflection" " typically requires a specialized reward model to detect misalignments and provide correction signals-toof, that are not yet available for WFMs. Second, current WFMs are not trained to understand or act on such feedback without additional fine-tuning. Therefore, we focus on strategy (2)-fest-timee search-as as the primary action to explore test-time scaling laws. We encourage the WFM to explore multiple possible continuations through i.i.d. sampling from its output distribution, following prior work Snell et al. (2024); Guo et al. (2025b). This approach avoids the need to generate reflection-basee based feedback or train the WFM to interpret and act on such feedback.
Another possible way to introduce randomness in video generation is by keeping the prompt fixed while sampling diverse input frames, potentially using text-to-image or text-to-video models to initialize different starting conditions. However, this strategy is not suitable. First, introduces inherent variation in the inputs, making the resulting generations not directly comparable. For instance, larger one, which can significantly affect evaluation metrics such as 3D consistency. Second, WFM may only rely on video input, in COSMOS 4B.
input may produce small object while another generates
5

---

Published as a conference paper at COLM 2025
Existiing TTS nethods
practical efficient adaptation video generation
<0. 82s video decoding
X
Output Rewar rd Model
pronpt
Rule-based verifiers
Smout
top-k
selection
P
Process Reward Model
Bean Search
w,
Prob
Figure Compared to ORM and PRM, our proposed beam search with probability is more efficient and practical for WFMs with three key efficiency designs.
4.4
Scaling via Test-Time Verifiers: Search Algorithm
Within the SWIFT framework, one can seamlessly adapt various test-time scaling methods.
4.4.1 Naive best-of-N to verify test-time scaling law
We begin with the simplest possible test-time scaling strategy: best-of-N search. For each input, we sample N independent video continuations, fully generate each one, score them using our rule-based verifiers R, and then select the highest-scoring video (Figure 4, left). This approach mirrors the "best-of-N" output selection commonly used in LLM research (ORM). Although conceptually trivial, best-of-N serves as a crucial start to investigate test-time scaling laws in WFM.
From this experiment, we draw two key observations:
Observation 1: Test-Time Scaling Exists in WFM: Even modest increase in the number of sampled continuations consistently improves generation quality across most evaluated metrics. Simply investing more compute at inference produces higher-fidelity videos, demonstrating that WFMs exhibit clear test-time scaling law (detailed quantitative results in Figure 3, Table 1, and Table 2).
Observation Test-Time Scaling Is Surprisingly Compute-Optinal. Beyond confirming that test-time scaling improves video quality, we ask: how much extra inference compute does a smaller model need to rival a larger one? Strikingly, only two to four additional inference passes a 4B model match or exceed the output quality of a single pass by 13B model. In terms of FLOPs, this is comparable to running the larger model once. This highlights a compelling trend in WFMs: test-time scaling can offer a more compute-efficient alternative to model size increases-echen recent findings in LLMs Snell et al. (2024).
4.4.2 Adapting Test-Time Scaling Strategy to World Modeling Practically and Efficiently
Naive best-of-N sampling improves video quality, but it fails to exploit the inherently sequential nature of autoregressive video generation (see our MDP formulation in Sec. 4.1). Intuitively, one can check video quality step-wise and select the best path generate subsequent frames, similar to the Process Reward Model (PRM) in the LLM Ma et al. (2023).
However, straightforward PRM implementations run into one fatal inefficiency due to:
Challenge: Expensive decoding via a diffusion- decoder: WFMs rely on a diffusion-based denoising tokenizer to produce high-quality frames. This process takes 137.2 A6000, roughly as long as autoregressively generating those frames. Hence, performing full diffusion decode step-wise creates substantial overhead. While test-time scaling inherently trades extra compute for better output quality, an efficient strategy must avoid wasted decoding and instead focus compute exploring promising candidate trajectories.
To address this challenge, we introduce several key contributions to design a practical test-time scaling framework for WFMs, shown in Figure 4:

---

Published as a conference paper at COLM 2025
1. Fast-Tokenizer to Accelerate Decision Process To avoid running the diffusion
decoder at every generation step, we leverage the WFM's discrete-token decoder
as lightweight proxy. This "fast tokenizer" con- verts intermediate latent outputs into video frames in only s0.015 versus =130 for full diffusion decode. We find that the reward scores computed from fast-tokenized outputs show strong correlation with those from the diffusion decoder (see Figure 5). This tight alignment confirms that the fast tokenizer offers a reliable and drastically cheaper feedback sig- nal, enabling early pruning of low-quality trajectories and allowing compute to focus on more promising candidate paths.
Diffiziion Decoder
Takenizer Decader
1.25
2.00
1.76
Figure Scores for videos decoded by diffusion decoder (slow) and tokenizer decoder (fast) follow the same trend.
2. Probability-Baser Top-K Pruning for N Samples Video generation is inherently se- quential: early frames influence-buu do not guarantee-the quality of later frames. Rather than greedily selecting the single highest-scoring continuation at each step (as in basic PRM), we maintain a small pool of promising candidates. Concretely, at timestep we sample N next-step continuations {s}
{ i=1 and compute their verifier scores {r Instead of deterministically choosing the top-K by score, we perform probability-based selection using a softmax over verifier scores:
exp(ri/T)
Bi11 SampleTopK ((s)), (p:).K),
Pj
1 exp(r//T))
where 7 is a temperature hyperparameter controlling randomness, and SampleTopp draws K unique candidates without replacement. This stochastic top-K strategy introduces con- trolled exploration, yielding consistently better performance than both top-1 sampling and deterministic top-K sorting (Figure 6).
3. Beam-Search Algorithm to Maintain Efficiency To prevent an exponential explosion of candidate trajec- tories, we adopt that bounds search complexity at each generation step. Specifically, we maintain a fixed beam of K partial video sequences. each timestep, every beam member spawns M new continuations, yielding M x K candi- dates. We score all candidates using our fast tokenizer verifier, then prune back to the top K for the next step. By capping the beam size, this approach keeps branch- ing growth linear in sequence length, sharply reduc- ing wasted computation and ensuring inference cost remains predictable.
Model ineformance Cumpar son
1:35
procedure
1:30
0.20
0.5
0.10
0.00
Vadaa
Figure 6: Improvement based on COSMOS-4B using different search al- gorithms. Beam search with probabil- ity boosts the performance most.
Experiments
Experimental Settings. As aforementioned, we use autonomous driving as the test case to demonstrate the test-time scaling on world models. Here we focus on two modalities: image to video, and image+text to video. For autonomous driving, estimating coherent futures requires least three essential priors that inherently govern the future motion of instances in the scene: position, velocity, and acceleration, shown in Gao et al. (2024); Hu et al. (2024). So we use the variants that take 9 video frames as input.
Datasets and Metrics. We collect total of 900 input videos from the test splits of two representative autonomous driving datasets: nuScenes and Waymo, each containing 150 scenes. Specifically, sample input video from each nuŠcenes test scene and five inputs from each Waymo test scene. The accompanying text prompts are generated using a prompt upsampling model, which is fine-tuned from the base model in Mistral & NVIDIA.
7

---

Published as a conference paper at COLM 2025
by the COSMOS framework, ensuring that the prompts are aligned with the style and semantics expected by the world model.
For evaluation, following convention in autonomous driving field, adopt FID and FVD to assess the overall quality of the generated video frames and sequences, respectively. Note that nuScenes and Waymo are from two data sources thus have different distribution, so we report FVD and FID on each dataset. To provide additional insight, we also evaluate using the VBench and VideoScore (VQ, TC, DD, FC) benchmarks. Since our rule-based rewards partially overlap with the metrics in VBench, we use the remaining two dimensions (motion smoothness (MS), imaging quality (IQ)) from VBench that also focus on temporal consistency to perform cross-validation of our evaluation. To aggregate performance across multiple evaluation dimensions, we compute the geometric mean of the normalized scores, providing a single summary metric that reflects overall quality (as in Figure 3 and Figure 6).
5.1
Quantitative Results
Before conducting our test- time scaling experiments, we perform an ablation study on evaluation metrics to identify which ones are most suitable for use as reward functions in the autonomous driving domain. Using COSMOS- 4B as the base model, we apply naive best-of-N selection strategy and evaluate the final outputs using FVD.
Reward Ablation.
>emppna Consaberry
650
Perceptual Duality
3D Corsizittiy
640
Temporll Lonssterey + Peceppial Quality
630
600
190
The results, shown in Figure 7, reveal that tempo- ral consistency and perceptual quality contribute most significantly to performance improvements. In contrast, 3D consistency shows limited ben- efit-likely due to the nature of driving scenes, where long-range point clouds introduce noise and reduce the reliability of MSE-based geometric evaluation. We do not consider spatial relationship awareness in this setting since the spatial configurations in driving environments tend to be straightforward (e.g., vehicles aligned along roads), resulting in few violations of spatial constraints implied by the prompt.
10
Best
Figure 7: COSMOS-4F using different met- rics reward. We use Temporal Consis- tency and Perceptual Quality by default.
Model
FVD
FID
IQ
MS
VQ
TC
DD
FC
637.08/120.30
67.75/10.58
63.48
0.9822
3.86
3.56
3.86
3.68
622.82/120.16
58.93/10.27
63.97
0.9831
3.86
3.56
3.86
3.68
4B
613.77 117.39
52.01/10.25
64.33
0.9833
3.87
3.57
3.86
3.68
599.10/
116.14 199.10/116.14
19.27/10.14
64.65
0.9836
3.88
3.58
3.87
3.70
16
599.08 / 120.04
45.75/10.14
64.82
0.9839
3.90
3.59
3.89
3.73
12B
560.86/117:23
67.10/10.67
63.73
0.9807
3.94
3.63
3.93
3.76
Table 1: Comparison of COSMOS-4B and COSMOS-12B using naive best-of-N sampling. FVD and FID scores are reported separately for the nuScenes (left) and Waymo (right) datasets, due to significant differences in their underlying data distributions.
Naive Best-of-N. As shown in Table 1, we begin with a simple best-of-N selection strategy at the output level to demonstrate the effectiveness of test-time scaling. Our key obser- vations are as follows: 1)Test-time verification improves generation quality. Compared to the COSMOS-4B baseline, integrating a best-of-N (or ORM) as verifier significantly enhances performance across nearly all evaluation dimensions. 2) Test-time scaling follows clear trend. As increases, performance improves consistently, indicating strong scaling law even at inference time. 3) Compute-optimalitt at small N. Notably, best-of-2 strategy delivers substantial gains without exceeding the inference- titeencee-imeeeee compute budget of a much larger pretrained model (e.g., COSMOS-12B), demonstrating the practical value of lightweight scaling. While smaller models may have a performance gap compared to larger models in approximating the true data distribution, this limitation can be partially mitigated through
8

---

Published as a conference paper at COLM 2025
effective test-time selection. Moreover, prior work (e.g., Gao et al. (2024)) has shown that automatic metrics such as FVD do not always align well with human-perceiven quality. As result, we place greater emphasis multi-aspect evaluation across diverse benchmarks. model is considered improved when shows consistent gains across the majority of these evaluation dimensions.
Efficient Beam Search with Probability. To support real-world deployment, a search strategy that both efficient and reliable is critical. In Table 2, we compare our proposed probabilistic beam search against straightforward Process Reward Model (PRM) strategy, which greedily selects the top-1 token at each step based solely on reward scores.
Our findings reveal that current autoregressive world models-much like LLMs and LMMs-ofter suffer from unstable and inconsistent decoding paths. As result, incor- porating verification mechanism that can identify and follow more reliable generation trajectories crucial. Our probabilistic beam search strikes a strong balance between perfor- mance and robustness, achieving better results without incurring additional overhead.
Model
N
Alg.
FVD
FID
IQ
MS
VQ
TC
DD
FC
637.08/120.30
67.75/10.58
63.48
0.9822
3.86
3.56
3.86
3.68
PRM
614.00/116.51
52.68/11.48
63.79
0.9836
3.68
3.38
3.68
3.48
4B
4
Ours 612.68/11 /114.2
50.39/10.35
64.39
0.9837
3.86
3.56
3.85
3.67
PRM
616.89/122 /121007
47.29/10.33
64.87
0.9844
3.89
3.58
3.88
16
Ours
590.34/120.48
43.69/10.27
64.98
0.9846
3.92
3.64
3.90
3.74
12B
560.86/117.23
67.10/10.67
63.73
0.9807
3.94
3.63
3.93
3.76
Table 2:
Comparison between COSMOS-4B and COSMOS-12B under different search algo- rithms. N sample number. For our beam search with probability, M set as sqrt(N).
We also investigate the text-image-oo-video modality in Table and the findings are aligned with those on the image-to-video modality.
Model
N
Alg.
FVD
FID
IQ
MS
VQ
TC
DD
FC
728.68/126.63
59.71/10.47
63.48
0.9822
3.80
3.44
3.81
3.63
5B
ORM
659.79/111.96
59.45/10.01
63.90
0.9840
3.89
3.55
3.88
3.71
PRM
641.5/112.15 52.60/10.07
64.39
0.9843
3.89
3.55
3.88
3.71
4
Ours
628.01/110.02
50.16/9.98
64.77
0.9848
3.91
3.57
3.92
3.73
13B
616.92/109.77
59.71/11.48
63.79
0.9834
3.92
3.55
3.94
3.75
Table 3: Comparison between COSMOS-5B and COSMOS-13B under different search algo-
rithms. N is sample number. For our beam search with probability, M is set sqrt(N).
Human Evaluation. To bet- ter demonstrate the perfor- mance gain over larger pre- trained model, we opt for
1.0
1.0
0.8
Selection Ralio
0.8
Selection
human evaluation for more faithful analysis. Follow- ing recent advances Bar-Tal et al. (2024); Blattmann et al.
2.45
0.4
0.4
COSMOS-120
COSMUS-40 46 ORN N=2
COSM0S-120 COSMUS 46 Ueam w Prob
(2023a); Chen et al. (2023), Figure 8: COSMOS-4B with test-time scaling vs. COSMOS-12B. we adopt the Two-Alternative Forced Choice (2AFC) protocol. Specifically, participants are shown side-by-side video pairs and asked to select the one they find better. We collect total of 3,685 responses from 24 participants. As shown in Figure 8, outputs from the smaller model enhanced with test-time scaling often preferred over those from the larger baseline. While this study focuses

---

Published as a conference paper at COLM 2025
prompt
witt The non Taared prrertiggtt tryffic secegnaee sasucles inclade tiboer 2UV, isteriectin lie- some cale aur early nomnang Late everdng.
Figure 9: COSMOS-4B without (top) and with ORM (middle) and our (bottom) test-time scaling.
wailing meditalll light. Ther bi flllriedd streati ghex and tratic signsts, with grewn lasde and tree in the backgnound. Venidtee includs siive SUV, stepoed overcast, momiing evening.
seation,
Figure 10: COSMOS-5B without (top) and with ORM (middle) and our (bottom) test-time scaling.
scaling smaller models, note that applying test-time scaling to larger models can further push performance toward the frontier-an exciting direction we leave for future work.
5.2 Qualitative Results
We present qualitative examples in Figure 9 and Figure 10. Each example includes video outputs under different test-time scaling strategies. The baseline model frequently produces inconsistent objects, unstable backgrounds, or unnatural appearances. In contrast, test-time scaling consistently mitigates these issues isstes-enhancing enhancing temporal consistency and improving overall visual fidelity.
Conclusion
In this work, we presented SWIFT, the first test-time scaling framework specifically de- signed for world foundation models (WFMs). Addressing the high computational cost and data limitations of training and scaling WFMs, SWIFT offers an efficient alternative by reallocating computation during inference. Our framework combines a modular evaluation toolkit with process-leve test-time strategies, including fast tokenization, probability-based Top-K pruning, and beam search.
Empirical results on the COSMOS model demonstrate that test-time scaling not only im- proves output quality, but does so in a compute-optimal manner manner-allowing smaller models to match or even outperform larger ones under the same compute budget. These findings es- tablish that test-time scaling laws hold for WFMs and open up a practical, scalable pathway for deploying WFMs more efficiently, without retraining or model enlargement.
10

---

Published as a conference paper at COLM 2025
References
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. ar Xiv preprint ar Xix::2501.03555, 2025.
Hidehisa Arai, Keishi Ishihara, Tsubasa Takahashi, and Yu Yamaguchi. Act-bench: Towards
action controllable world models for autonomous driving. arXio preprint ar arXtt:2412.05337, 2024.
Omer Bar-Tal Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, et al. Lumiere: A space-time diffusion model for video generation. In SIGGRAPH Asia 2024 Conference Papers, pp. 1-11, 2024.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXie:2311.15127, 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: Fisgurresluttio video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22563-22575, 2023b.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint ar arXiv:2310.19512 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ar preprint ar xrXio:2110.14166 2021.
Matthew DeLorenzo, Animesh Basak Chowdhury, Vasudev Gohil, Shailja Thakur, Ramesh Karri,Siddharte Siddharth Garg, and Jeyavijayan Rajendran. Make every move count: Llm-based high-quality rtl code generation using mcts. arXiv preprint ar arXiv:2402.03289 2024.
Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation without vector quantization. ar Xiv preprint ar dxXio:2412.14169, 2024.
Rinon Gal, Adi Haviv, Yuval Alaluf, Amit H Bermano, Daniel Cohen-Or, and Gal Chechik. Comfygen: Prompt-adaptive workflows for text-to-image generation. arXiv preprint arXi::2410001331, 2024.
Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. Xrv preprint ar arXiv:2405.17398 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl:: Incentivizing reasoning capability in llms via reinforcement learning. ar preprint arXin:2501.112443 2025a.
Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng. Can generate images with cot? let's verify and reinforce image generation step by step. arXio preprint ar XxXi:22501.1326, 2025b.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint ar arXio:1803.10122, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint ar xrXe:11912.01633, 2019.
Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, et al. Videoscore: Building automatic metrics to simulate fine-grained human feedback for video generation. arXie preprint arXi::2406.15252, 2024.
11

---

Published as a conference paper at COLM 2025
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022.
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXit preprint ar Xix:::2009 17080, 2023.
Xiaotao Hu, Wei Yin, Mingkai Jia, Junyuan Deng, Xiaoyang Guo, Qian Zhang, Xiaoxiao Long, and Ping Tan. Drivingworld: Constructingworldd model for autonomous driving via video gpt. Xrxiv preprint ar arXiv:2412. 19505, 2024.
woryd
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, lianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11807-218188 2024.
Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint ar drXiv:2410.05954, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Twasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Inference-time scaling for diffusion models beyond scaling denoising steps. ar Xiv preprint ar arXio:2501.0973. 2025.
Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. Let's reward step by step: Step-level reward model the navigators for reasoning. arx preprint ar xxXio:2310.10000 2023.
Mistral and NVIDIA. Mistral-neemo-11b-inntuu A 12b parameter large language model. https://mistree /tt:::////ssrall.mmm wwt.eww//msstralnnnneeeee
2024.
Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-codd generation with execution. In International Conference on Machine Learning, pp. 26106-261288 PMLR, 2023.
Masashi Okada and Tadahiro Taniguchi. Dreamingv2: Reinforcement learning with dis- crete world models without reconstruction. In 2022 IEEE/RS] International Conference on Intelligent Robots and Systems (IROS), pp. 985-991. IEEE, 2022.
OpenAI. Learning to reason with 1lms. dearning-to-reesonn-ii wtwwwwhhhlmm////mmmm Accessed: Month Day, Year.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arxiv preprint wxXiv:2408.03311 2024.
Linzhuang Sun, Hao Liang, Jingxuan Wei, Bihui Yu, Conghui He, Zenan Zhou, and Wentao Zhang. Beats: Optimizing Ilm mathematical capabilities with backverify and adaptive disambiguate based efficient tree search. arXiv preprint ar Xit:2400.11972. 2024.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. arxiv preprint ar 2r1222312.0,8935, 2023.
12

---

Published as a conference paper at COLM 2025
Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. arxio preprint arXit:2501.12387, 2025.
Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting URL https://arxio. org/abs/2402.10220 2024.
Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning a physics simulator from video. Advances in neural information processing systems, 30, 2017.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Huajian Xin, ZZ Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du, et al. Deepseek-prover-t 5: Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search. arXio preprint arXiv:2408.08152, 2024.
Xiangyuan Xue, Zeyu Lu, Di Huang, Wanli Ouyang, and Lei Bai. Genagent: Build collabo- rative ai systems with automated workflow generation-case studies on comfyui. Xiv preprint ar XrXiv:2409.01392 2024.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: Lim self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735-64777 2025.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. ar preprint arXit:2410.10762,22 2024.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v2. Breaking the barrier of closed-source models in code intelligence. arXiv preprint X1X.:::2440.11311., 2024.
A More Related Works
World Models and Video Generative Models World models Ha & Schmidhuber (2018) aim to learn compact representations of the real world from sensory data, enabling reliable prediction future states. Traditional physics-basee controlled, low-dimensional environments but struggle with generalization across diverse scenarios. Recent advances Hafner et al. (2019); Okada & Taniguchi (2022) leverage deep learning construct world models directly from visual inputs, often employing recurrent architectures or generative approaches to capture dynamic real-world processes.
models Watters et al. (2017) excel in
Video generative models have progressed from early low-resolution systems modern frameworks that synthesize high-quality, realistic videos. Two primary approaches have emerged: diffusion-based models Ho et al. (2022); Blattmann et al. (2023b), known for their superior visual quality, and autoregressive models Deng et al. (2024); Jin et al. (2024), which effectively capture temporal dependencies by generating frames sequentially. While diffusion methods excel in visual fidelity, autoregressive techniques benefit from advances in language modeling and offer improved integration of complex reasoning. Our study leverages an autoregressive approach within world foundation models, demonstrating that scaling test-time computation can enhance verification and overall performance in autonomous driving contexts.
13

---

Published as a conference paper at COLM 2025
Scaling Test-Time Computation. Inspired by how humans allocate more cognitive effort to tackle complex problems, recent work has explored scaling test-time computation to improve the performance of Large Language Models (LLMs) across domains such as mathe- matical reasoning Wang et al. (2023); Zhang et al. (2025); Xin et al. (2024); Sun et al. (2024), code generation DeLorenzo et al. (2024); Ni et al. (2023); Zhu et al. (2024), and multi-step planning Zhang et al. (2024); Gal et al. (2024); Xue et al. (2024). One class of approaches enhances the input side, leveraging techniques like Chain-of-Thooue Thoouhh (CoT) prompting Wei et al. (2022); Kojima et al. (2022) to guide models through more structured reasoning, Others focus on the output side, introducing strategies such as self-consistency Wang & Zhou (2024), iterative decoding, or verifier-based selection Cobbe et al. (2021); Lightman et al. (2023); Snell al. (2024) improve response quality. Verifier-based methods, in particular, have proven to be robust and general-purpose. Outcome Reward Models (ORMs) evaluate final outputs to select the best among multiple candidates Cobbe et al. (2021), while Process Reward Models (PRMs) provide finer-grained feedback during generation by assessing intermediate reasoning steps Lightman et al. (2023); Ma et al. (2023). Recent work has shown that increasing inference-time compute rather than scaling model size-car can yield stronger gains Snell et al. (2024). Notably, OpenAl's ol model OpenAl demonstrates the effectiveness of such techniques across complex reasoning tasks. Building on these insights, we investigate whether verifier-based test-time scaling strategies can be adapted to enhance autoregressive video generation in world foundation models, where sequential consistency and high-fidelity synthesis are critical.
14


Reviews:
[{"review_id": "jk5MZjVFSt", "reviewer": "Reviewer_qtYN", "summary": "The paper studies test-time compute in the context of world foundation models. In particular, they (1) observe and quantify improvements in generation quality (including things like physical consistency, perception etc) of world foundation models with difference inference compute strategies (e.g. best of N, beam search). (2) They study reward design in detail, formulating video generation as an MDP.\n\nOverall this is a nice paper with interesting insights (e.g., it was interesting to see different process rewards at play -- it wasn't obvious which ones would be most effective). However, I am not entirely sure the paper is within scope of COLM. While the COLM call for papers is not entirely restricted to _language_ models (it does mention applications to multimodality and other domains), the current paper is entirely focused, as far as I can tell, towards video models, and is perhaps more suitable for a less LLM-specialized venue?", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Studies an interesting problem -- inference time scaling quality for video generation models. This has practical impact, as video gen models are costly to train, and one could simple apply more inference compute to smaller models for better quality.\n- Good empirical results -- e.g., shows evidence that test-time scaling laws apply to WFMs, showing that a 4B parameter model with test-time scaling can match or outperform a 13B model while using similar compute.\n- Solid technical contribution along various axes -- presents a system with multiple components (fast tokenizer, probability-based pruning, efficient beam search) designed specifically to make WFM inference efficient. Additionally, it was nice to see the effect of various rewards specific to video gen.", "reasons_to_reject": "- The paper doesn't seem very in scope for the venue, focusing entirely on video gen models.\n- Some of the emphasized results feel a bit obvious -- e.g., \"Observation 1\" that test-time scaling and improvements exist for WFMs seems fairly intuitive.  \n- It's unclear how generalizable the results are, since the authors studies one large WFM (given compute constraints). E.g., the results on compute-optimality at small N, motivating a best-of-2 strategy -- how general could this be?\n- Some sentence structure and grammar issues, e.g. \"similar to the Process Reward Model (PRM) in the LLM Ma et al. (2023)\"\n\nMy primary concern remains appropriateness for the venue, with other concerns being significant but secondary.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qSBi8vDVS8", "reviewer": "Reviewer_oo4B", "summary": "The paper introduces SWIFT, a test-time scaling framework specifically designed for World Foundation Models (WFMs), which are models that simulate the physical world by predicting future states from current observations and inputs. The authors address the challenge that WFMs require substantial computational resources for pretraining and are constrained by available data during post-training. Instead of building larger models or retraining, they propose using additional computation at inference time (test-time scaling) to improve performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1/ Addresses a significant practical challenge in WFMs - the high computational cost of training and inference\n\n2/ Provides a compelling alternative to simply building larger models, showing smaller models can achieve comparable performance with appropriate inference strategies\n3/ Introduces a much-needed evaluation toolkit specifically designed for WFMs, filling a gap where existing video generation benchmarks don't align with physical modeling goals\n\n4/ The fast tokenizer approach intelligently avoids running expensive diffusion decoding at every step\n\n5/ Demonstrates compute-optimal scaling, where a 4B model with test-time scaling can match a 12B/13B model's performance\n\n6/ Includes both rigorous quantitative metrics and human evaluation to validate the approach", "reasons_to_reject": "While the framework shows promising results, there's limited ablation of hyperparameters within SWIFT (such as the temperature parameter in probability-based selection)\n\nThe human evaluation includes only 12 participants, which may be insufficient given the complexity of video quality assessment\n\n\nThe test-time scaling approach still requires multiple inference passes, which may present deployment challenges in resource-constrained environments", "questions_to_authors": "The temperature parameter τ in your probability-based Top-K pruning seems important for balancing exploration and exploitation. Could you elaborate on how sensitive your results are to this parameter and how you selected its value?", "extra_scores": {}}, {"review_id": "qKaXBEQe5q", "reviewer": "Reviewer_vAmQ", "summary": "This paper discusses test-time scaling for video-generating world foundational models. A retraining-free test-time scaling framework, **SWIFT**, is proposed, which features: **(1)** alignment with goals of physical realism and consistency, and **(2)** a test-time inference strategy supporting CoT, ToT, etc., tailored for WFMs which typically use diffusion decoders.\n\n**The evaluation metric includes**: (1) 3D consistency evaluated with 3D foundational models; (2) Temporal Consistency evaluated with CLIP, DINO similarity scores; (3) Spatial Relationship Awareness, evaluated with ; (4) Perceptual Quality, assessed with LAION aesthetic predictor; and (5) Test-to-Video Alignment, evaluated with ClipScore and X-ClipScore; and this framework is extendable to other metrics.\n\n**For the Inference-time scaling method**, the authors discussed the reward choices, the action designs, and the inference-time search algorithm.\n\n**For reward choices**, they compare preference-based rewards (i.e., VideoScore) with Rule-based rewards for visual quality and temporal consistency (i.e., aesthetic quality and object permanence) on visual quality and temporal consistency on the task of autonomous driving.\n\n**For action designs**, they discuss why they use test-time search to explore inference-time scaling laws.\n\n**For Search Algorithms**, they (1) validate the usefulness of inference-time scaling with BoN methods; and (2) find that WFM's discrete-token decoder is a lightweight proxy to diffusion-based heavy decoders; (3) use Probability-based Top-k pruning as a improved BoN method and (4) use beam-search to maintain efficiency.\n\n**For experimental validation**, the authors conduct experiments based on driving scenarios in nuScenes and Waymo, evaluating FVD, FID and further metrics (including VQ, TC, DD, FC, MS, IQ, human evaluation, etc. ). The authors (1) conduct reward ablation to choose Temporal Consistency + Perceptional Quality as scaling reward; (2) show conceptually that Inference-time scaling can improve performance for small models with BoN; and (3) their proposed probabilistic beam search strikes balance between performance and robustness.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a **timely discussion** about improving performance of World Foundation Models, especially the video models, with Test-time scaling. The paper **validates the idea** that with test-time designs (i.e., reward designs, search algorithm designs, etc.), we can use smaller amount of compute to improve performance: small models + test-time inference can be compute-optimal compared to larger models, for World Foundation Models.\n\nThe evaluation framework proposed is **extendable**, and could be of good impact to the community to evaluate further models potentially.\n\nThe methods proposed looks **good and reasonable**, and comparisons with baselines and ablation studies indeed show that the authors' methods can improve performance while save cost.\n\nThe paper is **well written** with clear presentations about its methods, experiment designs and results.", "reasons_to_reject": "This work is mainly based on one series of WFM (i.e., the COSMOS-4B and COSMOS-12B models). Potential discussions about other WFMs could make this work even sounder.\n\nThe fast Tokenizer decoder is provided by the COSMOS work, and the intuition that using a tokenizer decoder is faster than diffusion decoder, is straightforward.\n\nA minor point: the evaluation metrics and discussions proposed are mainly for World Foundation Models which generate videos. Of-course this is very important and timely since many WFMs currently generate videos, but potential discussions about WFMs in other modalities (e.g., 3D meshes, 3D videos?) could be even more inspiring.", "questions_to_authors": "When it comes to comparing Rule-based rewards and Preference-based rewards (around Figure 3), the rule-based rewards are chosen as visual quality and temporal consistency scores, which I think is closer to the evaluation metric (i.e. visual quality and consistency here)? Hence is it natural that VideoScore-based reward cannot outperform rule-based rewards, since the output is evaluated with these rules? While for the FVD metric you use in Figure 7, the output is not evaluated with these rules, so perhaps Preference-based rewards (e.g. VideoScore-based reward) can out-perform rule-based rewards? Could you conduct experiments with preference-based rewards to validate (or diss-validate) this?\n\nI appreciate this work in general; I'm not asking the authors to conduct all experiments I mentioned (or implied) in weakness and questions section: some of these experiments are indeed very resource-consuming to carry out.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper provides an interesting illustration of how test-time scaling methods from LLM reasoning can be directly applied to world models to improve the performance and accuracy of world modeling. Reviewers indicated their excitement about the paper, and papers seems well suited to the CoLM community. Congratulations to the authors!
Decision: Accept