Field-DiT: Diffusion Transformer on Unified Video, 3D, and Game Field Generation

Authors: Kangfu Mei, Mo Zhou, Vishal M. Patel
Primary area: generative models
Keywords: generative model, diffusion model, diffusion probabilistic fields, world model

Abstract:
The probabilistic field models the distribution of continuous functions defined over metric spaces. While these models hold great potential for unifying data generation across various modalities, including images, videos, and 3D geometry, they still struggle with long-context generation beyond simple examples. This limitation can be attributed to their MLP architecture, which lacks sufficient inductive bias to capture global structures through uniform sampling. To address this, we propose a new and simple model that incorporates a view-wise sampling algorithm to focus on local structure learning, along with autoregressive generation to preserve global geometry. It adapts cross-modality conditions, such as text prompts for text-to-video generation, camera poses for 3D view generation, and control actions for game generation. Experimental results across various modalities demonstrate the effectiveness of our model, with its 675M parameter size, and highlight its potential as a foundational framework for scalable, architecture-unified visual content generation for different modalities with different weights. Our project page can be found at https://kfmei.com/Field-DiT/.

Decision: accept (poster)
Reviewer scores: 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730363840525

thread_type:
official_review

<thread_object>
creator: reviewer_x2pz

<content>
summary:
This paper focuses on unifying data generation across various modalities, including images, videos and 3D geometry. It introduces a view-wise sampling algorithm along with autoregressive generation to improve the performance. The proposed framework can handle various modalities with good performance.

strengths:
1. The proposed method is a unified framework for various modalities. This is a relatively new task that might benefit the community.

2. Extensive experiments and ablation studies demonstrate the effectiveness of the proposed method.

weaknesses:
1. **The motivation is unclear.** From the introduction section, the main motivation to use Diffusion Probabilistic Field (DPF) is handling various modalities together with a unified model. As for the unified models, what I understand is a single model that can generate different modalities. However, from the description in the method and experiment sections, each modality has different coordinate-signal pairs and the models are trained for each modality separately. If so, such a method cannot be regarded as a unified framework in my view.

2. **Comparison with conventional diffusion models.** In Line 142-144, when comparing DPF with conventional diffusion models, the main difference is that DPF can be applied to sparse observation of fields. However, in the view-wise sampling subsection (Line 244), each time sample the tokens in n views, which is a dense modeling instead of sparse sampling. 

3. **Comparison with DPF.** In my view, the main contribution of DPF is the context query pairs sampling and optimization. However, in Line 502, this paper mentions that the context query pairs are not used, which confuses me about the training objective in this paper. Does this paper use the diffusion optimization objective like epsilon-prediction or velocity-prediction? If so, the method is almost the same with DiT.

4. **Limited performance.** I do not see a part describing the dataset and hyperparameters used for training. So I assume the model is trained on each benchmark. If so, the performance is far from satisfactory since the compared methods are generalizable ones instead of fitting to a benchmark.

questions:
I am not an expert in Diffusion Probabilistic Fields, and the writing of this paper makes me even more confusing. I hope the authors could improve the writing and explain more background and related work. In addition, most of my concerns are about the explanation of the method and motivation. Please refer to weaknesses for more details.

## Post Rebuttal
After discussing with the authors, I have decided to assign a reject score to this paper. Below are the key concerns that remain unaddressed:

- **Limited Contribution of the Unified Architecture**: The contribution of a unified architecture trained separately for each dataset is limited, especially since a similar concept has already been introduced by DPF.  
- **In-Distribution vs. Out-of-Distribution Performance**: For a generalizable model, the in-distribution performance should surpass out-of-distribution performance. For instance, in the comparison with Zero-1-to-3 in 3D generation, this expectation is not met.  
- **Video Generation Quality**: The qualitative results for video generation on the rebuttal page are unsatisfactory. None of the examples demonstrate temporally consistent motion. In contrast, the webpage for the compared method, [Latte](https://maxin-cn.github.io/latte_project/), showcases much better video generation results.  
- **Game Generation Quality**: The qualitative results for game generation are also unsatisfactory. Comparing the author-provided [game recording](https://www.youtube.com/watch?v=Gum4GI2Jr0s) with the results on the rebuttal page, the color of the bricks at the bottom of the generated videos is inconsistent. Furthermore, the generated scenes are limited, lacking elements such as flowers and turtles present in the recording, as well as any underground scenarios.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730611703813

thread_type:
official_review

<thread_object>
creator: reviewer_sd88

<content>
summary:
This paper proposed a novel method for unified video, 3D, and game generation, by learning a DiT based mapping from metric space to signal space, which is able to process high-resolution inputs by the proposed view-wise sampling strategy, as well as maintaining global struture with introduced inductive bias such as text prompts. Results have demonstrated the effectiveness of the proposed method.

strengths:
1. With the view-wise sampling strategy, this method can scale up to high-resolution inputs
2. By introducing long context conditioning, cross-view consistency can be avoided to some extent

weaknesses:
1. The novelty is limited. From my perpective, the method proposed in this paper simply alters the sampling strategy of existing approaches through a straightforward design change, which trades off a reduced number of sampled views for a higher input resolution. Though the introduction of long context conditioning can compensate the global structure, this operation is common, and i don't this operation is powerful enough to recover the information lost during the process of view-wise sampling.

2. Since the method amis to learn a mapping from input coordinates to output properties, i think some other methods should also be compared, such as SIREN[1], and the difference between them should be clarified.

3. I'm wondering that whether the proposed method can be applied to more complex scenes generation, instead of simple objects in the task of 3D novel view synthesis.

[1] Implicit Neural Representations with Periodic Activation Functions, NeurIPS 2020.

questions:
Please see weakness. I'm gald to increase my scores if my concerns can be addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721541827

thread_type:
official_review

<thread_object>
creator: reviewer_4gwr

<content>
summary:
This paper presents a transformer-based diffusion field model to better capture global structures and long-context dependencies. It does that by introducing a view-wise sampling algorithm and incorporating autoregressive generation. The proposed method is a general framework that can be applied to multiple modalities, such as video, 3D and game. Extensive experiments are conducted to validate the effectiveness of the proposal.

strengths:
S1. The paper is well-written and mostly clear.

S2. The proposed view-wise sampling algorithm is interesting and novel.

S3. Exploiting autoregressive generation to preserve global geometry is reasonable. 

S4. The experiments are extensive, especially including various tasks.

weaknesses:
W1. As autoregressive generation is typically slower than parallel generation due to its sequential nature, the authors are encouraged to discuss the inference time of the proposed method and baseline methods.

W2. As shown in Table 1, the proposed method achieves better performance against baseline methods on image and video, but worse FID and LPIPS scores on 3D generation task. The authors are encouraged to discuss this phenomenon.

questions:
Please see the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732543382595

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Follow-Up on Paper #824

comment:
Dear Reviewers,

The discussion period will close in less than two days, and we haven’t heard back from you yet. We truly appreciate the time and effort you’ve dedicated to reviewing our paper. Could you please let us know if you have any follow-up concerns about our response?

Best regards,

Authors of Paper #824
</content>
</thread_object>

</thread>
<thread>
creation date:
1732841971486

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank You and Wishing You a Wonderful Holiday

comment:
We sincerely thank all the Reviewers for their thoughtful and valuable feedback. We hope you have a wonderful and safe Thanksgiving celebration with your friends and family.

As our current scores remain mixed without convergence, we kindly invite the Reviewers to continue the discussion upon returning from the Thanksgiving holiday.

Warm regards,

Authors of A Simple Diffusion Transformer on Unified Video, 3D, and Game Field Generation
</content>
</thread_object>

</thread>
<thread>
creation date:
1733264266289

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal and Discussion

comment:
Dear Area Chair and Reviewers,

We sincerely appreciate the constructive feedback and insightful suggestions throughout the review process. We are pleased to note the positive outcomes from the discussion phase:

- **Reviewer 4gwr** upheld the marginally positive score, acknowledging that our detailed responses effectively addressed concerns about inference efficiency and tradeoffs in video and 3D generation tasks.

- **Reviewer Sd88** increased their score after clarifications regarding the novelty of our unified architecture, discussions on comparison with SIREN, and considerations for complex scene generation in future work.

- We also note that **three out of four concerns initially raised by Reviewer x2pz**—including The motivation is unclear, Comparison with conventional diffusion models, and Comparison with DPF—were not reiterated in their subsequent comments following our responses. While this does not explicitly confirm resolution, it suggests that our clarifications may have alleviated these points to some extent. We take this as partial progress in addressing the reviewer’s feedback.

As the discussion concludes, we would like to summarize the key points addressed:

**Novelty and Contribution of Unified Architecture**

Our work introduces a novel architecture-unified approach capable of modality-specific parameterization for video, 3D, and game generation. This method integrates view-wise sampling and autoregressive modeling to preserve spatial-temporal consistency across modalities. While each modality is trained with distinct parameters, the architecture offers extensibility across domains. Comparisons highlighted our method’s strengths in balancing computational efficiency with quality.

**Advancements in Long-Context Sampling**

The proposed framework demonstrates long-context consistency, particularly in video and 3D sequences. We conducted detailed comparisons with models like Zero-1-to-3 and conventional diffusion methods, underscoring our contributions to reducing computational burdens without compromising reconstruction fidelity.

**Improved Presentation and Clarity**

We revised the manuscript to address ambiguities, including terminology adjustments (e.g., "architecture-unified" vs. "unified framework") and expanded discussions on dataset applicability and metric interpretations.

**Significance and Future Directions**

Our contributions extend beyond specific benchmarks, offering a foundation for further exploration in scalable, modality-unified generative models. As reviewers noted, future iterations could enhance qualitative outputs and expand evaluations on complex datasets like OpenVid-1M and Panda-70M.

Finally, we thank the reviewers for their invaluable comments, which have significantly improved our work. We remain committed to advancing the intersection of generative AI and unified architectures for visual content generation.

Best regards,

Authors of A Simple Diffusion Transformer on Unified Video, 3D, and Game Field Generation
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper attempts to define a single flexible architecture which can be applied to many different, diverse settings. The paper proposes to use Diffusion Probabilistic Fields, which maps the observations from a metric space (e.g. coordinate space) to the signal space (e.g. RGB). The paper's specific contribution is to identify that there is an intermediate point in the sampling space of the current two methods: diffusion, which always operate on complete data, and DPF, which only samples a very small subset of the data such as individual pixels. This paper instead proposes to define an intermediate level of sparsity such as subsets of views (e.g. comparing two full frames to a new generated one). They call this "view-wise sampling" and use auto-regressive generation to apply it. The paper demonstrates reasonable results on 3 different diverse tasks (video generation, 3D generation, and game generation), though it clearly lags behind SOTA, domain-specific techniques on each.

The strength of the paper is in its proposed view-sampling scheme, which while somewhat incremental, is interesting and relevant as well as the overall interesting topic: defining architectures for extremely diverse tasks using an adaptation to DPF. The main weaknesses are the fairly mediocre performance in the subtasks and the fact that the writing before rebuttal edits does not do a very good job of explaining: 1) that only the architecture unifies, 2) that their method mainly changes the sampling of DPF. For example, even the title seems to suggest something other than a sampling change.

I tend to agree with Reviewers 4gwr and Sd88 that the paper has limited novelty but a solid increment (in the view-wise sampling and autoregressive application). And I think the weakness of the results is made up by the fact that the problem itself is interesting and more niche. I recommend acceptance (poster).

Note authors should fix typos in in Figures 1 and 2 "filed"?