CaLMFlow: Volterra Flow Matching using Causal Language Models

Authors: Sizhuang He, Daniel Levine, Ivan Vrkic, Marco Francesco Bressana, David Zhang, Syed A Rizvi, Yangtian Zhang, Emanuele Zappala, David van Dijk
Primary area: generative models
Keywords: Flow matching, operator learning, large language models, single-cell transcriptomics, causal language models, integral equations

Abstract:
We introduce CaLMFlow (Causal Language Models for Flow Matching), a novel framework that casts flow matching as a Volterra integral equation (VIE), leveraging the power of large language models (LLMs) for continuous data generation. CaLMFlow enables the direct application of LLMs to learn complex flows by formulating flow matching as a sequence modeling task, bridging discrete language modeling and continuous generative modeling. Our method implements tokenization across space and time, thereby solving a VIE over these domains. This approach enables efficient handling of high-dimensional data and outperforms ODE solver-dependent methods like conditional flow matching (CFM). We demonstrate CaLMFlow's effectiveness on synthetic and real-world data, including single-cell perturbation response prediction, showcasing its ability to incorporate textual context and generalize to unseen conditions. Our results highlight LLM-driven flow matching as a promising paradigm in generative modeling, offering improved scalability, flexibility, and context-awareness.

Decision: reject
Reviewer scores: 3, 8, 3

Reviews and discussion:
<thread>
creation date:
1730647315600

thread_type:
official_review

<thread_object>
creator: reviewer_lgve

<content>
summary:
This paper proposes CALMFLOW, an autoregressive modeling method for continuous data. It tokenizes continuous data into small time segments and uses numerical solutions of ODEs to model the data. Experiments are conducted on synthetic data and real cell-perturbation data to evaluate the approach.

strengths:
The writing is clear and accessible.

The idea is easy to understand.

Experiments are conducted on both synthetic and real data, enhancing the validity of the results.

weaknesses:
Trivial Task: The paper focuses on continuous data modeling, which may not present a sufficiently challenging or novel task. The modeling of Gaussian distributions and the MNIST dataset appears trivial, and many existing multimodal methods can already handle cell data modeling effectively.

Mismatched Motivation and Experiment: Although the paper initially highlights the difficulty of solving ODEs, the experiments focus only on continuous data modeling and do not fully support or address the stated challenges with ODEs.

Overclaim of Novelty: The claimed tokenization method seems indistinguishable from standard window-based segmentation techniques, which questions the novelty of the approach.

questions:
Problem Definition: What is the task setup for the synthetic experiments? How many training and test data samples are used, and what does each training sample look like (e.g., is it a two-dimensional Gaussian distribution)?

Parameter Comparison: There is no parameter comparison between different methods, which makes the evaluation less fair for data-fitting tasks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708927438

thread_type:
official_review

<thread_object>
creator: reviewer_kkhb

<content>
summary:
The paper introduces CaLMFlow, a framework that leverages LLM for continuous data generation. The paper formulates flow matching as a sequence modeling task, and applies LLMs to learn the complex flows. By tokenizing the space and time, the approach enables efficient handling of high-dim data. On a range of tasks (e.g., single-cell perturbation response prediction), the method shows strong performance.

strengths:
The technical idea is original and natural, blending LLMs into the framework of VIE flow matching. 

Extensive experiments were carried out on a range of tasks, covering synthetic and real-world data. 
The analysis and ablation studies also show the importance of each component/technique in the method, providing insights for future improvements. 

The paper is generally clear.

weaknesses:
The method seems general but the only kind of real data in the experiments are single-cell data. Without further evidence, it is hard to judge whether the significance will be high or not, broad or not.

The writing has some typesetting issues. E.g., 
- line-213, $T$ tokens 
- line-219, $D_{text{in}}$

questions:
What are the author's thoughts on generalizing the method to other kind of spatiotemporal data, e.g., high-dim time series, video data, etc?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730844920051

thread_type:
official_review

<thread_object>
creator: reviewer_htmr

<content>
summary:
In this paper, the authors propose to use an autoregressive (language) model to imitate a (discretized) flow from the prior to the target distribution. This has a clear advantage, as such an autoregressive model with an unbounded context can take into account non-local structures across multiple timesteps, unlike previous approaches that model a purely local transition (a la “difference”). Once training is over (in the next-token prediction way), one can simply run the autoregressive model forward starting from a sample drawn from the prior distribution to draw a sample from the target distribution.

strengths:
It is only natural to extend any of these iterative refinement based approaches to learning to sample from a complex distribution to make each refinement step less local and more global. Although it is natural, it has been challenging to do so until recently, as it was not clear whether we can build a powerful neural net that can take as input a long sequence of a trajectory and use it properly. With the recent advances in language models, this doubt is no more, and the authors in this paper demonstrate that indeed we can use such an autoregressive model to build a better sampler based on iterative refinement. Despite the unnecessarily convoluted way of presenting it via so-called Volterra flow, the idea is extremely straightforward, and the authors’ limited experiments do support that there is a benefit to be had from such non-local iterative refinement. It is furthermore interesting to see that they could easily extend it by prefixing their iterative refinement sampler with a natural language description to make it language conditional.

weaknesses:
Unfortunately the current manuscript is extremely difficult to read. One reason i can point out is due to the lack of clear exposition on how this underlying autoregressive model looks like; what does it take as input, and how the prefix is processed to result in the prediction of the next step’s refined observation? Furthermore, the authors’ use of the term “tokenization” confused me quite a lot, as “tokenization” is often used to refer to as the process by which we quantize a continuous observation into a finite set of discrete tokens, while the authors here are referring to discretizing continuous time and space. Such issues with presentation make it pretty much impossible for me to recommend the manuscript in the current form to be accepted at the venue. I would suggest the authors make a major revision to present the proposed approach (which I think may have an interesting impact on the community) more clearly and thoroughly.

questions:
This is more of a question to the other reviewers as much as it is to the authors. I am not too familiar with usual practices when these flow-based approaches are introduced and compared. Are these experiments standard ones that people have used to compare different iterative refinement based samplers? The first set of experiments on mixtures of Gaussians look extremely low-dimensional, and the second set of experiments on single cell sequencing data look relatively unconventional (I like this problem but it is difficult for me to understand that this is one of the standard problems people use to compare these samplers.) I may be completely wrong here (and if so, please do point out,) but I would like the authors to clarify what are more or less standard benchmarks in this area and how the proposed approach compares to the existing ones on those.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732294252533

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank the reviewers for their comments and suggestions for the manuscript. With only a few days left in the discussion period, we would be grateful to hear your thoughts and responses to our comments, answers, and revised manuscript.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces CaLMFlow (Causal Language Models for Flow Matching), a novel framework that reformulates flow matching as a Volterra integral equation (VIE) (instead of a differential equation), leveraging large language models (LLMs) for continuous data generation. The authors claim that CaLMFlow bridges discrete language modeling and continuous generative modeling by implementing tokenization across space and time, enabling efficient handling of high-dimensional data. The method is demonstrated on synthetic and a real-world data application (single cell sequencing dynamic), showcasing its ability to incorporate textual context and generalize to unseen conditions.

**Strength: Timely, Innovative Connection of two Active Areas of Research**. There is a long line of work aiming to repurpose neural network architectures typically used in language modeling (RNNs, LSTMs, Transformers) for continuous data domains. This paper fits withing that broader literatures, approaching this problem by marrying LLMs and flow matching, two very active areas of research, but so far mostly independent. Therein lies, in my view, the main strength of the paper: an innovative approach that presents a unique application of LLMs to flow matching, bringing the promise of flow matching to other domains. The use of integral (instead of differential) equations to model flow matching is interesting, but not an original contribution of this work (see Zappala et al.). Furthermore, I think its use requires more motivation. The paper makes a broad argument that IEs should be preferred over ODEs, but there's inherent computational-robustness tradeoffs that are not discussed here in depth. Additionally, the paper implicitly assumes that the conditions under which IEs might be preferable (singularities, non-locality, weak solutions) are ipso facto true in the applications of interest, but this does not seem to be case in the synthetic Gaussian experiments (despite their high-dimensionality), and unclear in the single cell perturbation ones. A stronger case for VIEs would have been made with an application where ODEs provably suffer from such issues, or to demonstrate that the decreased performance of ODE-based CFM in table 1 is due to stiffness or instability and not implementation issues. 

**Weakness: Methodological choices lack motivation (see above)**. 

**Weakness: Clarity and Presentation**. The reviewers noted that the manuscript is difficult to read due to unclear exposition and confusing terminology, particularly around the use of “tokenization.”. The revised version made important improvements in clarity, but introduced substantial changes. While such improvements are welcome, the fact that such an extensive rewriting was necessary suggests that the paper was submitted in a form not ready for prime time. Furthermore, while the revised version is definitely easier to read, many of the points raised by reviewer ktMR about missing details still stand. 

**Weakness: experimental evaluation**. As observed by all three reviewers, the experimental framework consists of very simplistic synthetic datasets and non-standard (for flow matching) single-cell generation task. This makes it hard to compare the method to alternatives. Additionally, there does not seem to be experimental results comparing the computational cost of this method against the baselines. This is particularly problematic given the known robustness-computation tradeoff of solving integral vs differential equations. 

Overall, I think this is a borderline paper with high unrealized potential that would benefit from a broader evaluation, rewriting, and an additional round of peer review.