INRFlow: Flow Matching for INRs in Ambient Space
Authors: Yuyang Wang
Anurag Ranjan
Joshua M. Susskind
Miguel Ángel Bautista
Venue: ICML 2025 poster (2025)
Keywords: Flow Matching
INRs
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: A flow-matching model for learning probabilistic INRs at high resolution for multiple data domains

Abstract: Flow matching models have emerged as a powerful method for generative modeling on domains like images or videos, and even on irregular or unstructured data like 3D point clouds or even protein structures. These models are commonly trained in two stages: first, a data compressor is trained, and in a subsequent training stage a flow matching generative model is trained in the latent space of the data compressor. This two-stage paradigm sets obstacles for unifying models across data domains, as hand-crafted compressors architectures are used for different data modalities. To this end, we introduce INRFlow, a domain-agnostic approach to learn flow matching transformers directly in ambient space. Drawing inspiration from INRs, we introduce a conditionally independent point-wise training objective that enables INRFlow to make predictions continuously in coordinate space. Our empirical results demonstrate that INRFlow effectively handles different data modalities such as images, 3D point clouds and protein structure data, achieving strong performance in different domains and outperforming comparable approaches. INRFlow is a promising step towards domain-agnostic flow matching generative models that can be trivially adopted in different data domains.

INRFlow: Flow Matching for INRs in Ambient Space
Yuyang Wang' Anurag Ranjan
12 Josh Susskind
Miguel Angel Bautista'
Abstract
proposed following this idea and pushing the quality of generated samples in many different domains, including images (Dai et al., 2023; Rombach et al., 2022; Preechakul et al., 2022), 3D point clouds (Luo & Hu, 2021), graphs (Hoogeboom et al., 2022) and video (Ho et al., 2022a). More recently. flow matching (Lipman et al., 2023) and stochastic interpolants (Ma et al., 2024) have been proposed as generalized formulations of the noising process, mov- ing from stochastic gaussian diffusion processes to general paths connecting a base (e.g. Gaussian) and target (e.g. data) distribution.
Flow matching models have emerged as power- ful method for generative modeling on domains like images or videos, and even on irregular or unstructured data like 3D point clouds or even protein structures. These models are commonly trained in two stages: first, a data compressor is trained, and in subsequent training stage a flow matching generative model is trained in the latent space of the data compressor. This two- stage paradigm sets obstacles for unifying models across data domains, as hand-crafted compressors architectures are used for different data modalities. To this end, we introduce INRFlow, domain- agnostic approach to learn flow matching trans- formers directly in ambient space. Drawing inspi- ration from INRs, we introduce conditionally independent point-wise training objective that en- ables INRFlow to make predictions continuously in coordinate space. Our empirical results demon- strate that INRFlow effectively handles different data modalities such as images, 3D point clouds and protein structure data, achieving strong per- formance in different domains and outperforming comparable approaches. INRFlow is a promis- ing step towards domain-agnostic flow matching generative models that can be trivially adopted in different data domains.
In practice, these iterative refinement approaches are com- monly applied in a latent space obtained from a pre-trained compressor model. Thus, the training recipe consists of two independent training stages: compressor (auto-encoder) training and subsequent generative modeling in latent space. General purpose transformer architectures have been used in the generative modeling step in latent space (Peebles & Xie, 2023; Ma et al., 2024; Esser et al., 2024). However, the first stage compressor uses architectures that are specific to the data domain, requiring hand-crafted inductive biases (i.e. ConvNets for image data (Rombach et al., 2022), PointNet for point clouds (Vahdat et al., 2022), Evoformer for protein structures (Jumper et al., 2021)).
We see this as one of the core issues preventing the ML com- munity to develop truly domain- aggosticc generative models that can be applied to different data domains in a trivial man- ner. Our goal in this paper to provide powerful single training stage approach that is donain-agnostic and simple to implement in practice, thus dispensing with the complex- ities of two-stage training recipes and enabling modeling of different data modalities directly in ambient (i.e. data) space.
1. Introduction
Recent advances in generative modeling have enabled learn- ing complex data distributions by combining both powerful architectures and training objectives. In particular, state-of- the-art approaches for image (Esser al., 2024), video (Dai al., 2023) or 3D point cloud (Vahdat et al., 2022) gener- ation are based on the concept of iteratively transforming data into Gaussian noise. Diffusion models were originally
It is worth noting that training diffusion or flow matching models in ambient space indeed possible when using do- main specific architecture designs and training recipes. In the image domain, approaches have exploited its dense na- ture and applied cascaded U-Nets (Ho et al., 2021; 2022b), joint training of U-Nets at multiple resolutions (Gu et al., 2023), multi-scale losses (Hoogeboom et al., 2023) or U-Net transformer hybrids architectures (Crowson al., 2024), ob- taining strong results. However, developing strong domain- agnostic models, using general purposes architectures that
Apple, Machine Learning Research "Work done while Ap- plc. Correspondence to: Yuyang Wang <yuyangw @ wrrrrangw@@ppll.cooo applc.com>, Miguel Angel Bautista <mbautistamartin @ aop..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

INRFlow: Flow Matching for INRs in Ambient Space
[0.1]
Value
Encoder
Condinatt
Z ft
Cocrdinate
Value Y, E R
Velocity
x X
Decoder (X-Attn)
G
WH x 5
pisels indepandently
WH 3
(a)
(b) ImageNet-222
(c) Image-to to-3D
(d) Protein Folding
Figure 1. (a) High level overview of INRFlow using the image domain as an example. Our model can be interpreted as an encoder-decoder model where the decoder makes predictions independently for each coordinate-val valuee pair given イ₁,- For different data domains, the coordinate and value dimensionality changes, but the model is kept the same. (b) Samples generated by INRFlow trained on ImageNet 256x256. Image-to-3D point clouds generated training INRFlow Objaverse (Deitke et al., 2023). (d) Protein structures generated by INRFlow trained on SwissProt (Boeckmann et al., 2003). GT protein structures are depicted in green while the generated structures by INRFlow are show in orange.
can applied across different data domains remains an important open problem.
13). Our contributions are summarized as follows:
In this paper, answer three part question: Can learn flow matching models in ambieni space, in single training stage and using domain agnostic architecture? Our goal is to unify different data domains under the same training recipe. To achieve this, we introduce INRFlow (INRFlow), see Fig. I(a). INRFlow makes progress towards the goal of unifying flow matching generative modeling across data domains. Drawing inspiration from INRs, formulate conditionally independent point-wise training objective that enables training directly in ambient space and can be densely (e.g. continuously) evaluated during inference. In the image domain, this means that INRFlow pixel value given its coordinate
We propose INRFlow, flow matching generative trans- former that works ambient space to enable single stage generative modeling different data domains.
Our results show that INRFlow, though domain- agnostic, achieves competitive performance on image and 3D point cloud generation compared with strong sppeifc baselines.
Our point- wise training objective allows for efficient training via sub-sampling dense domains like images while also enabling resolution changes at inference time.
models the probability (i.e. the probabilistic extension of an INR), allowing to generate images at different resolution than the used during training (see Fig. 4(a)). We show generated samples from INRFlow trained on ImageNet-256 in Fig. 1(b), image- to-3D on Objaverse in Fig. 1(c) and protein structures on SwissProt Fig. 1 (dd (see additional samples Fig. 9, 12,
2. Related Work
Diffusion models have been the major catalyzer of progress in generative modeling, these approaches learn to reverse forward process that gradually adds Gaussian noise to corrupt data samples (Ho al., 2020). Diffusion models
2

---

INRFlow: Flow Matching for INRs in Ambient Space
notable for their simple and robust training objective. Extensive research has explored various formulations of the forward and backward processes (Song et al., 2021; Ris- sanen et al., 2022; Bansal et al., 2022), particularly in the image domain. In addition, different denoising networks have been proposed for different data domains like images (Nichol & Dhariwal, 2021), videos (Ho et al., 2022a), and geometric data (Luo & Hu, 2021). More recently, flow matching (Liu et al., 2023; Lipman al., 2023) and stochas- tic interpolants (Ma et al., 2024) have emerged as flexible formulations that generalized Gaussian diffusion paths, al- lowing to define different paths to connect a base and a target distribution. These types of models have shown in- credible results in the image domain (Ma et al., 2024; Esser et al., 2024) when coupled with transformer architectures (Vaswani et al., 2017) to model distributions in latent space learnt by data compressors (Peebles & Xie, 2023; Ma et al., 2024; Rombach et al., 2022; Vahdat et al., 2022; Zheng et al., 2023; Gao et al., 2023). Note that these data com- pressors use domain specific architectures with hand-crafted inductive biases.
from 2D pixel coordinates æ E R2 to corresponding RGB values y E R thus f : IR2 - R3, where each image is different map. For 3D point clouds, f can be interpreted as a deformation that maps coordinates from fixed base configuration in 3D space to a deformation value also in 3D space, f : IR + IR" as in the image case, each 3D point cloud corresponds to a different deformation map f. For ease of notation, we define coordinates æ and values y of any given map f as f and ys. respectively. Fig. I(a) shows an example of such maps in the image domain.
In practice, analytical forms for these maps f are unknown. In addition, different from previous approaches (Dupont et al., 2022a; Du et al., 2021a), we do not fit separate INRs to each data sample via reconstruction, since that would involve a separate training stage fitting an MLP for each map (Dupont et al., 2022a; Bauer et al., 2023; Du et al., 202 As result, we assume we are only given sets of corresponding coordinate and value pairs resulting from observing these maps at a particular sampling rate (e.g. at a particular resolution in the image case). In the following, we develop an end-to-end approach that can directly take these coordinate-value sets as training data and train model that extends INRs to the probabilistic setting.
In an attempt to unify generative modeling across various data domains, continuous data representations (also referred to as implicit neural representation, neural fields neural operators) have shown potential in different approaches: From Data to Functa (Functa) (Dupont et al., 2022a), Gen- crative Manifold Learning (GEM) (Du et al., 2021a), and Generative Adversarial Stochastic Process (GASP) (Dupont et al., 2022b) have studied the problem of generating contin- uous representations of data. More recently Infinite Diffu- sion (Bond-Taylo & Willcocks, 2023) and PolyINR (Singh et al., 2023) have shown great results in the image domain by modeling images as continuous functions. However. both of these approaches make strong assumptions about image data. In particular, (Bond-Taylor & Willcocks, 2023) inter- polates sparse pixels to an euclidean grid to then process it with U-Net. On the other hand, (Singh et al., 2023) uses a patching and 2D convolution in the discriminator. Our ap- proach also relates to DPF (Zhuang et al., 2023), diffusion model that acts on function coordinates and can be applied in different data domains on a grid at low resolutions (i.e. 64 x 64). Our approach is able to deal with higher resolu- tion functions (e.g. 256x256 vs. 64x64 resolution images) on large scale datasets like ImageNet, while also tackling unstructured data domains that do not live on an Euclidean grid (e.g. like 3D point clouds and protein structures).
3.2. Flow Matching and Stochastic Interpolants
We consider generative models that learn to reverse a time- dependent forward process that turns data samples (i.e. maps f in our case) f q(f) into noise ~ I).
Je = 0+ + Oze
(I)
Both flow matching (Lipman et al., 2023) and stochastic interpolant (Ma et al., 2024) formulations build this forward process in Eq. 1 so that it interpolates exactly between data samples f at time L = and at time / = 1, with E [0, 1]. In particular, (f( ~ .N(0, and po( R q(). In this case, the marginal probability distribution pc(J) of ʃ is equivalent the distribution of the probability flow ODE with the following velocity field (Ma et al., 2024):
deft = (fffdd
(2)
where the velocity field is given by the following conditional expectation,
3. Method
u.( = Eldesele = ʃ] = = d.a,E = dro, = =s].
3.1. Data as Continuous Coordinate- + Value Maps
(3)
We interpret our empirical data distribution q to be com- posed of maps f ~ (f). These maps take coordinates x as input values y as output. For images, maps are defined
Under this formulation, samples fo ~ -Po(f) are gener- ated by solving the probability flow ODE in Eq. 2 back- wards in time (e.g. flowing from to f = 0), where
3

---

INRFlow: Flow Matching for INRs in Ambient Space
Po(J) R q(J). Note that both the flow matching (Lipman et al., 2023) and stochastic interpolant (Ma et al., 2024) formulations decouple the time-dependent process formu- lation from the specific choice of parameters au and o allowing for more flexibility. Throughout the presentation of our method we will assume a rectified flow (Liu et al., 2023; Lipman et al., 2023) or linear interpolant path (Ma et al., 2024) between noise and data, which define a straight path to connect data and noise: fa = (1 fo + tc. Note that our framework for learning flow matching models for coordinate-value sets can be used with any path definition. Compared with diffusion models (Ho et al., 2020), linear flow matching objectives result in better training stability and more modeling flexibility (Ma et al., 2024; Esser et al., 2024) which we observed in our early experiments.
are not independent from each other and natural images show strong both short and long spatial dependencies across pixels. In order to solve this, we introduce a latent variable f. that encodes contextual information from a set of given coordinate and value pairs of ft- This contextual latent vari- able allows us to formulate the learnt velocity field to be conditionally independent for coordinate-value pairs given z,, The final point-wise conditionally independent CFM loss, which we denote CICFM loss is defined as:
Ee-ee00.1,1/-a//lllloood Us. (25,.) ur (x ys 1
(4)
where the target velocity field (z ylc) is defined as a rectified flow (Liu et al., 2023; Lipman et al., 2023; Ma et al., 2024): (,( = I-- U/.
3.3. INRFlow
One of the core challenges of learning this type of generative models is obtaining a latent variable 2,. that effectively captures intricate dependencies across the domain of the function, specially for high resolution stimuli like images. In particular, the architectural design decisions are extremely important to ensure that zy, does not become bottleneck during training. In the following we review our proposed architecture
We now turn to the task of formulating a flow matching training objective for data distributions of maps f. We recall that in practice we do not have access to analytical or parametric form for these maps f (e.g. we do not requite to pretrain INRs for each training sample), and we are only given sets of corresponding coordinate 2 and value ys pairs resulting from observing the mapping at a particular rate. As a result, we need to formulate a training objective that can take these sets of coordinate-valus as training data.
3.4. Network Architecture
In order to achieve this, we first observe that the target locity field u, (ff))d, can be decomposed across both the domain and co-domain of ft, resulting in poiet-wise ve- locity field uz f y,,, )d,, defined for corresponding coor- dinate and value pairs of ft. As an illustrative example in the image domain, this means that the target velocity field can be independently evaluated for any pixel coordinate 2 with corresponding value yf, so that u. ua( fo yr,) E IR Note that one can always decompose target velocity fields in this way since the time-dependent forward process in Eq. I aggregates data and noise independently (e.g. point-wise) across the domain of f. Again, using the image domain as an example, the time-dependent forward process of a pixel at coordinate y is not dependent on other pixel positions or values.
We base our model on the general PerceiverlO design (Jae- gle et al., 2022), Fig. 2 illustrates the architectural pipeline of INRFlow. At a high level, our encoder network takes a set of coordinate-valle pairs and encodes them to Icarnable latents through cross-attention. These latents are then up- dated through several self-attention blocks to provide the final latents zf, E R. To decode the velocity field for given coordinate-value pair we perform cross attention to fe, generating the final point-wiig wise prediction for the velocity field (2 y5. 4[z,))-
The encoder of a vanilla PerceiverIO relies solely on cross- attention to the latents 2f, E <. learn spatial connec- tivity patterns between input and output elements, which we found to introduce strong bottleneck during training. To ameliorate this, we make a key modifications to boost the performance. Firstly, our encoder utilizes spatial aware latents where each latent is assigned a "pseudo" coordi- nate. Coordinate- valuee pairs are assigned to latents based on their distances on coordinate space. During encoding. coordinate-valle value pairs interact with their assigned latents through cross cross-attention attention, this means that each of the L latents
Our goal now is to formulate training objective to match this piint-wise independent velocity field. We want our neu- ral network Uo parametrizing the velocity field to be able to independently predict a velocity for any given coordinate and value pair 3 and y. However, this point-wise in- dependent prediction is futile without access to additional contextual conditioning information about the underlying function fr at time t. This is because even if the forward process is pisn independent, real data exhibits strong dependencies across the domain f that need to be captured by the model. For example, in the image domain, pixels
only attends to set of neighboring coordinate-vall Latent vectors are then updated using several self- self-attention blocks. These changes in the encoder allow the model to effectively utilize spatial information while also saving com- pute when encoding large coordinate- value sets on ambient
pairs.
4

---

INRFlow: Flow Matching for INRs in Ambient Space
Encoder
XN
Coord
Value Y/
Spatial aware latents
ft
Self Attention
R3
R3
R
R
Or
M
Or
Velocity
Cross Attention
X Coord
4/ Value
R² + R3
R² R
Decoder
Figure 2. Architecture of proposed INRFlow for different data domains including images and 3D point clouds. Note that models are trained for each data domain separately. Each spatial aware latent takes in a subset of neighboring context coordinate-valle sets in coordinate space. The latents then updated through self-attention. Decoded coordinate-value pairs attend to the updated latents to decode the corresponding velocity.
Model
FTHQQ2266
Church-256
space.
Domain specific models
CIPS (Amokhin et al., 2021)
5.29
10.80
4. Experiments
StyleSwin (Zhang al., 2022)
3.25
8.28
UT (Bond-Taylor et al., 2022)
3.05
5.52
StyleGANZ (Karras et 2020)
evaluate INRFlow on two challenging problems: image generation (FFHQ-256 (Karras al., 2019), LSUN-Churee 256 (Yu et al., 2015), ImageNet- 128/256 (Russakovsky al., 2015)), image-to-3D point cloud generation (Obja- verse (Deitke et al., 2023)) and protein folding (SwissProt (Boeckmann et al., 2003)), Note that we use the same train- ing recipe for all tasks, adapted for changes coordinate- value pair dimensions in different domains. See App. for more implementation details and training settings.
Fincciion space models GEM (Da et al., 2021h)
35.62
87.57
GASP (Dupont et al., 2022c)
24.37
37.46
Do-Diff (Bond-Taylor Willcooks, 2023)
INRFlow B (ours)
2.46
7.11
INRFlow (purs)
Table 1. FIDeLPP (Kynkäänniemi al., 2023) results for state-of- the-art function space approaches.
proaches, training INRFlow on ImageNet at both 128 128 and 256x256 resolutions. On ImageNet-122. shown in Tab. 2, INRFlow achieves an FID of 2.73, which is a a competitive performance in comparison to diffusion or flow- based generative baselines including ADM (Dhariwal Nichol, 2021), CDM (Ho al., 2021), and RIN (Jabri al., 2023) which domain-specific architectures for image generation. Besides, comparing to PolyINR (Singh et al., 2023) which also operates on function space, INRFlow achieves competitive FID, while obtaining better IS, preci- sion and recall. In addition, we report results of INRFlow for ImageNet-256 on Tab. 3. We observed that INRFlow slightly outperformed by latent space models like DiT (Pee- bles & Xie. 2023) and SiT (Ma al., 2024). We highlight that these baselines rely pre-trained VAE compressor that was trained on datasets (i.e. 9.29M images) that are much larger than ImageNet, while INRFlow was trained only with ImageNet data. In addition, INRFlow achieves better performance than many of the baselines trained only with ImageNet data including ADM (Dhariwal & Nichol,
4.1. Image Generation
Given that INRFlow is an probabilistic extension of INRs we compare with other generative models of the same type, namely approaches that operate in continuous func- tion spaces. Tab. 1 shows comparison of different image domain specific, as well as, function space models (e.g. approaches that model Flow surpasses other generative models in function space on both FFHQ (Karras et al., 2019) and LSUN-Church (Yu al., 2015) at resolution 256 x 256. Compared with genera- tive models designed specifically for images, INRFlow also achieves comparable better performance. When scaling up the model size, INRFlow-L demonstrates better perfor- mance than all the baselines on FFHQ-256 and Church-256, indicating that INRFlow can benefit from increasing model sizes.
signals). INR-
We also evaluate the performance of INRFlow on large scale settings previously untapped for domain agnostic ap-
5

---

INRFlow: Flow Matching for INRs in Ambient Space
Class-Conditioaal ImageNet 128x128
generation).
Model
FIDL
1St
Prect
Reet
Tab. 4 shows the performance of INRFlow in comparison with recent baselines on Objaverse. We report ULIP-I (Xue et al.. 2024) and P-FID (Nichol et al., 2022) following CLAY (Zhang et al., 2024). PointNet++ (Qi et al., 7a;b; Nichol et al., 2022) is employed to evaluate P-FID. ULIP-I is an analogy to CLIP for rext-to-image generation. ULIP-I is measured as the cosine similarity between point-cloud features from ULIP-2 model (Xue et al., 2024) and image features from CLIP model (Radford et al., 2021). The per- formance numbers of baseline models are directly borrowed from CLAY (Zhang et al., 2024). We calculate the met- rics of our INRFlow on 10k sampled point clouds. In our case, P-FID is measured on point clouds with 4096 points following Shape-E (Jun & Nichol, 2023) while ULIP-I is measured point clouds with 10k points following ULIP- 2 (Xue et al., 2024). Note that since CLAY (Zhang et al., 2024) is not open-source, we do not have the access to the exact evaluation setting or the conditional images rendered from Objaverse. But all evaluation settings of INRFlow are provided for reproduction purpose. As shown in Tab. 4, our INRFlow achieves strong performance on image-to-3D generative tasks. Compared to CLAY (Zhang et al., 2024), which is a 2-stage latent diffusion model, INRFlow demon- strates very strong performance on both ULIP-I and P-FID.
Adversarial BigGAN-deep (Brock et al., 2019)
6.02
145.8
0.86
0.35
PolyINR (Singh et al., 2023)
179.0
0.70
0.45
Difiisson models
CDM (w/clg) (Il al., 2021)
3.52
128.0
ADM (w/ <fg) (Dhariwal & Nichol, 2021)
2.97
141.3
0.78
0.59
RIN (Jabri al., 2023)
2.75
144.0
INRFlow-XC Xt. (ours) 1.5)
2.73
187.6
0.80
0.58
Table Benchmarking classicon image generation on Im- ageNet 128x128.
2021), CDM (Ho et al., 2021) and Simple Diffusion (U- Net) (Hoogeboom et al., 2023) which all use CNN-based architectures specific for image generation. Note that this is consistent with the results show in Tab. I, where INRFlow outperforms all function space approaches.
When comparing with approaches using transformer archi- tectures we find that INRFlow obtains performance com- parable RIN (Jabri et al., 2022) and HDiT (Crowson et al., 2024), with slightly worse FID and slightly better IS. However, INRFlow is a domain-agnosti architecture that can be trivially applied to different data domains like 3D point clouds or protein structure data (see Sect. 4.2 and Sect. 4.3). For completeness, we also include billion scale U-Net transformer hybrid models, Simple Diffusion (U-ViT 2B) and VDM++ (iT 2B). We highlight that the sim- plicity of implementing and training INRFlow models in practice, and the trivial extension to different data domains are strong arguments favoring INRFlow. Finally, compar- ing with e.g. PolyINR (Singh et al., 2023) which is also a function space generative model we also find comparable performance, with slight worse FID but better Precision and Recall. It is worth noting that (Singh et al., 2023) applies pre-trained DeiT model as the discriminator (Singh al., 2023). Whereas our INRFlow makes no such assumption about the function or pre-trained models, enabling to triv- ially apply INRFlow to other domains like 3D point clouds (see Sect. 4.2) or protein folding.
Fig. 4(b) show examples of sampled point clouds and cor- responding conditional images. As discussed in $4.4, IN- RFlow trained on Objaverse also enjoys the flexibility of resolution agnostic generation. In the App. Fig. 12 we show additional results sampled with more points than what the model was trained on. As shown, INRFlow learns to gener- ate 3D objects with rich details that match the conditional images ultimately being able to generate a continuous sur- face.
4.3. Protein Folding
We now showcase the domain-agnostic prowess of INR- Flow applying it to the protein folding problem (Jumper et al., 2021). From an ML perspective, this problem is a conditional 3D generation problem where we are given the amino-acid sequence (e-g. sequence of discrete symbols from vocabulary of 20 possible amino-acids) and we need to generate a 3D coordinate for each atom in the protein, where different amino-acids can have different numbers of atoms. In our experiments we use SwissProt set (Boeck- mann et al., 2003) taking the ground truth structures from the AlphaFold Database (Varadi et al., 2022).
4.2. Image-to-3D
We also showcase that INRFlow can directly integrate con- ditional information like images. We train an image-to- point-cloud INRFlow model on Objaverse (Deitke et al., 2023), which contains 800k 3D objects of wide variety, to illustrate the capability of INRFlow on large-scale 3D gen- erative tasks. In particular, conditional information (i.e., image) is integrated to our model through cross-attention. We train INRFlow with batch size 384 for 500k iterations. During sampling, we use an Euler-Maruyama sampler (Ma et al., 2024) with 500 steps to generate point clouds (see App. A.2 for more details on architecture and objaverse data
We adapt the coordinate and signal in INRFlow that the coordinate becomes feature representation of each atom in the protein and the signal represents the 3D point for that particular atom. This is similar to recent work on protein folding and conformer generation, which directly predicts
6

---

INRFlow: Flow Matching for INRs in Ambient Space
Clas-Conditional ImageNet 256x256
Model
Agnostic
Tr. Samples
Params
it.
FID4
1st
Precision"
Recallf
Adversarial models
BigGAN-ddepp (Brock al.. 2019)
1.28M 1.28M
0.87
PolyINR (Singh 2023)
2.86
241.4
0.71
0.39
Latent space with pretrained
DiT-XL (c) (Peebles Xie, 2023)
9.23M
675M
2.27
278.2
0.83
0.57
SIT-XL (cfg=1.5,, SDE) (Ma et al., 2024)
X
9.23M
675M
1.B
2.06
270.2
0.82
0.59
Ambient space
ADM (Dhuriwal & Nichol, 2021)
X
1.28M
554M
507M
10.94
100.9
0.69
0.63
CDM (Ho et al., 2021)
X
1.28M
4.88
158.7
Simple Diff. (U-Net) (Hoogehoom al., 2023)
1.28M
3.76
171.6
RIN (labri et al., 2023)
1,28M
3.42
182.0
HDiT (efg=1.3) (Crowson 2024)
1.28M
557M
742M
3.21
220.6
Simple Dift. (U-VIT) (Hoogeboom al., 2023)
X
1,28M
2B
1.4B
2.77
211.8
VDM++ (U-VIT) (Kingma& & Gao, 2023)
X
1.28M
2B
1.4B
2.12
267.7
INRFlow-XI (ours) (.))=1.55
1.28M
733M
780M
3.74
228.8
0.82
0.52
Table 3. Top performing models for class-conditional image generation on ImageNet 256x256.
LDDT: 0.88
LDDT: 0.89
LDDT: 0.93 TM-Score: 0.87
LDDT: 0.77 TM-Score: 0,83
TM-Score: 0.85
TM-Score: 0.83
Figure Examples protein structures predicted by INRFlow on SwissProt, together their LDDT scores. The GT structures are depicted in green while the generated structures are show in orange. INRFlow accurately captures the global spatial distribution of protein backbones generating reasonable structures different protein sequences.
Model
ULIP-T
P-FID 4
spaces for future work developing protein folding network architectures that can leverage general purpose transformer blocks.
Shap-E (Jun & Nichol, 2023)
0.1307
Michelangelo (Zhao al., 2024)
0.1899
CLAY (Zhang al., 2024)
0.2066
0.9946
INRFlow (ours)
0.2976
0.3638
4.4. Resolution Agnostic Generation
An interesting property of INRFlow is that decodes each coordinate-value pair independently, allowing resolution changes of resolution for generation. At inference time the
Table image-conditioned 3D point cloud generation perfor-
mance on Objaverse.
define many coordinate-value pairs de- sired where the initial value of each pair t 1 is drawn from Gaussian distribution. We now quantitatively eval- uate the performance of INRFlow in this setting. In Tab. we compare the FID different recipes. First, INRFlow is trained on FFHQ-256 and bilinear or bicubic interpola- tion is applied the generated samples get images directly generate
atom coordinates as opposed to frames of reference (Wang et al.; Abramson et al., 2024). For the spatially- aware latents we aggregate information from all the atoms corresponding the same amino-acid into single latent vector (see App. A.3 for more details architecture and training recipe).
We show qualitative results on Fig. 3 and additional results on Fig. 13. INRFlow accurately captures the global dis- tribution of protein backbones generating reasonable 3D structures for different protein sequences. These initial re- sults are very encouraging and open up exciting design
512. On the other hand, INRFlow images at resolution 512 by simply increasing the number of coordinate-valus pairs during inference without further tuning. As shown in Tab. 5, INRFlow achieves lower FID

---

INRFlow: Flow Matching for INRs in Ambient Space
256
5122
1024
2048
16k
32k
128k
(e)
(b)
Figure 4. Examples of resolution agnostic generation for INRFlow models trained ImageNet-256 (a), and Ohjaverse- 6k (b). To generate samples at higher resolutions than the one in training we fix the initial noise seed and increase the number of coordinate-value pairs evaluated by the model. Even though INRFlow was only train with samples at fixed resolution (256 for ImageNet and 16k for Objaverse). it can still generate realistic samples at higher resolutions. These results show that INRFlow is learning a continuous probability density field.
when compared with other manually designed interpolation methods, showcasing the benefit of developing generative models on ambient space.
use large amounts of expensive high resolution data, which can be hard to collect some domains.
5. Conclusion
INRFlow Bilincar Bicubic
We introduced INRFlow, a flow matching generative model for continuous function spaces designed to operate directly in ambient space. Our approach dispenses with the practical complexities of training latent space generative models (Du et al., 2021a; la; Dupont et al., 2022a), such as the dependence on domain- -spaiiii compressors for different data domains tuning of hyper-parametert of the data compressor (i.e. adversaria weight, KL term, etc.). Inspired by deterministic encoding of INRs, we introduced a conditionally indepen- dent point-wise training objective that decomposes the target vector field and allows to continuously evaluate the gener- ated samples, similar to INRs, enabling resolution changes at inference time.
FID(J)
23.09
35.05
24.34
Table FID different super resolution methods to generate images resolution 512 x 512 for INRFlow trained on FFHQ- 256.
We show examples of resolution agnostic generation for INRFlow models trained on ImageNet-256 Fig. 4(a) and Objaverse- 16k in Fig. 4(b). Even though INRFlow was only train with samples at a fixed resolution (256 for ImageNet and 16k for Objaverse), it can still generate realistic samples higher resolutions. For example, Fig. 4(a) shows samples generated from INRFlow trained on ImageNet-256 and sam- pled at resolutions up to 2k (see additional examples in Fig. 14). Fig. 4(b) shows point cloud with up to 128k points from INRFlow trained on Objaverse with only 16k points points per sample. It is worth noting that in INRFlow there's 's no cascading (Ho et al., 2022b) or multiple resolution training (Gu al., 2023), we simply fix the initial noise seed and increase number of coordinate-value pairs that are evaluated. These results show that INRFlow is not trivially overfit- ting to the vendinatt-value pairs in the training but rather learning a continuous probability density field in space from which an infinite number of points could be sampled. Gen- erally speaking, this also provides the potential to efficiently train flow matching generative models without the need to
Our results both image, 3D point cloud and protein fold- ing benchmarks show the strong performance of INRFlow, as well as, its trivial adaption across modalities which we believe is the cornerstone of a good generative modeling architecture. conclusion, INRFlow represents a promis- ing direction for flow matching generative models, offering powerful and domain-agnostic framework. Future work could explore further improvements in training efficiency applying tricks orthogonal to contribution (Schwag et al., 2024) and investigate co co-training of multiple data domains to enable multi-modality generation in an end-to-end learn- ing paradigm.
8

---

INRFlow: Flow Matching for INRs in Ambient Space
Impact Statement
Proceedings of the European Conference on Computer Vision (ECCV), 2020.
This paper concerns the generative modeling methodology. While we do not see immediate societal implications from our technical contribution, there are potential consequences when it is used in training foundational generative models.
A. Chang, T. Funkhouser, I., Guibas, et al. Shapenet: An information-rich 3d model repository. arXiv, 2015.
Katherine Crowson, Stefan Andreas Baumann, Alex Birch, Tanishq Mathew Abraham, Daniel Z Kaplan, and Enrico Shippole. Scalable high-resolution pixel-space image synthesis with hourglass diffusion transformers. arXiv preprint arXiv:2401. 1605, 2024.
References
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans,
Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Xiaoliang Dai, Ji Hou, Chih- Cho Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhanc- ing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309. 15807, 2023.
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Korzhenkov. Image Pixel Syn-
Generators with thesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14278- 14287, 2021.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Obja- verse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp- 13142-13153, 2023.
A. Bansal, E. Borgnia, H. Chu, J. Li, H. Kazemi, F. Huang, M. Goldblum, J. Geiping, and T. Goldstein. Cold diffu- sion: Inverting arbitrary image transforms without noise. arXiv, 2022.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, 2021.
Matthias Bauer, Emilien Dupont, Andy Brock, Dan Rosen- baum, Jonathan Schwarz, and Hyunjik Kim. Spatial functa: Scaling functa to imagenet classification and gen- eration. arXiv preprint 3.Xi::2302.033000 2023.
Y. Du, K. Collins, J. Tenenbaum, and V. Sitzmann. Learning dgnnal-agnsstc manifolds of neural fields. In NeurIPS, 2021a.
Brigitte Boeckmann, Amos Bairoch, Rolf Apweiler, Maric- Claude Blatter, Anne Estreicher, Elisabeth Gasteiger, Maria J Martin, Karine Michoud, Claire O' Donovan, Is- abelle Phan, et The swiss-prot protein knowledgehase and its supplement trembl in 2003. Nucleic acids research, 31(1):365-370 2003.
Yilun Du, Katic Collins, Josh Tenenbaum, and Vincent Sitz- mann. Learning Signal-Agnostic Manifolds of Neural Fields. Advances in Neural Information Processing Sys- lems, 34:8320-833 8331, 2021b.
E. Dupont, H. Kim, S. Eslami, D. Rezende, and D. Rosen- baum. From data to functa: Your data point is a function and you should treat it like one. In ICML, 2022a.
Sam Bond- Bond-Taylor and Chris G Willcocks. Infinite resolution diffusson with subsampled mollified states. arXiv preprint 1rXiv:2303.18244 2023.
E. Dupont, Y. Teh, and A. Doucet. Generative models as distributions of functions. In AISTATS, 2022b.
Sam Bond-Taylor, Peter Hessey, Hiroshi Sasaki, Toby P Breckon, and Chris G Willcocks. Unleashing Transform- ers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High- Rigg-tReoolution Image Generation from Vector-Quantized Codes. In European Conference on Computer Vision, pp. 170-188. Springer, 2022.
Emilien Dupont, Yee Whye Teh, and Amaud Doucet. Gen- erative Models as Distributions of Functions. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 2989-3015. PMLR, 2022c
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, al. Scaling recti- fied flow transformers for resouutioo image synthesis. arXiv preprint arXiv:2403. 0..4403.33006 03206, 2024.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2019.
Ruojin Cai, Guandao Yang, Hadar Averbuch-Elie Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hari- haran. Learning gradient fields for shape generation. In
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and
Shuicheng Yan. Masked diffusion transformer a strong

---

INRFlow: Flow Matching for INRs in Ambient Space
image synthesizer. In Proceedings of the IEEE/CVE In- ternational Conference on Computer Vision, pp. 23164- 23173, 2023.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and Improv- ing the Image Quality of StyleGAN. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 81 1110-1119 2020.
vision
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang. Joshua M Susskind, and Navdeep Jaitly. Matryoshka diffusion mod- els. In The Twelfth International Conference on Learning Representations, 2023.
Diederik Kingma and Ruiqi Gao. Understanding the diffusion objective as weighted integral of elbos. arXiv preprint arXiv:2303. 00848, 2023.
Ho, A. Jain, and P. Abbeel. Denoising diffusion proba- bilistic models. In NeurIPS, 2020.
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen, The Role of ImageNet Classes in Fréchet Inception Distance. In International Confer- ence Learning Representations, 2023.
J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. Fleet. Video diffusion models. In ICLR, 2022a.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded dif- fusion models for high fidelity image generation. arXiv preprint 10v: 15282, 2021.
Ruihui Li, Xianzhi Li, Ke-Hei Hui, and Chi- Wing Fu. SP- GAN:sphere- guided 3d shape generation and manipula- tion. ACM Transactions on Graphics (Proc. SIGGRAPH), 40(4), 2021.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded Dif- fusion Models for High Fidelity Image Generation. Mach. Learn. Res., (3(4))11-33, 2022b.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maxim- ilian Nickel, and Matt Le. Flow matching for generative modeling. In ICLR, 2023.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023.
E. Hoogeboom, V. Satorras, C. Vignac, and M. Welling. Equivariant diffusion for molecule generation in 3d. In ICML, 2022.
S. Luo and W. Hu. Diffusion probabilistic models for 3d point cloud generation. In CVPR, 2021.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. Sim- ple Diffusion: End-to-End Diffusion for High Resolution Images. arXiv preprint arXiv:2301.11099 2023.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Fijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint 200.002001.087400 2024.
Allan Jabri, David Fleet, and Ting Chen. Scalable Adaptive Computation for Iterative Generation. arXiv preprint arXiv:2212.11972 2022.
A. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In ICMI, 2021.
Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. In ICML, 2023.
Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for gen- erating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08755 2022.
A. Jaegle, S. Borgeaud, J. Alayrac, et al. Perceiver io: A general architecture for structured inputs & outputs. In ICLR, 2022.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasu- vunakool. Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583 589, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov. Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199 2023.
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02443. 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
J. Plucker. Analyticci-Geomerrichh Geometrische Entwicklungen, Er- ster Band. Creative Media Partners, LLC, 2018. ISBN 9780274296538. URL https://books. http//bbookk..ggge..... google. com/books? htmp/bookks/iddgggEEuAAAAAA
T. Karras, S. Laine, and T. Aila. A style- generator ar- chitecture for generative adversarial networks. CVPR, 2019.
10

---

INRFlow: Flow Matching for INRs in Ambient Space
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizad- wongsa, and Supasorn Suwajanakorn. Diffusion Autoen- coders: Toward Meaningful and Decodable Represen- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10619- 10629, 2022.
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic acids research, 1D3::D439-D44, 2022.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017a.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, Advances in neural information processing systems, 30, 2017.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. PointNet++: Deep Hierarchical Feature Learning on Point Sets in Metric Space. Advances in neural information processing systems, 30, 2017b.
Yuyang Wang. Ahmed AA Elhag, Navdeep Jaitly, Joshua M Susskind, and Miguel Ángel Bautista. Swallowing the bitter pill: Simplified scalable conformer generation. In Forty-firsi International Conference on Muchine Leurn- ing.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. Interna- tional Conference on Machine Learning, ICML, 2021.
Jeremy Wohlwend, Gabriele Corso, Saro Passaro, Mateo Reveiz, Ken Leidal, Wojtek Swiderski, Tally Portnoi, Itamar Chinn, Jacob Silterra, Tommi Jaakkola, et al. Boltz-1: Democratizing biomolecular interaction model- ing. hioRxiv, pp. 2024-11, 2024.
S. Rissanen, M. Heinonen, and A. Solin. Generative mod- elling with inverse heat dissipation. arXiv, 2022.
Le Xue, Ning Yu, Shu Zhang, Artemis Panagopoulou, Jun- nan Li, Roberto Martin-Martin, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, et al. Ulip-2: To- wards scalable multimodal pre-training for 3d understand- ing. In CVPR. pp. 27091-27101 2024.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Om- mer. High-resolution image synthesis with latent diffu- sion models. In CVPR, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpa- thy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.
Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Yu Liu, Serge Belongie, and Bharath Hariharan. PointFlow: 3D point cloud generation with continuous normalizing flows. In ICCV, 2019.
Vikash Sehwag, Xianghao Kong, Jingtao Li, Michael Spranger, and Lingjuan Lyu. Stretching each dollar: Dif- fusion training from scratch on a micro-budget. arXiv preprint ar arXiv:2407.15811, 2024.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianx- iong Xiao. LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop. arXiv preprint 1506::1506.03666 2015.
Rajhans Singh, Ankita Shukla, and Pavan Turaga Poly- nomial implicit neural representations for large diverse datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2041- 2051, 2023.
Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo. StyleSwin: Transformer-Based GAN for High- Resolution Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11304-11314, 2022.
J. Song, C. Meng, and S. Ermon. Denoising diffusion im- plicit models. In ICLR, 2021.
Dongsu Zhang, Changwoon Choi, Jeonghwan Kim, and Young Min Kim. Learning generate 3d shapes with generative cellular automata. In International Conference on Learning Representations, 2021.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. CLAY: A controllable large- generative model for creating high-quality 3d assets. ACM TOG, 43(4): 1-20, 2024.
Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, Karsten Kreis, et al. Lion: Latent point diffu- sion models for 3d shape generation. Advances in Neural Information Processing Systems, 15:10021-10039, 2022.

---

INRFlow: Flow Matching for INRs in Ambient Space
Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng, Rui Wang, Pei Cheng, Bin Fu, Tao Chen, Gang Yu, and Shenghua Gao. Michelangelo: Conditional 3d shape generation based on ceape--mage-tex-e aligned latent representation. NeurlPS, 36, 2024.
of function evaluation). Note that for models that achieve better performance than INRFlow, many of them are trained for more iterations. In addition, at inference time INRFlow applies simple first order Euler sampler with 100 sampling steps, which uses less NFE than many other baselines.
Hongkai Zheng, Weili Nie. Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers, arXiv preprint arXiv:2306.0940 09305, 2023.
Model
Layers
Hidden size
#Lateents
Heads
Decuder layers #Params
INRF
12
384
1024
6
1
35M
INRFlow-B
12
768
1024
12
138M
INRFlow-L
24
1024
1024
16
1
458M
INRHow-XL
28
1152
1024
16
2
733M
Peiye Zhuang, Samira Abnar, Jiatao Gu, Alex Schwing, Josh Susskind, and Miguel Angel Bautista. Diffusion probabilistic fields. In ICLR, 2023.
Table 6. Detailed configurations of INRFlow for image generation.
A.2. Image-to-3D
A. Model Configuration and Training Settings
For image-to-3DD point cloud generation we trained INR- Flow on Objaverse (Deitke et al., 2023), which contains 800k 3D objects of great variety. In particular, condi- tional information (i.e., an image) is integrated to our model through cross- prossaattention For each object in Objaverse, we sample point cloud with 16k points. To get images for con- ditioning, each object is rendered with 40 degrees field of view, 448 x 448 resolution, at 3.5 units on the opposite sides of and axes looking at the origin. We extract features via DINOv2 (Oquab et al., 2023) which is con- catenated with Plucker ray embedding (Plucker, 2018) of each patch in DINOv2 feature. In each block, the learnable latent vector 2,, cross attends to image feature. During train- ing, the image conditioning is dropped randomly with 10% probability. Therefore, model can also benefit from pop- ular classifier-free guidance (CFG) to increase the guidance strength. The model is trained with batch size 384 for 500k iterations. During sampling, we use an Fuler-Maruyama sampler with 500 steps to generate point clouds. We train an INRFlow-XX. size model of 866M parameters similar to the one reported in Tab. 11.
A.1. Image
We provide detailed model configurations and training set- tings of INRFlow for image generation in Tab. 6. We de- velop model sizes small (S), base (B), large (L), and extra large (XL) to approximately match the number of parame- ters in DiT' (Peebles & Xie, 2023). For image experiments we implement the "psuedo" coordinate of latents as 2D grids and coordinate-value pairs are assigned different latents based on their distances to the latent coordinates. To embed coordinates, we apply standard Fourier positional embed- ding (Vaswani et al., 2017) for ambient space coordinate input in both encoder and decoder. The Fourier positional embedding is also applied to the "psuedo" coordinate of latents. On image generation, we found that applying ro- tary positional embedding (RoPE) (Su al., 2024) slightly improves the performance of INRFlow. Therefore, RoPE is employed for largest INRFlow-XL. model. For all the models we share the following training parameters except the t in inngstttpssss across different experiments. On image generation, all models are trained with batch size 256, except for INRFlow- -XL reported in Tab. 2 and Tab. 3, which are trained for 1.7M steps with batch size 512.
One particularity for image-to-3D point cloud generation is that we assign input elements to latents through a hash code, so that neighboring input elements are likely (but not certainly) to be assigned to the same latent token. We found that the improvements of spatial aware latents in 3D to not be as substantial as in the 2D image setting, so we report results with a vanilla Perceiver1O architecture for simplicity. To embed coordinates, we apply standard Fourier positional embedding (Vaswani et al., 2017) for ambient space coordinate input in both encoder and decoder.
fault training config:
optimizer=' Addmm adla__betal00-1 adam_beta2=0. 999
adam_eps=1e-8
learning_rate==le4 weight_decay=0.0
gradient_cciipnomm=2
A.3. Protein Folding
ema_ ema_decay=
In our experiments we use SwissProt set (Boeckmann et al., 2003) taking the ground truth structures from the AlphaFold Database (Varadi et al., 2022). We select a random set of 10k protein structures to train INRFlow. In this setting, the coordinate-valus pairs represent atoms in the protein, where
In Tab. 7, we also compare the size of models trained on ImageNet-256, training cost (i.e. product of batch size and training iterations), and inference cost (i.e. NFE, number
12

---

INRFlow: Flow Matching for INRs in Ambient Space
Model
Train data
parums
bs.
NFE
FID
1
ADM (Dhariwal & Nichol, 2021)
1.28M
554M
507M
1000
10.94
100.9
RIN (Jabri 2023)
1.28M
410M
614M
1000
3.42
182.0
HDiT (Crowson et al., 2024)
1.28M
557M
742M
100
3.21
220.6
Simple Diff. (U-VIT 2B) (Hoogehoom et al,, 2023)
1.28M
2B
1B
2.77
211.8
DIT-XL (Peebles & Xic, 2023)
9.23M
675M
1.8B
250
2.27
278.2
VDM++ (U-VIT 2B) (Kingma & Gao, 2023)
1.28M
2B
1.4B
512
2.12
267.7
SiT-XL (Ma et al., 2024)
9.23M
675M
1.8B
500
2.06
270.2
INRFlow-XI. (ours)
1.28M
733M
870M
100
3.74
228.8
Table Comparison of INRFlow and baselines in params and training cost (i.e. product of batch size and training iterations). Some numbers are borrowed from (Crowson et al., 2024).
the "coordinate" part is a set of features of that particular atom. In particular, we use the atom features shown in Tab. 8 together with the cigenvectors of the graph laplacian for each residue, as previously done in (Wang et al.). In particular, we found beneficial also concatenate to the atomic features the amino-acid embedding of its corresponding amino-acid, which we obtain from pre-trained ESM model (ESM- 650M), a language model for protein sequences masking only on sequence data and not 3D structures.
INRFlow-S
120
INRFIOW-B
INRFlow-L INRFION-XL
100
FID-50K
80
60
40
We aggregate information into spatially-aware latents by gooss-atteddigg all atoms belonging to a particular amino- acid with its particular latent. In practice, since the number of atoms for each aminoacid type is always fixed, might also simply use linear layer. Finally, for this task we train a XL size model for 100k iterations with batch size 256.
20
100k
200k
300k
400k
500k
650k
Training Steps
Figure 5. FID-50K over training iterations with different model sizes, where we see clear benefits scaling up model sizes.
B. Performance vs Model Size
40
INRFlow-L-linca INRFlow XL linear
To demonstrate the scalability of INRFlow we train models of different sizes including small (S), base (B), large (L), and extra-large (XL) on 255. We show the perfor- mance of different model sizes using FID-50K in Fig. 5(a). We observe a clear improving trend when increasing the number of parameters as well as increasing training steps.
35
INRFloW-XL
30
DIT
25
HID-Sak
20
This demonstrates that scaling the total training Gflops is important to improved generative results as in other VIT- based generative models (Peebles & Xie, 2023; Ma al.,
1.5
14 5s
10
3.92
3.74
2024).
VAE floos
D.T. flops
2.77
C. Performance vs Training Compute
0.5
1.0
1.5
2.0
Iraining compute (tfflops)
1e12
We compare the performance vs total training compute of INRFlow and DiT (Peebles & Xie, 2023) in Gflops INRFlow-linear denotes the variant of INRFlow where the cross-attention in the spatial aware encoder is replaced with grouping followed by linear layer. We found this could be an efficient variant of standard INRFlow while still achiev- ing competitive performance. Fig. 6 shows the comparison of the training compute in Gflops vs FID-50K between INR- Flow and latent diffusion model DiT (Peebles Xie, 2023) including the tranining compute of the first stage VAE, We
Figure 6. Comparing the performance vs total training compute comparison of INRFlow and DiT (Peebles & Xie, 2023).
estimate the training cost of VAE based the model card listed in HuggingFace", As shown, the training cost of VAE
'nttpa://huggt colacc..cc//tte /taaaii// sd sd-vae- ft-mse
13

---

INRFlow: Flow Matching for INRs in Ambient Space
Name
Description
Range
atomic
Atom type
one-hot of 35 elements dataset
degree
Number of bonded neighbors
(2:0 S z S 6, x, 2)
charge
Formal charge of atom Implicit valence of atom Hybrization type
{ < 1, 2
valence
<2:0<256,765
hybrization
{sp, sp'. spp,sp'',
other)
aromatic
Whether on a aromatic ring number of rings atom is in Amino-acid embedding
[Tree, False)
jum_rings
{# : 0 < x S E
ESM embedding
# R22200
Table 8. Atomic features included in INRFlow for protein folding.
is not negligible and reasonable models with FII) R 6.5 can be trained for the same cost.
90
train -t0K
train it -00K
train it. 300K
Admittedly, under equivalent training Gflops, INRFlow achieves comparable but not as good performance as DiT in terms of FID score (with a difference smaller than 1.65 FID points). We attribute this gap to the fact that DiT's VAE was trained dataset much larger than ImageNet, using a Noomain-specifie architecture (e.g. a convolutional U-Net). We believe that the simplicity of implementing and training INRFlow models in practice, and the trivial extension to different data domains (as shown in Sect. F) are strong arguments to counter an FID difference of smaller than 1.65 points. In addition, applying masking tricks orthogonal to our approach like the ones in (Sehwag et al., 2024) can help mitigate the training compute difference.
train it. 400K
train 500K
60
10K
50
30
20
210
2
2
2
244
QQueries (log scale)
Figure 7. FID-50K over training iterations with different number of decoded coordinate value pairs during training and the corre- sponding compute cost for a single forward pass.
In addition, due to the flexibility of cross-attention decoder in INRFlow, one can easily conduct random sub-sampling to reduce the number of decoded coordinate-valle pairs during training which ca also saves computation. Fig. shows how number of decoded coordinate- value pairs affects the model performance as well as Gflops in training. An image of res- olution 256 x 256 contains 65536 pixels in total which is the maximal number of coordinate-value pairs during training. As see in Fig. 5(b), a model decoding 4096 coordinate-valus pairs saves more than 20% Gflops over one decoding 16384.
and report results in the table below. INRFlow achieves the best performance when using the default grid pseudo coor- dinates. When using randomly sampled pseudo coordinates we observe drop in performance.
We attribute this to the fact that when ceeudd-corddinaes are randomly sampled, each spatial-aware latent effectively does a different amount of work (since pixels only cross- attend to the nearest pseudo-coordinate)) This unbalanced load across latents makes encoding less efficient. There are few different ways to deal with this without neces- sarily relying on a grid, one is to cluster similar pseudo- coordinates to provide an equidistant distribution in 2D space (i.e. KMeans++ initialization), another one is to in- crease the number of spatial- awware latents so that each latent has to do less work. We empirically see that both of this options are effective. Ultimately, having pseudo-coordinate lie grid strikes good balance of efficiency and effec- tiveness.
D. Spatial-aware Latent
Psuedo-cnond # latents FID-CLIP(() FID(.)
grid
1024
7.32
9.74
random
1024
11.66
17.99
KMeans++
1024
10.42
15.56
random
2048
8.95
11.86
Table 9. Performance of INRFlow on protein folding.
In image domain, we define pseudo coordinates to lie on a 2D grid, which results in pixels grouping as patches of same size. Tab. 9 lists the results of an ablation study on
E. Architecture Ablation
LSUN-church- 256 to compare different pseudo pseudo-coordinates. We trained all models for 200K steps with batch size 128
We also provide an architecture ablation in Tab. 10 show- casing different design decisions. We compare two variants
14

---

INRFlow: Flow Matching for INRs in Ambient Space
Model
(ID(L) Precision(1)
Recall(t)
Finally, we also report results for larger model INRFlow-L. (with x2 the parameter count LION) investigate how INRFlow improves as with increasing model size. We ob- serve that with increasing model size, INRFlow typically achieves better performance than the base version. This further demonstrates scalability of our model on ambient space of different data domains. More point cloud samples can be found in Appendix J.
PerceiverIO
65.09
0.38
0.01
INRFlow (ours)
7.03
0.69
0.34
Table 10. Benchmarking vanilla PerceiverIO and INRFlow with spatially aware latents on LSUN-Church-256 (Yu al., 2015).
of Transformer-based architectures INRFlow: vanilla Per- ceiverlO that directly operates on ambient space, but without using spatial aware latents and INRFlow. As it can be seen, the spatially aware latents introduced in INRFlow greatly improve performance across all metrics in the image do- main. justifying our design decisions. We note that we did not observe the same large benefits for 3D point clouds, which we hypothesize can be due to their irregular structure.
G. Quantitative Results on Protein Folding
This section includes the quantitative evaluation of the pro- tein folding task. In particular, we randomly selected 512 proteins from the AFDs-SwissProt dataset (Varadi et al., 2022) and use them as a test set. We compare INRFlow with an open-source replication of AlphaFold3 (Abramson et al., 2024) (ie. Boltz-1 (Wohlwend et al., 2024)), which is the SOTA approach for protein folding. Noted that AlphaFold3 is extremely domain-specifee using complex and curated architectural designs for protein folding. For example, it relies on multi-sequence alignment and template search on existing proteins. also designs a triangle self-attention
F. Unconditional 3D Point Cloud generation
Model
Layers
Hidden size
#Latents
Heads
Decoder layers 4Params
INRFlow-R B
512
1024
4
10SM
INRFlow-L.
12
512
1024
204M
INRFlow-XL
1152
1024
866M
update for atomic coordinates. Whereas INRFlow makes assumptions about data domain and effectively models different tasks under equivalent architecture and train- ing objective. We report C. -LDT and TM-score which are commonly used metrics to evaluate how predicted pro- tein structures align with ground truth (Tab, 13). Results indicate that INRFlow, which uses a domain-agnostic archi- tecture performs decently well on protein folding even when compared to SOTA models that require intricate domain- expertise embedded in the architecture. Note that we have not optimized INRFlow for folding experiment.
Table 11. Detailed configurations of INRFlow for point cloud gen-
eration.
For completeness we also tackle unconditional 3D point cloud generation on ShapeNet (Chang et al., 2015). Note that our model does not require training separate VAEs for point clouds, tuning their corresponding hyper-parameters or designing domain specific networks. We simply adapt our architecture for the change in dimensionality of coordinate- value pairs (e.g. f IR.2 + R33 for images to f R33 - R for 3D point clouds.). Note that for 3D point clouds, the coordinates and values equivalent. In this setting, we compare baselines including LION (Vahdat et al., 2022) which is a recent state-of-the-att approach that models 3D point clouds using a latent diffusion type of approach. Fol- lowing (Vahdat et al., 2022) we report MMD, COV and 1-NNA as metrics. To have a straightforward comparison with baselines, we train INRFlow-B with approximately match the number of parameters LION (Vahdat et al., 2022) (110M for LION vs 108M for INRFlow, see Tab. 11) on the same datasets (using per sample normalization as in Tab. 17 in Vahdat et al. (2022)). On ShapeNet, INRFlow models are trained for 800K iterations with a batch size of 16.
in the protein
H. Implementation of Resolution Agnostic Generation
One can interpret the spatial-aware latents computed from INRFlow's 's encoder as the "latent codes that are transformed into network parameters". For INRFlow these latents codes are used to compute v, k for the cross-attention block in the decoder, which takes in queries q (i.e. coordinate-vall pairs) at any resolution. The only thing that need to do in order to obtain consistent outputs across different query resolutions is to keep the resolution of the encoder fixed, while the decoder can be queried at a different res- olution. This is trivially achieved by employing a simple sub- sampling operation (i.e. grid sub-sampling) and keeping the sub-sampling operation fixed during inferenc (Fig. 8). This simple technique allows to change the resolution at inference time without any other additional tricks regarding noise alignment at different resolutions and produces crisp and consistent examples at higher resolutions that the one
We show results for category specific models and for an un- conditional model jointly trained on 55 ShapeNet categories in Tab. 12. INRFlow-B obtains strong generation results on ShapeNet despite being a domain agnostic approach and outperforms LION in most datasets and metrics. Note that INRFlow-B B has comparable number of parameters and the same inference settings than LION so this fair comparison.
15

---

INRFlow: Flow Matching for INRs in Ambient Space
MMD
COVT (%)
1-NNA, (%)
Category
Model
CD
EMD
CD
EMD
CD
EMD
ShapeGF (Cai al., 2020)
0.3130
0.6365
45.19
40.25
81.23
80.86
SP-GAN (Li al., 2021)
0.4035
0.7658
26.42
24.44
94.69
93.95
Airplane
GCA (Zhang et al., 2021)
0,3586
0.7651
38.02
36.30
88.15
85.93
LION (Vahdat et al., 2022) (110M)
0,3564
0.5935
42.96
47.90
76.30
67.04
INRFlow-B (ours) (108M)
0,2861
0.5156
43.38
47.54
75.55
64,95
INRFlow-L. (ours)
0.2880
0.5052
44.44
47.16
62.20
62.96
ShapeGF (Cai al., 2020)
3.7243
2.3944
48.34
44.26
58.01
61.25
SP-GAN (I.i al., 2021)
4.2084
2.6202
40.03
32.93
72.58
83.69
Chair
GCA (Zhang et al., 2021)
4,4035
2.5820
45.92
47.89
64.27
64.50
LION (Vahdat ct al., 2022) (1 (110M)
3.8458
2,3086
46.37
50.15
56.50
53.85
INRFlow-B (ours) (108M)
3,6310
2.1725
46.67
53.31
55.43
51.13
INRFlow-L (ours)
3.5145
2.1860
49.39
49.84
50.52
51.66
ShapeGF (Cai al., 2020)
1.0200
0.8239
44.03
47.16
61.79
57.24
SP-GAN (Li al., 2021)
1.1676
1.0211
34.94
31.82
87.36
85.94
Car
GCA (Zhang et al., 2021)
1.0744
0.8666
42.05
48.58
70.45
64.20
LION (Vahdat et al., 2022) (1 (110M)
1,0635
0.8075
42.90
50.85
59.52
49.29
INRFlow-B (ours) (108M)
0,9923
0.7692
43,46
47, /4
60.36
53.27
INRFlow-Le (ours)
0.9660
0.7846
44.03
48.86
53.83
54.55
LION (Vahdat et al., 2022) (110M)
3,4336
2.0953
48.00
52.20
58.25
57.75
All (55 cat)
INRFlow-B (ours) (108M)
3.2586
2.1328
49.00
50.40
54.65
55.70
INRFlow-L (ours)
3.1775
1.9794
49.80
52.39
51.80
53,90
Table 12. Generation performance metrics on Airplane, Chair, Car and all 55 categories jointly. All models were trained on the ShapeNet dataset from PointFlow (Yang et al., 2019). Both the training and testing data normalized individually into range [-1, 1].
Model
C. -DDD(())) TM-Score(t)
I. Additional ImageNet Samples
Boltzl (Wohlwend al., 2024)
0.923
0.812
We show uncurated samples different classes from INRFlow-XL trained on ImageNet-256 in Fig. and Fig. 10. Guidance scales CFG are set as 4.0 for loggerhead turtle, macaw, otter, coral reef and 2.0 otherwise.
INRFlow
0.722
0.664
Table Performance of INRFlow on protein folding.
ST2W012
J. A Additional ShapeNet Samples
Tra tin (eg
We show uncurated samples from INR trained jointly on ShapeNet categories in Fig. 11.
Encoder
Secmerpeed
K. Additional Objaverse Samples
We show additional Objaverse samples from INRFlow in Fig. 12
L. Additional SwissProt Samples
Figure 8. Illustration of resolution agnostic sampling. When gen- erating higher-resolution images in inference (e.g., 512x512), 256x28 256 coordinate-value pairs (consistent to setting training) are selected through grid subsampling and are fed to the encoder. The decoder takes predict velocities of all the pixel values. The model repeats the process in inference generate x12 image.
We show additional SwissProt samples from INRFlow in Fig. 13
full 512x512 coordinate-vee
pairs to
M. Additional Resolution Agnostic Image Samples
We show additional samples generated at different resolu- tions from INRFlow trained ImageNet-256 in Fig. 14.
used training (more results see Fig. and 14).
16

---

INRFlow: Flow Matching for INRs in Ambient Space
Figure 9. Uncurated samples of class labels: loggerhead turtle (33), macaw (88), golden retriever (207), otter (360) and red panda (387), panda (388) from INRFlow trained on 155.
17

---

INRFlow: Flow Matching for INRs in Ambient Space
Figure 10. Uncurated samples of class labels: palace (698), space shuttle (812), ice cream (928), pizza (963), coral reef (973), and valley (979) from INRFlow trained ImageNet-256.
18

---

INRFlow: Flow Matching for INRs in Ambient Space
Figure 11. Additional uncurated ShapeNet generations using 2048 points from the unconditional model jointly trained on 55 categories
19

---

INRFlow: Flow Matching for INRs in Ambient Space
Condition Image
16k
32k
64k
128k
Figure 12. Image-conditioned point clouds with 16k, 32k, 64k, and 128k points generated from INRFlow trained Objaverse (training with 16k points, CFG scale 5.0).
20

---

INRFlow: Flow Matching for INRs in Ambient Space
Figure 13. Additional examples of protein structures predicted by INRFlow on SwissProt (Boeckmann et al., 2003)
21

---

INRFlow: Flow Matching for INRs in Ambient Space
256
5122
10242
2048
Figure 14, Images generated at 256, 512, 1024, and 2048 resolutions from an INRFlow trained on ImageNet-256.
22


Reviews:
[{"review_id": "XnvaXEvlkR", "reviewer": "Reviewer_EhwM", "summary": "This paper proposes INRFlow, a novel domain-agnostic approach to learn flow matching in ambient space without the need of a pretrained domain-specific encoder. INRFlow has been evaluated on three tasks: image-to-image generation, image-to-3D point cloud generation, and protein folding. The effectiveness has been demonstrated.\n\nStrength:\n1. The strength compared with SOTA baselines is highlighted.\n2. The domain-agnostic flow matching is very attractive.\n\nWeakness and Questions:\n1. The key novelty -- spatial-aware latents -- is not well explained in either the main text or the appendix. How do you select the the pseudo coordinates? I can imagine the selection of these the pseudo coordinates can significantly impact the latent code quality and the generation quality. Can you do some ablation studies on this?\n2. While Figure 2 shows that the conditions can be both images and 3D point clouds, the experiments only focus on images as the condition. This causes some confusion.  Is it possible to use 3D point clouds as conditions?\n3. \"We found that the improvements of spatial aware latents in 3D to not be as substantial as in the 2D image setting.\" The pseudo coordinates are in 2D or 3D? Is this caused by the selection of the pseudo coordinates? \n4. Do you have some quantitative evaluation on the protein folding task?\n\nI am willing to improve my scores based on authors' answers to my question.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting the strength of INRFlow as a domain-agnostic flow matching model and thoughtful comments. Please find point-by-point response to your questions below.\n\n1. How do you select the pseudo coordinates?[...]Can you do some ablation studies on this?\n    - In image domain, we define pseudo coordinates to lie on a 2D grid, which results in pixels grouping as patches of same size. We add an ablation study on LSUN-church-256 to compare different pseudo-coordinates. We trained all models for 200K steps with batch size 128 and report results in the table below.  INRFlow achieves the best performance when using the default grid pseudo coordinates. When using randomly sampled pseudo coordinates we observe a drop in performance. \n    - We attribute this to the fact that when pseudo-coordinates are randomly sampled, each spatial-aware latent effectively does a different amount of work (since pixels only cross-attend to the nearest pseudo-coordinate). This unbalanced load across latents makes encoding less efficient. There’re a few different ways to deal with this without necessarily relying on a grid, one is to cluster similar pseudo-coordinates to provide an equidistant distribution in 2D space, another one is to increase the number of spatial-aware latents so that each latent has to do less work. We empirically see that both of this options are effective. Ultimately, having pseudo-coordinates lie on a grid strikes a good balance of efficiency and effectiveness. \n|pseudo coordinate|\\# latents|FID-clip|FID|\n|-|-|-|-|\n|grid|1024|7.32|9.74|\n|random|1024|11.66|17.99|\n|KMeans++|1024|10.42|15.56|\n|random|2048|8.95|11.86|\n2. While Fig. 2 shows that conditions can be both images and 3D point clouds[...].This causes some confusion. \n    - In Fig 2 we tried to show that both images and 3D point clouds can be modeled by INRFlow in the same way (via spatial-aware latents). We're aware how this leads to confusion and have edited the figure to make it clear (https://docs.google.com/document/d/e/2PACX-1vRVsqNt103C0EFDMdvCnWF8_2XLUps3ztW4Z5Fyu-ZHMMIKrn-Rg3TR7aep3wMvnQB-1Yivb3HeTO45/pub). \n3. \"Improvements of spatial-aware latents in 3D to not be as substantial as in the 2D image setting.\" [....] Is this caused by selection of the pseudo coordinates?\n    - For pointclouds, pseudo coordinates are defined in 3D space as opposed to in 2D for images. There’re two main reasons that we found for 3D pointclouds not benefiting from spatial-aware latents:\n        - Pointclouds are sparse representation of 3D geometry. If we define a fixed set of pseudo-coordinates as a 3D grid, a substantial amount of latents will encode empty space, which is not efficient for encoding (similar to voxel representations). Therefore, we opt for vanilla PerceiverIO for 3D point cloud generation obtaining great results without sacrificing efficiency.\n        - In 3D, we learn a deformation field from a prior 3D pointcloud distribution (eg. a 3D gaussian) towards a training set of shapes. Given that the geometry of the pointcloud changes at different timesteps and for different samples, it’s not trivial to define a fixed set of pseudo-coordinates to effectively cover the full geometry. We believe that timestep-aware and sample-aware pseudo-coordinates are very interesting to study in the future and are part of future work.\n4. Quantitative evaluation on protein folding task\n    - We have included a quantitative evaluation of the protein folding task. In particular, we randomly selected 512 proteins from the AFDB-SwissProt dataset and use them as a test set. We compare INRFlow with an open-source replication of AlphaFold3 [1] (ie. Boltz-1 [2]), which is the SOTA approach for protein folding. We note that AlphaFold3 is extremely domain-specific, using complex and curated architectural designs for protein folding. For example, it relies on multi-sequence alignment and template search on existing proteins. It also designs a triangle self-attention update for atomic coordinates. Whereas INRFlow makes no assumptions about data domain and effectively models different tasks under an equivalent architecture and training objective. \n    - We report $C_\\alpha$-LDDT and TM-score (higher the better) which are commonly used metrics to evaluate how predicted protein structures align with ground truth. Results indicate that INRFlow, which uses a domain-agnostic architecture performs decently well on protein folding even when compared to SOTA models that require intricate domain-expertise embedded in the architecture. Note that we have not optimized INRFlow for model size or other hyper-parameters in the protein folding experiment. \n|Model|$C_\\alpha$-LDDT|TM-score|\n|-|-|-|\n|AlphaFold3 (Boltz-1)|0.923|0.812|\n|INRFlow|0.722|0.664|\n\n[1] Abramson, J., et al. \"Accurate structure prediction of biomolecular interactions with AlphaFold 3.\" Nature (2024).\n\n[2] Wohlwend, J., et al. \"Boltz-1: Democratizing Biomolecular Interaction Modeling.\" bioRxiv (2024).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes. The experimental designs make sense to me.", "relation_to_broader_scientific_literature": "This work is highly related to the diffusion models, flow matching, and generative AI works.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b99aVMkvjx", "reviewer": "Reviewer_GGzL", "summary": "# Update\n\nThe authors have adequately addressed my concerns, and I expect that they incorporate:\n\n1. more explanation on how to consistency generate images at different resolutions, and \n2. explanation on the difference between their proposed methods and other function-space methods I mentioned\n\nto the final version of the paper.\n\nI decided not to change the score as I have already given a 4.\n\n# Old Summary\n\nThe paper presents a  model architecture and a training algorithm for generative models that can generate data in diverse domains as long as data in these domains can be represented as sets of discrete samples of a function $f: \\mathcal{X} \\rightarrow \\mathbb{R}^d$. These domains include images ($\\mathcal{X} = \\mathbb{R}^2$, $d = 3$), point clouds ($\\mathcal{X} = \\mathbb{R}^3$ , $d = 3$), and protein structures ($\\mathcal{X} =$ a set of descriptors of amino acids in a sequence, $d = 3$). \n\nMore concretely, a data item is a finite set $f = \\\\{ (x,y) : x \\in \\mathcal{X}, y \\in \\mathbb{R}^d \\\\}$, which is a \"function\" in the sense that, if $(x,y_1)$ and $(x,y_2)$ are members of $f$, then $y_1 = y_2$. Another way to view a data item is to say that $f = \\\\{ (x_1, y_1), (x_2, y_2), \\dotsc, (x_N, y_N) \\\\}$. We may now let $\\mathbf{x} = (x_1, x_2, \\dotsc, x_N)$ and $\\mathbb{y} = (y_1, y_2,. \\dotsc, y_N)$ be tensors. A data item is thus equivalent to an ordered pair $(\\mathbf{x}, \\mathbf{y})$ of tensors whose first dimensions have the same size $N$.\n\nThe idea is to create a flow matching model that represents a stochastic process that transforms the data distribution to a distribution of $(\\mathbf{x}, \\mathbf{y})$ where $\\mathbf{y}$ is standard Gaussian noise. Let $\\mathbf{y}\\_t = (1-t) \\mathbf{y} + t\\xi$ where $\\xi \\sim \\mathcal{N}(0,I)$. We train a neural network $v\\_\\theta$ so that $v\\_\\theta(\\mathbf{x}, \\mathbf{y}_t, \\theta)$ gives a velocity vector $\\mathbf{v}$ on the trajectory that connect $\\mathbf{y}_0$ (a well-formed data item) to $\\mathbf{y}_1$ (a Gaussian noise tensor). This can be done by minimizing the followng loss\n\n\\begin{align*}\n\\mathcal{L}\\_{CICFM} = E\\_{\\substack{t \\sim \\mathcal{U}[0,1],\\\\\\\\ (\\mathbf{x},\\mathbf{y}) \\sim p_{\\mathrm{data}}, \\\\\\\\ \\xi \\sim \\mathcal{N}(0,I) }} \\Big[ v_\\theta(\\mathbf{x}, \\mathbf{y}_t, t) - u_t(\\mathbf{x}, \\mathbf{y}_t | \\xi)  \\Big]\n\\end{align*}\n\nwhere (again) $\\mathbf{y}_t = (1-t)\\mathbf{y} + t\\xi$, and \n\n\\begin{align*}\nu_t(\\mathbf{x},\\mathbf{y}_t | \\xi) = \\frac{\\xi - \\mathbf{y}_t}{1 - t}.\n\\end{align*}\n\nTo sample a data point, one starts with a noisy data item $(\\mathbf{x}, \\xi)$ where $\\xi \\sim \\mathcal{N}(0,I)$. Then, one uses the trained network above to solve the differential equation $\\partial \\mathbf{y}\\_t / \\partial t = v_\\theta(\\mathbf{x}, \\mathbf{y}_t, t)$ with the initial condition $\\mathbf{y}_1 = \\xi$ and find $\\mathbf{y}_0$.\n\nThe paper proposes a model architecture for $v_\\theta$ so that the data size $N$ may be different among items in the dataset. Moreover, at test time, $N$ (and, to a certain extent, $\\mathbf{x}$) can take on unseen values. This is done by dividing the networks into two parts: the encoder and the decoder. The encoder takes $(\\mathbf{x}, \\mathbf{y}_t)$ as input and produces a latent code $\\mathbf{z}$. The decoder, on the other hand, operates on each \"row\" $(x\\_i,y\\_{t,i})$ of the input $(\\mathbf{x}, \\mathbf{y_t})$ independently, and uses $\\mathbf{z}$ to model correlations between rows.\n\nThe paper claims that their method is effective in creating generating models for (1) images, (2) 3D point clouds, (3) 3D point clouds conditioned on images, and (4) protein structures. All the generative models for all domains have the same general architectures. Due to the proposed architecture, a trained model can generate images with different resolutions and point clouds with different number of points from those that are found in the training dataset.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments which helped substantially improve the clarity of the submission. Please find point-by-point response to your questions below.\n\n1. Clarification of how upsampling in Figure 4 is conducted.\n    - We agree with the reviewer that description of these experiments/implementation caused confusion and we will clarify our explanation in the final version of the paper. The resolution-free inference process is performed in the following manner. Let’s assume we have an INRFlow model trained at resolution $N=256$ for which we want to run inference at resolution $M=512$.  We start by drawing $M^2$ pixel values from the gaussian prior at timestep t=1.  We then take $N^2$ pixels from these $M^2$ pixels (eg. via simple grid sub-sampling) and feed this set of sub-sampled $N^2$ pixels to the encoder of INRFlow to compute spatial aware latents. Once these latents are computed, we feed the $M^2$ pixel coordinates and values to the decoder of INRFlow to compute cross-attention with the spatial-aware latents, which will give us $M^2$ pixels values for the next timestep. At the next timestep, we again repeat the process where we take $N^2$ pixels from the newly computed $M^2$ pixel values via sub-sampling to again feed through the encoder.  In this setting, the spatially-aware latents act as the “neural network” parameters in GEM, GASP or PolyINR. However, instead of having these parameters explicitly parametrize a neural network or a generator, we tap into them via a cross-attention block in the decoder. To summarize: the resolution of the inputs that go into the encoder does not change at inference. We just let those inputs be a sub-sample of a higher resolution noisy image. Since the decoder of INRFlow can be evaluated continuously at any resolution, we can use it to generate an image at a higher resolution that the one seen by the encoder. We will update our explanation in the paper and include a figure in the appendix with a visual explanation of the process (link to figure: https://docs.google.com/document/d/e/2PACX-1vRVsqNt103C0EFDMdvCnWF8_2XLUps3ztW4Z5Fyu-ZHMMIKrn-Rg3TR7aep3wMvnQB-1Yivb3HeTO45/pub)\n\n2. Typo at Equation 5.\n    - We thank the reviewer for pointing out the typo, which we agree is incorrect. We will fix this in the final version of the paper\n\n3. Missing reference: the paper may cite papers on diffusion autoencoders such as [1] because INRFlow architecture is quite similar to that of a diffusion autoencoder. \n    - We thank the reviewer for pointing out this work. We will include and discuss it in the related work section.\n\n4. I think the paper should point out a significant difference between their approach and those taken by other \"function space models\" such as PolyINR, GEM, and GASP. For these approaches, it is possible to sample a random function f and then evaluate the function mulitiple times. This is done by first sampling a latent code z, and then use z to parameterize a neural network. For GEM and GASP, the latent codes are transformed into network parameters. For PolyINR, z is turned into parameters of a generator network. Once this neural network is obtained, we can evaluate it to get the output at any input coordinates and as many input coordinates as we want. \n    - The reviewer brings up a great point, which definitely needs clarification. We believe that the gap in our explanation was that one can interpret the spatial-aware latents computed from INRFlow’s encoder as the “latent codes that are transformed into network parameters” (as the reviewer explained). For INRFlow these latents codes are used to compute $k,v$ for the cross-attention block in the decoder, which takes in queries $q$ (ie. coordinate-value pairs) at any resolution. \n    - The only thing that we need to do in order to obtain consistent outputs across different query resolutions is to keep the resolution of the encoder fixed, while the decoder can be queried at a different resolution. This is trivial to achieve by employing a simple sub-sampling operation (ie. grid sub-sampling) and keeping the sub-sampling operation fixed during inference. We will update our explanation in the paper and include a figure in the appendix with a visual explanation of the process  (link to figure: https://docs.google.com/document/d/e/2PACX-1vRVsqNt103C0EFDMdvCnWF8_2XLUps3ztW4Z5Fyu-ZHMMIKrn-Rg3TR7aep3wMvnQB-1Yivb3HeTO45/pub). This simple technique allows us to change the resolution at inference time without any other additional tricks regarding noise alignment at different resolutions and produces crisp and consistent examples at higher resolutions that the one used in training (see Fig 4 and Tab 5).\n\n[1] Preechakul et al. \"Diffusion Autoencoders: Toward a Meaningful and Decodable Representation.\" 2022.", "claims_and_evidence": "The paper makes several important claims.\n\n1. A single architecture can be used to model data in different domains.\n2. The paper's proposed architecture and training method yield effective generative models for the domains tested.\n3. Trained models can generate data at resolutions different from those used in training time.\n\nI believe these claims are well substantiated. The variants of the same architecture are given in the Appendix, and the authors are able to reuse these architecture on different modalities. The scores that the train models achieve are competitive with baselines. Figure 4 show images with different resolutions and point clouds with different number of points.", "methods_and_evaluation_criteria": "I believe that the datasets and the evalution metrics used are reasonable.\n\nHowever, I think the paper is not clear on how it achieves \"resolution agnostic generation.\" This is especially true on how it generates samples in Figure 4 where the images and meshes at different resolutions are similar to one another. The paper say \"we simply fix the initial noise seed and incrase number of coordinate-value pairs that are evaluated.\" What is unclear is the process of \"fixing the noise seed\" as it can means several things. It can either mean (a) fixing the seed of the random number generator or (b) fixing the noise signal $\\mathbf{y}_1$ at a low resolution and then deriving from it $\\mathbf{y}_1$ at other resolutions.\n\nI don't think Option (a) makes sense. For example, let's say we fix a random seed of 42. When we want to generate a (1D) image with 2 pixels, randomization might give us the following $(x,y)$ pairs.\n\n(0.0, 0.7), (0.5, 0.4)    (Let's call this F.)\n\nHowever, if we keep the same random seed and try to generate a 1D image with 4 pixels, then we might end up with the following $(x,y)$ pairs:\n\n(0.00, 0.7), (0.25, 0.4), (0.50, 0.9), (0.75, 0.3)   (Let's call this G.)\n\nHere, we assume that fixing a random seed would give the same sequence of random numbers. The issue is that F and G would likely to yield very different final images. A sequence that would yield an image similar to F would be an upsampled version of F, which G is clearly not one.\n\nAs a result, Option (b) seems to make more sense. However, the paper never explains how it upsamples a random noise vector to a higher resolution. As a result, it is unclear to me how the images in Figure 4 were generated, and this makes the result there not reproducible.", "theoretical_claims": "I think the proposed training algorithm is correct. However, I think Equation (5)\n\n\\begin{align*}\nu_t(\\mathbf{x}_f, \\mathbf{y}_f | \\epsilon) = (1 - t)\\epsilon + t \\mathbf{y}_f\n\\end{align*}\n\nis wrong, and I surmise this is a typo. It should have been\n\n\\begin{align*}\nu\\_t(\\mathbf{x}\\_{f\\_t}, \\mathbf{y}\\_{f\\_t} | \\epsilon) = \\frac{\\epsilon - \\mathbf{y}\\_{f\\_t}}{1-t}.\n\\end{align*}\n\nFor reference, see Equation (20) in the Lipman et al's flow matching paper [1]. Moreover, if we use the fact that $\\mathbf{y}_{f_t} = (1-t)\\mathbf{y}_f + t\\epsilon$ as defined in Section 3.2 of othe paper, then we have that \n\n\\begin{align*}\nu\\_t(\\mathbf{x}\\_{f\\_t}, \\mathbf{y}\\_{f\\_t} | \\epsilon) = \\frac{\\epsilon - (1-t)\\mathbf{y}\\_f - t\\epsilon }{1-t} = \\frac{1-t}{1-t}(\\epsilon - \\mathbf{y}\\_f) = \\epsilon - \\mathbf{y}_f\n\\end{align*}\n\nThis matches with the target value $X_1 - X_0$ in Equation 1 of the rectified flow paper [2].\n\n*Citation*\n* [1] Lipman et al. \"Flow Matching for Generative Modeling.\" 2022.\n* [2] Liu et al. \"Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow.\" 2022", "experimental_designs_or_analyses": "I do not find issues with experimental designs and analyses.", "relation_to_broader_scientific_literature": "I think the paper should point out a significant difference between their approach and those taken by other \"function space models\" such as PolyINR, GEM, and GASP. For these approaches, it is possible to sample a random function $f$ and then evaluate the function mulitiple times. This is done by first sampling a latent code $z$, and then use $z$ to parameterize a neural network. For GEM and GASP, the latent codes are transformed into network parameters. For PolyINR, $z$ is turned into parameters of a generator network. Once this neural network is obtained, we can evaluate it to get the output at any input coordinates and as many input coordinates as we want. For example, if the function models an image, we can use it to generate images at $64 \\times 64$, $128 \\times 128$, $256 \\times 256$, or any other resolutions that we want by simply feeding the function with an appropriate grid of 2D coordinates.\n\nHowever, I don't think INRFlow has the above capability. It is not possible to sample a random image function and then generate the same image at resolution $64 \\times 64$, $128 \\times 128$, and $256 \\times 256$ afterwards. This is because, to generate images at different resolutions with INRFlow, one has to start from point samples of noisy images $\\\\{ (x,y) \\\\}$ at the specified resolutions. The crucial difference here is that, for INRFlow, the noise vector $y$ has to be supplied for each input coordinate $x$. However, the approaches discussed above do not to specify $y$. To make sure that the images at different resolutions are consistent with one another, it is necessary to make sure that the noise $y$ for each resolutions are properly correlated. I believe this is harder than what is done with PolyINR, GEM, and GASP (i.e., nothing). Moreover, as pointed out earlier, the way to ensure consistency between different resolutions are not adequately explained in the paper.", "essential_references_not_discussed": "I believe that the references are adequate. However, the paper may cite papers on diffusion autoencoders such as [1] because INRFlow architecture is quite similar to that of a diffusion autoencoder.\n\n*Citation*\n* [1] Preechakul et al. \"Diffusion Autoencoders: Toward a Meaningful and Decodable Representation.\" 2022.", "other_strengths_and_weaknesses": "I think this paper is an interesting take on a unified architecture to model multiple types of signals.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please specify how to noise vectors are initialized to generate images and meshes in Figure 4 to resolve the clarity issue I pointed out the the \"Methods and Evaluations\" section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JSOTQkrzGI", "reviewer": "Reviewer_RpjP", "summary": "This paper presents INRFlow, a novel domain-agnostic generative model that operates in ambient space, eliminating the need for hand-crafted data compressors in different domains. The key innovation is a conditionally independent point-wise training objective, allowing INRFlow to model continuous coordinate-value mappings across diverse data types, including images, 3D point clouds, and protein structures.\n\nThe authors claim the following contributions:\n- Proposing INRFlow, a flow-matching generative transformer that works on ambient space to enable single-stage generative modeling on different data domains.\n- Empirical results show that INRFlow achieves competitive performance on image and 3D point cloud generation compared with strong domain-specific baselines.\n- Allowing resolution changes at inference time.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging INRFlow’s competitive experimental performance and flexibility in inference. We also appreciate your thoughtful comments which help substantially improve the quality of our work. Please find point-by-point response to your questions below.\n\n1. How are spatial aware latents assigned?\n    - The input coordinate-value pairs first cross attend to spatial aware latents (which are free parameters) and then self-attention is applied to the latents.  INRFlow assigns each latent with a pseudo coordinate and each coordinate-value pairs is assigned to its closest latent. For example, in image generation, we define the pseudo coordinates for latents to lie in a 2D grid by default. Thus pixels within a patch cross attend to one latent, which is located at the center of that patch. We notice that the current description can be unclear and will further clarify the spatial-aware latents definition in the final version of the paper. \n\n2. An ablation study would strengthen the empirical evaluation by providing insights into key design choices. For example, investigating the impact of attended coordinates L, performance as a function of training sample size, and other architectural decisions would improve the paper’s clarity and robustness.\n    - We have performed an additional ablation study on selection of pseudo-coordinates for latents and the number of training samples on LSUN church 256. Results are shown in the table below (all models are trained for 200K iterations with batch size 128). We note the following and answer the reviewers questions:\n        - The process for assigning pseudo-coordinates impacts overall performance. By default, we let pseudo-coordinates lie on a 2D grid which cover the whole image. Changing that to randomly assigned psuedo-coordinates (drawn from a uniform distribution) decreases performance due to reduced image coverage, as show by the difference between rows 1 and 2 of the table. \n        - Increasing the number of latents $L$ has a positive impact on performance, even when defining pseudo-coordinates are random, as show in rows 2 and 3. \n        - Finally, we also show performance as a function of training sample size, where training INRFlow on a subset of the full dataset (e.g., 15k ($\\sim$12.5%), 30k ($\\sim$25%) and 60k ($\\sim$50%) samples), the model achieves comparable performance to using the full dataset only using ~50% of the training data with a linear drop in performance after that. This indicates the model is capable of learning the distribution of dataset with limited data samples and generating realistic samples. \n         | Model   | pseudo coordinate | \\# latents | \\# Data | FID_clip | FID   |\n         | ------- | ----------------- | ---------- | ------- | -------- | ----- |\n         | INRFlow | grid              | 1024       | 126k    | 7.32     | 9.74  |\n         | INRFlow | random            | 1024       | 126k    | 11.66    | 17.99 |\n         | INRFlow | random            | 2048       | 126k    | 8.95     | 11.86 |\n         | INRFlow | grid              | 1024       | 60k     | 7.62     | 10.6  |\n         | INRFlow | grid              | 1024       | 30k     | 7.68     | 12.14 |\n         | INRFlow | grid              | 1024       | 15k     | 12.17    | 25.52 |\n\n3. In the flow matching framework, the objective is to map $\\mathbb{R}^d \\rightarrow \\mathbb{R}^d$. How did you handle this in the image-to-point-cloud setup, where the mapping involves $\\mathbb{R}^3 \\rightarrow \\mathbb{R}^3$?\n    - In 3D point cloud generation, the mapping is defined as $\\mathbb{R}^3 \\rightarrow \\mathbb{R}^3$, where the input and output share the same space. Conceptually, one can think of the signal space as the deformation of the input (ie. a deformation from a gaussian distribution in 3D to a particular object shape). Namely given a set of points in 3D space, the model learns to predict the transformation of the point cloud in the same 3D space which leads to semantically meaningful objects, conditioned on an input image.  \n\n4. Does INRFlow exhibit permutation equivariance? Specifically, if the input coordinates are permuted, does the output follow the same permutation?\n    - Yes, INRFlow preserves permutation equivariance in decoding. The decoder in INRFlow is implemented as a cross-attention block and therefore inherits the permutation equivariance property. Namely, if one permutes the query coordinate-value pairs, the decoding results will be permuted accordingly. \n    - In the encoder, INRFlow is permutation invariant. In particular, if one changes the order of input coordinate-value pairs in encoder, the assignment to spatial-aware latents stay the same which results in unchanged spatial-aware latents $z_{f_t}$. This guarantees that the learned $z_{f_t}$ well models the mapping from coordinate space to signal space.", "claims_and_evidence": "See below.", "methods_and_evaluation_criteria": "See below.", "theoretical_claims": "See below.", "experimental_designs_or_analyses": "See below.", "relation_to_broader_scientific_literature": "See below.", "essential_references_not_discussed": "See below.", "other_strengths_and_weaknesses": "#### Pros:  \n- The proposed method, INRFlow, is novel, offering a simple yet effective approach that is domain-agnostic, meaning the same architecture can be applied to images, 3D point clouds, and protein structures, demonstrating its adaptability.  \n- The experimental evaluation is comprehensive, covering multiple domains. INRFlow performs well, achieving comparable or superior results to domain-specific architectures.  \n- Resolution-agnostic generation: INRFlow can produce outputs at arbitrary resolutions, providing more flexibility than traditional generative models.  \n- Single-stage training: The method eliminates the complexity of two-stage training pipelines, making it easier to implement and extend.  \n\n#### Cons:  \n- The writing quality could be improved, particularly in the method section:  \n    - \"At a high level, our encoder network takes a set of coordinate-value pairs and encodes them to learnable latents through cross-attention. These latents are then updated through several self-attention blocks to provide the final latents.\" – Figure 2 does not depict cross-attention in the encoder, which may confuse readers. The description should clarify that cross-attention occurs after the self-attention blocks.  \n    - \"Firstly, our encoder utilizes spatial aware latents where each latent is assigned a “pseudo” coordinate. Coordinate-value pairs are assigned to latents based on their distances on coordinate space.\" – This explanation is unclear. If the authors mean that higher resolution creates pseudo labels, further details are needed. How are these pseudo labels assigned? Interpolation? KNN? A more detailed explanation is necessary.  \n\n- An ablation study would strengthen the empirical evaluation by providing insights into key design choices. For example, investigating the impact of attended coordinates $L$, performance as a function of training sample size, and other architectural decisions would improve the paper’s clarity and robustness.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. In the flow matching framework, the objective is to map $\\mathbb{R}^d \\rightarrow \\mathbb{R}^d$. How did you handle this in the image-to-point-cloud setup, where the mapping involves $\\mathbb{R}^2 \\rightarrow \\mathbb{R}^3$?  \n2. Does INRFlow exhibit permutation equivariance? Specifically, if the input coordinates are permuted, does the output follow the same permutation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7oUVDuGil4", "reviewer": "Reviewer_D7hb", "summary": "Current flow matching (FM) methods are usually trained in two-stage paradigm, which sets obstacles for unifying models across data domains. To deal with this, this paper introduces INRFlow. In the proposed method, they estimate the map from coordinate to value via FM in a pointwise manner. To further model spatial dependency, they introduce latent variable via self-attention in the encoder and cross-attention in the decoder. They then applies the proposed INRFlow into image generation, image-to-3D point clound generation and protein generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. There may raise some concerns on novolties, as the proposed method is essentially combination of INR with flow matching, and using attention to handle latent variables for context information. Although the architecture is absolutely new and novel, the proposed method can solve problem well. Therefore, I would suggest borderline accept.\n    - We want to first thank the reviewer for acknowledging that our proposed INRFlow solves different problems with a new and novel architecture. However, we want to kindly point out that our work is more than a simple combination of INR and flow matching. Though they are previous works investigating combining INR with generative models [1, 2, 3]. Which required complex multi-stage training recipes [1,2] and were not able to scale to high resolution signals [2,3]. In fact, most of the experiments are conducted on low-resolution images like 32 $\\times$ 32 and 64 $\\times$ 64. Our novel architecture and training objective allows us generate images up to 2048 $\\times$ 2048 resolution, which is a drastic change with respect to existing work.\n    - We want to emphasize that it’s non-trivial to obtain these results at large-scale data regime that we have. Finally, our point-wise training objective allows for efficient training via sub-sampling dense domains which previous works did not explore and allow us to tackle high-resolution data. This also enables INRFlow to do inference at arbitrary resolution in inference time without additional training (Figure 4).  \n\n[1] Du, Yilun, et al. \"Learning signal-agnostic manifolds of neural fields.\" Advances in Neural Information Processing Systems 34 (2021): 8320-8331.\n\n[2] Dupont, Emilien, et al. \"From data to functa: Your data point is a function and you can treat it like one.\" arXiv preprint arXiv:2201.12204 (2022).\n\n[3] Zhuang, Peiye, et al. \"Diffusion probabilistic fields.\" The Eleventh International Conference on Learning Representations. 2023.", "claims_and_evidence": "The claims made in the submission is clear and convincing.", "methods_and_evaluation_criteria": "The goal is to train a unifying model in ambient space, using domain agnostic architecture. The proposed methods can solve the problem well. The dataset used (FFHQ-256, LSUN-Church-256, ImageNet-128/256 etc.) are diverse and standard, which shows the generality of the proposed method.", "theoretical_claims": "There's no theory part in this paper.", "experimental_designs_or_analyses": "The experiement is extensive and sound.", "relation_to_broader_scientific_literature": "The paper use INRs in the flow matching context with self- and cross-attention for latent variables, and the same strategies can be applied to other methods.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strength: the idea is simple and clear. Writing is easy to follow.\n\nWeakness: There may raise some concerns on novolties, as the proposed method is essentially combination of INR with flow matching, and using attention to handle latent variables for context information. Although the architecture is absolutely new and novel, the proposed method can solve problem well. Therefore, I would suggest borderline accept.", "other_comments_or_suggestions": "In 3.3 and 3.4, the paper use $x_{f_t}$ to denote coordinate at time t. However, the coordinate should be static, which serve as covariates for neural net approximating vector fields? If I'm correct, I would suggest use $x_f$ consistently for the entire paper (e.g. match to Figure 2)", "questions_for_authors": "The paper is clearly written.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The proposed approach solves an important problem of domain agnostic generative model architecture. The results indicate strong performance. The reviewers recognise the value of the contribution. 
One reviewer's recommendation stays bellow accept. However, this reviewer remained unresponsive during the rebuttal phase.
Decision: Accept (poster)