Normalizing Flows are Capable Generative Models
Authors: Shuangfei Zhai
Ruixiang ZHANG
Preetum Nakkiran
David Berthelot
Jiatao Gu
Huangjie Zheng
Tianrong Chen
Miguel Ángel Bautista
Navdeep Jaitly
Joshua M. Susskind
Venue: ICML 2025 oral (2025)
Keywords: Normalizing flows
autoregressive models
likelihood estimation
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We show that normalizing flows can work great as a generative modeling principle, and propose a simple architecture and set of techniques to achieve it.

Abstract: Normalizing Flows (NFs) are likelihood-based models for continuous inputs. They have demonstrated promising results on both density estimation and generative modeling tasks, but have received relatively little attention in recent years. In this work, we demonstrate that NFs are more powerful than previously believed. We present TarFlow: a simple and scalable architecture that enables highly performant NF models. TarFlow can be thought of as a Transformer-based variant of Masked Autoregressive Flows (MAFs): it consists of a stack of autoregressive Transformer blocks on image patches, alternating the autoregression direction between layers. TarFlow is straightforward to train end-to-end, and capable of directly modeling and generating pixels. We also propose three key techniques to improve sample quality: Gaussian noise augmentation during training, a post training denoising procedure, and an effective guidance method for both class-conditional and unconditional settings. Putting these together, TarFlow sets new state-of-the-art results on likelihood estimation for images, beating the previous best methods by a large margin, and generates samples with quality and diversity comparable to diffusion models, for the first time with a stand-alone NF model. We make our code available at https://github.com/apple/ml-tarflow.

Normalizing Flows are Capable Generative Models
Shuangfei Zhai'
Ruixiang Zhang'
Preetum Nakkiran'
David Berthelot Navdeep Jaitly"
Jiatao Gu Huangjie Zheng' Josh Susskind
Tianrong Chen Miguel Angel Bautista
Abstract
Diffusion
NF(OUrs)
36
Normalizing Flows (NFs) likelihood-based models for continuous inputs. They have demon- strated promising results on both density estima- tion and generative modeling tasks, but have re- ceived relatively little attention in recent years. In this work, we demonstrate that NFs are more powerful than previously believed. We present TARFLOW: a simple and scalable architecture that enables highly performant NF models. TARFLOW can be thought of as Transformer-basee vari- ant of Masked Autoregressive Flows (MAFs): it consists of a stack autoregressive Transformer blocks on image patches, alternating the autore- gression direction between layers. TARFLOW straightforward to train end-to-end, and capable of directly modeling and generating pixels. We also propose three key techniques to improve sam- ple quality: Gaussian noise augmentation dur- ing training, a post training denoising procedure, and an effective guidance method for both class- conditional and unconditional settings. Putting these together, TARFLOW sets new state-of-the- art results on likelihood estimation for images, beating the previous best methods by large mar- gin, and generates samples with quality and di- versity comparable to diffusion models, for the first time with a stand-alone NF model. We make our code available at https//gittub.com/lppeennt tarflow.
EPD
NFDW
Figure I. TARFLOW demonstrates substantial progress in the do- main of normalizing flow models, achieving state-of-the-art re- sults in both density estimation and sample generation. Left: We show the historical progression of likelihood performance Ima- geNet 64x64, measured in bits per dimension (BPD), where our model significantly outperforms previous methods (see Table 2 for details). Right: Selected samples from our model trained on ImageNet 128x 128 demonstrate unprecedented image quality and diversity for normalizing flow model, establishing new bench- mark for this class of generative models.
which transform a data distribution into simple prior distribution (such Gaussian noise), keeping track of like- lihoods via the change of variable formula. Normalizing Flows enjoy many unique and appealing properties, includ- ing exact likelihood computation, deterministic objective functions, and efficient computation of both the data gener- ator and its inverse. There has been a large body of work dedicated studying and improving NFs, and in fact NFs were the method of choice for density estimation for num- ber of years (Dinh al., 2017; Kingma & Dhariwal, 2018; Chen et al., 2018; Papamakarios et al., 2017; Ho et al., 2019). However in spite of this rich line of work, Nor- malizing Flows have seen limited practical adoption in stark contrast to other generative models such as Diffusion Models (Sohl-Dickstein et al., 2015; Ho et al., 2020) and Large Language Models (Brown et al., 2020). Moreover, the ofate-of-the-art in Normalizing Flows has not kept pace with the rapid progress of these other generative techniques, leading to less attention from the research community.
1. Introduction
Normalizing Flows (NFs) are a well-established likelihood based method for unsupervised learning (Tabak & Vanden- Bijnden, 2010; Rezende & Mohamed, 2015; Dinh et al., 2014). The method follows a simple learning objective,
'Apple. Correspondence to:
Shuangfei Zhai
<szhai apple..opp..cm......
It is natural to wonder whether this situation is inherent i.c., are Normalizing Flows fundamentally limited as a mod- eling paradigm? Or, have we just not found an appropriate
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Normalizing Flows are Capable Generative Models
way to train powerful NFs and fully realize their potential? Answering this question may allow us reopen alterna- tive path to powerful generative modeling, similar to how DDPM (Ho et al., 2020) enlightened the field of diffusion modeling and brought about its current renaissance.
standard image modeling tasks.
We highlight our main results in Figure 1, and summarize our contributions as follows.
We introduce TARFLOW, a simple and powerful Trans- former based Normalizing Flow architecture.
In this work, we show that NFs are more powerful than previously believed, and in fact can compete with state- of-the-art generative models on images. Specifically, we introduce TARFLOW (short for Transformer RetoRegressive Flow): powerful NF architecture that allows one to easily scale up the model's 's capacity; as well as a set of techniques that drastically improve the model's ' generation capability.
We achieve state-of-the-art results on likelihood estima- tion on images, achieving a sub-3 BPD on ImageNet 64x64 for the first time.
We show that Gaussian noise augmentation during training plays critical role in producing high qual- ity samples.
On the architecture side, FARFLOW is conceptually similar to Masked Autoregressive Flows (MAFs) (Papamakarios al., 2017), where we compose deep transformation by iteratively stacking multiple blocks of autoregressive trans- formations with alternating directions. The key difference is that we deploy a powerful masked Transformer (Vaswani et al., 2017) based implementation that operates in a block autoregression fashion (that is, predicting a block of dimen- sions a time), instead of simple masked MLPs used in MAFs that factorizes the input on per dimension basis.
We present a post-training score-based denoising tech- nique that allows one to remove the noise in the gener- ated samples.
We show that guidance compatible with both class conditional and unconditional models, which drasti- cally improves sampling quality.
In the context of image modeling, we implement each au- toregressive flow transformation with a causal Vision Trans- former (ViT) (Dosovitskiy et al., 2021) on top of a sequence of image patches, given a particular order of autoregression (e.g., top left to bottom right, or the reverse). This ad- mits powerful non-lineat transformation among all image patches, while maintaining a parallel computational graph during training. Compared to other NF design choices (Dinh et al., 2017: Grathwohl et al., 2019; Kingma & Dhariwal, 2018; Ho et al., 2019) which often have several types of interleaving modules, our model features a modular design and enjoys greater simplicity, both conceptually and practi- cally. This in return allows for much improved scalability and training stability, which is another critical aspect for high performance models. With this new architecture, we can immediately train much stronger NF models than previ- ously reported, resulting in state-of-the-art results on image likelihood estimation.
Table Notation.
Notation
Meaning
Pdlata (x)
training distribution
Pmodel()) f(x)
model distribution
the forward flow function
f((2)
the forward function for the t-th flow block
1², v''aa
learnable causal functions in the 1-th flow block the noise distribution the noisy data distribution
(e(e)
Q(v)
p(z)
the discrete model distribution
2. Method
2.1. Normalizing Flows
Given continuous inputs x Pdatas x E RD, a Nor- malizing Flow learns a density Pmoode via the change of variable formula (m) = (f()(x)) (e(())))))) ) where f : R. D RD is an invertible transformation for which we can also compute the determinant of the Jacobian det( df()) ); Po is prior distribution. The maximum likelihood estima- tion (MLE) objective can then be written
On the generation side, we introduce three important tech- niques. First, we show that for perceptual quality, it is critical to add moderate amount of Gaussian noise to the inputs, in contrast to small amount of uniform noise commonly used in the literature. Second, we identify a post- training score based denoising technique that allows one to remove the noise portion of the generated samples. Third, we show for the first time that guidance (Ho & Salimans, 2022) is compatible with NF models, and we propose guid- ance recipes for both the class conditional and unconditional models. Putting these techniques together. are able to achieve ofate-of-the-art sample quality for NF models
(d((x)
min log (o()()) log((det
1)-
(1)
dr
In this paper, we let po be standard Gaussian distribution (0, Ip), so Equation 1 can be explicitly written as
min 0.5 0.51 - -lo(( det (d((z) J
1),
(2)
dr
2

---

Normalizing Flows are Capable Generative Models
Unpatchify
2
2,
2
2,
2
12:
expl-af(2
a1
2,
2
Transformer AR Flow Block F(-)
xT
Causal Transformer
2
i
7,1
2
2;
Z'
2
2'
21
4
2
2
2;
Permutation m'(2')
2'
z:
4
Z
2
4
Z
2
Transformer AR Flow Block
Patchify
Figure Left, TARFLOW consists flow blocks trained end end; Right, zoom-in view of each flow bock, which contains sequence permutation operation, standard causal Transformer, and an affine transformation to the permuted inputs.
where have omitted constant terms. Equation 2 bears intuitive interpretation: the first term encourages the model to map data samples r to latent variables = f(z of small norm, while the second term discourages the model from "collapsing" - i... the model should map proximate inputs to separated latents which allows to fully occupy the latent space. Once the model is trained, one automatically obtains generative model via Po(z), x f-1(()))))) (2).
inputs {2t [E(]
zt n (2
x (i(Ett) exx(-a(((55)) i> 0
i 0 (3)
2/41
Note that since μ is causal, the i-token of its output (x()) only depends E as written explicitly above. Iterat- ing the above for t 0, 0, (T' 1) yields the output z =: f(x). The inverse function ---((=²) is given by iterating the following flow to obtain 2" from tt
2.2. Block Autoregressive Flows
One appealing method for constructing a deep normalizing flow is by stacking multiple layers of autoregressive flows. This was first proposed in IAF (Kingma et al., 2016) in the context of variational inference, and later extended by MAF (Papamakarios et al., 2017) as standalone density models.
1 = gtt o (xxp((((())) 2' (x') (x))
+
i> 0 (4)
In this paper, we consider a generalized formulation of MAF block autoregressive flows. Without loss of gen- erality, assume an input presented in the form of se- quence E Nx,, where N is the sequence length and D is the dimension of each block of input. Let T E N be the number of flow layers in the stack of flows. Sub- scripts denote indexing along the sequence dimension, e.g.
This yields x == 00 as the final iterate. As for the choice of permutations x,, f in this work we set all n' as the reverse function >²(=); = -N-1-i. except for d0 which is set as identity. Ultimately, the entire flow transformation consists of T flows (f*), and in each flow the input is first permuted then causally transformed with learnable element-wiss sub- tractive and divisive terms (.),), exp(a)())).
I; E IRD, and superscripts denotes flow-layer indices (see Figure 2). We then specify flow transformation 2T
It is worth noting that Equation degenerates to MAF when = 1. Intuitively, D plays a role balancing the difficulty of modeling each position the sequence and the length of the entire sequence. This allows for extra modeling flexibility compared the naive setting in MAF, which will become clearer in the later discussions.
f(x) (f fT' (z) as follows. First, we choose ) any fixed set of permutation functions along the sequence dimension. The t-th flow, ft, is parameter- ized by two learnable functions μ², a' IR Nx + NDD which are both causal along the sequence dimension.
We initialize with 20 : x. Then, the t-th flow transforms at NxD into 2-1 Nx by transforming a block of
In each flow transformation ft, there are two operations. The first permutation operation n' is volume preserving, therefore its log determinant of the Jacobian zero. The second autoregressive step has Jacobian matrix of lower
3

---

Normalizing Flows are Capable Generative Models
triangular shape, which means its determinant needs only account for the diagonal entries. The log determinant of the Jacobian then readily evaluates to
design:
training stability - i.e., training our model should be as easy as training a standard Transformer.
Combining the architecture and the loss (Equation 6) to- gether, we have complete recipe for a simple, scalable, and trainable NF model. See Figure 2 for an illustration of the architecture.
(2' d2a
N- 1 カー Σ
log( ddt(
1|)
(5)
3=0
Putting them together, the training loss of our model can be written as
2.4. Noise Augmented Training
It is considered a common practice to introduce additive noise to the inputs during the training of NF models (Dinh et al., 2017; Ho et al., 2019). The usage of noise has mostly been motivated from the likelihood perspective, where adding uniform noise whose width is the same as the pixel quantization bin size to images allows one to "dequantize" the discrete pixel distribution to a continu- ous one. Formally speaking, instead of directly modeling the training data distribution Pdata. we model a noise aug- mented distribution g(y) = Se Peita - e e)p. (e)de. With finite training set X. this can be explicitly rewritten as (v) = |&] ΣxEx Py - x). When evaluating likeli- hood, we follow the literature (Dinh et al., 2017) and let (c(-) (c; O, bin), where bin is the quantization bin size
T-1
min 0.51
(6)
which simply consists of a square term and a sum of linear terms.
2.3. Transformer Autoregressive Flows
Architecture design is arguably the most challenging aspect of NF models. We suspect that large part of the reason that NF' have not been as performant as other families of models is the lack of an architecture that allows for stable and scalable training.
To this end, we resort to a Transformer-basee architecture, TARFL.W, with a design philosophy that features simplic- ity and modularity. In particular, we realize the fact that Equation 3 favors a parallel implementation with attention masks. This follows the same spirit as the original MAFs, but we replace the MLP based implementation with much more powerful Transformer backbone which has proven track record of success across both discrete and continuous domains. This seemingly simple change allows one to fully unlock the potentials of autoregressive flows, to degree that has never been previously shown or expected.
(e.g.. T28 for 8-bit pixels normalized to the range of [-1, ID. We can then compute likelihood w.r.t. the discrete inputs ă with p(ã)
(o.n Pmodel + e)de.
For better perceptual quality during sampling, however, we show that it is critical to set (-(-) as a Gaussian distribution NN: 0, I2 whose magnitude o is small but larger than that of the pixel quantization bin size. To put it into context, with image pixels in 11, an optimal a of p.(-) for sample quality is around 0.05, whereas the standard deviation of the dequantization uniform noise is merely 0.002, order of magnitude smaller.
We now consider the concrete case of modeling images, with the discussions generalizable to other domains. Given an image of shape C. x H x W, where C, H, W are the channel size, height and width of the image, respectively, we first convert to a sequence of patches with a patch size of S. This gives
Why is this the case? There are two factors which could be important. First, training a NF model with good gener- alization is inherently a challenging task. Without adding noise, the inverse model f (z) is effectively trained on discretized inputs z, of the same size the training set. During the inference, however, f is expected to gener- alize on a much denser input distribution (e.g., Gaussian), which poses an out-of- dittof-diitriuttion problem that hinders the sampling quality. Adding noise therefore serves a simple purpose of enriching the support of the training distribution, hence the support of the inverse model f-1 Second, us- ing Gaussian noise instead of uniform is also critical, as the former effectively stretches the support of the training distribution to the ambient input space, with the mode of the density placed at the original data points. Although this makes it less straightforward to convert the learned density (vy) to a discrete data probability, but we will later see that it greatly enhances the sampling quality.
sequence representation of & E XD N = HW D CS?.. Similarly, the input of each flow transform z will have the same size as x. We can then readily apply a standard Vision Transformer with causal attention masks to implement the transformation of a single autoregressive pass f Importantly, the Transformer can have arbitrary depth and width, completely independent of the input's 's dimension.
When stacking multiple autoregressive transformations, the entire model can be viewed as variant of a Residual Net- work. More specifically, the network consists of two types of residual connections: the first over the hidden layers in- side the causal Transformer, the second over the latents 2/. This ensures another important factor of the architecture
4

---

Normalizing Flows are Capable Generative Models
Figure 3. Images of various resolutions generated by FARFLOOW models. From left to right, top to bottom: 256x256 images on AFHQ, 128x128 and 64x64 images on ImageNet.
2.5. Score Based Denoising
2.6. Guidance
Training with noise augmentation introduces an additional challenge: models trained on the noisy distribution (yy) nat- urally generate outputs that mimic noisy training examples, rather than clean ones. This results in samples that are less visually appealing. As remedy, we propose straightfor- ward training-fre free technique that effectively denoises the generated samples, by drawing inspiration from score-based generative models.
An important property of state-of-the-art generative models is their ability be controlled during inference. Normal- izing flows have conventionally relied on low temperature sampling (Kingma & Dhariwal, 2018), but it's only appli- cable to the volume preserving variants and also introduces severe smoothing artifacts.
On the other hand, guidance in diffusion models (Dhariwal & Nichol, 2021; Ho & Salimans, 2022) have achieved great success in this regard, which allows one to trade-off diversity for improved mode seeking ability, Surprisingly, we found that our models can also be guided, offering very similar flexibility to the case in diffusion models.
The idea is follows. Consider the joint distribution (ar y) where z Pdata and y = + for ~ NN, o²1). By definition, y is marginally distributed as the noisy data dis- tribution By Tweedie's formula, we have
In the conditional generation setting, guidance can be ob- tained in almost the exact same way as classifier free guid- ance (CFG) (Ho & Salimans, 2022) diffusion models. We first override the notation by letting p1(:: c), (:(:)) be the class conditional predictions, and (1 0), (=; 0) be the unconditional counterparts. In practice, the unconditional predictions can be obtained by randomly dropping out the class label during training, similar to (Ho & Salimans, 2022). For each flow block t, we modify the reverse function in Equation 4 to
E w] - y + 2 V log)(y).
Therefore, given a noisy sample y we can denoise it to a clean sample â == E[x y) if we know gradients of the log- small,
likelihood log g Under the condition when we have [f(x v) x. Now further assuming that the model (-) is well trained, we can use the same formula to denoise a sample from the model, using Pmodel in place of The complete sampling procedure can be written as:
Po.y:= := f-) == = + o² o²V, log Pmodel (v). (8)
x = 2 ++1 O exp(at(E(sic (f w)) + A w).
(9)
Here we generate x with the guided predictions Af(:c, af, w) under guidance weight w, which are defined
is worth noting that our score based denoising technique uses only the TARFLOW model itself, without requiring any extra modules. The fact that this works well (as demon- strated later in Sec. 3.3) suggests that learning the density of the noisy distribution is sufficient for recovering the score function, which is an interesting and significant result on its own in the context of score based generative models.
A w) = (1+ (1+ w/i(s 0). af w) (1 w)) wa)
(10)
Intuitively, under positive guidance w > 0, Equation 10
5

---

Normalizing Flows are Capable Generative Models
modifies the updates of sampling to guide conditional vari- ables oj away from predictions from an unconditional model, therefore converging more towards the class model of
Table 2. Bits per dim evaluation on unconditional ImageNet 64x64 test set. We denote the TARFLOW configuration in the format IP-Ch-T-K-p.J-
Model
Type
BPD 4
We have also discovered as method for applying guidance to unconditional models, see Appendix Section B for details.
Very Deep VAE (Child, 2021)
VAE
3.52
Glow (Kingma & Dhariwal, 2018)
Flow
3,81
Flow++ (Ho et al., 2019)
Flow
3.69
3. Experiments
PixelCNN (van Oord et al., 2016a)
AR
3.83
SPN (Menick & Kalchbrenner, 2019)
AR
3.52
We perform our experiments unconditional ImageNet 64x64 (van den Oord et al., 2016b), as well as class con- ditional ImageNet 64x64, ImageNet 128x 128 (Deng et al., 2009) and AFHQ 256x256 (Choi et al., 2020).
Sparse Transformer (Child et al.. 2019)
AR
3,44
Routing Transformer (Roy al., 2021)
AR
3.43
Improved DDPM (Nichol & Dhariwal, 2021)
Diff/FM
3,54
VDM (Kingma al., 2021)
Diff/FM
3.40
Flow Matching (Lipman et al., 2023a)
Diff/FM
3.31
Our models are implemented as stacks of standard causal Vision Transformers (Dosovitskiy al., 2021). In each AR flow block, the imputs are first linearly projected to the model channel size, then added with learned position embeddings. For class conditional models, we add an immediate class embedding on top of it. We use attention head dimensions of 64 and an MLP latent size 4x that of the model channel size. The output layer of each flow block consists of two heads per position, corresponding to / a/, respectively, and they are initialized as zeros. All parameters trained end-to-end with the AdamW optimizer with momentum (0.9. 0.95). We use a cosine learning rate schedule, where the learning rate is warmed up from 10-6 to 10-4 for one epoch, then decayed to 10-6, 60-6,0 We use a small weight decay of 10-4 to stabilize training.
NEDM (Bartosh ct al., 2024)
Diff/FM
3.20
TARFloW 12-768-8-8-4(0, 128
1 (
NF
2.99
Table 3. Fréchet Inception Distance (FID) evaluation Condi- tional ImageNet 64x64. We denote the TARFLOW contiguration in the format (P-Ch-T-K-pel-
Model
Type
FID +
EDM (Karras et al., 2022)
Diff/FM
1.55
iDDPM (Nichol & Dhariwal, 2021)
Dift/FM
2.92
ADM(dropout) (Dhariwal & Nichol, 2021)
Diff/FM
2.09
IC-GAN (Casanova et al., 2021)
GAN
6.70
BigGAN (Brock et al., 2019)
GAN
4.06
CD(LPIPS)(Song et al., 2023)
CM
4.70
We adopt simple data preprocessing protocol, where center crop images and linearly rescale the pixels to [-1, 1]. For each task, we search for architecture configurations consisting of the patch size (P), model channel size (Ch), number of autoregressive flow blocks (T) and the number of attention layers each flow (K). For generation tasks, we also search for the best input noise that yields the best sampling quality. We denote a TARFLOW configuration as P-Ch-T-K-pe-
iCT-deep(Song & Dhariwal, 2023)
CM
3.25
TARFloow [4-1024-8-8-N 0.05")] (Ours)
NF
3.99
TARFLOW [2- 2--76-8-8-- 0.05*) (Ours)
NF
2.90
TARFLOW [2- 80 0.05 (Ours)
NF
2.66
Table 4. Fréchet Inception Distance (FID) evaluation on Condi- tional ImageNet 128 128. We denote the TARFLOW configura- tion in the format [P-Ch-T-K-p.].
Model
Type
FID +
3.1. Likelihood
ADM-G (Dhariwal Nichol, 2021)
Diff/FM
2.97
CDM (Ho et al., 2022)
Diff/FM
3.52
Likelihood estimation provides direct assessment of a nor- malizing flow architecture's 'ss modeling capacity, it aligns precisely with the model's ss training objective. For evaluating likelihood on image data, unconditional ImageNet 64x64 has acted as the de facto benchmark dataset. Its relatively large scale and inherent diversity pose significant challenges for model fitting, making it an ideal testbed where improve- ments typically stem from enhanced model capacity rather than regularization techniques.
Simple Diff (Hongehoom et al., 2023)
Diff/EM
1.94
RIN (Jabri et al., 2023)
DifI/FM
2.75
BigGAN (Brock al., 2019)
GAN
8.70
BigGAN-doep (Brock et al., 2019)
GAN
5.70
TArFlow 84-1002-----V 0.05*)] (Ours)
NF
5.29
TARFLoW [4- 100-1-00----- 0.15") (Ours)
NF
5.03
During both training and evaluation, we apply uniform noise u(00, T28 to the data, which corresponds to the "dequantiza- tion" noise (Dinh et al., 2017). We do not use any additional
data augmentation techniques during training. As shown in Table 2 and visualized in Figure 1, our approach estab- lishes new state-of-the-an art result test set likelihood, by a significant margin over all previous models.
6

---

Normalizing Flows are Capable Generative Models
0 0.05
0 = 0.15
Conditionl
Unconditinal
FID Score (Conditiona Model)
Score (Uncond diniddiionall Model)
raw samples
denoised
Raw Samples
Denoised Samples
Classifer-Free Guidance Weight
FID Score (Denclsed Samples)
FID Score Samples)
1.3-
16
Figure 5. Guidance weight w vs FID for both the conditional and unconditional models (with r = L.5) on ImageNet 64x64. Note the axis's 's scale difference between the two settings.
60-
80-
We then perform quantitative evaluations in terms of FID, on the ImageNet models. For each setting, we randomly generate 50K samples, and compare with the statistics from the entire training set. We search for the best guid- ance weights (and attention temperature in the unconditional case). The results summarized in Table 3, 6, 4. In all settings, we see that TARFLOW produces competitive FID numbers, often times better than strong GAN baselines, and approaching results from recent Diffusion Models. It is also interesting to note that we found no publicly reported NF based FID numbers on the ImageNet level datasets, most likely due the lack of presentable results from the NE community.
20-
0.01
0.35
Input Noise Standard Deviation (o)
Figure 4. Top:
The effect of imput noise a and denoising, all sam- ples are generated with guidance weight
on ImageNet 128x128 from the same initial noise, hetter viewed when zoomed in. Bottom: Sample FID vs input noise a on ImageNet 64x64, with and without denoising. Before denosing, it first appears that small a has the best FID, due the smaller amount of noise present in the raw samples. However, after denoising with Equa- tion 8, slightly larger favors better FID and demonstrates more consistent shapes. Note that the scale of the right y-axis differs from that of the left.
3.3. Ablation on Noise Augmentation and Denoising
We then study the role of input noise pe- We first experi- mented with the 'dequantization' uniform noise and found that sampling experiences constant numerical issues and was not able to produce sensible outputs. We hypothesize the reason being that a narrow uniform noise makes the flow transformation ill-conditioned, as forces a model to map low entropy distribution to an ambient Gaussian distribution.
3.2. Generation
Next, we evaluate TARFLOW's sampling ability in class conditional (ImageNet 64x64, ImageNet 128x128, AFHQ 256x256) as well as unconditional (ImageNet 64x64) set- tings. Our experimental protocol is largely the same as previously mentioned, except that we adopt random hori- zontal image flips. For the class conditional models, we randomly drop the class label with probability of 0.1.
Next, experiment with different Gaussian noise levels during training on class conditional ImageNet 64x64. We use an architecture configuration of 4- 10244-8--, and vary a in (0.01, 0.05, 0.2, 0.5). For fast experimentation, we train all models for only 100 epochs with a batch size of 512. We evaluate these models with a guidance w and plot the 50K sample FIDs before and after the score based denoising. For visual comparison, we also train two models ImageNet 128x models, with the architecture 4- 1-24-88- and noise E [0.05, 0.15], respectively. We show the FID curves together with the visual examples in Figure 4. There are two important observations. First, naively increasing the noise level on the surface appears to
We first show qualitative results in Figure 3, which are ob- tained with the sampling procedure in Equation 8. We see that TARFLOW generates diverse and high fidelity images all settings. Also, TARFLOW seems to demonstrate great robustness w.r.t. the data size and resolution. For instance, works well on both large diverse dataset (ImageNet, 1.3M examples, IK classes) and small but high resolu- tion one (AFHQ, 15K examples in 3 classes and 256x256). Visually, these samples are comparable to those generated by Diffusion Models, which marks a large improvement from the previous best NF models. We include more quali- tative results in the Appendix.
7

---

Normalizing Flows are Capable Generative Models
Loss
FID Score
Minimum Loss
Minimum FID
Training FID
-2.15
Negative Log Likellhaad Lass
Frechet Inception Distance
-2.66
-2
-38
20
20
FID Score
-2.17
-0.17
Traihing
15
10
-2 18
10
-215
231
231-
1d54
2:22
6451
6x4
(a)
Epochs
(b)
Network Depth Configuration
Network Depth Contiguration
Figure 6. (a) A typical training run on ImageNet 64x64. Our loss smoothly decreases during training, and is positively correlated with FID. (b) Depth configuration (in the form of 7 x K)) training loss. Overall, see strong positive correlation between the training loss and FID. Left, the optimal training loss happens when the capacity is evenly allocated to number of blocks and number of layers per block. Interestingly, the special case of block degenerates to an incapable model, which has both high loss and FID equivalent to random guess 267). Right, increasing both the number of blocks and number of layers per block improves the model's loss and sampling quality,
hurt the raw samples' quality. However, this is no longer the case after applying the denoising step in Equation 8. Denoising successfully cleans up the noisy raw samples, and as a result the best visual quality occurs at a moderate (but still relatively small) amount of noise. This verifies the necessity of our proposed sampling procedure, whereas the combination of noise augmented training and the score based denoising step work organically together produce the best generative capability. See the Appendix for more visualizations of the effect of denoising.
size, especially model's depth. Depth plays vital role in our model, as we need to have a sufficient number of flow blocks, as well as number of layers within each block. This deep transformation then poses questions on architecture design as well as its trainability.
We answer this question by performing two sets of ablations conditional ImageNet 64x64. In the first set of experi- ments, we train a set of models who share the same number of combined layers T x K: and in the second, we increase a base model's depth by increasing either T or K. The results are shown in Figure 6(b). First of all, we observe again the strong positive correlation between the loss and FID curves, across different architectures. This points a nice property of NF models where improving the likelihood (i.e., the loss) directly leads to improved generative modeling capabilities. Second, there is a U-shape distribution w.r.t. the T x K configuration, and it appears that the best trade-off occurs when T = K. The case of T = is also interesting, as it corresponds to a special case of a single direction autore- gressive model on image patches. It is obvious that this model fails to fit the data, both in terms of loss and FID. This is in contrast to the T 2 configuration which has much more reasonable performance. Lastly, increasing either T or K is effective in improving the model's capacity. Putting these observations together, we see that FL.ow demonstrates promising scaling behaviors, which makes it a particularly appealing candidate for exploiting the wide abundance of power of modern compute infrastructures.
3.4. A Ablaioon on Guidance
To see the effect of guidance, we perform qualitative and quantitative evaluations on both the class conditional and unconditional versions of ImageNet 64x64. The results are shown in Figure 5 and 7 (in Appendix). In terms of FID, the guidance weight w plays an effective role for both mod- els. Visually, it also clear that guidance allows the model to converge to more recognizable modes, presenting more aesthetic samples. Interestingly, this is also somewhat true for the unconditional models, whereas both the guidance weight w and attention temperature contribute to the de- gree of guidance. We show more guidance comparisons in the Appendix.
3.5. Ablation on Model Scaling
Regarding scaling, we first show a typical training loss curve together with an online monitoring of the model's sample quality in terms of FID (we use 4096 samples for efficiency). This is shown in Figure 6(a). We see that the loss curve is smooth and monotonic, and it has a strong positive correlation with the FID curve.
3.6. Comparison with VP and Channel Coupling
We also ablation two important design choices of TARFLOW: the non-volume preserving (NVP) and autoregressive aspect. We train two baseline models: one to change NVP to VP
We proceed to discuss another design question: the model's
8

---

Normalizing Flows are Capable Generative Models
key component for realizing the true potential of the NE principle.
Table 5. VP and channel coupling w.r.t. FII ImageNet 64x64,
Guidance
TARFLOW
VP
channel coupling
Autoregressive Normalizing Flows. IAF (Kingma et al., 2016) introduced dimension affine transformations conditioned on preceding dimensions for variational infer- ence, and MAF (Papamakarios et al., 2017) leveraged the MADE (Germain et al., 2015) architecture to construct in- vertible mappings through autoregressive transformations. Neural autoregressive flow (Huang et al., 2018) replaces the affine transformation in MAF by parameterizing a mono- tonic neural network, at the cost of losing analytical invert- ibiliiy. T-NAF (Patacchiola et al., 2024) extends NAF by introducing a single autoregressive Transformer. Block Neu- ral Autoregressive Flow (Cao et al., 2019) fits an end-to-end autoregressive monotonic neural network, rather than NAF's dimension -wise sequence parameterization, but also sacri- fices analytical invertibility. TARFL.Ow differs from these as we show that it is sufficient to stack multiple iterations of block autoregressive flows with standard Transformer model in alternating directions, without the need for other types of flow operations.
25.3
81.5
50.3
2
5.7
51.0
20.4
(done by setting a-(() to 0 in Equation 3) and the other a channel coupling baseline by using the same architecture but removing the causal masks. We use the same 0.05 Gaussian noise on both, and found that and denoising consistently help. For the VP baseline, we found that also need to learn the prior's variance as the latents tend to have very small magnitudes which makes sampling from the standard Gaussian prior immediately fail. All models share similar training costs which makes a fair comparison. The results are shown in Table 5. We see that both variants significantly under perform TARFLOW. Also, interestingly, guidance consistently improves both settings.
4. Related Work
Coupling-based Normalizing Flows. NICE (Dinh et al., 2014) introduced additive coupling layers to construct the transformations and simplified the computation of the Ja- cobian determinant. RealNVP (Dinh al., 2017) extended this approach by incorporating scaling and shifting opera- tions to enhance the model's expressiveness. Glow (Kingma & Dhariwal, 2018 8) advanced these models by introducing invertible 1 x 1 convolutions, achieving improved results in image generation tasks. Flow++ (Ho et al., 2019) further introduced Icarned dequantization noise, a sophisticated coupling layer and attention mechanisms to enhance the model's is expressiveness. What's shared in common among these designs is that they need carefully wired and restrictive architectures, which poses great a challenge in scaling the model's 's capacity.
Probability Flow in Diffusion & Flow Matching, Diffu- sion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021) generate data by simulating Stochastic Differential Equations. Song et al. (2021) provided a deter- ministic Ordinary Differential Equation (ODE) counterpart to this generative approach, also known as the probability flow (Song et al., 2021). Similarly, Flow Matching (Lipman et al., 2023a) proposes to learn such ODEs by training on linear interpolants of data and noise with the velocity pre- diction objective. Importantly, TARFLOW differs from these instances as it is directly trained with the MLE objective, without the need for excessively large Gaussian noise during training.
See also Sec. A in Appendix for additional related works.
Continuous Normalizing Flows. Neural Ordinary Dif- ferential Equations (Chen et al., 2018) based Continuous Normalizing Flows is an alternative NF design principle. In this framework, the invertibility of the network is inherently satisfied, and the computation of the Jacobian determinant within the normalizing flow is reduced to calculating the trace of the Jacobian. FFJORD (Grathwohl et al., 2019) further simplifies the expensive Jacobian computation by employing Hutchinson's trace estimator (Hutchinson, 1989). However, these models often suffer from numerical instabil- ity during training and sampling, which has been extensively analyzed in (Zhuang et al., 2021; Liu al., 2021). The ex- pressive capability can be further improved by augmenting auxiliary variables (Dupont et al., 2019; Chalvidal et al., 2021). In comparison, TARFLOW enables an unconstrained architecture design paradigm by fully taking advantage of the power of causal Transformers, which we believe a
5. Conclusion
We presented TARFLOW, a Transformer-based architecture together with a set of techniques that allows us to train high- performance normalizing flow models. Our model achieves state-of-the-arr results on likelihood estimation, improving upon the previous best results by a large margin. We also show competitive sampling performance, qualitatively and quantitatively, and demonstrate for the first time that nor- malizing flows alone are a capable generative modeling technique. We hope that our work can inspire future inter- est in further pushing the envelope of simple and scalable generative modeling principles.
9

---

Normalizing Flows are Capable Generative Models
Acknowledgements
Romero-Soriano, A. Instance-condttioned GAN. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 27517- 27529, 2021. URL https: httt:p://procceednng.
We thank Yizhe Zhang, Alaa El-Nouby, Arwen Bradley, Yuyang Wang. and Laurent Dinh for helpful discussions. We also thank Samy Bengio for leadership support that made this work possible.
Impact Statement
neurips cc/ https/aaappee/e//aaaaaaa//////// e eTc=288boFA44145550001111a7a7 html.
This paper concerns the generative modeling methodology. While we do not see immediate societal implications from our technical contribution, there are potential impacts when it is used in training foundational generative models.
Chalvidal, M., Ricci, M., VanRullen, R., and Serre, T. Go with the flow: Adaptive control for neural odes. In 9th International Conference
Learning Representa- tions, ICLR 2021, Viriual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL /ttp:::///peeeeeie.. net/ / forum? wiw.mmmiiigggge iddaait 4HdDNa.
References
Bartosh, G., Vetrov, D., and Naesseth, C. A. Neu- ral flow diffusion models: Learnable forward process for improved diffusion modelling. ArXiv preprint, abs/2404.12944 2024. URL 3bs/2404.1295
Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pix- els. In Proceedings of the 37th International Confer- ence on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Ma- chine Learning Research, pp. 1691-1703, PMLR, 2020, URL http::/prooceddng mlr ptt.sppreesvvv11 chen20s.hta
Brock, A., Donahue, J., and Simonyan, K. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference Learning Representa- tions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:: /pppneeveeee fott/form?? i wtwp::xxxxx....oo
Chen, T., Gu, J., Dinh, L., Theodorou, E., Susskind, M., and Zhai, S. Generative modeling with phase stochas- tic bridge. In The Twelfth International Conference Learning Representations, 2024.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.. Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., feererrtVosss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
Chen, T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. Neural ordinary differential equations. In Bengio, S.. Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi. N., and Gamett, R. (eds.). Advances in Neural Information Processing Systems 31: Annual Con- ference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 6572-6583, 2018. URL httpt: httttpsss//proceeiingss neur Souripsaa ps http://eeepeee/eeee/eeeaaa 6o288ffbbideeed33222222666633333 htw.:b599339b----baaac--- Abstract html.
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: btttps:/pprooedinggg
Language
/prpceedings...
neurips. oc http://scc.cpaappee////////a/a// 4457c0d@bffb49674118888888 html.
Abstract.
Child, R. Very deep vaes generalize autoregressive models and can outperform them on images. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: http:///ppeeeeieeeeeemm net n id=RLRXCV6Dbe..
Cao, N. D., Aziz, W., and Titov, I. Block neural autore- gressive flow. In Globerson, A. and Silva, R. (eds.), Proceedings of the Thirty-Fifth Conference on Uncer- tainty in Artificial Intelligence, UAI 2019, Tel Aviy, Israel, July 22-25, 2019, volume 115 of Proceedings of Machine Learning Research, pp. 1263-1273. AUAI Press, 2019. wnw.oeeeeedii.eg.....ooo ngs. de de- caw-0a020...omm aato020a.htm...
URL
press paress/vlle vv155 wll...l...... mlr. w.p..sesss.
Child, R., Gray, S., Radford, A., and Sutskever, I. Gener- ating long sequences with sparse transformers. ArXiv preprint, abs/1904.10509, 2019. URL https:// arxiv.org/abs/1990
Casanova, A., Careil, M., Verbeek, J., Drozdzal, M., and
10

---

Normalizing Flows are Capable Generative Models
Choi, Y., Uh, Y., Yoo, J., and Ha, J. Stargan v2: Diverse im- age synthesis for multiple domains. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 8185-8194, IEEE, 2020. doi: 10. P0.14000 2020.00821. URL https: /tt.//di.../ org/1ogg////o. 1109/ CVPR42600 2020. 00821.
Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 3134-
3144, 2019. URL https: http///eooceeen.../////////ss/se
neurips ctc.ipssccpptee httpr//sscepappe////////aeeeee
21 html.
01888854499990 d241 981 cec3c- Abstract.
Esser, P., Rombach, R., and Ommer, B. Taming trans- formers for high-resolution image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 12873-12883, Computer Vision Foundation / IEEE, 2021. doi: 10.1009CCVPR4644.2222...2222
Deng, J., Dong, W., Socher, R., Li, and Li, F. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEEE Com- puter Society, 2009, doi: URL https: attps::/dei..rr//// 5206848.
URL https::/ tww.:s..sshecccccc..ooo.oo com/ Content//cVP2222///mm///s//
otg./.oorg//00 http://..11119/911 11w.11119///e 2009.
Transformers_ for httpereeere_oo__tggghhheeeooooo Resouution Image_ Slage_Synthesis_cn cvp____________ 2021 2aapaaae11eapppeeeeeeeeee html.
Dhariwal, P. and Nichol, A. Q. Diffusion models beat gans on image synthesis. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances Neural Infor- mation Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 8780-
Germain, M., Gregor, K., Murray, I., and Larochelle, H. MADE: masked autoencoder for distribution estimation. In Bach, F. R. and Blei, D. M. (eds.), Proceedings of the 32nd International Conference Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 881- 889. JMLR.org, 2015. URL http: http:://pooceeeeeeeeeeee mir.press svreess/v37. v /ttp////eemnnnnm// htmll
8794. 2021. URL https: https://prpreeeee hptp:://///ccccogg......
neurips oc 9aad2aalessaaaaee772222eeffffff.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., and Bengio, Y. Generative adversarial nets. In Ghahramani, Z. Welling, M., Cortes, C., Lawrence, N. D., and Weinberger, K. Q. (eds.), Advances in Neural Informa- tion Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 2672- 2680, 2014. URL https: http:::/pproceeinng
html.
Dinh, L., Krueger, D., and Bengio, Y. Nice: Non-linear independent components estimation. International Con- ference on Learning Representations workshop Track, 2014.
Dinh, L., Sohl-Dickstein, J., and Bengio, S. Density es- timation using real NVP. In 5th International Confer- ence Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL httpp::///et //ppnreiiee.....w....o.o net/ htte/forummiiiee hitphikkkdddkkkkk
wwaripss.cc/papee//2000 /hw.h/aahh//// $cale9012f661f88f0644ccc..bb...9 html.
Abstract.
Grathwohl, W., Chen, R. T. Q., Bettencourt, J., Sutskever, I., and Duvenaud, D. FFJORD: free-form continuous dynamics for scalable reversible generative models. In 7th International Conference on Learning Representa- tions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.ne,, 2019. URL htthtt:s////ot http::::////peeeveeee. net/ forum? wattoorrmmid=rreeeeeee d=
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.. Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Con- ference on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:://// htttp:://pperreviii net/ forum?
Gu, J., Wang, Y., Zhang, Y., Zhang, Q., Zhang, D., Jaitly, N., Susskind, J., and Zhai, S. Dart: Denoising autore- gressive transformer for scalable text-to-image gener- ation. ArXiv preprini, abs/2410.081559 2024. URL /210 08159.
dNTTT.
Dupont, E., Doucet, A., and Teh, Y. W. Augmented neural odes. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d'Alché-Bue, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural
Ho, J. and Salimans, T. Classifier-free diffusion guidance. ArXiv preprint, abs/2207. 2022. URL https: //arxiv. (arrdw..rrg/bb//2207.222

---

Normalizing Flows are Capable Generative Models
Ho, J., Chen, X., Srinivas, A., Duan, Y., and Abbeel, P. Flow++: Improving flow-based generative models with variational dequantization and architecture design. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learn- ing, ICML 2019, 9-15 June 2019, Long Beach, Califormia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2722-2730. PMLR, 2019. URL http:// proceedings ngg mlm press /v./sss///////// v97 ha.hmm.
Machine Learning Research, pp. 14569-14589, PMLR, 2023. htthttps: /tttttp::/porceee http///ooeddgggeeeeeee ml r..esss//// v202/jabri23a, html.
Kang. M., Zhu, J., Zhang, R., Park, J., Shechtman, E., Paris, S., and Park, T. Scaling up gans for text-to- image synthesis. IEEF/CV Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancou- ver, BC. Canada, June 17-24, 2023, pp. 10124-10134. IEEE, 2023. doi: 10. (0.1100/CVVR5227792222.00 URL https:: /tle::/doooio htt.dii.oggg....o
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual. 2020. URL https: httppas/pprrcceciinig
Karras, T., Laine, S., and Aila, T. style-based generator architecture for generative adversarial networks. In IEEE Conference
Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20. 2019, pp. 4401-4410. Computer Vision Foun- dation / IEEE, 2019. doi: 10.1109/CVPR.2019.000 com/
http///eeoc.c.../////e////e.../p neurips. httcc///aae wwtpc//ppppp//2////eaaee 4c5bcffc3588a500667TT htt.://aaabb.1111a-aaab-bbee html.
URL contenntccvPp_2011//tnn//rrere
Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23:47:1- 47:33, 2022. URL http: jmlr http:imtl...oeeeeeeeeeeeoo wtgppapers//////ooooooooggoseare 028/2100635 .ttml.
for for__eeneratiive_ddveessrrllllld CXPR_2019_paper.htm
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion based generative models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Infor- mation Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Hoogeboom, E., Heek, J., and Salimans, simple diffu- sion: End-to-end diffusion for high resolution images. In Krause, A., Brunskill, E., Cho. K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Confer- ence on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 13213-13232. PMLR, 2023. URL htwps: http:::ppp///pppeeeeeeee mmlr.press. v2 http///hoobboom2aa... html.
Kingma, D., Salimans, T., Poole, B., and Ho, Varia- tional diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
Huang, C., Krueger, D., Lacoste, A., and Courville, A. C. Neural autoregressive flows. In Dy, J. G. and Krause, A. (eds.), Proceedings of the 35th International Conference Machine Learning. ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Pro- ceedings of Machine Learning Research, pp. 2083-2092. PMLR, 2018. URL http: http///poccceeeneeeeeeeee mlr. press/ / vv80 /hhunngl8e html.
Kingma, D. P. and Dhariwal, P. Glow:
Generative flow with invertible Ixl convolutions. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 10236- 10245, 2018. URL https://proceeding
Hutchinson, F. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Com- munications in Anaristies-Simlltiion and Computation, 1863):0059-006, 1989.
neur / http///11///haa/// 1L3300a2226200222cc1111222233333 ott.rarrrr.tt.11122oo.tttta.aa.. Abstract. html.
Jabri, A., Fleet, D. J., and Chen, T. Scalable adap- tive computation for iterative generation. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato. S., and Scarlett, J. (eds.), International Conference on Ma- chine Learning. ICML 2023. 23-29 July 2023, Hon- olulu, Hawaii USA, volume 202 of Proceedings of
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In Bengio. Y. and LeCun, Y. (eds.), 2nd Interna- tional Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http: / /arxivv. .org/ abs/1312. 6114.
12

---

Normalizing Flows are Capable Generative Models
Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M. Improved variational infer- ence with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
Papamakarios, G., Murray, I,, and Pavlakou, T. Masked autoregressive flow for density estimation. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp.
Li, I., Tian, Y., Li, H., Deng, M., and He, K. Autore- gressive image generation without vector quantization. ArXiv preprint, abs/2406.1 1838, 2024. URL https: //arxiv. org/ 2bbs/206. 11838.
2338 2348, 2017. URL https: https:/pprocediingg pttpp/eecceeeeeecc.ee neurips. neurips.cc/ cttp://cccpppppp/eeaeeeeeee (cc1188662266122222f666666666666 html.
Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1- 5, 2023. OpenReview.net, 2023a. URL https:// openreview net pdf PUMMRDJIST..
Patacchiola, M., Shysheya, A., Hofmann, K., and Turner, R. E. Transformer neural autoregressive flows. ArXiv preprint, abs/2401.018 arxiv. wwtpg//////////////////////aaee/ abs /4001. 01855.
2024. URL https://
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May
5. 2023. OpenReview.net, 2023b. URL https:/ openreview net. mett/pdffiieePprMoDOTTT
Pooladian, A., Ben-Hamu, H., Domingo-Enrich C., Amos, B., Lipman, Y., and Chen, R. T. Q. Multisample flow matching: Straightening flows with minibatch couplings. Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, (eds.), International Confer- ence on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 28100-28121) PMLR. 2023. httphttsss//prrceeeeee / press/ v202 htt2//oooldaaan22... html.
Liu, G., Chen, T., and Theodorou, E. A. Second-order neural ODE optimizer. In Ranzato, M., Beygelz- imer, A.. Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Pro- cessing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 25267- 25279, 2021. URL http:ss:///roceednngs..
mlr
neurips.com pthip//ss///pper///2////ss//// 0r02e412227122222770000877788883 html.
Razavi, A., van den Oord, A., and Vinyals, O. Generating diverse high-fidelity images with VQ-VAE-2. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' d'Alché-Bur -uce F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural In- formation Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pp. 14837-14847, 2019. URL https::/ppoceeednggs
Menick, J. and Kalchbrenner, N. Generating high fidelity im- ages with subscale pixel networks and multidimensional upscaling. In 7th International Conference on Learn- ing Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: //openreviews www.ereewiewww..eeeeeeee htt.i/ewwwee//oounnnee forum? 1
neur peurips.cc/pppe 201 9/hash/ E html.
Nichol, A. Q. and Dhariwal, P. Improved denoising dif- fusion probabilistic models. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Con- ference on Machine Learning, 1CML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Ma- chine Learning Research, pp. 3171-8171. PMLR, 2021. URL htth: httpp/ppcceenn...eeee ml www.ss/essso..e. /v139/ nichol21a, html.
Rezende, D. and Mohamed, S. Variational inference with normalizing flows. In Bach, F. R. and Blei, D. M. (eds.), Proceedings of the 32nd International Confer- ence Machine Learning, ICML 2015. Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1530-1538. JMLR.org, 2015. URI. hwtp: http:::///ooccoeeeeeeoeoooooeeee ngs..nn.......... r.press/ html.
Noroozi, M. Self-labeled conditional gans. ArXiv preprint, abs/2012.02162, 2020. URL https: http///axxxvvoo// org/ abs/2012. 2. 021.2.
Roy, A., Saffar, M., Vaswani, A., and Grangier, D. Effi- cient content-based sparse attention with routing trans-
13

---

Normalizing Flows are Capable Generative Models
formers. Transactions of the Association for Compula- fional Linguistics, 9:53-68, 2021. doi: 10.1162/tacl a_00353. URL https://aae https://sclanthhology org/ 2021. tacl-1.4.
Tschannen, M., Pinto, A. S., and Kolesnikov, A. Jet- former: An autoregressive generative model of raw im- ages and text. ArXiv preprint, abs/2411.19722, 2024. URL https: http//axxivv..g/gaa..... wtt.ooogggaaaagoooooooo
Sherstinsky, A. Fundamentals of recurrent neural network (rnn) and long short-term memory (Istm) network. Phys- ica D: Nonlinear Phenomenu, 404:132306, 2020.
Tschannen, M., Eastwood, C., and Mentzer, F. Givt: Gen- erative infinite-vocabulary transformers. In European Conference on Computer Vision, pp. 292-309. Springer, 2025.
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamies. In Bach, F. R. and Blei. D. M. (cds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 2256-2265. JMLR.org 2015. URL http:: http:::/prcceee.nnngnn v37/soh1-dicksteiinee html.
van den Oord, A., Kalchbrenner, N., Espeholt, L., Kavukcuoglu, K., Vinyals, O., and Graves, A. Con- ditional image generation with pixelenn decoders. In Lee, D. D., Sugiyama, M., von Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Infor- mation Processing Systems 29: Annual Conference Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4790- 4798, 2016a. URL https: http::://proceedingg
mlr. prrss//
neur ips. # Abstraat.. html.
wwtpccccppppppppccccc 220 6/hash/
Song, Y. and Dhariwal, P. Improved techniques for train- ing consistency models. In The Twelfth International Conference on Learning Representations, 2023.
van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. Pixel recurrent neural networks. In Balcan, M. and Weinberger, K. Q. (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, vol- ume 48 of JMLR Workshop and Conference Proceed- ings, pp. 1747-1756. JMLR.org, 2016b. URL http:// proceedings. ml www.sppe/e//oorreseeoe 6 html..
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021. URL https:tt//cc....c http:///ppppeeeeeeeeeooee Forum?
Song. Y., Dhariwal, P., Chen, M., and Sutskever, I. Con- sistency models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Inter- national Conference Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 32211- 32252. PMLR, 2023. URL https: / /tte::/prrroooo /ptppppeeeeens... www.presss/eeee /v202/22222/// /ttpee////222//oggg33333 song23a. html.
van den Oord, A., Vinyals, O., and Kavukcuoglu, K. Neural discrete representation learning. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M.. Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6306 6315, 2017. URL https: htttts::/proceeeeee pttp:///ccccccc..../... neurips cc. wwurips..c/ppaerr/// / 201ha///////////// 0.gaaf17eccaaaa000eeee9693333333 Abstaaaat......taaattttttt...... html.
mlr. wwr.press.ee
Sun, P., Jiang, Y., Chen, S., Zhang. S., Peng. B., Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation. CoRR, abs/2406.006255, 2024. doi: 10448550ARRXIV2240606222 URL ht https: // //doi. . org/ 10. (al///55//iiiii .20............. 2406. 06525.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Atten- tion is all you (ness 2017. Advances in neural information processing systems, 30, 2017.
Tabak, E. G. and Hunden-Eiijded E. Density estimation by dual ascent of the log-likelihood. Communications in Mathematical Sciences, 8(1):217-233, 2010.
Wiatrak, M., Albrecht, S. V., and Nystrom, A. Stabiliz- ing generative adversarial networks: A survey. ArXiv preprint, abs/1910.00927,, 2019. URL https:// arxiv. orx.v.org/aae abs 1910111111.1 00927.
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang, L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. ArXiv preprint, abs/2404.02205,, 2024. URL https: http///axxvvvv/// abs/2404 02905.
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B, K., et al. Scal- ing autoregressive models for content-rich text-to-image
14

---

Normalizing Flows are Capable Generative Models
generation. Transactions Machine Learning Research, 2(3):5, 2022.
Zheng, Z., Peng. X., Yang. T., Shen, C., Li, S., Liu, H., Zhou, Y., Li, T., and You, Y. Open-sora: Democratizing efficient video production for all, 2024. URL https: //github. com hpca tepchOpen-eeeeeeeeeeeeeeeeeeee Sora.
Zhuang. J., Dvomek, N. C., Tatikonda, S., and Duncan, S. MALI: A memory efficient and reverse accurate integrator for neural odes. In 9th International Con- ference on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htttps://openreeee/ net forum?
fd-blfsjHeeM_e.
15

---

Normalizing Flows are Capable Generative Models
A. Additional Related Work
Autoregressive Models for Image Generation Many efforts (van den Oord et al., 2016b;a; Esser et al., 2021; Razavi et al., 2019) have been made apply autoregressive sequential methods image generation. PixeIRNNN(an den Oord et al., 2016b) is a pioneering work in this field. This approach views an image as sequence of data, modcling the distribution of each subsequent pixel conditioned all previously generated pixels through RNN architecture(Shersttssky 2020). This methodology is readily adaptable to masked convolutional structures(van van den Oord et al., 2016a), where the prediction of the next pixel is based on its neighboring pixels, bypassing the use of a traditional convolutional kernel. The transformer model has been successfully applied to image generation tasks. Chen et al. (2020) introduced ImageGPT, an autoregressive model that predicts pixels sequentially in raster order. More recently, Yu et al. (2022) introduced Parti, a scalable encoder-decoder transformer for text-to-image generation, which conceptualizes the task as a sequence-to-sequeee 2024) begins with low-resolution images in the latent space, effectively predicting the next level of resolution and yielding impressive outcomes. Studies like (Yu et al., 2022; Gu et al., 2024; Sun et al., 2024) further demonstrate the scalability and effectiveness of autoregressive models in producing high- nigh-dimensional images. Recent work MAR (Li et al., 2024) introduced diffusion models for autoregressive latent token prediction as an alternative to vector quantization approaches for image generation. GIVT (Tschannen et al., 2025) employed transformer decoders to model latent tokens generated by a VAE encoder, while incorporating a Gaussian Mixture Model (GMM) in place of categorical prediction for likelihood modeling. Concurrently to our work, JetFormer (Tschannen al., 2024) further extended the approach by substituting the VAE with a coupling-bass baseid normalizing flow model, and used an autoregressive Transformer with GIVT's GMM prediction head to model sequences of latent tokens. While TARFLOW employs a causal Transformer architecture similar these approaches, it operates differently by processing continuous data directly with single model, thus avoiding the complexity of input discretization, or the need for separate image tokenization and autoregressive modeling stages.
problem. VAR(Tian et al.,
Diffusion models, other generative models Diffusion models (Ho et al., 2020; Song et al., 2021) are emerging generative models that achieve appealing results. Stable Diffusion (Podell et al., 2024) and OpenSora (Zheng et al., 2024) push the boundaries of diffusion models' capabilities, demonstrating their ability to generate extremely high-dimensional data. Besides, Variational Autoencoders (VAEs) (Kingma & Welling, 2014) and Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are also popular generative models. By avoiding the posterior collapse issue, VQ-VAE (van den Oord et al., 2017) demonstrates impressive generative performance and subsequently serves an essential component in the later latent diffusion model (Podell et al., 2024). In the realm of GANs, Karras et al. (2019); Kang et al. (2023); Brock et al. (2019) showcase the remarkable capability of GANs to generate high-resolution images with comparatively cheap inference costs, though the training stability of GANs remains challenging (Wiatrak et al., 2019). TARFLOW represents an orthogonal learning paradigm to these methods, with its unique benefits and challenges.
B. Guidance
In addition conditional guidance, we also introduce novel method for guiding unconditional models. The basic idea is to construct predictions of inferior quality, analogous to the role of unconditional ones. In order to do so, we override the notation yet again and introduce μ ((::)) r), af((:)) Here we have let the predictions μÝ of take an additional parameter which corresponds to a manually injected temperature term to all the attention layers in the Transformer for f Namely, for each attention layer in f', we divide the attention logits by т, before normalizing it with the Softmax function. A T larger or smaller than makes the attention overly smooth or sharp, either way reducing the Transformer's 's ability to correctly predict the next variable's transformations.
We can then similarly write out the unconditional guided predictions as
A((L)T = (1 x) w) - a(( (1 + w)a wa
wH((((s
(1I)
where increasing either or |7 17-11 1| corresponds to stronger guidance.
Lastly, for both the conditional and unconditional cases, it is possible to assign a different guidance weight wi depending on the flow and position index t, i. We have preliminarily explored linearly increased w; as function of i, as in w; = T-1 w, and we have found this to achieve better sampling results w.r.t. FID than uniform guidance weights. We leave the thorough exploration of the optimal guidance schedule as future work.
16

---

Normalizing Flows are Capable Generative Models
Table Fréchet Inception Distance (FID) evaluation on Unonditional ImageNet 64x64. We denote the TARFLOW configuration in the format p.-.h.T-K-p...
Model
Type
FID 4
MFM (Pooladian et al., 2023)
Dift/FM
11.82
FM (Lipman al., 2023b)
Diff/FM
13,93
AGM (Chen et al., 2024)
Diff/IM
10.07
IC-GAN (Casanova al., 2021)
GAN
10,40
Self-sup GAN (Noroozi, 2020)
GAN
19.20
TARFLOW [2-768-6 8 8-V(0,, 0.05")] (Ours)
NF
18.42
C. Experimental details
Our models are implemented with PyTorch, and our experiments are conducted A 100 GPUs. We by default cast the model to bfloat16, which provides significant memory savings, with the exception of the likelihood task where we found that float32 is necessary to avoid numerical issues. All of our jobs are finished within 14 days of training, though we believe that the models should get better if trained longer. We summarize the hyperparameters for our best jobs in Table 7.
Table 7. Hyper parameters for the best performing model on cach task.
Task
Patch Size
Channels
Num Flows
Layers per Flow
Input Noise
Batch size
Epochs
Num GPUs
Uncond ImageNet 64x64 (likelihond)
768
8
8
24(0, Tæx
384
60
32
Uncond ImageNet 64x64 (gcocration)
2
768
8
8
[0 0.05
256
200
8
Cond ImageNet 64x64
1024
8
8
[[0., 0.05²)
768
320
32
Cond ImageNet 128x128
1024
8
[0 0.152)
320
32
Cond AFHQ 256x256
768
8
8
0.07%
256
4000
8
D. Inference Implementation
Although main focus in this paper has been on training capable generative models, it is still worth commenting on the sampling efficiency of our method. Sampling from a TARFLOW involves reversing a series of causal Transformers. Unlike the training model where the autoregressive flow can be computed in parallel with causal masks, the reverse step is inevitably sequential with respect to the sequence direction. In practice, we resort to a KV-cache based implementation, which is a standard practice in the context of LLMs, and we found that it greatly speeds up the sampling over a naive implementation. For instance, sampling from guided batch of 32 samples from the 2-768-8-8-2 8-8-8 0.05?) ImageNet 64x64 model takes about 2 minutes on a single A100 GPU. Although efficient sampling is not the focus of this work, we believe that there is great room for improvement in this regard, and leave it future work.
Another component in our sampling pipeline is the score based denoising step. The time of this step is equal to two forward model calls, which usually happens in matter of seconds. A practical bottleneck is that this step is more memory consuming than the flow reverse step, due to the need caching all intermediate activations for back propagation. In theory, this can be further alleviated by adopting techniques like gradient checkpointing, essentially trading time for memory.
D.1. Visualizing Sample Trajectory
Thanks to the residual style composition of TARFLOW, we can also visualize the generation process by reshaping each {2*} to the pixel space. We visualize two sampling sequences with the ImageNet 128x 128 model in Figure Interestingly, the sample trajectories highly resemble those from Diffusion model, in the sense that the initial noise gradually transformed into visible inputs though they are trained with completely diffferent objectives.
E. A Additional samples
Next we show more uncurated samples from four generation tasks, demonstrating the raw samples, guided samples as well as denoised samples in Figure 9, 10, 11 and 12.
17

---

Normalizing Flows are Capable Generative Models
(d) guidance (no guidance)
(a) guidance (no guidance)
(e) guidance 0.5, 1.2
guidance w 0.5, T 1.5
(b) guidance w = 2
guidance
1.2
(c) guidance ur
(h) guidance u = 1, T 1.5
Figure 7. Left: Varying guidance weight with the class conditional model on ImageNet 128x128, here show 4 samples from the ImageNet class 849 ("tcapot"); Right: Varying guidance weight and attention temperature for the uncondtional ImageNet 64x64 model.
Figure 8. From left to right, the sampling trajectory from the model on ImageNet 128x128 with 8 flow blocks, The visualization includes the final denoising step.
18

---

Normalizing Flows are Capable Generative Models
(a) guidance w 0 (no guidance), noisy
(b) guidance w 0 (no guidance). denoised
(c) guidance noisy
(d) guidance 2, denoised
Figure 9. Uncurated samples with a fixed set of initial noise from the model trained on conditional ImageNet 64x64.
19

---

Normalizing Flows are Capable Generative Models
(a) guidance w 0 (no guidance), noisy
(b) guidance w 0 (no guidance). denoised
(c) guidance 0.15, 0.2, noisy
(d) guidance 0.15, 0.2, denoised
Figure 10. Uncurated samples with fixed set of initial noise from the model trained on unconditional ImageNet 64x64.
20

---

Normalizing Flows are Capable Generative Models
(a) guidance w = (no guidance), noisy
(b) guidance w
0 (no guidance), denoised
(c) guidance = 2.5, noisy
(d) guidance 2.5, denoised
Figure II. Uncurated samples with fixed set of initial noise from the model trained on conditional ImageNet 128
21

---

Normalizing Flows are Capable Generative Models
(a) guidance w = (no guidance), noisy
(b) guidance w 0 (no guidance), denoised
(c) guidance 2, noisy
(d) guidance denoised
Figure 12, Uncurated samples with fixed set of initial noise from the model trained on conditional AFHQ 256x256.
22


Reviews:
[{"review_id": "JAVzAOStdp", "reviewer": "Reviewer_M2FB", "summary": "This paper scales up masked autoregressive flow (MAF) with powerful transformer architecture along with several techniques such as classifier guidance, noise augmentation and achieve good performance on many datasets including high resolution AFHQ and multimodal dataset Imagenet.\n\n## update after rebuttal\nThe rebuttal has resolved my concerns. After careful consideration, I decided to vote Accept for this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our contributions and we answer each of the questions below. \n\nFirst of all, please note that we do have a supplementary material where we include more experimental settings, related work and results which we believe might be interesting to the reviewer. \n\n\n## Q: The sampling time is very slow compared to standard autoregressive models.\n\nA:  Given a sequence length N (ie, number of patches), number of AR flows T and the Transformer depth for each flow L, the inference cost of TarFlow evaluates to $O(TLN^2)$. This cost will be equivalent to that of an autoregressive model on the same sequence but with a depth of $TL$. Therefore, the inference speed comparison between TarFlow and an AR model boils down to the combined depth, and the two families of models actually have the same inference time complexity given the same depth budget. \n\n\n## Q: The architecture of model is quite heavy 8x8 = 64 transformer blocks, which is about 2 time larger than DiT XL\n\nA: Related to the previous question, our design philosophy has been to train strong models and prove that NFs are capable learning objectives, whereas we did not emphasize on reducing the total depth. As for the comparison to DiT, we argue that they are also not directly comparable, due to two reasons. 1. DiT conducts experiments in the latent setting, which is known to simplify the modeling difficulty, whereas TarFlows are pixel based models. 2. The inference mode of DiTs is actually much deeper than TarFlow, due to the need of calling multiple NFEs. \n\n\n## Q: The method is not new and a little bit limited novelty since classifier free guidance and data augmentation are existed techniques\n\nA: We respectfully disagree with the reviewer’s assessment on novelty. It is indeed true that all the essential ingredients in this work has been previously established techniques, this includes MAF, Transformers, noise augmentation, denoising and guidance. However, in the context of normalizing flows, the exact recipe of using them has been unknown, and as the reviewer already acknowledged, we are the first work to show the full potential of correctly combining them together. The fact that these individual techniques are standard only strengthens the simplicity aspect of our work, and it should not be treated as a penalty for novelty. In fact, the same argument can be made for many groundbreaking works — for example, denoising autoencoders are well studied subjects [1], but correctly applying it to generative modeling [2, 3] is still highly novel and it gives rise to the diffusion and score based generative model revolution. \n\n\n## Q: Is the anyway to develop faster inference model based on TarFlow ?\n\nA:  Although we did not focus on inference speed in this work, we are excited by several possibilities of improving it in the future work. First of all, there is a large body of literature (eg, speculative decoding [4]) dedicated to speeding up the inference of autoregressive models, which TarFlow can already benefit from, due to its autoregressive architecture. Another promising direction is distillation, which has achieved great success in diffusion models [5]. And we believe that distillation should be naturally compatible with TarFlow, and arguably more so than with diffusion models, as we have an explicit bijective mapping between an input x and noise z. \n\n# References\n[1] Extracting and composing robust features with denoising autoencoders, Vincent et al, ICML 2008\n\n[2] Denoising Diffusion Probabilistic Models, Ho et al, NeurIPS 2020\n\n[3] Generative Modeling by Estimating Gradients of the Data Distribution, Song & Ermon, NeurIPS 2019\n\n[4] Fast inference from transformers via speculative decoding, Leviathan et al, ICML 2023\n\n[5] Progressive Distillation for Fast Sampling of Diffusion Models, Salimans & Ho, ICLR 2023", "claims_and_evidence": "Yes, paper provides clear representation and convincing evidence to all claims", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria are standard for generative model.", "theoretical_claims": "There is no theoretical claim in this paper.", "experimental_designs_or_analyses": "The experimental design is well-conducted and support all the paper's claim.", "relation_to_broader_scientific_literature": "This paper scale up MAF, an existing technique to train normalizing flow to show the scalability of normalizing flow class of generative models. The paper's result is quite interesting and promising.", "essential_references_not_discussed": "No, I find the reference is sufficient", "other_strengths_and_weaknesses": "**Strength**\n\nThe paper presentation is clear and easy to understand.\n\nThe experiment shows promising potential for normalizing flow to scale up, in comparison to diffusion, autoregressive and GAN models\n\nThe ablation is fully provided\n\nSeveral proposed techniques such as noisy augmentation, transformer backbone for MAF and cfg are first-time investigated in MAF\n\n**Weakness**:\n\n\nThe sampling time is very slow compared to standard autoregressive models.\n\nThe architecture of model is quite heavy 8x8 = 64 transformer blocks, which is about 2 time larger than DiT XL\n\nThe method is not new and a little bit limited novelty since classifier free guidance and data augmentation are existed techniques", "other_comments_or_suggestions": "No", "questions_for_authors": "Is the anyway to develop faster inference model based on TarFlow ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "X09CviUMxt", "reviewer": "Reviewer_jjSB", "summary": "This paper proposes to integrate visual transformer architecture into Real NVP Normalizing Flows. Over the past years normalizing flows has been inferior to other types of generative models; particularly when compared to diffusion models. This paper claims that the reason for that is the design limitation of normalizing flows, therefore by using transformer backbones with causal attention mechanisms they are able to improve the generation performance and enhance the scalability of the model. \n\nThe approach shares similarity with Masked Autoregressive Flows (MAF), where it is extended to patches rather than pixels. Additionally, the input of the model is injected with small noise for smoothing (in probability terms) the discrete data distribution. Which in turn, requires an extra final denoising step, performed using a score based model.\n\nThe paper also presents a guidance mechanism, similar to the one used in diffusion models. Classifier free guidance in particular. Enabling the generation to be controlled externally.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review, and please see our responses below. \n\n## Q:  Scalability claim\n\nA: Our claim about the scalability of TarFlow is within the context of different model sizes and training FLOPs on a given dataset, which is supported by evidence in Sec 3.5 and Figure 6. Moreover, for the comparison of ImageNet 128x128 vs 64x64, we do not agree that the former is worse. Note that the absolute FID numbers are not comparable across datasets, but it is pretty clear that our ImageNet 128x128 samples are visually better than those on 64x64 (Figure 11 vs Figure 9). \n\nIn addition, we have conducted further experiments on ImageNet 256x256 with a model of ~1.4B parameters, and we are able to achieve very competitive results, with a 50K sample FID of 4.00 on pixel space. This number again places TarFlow in a diffusion dominated regime, see the table below as a comparison.\n\n| Method (modeling pixels) | ImageNet 256x256 FID 50K |\n| --------  | ------- |\n| Simple Diffusion | 3.75|\n| ADM-G | 4.59 |\n| RIN | 4.51 |\n| TarFlow | 4.00 |\n\n## Q: Denoising & Section 2.5\nA: We believe that the reviewer has a serious misunderstanding of our contributions. To clarify, denoising is performed with the same TarFlow model we train, and **we do not use a separate denoising network**. The exact recipe is explained in Equation 8, where $\\log p_{model}(y)$ corresponds to the negative training loss of the TarFlow model. This also explains why Section 2.5 is a critical piece of our method, as it is not previously clear that NF models empirically give rise to accurate score estimates, making it a suitable choice for denoising in line with the Tweedie’s formula. Note that this point is also correctly recognized and acknowledged by Reviewer EWAZ as a strength of our paper.\n\nAs for speed, the denoising step adds a minimum overhead, due to its parallel nature. For actual measurements, we benchmarked the model we used for ablations, ie, the one from Figure 4, it takes 13.5 seconds to generate a noisy batch of examples, and 0.14s to denoise them. \n\n## Q: Results without noise injection\nA: We assume that the reviewer is interested in the dequantization uniform noise setting, which is a standard in the NF literature [1] — and also the simplest way to make modeling pixels a valid density estimation problem (see, eg, Sec 3.1 of [3] for a discussion on this). As noted in the first paragraph of Section 3.3, this setting has poor numerical stability which makes it difficult to compare with other settings fairly. Nonetheless, during the rebuttal period we managed to train such a model by using fp32 instead of our default bf16, and performed 50K FID evaluation by skipping many samples with NaNs. See results in the table below.\n\n## Q: Non-causal transformer\nA: First, the causal architecture is a necessary component for implementing the AR flow. More specifically, it allows us to compute the determinant analytically, and also invert the transformation explicitly, due to the Jacobian of the causal transformation being lower triangular. Both conditions will break if one directly applies a non-causal Transformer. That being said, the closest baseline we can think of that’s using a non-causal Transformer is to follow the channel coupling design from [1].  We performed such an ablation, and the result is shown below.\n\n## Q: Normalizing Flows other than Real NVP\nA: We have also trained a volume preserving version, by enforcing the logdet terms to be zero. See results below.\n\n### Summary of additional ablations, where the baseline experimental setting follows that of Figure 4. All variants have inferior results than the default TarFlow setting.\n\n| Variant   | ImageNet 64x64 FID 50K, cfg = 0 | ImageNet 64x64 FID 50K, cfg = 2|\n| --------  | ------- | ------- |\n| TarFlow default |25.3| 5.7 |\n| dequantization uniform noise | 43.6 | 21.9 |\n| non-causal architecture with channel coupling | 50.3 | 20.4 |\n| volume preserving   | 81.5 | 51.0 |\n\n## Q: Novelty\nA: We believe that TarFlow has significant novelty compared to [2]. The similarity is that they both apply Transformers to MAF. However, it is important to note that [2] does not provide a scalable recipe for high dimensional inputs. In particular, [2] only considers models with a single flow, whereas TarFlow stacks multiple flows with alternating directions, and show that this can be trained well. As shown in Figure 6(b) of our paper, using one flow (T=1) significantly limits the model’s capacity and it becomes a degenerated model for images. The other obvious difference lies in our usage of noise augmented training, denoising and guidance. All these critical pieces are completely missing in [2].\n\n# References\n[1] Density estimation using Real NVP, Dinh et al, ICLR 2017\n\n[2] Transformer Neural Autoregressive Flows, Patacchiola et al., 2024\n\n[3] Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design, Ho et al, ICML 2019", "claims_and_evidence": "- The paper claims and shows empirically that Normalizing Flows indeed need a more strong deep network architecture to be able to compete with state-of-the-art diffusion models. \n\nThe paper claims that the approach is scalable, similar to what has been done in diffusion models, however:\n\n- The approach is not tested on ImageNet $256\\times 256$, it is only trained on AFHQ at this resolution, which is a relatively homogeneous dataset, and there aren't any quantitative metrics that support this claim.\n\n- Comparing the reported results at $128\\times 128$ actually contradicts with the scalability claim. In contrast to $64\\times 64$, where the FID score is inline with the diffusion scores, the result on $128\\times 128$ is actually worse.", "methods_and_evaluation_criteria": "Strengths:\n- The approach is evaluated properly on different datasets.\n\n- The results shown in the paper are promising, competitive with state-of-the-art diffusion models.\n\nWeaknesses:\n\n- The paper is missing an ablation study on the effect of the denoiser choice.\n\n- The results without the noise injection are not shown in the paper, especially that the paper emphasizes the importance of it. I think it is very important to highlight the drawbacks of not adding the noise more clearly (diversity, mode collapse, textures, etc.).\n\n- The paper shows the results only for causal transformer network, I think there should be an additional ablation study that demonstrate why this choice is preferable over non-causal transformers.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The design scheme is straightforward, the paper uses transformer based architecture to predict the coupling of the bijection in an auto-regressive manner.", "relation_to_broader_scientific_literature": "The paper highlights the advantage of using capable architectures, transformer networks in particular,  in Normalizing Flows. Prior flow based models struggled to compete with state-of-the-art diffusion models, and this work offers a very competitive alternative (for lower resolutions at least).", "essential_references_not_discussed": "Although it is mentioned in the related work section, but it would be much better to discuss the main differences between this method and the one proposed in Patacchiola et al., 2024.", "other_strengths_and_weaknesses": "Other strengths:\n\n- The paper is clear and well written.\n\nOther weaknesses:\n\n- The results without the noise injection are not shown in the paper, especially that the paper emphasizes the importance of it. I think it is very important to highlight the drawbacks of not adding the noise more clearly (diversity, mode collapse, textures, etc.).\n\n- Obtaining high-fidelity generation results requires using a denoising network for post-processing the output.\n\n- The novelty of the approach is relatively limited, particularly compared to Patacchiola et al., 2024.\n\n\n- Normalizing Flows other than Real NVP are not examined.", "other_comments_or_suggestions": "- Section 2.5 can be reduced, there is no need to provide redundant details about Tweedie’s relation of score and denoising if your target is just to denoise the final image.", "questions_for_authors": "- What is the denoiser architecture used in the final stage? \n\n- What is the overhead and added computational cost of using a denoising network?\n\n- How does not adding noise at all affects the results? in terms of diversity, mode collapse, and quantitatively (FID).\n\n - In terms of computations, how many \n\n- How important is the causality part of the method? Does the method also work if the transformer was not causal? How does that affect the results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UmiHeZzctq", "reviewer": "Reviewer_EWAZ", "summary": "The paper presents a normalizing flow architecture and training pipeline for image generation that significantly improves previous normalizing flow models and obtains competitive performance when compared with diffusion models and GANs. The architecture uses a masked transformer backbone to implement RealVPN-type partitioned layers. Training is performed on noised images with a fixed level of noise, which is then removed at inference time using Tweedie’s formula, in what essentially is a single step of denoising diffusion. Importantly, this is done using the existing network without additional training. The paper also introduces an interesting guidance scheme inspired by classifier-free guidance but at the level of the attention layers.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our contributions and we agree most of your assessments. \n\n## Q:  Limitations of likelihood-based models\n\nA: An interesting point of discussion the reviewer brought up is the fundamental limitations of likelihood-based models. We agree with the reviewer that likelihood method, especially density estimation such as NFs, can be ill behaved on real data that’s distributed on a narrow manifold. And we also agree that TarFlow needing input noise resonates with this view, which put in other words, suggests that density estimation is indeed a different task than generative modeling. We also believe that this observation is consistent with the findings in diffusion models, whereas the reweighting the likelihood based loss function is beneficial for sampling quality [1, 2].\n\n\n## Q: Significance of the BPD benchmark\n\nA: A related question is whether one should value the strong BPD results that TarFlow achieves. We believe so, based on a subtle yet interesting point detailed below. Note that the BPD metric evaluates the discrete probability of quantized pixels, rather than the continuous density of real pixel values. This is a key difference because, unlike continuous density, discrete probability can always be faithfully evaluated even when the input has much lower intrinsic dimension (eg, concatenating a constant dimension to a discrete variable multiplies the joint discrete probability by 1, instead of infinity). When using a density estimation method to evaluate BPD, one would first inflate the discrete values $\\tilde{x}$ to a local hyper cube $C(\\tilde{x})$ that is disjoint from other discrete points (which is the case for the dequantization uniform noise), and convert the density model to discrete probability via integration, as in $\\tilde{p}(\\tilde{x}) = \\int_{x \\in C(\\tilde{x})} p_{model}(x)dx$ (this is alo explained in the first paragraph of Section 2.4). Due to the same reasoning, the discrete probability result (hence BPD) is comparable among difference families of models, including discrete modeling methods using autoregressive models and continuous ones such as diffusion and NFs. In addition, the BPD metric has a concrete grounding itself, which essentially translates to the theoretical lower bounds for lossless compression of the target distribution [3], which is another way to prove that BPD does not degenerate. Therefore, we believe that TarFlow achieving SoTA BPD among different modeling methods is a strong indication of its raw modeling capacity.\n\n\n## Q: temperature based guidance for LLMs\n\nA: We'd like first to confirm that we did come up with the attention based temperature guidance on our own, and we are not aware of a similar idea being deployed in other context such as LLMs. Like the reviewer, we are also intrigued by its generality and we believe that it does have potential to be applied to all Transformer based generative models as well. For LLMs, specifically, we speculate that guidance had received relatively little exploration likely due to the prevalence of finetuning, and it would indeed be interesting future work to further investigate its compatibility with our temperature based guidance. \n\n# References\n[1] Denoising Diffusion Probabilistic Models, Ho et al, NeurIPS 2020\n\n[2] Variational Diffusion Models, Kingma et al, NeurIPS 2021\n\n[3] IDF++: Analyzing and Improving Integer Discrete Flows for Lossless Compression, Berg et al, ICLR 2021", "claims_and_evidence": "The claims are convincing and well supported by the experiments, with several ablation studies elucidating the relative contributions of the novel ideas.\nFor reasons explained below, I do not think that the exceptional likelihood performance is particularly interesting, since the data approximately lives in a lower-dimensional manifold.", "methods_and_evaluation_criteria": "The methods and evaluation metrics are in line with the modern literature. Several ideas from transformers and diffusion models are integrated elegantly in the normalizing flow framework. The authors show a good mastery of the modern generative modeling landscape.", "theoretical_claims": "The paper does not make precise theoretical claims. The formulas and derivations are solid.", "experimental_designs_or_analyses": "The experiments follow standard image generation and evaluation approaches widely used in the image generation literature.", "relation_to_broader_scientific_literature": "The paper offers an elegant integration of several modern generative modeling ideas\"\n1) It combines the general RealNVP framework with a masked transformer architecture.\n2) It uses a guidance scheme inspired by classifier-free guidance in diffusion models. However, here the guidance is used within the different layers opf the normalizing flow architecture.\n3) It uses a denoising technique that uses the basic formulas of generative diffusion models.", "essential_references_not_discussed": "The literature coverage concenring normalizing flows, diffusion models and tautoregressive models is appropriate.", "other_strengths_and_weaknesses": "Strengths:\nIt is very nice to see modern work on normalizing flows, and I think that the authors did a great job in upgrading these models to near the current SOTA using both ideas from transformers and diffusion models.\n\nThe main strengths of the paper are:\n\nThe proposed architecture is both elegant and powerful and I think it could offer a starting point for a renaissance in normalizing-flow research even outside computer vision. \nThe guidance scheme, if original, is very interesting and it could be applied to all sorts of transformer architectures.\nWhile it is somewhat disappointing that the model needs to work on noised data, I appreciated the elegance of the denoising solution inspired by diffusion theory. I really appreciate how the authors are borrowing elegant ideas from different approaches while integrating them elegantly within the normalizing flow framework.\nWeaknesses:\nWhile the results are convincing, the paper is also a clear indication that normalizing flows, even with very modern components, fail when trained to generate noiseless images.\n I think that this simply reflects a fundamental problem with likelihood-based models such as NFs when trained on data such as images that are supported on a lower dimensional manifold-like structure. The issue is that the likelihood diverges if the model approaches the correct support of the data, regardless of how well it is fitting the distribution restricted to the manifold. This phenomenon is known as ‘manifold overfitting’ [CITE]. In the case of NFs on noiseless images, the optimum of the loss is situated near or at a singular point of non-invertibility, which leads to unstable training. So said, this is not really a weakness of the paper, which does a good job at characterizing this behavior empirically. \n\nDue to manifold overfitting and divergence of the likelihood, I do not think that the likelihood values are of particular interest, since on manifolds a bad generative model can reach arbitrarily high likelihood simply by fitting some section of the support of the data. In fact, in this case we see exceptionally high likelihood values together with FIDs that are good but not exceptional. Again, I do not consider this as a real weakness of the paper, but I do not put much weight in the exceptionally good BPD numbers.", "other_comments_or_suggestions": "None", "questions_for_authors": "I am really intrigued by your guidance method. Are you sure that it is not currently used in the LLM literature? I do not know the literature well enough to know. If not, it would be very interesting to use it in more conventional transformers as well.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "INUAYb62Vu", "reviewer": "Reviewer_fuKH", "summary": "The paper proposes a new architecture called TarFlow, which is a Transformer-based variant of Masked Autoregressive Flows (MAFs). TarFlow achieves a high-performance normalizing flow (NF) model by stacking autoregressive Transformer blocks on image patches and alternating the autoregressive direction between layers. Additionally, the paper introduces three techniques to improve sample quality: Gaussian noise augmentation, a post-training score-based denoising technique, and efficient guidance recipes for both the class conditional and unconditional models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Q: Motivation and difference to flow matching\n\nA: We’d like to clarify that the exact notion of Normalizing Flow (NF) we consider is here fundamentally different from the modern notion Flow Matching (FM) method. In our paper, we follow the conventional notation of NFs that exclusively refer to a maximum likelihood method that uses the change of variable technique of probability [1, 2, 3]. One of their distinct properties is that the training loss can be **exactly computed** for each training example. Flow matching [4] on the other hand, denotes a training technique that can be considered a variant of diffusion models, see [5] for such illustration, which does not follow a maximum likelihood objective. Also, the training loss of FM relies on a **stochastic approximation** of an expectation over time steps, and over the noise within each step. Moreover, it was shown that FM’s training objective follows a modified variational lower bound of likelihood [8], which again is fundamentally different from NFs's exact likelihood nature. In fact, this difference is also documented in [4], see the first paragraph of Section 5. The confusion may stem from the reference to continuous normalizing flow (CNF) in [4], which considers a **generalized notion of NFs** which deterministically map noise to data, regardless their training objectives. However, please note that the CNF here is mathematically equivalent to the probability flow [7], or ODE/DDIM style inference path in diffusion models [6]. This is also the reason why we have categorized all diffusion model and flow matching results under the term Diff/FM in our comparisons, see eg Table 2. \n\nWe hope this clarifies the difference, and we will be happy to make this more explicit in our paper, constraining the scope of our discussion of NFs to methods that directly follow the MLE objective with the change of variable formula.\n\n## Q: Label conditioning implementation\n\nA: We adopt a simple implementation of label conditioning, where we add the label embedding directly to the position embeddings in each flow block. During guidance, the unconditional predictions are obtained by averaging the label embeddings of all classes and adding it to the position embeddings.\n\n\n## Q: Introduction to BPD\n\nA: Thank you for the suggestion, we are happy to provide more context on it. In simple terms, BPD measures the average log probability for all the pixels of an image, where the probability is computed as a discrete one among 256 possible pixel values. \n\n\n## Q: Results worse than diffusion models\n\nA: We agree that the current results of TarFlow is still lacking behind the best tuned diffusion models. We argue that for any new type of method, it usually takes time for it to be collectively improved by a community before it reaches the SoTA performance, and this was definitely the case for diffusion models too. We look forward to working with the generative modeling community together to further improve the upper bound of normalizing flow methods. \n\n\n## Q:  Comparison with other flow and autoregressive model\n\nA: Our comparisons have been focusing on mainstream continuous modeling method, and we have identified GANs, diffusion/flow matching models as the representative categories. Also note that we did not include traditional NF baselines in the FID comparisons as they were generally under performing and we were not able find comparable FID results reported in the literature. Following the reviewer's suggestion, we will also include representative baselines from Autoregressive models. Also, we would be happy to include more baselines for comparison if the reviewer is able to provide more specific references. \n\n## Q: Figure 4 not clear\n\nA:  We apologize for the visualization clarity, but the main cause for it is that the noise we apply is indeed very small and it does require a closer look to recognize the effect of denoising. We refer the review to more visualizations in the supplementary material, ie Figure 9-12, which should allow one to observe the effect of noise more clearly.\n\n# References\n[1] Density estimation by dual ascent of the log-likelihood, Tabak et al, Communications in Mathematical Sciences, 2010\n\n[2] Variational inference with normalizing flows, Rezende & Mohamed, ICML 2015\n\n[3] Nice: Non-linear independent components estimation, Dinh et al, ICLR 2014\n\n[4] Flow Matching for Generative Modeling, Lipman et al, ICLR 2023\n\n[5] Diffusion Meets Flow Matching: Two Sides of the Same Coin, Gao et al, ICLR Blogposts 2025\n\n[6] Denoising Diffusion Implicit Models, Song et al, ICLR 2021\n\n[7] Score-Based Generative Modeling through Stochastic Differential Equations, Song et al, ICLR 2021\n\n[8] Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation, Kingma & Gao, NeurIPS 2023", "claims_and_evidence": "I think the claims regarding the normalizing flow part is solid, but I'm not so persuaded in terms of motivation part, which claims that the normalizing flow have seen limited practical adoption. Shouldn't flow matching techniques has already been widely applied? Can author address the foundamental differences of the proposed TarFLow and current flow matching methods?", "methods_and_evaluation_criteria": "1. In the section that introduces guidance technique, I'm wondering how the class label conditioning is added in the transformer architecture, through input concatenation or cross-attention?\n\n2. The tasks and benchmark datasets make sense, but I think the author should give a little bit introduction to the less commonly-used evaluation criteria like Bits per dim (BPD) evaluation in table 2.", "theoretical_claims": "This paper does not propose theoretical claims that needs mathematical derivation or proof. Its contribution mostly lies in the design of Transformer-based normalizing flow architecture.", "experimental_designs_or_analyses": "1. As mentioned before, I think this paper lacks comparison and discussion with flow matching method, which should be closely related to the realm of normalizing flow.\n\n2. From the quantitative experiments, the performance of the TarFlow is not comparable with diffusion-based method, can the author provide analysis regarding this phenomenon?\n\n3. In generation quantitative experiments, it lacks comparison with other flow and autoregressive model, please add more baseline results.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "One subtle suggestion is that the top visualization part of figure 4 is not clear enough. It's hard to distinguish the difference between the raw samples and denoised samples.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper has solid consensus for acceptance. It tackles an interesting problem of making normalizing flows truly performant models, rather than just density estimators.
Decision: Accept (oral)