Boosting Summarization with Normalizing Flows and Aggressive Training
Authors: Yu Yang
Xiaotong Shen
Venue: EMNLP 2023 Main (2023)
Keywords: summarization
normalizing flows
posterior collapse
aggressive training
TL;DR: This paper proposes a normalizing flows-based variational encoder-decoder framework for Transformer-based summarization models.

Abstract: This paper presents FlowSUM, a normalizing flows-based variational encoder-decoder framework for Transformer-based summarization. Our approach tackles two primary challenges in variational summarization: insufficient semantic information in latent representations and posterior collapse during training. To address these challenges, we employ normalizing flows to enable flexible latent posterior modeling, and we propose a controlled alternate aggressive training (CAAT) strategy with an improved gate mechanism. Experimental results show that FlowSUM significantly enhances the quality of generated summaries and unleashes the potential for knowledge distillation with minimal impact on inference time. Furthermore, we investigate the issue of posterior collapse in normalizing flows and analyze how the summary quality is affected by the training strategy, gate initialization, and the type and number of normalizing flows used, offering valuable insights for future research.

Boosting Summarization with Normalizing Flows and Aggressive Training
Yu Yang
Xiaotong Shen University of Minnesota xshen@umn. edu
University of Minnesota yang6367@umn. edu
Abstract
these issues by introducing uncertainty in predic- tions through learning probability distribution over latent variables. variational model enables diverse text generation (Du et al., 2022), smoother output spaces, and semantically meaningful latent codes (Wang et al., 2019) that guide the generation of coherent and informative summaries.
This paper presents FlowSUM, normaliz- ing flows-based variational encoder-decoder framework for Transformer-based summariza- tion. Our approach tackles two primary chal- lenges variational summarization: insuffi- cient semantic information in latent representa- tions and posterior collapse during training. To address these challenges, we employ normal- izing flows to enable flexible latent posterior modeling, and we propose a controlled alter- nate aggressive training (CAAT) strategy with an improved gate mechanism. Experimental results show that FlowSUM significantly en- hances the quality of generated summaries and unleashes the potential for knowledge distilla- tion with minimal impact on inference time. Furthermore, investigate the issue of pos- terior collapse in normalizing flows and ana- lyze how the summary quality affected by the training strategy, gate initialization, and the type and number of normalizing flows used, offering valuable insights for future research.
Nonetheless, existing variational models have not fully achieved the aforementioned desirable properties due two main challenges. Firstly, the semantic information in the source text may pos- sess complex structure. However, since introduc- ing latent variables complicates parameter estima- tion, many current models (Fu et al., 2020; Zheng et al., 2020) represent latent codes using Gaussian distribution, which is insufficient for capturing the intricacies of the latent space and could potentially reduce model performance. To enrich latent distri- butions, researchers suggest replacing the highly re- stricted isotropic Gaussian with normalizing flows (Rezende and Mohamed, 2015). Normalizing flows can generate complex distributions while preserv- ing density in an analytical form, and they have been integrated into variational autoencoder (VAE) (Kingma and Welling, 2014; Rezende et al., 2014) and variational encoder-decoder (VED) (Serban et al., 2017: Zhou and Neubig, 2017) frameworks to better approximate the latent posterior. This ap- proach has found application in various domains, including text generation (Wang et al., 2019), neu- ral machine translation (Setiawan et al., 2020), and dialogue generation (Luo and Chien, 2021). De- spite this progress, the operating characteristics of normalizing flows summarization tasks have yet to be investigated.
1 Introduction
Abstractive summarization (See et al., 2017; Paulus et al., 2018; Wang et al.. 2018) aims to gener- ate summaries by rephrasing or introducing novel words to capture the most salient information in the source text. Many abstractive summarization models (Liu and Lapata, 2019; Zhang et al., 2020a; Rothe al., 2020; Raffel et al., 2020) are based on the Transformers architecture (Vaswani et al., 2017) and have consistently produced ofate-of-the- art summarization quality. However, issues such exposure bias (Ranzato et al., 2016; Qi et al., 2020), lack of text generation diversity (Holtzman et al., 2020), and insufficient capturing of semantic information (Reimers and Gurevych, 2019; Wang al., 2020) remain.
Secondly, reported by previous studies (Bow- man al., 2016; Kingma al., 2016; Chen et al., 2017), variational models tend to experience pos- terior collapse during training, which occurs when the KL term vanishes to zero, indicating that the model fails to learn meaningful latent codes. This
Variational models have gained increasing re- search interest (Zhang et al., 2016; Su et al., 2018; Wang et al., 2019; Fu et al., 2020) as they address

---

problem becomes more severe when modeling dis- crete data with strong auto-regressive decoder (He et al., 2019), which the case for Transformer- based summarization models. To resolve this is- sue, several solutions have been proposed, such employing less auto-regressive decoder net- work (Yang et al., 2017; Semeniuta et al., 2017;
This article consists of five sections. Section 2 provides an overview of normalizing flows, VED, and a summary of related studies. Section 3 de- scribes the proposed model architecture and the training strategies employed. Section 4 presents the experimental setup and results, and Section 5 concludes the paper with some discussions.
Shen al., 2018 modifying the training ob- jective (Zhao et al., 2017; Tolstikhin et al., 2018;
2 Backgrounds
Prokhorov al., 2019), and proposing new train- ing strategies (Kim al., 2018; He et al., 2019). However, most existing work focuses on the VAE framework with Gaussian latent distribution, yet limited work considers the VED framework with normalizing flows. In particular, two questions re- main unclear: (1) when the latent distribution is modeled by normalizing flows, does the posterior collapse problem still exist? (2) when posterior collapse exists, what are the appropriate strategies to achieve good summarization quality within the VED framework?
2.1 Normalizing Flows
Normalizing flows (NF) (Rezende and Mohamed, 2015) is a type of generative model that has gained popularity in recent years. The fundamental idea in- volves mapping simple probability density (e.g., Gaussian) to a more complex one through se- ries of invertible transformations. One of the key advantages of NF is that it allows for exact likeli- hood evaluations, which is crucial for many appli- cations such as density estimation (Papamakarios et al., 2017), data generation (Tran et al., 2019), and variational inference (Kingma et al., 2016). A flow-based model consists of two components:
This paper introduces FlowSUM' normaliz- ing flows-based VED framework for Transformer- based summarization, along with controlled al- ternate aggressive training (CAAT) strategy and refined gate mechanism resolve the two challeng- ing issues. Our contributions include:
base distribution (u(u) and a transformation f(-) R ID. where must be invertible and both f and f-1 must be differentiable. Let f(u) where ~ Pu(u), then the density of can be obtained via a change of variables (Bo- gachev, 2007):
1. We employ normalizing flows enrich the latent posterior distribution and integrate the latent code into Transformer-baat models in a plug-and-play manner, demonstrating its effectiveness through extensive experiments.
Px Px(x) Pu (u) det [1 = =P(s '(x)) |det Jx)
(1)
In this paper, we examine several NFs, including planar flows (Rezende and Mohamed, 2015), radial flows (Rezende and Mohamed, 2015), Sylvester flows (van den Berg et al., 2018), real-valued non-volume preserving (RealNVP) transformation (Dinh et al., 2017), inverse autoregressive flow (IAF) (Kingma er al., 2016), rational-quadratic neu- ral spline flows (RQNSF) (Durkan et al., 2019), and rational-lincar neural spline flows (RLNSF) (Dolatabadi et al., 2020). We delegate the detailed discussion of transformation and invertibility to Ap- pendix J. Throughout the paper, for each type, we compose K layers of transformation fK ......o0 (1(.,, which remains invertible and differentiable.
2. We propose a controlled alternate aggressive training strategy and refined gate mechanism to mitigate the posterior collapse problem and improve training efficacy.
3. Our findings suggest that FlowSUM facilitates knowledge distillation while having a negligi- ble effect on inference time, implying normal- izing flows' potential for transferring knowl- edge from advanced large language models.
4.
We investigate the posterior collapse problem for different normalizing flows and examine how the quality of summary is impacted by the training strategy, gate initialization, and the type and depth of normalizing flows.
2.2 Variational ChcodersDecoderr
Variational encoder-decoders (VEDs) (Zhang al., 2016; Serban et al., 2017: Zhou and Neubig, 2017; Shen et al., 2018b), which can be seen as an exten- sion of variational autoencoders (VAEs) (Kingma
'Code is available at yuyangstat/Flowsum
com/

---

and Welling, 2014; Rezende et al., 2014), have been widely used to understand the conditional data generation process. Given an input æ, the framework posits the existence of a latent variable 2 ~ p(= k; Ø), and the generation of y relies on p(y| 2; 0). With this premise, the conditional data generation can be formulated as in Eq. 2.
for instance, BART (Lewis et al., 2020), MASS (Song et al., 2019), UniLM (Dong et al., 2019). ProphetNet (Qi et al., 2020), and T3 (Raffel et al., 2020). In addition, Zhang et al. (2020a) proposed PEGASUS with pre-training objective tailored for abstractive summarization, achieving signifi- cant improvements across multiple datasets.
p(y 2; 0) - / p(z x; 0)p(v 2, 02
2.3.2 Variational Summarization
Variational summarization models come in two dif- ferent flavors: unsupervised and supervised. In the unsupervised domain, researchers commonly uti- lize variational autoencoders in conjunction with specific control mechanisms for summary genera- tion, as exemplified by prior work such as Schu- mann (2018); Chu and Liu (2019); Brazinskas et al. (2020). In the supervised realm, there are generally two primary approaches. The first ap- proach models the conditional probability of the target sentences p(y x)) in Eq. 2, whereas the second approach models the joint probabil- ity of the source and target sentences p(r, y) with I p(=)p(r) =))(v | 2, 2,2) Our model belongs to the first category, akin to prior studies like Setiawan et al. (2020); Fu et al. (2020). In contrast, other works, including Zheng et al. (2020); Nguyen et al. (2021); Zou et al. (2021), adopt the second type by jointly modeling topics and sequence-to-sequuee generation. Most of them assume a simple Gaus- sian latent prior, except for Nguyen et al. (2021), which employs normalizing flows to model neural topic models and enrich global semantics. How- ever, they did not specify the choice of normalizing flows and how they addressed posterior collapse. To the best of our knowledge, there remains limited research on the application of normalizing flows in variational summarization models and their operat- ing characteristics.
Since the marginal ply zy Φ, B) is intractable, we employ variational inference to estimate the param- eters. This involves maximizing the evidence lower bound (ELBO), a surrogate of the log-likelihood, as defined in Eq. 3. The underlying idea is to propose a parameterized distribution g(= 2, U: vb), known the variational posterior, to approximate the true posterior distribution p(= 2. y; Φ, 8)- The greater the flexibility in q(= xz, y; ()), the better the ap- proximation, and the more effective the surrogate ELBO becomes. See more details in Appendix B.
ELBOvED
E pogplyl 2; KL(((> (.llpl [)) v(simam))
(3)
For summarization, we parameterize p(y x, 2; encoder-decoder model that generates sum- maries conditioned on the input text and latent code.
2.3
Related Work
2.3.1
Transformer-hased Summarization Models
Transformer-batt models equipped with pre- training and fine-tuning techniques have enjoyed significant success in many NLP tasks, including text summarization. Liu and Lapata (2019) pro- posed BertSUM for extractive and abstractive tasks, utilizing the pre-trained BERT encoder (Devlin al., 2019). To better align the pre-traine pre-trained en- coder for document understanding with the decoder trained from scratch for text generation, Rothe et al. (2020) demonstrated the effectiveness of leverag- ing pre-trained BERT (Devlin et al., 2019), GPT-2 (Radford et al., 2019), and RoBERTa (Liu et al., 2019) checkpoints to build sequence-to-scquenec (S2S) models for tasks including summarization. Another approach is to address both document un- derstanding and generation in a unified framework by first pre-training some general-purpout S2S models and then fine-tuning on downstream tasks,
3 Normalizing Flows Enhanced Summarization Model
3.1
FlowSUM Model Architecture
As illustrated in Fig. 1, FlowSUM consists of three components: an NF latent module, Transformer- based encoder-decoder, and refined gate mecha- nism. The NF latent module focuses modeling the variational posterior q(2 xr, ys v)), whereas the encoder-decoder, combined with the refined gate, models the conditional generation p(y) x. 2; A) with latent code. As simplification, ditional prior p(2 xx; Φ) is a standard Gaussian as
we assume the con-

---

code 2K fx 0...0 o E R Note that when K = 0. the model reverts to the traditional VED framework, and we refer to this degenerated version as VEDSUM.
Hoad Layer
Feffneed Gate
Encoder-Decoder Our model adopts the Transformer-base encoder- decoder. The encoder processes the input text and learns sequence of hidden representations, and the decoder generates a summary based on the en- coder's hidden states and the previously generated tokens. We incorporate the latent information into the decoder with a gate mechanism, which mixes the latent vector 2к with the decoder's last layer of hidden states {hy [h}]ー=1- As pointed out in Gu et al. (2020), the saturation property of traditional gating mechanisms hinders gradient-based optimization. Therefore, following their proposal, use fined gate mechanism designed allow for better gradient flow. Let a--) be the sigmoid function. We generate the gated fused hidden states (hg]g=1 as in Eq. 4.
Gated Transformer-based
+1
Decoder Block
M2
900
fx txo..og 4
Figure 1: FlowSUM Model Architecture, including an NF latent module (in purple), a Transformer-bawee encoder-dee decoder (in green), and a refined gate mecha- nism (in orange)
in Setiawan al. (2020). Throughout this section,
let e be the embedding size, m, be the length of the input source and target summary respectively, l be the latent dimension of the NF latent module, d be the dimension of the decoder's hidden states, {r:)m = be the input source text, ( be the target summary text, and T E R be the average embedding of the untruncated input source text?.
# 3 E Rd, where V Raxt
fj = (W' [kgizk] Rd, where W' e Rdx2d
Ty 8 (W" [byj <k]] E ERA, where W E Rdd2d
9j (1 rÿ). f3 + +r (1 (1-(1-1)))) Rd
h/S = (1 9,) ) h1 + 2/x c IRdd
NF Latent Module. To model the variational pos- terior q( æ, y: t) we follow Zhou and Neubig (2017) and assume all the information in y is con- tained in .². Therefore, we have g(2 2t, y/5 v/ q(2 az 1b), which allows us to parameterize q(> v) with neural networks (NNs) and normalizing flows using the amortization and reparameteriza- tion tricks (Kingma and Welling, 2014). The NFF latent module comprises of an inference network go(+) and a normalizing flows model. The inference network takes E as input and produces two output vectors, 140 C Re and log(Øo) E RÉ Using the reparameterizatioon trick, random sample 2n E R is drawn from N(uo, diag(a))). Afterward, the normalizing flows model applies a sequence of K invertible transformations to 2g to obtain the latent
(4)
Afterward, the fused hidden states are passed to a language model (LM) Head layer, where they are transformed into vectors modeling the probabilities of each word in the vocabulary.
3.2 Training Objective
Traditional VEDs usually assume g(= x;) to be Gaussian, allowing analytical computation of the KL. term in ELBO. However, in normal- izing flows-based VED, the variational posterior q(> x) = gk () can be complex and hence the KL term Eq. 3 lacks an analytical form. Therefore, we rewrite the ELBO via a change of variables to enable analytical evaluation":
ELBONF-VED
LLet V be the vocabulary size, V be the input em- beddings, and W be the Bag-of- Words (BoW) of the input source text, then F = (Σ b. E.E b.) E R addition, when we don't 't truncate input text, 8" 1 m holds. However, we truncate the input due to encoder con- straints, then b71 m. and the BoW vector will contain information that would otherwise have been lost. See detailed discussion in Appendix C.
= (o))) [log (y/ x =.) + log (ZK xx]]
Bxo(ea) [log go (=o) ΣK log |dct Jr, (zk- |i]]
(5)
"The log-determinant of the Jacobian at each layer is recorded along the forward call for loss computation. 'See derivation in Appendix B Eq. 9.

---

where 90 is zo's probability density function, Gaussian distribution modeled by NNs, and det Je (-) is the determinant of f& s Jacobian.
(2020): Shrink and Fine-Tune (SFT) and Pseudo- labels (PL). SFT shrinks the teacher model and re-finetunes the shrunk model. In contrast, the PL method initializes the student model with the compressed version produced by SFT and then fine- tunes using the pseudo-labeled data generated by the teacher model. In this study, we fine-tune the model on the augmented data with both original and pseudo-labeled data, enabling it to more ef- fectively switch between generated summaries and ground truth, thereby mitigating exposure bias.
Let LCE denote the cross-entropy loss and Lvi denote the loss introduced by the variational latent module. Applying the idea of Monte Carlo to Eq. 5. we obtain the training objective as below. Note that Lvi is Monte Carlo estimate of the KL diver- gence between the variational posterior qk and the conditional prior distribution p(zk x))
[ - LCE + Lvi
- logp (3 { 2K, (k,)
4 Experiments
(6)
+ log (20) - Eim log det J (2k-1))
4.1 Datasets
log (zK .r)
We evaluate the effectiveness of FlowSUM six public benchmark datasets?, including CNN/Daily Mail (CNN/DM) (Hermann et al., 2015), XSum (Narayan et al., 2018), Multi-News (Fabbri et al., 2019), arXiv, PubMed (Cohan et al., 2018), and
3.3 Mitigating Posterior Collapse
To remedy posterior collapse, we consider two strategies, aiming to preserve the expressiveness of the latent variable and improve the overall sum- mary quality. The first approach, called 3C-VAE (Prokhorov et al., 2019), replaces the KL term with B CI, where a scaling factor, and C is a threshold that regulates the magnitude of the KL term. When C 0, the KL term is expected to be discouraged from getting close to O.
SAMSum (Gliwa et al., 2019). These datasets ex- hibit various summary styles and lengths, and their corresponding statistics are shown in Table 1. Refer to Appendix E for more details.
Split (train/val/test)
Avg. doc length
Avg. summary length
Datascts
We propose the second approach, Controlled Al- ternate Aggressive Training (CAAT), inspired by the lagging inference strategy (He et al., 2019). This strategy uses the observation that the infer- ence network cannot accurately approximate the true posterior in the initial stages of training. As outlined Alg. in Appendix A, CAAT comprises two stages. In the first stage, we alternately update the variational parameters and the entire parame- t for specified number of steps. In the second stage, we train all parameters jointly, as in basic VAE training. for the remainder of the training.
CNN/DM
287113/13368/1 1490
781
56
Multi-News
44972/5622/5622
2103
264
arXiv
201037//44366444)
4938
220
PubMed
119924/6633/6658
3016
203
XSam
204045/1133211
1334
431
23
SAMSum
14732/818/819
94
20
Table 1: Statistics of Summarization Datasets.
4.2 Implementation Details
We configure the inference net go (()g()) to be a feedforward neural network and set the latent di- mension & to 300 and the number of NF layers K {2, 4, 6, 8}. For models that use VAE, we set 3 and C 0.1, and for those using CAAT, conduct one epoch of aggressive training with Nalt = 15 and two epochs of non-aggressive train- ing. See more details in Appendix G.
3.4 NF-enhanced Knowledge Distillation
Normalizing flows can learn complex and multi- modal distributions (Papamakarios al.. 2017), which makes them promising approach for knowl- edge distillation tasks that involve integrating infor- mation from multiple sources (Hinton et al., 2015). To investigate the impact of normalizing flows on knowledge distillation, we adopt two knowl- edge distillation methods by Shleifer and Rush
4.3 Baselines
We use BART (Lewis et al., 2020) and BERT2BERT (Rothe et al., 2020) two backbone models. We refer to the PL knowledge distilled
'In our preliminary experiments, we find that if we alter- nate between variational and encoder-dccoder parameters, the training becomes unstable and generates NaN values. There- fore, we alternate between variational and all parameters.
We access them through Hugging Face Datasets, which provides reproducihle code for processing texts and generating ttain/vaiiaaiionfestt splits.

---

FlowSUM HlowSUM-PLKD. Our comparison involves the following baselines: PG+Cov (See
ROUGE
Model
L
et al.. 2017), BERT2BERT (Rothe et al., 2020),
PG+Cov (See et al., 2017)
39.53
17.28
36.38
BERT2BERT (Rothe et al., 2020)
41.28
18.69
38.09
BERTSUM (Liu and Lapata, 2019), BART (Lewis et al., 2020), PEGASUS (Zhang et al., 2020a),
BERTSUM (Liu and Lapata, 2019)
42.13
19.60
39.18
BART (Lewis et al., 2020)
44.16
21.28
40.90
VHTM (Fu et al., 2020), TAS (Zheng et al., 2020), and PEGASUS++FOw-NTT (Nguyen et al., 2021). See Appendix F for more detailed descriptions.
PEGASUS (Zhang et al., 2020a)
44.17
21.47
41.11
VHTM (Fu 2020)
40.57
18.05
37.18
TAS (Zheng al., 2020)
44.38
21.19
41.33
PEGASUS+NIM (Nguyen al., 2021)
44.52
21.95
41.39
4.4 Results
VEDSUM (BERT2BERT)
40.89
18,28
37.95
FlowSUM (BERT2BERT)
41.51
18.81
38.56
4.4.1
Automatic Evaluation
VEDSUM (BART)
44.36
21.09
41.37
We evaluate the generated summary quality us- ing ROUGE scores (Lin, 2004) and BERTScore (Zhang et al., 2020b) Specifically, we utilize the overlap of unigrams and bigrams (ROUGE-1 and ROUGE-2) to evaluate the informativeness, and the longest common subsequence (ROUGE-L) for fluency. Moreover, we report BERTSScree, which gauges semantic similarity based on contextual em- beddings. Furthermore, we present rep-w (Fu et al., 2021)" and the average length of summaries to gain a better understanding of the quality.
FlowSUM (BART)
44.64
21.36
41.65
FlowSUM-PLKD (BART)
44.59
21.49
41.59
Table 2: Comparison with baselines on CNN/DM.
ROUGE
Model
PG+Cov (See et al., 2017)
28.10
8.02
21.72
BERTSUM (Liu Lapata, 2019)
38.81
16.50
31.27
BART (Lewis et al., 2020)
45.14
22.27
37.25
PEGASUS (Zhang et al., 2020a)
47.21
24.56
39.25
TAS (Zheng al., 2020)
44.63
21.62
36.77
PEGASUS+NIM (Nguyen al., 2021)
49.57
25.08
41.81
We compare the proposed model against base- line models in ROUGE scores Tables 2 and 3. On CNN/DM, FlowSUM (BERT2BERT) greatly out- performs BERT whereas VEDSUM adds noise to the model and leads to a decrease in per- formance. With the BART backbone, FlowSUM achieves an absolute improvement over the BART model with +0.48, +0.08, and +0.75 in R-1, 2, and L scores, respectively. However, XSum, the variational models do perform well when the gold summaries involve only one sentence. VED- SUM leads significant decrease performance, whereas with FlowSUM, the decrease in ROUGE scores is less severe, leading to +0.12, 15,, and -0.25 in R-1, 2, and L scores, respectively. Table 4 uses BART the backbone and com- pares BART, VEDSUM, and FlowSUM across all datasets. Overall, variational models produce sum- maries of superior quality for datasets with long summaries, such as CNN/DM, Multi-News, arXiv, and PubMed, and FlowSUM further enhances the performance beyond VEDSUM. However, when comes to datasets featuring short summaries such XSum and SAMSum, the variational compo- nent markedly diminishes the model performance.
VEDSUM (BART)
43.62
20.27
35.06
FlowSUM (BART)
45.26
22.12
37.00
FlowSUM-PLKO (BART)
45.54
22.67
37.38
Table 3: Comparison with baselines on XSum.
We hypothesize that brief summaries may be more susceptible to disturbances and are more prone to being affected by noise. Nevertheless, incor- porating NF modules alleviates these reductions and accomplishes comparable outcomes. Further- more, we observe that both variational models tend to generate lengthier summaries, while FlowSUM exhibits fewer issues with repetition compared VEDSUM.
4.4.2 On NF-enhanced Knowledge Distillation We use PEGASUS as the teacher model to generate pseudo-labels on the CNN/DM training set. In this study, we explore the effects of knowledge distilla- tion on BART and DistiBART,, a shrunken version of BART. We examine two variations of Distil- BART: dBART-6-66, which replicates 6 layers 10 of the BART encoder and decoder, and UBART-12-3, which duplicates all layers of the BART encoder and 3 layers'! of the decoder.
"We obtain both metrics using Hugging Face Evaluate report the Fi scores.
Table 5 presents the impact of the PL approach on the original BART model. Training the BART
'rep-w is calculated as the proportion of the current token that appears in the previous v tokens. Refer to Appendix D for the detailed definition.
The 0, 2, 4, 9, and 11th layer.
"he 0, 6, and 11th layer.

---

Model
ROUGE
1/2/L
BERT- Score
rep-w.
Length
Model
ROUGE 1
2
BERT
Score
Length
40.90
CNN/DM
VEDSUM
44.34
21.00
41.37
89.20
88.63
BART
14.16/21.28/40
89,40
8.31
84.11
SUM (Planar)
44.62
21.32
41.64
89.20
90.78
VEDSUM 14.34/21.09/41..
89.20
8.43
88.63
Flo (RQNSF)
21.36
41.65
FlowSUM
44.64/21.36/41.66
89.46
8.43
92.24
PEGASUS
44.17
21.47
41.11
89.52
77.84
Multi-News
FLRD
42.83
20.16
39.98
89.04
100.52
VHDSUM-PLKD
44.45
21.25
41,45
89.41
93.42
BART 42.36/15.34/36.67
86.69
9.76
133.42
FlowSUM-PLAN (Planar)
41.15
89.34
VEDSUM 43.91/16.68/38.111
87.00
9.95
128.79
FlowSUM PLKD (RQNST)
44.59
21.48
41.59
89.47
84.75
FlowSUM
44.42/11.01/38.36
$7.09
9.91
128.87
Table 5: PL Knowledge Distillation on BART on
arXiv
CNN/DM.
BART 42.55/15.92/37.89
85.35
17.23
130.68
VEDSUM 43.05/16.34/38.22
85.44
16.63
130.92
FlowSUM
43.11/16.26/38.11
85.45
16.55
132.88
Model
ROUGE
BEKT
Length
Parans (MM)
Inference
PubMed
BART 41.57/16.72/36.94
84.65
13.26
136.10
6
12.7820.2439972
88.98
67.42
2:00
170.5
FlosSUM
43 43.41/20.53/000
89.18
91.25
238
234.9
VEDSUM
21.21/119.20/993
85.07
12.76
138.70
FboSUUM
4370020.71440.75
89.24
91.10
238
239.7
FlowSUM
44.55/19.5039959
85.16
12,59
138.09
XSum
12ARE-223
13.3900.57400 44
89 20:20
85-48
255
199
POoSUM
13.522086100399
89.28
8374
263
190.7
BART 45.14/22.27/337.25
92.16
4.63
25.54
FlwSUM
440521.0644107
84.48
263
200.4
VEDSUM 43.62/20.27/35.00
91.75
5,96
31.22
FlowSUM
45.26/22. 12/37.00
92.13
4.95
28.71
Table 6: Knowledge Distillation on DistilBART on
CNN/DM.
SAMSum
BART
13.19/28.19/49.03 19/49.03
92.68
6.71
30.00
VEDSUM $1.91/26.74447.441
92.40
7.53
30.92
achieves R-1/2/L improvements of 0.92/0.47/1.0
FlowSUM 53.13/28.49499.00
92.67
6.59
29.77
over dBART-6-6, and SIMMSSUMPLKKE (dBART- 12-3) achieves improvements of 0.66/0.49/0.63 over dBART-12-3, much more than the SFT ap- proach. Furthermore, FlowSUM does not intro- duce additional computational burden at inference, and the time cost is primarily related to the length of the generated summaries.
Table 4: Comparison of BART, VEDSUM (BART), and FlowSUM (BART) on all six benchmarks.
model augmented data worsens the performance compared training the original data. In con- trast, VEDSUM-PLKD achieves improvements in all three ROUGE scores, and FlowSUM-PLKD with RQNSF achieves the highest R-2 score, albeit with some sacrifice in R-1 and '2 However, planar flows appear to be unsuitable for knowledge distillation via PL. To better understand FlowSUM- PLKD, we visualize the latent distribution (see Ap- pendix I) and demonstrate how the NF's ability to capture multi-modality could account for its im- pressive performance.
4.4.3 Analysis on NF Types and Depth
We investigate the effect of NF types and the num- ber of NF layers the Multi-News dataset Ta- ble 7 explores the effect of NF types. Simple flows like Planar and Radial yield inferior performance compared to the VAE counterpart, whereas more complex flows tend to achieve greater improve- ments. Overall, IAF and RQNSF emerge as the west-performing NF types.
Table 6 investigates the two DistilBART vari- ants with RQNSF. With FlowSUM, both variants achieve improvements, suggesting that NF is bene- ficial for the SFT approach. Previous experiments from Shleifer and Rush (2020) showed that PL per- formed worse than SFT on CNN/DM. However, our experiments reveal that the NF latent module unleashes the potential of PL. When trained on augmented data, FlowSUM-PLKK (dBART-6-6)
Table 8 delves further into IAF and RQNSF, in- vestigating the effect of NF depth. The findings indicate that adding more layers does not always lead to improved performance. We hypothesize that when the incoder-decodee model is well-eraineed, the increased complexity of the NF' module may introduce more noise, outweighing the benefits of better latent modeling and subsequently worsening the summary quality.
TThis can he explained by the teacher model's is worse per- formance in these two metrics.
We choose Multi-News due to its smaller size, enabling us to conduct experiments with reduced computational cost.

---

ROUGE1 1
BERT
ROUGE
KL
Model
rep-w..
Length
Model
Training
1/2/L
Score
1.
Divergence
BART
42.56/15,
13.56/15.38/36.77 35/3667
86.69
9.76
133.42
VEDSUM VEDSUM
standard
43.91
38.10
VEDSUM
43.91/16.68/38.11 10
87.04
9.95
128.79
Bc-
43.78
16.54
37.96
0,0082
FlowSUM (Planar)
43.85/16.61/35
87:03
10.04
128.84
SUM (Radial)
13.84/16.00 68/37779
87:04
9.92
128.72
FlowSUM (Planar)
standard
43.85
16.61
37.97
0,2719
SUM (Sylvester)
44.18/16.7188 71/33.151
87.08
9.80
128.76
FlowSUM (Planor)
43.68
16,47
37.85
0.1815
FlowSUM (RealNVP)
44. 44.19/16.4/38.11
87:05
9.81
128.76
FlowSUM (Radial)
standard
43.63
16.37
37.82
0.0121
PloWSUM (IAF)
44.42/17.01/38..6
87.09
128.87
FlowSUM (Radial)
AA-VAE
43.84
37.98
0.0096
Flow SUM (RI NSF)
44 14.25/16.86/38. 14
87.06
9.80
128.80
FlowSUM (RQNSF)
44.31/16,98338227
87.07
128.81
FlowSUM (Sylvestrr)
standardd
43.68
16.51
31.87
0.0841
FlowSUM (Syivester)
Ac-VAE
44.18
16.71
38.15
0.0348
Table 7: Effect of NF Types Multi-News.
FlowSUM (RealNVP)
standard
44.19
16.64
38.15
4.7986
FlowSUM (ReaINVP)
3c-VAE
43.71
16.54
37.85
7.8938
FlowSUM (RealNVP)
44.12
16.82
38.11
5,2107
ROUGE
BERT
FlowSUM (IAF)
standard
43.87
16.62
37.97
3,9146
Model
1/2/L
Score
rep-w
Length
FlowSUM (LAF)
VA-
43.81
16.58
37.91
3.9128
FlowSUM (LAF-4)
H4.30/17.03/38.22
87.05
9.82
128.81
FlowSUM (IAF)
CAAT
44.30
17.03
38.22
2.1108
FlomSS/M ([AR-)
44.42/17.01/38,.
87.09
128.87
FlowSUM (RLNSF)
standard
44.25
16.86
38.14
104,9667
FloSUUM (LAF-S)
44.18/16.9038..8 16
87.04
9.88
128.84
FlowSUM (RLNSF)
VA-
44.25
16,86
38.14
104,9667
FloSSUM (RQNSF-2)
44.15/16.88/8 88/38.20 20
87.04
9.94
128.83
FlowSUM (RLNSF)
CAAT
44.14
16.82
38.05
95.3774
FlowSU/M (RONSF.4)
44.31/16,98/38.77
87.07
9.91
128.81
FlowSUM (RQNSF)
standard
44.18
16.76
38.18
127.8106
FlowSUM (RQNSF-6)
44.15/1688//88.1 18
87.06
9.87
128.92
FlowSUM (RQNSF)
Ac-VAE
44.18
38.18
127.8106
FlowSUM (RQNSF)
CAAT
44.31
16.98
38.27
107.0794
Table Effect Number of NF Layers on Multi-News.
à VEDSUM and HowSUM with radial dlows have CAAT results as tmining unstable and generates values.
Table 9: Effect of Training Strategies.
4.4.4 Analysis on Training Strategies
We implement standard VAE training, VAE, and CAAT on VEDSUM and FlowSUM models, and we evaluate their effectiveness with different types of normalizing flows. Table 9 shows that VEDSUM and FlowSUM models with residual flows, including planar, radial, and Sylvester flows, suffer from posterior collapse, whereas those with more complex flows do not. Moreover, applying 9c- VAE to VEDSUM and FlowSUM models with residual flows does not effectively mitigate poste- rior collapse but even exacerbates the issue. Fur- thermore. for models with planar, RealNVP, and IAF flows, training with VAE worsens ROUGE scores, while for radial and Sylvester flows, it im- proves performance. Notably, the two neural spline flows are not impacted by 8c-VAt VAE training.
ROUGE
Training
Gate Initialization
2
L
standard standard CAAT CAAT
standard
40.82
18.29
37.92
near-zero
40.98
18.36
38.09
standard
41.51
18.81
38.56
near-zero
41.13
18.57
38.21
Table 10: Effect of CAAT and Gate Initialization.
imately 0.05, Our experiments using FlowSUM (BERT2BERT) with RQNSF as the base model reveal that CAAT + Standard Gate Score Initial- ization yields the best results and the most stable training process, illustrated in Table 10 and Fig- ures 2 to 3 in Appendix H. This suggests that by setting large initial gate score and forcing the model to learn from the NF latent module, we can better capture latent code information.
Concerning CAAT, we note that applying it to treat severe posterior collapses such as VEDSUM and FlowSUM with residual flows can cause in- stability in training while producing NaN values. Hence, it is only effective for models with KL di- vergence that is not close to zero. Nonetheless, when applicable, CAAT enhances the quality of summaries, particularly when utilized with the top- performing NFs, namely IAF and RQNSF.
Conclusions and Discussions
This paper introduces FlowSUM, a normalizing flows-based Variational Encoder-Decoder (VED) framework for text summarization. It outperforms leading non-latent model across multiple datasets. This enhanced performance is attributed to the flexi- ble posterior distributions provided by normalizing flows. We also analyze the operating characteristics and the posterior collapse problem of normalizing flows and propose an effective training strategy for complex flows. Moreover, we demonstrate that in-
In addition, explore the impact of gate score initialization. The standard method initializes gat- ing weights with small deviations from zero, result- ing in an initial gate score close to 0.5. In contrast, the near-zero initialization method initializes gating weights such that the resulting gate score is approx-

---

corporating normalizing flows is highly effective for knowledge distillation with minimal impact on inference time.
Acknowledgements
This work was supported in part by NSF grant
and NIH grants RO1AG069895, ROIAG065636, R01AG074858, UO1AG073079.
FlowSUM illustrates the advantages of incorpo- rating flexible latent modeling. Considering the remarkable achievements of Latent Diffusion Mod- els (LDMs) in generating images (Rombach et al., 2022), adopting LDMs for capturing latent repre- sentation may produce comparable even superior outcomes in text summarization. In this scenario, the gating mechanism may not be an appropriate choice. direct correlation between the latent vec- tor and the target text may be more suitable for executing the diffusion process. Enhancing the ar- chitecture to leverage diffusion models could be potential avenue for future research.
References
Vladimir Bogachev. 2007. Measure Theory. Springer.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, An- drew Dai, Rafal Jozefowicz, and Samy Bengio, 2016. Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference on Com- putational Natural Language Learning, pages 10-21, Berlin, Germany, Association for Computational Lin- guistics.
Arthur Brazinskas, Mirella Lapata, and Ivan Titov. 2020. Unsupervised opinion summarization as copycat- review generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACL 2020, Online, July 5-10, 2020, pages 5151-5169, Association for Computational Linguis- tics.
Limitations
FlowSUM has demonstrated excellent results on datasets with long summaries. However, its perfor- mance short-summary datasets like XSum and SAMSum has been unsatisfactory. The underlying cause could be attributed suboptimal hyperpa- rameter tuning or the incompatibility of FlowSUM with short summaries. Additional investigations are needed to identify the root cause.
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Πlya Sutskever, and Picter Abbeel. 2017. Variational lossy autoen- coder. In Sth International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open- Review.net.
Furthermore, we did not fine-tune the hyper- parameters of the normalizing flows model, such as the latent dimension, the number of bins in spline coupling layers, and the neural network in IAF, Re- opted
Eric Chu and Peter J. Liu. 2019. Meansum: A neural model for unsupervised multi-document abstractive summarization. Proceedings of the 36th Inter- national Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Re- search, pages 1223-1233 PMLR.
alNVP, RLNSF, and RONSF. Moreover, for small batch size due memory limitations. Adjusting these hyperparameters could potentially enhance the model's ' performance.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim. Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse- aware attention model for abstractive summarization of long documents. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Short Papers), pages 615-621 New Or- lcans, Louisiana. Association for Computational Lin- guistics.
Due to limited computational resources, we uti- lized BART and BERT2BERT as the backbone models instead of newer architectures. Further re- search may focus verifying the effectiveness of FlowSUM more advanced structures.
Ethics Statement
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Our research entailed developing new text sum- marization framework. Although no private data were utilized, we acknowledge the potential soci- etal impacts of our work. Therefore, we adhered to pertinent ethical guidelines and implemented rig- orous procedures to guarantee the accuracy of our results.

---

Laurent Dinh. David Krueger, and Yoshua Bengio. 2015, NICE: non non-linear independent components estima- Learning
Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. 2021. A Theoretical Analysis of the Repeti- tion Problem in Text Generation. In Proceedings of the AAA/ Conference on Artificial Intelligence, vol- ume 35, pages 12848-12856
tion. In 3rd International Conference Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings.
Laurent Dinh, Jascha Sohl-DDicsttei, and Samy Bengio. 2017. Density estimation using real NVP. In 5th International Conference on Learning Representa- tions, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net,
Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. 2015. MADE: masked autoen- coder for distribution estimation. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Pro- ceedings, pages 881 881- JMLR.org.
Hadi Mohaghegh Dolatabadi, Sarah M. Erfani, and Christopher Leckie. 2020. Invertible generative mod- eling using linear rational splines. In The 23rd In- ternational Conference
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alck- sander Wawer. 2019. SAMSum corpus: A human- annotated dialogue dataset for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pages 4236-4246. PMLR.
Li Dong, Nan Yang. Wenhui Wang. Furu Wei, Xi- aodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Pro- cessing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13042-13054.
Albert Gu, Çaglar Gülçehre, Thomas Paine, Matt Hoff- man, and Razvan Pascanu. 2020. Improving the gating mechanism of recurrent neural networks. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Vir- rual Event, volume 119 of Proceedings of Machine Learning Research, pages 3800-3809, PMLR.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg- Kirkpatrick. 2019. Lagging inference networks and posterior collapse in variational autoen- coders. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Wanyu Du, Jianqiao Zhao, Liwei Wang, and Yangfeng Ji. 2022. Diverse text generation via variational encoder-decoder models with gaussian process priors. arXiv preprint 2rXiv:2204.01227.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. 2019. Neural spline flows. In Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurlPS 2019, December 8-14. 2019, Vancouver, BC, Canada, pages 7509-7520.
Karl Moritz Hermann, Tomás Kociský, Edward Grefen- stette, Lasse Espcholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neu- ral Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1693- 1701.
Bryan Eikema and Wilker Aziz. 2019. Auto-encoding variational neural machine translation. In Proceed- ings of the 4th Workshop on Representation Learning for NLP, RepLANLP gust 2, 2019, pages 144-141, Association for Com- putational Linguistics.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in neural network, arXiv preprint
2019, Florence, Italy, Au-
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26- 30, 2020. OpenReview.net.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstrac- tive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 1074-1084, Florence, Italy, Asso- ciation for Computational Linguistics.
Yoon Kim, Sam Wiseman, Andrew C. Miller, David A. Sontag, and Alexander M. Rush. 2018. Semi- amortized variational autoencoders. In Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmsmässsnn, Stock- holm, Sweden, July 10-15, 2018, volume 80 of Pro- ceedings of Machine Learning Research, pages 2683- 2692. PMLR.
Xiyan Fu, Jun Wang, Jinghan Zhang, Jinmao Wei, and Zhenglu Yang. 2020. Document summarization with vhtm: Variational hierarchical topic-aware mecha- nism. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7740-7747.

---

Diederik P. Kingma and Jimmy Ba. 2015. Adam: method for stochastic optimization. In 3rd Inter- national Conference ICLR 2015, San Diego CA. USA. May 7-9, 2015, Conference Track Proceedings.
Conference on Empirical Methods in Natural Lan- guage Processing. pages 9443-9456, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Learning Representations.
George Papamakarios, lain Murray, and Theo Paylakou. 2017. Masked autoregressive flow for density estima- tion. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Informa- tion Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 2338-2347.
Diederik P. Kingma and Max Welling. 2014. Auto- encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Durk P Kingma, Tim Salimans, Rafal Jozefowiez, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improved variational inference with inverse autore- gressive flow. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A Deep Reinforced Model for Abstractive Summarization. 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. tosseuueeeeee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Victor Prokhorov, Ehsan Shareghi, Yingzhen Li, Mo- hammad Taher Pilchvar, and Nigel Collier. 2019. On the importance of the Kullback-Leibler divergence term in variational autoencoders for text generation. In Proceedings of the 3rd Workshop on Neural Gen- eration and Translation, pages 187. Hong Kong. Association for Computational Linguisties.
BART: Denoising
Weizhen Qi. Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. ProphetNet: Predicting future n-gram for gurvencccttSSequeeeeP--rrrinnn
Yhin-Yew Lin. 2004, ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational I. Linguistics.
In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2401-2410, Online. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl Blog, 188):9.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 11(5)55485-5551
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. / 1699.
Marc' Aurelio Ranzato, Sumit Chopra. Michael Auli, and Wojciech Zaremba. 2016. Sequence level train- ing with recurrent neural networks. In 4th Inter- national Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Tien-Ching Luo and Jen- onnTzung Chien. 2021. Varia- tional dialogue generation with normalizing flows. In IEEF International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021, pages 7778-7782. IEEE.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLL) pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't ' give the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807, Brussels, Bel- gium. Association for Computational Linguistics.
Thong Nguyen, Anh Tuan Luu, Truc Lu, and Tho Quan. 2021. Enriching and controlling global semantics for text summarization. In Proceedings of the 2021
Danilo Jimenez Rezende and Shakir Mohamed. 2015. Variational inference with normalizing flows. In Pro- ceedings of the 32nd International Conference on

---

Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Con- ference Proceedings, pages 1530-1538. Jog..
Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18). the 30th inno- vative Applications of Artificial Intelligence (IAAI- 18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2016 8, pages AAAT Press.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra, 2014. Stochastic backpropagation and ap- proximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning. volume 32 of Proceedings of Ma- chine Learning Research, pages 1278 -1266, Bejing, China. PMLR.
Xiaoyu Shen, Hui Su, Shuzi Niu, and Vera Demberg. 2018b. Improving variational encoder-decoders in dialogue generation. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Arti- ficial Intelligence (IAAI-18), and the 8th AAAI Sym- posium on Educational Advances in Artificial Intel- ligence (EAAA-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5456-5463. AAAI Press.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. 2022. High- resolution image synthesis with latent diffusion mod- els. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10674-10685 IEEE.
Sam Shleifer and Alexander M Rush. 2020. Pre- trained summarization distillation. arXiv preprint arXiv:2010.13002
Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for se- quence generation tasks. Transactions of the Associ- alion for Compulational Linguistics, 8:264-280.
Kaitao Song, Xu Tan. Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2019. MASS: masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California. USA, volume 97 of Proceedings of Ma- chine Learning Research, pages 5926-59366 PMLR.
Raphael Schumann. 2018. Unsupervised abstractive sentence summarization using length controlled vari- ational autoencoder. CoRR, abs// 809.05233.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguistics.
Jinsong Su, Shan WI, Deyi Xiong, Yaojie Lu, Xianpei Han, and Biao Zhang. 2018 Variational recurrent neural machine translation. In Proceedings of the Thirty-Second AAAI Conference Artificial Intelli- gence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium Educational Advances in Artificial In- relligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7. 2018, pages 5488-5495 AAAI Press.
Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth. 2017. A hybrid convolutional variational au- toencoder for text generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 627-637, Copenhagen, Denmark. Association for Computational Linguis- tics.
Esteban G Tabak and Cristina V Turner. 2013. A fam- ily of nonparametric density estimation algorithms. Communications Pure and Applied Mathematics. 66(2)145-164.
Tulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2017. A hierarchical latent vari- able encoder-decoder model for generating dialogues. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA.pages pages 3295-3301. AAAT Press.
Ilya O. Tolstikhin. Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf. 2018 Wasserstein auto- encoders. In 6th International Conference on Learn- ing Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net.
Dustin Tran, Keyon Vafa, Kumar Krishna Agrawal, Lau- rent Dinh, and Ben Poole. 2019. Discrete flows: Invertible generative models of discrete data. In Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancowver, BC, Canada, pages 14692-14701.
Hendra Setiawan, Matthias Sperber, Udhyakumar Nal- lasamy, and Matthias Paulik. 2020. Variational neu- ral machine translation with normalizing flows. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7771- 7777, Online. Association for Computational Lin- guistics.
Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomezak, and Max Welling, 2018. Sylvester normal- izing flows for variational inference. In Proceedings of the Thirty-Foure Fourth Conference on Uncertainty in
Dinghan Shen, Yizhe Zhang, Ricardo Henao, Qinliang Su, and Lawrence Carin. 2018a. Deconvolutional latent-variable model for text sequence matching. In

---

Artificial Intelligence, UAI 2018, Monterey, Califor- nia, USA, August 6-10, 2018. pages 393 402. AUAI Press.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. 2017. Information maximizing variational autoen- coders. arXiv preprint arXiv: 17Xiv:/006002: 0226..
Infovae:
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Chujie Zheng, Kunpeng Zhang. Harry Jiannan Wang. Ling Fan, and Zhe Wang. 2020. Topic-guided ab- stractive text summarization: a joint learning ap- proach. arXiv preprint arXiv:2010.10322
Chunting Zhou and Graham Neubig. 2017. Morphologi- cal inflection generation with multi-space variational mncoder-decoders In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Mor- phological Reinflection, pages 58-65, Vancouver. As- sociation for Computational Linguisties.
Wang. Junlin Yao, Yunzhe Tao, Zhong, Wei Liu, and Qiang Du. 2018. A reinforced topic- aware convolutional sequence-to-sequence model for abstractive text summarization. arXiv preprint orXiv:/805.036/1
Yicheng Zou, Lujun Zhao, Yangyang Kang. Jun Lin, Minlong Peng. Zhuoren Jiang. Changlong Sun, Qi Zhang, Xuanjing Huang, and Xiaozhong Liu. 2021. Topic friictoriented spoken dialogue summariza- tion for customer service with saliency-awart topic modeling. In Thirty-Fifih AAAI Conference on Ar- rificial Intelligence, AAAI 2021, Thirty-Third Con- ference on Innovative Applications of Artificial In- telligence, IAAI 2021, The Eleventh Symposium Educational Advances in Artificial Intelligence, EAAI 2021. Virtual Event, February 2-9, 2021, pages 14665-14673, AAAI Press.
Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. 2019. Topic-guided variational auto-encoder for text generation. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 166-177, Minneapolis, Min- nesota. Association for Computational Linguistics.
Zhengjue Wang. Zhibin Duan, Hao Zhang, Chaojie Wang. Long Tian, Bo Chen, and Mingyuan Zhou 2020. Friendly topic assistant for transformer based abstractive summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 485 985-497,00 On- line. Association for Computational Linguistics.
A Controlled Alternate Aggressive Training (CAAT)
Algorithm 1 Controlled Alternate Aggressive
Training (CAAT)
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg- KinkpKirkattric... 2017. Improved variational autoencoders for text modeling using dilated con- volutions. In Proceedings of the 34th International Conference Machine Learning, ICML 2017, Syd- ney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3881-3890. PMLR.
Input: number of aggressive training steps naggs maximum number of training steps Nmazi number of alternating steps nait-
0, &b + Initialize encoder decoder parameters and variational parameters respectively
for i 1.2, Bagg do X 4- Random data minibatch
3:
Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and
4:
if mod Natt = then
Min Zhang. 2016. Variational neural machine trans- lation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing, pages 521-530, Austin, Texas. Association for Computational Linguistics.
5:
Compute 90. < V Vxx xb)
6:
Update 0, za using gradients go.0
7:
else
Compute Iv Update ab using graidents 9Φp
O,
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter J. Liu. 2020a. PEGASUS: pre-training with ex- tracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, 1CML 2020, 13-18 July 2020, Virtual Event, volume 11 of Proceedings of Machine Learning Research, pages 11328-112 1139. PMLR.
10: for i = Ragg Nlgg + 1, ... nmax do
11:
X + Random data minibatch
12:
Compute 98. <- Veee 0, Update 0, 1b using gradients 90.v
13:
14:
if early stopping criterion is met then break
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020b. Bertscore: Eval- uating text generation with BERT. In 8th Inter- national Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
15:
Another advantage of the controlled alternate ag- gressive training (CAAT) strategy is that it provides us with more control. is commonly assumed that

---

allowing the model more freedom to learn, even if the NF latent module is not helpful, will not harm performance. However, our experiments suggest that this assumption does not hold, particularly for short-summary datasets where the model will not learn on its own to avoid hurting the original perfor- mance. The CAAT strategy allows us to effectively freeze the encoder-decoder parameters by setting Nogy and natt large values, ensuring that when the nf module is unhelpful, will not significantly harm performance.
(L(a(= (z,))lp(a (,,)))
(elle (2 xt, w/] -Bucolass) log 22,, 3., 1) p(a,,
Root log (2 x.. w/]
Motellnn) log p(),) pla,2)
p(xx)
p(a,=) p(xx p(x, v)
-Egcola.o) Jog ((2 a. Batala [log | [2, =)]
- Eggattn [lug x]] + Eg (lggíy [x)
= K L(1(2 | x. v) [|p( x)) By(ala [logp(y 1
+ Re [oog xx]]
00
B Deeper Dive into the Evidence Lower Bound (ELBO)
ELBOvED
log 2,, (hotela.nnloog(
K KLL((2 2,)))pp(> a))
log a)) KL[g(= a, lpl( 22,
log p(y ir)
Within the VED framework, the conditional data generation process can be expressed as follows:
EL ELBONE-EE
(alla) [logply [x + Ba(att) log p(2
Bg(ttee) log g(> x]]
p(u | x; 0.0 f p(z 12; xx;0)((v| 2, 22;;
mmo()) [logp() pogp(y (y) x, #x) + log ( xr]
(ep((s)) [log 9x (#x)
E [log p (y/ n. 2x) + logp (Ex xx)
Boo(
1o (zu) Σ log Idet Jyp (2x-13)
The subsequent challenge revolves around param- eter estimation. Typically, the conditional latent prior is assumed as p(z x; Φ) = (N, I) for simplification (hence eliminating the Φ parameter).
where go and gK the probability density func- tion for 2g and žς respectively.
Despite this, the likelihood p(y x;; 0) remains computationally intractable to evaluate. Variational inference tackles this issue by introducing a vari- ational distribution g(= v, y; v)) from specific parametric family, aiming to approximate the actual posterior p( x. y). Here, A denotes the model pa- rameters, and & refers to the variational parameters. Instead of attempting to estimate solely through maximizing the challenging log-likelihood, the ap- proach involves joint estimation of both 0 and zb by optimizing the ELBO.
C Discussion on q(> x. y) - q(= x))
we choose to assume q(2 x, y) = g(2 | x) for the following reasons. Firstly, this assumption is grounded in the nature of summarization, where y can be viewed as condensed form of x and hence it is sensible to assume all the information in y is contained in æ. Secondly, evidenced by Zhang et al. (2016). terior on both z and y. However, their approach suffers from difficulties during prediction. In pre- diction, the target text is not accessible, making it hard to sample from q(> xr y). Zhang et al. (2016) suggests taking the prior's 's mean as the latent code, but in our paper, the prior is Gaussian whereas the posterior is complex distribution modeled by normalizing flows, and taking such strategy would diminish the benefit of using normalizing flows. Thirdly, it has been shown empirically by Eikema and Aziz (2019) that by restricting the con- ditioning of the posterior to x alone, their model
plausible to condition the pos-
Examining Eq. and 8, it's evident that the ELBO represents lower bound of the log- likelihood. Moreover, smaller value of KT.(g(= z, x. x, y)) indicates closer alignment be- tween the variational posterior and the true pos- terior, thereby bringing the ELBO closer to the log-likelihood. This insight propels the adoption normalizing flows to model a flexible family of variational posterior.

---

achieves higher accuracy. Therefore, we consider g(z Fz, v) q(> æ) as our modeling strategy.
sequence Transformer architecture. In this paper. we use BART as the encoder-decoder backbone. PEGASUS (Zhang et al., 2020a) is a large Transformer- based S2S model, pre-trained on mas- sive text data using self-supervised objective called gap sentence generation, designed for ab- stractive summarization.
D Repetition Measures
Let represent the sentences in a result set D, |s| be the number of tokens in 8, St be the tth token, and Saj be the sub-sequence of from the ith token to the jth token. The rep-w (Fu et al., 2021) is then defined by Equation 10.
VHTM (Fu al., 2020) is a variational hierarchical model built on the PG network. models the topic proportion vector with isotropic Gaussian and fuses in topic information at diverse granularity levels. TAS (Zheng et al., 2020) is a topic-guided Transformer-based S2S model that injects the topic- word matrix into the LMHead layer and jointly trains the NTM and encoder model.
1
rep-w
11 8e E mmxx(t-w,1)dd-)
D
SED
2=2
(10)
E Datasets
PEGASUS+EIOw-NIM, (Nguyen et al., 2021) a topic-aware model built on PEGASUS. It utilizes a Flow-based NTM and a contextualized gating mechanism to integrate topic information into the encoder and the decoder.
CNN/Daily Mail (Hermann et al., 2015) consists of 312,085 online news articles, with one article paired with multi-sentence summary. We use the non-anonymized version as in See al. (2017) and follow the text processing" 14 in Lewis et al. (2020). XSum (Narayan et al., 2018) contains 227k BBC articles, each summarized a single sentence. Multi-News (Fabbri et al., 2019) is a multi- document dataset comprising 56k pairs of news articles and multi-sentence summaries.
G Implementation Details
G.1 NF Latent Module
We configure the inference net (=00[T) to be feed- forward neural network with three hidden layers of dimension E {300, 600), Tanh activations, and a 0.1 dropout rate. We set the latent dimension to 300 and the number of NF layers E {2, 4. 6, 8}. For spline coupling layers (RLNSF and RQNSF), we set the number of bins to 4, the bound to 3.0, the split dimension f/2, and the neural network have two hidden layers with the dimension l. For RealNVP, the split dimension f/2, and the neu- ral network has one hidden layer with a dimension of 10C. For IAF, the neural network features one hidden layer of the dimension 3 + 1. Moreover, we set 8 - 1 and C = 0.1 for models that use Se- VAF, and for models that use CAAT, we conduct one epoch of aggressive training with nhat 15, followed by two epochs of non-aggressive training.
arXiv, PubMed (Cohan al., 2018) are two scien- tific paper document datasets from arXiv.org (113k) and PubMed (215k). Each pair consists of scien- tific article's ' body document and its abstract.
SAMSum (Gliwa et al., 2019) includes 16k conver- sations annotated with summaries by linguists. Un- like structured texts, the information in dialogues is scattered across different speakers' utterances, increasing the summarization difficulty.
F Baseline Models
PG+Cov (See al., 2017) is pointer-generator (PG) network supplemented with a coverage mech- anism that addresses the Out-Of- Vocabulary prob- lem and minimizes word repetition.
BERT2BERT (Rothe et al., 2020) initializes both the encoder and the decoder with the pre-trained BERT checkpoints and adds cross-attention layers. BERTSUM (Liu and Lapata, 2019) builds top of BERT and applies fine-tuning scheduler to better align the encoder and the decoder.
G.2 Optimization
We train the models using the Adam optimizer (Kingma and Ba, 2015) with 31 - 0.9, f2 - 0.999, and e = 10 8. The initial learning rate is 5 x 10 5 We employ a lincar learning rate scheduler that in- creases the learning rate from 0 the initial learn- ing rate during the warmup stage and decreases it from the initial learning rate to 0 after the warmup stage. We also apply the gradient clipping tech- nique with a maximum gradient norm of 1.0. Fur-
BART (Lewis et al., 2020) is a pretrained denois- ing autoencoder with the standard sequence-to-
We update the data loading script following https:// gi titubb.co com/ www./coo//caeeeeee ceceeboookeseeercco wat./aairrr/isssaee

---

thermore, we terminate the training early when the perplexity fails to improve for eight or sixteen consecutive evaluation calls.
G.3 Model Hyper Parameters
Table 11 provides the hyper-paramess for the models discussed in Table 4 7, for the sake of reproducibility. To ensure fair comparisons, unless otherwise specified, the VEDSUM models typi- cally employ the same set of hyper-parameters as
(a) Gate Score
(b) Training Perplexity
their FlowSUM counterparts, except with standard training and no NF layers applied. Additionally, the models in Table 8 have the same parameersssssrss those in Table 7, except for the number of NF layers used. Lastly, in Table 9, all FlowSUM mod- els 4 NF layers and the same set of hyper- parameters those in Table 7 but vary in their training strategies.
(c) Evaluation Perplexity
Figure 2: Comparison of training strategies and gate initialization.
Experiments on Training Strategies and Gate Initialization
The training curves for the methods in Table 10 are illustrated in Figure 2. The plot demonstrates that the gate score decreases gradually and remains high during aggressive training when CAAT is com- bined with standard initialization. This combina- tion compels the model utilize the latent code information effectively. Moreover, as presented in Figure 2c, even though CAAT combined with standard initialization starts with a high perplex- ity, it achieves a lower perplexity level than other approaches by the end. By examining the training procedure in detail, Figure 3 further indicates that CAAT contributes to greater training stability than standard training.
CAAT
(b) Standard Training
Figure 3: A closer look at the training process: CAAT vs. Standard Training.
retel
jin
I Visualization of Latent Distribution
To gain better understanding of how normalizing flows contribute to knowledge distillation, we se- lected several examples from the CNN/Daily Mail and XSum datasets and visualized the resulting la- tent distribution generated by the FlowSUM-PLKKD model, as shown in Figure and 5. For both cases, the transformed latent code exhibited highly flexible distribution. Notably, in the CNN/Daily Mail example, the first dimension of the second example demonstrated clear bi-modal distribu- tion, indicating the model's ability capture in- formation from multiple sources. Similarly, in the XSum dataset examples, we observed distinct multi-modaa patterns.
300
Figure 4: Visualization of the first two dimensions of 20. ÆK. and (0, by FlowSUM-PLKD on CNN/DM. The right sub-figure demonstrates a clear bi-modality.

---

FlowSUM in Table 4
Number of epochs
Number of aggressive epochs
Batch
Inference net
Number of
Beum
Length penalty
Max input
Max turgel
Dataset
NF type
hidden dim
layers
tokons
tokkens
CNN/Daily Mail
300
RQNSF
4
4
2.0
1024
128
Multi-News
600
IAF
2.0
1024
128
atXiv
600
RQNSF
142
PubMed
16
600
RONSF
4
2.0
1024
142
XSam
600
RONSP
0.5
1024
62
600
RQNSF
Models in Table
Model
Number
Number apgressive epochs
NF type
Number
Beam size
Length penalty
input
target
epochs
size
hiddlen dim
NH layers
tokens
tokoons
VEDSUM
0
600
2.0
1024
128
FlowSUM (Planar)
600
1024
128
PloWSUM (RQNSF)
300
RONSF
2.0
1024
128
BART- PLKD
0
2.0
1024
128
VEDSUM-PIKKI
WOWSUUMPLLK (Planar
0
600
Planur RONSP
2.0
1024
128
SLoWSUM-PI (RQNSF)
300
2.0
1024
128
Models in Table
Number of epochs
Number of aggressive epochs
Batch
Inference net
Number of layers
Beam
Longth penalty
Max inpot
Max target tokenss
Model
NF type
hidden
dBART-6-6
FloSUUM
300
RONSF RONSP
4
2.0
1024
128
FlowSUM-PLXXT
300
2.0
1024
128
dRART-12.3
FloWSUM
300
RONSK
1024 1024
FloSUM PLKD
300
RONSF
2.0
128
Models in Table
Number of epochs
Training stralegy
Batch size
Inference net
Number of
Beam size
Length penalty
Max input
Max target
Model
NF type
hidden dim
NF layers
tokens
tokeess
Flo (Plsnar)
standard
600
FlowSUM (Radial)
RA-VAE
600
Radial
4
4
2.0
1024
128
FUowSUM (8ylvester)
Re-VAR
600
Sylvester
2.0
1024
128
SUM (ReaINVP)
standard
600
RealNVP
128
FUowSUM (IAF)
1/3 CAAT"
600
IAF
6
2.0
1024
128
Flow SUM (RLNSP)
Re-VAE
600
RLNSP
2.0
1024
128
(RQNSF)
600
128
12" means applicable.
CAAT: aggressive training I epoch and non-aggressive triningg for 2 epochs.
Table Model
Normalizing Flows
Fasmuil
Fa
Planar flow Proposed by Rezende and Mohamed (2015), the planar flow can be expressed as in Eq. 11. It applies contractions or expansions in the direction perpendicular to the hyperplane w
0. Its Jacobian determinant computed in time O(D) in Eq. 12, using the matrix determinant lemma. In addition, we need to note that this flow not invertible for all values of u and w. When the derivative the activa- tion function 1²(-) is positive and bounded from above, w TTu sup, k(]] is sufficient to ensure invertibility.5.
be
Figure 5: Visualization of the first two dimensions of 2u. 2k. and (O, /) by FlowSUM-PLKKD on XSum. Both sub-figures demonstrate distinct multi-modal patterns.
In our code, perform transformation u u] and restrict the activation h(-) to be one of leakyrelu, rolu, and tanh to meet this condition.
[log (1 + exp (w 'u))

---

Autoregressive Flows The masked autoregres- (11) sive flow (MAF) (Papamakarios et al., 2017) was motivated by MADE (Germain et al., 2015), which is an autoregressive model for density estimation. MAF generalizes the conditional distribution to be Gaussian and generates data in recursive way in Eq. 18. Given a data point x, the inverse trans- formation can be performed in parallel as in Eq. 19. The Jacobian of the inverse transformation is lower- -rian-tr by design due to the autoregres- sive structure, hence its absolute determinant can be expressed as in Eq. 20. The set of functions {f far} are autoregressive neural networks fol- lowing the approaches in MADE.
f(z) = z + uh w z tb);
det J 1 h' (wT 2 2 w Tu
(12)
where {u, W E R RJ are free parameters and h(-) is smooth element-wist non- non-linear acti- vation function with derivative 1'(-).
Radial flow The radial flow (Tabak and Turner, 2013; Rezende and Mohamed, 2015) takes the form of Eq. 13. It applies radial contractions and expan- sions around a reference point. Similar to the planar flow, we can apply the matrix determinant lemma to calculate the Jacobian determinant in (D) time, in Eq. 14. To guarantee invertibility, usually require a
= u. expa; Mix
(18)
f(z) = zz ==+ ()(a, (z -z);
(13)
where 14g = Sus (x1:-1), = Sas (X1:1-1) and u1 (O.
det -1+ ag (1 + 3h) 31+
(14)
1.2
U4 (xz Mi) exp (-ai)
(19)
where E R is the reference point, 3 E IR, a E I+ are free parameters, || zo|| is the norm of 20. and h(a,r) cttt Sylvester flow The Sylvester flows (van den Berg et al., 2018) generalize the planar flows to have M hidden units, as in Eq. 15. To achieve better computational efficiency, van den Berg et al. (2018) proposes the parameterization as in Eq. 16. with which the Jacobian determinnant reduces to Eq. 17 and can be computed in (M). Simi- lar to the planar flows, when h-(-) is positive and bounded from above, R.R.a. sup, h(z)) for all i L {1....,)) D} is sufficient to ensure invertibility.
J- exp
Σa
az
(20)
Likewise, the inverse autoregressive flow (IAF)
(Kingma et al., 2016) uses MADE with Gaussian conditionals and generates data in Eq. 21. Its Ja- cobian determinant has simple form in Eq. 22. The main difference between IAF and MAF lies in the history variables. MAF uses previous data variables X1 to compute 14 and a;s, whereas IAF uses previous random variables u1:111 for the computation. In terms of sampling and density evaluation, IAF can sample in parallel and need to evaluate sequentially, whereas MAF has to sample sequentially and can evaluate in parallel. Since we more about the sampling efficiency in varia- tional inference, we choose IAF in the paper.
f(z)= = Uh W z + +b)
(15)
where {U R DXM, W E DDX R are the free parameters and h(-) is element-wise activation function.
I; Uz exp Oi + Pi:
(21)
f(z) QR (RQ"2 z + b))
(16)
where 14 fpc (u and a Sac (u1+-1).
det J det (IN diag (N' (fQ*Z+b)) R.R.
(17)
|dct. J = exp
cxp
Σ a
(22)
where R. and R are upper triangular M x M matri-
ces, and Q (qı ... qM) consists of an orthonor- mal set of vectors.
Affine Coupling The affine coupling layer, pro- posed in NICE (Dinh et al., 2015) and later gen- cralized in RealNVP (Dinh et al., 2017) takes the following form.
I6 our code, we perform a transformation on 9 : B + -a + log (1 + ed) to guarantec invertibility.

---

and coupling layers (such as NICE and RealNVP) in that it has both an analytic single-pass inverse and sufficient flexibility, as demonstrated in Durkan et al. (2019).
Vidd
F1:d
DdD1D:D & exp (s (x1.a) (d)
yd L:D
(23)
where 8 Rd RD and R R
K Example Analysis
are scale and translation transformation function respectively, and @ the element-wise product. Its Jacobian determinant can be efficiently com- puted as det J exp Σ s (I Since the computation does not involve the Jacobian of f, we can make these two functions arbitrarily complex and use neural networks to model them. The coupling layers are usually composed of per- mutation layers to ensure every component gets modified, and since the Jacobian determinant of permutation is 1, the Jacobian determinant remains tractable.
In this section, analyze several instances from CNN/Daily Mail and XSum, showcasing diverse outcomes generated by different summarization models."
Spline Coupling Neural spline flows (Durkan al., 2019; Dolatabadi al., 2020) use monotonic rational -uandrati splines or monotonie rational- linear splines as the coupling transformation to achieve more flexibility and yet remain differen- tiable and invertible. The monotonic rational- quadratic spline uses K + 1 monotonically in- K
( ) to set up K bins, creasing knots { (x.4), y each of which is defined as rational-quadratic function" 17 that is monotonically increasing. It maps B, B] to [-B, B) and defines the trans- formation outside the range to be identity trans- formation. Let 8k and {(x) (x- (2++1-2** x²), the rational- quadratic function the kth bin takes the form of Eq. 24 and the Jacobian determinant of the rational- quadratic neural spline flows (RQNSF) can be writ- ten as in Eq. 25.
(y 611 v*) / (a. x*)
(()())) (*) + 2(k)+ (8) -2(() (200))))000111---))00)00 () ((T(()
(24)
(E)
det
dr
(()
k
(a(k)) [ 1)++1)ee2 22
(1-E)2]
-ILx
18(+++)++) 25 <(1 {)]²
(25)
The rational-linear neural spline flows (RLNSF) work similarly, except with monotonically increas- ing linear rational functions in each bin. Neural splines combine the best of autoregressive flows
17 A rutiona--gaadratic function is defined as the quotient of two quadratic polynomial functions.
It is worth mentioning that few of the grammatical errors in the summaries can be attributed to the source text itself.

---

Original Text (truncated); It looks like an ordinary forest, with moss climbing up the walls and brown leaves covering the floor. But if you look closely, you will see that this picture is not all seems. For the peaceful scene actually features a carefully painted female model. The amazing illusion is the work of German body-painting artist Joerg Duesterwald, who spent hours painting his model so she would blend in with her surroundings. The stunning set of pictures was taken in forest in Langenfeld, Germany, yesterday. Mr Duesterwald has been painting for more than 20 years.
Gold Summary: The illusion is the work of German body-patd soddtypanning artist Joerg Duesterwald, who spent hours painting his model. Stunning set of pictures was taken front rockface in forest in Langenfeld, Germany, yesterday.
Stunning set of images was taken in a forest near Langenfeld, Germany, yesterday by body-painting artist Joerg Duesterwald. It looks like an ordinary forest, with moss climbing up the walls and brown leaves covering the floor. But, if you look closely, you will see that this picture is not all it seems. For the peaceful scene actually features carefully painted female model.
BART:
VEDSUM:
The stunning set of pictures was taken in forest in Langenfeld, Germany, yesterday. looks like an ordinary forest, with moss elimbing up the walls and brown leaves covering the floor. But, if you look closely, you will see that this picture not all it seems. For the peaceful scene actually features a carefully painted female model.
Amazing illusion is the work of German body-painting artist Joerg Duesterwald. He spent hours painting his model she would blend in with surroundings. Stunning set of pictures was taken in forest in Langenfeld, Germany, yesterday.
FlowSUM:
Table 12: Example 6766 in the CNN/Daily Mail test set: FlowSUM BART VEDSUM.

---

Original Text (truncated); UFC light heavyweight champion Jon Jones ran from a crash that hospitalised pregnant woman - but quickly came back to grab 'a large handful of cash' from the car, witnesses told police. According to police, the accident occurred in southeastern Albuquerque just before noon on Sunday local time when the driver of a rented SUV jumped red light. The driver, whom an off-duty officer identified as Jones, ran from the scene but then returned for the cash before fleeing again, police said. 'Witnesses stated he shoved the cash into his pants and ran north jumping the fence,' the report said. Officers found pipe with marijuana in the vehicle as well as MMA and rental car documents in Jones' name, according to the police report. Police were searching for UFC champion Jon Jones in connection with a hit-and-run accident. Albuquerque police were seeking an arrest warrant for Jones on Monday, They said he would likely face a felony charge of leaving the scene of an accident since the woman broke her arm in the crash. Police said in a news release they'd 'd been unable to reach Jones or his lawyer. However, Jones handed himself in later the same day, with TMZ reporting he was being held at Bernalillo County Metropolitan Detention Center. According to the warrant, the pregnant woman told police she was driving when she was hit by silver Buick SUV. ... Although he is widely considered the world's best pound-for for-pound mixed martial artist, Jones has endured legal problems and questionable behaviour as champion.
Gold Summary: UFC light heavyweight champion Jon Jones ran from a crash that hospitalised a pregnant woman, witnesses told police. According to police, the accident occurred in Albuquerque just before noon on Sunday when the driver of a rented SUV jumped a red light. The driver, whom an dff- officer identified as Jones, ran from the scene but then returned for the cash before flecing again, police said. Jones is widely considered the best pound-for-pound mixed martial artist.
BART: A Albuquerque police were seeking an arrest warrant for Jones on Monday. They said he would likely face felony charge of leaving the scene of an accident since the woman broke her arm in the crash. However, Jones handed himself in later the same day, withTMZ reporting he was being held at Bernalillo County Mctropolitan Detention Center.
UFC light heavyweight champion Jon Jones ran from a crash that hospitalised a pregnant woman. Witnesses said he returned for 'a large handful of cash' from the car. Albuquerque police were seeking an arrest warrant for Jones Monday. They said he would likely face felony charge of leaving the scene of an accident since the woman broke her arm in the crash. Jones handed himself in later the same day.
VEDSUM:
FlowSUM: UFC light heavyweight champion Jon Jones ran from a crash that hospitalised a pregnant woman. Witnesses said he came back to grab 'a large handful of cash' from the car, witnesses told police. The driver, whom an off-duty officer identified as Jones, ran from the scene but then returned for the cash before fleeing ugain, police said. Officers found pipe with marijuana in the vehicle as well as MMA and rental car documents in Jones' name, according to the police report.
Table 13:
Example 4627 in the CNN/Daily Mail test set: FlowSUM VEDSUM > BART.

---

Original Text (truncated); An Icelandic duo has created a snack that is made using cricket flour. Called the Jungle Bar it also contains dates, sesame seeds and chocolate. Cricket flour is said to be good source of protein and other nutrients. The duo hopes it will encourage people in the West to eat more insects. The Jungle Bar is being developed by Icelandic duo Búi Bjarmar Aõalsteinsson and Stefán Atli Thoroddsen through their company Crowbar Protein. On Kickstarter they are seeking $10,000 ($15,000) for the insect-powerts protein bar. They previously rose to fame with their Fly Factory, a micro-factory that used larvae to create foods including chocolate cake and pâté. Ingredients. Dates, sesame, sunflower and pumpkin seeds, chocolate and cricket flour. Nutrition information. The bar is 50 grams (1.7 ounces), 200 calories, contains 8 grams of high quality protein (16 per cent of the bar) and has a shelf life of 1 year. The duo say that insects are a largely untapped source of nutrients in the Western world, and they hope their product could spark a change in diet habits. To make the bar, the team has cricket flour sent in, from farm-raises crickets that have been ground down. It is then mixed with the other ingredients make the unusual bar.
Gold Summary: Icelandic duo has created a snack that is made using cricket flour. Called the Jungle Bar it also contains dates, sesame seeds and chocolate. Cricket flour said to be a good source of protein and other nutrients. The duo hopes it will encourage people in the West to eat more insects.
An Icelandic duo has created a snack that is made using cricket flour. Called the Jungle Bar it also contains dates, sesame seeds and chocolate. Cricket flour is said to be a good source of protein and other nutrients. The duo hopes it will encourage people in the West to eat more insects. On Kickstarter they seeking $10,000 ($15,000) for the insect-powered bar.
BART:
The Jungle Bar is being developed by Icelandic duo Búi Bjarmar Adalsteinsson and Stefán Atli Thoroddsen through their company Crowbar Protein. Cricket flour is said to be a good source of protein and other nutrients. The duo hopes it will encourage people in the West to eat more insects. To make the bar, the team has cricket flour sent in, from farm-raised crickets that have been ground down.
VEDSUM:
FlowSUM: An Icelandic duo has created snack that is made using cricket flour. Called the Jungle
Bar it also contains dates, sesame seeds and chocolate. The Jungle Bar is being developed by Búi
Bjarmar Adalsteinsson and Stefán Atli Thoroddsen through their company Crowbar Protein. On
Kickstarter they are seeking $10,000 ($15,000) for the insect-powered protein bar. They previously
rose to fame with their Fly Factory, a micro-factory that used larvae to create foods including chocolate cake and pâté.
Table 14: Example 4120 in the CNN/Daily Mail test set: BART VEDSUM > FlowSUM.

---

Original Text (truncated): The Office for National Statistics said industrial output fell 0.7% compared with January, when dropped 0.3%. Unexpectedly warm weather drove the change, because it led fall in electricity and gas demand, the ONS said. Construction output fell by 1.7% in February, down from a revised January reading of zero growth. The construction figure, the biggest drop in nearly a year, was mainly the result of a 2.6% fall in the housebuilding sector, Meanwhile, the UK's deficit in goods and services widened to Af3.7bn in February, from revised figure of Åc3bn in January, According to the ONS, the deficit was fuelled by what it called "erratic items", such imports of gold and aireraft. "The overall trade deficit worsened, but excluding erratic items, the picture improved, imports fell more than exports," " said ONS senior statistician Kate Davies. Howard Archer, chief UK and European economist at IHS Markit, called the figures "a disappointing package of data for the UK economy which fuels suspicion that GDP growth slowed markedly, largely duc to consumers becoming more cautious". He added: "We suspect UK GDP growth in the first quarter of 2017 slowed to 0.4% quarter-on-quarter from 0.7% quarter-on-quarter in the fourth quarter of 2016 this would be the weakest growth rate since the first quarter of 2016."
Gold Summary: Activity in the UK's industrial and construction sectors shrank in February, new figures show.
BART:
UK industrial output fell for the second month in a row in February, official figures have shown.
VEDSUM: Industrial output in the UK fell for the second month in a row in February, official figures have shown.
Activity in the UK's industrial and construction sectors shrank in February, according to official figures.
FlowSUM:
Table 15: Example 2924 the XSum test set: FlowSUM > BART VEDSUM.

---

Original Text (truncated); December, the government announced finalised plans for a cull, initially in pilot areas, as a way to curb the spread of tuberculosis cattle. In applying for judicial review, the Badger Trust says culling will not stop TB and may in fact help spread it. Other campaign groups are considering action under the Bern Convention, which protects European wildlife. The government's ' plans are likely to result in farmers funding contractors to shoot badgers in number of areas of England, with two initial pilots in west Gloucestershire and west Somerset taking place later this year. "We have identified some serious flaws in the way by which the Secretary of State [Caroline Spelman] reached her decision cull badgers," said Gwendolen Morgan of Bindmans solicitors, lawyer for the Badger Trust. "Given that Defra's proposals come an enormous cost to farmers, and threaten to prompt rather than prevent the spread of disease, we hope that this ill-conceived decision will be struck down by the court." She pointed to government projections that culling would reduce TB incidence by 12-16% over nine years.
Gold Summary: The Badger Trust has launched new legal challenge to the government's plans to cull badgers in England.
BART:
The Badger Trust has launched a legal challenge to the government's plans cull badgers in England.
VEDSUM:
The Badger Trust is taking legal action against the Department for Environment, Food and Rural Affairs (Defra) over plans to cull badgers in England.
The Badger Trust has launched a legal challenge to the UK government's plans to cull badgers in England and Wales.
FlowSUM:
Table 16: Example 5737 the XSum test set: BART > FlowSUM VEDSUM.

---

Original Text (truncated); The response from many in that time has been: "Let's get on with it." That view was shared by the First Minister Carwyn Jones until recently when he altered his opinion and said that we should only start the official Brexit negotiations in the early part of next year. My sense is that the public will be flexible on the timing up to a point, as long as they are given clear sense of direction. The majority of the political establishment have had to come to terms with the fact that most people ignored their advice to remain. So much for being in touch with the electorate. In conversations with politicians on the remain side since, have come across mix of bewilderment, frustration and sadness. And while people like spend lot of time talking and writing about this subject least, Wales was carbon copy of England. In stark
Welsh political dynamic, contrast, those that supported leaving feel vindicated by their campaign, and now believe they are the ones in touch with vast swathes of the population. The referendum result was devastating indictment the effectiveness of the billions of pounds of EU funds spent trying to regencrate economically deprived communities. The brutal reality is that those who were most likely to vote to leave lived in communities where most BU money had been spent. It is an extraordinary paradox that raised eyebrows far further afield than Wales.
Gold Summary: It has been month since Wales voted to leave the European Union.
BART: has been more than year since the UK voted to leave the European Union
VEDSUM: It has been a year since the EU referendum result, and in that time have spent a great deal of time talking to politicians on both sides of the political spectrum about what they think about Brexit.
Since the referendum result on 23 June, have spent lot of time talking about the implications for Wales and the Welsh political establishment.
FlowSUM:
Table 17: Example 9512 in the XSum test set: BART > VEDSUM FlowSUM.


Reviews:
[{"review_id": "vJazuKoRve", "reviewer": "Reviewer_6vj4", "summary": "", "questions": "none", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nThank you for drawing attention to the overlooked reference. We highly value your thorough review of our manuscript. The reference you highlighted sheds light on an alternative approach to variational summarization—one that emphasizes unsupervised learning and explicit control over summary length using VAEs. This perspective represents a growing avenue in variational summarization, gaining momentum alongside the expanding availability of diverse large training corpus. Thank you for your recommendations. We will incorporate your invaluable feedback to enhance our work.\n\nBest regards,", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- great motivation, well outlined\n- well written paper\n- extensive experiments on multiple datasets\n- good ablation studies", "reasons_to_reject": "none", "questions_to_authors": "none", "paper_topic_and_main_contributions": "The authors introduce FlowSUM, a transformer-based encoder-to-decoder summarization framework that utilizes normalizing flows. They show the competitiveness to state-of-the-art summarization models and superiority over direct baselines. They also show the applicability of their model to knowledge destillation", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "https://arxiv.org/abs/1809.05233 seem to use VAEs for summarization but in a different way", "Typos_Grammar_Style_and_Presentation_Improvements": "didn't notice any"}, "extra_text": {}}, {"review_id": "4MlJRTflpU", "reviewer": "Reviewer_FZCB", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for pointing out the clarity concerns related to the figure. In the submitted manuscript, the NF latent module is colored in purple, encompassing the chain from $\\overline{x}$ to $z_K$. To address your concern, we intend to enhance the figure's clarity by assigning distinct colors to the three crucial components: the NF latent module in purple, the Transformer-based encoder-decoder (including $x_i$, Encoder Block, $x_i^{\\prime}, y_j$, Decoder Block, $h_j$, and LM Head Layer) in green, and the refined gate mechanism (comprising Refined Gate and $h_j^{\\prime}$) in orange. Additionally, we'll include clear figure captions to provide further clarity on the model architecture. We genuinely appreciate your diligence in reviewing our content and are dedicated to delivering a more refined version based on your insights.\n\nBest regards,", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The use of normalizing flows for variational summarization is a unique and promising idea, addressing important challenges in the field.\n2. The introduction of a new training strategy with an improved gate mechanism is a valuable contribution to addressing posterior collapse during training.\n3. The experiments are abundant.", "reasons_to_reject": "1. Figure 1 is not clear which does not contain the NF latent module as described in the content. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces \"FlowSUM,\" a novel variational encoder-decoder framework for Transformer-based summarization that addresses two key challenges: insufficient semantic information in latent representations and posterior collapse during training. To overcome these issues, the authors employ normalizing flows to enable flexible latent posterior modeling and propose a controlled alternate aggressive training (CAAT) strategy with an improved gate mechanism. Experimental results demonstrate that FlowSUM significantly improves the quality of generated summaries and facilitates knowledge distillation without a substantial increase in inference time. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tdmvSi5gD7", "reviewer": "Reviewer_BWR7", "summary": "", "questions": "See Reasons To Reject.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your comprehensive feedback.\n\nRegarding your first comment, the evidence lower bound (ELBO) is a surrogate for log-likelihood in parameter estimation within variational inference. Due to the intractability of $p(y | x)$, which prevents the direct use of maximum likelihood estimation for parameter estimation, we introduce a parametric distribution, the variational posterior $q(z | x, y)$, as an approximation to the true posterior $p(z | x, y)$.  As indicated in Appendix B, the KL divergence between these two distributions equates to the log-likelihood minus the ELBO. Hence, ELBO functions as a lower bound for log-likelihood, and the fidelity of $q(z | x, y)$ approximating $p(z | x, y)$ determines the efficacy of ELBO as a surrogate. We will update Section 2.2 to make the underlying idea clearer to the readers. Your observation regarding references for Variational Encoder-Decoders (VEDs) is duly noted. In addition to those cited in the Introduction, we plan to incorporate relevant references into Section 2.2, including Zhang et al., 2016; Serban et al., 2017; Zhou and Neubig, 2017; Shen et al., 2018, etc.\n\nTo address your second comment, we intend to use distinct colors for the three key components. Specifically, the NF latent module, which models the variational posterior, will remain purple, the Transformer-based encoder-decoder (including $x_i$, Encoder Block, $x_i^{\\prime}, y_j$, Decoder Block, $h_j$, and LM Head Layer) will be in green, and the refined gate mechanism (including Refined Gate and $h_j^{\\prime}$) will be highlighted in orange. Furthermore, we will enhance the figure with detailed captions to clarify the model architecture.\n\nIn response to your third comment, the backbone models are indeed applicable within the conventional VED framework. Notably, the integration of Pretrained Language Models (PLMs) within the traditional VED context aligns with a specific instance of FlowSUM, where the normalizing flow layer count $K = 0$.  Our earlier submission referenced this model as VAESUM and demonstrated its performance alongside FlowSUM. Given the underemphasized connection with FlowSUM, we plan to clarify this relationship in Section 3.1 and rename the model to VEDSUM for clarity.\n\nRegarding your fourth point, despite incorporating normalizing flows and the refined gate mechanism, the model architecture aligns with the traditional VED framework. Another difference lies in the proposed controlled alternate aggressive training (CAAT) strategy related to posterior collapse. Additionally, in terms of incorporating $y$ into the variational posterior, we choose to assume $q(z | x, y) = q(z | x)$ for the following reasons. Firstly, this assumption is grounded in the nature of summarization, where $y$ can be viewed as a condensed form of $x$ and hence it is sensible to assume all the information in $y$ is contained in $x$. Secondly, as evidenced by Zhang et al. 2016, it is plausible to condition the posterior on both $x$ and $y$. However, their approach suffers from difficulties during prediction. In prediction, the target text $y$ is not accessible, making it hard to sample from $q(z | x, y)$. Zhang et al. 2016 suggest taking the prior’s mean as the latent code, but in our paper, the prior is a Gaussian, and taking such a strategy would diminish the benefit of using normalizing flows. Thirdly, it has been shown empirically by Bryan Eikema & Wilker Aziz, 2018 that by restricting the conditioning of the posterior to $x$ alone, their model achieves higher accuracy. Therefore, we consider $q(z | x, y) = q(z | x)$ as our modeling strategy.\n\nThank you for pointing out typographical errors and abbreviations in the manuscript. We are committed to addressing these issues and ensuring abbreviations are clearly defined. We highly appreciate your insightful feedback for enhancing our paper's clarity and quality.\n\nBest regards,\n\n### References:\n\nZhang, Biao, et al. \"Variational Neural Machine Translation.\" *Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing*. 2016.\n\nSerban, Iulian, et al. \"A hierarchical latent variable encoder-decoder model for generating dialogues.\" *Proceedings of the AAAI conference on artificial intelligence*. Vol. 31. No. 1. 2017.\n\nZhou, Chunting, and Graham Neubig. \"Morphological inflection generation with multi-space variational encoder-decoders.\" *Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection*. 2017.\n\nShen, Xiaoyu, et al. \"Improving variational encoder-decoders in dialogue generation.\" *Proceedings of the AAAI conference on artificial intelligence*. Vol. 32. No. 1. 2018.\n\nEikema, Bryan, and Wilker Aziz. \"Auto-encoding variational neural machine translation.\" *arXiv preprint arXiv:1807.10564*(2018).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposed an NF-based VED framework with an aggressive training strategy for summarization tasks.\n2. Extensive experiments were conducted to reveal the effectiveness of the proposed method.", "reasons_to_reject": "1. What are the potential meanings in Eq. (2) for ELBO? Corresponding references to support the backbone of the proposed framework are required.\n2. Since the VED is the main framework in the proposed method, it is required to denote which parts in Figure 1 are variational posterior, variational prior, and encoder-decoder structure, respectively, making a better understanding.\n3. Based on PLMs, the backbone of the proposed method, VED, seems could be directly applied for summarization tasks, how about its performance?\n4. Unless NF and gate mechanism, what’s the main differences between the proposed VED and other VED frameworks? How about the performance of the model that introduces y into variational posterior instead of regrading p(z|x,y)=p(z|x)? Refer to reference: Variational Neural Machine Translation, EMNLP-2016.\n5. Typos: (1) The paper said $six sections$ in line 127, but only 5 sections are presented; (2) Abbreviate forms should be defined when the corresponding phrases first appear and be capital letters, i.e., lines 163, 285, 1041.\n", "questions_to_authors": "See Reasons To Reject.", "paper_topic_and_main_contributions": "This paper proposed an NF-based VED framework with an aggressive training strategy for summarization tasks. Extensive experiments were conducted to reveal the effectiveness of the proposed method. However, there are still something need to be improved.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "See Reasons To Reject.", "Typos_Grammar_Style_and_Presentation_Improvements": "See Reasons To Reject.", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces "FlowSUM," a novel variational encoder-decoder framework for Transformer-based summarization that solves two major problems: posterior collapse during training and a lack of semantic information in latent representations. To address these problems, the authors suggest a controlled alternate aggressive training (CAAT) technique with a better gate mechanism and use normalizing flows to enable flexible latent posterior modeling. According to experimental findings, FlowSUM greatly enhances the quality of generated summaries and speeds up knowledge distillation without dramatically lengthening inference times. 

Pros:
1. The paper is well-written
2. The approach is novel
3. Extensive experiments that show that the proposed methodology works fine
4. Good ablation study 

The authors addressed all comments (cons) that were raised by the reviewers in rebuttal.
Decision: Accept-Main