MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Space
Authors: Hanxing Ding
Liang Pang
Zihao Wei
Huawei Shen
Xueqi Cheng
Tat-Seng Chua
Venue: EMNLP 2023 Findings (2023)
Keywords: Natural Language Generation
Controllable Text Generation
Pretrained Language Models

Abstract: Multi-aspect controllable text generation aims to generate fluent sentences that possess multiple desired attributes simultaneously.
Traditional methods either require expensive iteration / searching within the discrete text space during the decoding stage, or train separate controllers for each aspect, resulting in a degradation of text quality due to the discrepancy between different aspects. To address these limitations, we introduce a novel approach for $\textbf{M}$ulti-$\textbf{a}$spect $\textbf{c}$ontrol, namely MacLaSa, that estimates compact $\textbf{La}$tent space for multiple aspects, and performs efficient $\textbf{Sa}$mpling with a fast sampler. To eliminate the domain discrepancies between different aspects, we first utilize a variational autoencoder (VAE) network to map text sequences from various data sources into close latent representations. The estimated latent space enables the formulation of joint energy-based models and the plugging in of arbitrary attribute discriminators to achieve multi-aspect control. Afterwards, we draw latent samples with a fast sampler based on ordinary differential equations and feed sampled examples to the VAE decoder to produce target text sequences. Experimental results demonstrate that MacLaSa outperforms strong baselines on both attribute relevance and textual quality while maintaining a high inference speed.

MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Space
Hanxing Dingl.2 1.2 Liang Pangl* Zihao Wei!.3 Huawei Shen
Xueqi Cheng Tat-Seng Chua
Instiitute of Computing Technology, Chinese Academy Sciences
2 University of Chinese Academy of Sciences
Sea-NExT Joint Lab, National University of Singapore
{dinghanxing188, pangliang, weizihao22z, shenhuawei, cxq)eict.ac.oo dcsctsênus, edu. sg
Abstract
Build Space
Find Target
Opeimizatinn saaos-dee Method
Multi-aspect controllable text generation aims generate fluent sentences that possess mul- tiple desired attributes simultaneously. Tradi- tional methods either require expensive itera- tion / searching within the discrete text space during the decoding stage, or train separate controllers each aspect, resulting in a degra- dation of text quality due to the discrepancy between different aspects. To address these limitations, introduce novel approach for Multi-aspar aspect control, namely MacLaSa, that estimates compact Latent space for multiple aspects, and performs efficient Sampling with a fast sampler. To climinate the domain discrep- ancies between different aspects, we first uti- lize variational autoencoder (VAE) network to map text sequences from various data sources into close latent representations. The estimated latent space enables the formulation of joint energy-based models and the plugging in ar- bitrary attribute discriminators to achieve multi- aspect control. Afterwards, we draw latent sam- ples with a sampler based on ordinary ferential equations and feed sampled examples to the VAE decoder to produce target text se- quences. Experimental results demonstrate that MacLaSa outperforms strong baselines both attribute relevance and textual quality while maintaining high inference speed.
Datarted Space
Iteration
Coastraint
(LM)
Prefis
Prefix- Method
Prefix
Prils
Prafie
Outlier
anl One T4
Our Method
Clasdifeer
Compact
Space
(VAR)
Figure 1: comparison of existing methods and our approach. Top: optimization based methods perform iteration searching in the distorted text space. Middle: prefix-based methods fuse multiple prefixes to obtain interpolation or average of these distribution centers. Bottom: our framework estimates compact latent space for better controllability and performs efficient sampling with a fast ODE-based sampler.
multi-aspect controllable text generation' (Kumar et al., 2021; Qian et al., 2022; Qin et al., 2022). For instance, dialogue system may require the control of emotions, persona, politeness, etc, at the same time. However, training multi-aspect controllers directly is difficult due the limited availability of sentences with multi-attribute anno- tations. Thus, recent works focus on training sepa- rate single-aspect discriminators or controllers for each aspect and combining them for multi-aspect controllable text generation (Mireshghallah et al., 2022; Qian et al., 2022).
1 Introduction
Attribute-ban -ased controllable generation aims to gen- erate text that exhibits desired attributes in cer- tain aspects (Zhang et al., 2022). Early work fo- cused on single-aspect control tasks and involved re-training tneettnnng language models (LMs) using well-labeled data, which resulted in good per- formance (Keskar al., 2019; Chan et al., 2021; Hao et al., 2021; Hu et al., 2017; Ficler and Gold- berg, 2017; Xiao et al., 2021). Recent studies fo- cus on a more challenging and practical setting.
As illustrated in Figure 1, recent works on multi-aspect controllable text generation task can be primarily categorized into two types. Firstly,
'The aspect can be sentiment or topic, and sentiment may have two attributes: positive and negative.
"Corresponding author

---

optimization-base methods either apply extra at- tribute classifiers to adjust the conditional prob- ability distributions of language model at every generation step (Dathathri et al., 2020; Krause et al., 2021; Yang and Klein, 2021), or regard the decoding process as an optimization objective and search for optimal soft-representations that satisfy multi-objectivv constraints (Kumar et al., Qin et al., 2022; Mireshghallah et al., 2022). However, from a distributional perspective, ciamiization-based methods often conduct com- plicated gradient-descent iterations or searching in the distorted text space, and the discrete na- ture makes it difficult to find high-quality texts, leading to poor linguistic quality and slow infer- ence speeds. Secondly, prefix-based methods are introduced to guide conditional generation using lightweight continuous task-specific vectors (Qian et al., 2022; Yang ct al., 2022). They typically train single-aspect prefixes separately and suffer from text quality degeneration when combining them for multi-aspect control due to the mutual interference between multiple prefixes. As depicted in Figure I, prefix- methods combine multiple prefixes to obtain the interpolation or average of these dis- tribution centers appraised by prefixes. However, there could be a mismatch between interpolation points and target intersection regions when the dis- tribution centers of different aspects are far away, leading to the degradation of textual fluency. There- fore, an ideal method for multi-aspect controllable generation should enhance controllability and tex- tual quality, while enabling rapid inference speeds.
from the distribution formed by multiple attribute classifiers. Ultimately, the selected latent vectors are input into a VAE decoder to generate target text sequences. In short, our approach improves control- lability and textual quality by estimating compact latent space to mitigate mutual interference among various aspects, and the fast ODE-based sampler contributes to efficient sampling.
We conduct experiments on the multi-aspect con- trol task with two attributes from the sentiment as- pect and four attributes from the topic aspect, with datasets IMDb movie reviews (Maas et al., 2011)
and AGNews (Zhang et al., 2015), respectively. Experimental results of both automatic and human evaluation demonstrate that our method achieves encouraging improvements in attribute relevance and text quality compared to previous strong base- lines. Our work also exhibits significant advantages in inference speed over existing baselines?.
2 Related Work
In this section, we discuss the related work on multi-aspect control. Recent researches on multi- aspect can be divided into two types: optimization- based methods and prefix-based methods.
Existing efforts on multi-aspect control typically combine many at- tribute controllers in the decoding stage to bias the language model for desired directions. Weighted- decoding methods focus on decomposing condi- tional probability through Bayesian factorization into language model and classifier (Dathathri et al.. 2020; Krause al., 2021; Yang and Klein, 2021; Liu et al., 2021; Gu et al., 2022a; Hallinan et al., 2023). Other approaches define controllable text generation as multi-objectivee optimization problem and find the optimal soft-representation sequences by specific sampling schemes or other gradient-based samplers (Lample et al., 2018; Bhat- tacharyya et al., 2021; Mireshghallah al., 2022; Qin al., 2022; Kumar et al., 2021, 2022). These optimization- based methods often require compli- cated iteration search in the nigh-dimensiona text space, leading to slow inference speed.
Optimization- Methods
In this paper, we introduce a new technique for multi-aspect controllable text generation, dubbed MacLaSa, which estimates compact space con- taining latent representations of various attributes and performs effective sampling using fast sampler based on ordinary differential equations (ODEs). To eliminate the domain discrepancies be- tween different aspects, we initially employ a VAE encoder network to map attribute-related sentences into latent representations and penalize the distance between each pair of aspect distribution centers. The acquired compact latent space aids in formulat- ing joint latent-space energy-based models (EBMs) and allows integrate arbitrary attribute dis- criminators to satisfy multi-aspect combinations. Subsequently, we utilize an efficient ODE-basee sampler (Song et al., 2021; Nie et al., 2021) to draw latent samples possessing desired attributes
Prefix-based Methods Recent work leverages the learned continuous task-specific vectors, which are called prefixes, as a lightweight alternative to guide the language model to generate desired
"Our code is available at https://github.co comb TrustedLLM/MacLaSaa

---

attribute text (Li and Liang. 2021; Yu et al., 2021: Zhao et al., 2020; Qian et al., 2022; Yang et al., 2022; Huang et al., 2023). Contrastive Pre- fixes (Qian et al., 2022) utilize the opposite re- lationship between different attributes to help to train singlc-aspect prefixes and combine them for multi-aspect control. Tailor (Yang et al., 2022) pro- vides multi-aspect prefix mask and re-indexing position-ids sequence to bridge the gap between sin- gle and multi aspect control. Nevertheless, these learned controllers in prefix-based methods may prefer different language habits, resulting in tex- tual quality degeneration when combining them for multi-aspect control.
multi-aspect sequences.
3.1 Task Definition
First, we present the task definition of multi-aspect controllable text generation. Suppose we have N aspects, represented by A {AA...... AN }. where each aspect A, contains |An attributes, given by {a/.. an [lll). The goal of multi-aspect control is to generate sentences that possess mul- tiple attributes = {ai, ... ; . simultaneously. For instance, we may expect our model to produce sentence with attribute of (from aspect A1) and attribute a (from aspect A2).
Our training samples are organized and labeled according to their corresponding aspects and at- tributes. S% denotes the index set of sentences with attribute az. As result, we have S, = 0 Aud Sh. which represents the index set containing all sen- tences within aspect An- Likewise, S U S, signifies the indices encompassing our entire train- ing dataset. We use x to represent an arbitrary sentence and 2 to indicate its latent representation. It is worth noting that our training corpus con- tains only single-aspect labeled sentences, making it infeasible to directly train multi-aspect control- lable text generative model.
There is also a line of work that manipulates la- tent variables in the latent space (Gu et al., 2022c,b; Liu et al., 2022). Gu et al. (2022c) map attribute- related sentences to the latent space and then de- signs a heuristic searching algorithm to approach intersection regions of the different attributes for generation. Despite their efficiency, they still suffer from the unstable controllability due to the rare intersections of diffferent attributes. LatentOps (Liu al., 2022) executes composable control opera- tions within the low-dimensional continuous latent space. However, it does not adequately consider the discrepancy between various aspects, resulting in suboptimal performance when controlling multiple attributes simultaneously.
3.2 Building Latent Space
To estimate a compact, continuous latent space that outlines the latent distribution of interest and facilitates subsequent sampling processes, we uti- lize a VAE network equipped with pre-trained lan- guage models to encode any single-aspect sen- tence x to its hidden representation = using z Encoder (z). The encoded latent representations constitute the estimated attribute space.
3 Methodology
In this section, we first present the task definition of multi-aspect controllable text generation ($3.1) I). Next, we describe how to build the compact latent space ($3.2), how to define the joint EBMs on the latent space ($3.3), and how to sample from the EBMs to generate the final results ($3.4).
We expect the latent space to be sufficiently com- pact while ensuring that latent representations from various aspects maintain their semantic meanings. To accomplish this, propose the following three training objectives:
The overall structure of MacLaSa is illustrated in Figure Our approach primarily relies on the variational autoencoder architecture for manipulat- ing latent spaces. To weaken the mutual interfer- ence among different aspects, we initially employ the VAE encoder to estimate a continuous low- dimensional latent space, incorporating additional losses to ensure its compactness. Subsequently, we establish joint aatent-space energy-basee mod- integrate multiple constraint
ELBO Loss LE We adopt the basic Evidence Lower Bound (ELBO) objective to learn smooth latent space and force the decoder to map any given latent vector into its original text
Le B (xls)
po (x]=)) + (L (2]))))l)e)r ll (=)),
els, which allow functions for guiding sophisticated multi-aspect control. Finally, we utilize a fast ODE-b sam- pler to draw samples from the EBMs and input them into the VAE decoder to generate the desired
where Pprior (z) ) is standard Gaussian distribution as the prior, and (T.(-Il-) is the Kullack-Leibler di- vergency. The first term encourages z to encode more relevant content information for reconstruct-

---

Build Latent Space
Formulate Joint Latent-Space EBMs
Text Space
E
Aspect
Aspect Ag
VAE Encoder
VAE Decoder
En (agl2) (o log Σ exp((x())aaa)
F3
Sample with ODE
Chasifiiation Loss Ag
dx B()) Σ
Aspect Discrppnccy Loss Ap
Sample from PLAN
Multi-
uspect
Decoder
Latent Space
z(T
z
Sentence
Figure 2: An overview of MacLaaSa. Left: Build latent space for MacLaSa. We utilize the VAE framework with additional losses to build compact latent space. Top Right: Formulate joint EBMs. We formulate the latent-space EBMs latent representation and attribute to facilitate plug of multiple attribute constraint classifiers. Bottom Right Sample with ODE. We adopt fast ODE-based sampler to perform efficient sampling from the EBMs, and feed samples to the VAE decoder to output desired multi-aspect sentences.
ing the original text with the VAE decoder po- The KL divergence forces the variational distribu- tion (0(2(x) match the prior.
centers of other aspects. Minimizing Ln allows the model reduce the discrepancy between dif- ferent aspects, and helps eliminate the mutual interference among them.
Classification Loss Lc We propose the classifi- cation loss Lc force the mapped representations to preserve their original attribute information and help the model to distinguish representations of different attributes from the same aspect. We in- troduce independent classification layers for each aspect and train them by minimizing the negative log-likelihood of the corresponding attribute a'::
Totally, our learning objective is:
wiLF + waCo wyLn.
(4)
We update parameters d. 0 and { for the encoder, decoder, and the classifier layers.
3.3 Formulating Joint Latent-Space EBMs
In order to satisfy the requirement of controlling multiple attributes simultaneously, we leverage the compositionality of EBMs and formulate the joint distribution for the latent representations and tar- get attribute by incorporating any constraint(c.g. attribute classifiers) into the energy function E(-)- To begin with, we define the following joint dis- tribution on both the latent representation a and desired attributes a as:
[A..
Lc
log Pea (0% 1
where Pa, is classifier that distinguish attributes {aA) from aspect A, with parameter TT-
Aspect Discrepancy Loss Li To reduce the dis- tribution discrepancy between different aspects, we introduce the aspect discrepancy loss (Gu et al., 2022c) to penalize the distance between distribu- tion centers of each two aspects:
p(z, a) Ppta(())p(aaa) Pp)n(()) F(al)) /Z, (5)
where = -- the normalization term. Ppeior is the Gaussian prior distribution, and p(a|z) follows Boltzmann distribution. In this work, We assume the target attributes are indepen- dent with each others. We then formulate F(a|2) as the energy-based models that combine arbi- trary attribute classifiers based our needs:
CD Σ
Σ SES
Σ JES
(3)
SSn
SSngl
which calculates the Euclidean distance between distribution centers. practice, we use batch- level approximation by taking the average repre- sentations each aspect in each mini-batchh as the estimated center and calculating the distances to
N
E(als) Σ
Xo Er, (0)]2))
(6)

---

An E IR is the balanced weight to balance the per- formance among attributes from different aspects, The energy function En (=) is defined as the negative log probability of target attribute of:
drawing z(T) (O, and solving the (1) with a differential neural ODE solver (Chen al., 2018) to obtain z0)). Then x(0) is fed to the VAE decoder po to produce target text sequences that possess multiple attributes simultaneously.
bin (a)l2) = (x) [ch] - log Σ exp (fm(#) (
To narrow the inevitable gap between the prior distribution Pprior (2) and the learned VAE posterior (=(=)²)) on Z, following previous work (Li et al., 2020; Hu and Li, 2021; Liu et al., 2022), we fit a simple single-layer generative adversarial network (GAN) (Goodfellow et al., 2014), PGAN(2), on the learned latent space and draw 2(7) from PGAN(2). We study the impact of PGAN in $4.5.
(7) where [2) is the multi-class attribute classifier trained on the frozen latent space, and fn (2)
is the output unnormalized logits for attribute a. After the training of VAE, we fix the entire VAE encoder and map the input text with attribute anno- tations into the latent space, then ask the classifier to predict target attribute label given the latent vec- tor. Training attribute classifiers fn (z) ) in the frozen low-dimensional latent space is efficient, which en- ables us to plug in different attribute classifiers to guide complex multi-aspect control.
Experiments
In this section, demonstrate the effectiveness of our proposed MacLaSa in the multi-aspect control setting through both automatic and human evalu- ations. Additionally, we provide further analysis and visualization on efficiency, and case studies.
3.4 Sampling from EBMs with ODE
After the acquisition of the joint distribution p(z, a), we would like to draw latent representa- tions given the target attribute values a. To en- sure high-quality and efficient sampling, we adopt a fast ODE- sampler to draw samples from the energy based models.
4.1 Experimental Setups
Datasets We conduct experiments for controlling two aspects: sentiment and topic, simultaneously. We adopt the LMDb movie reviews (positive and negative) (Maas et al., 1) for sentiment control and AGNews dataset (World, Sports, Business and Sei./Tech) (Zhang al., 2015) for topic control. Following previous work (Qian al., 2022; Gu et al., 2022c), we randomly sample 20k sentences from each dataset for each attribute to train our method. For evaluation, consistent with previous work (Dathathri et al., 2020; Krause et al., 2021; Yang and Klein, 2021: Liu et al., 2021; Gu et al., 2022a), we choose the same 15 attribute-unrelatd prompts and ask the model to complete 50 sen- tences with the desired attributes starting with each prompt.
Prior work (Song et al., 2021) shows that control- lable generation p(z/aa a)) can be achieved by solving
the following ordinary differential equation (ODE):
dx - 8(t) [a + v, log a)) dr,
(8)
where B(t) is a time-variant diffusion coefficient that has the form ((t) - Buin + (Amax Smin) t. t is the timestep from T to 0, and pr a) denotes the join distribution of data and attribute at time t.
In our work, we adapt the ODE from (8) into the low-dimensional latent space, which gives:
dz = 3() |a + V. pog, at
2 3(0) lz V (a(z)) + (og)) dt.
MacLaSa Settings For the proposed MacLaSa, employ BERT-base and GPT-2 medium ini- tialize the encoder and decoder networks in VAE, respectively. The dimension of the latent space is 128. We also apply a cyclical schodule for KL weight and a KL thresholding scheme to allevi- ate the notorious KL vanishing issue (Bowman et al., 2016). During the training stage, we use the AdamW (Loshchilov and Hutter, 2017) optimizer with a learning rate of 8e-5. The number of train- ing epochs 50. We also randomly select 10k lk
Note that (z) = (O, is time-invariant for t E [0, 7]. Since the classifier fx(=) in (7) is fixed, Er(a]z) is also time-invariant and we have [x(a = E(alz). The above ODE becomes:
de ! p(t) v.E(a ! 1lell dt !
= 7 [(tt)=+((alz)) de
(10)
= - VannE dI.
Now we can easily sample latent samples by
htt hhttts://github.comm/eet

---

examples to train validate attributes classifiers in the latent-space EBMs. our experiments, wi, w2 and w3 are set to 1. During the inference stage, we set 5min - 0.1 and fmax - 20 for the timevvariant diffusion coefficient/ 3,. We also manually tune the weight A of different attributes to balance them. All experiments are conducted on a single NVIDIA V100 32GB GPU.
and Distinct-2 in our paper.
Human Evaluations In addition to automatic evaluations, we conduct human evaluations com- pare our method's performance with that of the baseline models. We enlist four annotators with high-level language skills to carry out the human evaluation. Annotators are instructed to assess at- tribute relevance, fluency, and diversity on scale of 1 1-5, with denoting "very low" and 5 represent- ing "very high." Morcover, we direct the annotators not to consider linguistic quality when evaluating attribute alignment and vice versa. We randomly select 800 generated sentences (100 for each com- bination) and shuffle them for evaluation with each method. The scores are then averaged to derive the final human evaluation results.
4.2 Baseline Models
We compare with three types of baseline models: (I) optimization-baseed methods: PPLM (Dathathri al., 2020) prok-propagates gradients of extra attribute classifiers to guide conditional genera- tion at every decoding step. DEXPERTS (Liu et al., 2021) reweights the predictions of language models based on expert (and anti-expert) opinions for effective attribute control. MaRCo (Hallinan et al., 2023) achieves controllable generation using likelihoods under expert LM and a anti-expert LM find candidate words to mask and replace. Mix&Match (Mireshghallah et al., 2022) uses a Hutropolis-Hasttnn sampling scheme to draw sam- plers from an energy-based model that combines multiple attribute discriminators. (2) Prefix-based methods: Contrastive Prefixes (abbreviated as Con- trastive) (Qian et al., 2022) trains prefixes for each aspect while the combination of them can achieve multi-aspect control. We also compare with re- cent approaches that manipulate the latent space, including: LatentOps (Liu et al., 2022) performs composable text operations in the low-dimensional latent space, and Distribution (Gu et al., 2022c) searches for the intersection areas of multiple at- tribute distributions for generation.
4.4 Main Results
Automatic Evaluations We conduct experi- ments in the two-aspect control setting and com- pare our method with several strong baselines. The results of automatic evaluation are depicted in Ta- ble 1. We calculate the average correctness scores of eight attribute combinations as the final results for each method. We also report the standard de- viations, which stand for the stability of models among different runs. Moreover, we assess the av- erage inference time required to generate a single sentence for each method.
We note that existing baselines excel in individ- ual evaluation metrics but struggle to concurrently achieve good controllability and superior linguistic quality, which is essential for multi- aulti- control. PPLM and MaRCo can generate fluent sentences but fall short in attribute accuracy. In contrast, Mathh demonstrates strong attribute control- lability, yet the text quality is subpar. Moreover, optimization-based methods, including PPLM and Mix&Match exhibit severe slow inference speeds due to their complex iterations or searching in the nigh-dimensional text space. The Contrastive method attains a high correctness score in multi- aspect control by training separate continuous pre- fix vectors for each aspect. However, the mutual in- terference of different prefixes results in diminished text quality. LatentOps has average performance over baseline models. The Distribution method generates highly fluent texts with good attribute correctness scores but lacks textual diversity. MacLaSa showcases notable performance boost in average correctness scores, achieving an
4.3 Evaluation Metrics
Automatic Evaluations We adopt three auto- matic evaluations metries measure the perfor- mance on the two-aspect control task. Correctness evaluates the success rate of controlling the two as- peets simultaneously. We finetune two RoBERTa- Large (Liu et al., 2019) discriminators on the IMDb dataset for sentiment aspect, and the AGNews dataset for topic aspect. We use the two attribute discriminators to compute the fraction of sentences that contain pre-specified attributes. Perplexity (PPL) is an automatic metric of text fluency. We feed generated test sentences to GPT2-Large model and report the perplexity score. Distinct- ness (Li et al., 2016) is a n- n-gram--aase based metric for evaluating textual diversity, we report Distinct-1

---

Correctness (%)
Text Fluency
Diversity
Efficiency
Method
Senti. & Topic Acc. 1
PPL 4
Distinct-1 1
Distinct-2
Time (8)
optimizaaiinn based method
PPLM
18.14 0.45
25.59 + 1.09
0.23
0.64
40.56
DEXPERTS
23.93 + 1.11
38.70 = 2.51
0.23
0.70
0,64
MaRCo
27.81 1.94
18.87 = 1.85
0.18
0.58
0.40
Mix&Match
50.17 2.07
68.72 = 0.97
0.36
0.84
164.60
prefix-based method
Contrastive
53.02 + 1.52
52.56 + 11.97
0.22
0.71
0.59
method that manipulates latent space
LatentOps
44,41 1 5.72
26.11 +: 1.46
0.16
0.55
0,10
Distribution
49.79 1.99
12.48 1: 0.52
0.08
0.28
0.04
MacLaSa
59.18 1 0.81
28.19 1 1.26
0.16
0.60
0,10
w/o Le
47.54 12.67
27.91 +: 1.10
0.15
0.57
0.10
w/o Lp
51.18 1: 3,90
28.49 1 0.44
0.18
0.62
0.10
Table I: Automatic results multi-aspect control. We average the correctness scores of cight combinations(two sentiment attributes x four topic attributes) as the final results for each method. Detailed results of each combination are listed in Appendix A.
Method
Correctnesst
Fluencyt
Diversity
Correctness (%)
Text Quality
Method
PPLM
1.96
2.67
2.54
Sentiment & Topic
PPL.
DEXPERTS
1.98
2.38
1.88
Random
13.24
32.28
MaRCo
2.08
2.78
2.65
I.D
26.93
6.70
Mix&Match
1.21
1.38
2.13
ODE (MacLaSa)
58.
26.86
Contrastive
2.04
2.29
2.38
w/o PGAN
37.82
36.93
LatentOps
2.21
2.21
2.38
Distribution
2.67
2.67
2.63
MacLaSa
3,54
3.25
2.96
Table 3: Automatic results of comparison of different samplers.
Table 2: Human evaluations on multi-asee multi-aspect control.
evaluation results, annotators favor our approach as delivers the highest text quality among the baselines. Overall, our model demonstrates supe- rior performance in both attribute correctness and textual quality.
11.62% improvement compared to the strongest baseline. This result highlights our superior- ity in multi-aspect controllability. Additionally, MacLaSa displays good linguistic quality com- pared to previous method, emphasizing the ben- efits of learning compact latent space. Our ap- proach also exhibits substantial advantages in gen- eration efficiency. Compared to the parameter- efficient pbase Contrastive method, our method demonstrates remarkable 5.9x faster in inference speeds. In summary, MacLaSa surpasses existing baselines in attribute correctness and tex- tual quality while keeping high inference speeds.
Both automatic and human evaluations demon- strate that proposed MacLaSa outperforms other baseline models terms of attribute correct- ness and linguistic quality, while maintaining a high inference speed.
4.5 Analysis
Effects of VAE Losses conduct an ablation study to verify the effects of the classification loss Lo and aspect discrepancy loss Lp. The results are shown in Table 1. Removing Le causes the latent space to collapse completely. The correct- scores drop drastically as the model can hardly distinguish between representations of different tributes within the same aspect. Removing LD de- grades attribute correctness since we cannot allevi- ate domain gaps between different data sources. In- terestingly, without Lp, the distance between sam-
Human Evaluations The human evaluation re- sults for the multi-aspect control task can be found in Table 2. The inter-annotator agreement is 0.32 in Fleiss' k, indicating a fair agreement. Generally, the human judgment on attribute correctness aligns well with the results of the automatic evaluation. Our method excels in attribute control, achieving a correctness score of 3.54. Contrary to the automatic

---

Model
Generated Sentences
Positive Negative
Once upin a time, not so had ASTRONAUT Trigic end- Marinces collapse to sweep from the cliff in the AL wild- goose division.
Contrastive
Sports
The country turus its tail WESTMINSTER is in many ways and SOUTHAMPTON BALL seems to have the same hor- ing name all NBA names
The president of the country not to be? Your unk will not like your unk, your unk says the singer, doesn.
The last time was bad. The first time bad. The third time is bad, And the fourth is worse.
Distribation
The horse is had. The horse was had in the first round of contest, showing a loss to rival South Korea after an earlier victory.
The road hud. mad was had the first round of com- petition, ending up with record-breaking 30-year drought the U.S. Open...
Sen Censr Topic Center
The president of the country can't t hang to results. After Sunday' dehaccle in Philadelphin, there is little hope that Todford University can maintain its ranking in the top 10 in the country.
Figure 3: Projection of four attributes of two aspects from latent space via t-SNE.
MacLaSa
The horse was all wrong: Rossi Causes world championship leader Valentino Rossi suffered from an unusual
ple points from different aspects increases, lead- ing our model to generate sentences mapped from sparser regions. This results in a minor decrease in fluency while slightly increasing diversity.
The last they clashed, they lailed meet expectations for this matchup to be made at the WNBA Finals and they have not bocn...
Table 4: Example cases of generated sentences with at- tribute combination negative and sports. Red highlights sentiment-related contents. Blue highlights topic-related contents. Underlined are the input prompts.
Effects of Samplers To demonstrate the supe- riority of our ODE-based sampler, we compare it with other standard samplers. For fair compar- ison, we fix the parameters of VAE and choose different samplers for multi-aspect control text gen- eration. We first implement a random sampler by directly drawing samples from the latent space us- ing PGANddescribed in $3.4). We also compared it with a gradient- based sampler using Langevin Dynamics (Kumar et al., 2022; Qin et al., 2022). The automatic evaluation results are shown in Ta- ble 3. Random sampling directly from the latent space can only generate representations with sin- gle attributes, highlighting the necessity of using a specific sampler. While the LD-based sampler can generate high-quality sentences, it sacrifices attribute alignment, resulting in low attribute rel- evance. This may be because LD is sensitive and unrobust to hyperparameters (Nie et al., 2021). In contrast, our ODE-based sampler outperforms LD in terms of attribute alignment and textual diversity.
we use the t-SNE technique (Van der Maaten and Hinton, 2008) to visualize part of our estimated latent space with four attributes: positive, nega- five, world and sports in Figure 3. As shown, (1) attribute distributions within the same aspect are well separated due to the classification loss Lo that helps our model distinguish mutually exclusive at- tributes. (2) The distribution centers of sentiment and topic aspects are close to each other because we introduced Lp to penalize the distance between them to eliminate domain gaps, which helps gener- ating high-quality multi-aspect sentences. We also notice that the combination of negative- is tighter than that of negative-sports because world news often covers negative events such as war, dis- ease, and famine. This observation aligns with our experimental results in Appendix A.
To investigate the impact of PGAN, we conduct experiments by removing the GAN network and directly drawing latent representations from the standard Gaussian distribution N I). As shown in Table 3, without the GAN, model cannot accurately estimate the attribute space, resulting in decreased attribute relevance and textual quality.
4.6 Case Study
To better understand the benefits of learning a compact latent space for generative models, we randomly present generated examples in Table 4. When generating sentences with the attribute combination negative and sports, the Contrastive method can generate attribute-related words like "tragic" and "NBA"; however, the semantic coher- ence of the sentences is insufficient. This observa-
Visualization of Latent Space To provide an intuitive impression of the estimated latent space,

---

tion is consistent with the results of both automatic and human evaluations (see $ 4.4). One possible ex- planation is that the prefixes used for sentiment and topic control are trained independently, causing the two learned prefixes to exhibit different language habits and leading to incoherent expressions when combined for multi-aspect control. Conversely, the Distribution method can generate fluent sen- tences that display multiple attributes but struggles with varying expressions. For instance, Distribu- tion tends to use the word "bad" to convey nega- tive emotions, and its sentence structure is often repetitive, such as "The [noun] was bad in the first round of"" Our proposed MacLaSa can generate numerous aetribute-related content, such as "there is little hope" and "world championship leader", in fluent manner. By minimizing the discrepancy between sentiment and topic representations in the latent space, we merge high-quality representations related to attribute information, resulting in more coherent expression.
skill and experience to address. The KL vanishing problem refers to the situation where, during the training process, the KL divergence term may ap- proach zero. This can lead to a poorly constrained latent space, resulting in the model generating sam- ples that lack diversity and are not representative of the true data distribution. To tackle this issue, adopt several techniques, which are described in $ 4.1.
Ethics Statement
We honor and support the EMNLP code of Ethics. The paper focuses on controlled text generation, which aims to generate text with desired aspects. We recognize that controlled text generation may be misused to generate harmful texts, e.g., fake news. However, our method can also help to elim- inate toxic information in pre-trained language models by introducing specific attribute classifiers. Overall, it is meaningful to continue research into this work based on predecessors. Besides, the datasets used in this paper are all from previously published work and do not involve privacy or ethi- cal issues.
5 Conclusion and Future Work
In this study, we introduce a novel method, namely MacLaSa, for multi-aspect controllable text gen- eration that estimates a compact, low-dimensional latent space and employs a fast ODE-based sam- pler for efficient sampling. Our experiments on the two-aspect control task demonstrate the effective- ness and efficiency of our approach. Additionally, we carry out in-depth analytical experiments to emphasize the impact of each module and visual- ize the estimated latent space. In the future, we aim to expand our work by incorporating arbitrary attribute discriminators into the diffusion process using plug-and-pa and-play approach. Furthermore, plan to explore more powerful models enhance the linguistic quality of generated sentences.
A Acknowledgements
This work was supported by the National Key R&D Program of China (2022YFB3103700, 2022YFB3103704), the National Natural Science Foundation of China (NSFC) under Grants No. 62276248, and the Youth Innovation Promotion As- sociation CAS under Grants No. 2023111, Liang Pang is also supported by Beijing Academy of Ar- tificial Intelligence (BAAI).
References
Sumanta Bhattacharyya, Amirmohammad Rooshenas. Subhajit Naskar, Simeng Sun, Mohit lyyer, and An- drew McCallum. 2021. Energy-basee -ased reranking: Improving neural machine translation using energy- based models. In Proceedings of the 59th Anmual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Limitations
One of the limitations of the current MacLaSa ap- proach is that when a new aspect or attribute is in- troduced, the entire VAE framework needs to be re- trained to accommodate the unseen attributes. This retraining process can often be time-consuming and computationally expensive, posing a signif- icant challenge in dynamic environments where new aspects may frequently emerge.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals. An- drew M. Dai, Rafal Józefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016.
Moreover, due to the notorious KI. vanishing issue, the training process of the VAE framework is not stable and requires a significant amount of

---

Alvin Chan, Yew-Soon Ong, Bill Pung, Aston Zhang, and Jie Fu. 2021. Cocon: A self-supervieed approach for controlled text generation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
Thirty-Fifth AAAI Conference on Artificial Intelli- gence, AAAI 2021, Thirty- Conference on In- novative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2021. Vir- tual Event, February 2-9, 2021, pages 12955-12962. AAAT Press.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. 2018. Neural ordinary differential equations. In Advances in Neural Information Pro- cessing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018. December 3-8, 2018, Montréal, Canada.
Zhiting Hu and Li Erran Li. 2021. A causal lens for controllable text generation. In Advances in Neural Information Processing Systems 34; Annual Confer- ence on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski. and Rosanne Liu. 2020. Plug and play language models: A simple approach to controlled text generation. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Zhiting Hu, Zichao Yang. Xiaodan Liang, Ruslan Salakhutdinov, and Erie P Xing. 2017. Toward con- trolled generation text. In International conference on machine learning, pages 1597-1596. PMLR.
Xuancheng Huang. Zijun Liu, Maosong Sun Peng Li, Tao Li, and Yang Liu. 2023. An extensible plug- and-play method for multi-aspect controllable text generation. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics. Association for Computational Linguisties.
Jessica Ficler and Yoav Goldberg, 2017. Controlling linguistic style aspects in neural language generation. CoRR, abs/ 0as//707.020 707.02633.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative ad- versarial nets. In Advances in Neural Information Processing Systems 27: Annual Conference on Neu- ral Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for con- trollable generation. CoRR, abs//909.05885
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann. Nitish Shirish Keskar, Shafiq R. Joty, Richard Socher, and Nazneen Fatema Rajani. 2021. Gedi: Genera- tive discriminator guided sequence generation. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021.
Yuxuan Gu, Xiaocheng Feng, Sicheng Ma, Jiaming Wu, Heng Gong, and Bing Qin. 2022a. Improving control- lable text generation with position-aware weighted decoding. In Findings of the Association for Com- putational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022.
Yuxuan Gu, Xiaocheng Feng, Sicheng Ma. Lingyuan Zhang. Heng Gong, and Bing Qin. 2022b. Con- trollable text generation via probability density estimation in the latent space. arXiv preprint arXiv:2212.08307.
Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yu- lia Tsvetkov. 2021. Controlled text generation continuous optimization with multiple constraints. In Advances in Neural Information Processing Sys- tems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual.
Yuxuan Gu, Xiaocheng Feng. Sicheng Ma, Lingyuan Zhang, Heng Gong, and Bing Qin. 2022c. A distri- butional lens for multi-aspect controllable text gen- eration. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022.
Sachin Kumar, Biswajit Paria, and Yulia Tsvetkov. 2022. Constrained sampling from language models via langevin dynamics in embedding spaces. CoRR, abs/2205. 12558.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. 2018. Phrase- based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Skyler Hallinan, Alisa Liu, Yejin Choi, and Maarten Sap. 2023. Detoxifying text with marco: Controllable re- vision with experts and anti-experts. of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). ACL 2023, Toronto, Canada, July 9-14, 2023, pages 228-242. Association for Computational Linguistics.
Proceedings
Chunyuan Li. Xiang Gao. Yuan Li, Baolin Peng, Xiujun
Li. Yizhe Zhang, and Jianfeng Gao. 2020. Optimus: Organizing sentences via pre-trained modeling of a latent space. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Process- ing (EMNLP).
Changying Hao, Liang Pang, Yanyan Lan, Yan Wang. Jiafeng Guo, and Xueqi Cheng. 2021. Sketch and customize: A counterfactual story generator. In

---

Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao. ob- jective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Compula- tional Linguistics: Human Language Technologies.
Lianhui Qin, Sean Welleck, Daniel Khashabi. and Yejin Choi. 2022. COLD decoding: Energy-based constrained text generation with langevin dynamics. CoRR.
and Bill Dolan. 2016. A diversity-pronat
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2021. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
Xiang Lisa Li and Percy Liang. 2021.
Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 1Ith International Joini Conference on Natural Longuage Processing ( (Volume Long Pupers).
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Alisa Liu, Maarten Sap. Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021. Dexperts: Decoding-tin -rtoodiig-tteeeeeeo con- urllld text generation with experts and anti-expe expeetss In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Lan- guage Processing, ACI/IJCNLP 2021. (Volume 1: Long Papers), Virtual Event, August 1-6, 2021.
Fei Xiao, Liang Pang, Yanyan Lan, Yan Wang, Huawei Shen, and Xueqi Cheng. 2021. Transductive learning for unsupervised text style transfer. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Vir- tual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages 2510-2521, Association for Computational Linguistics.
Guangyi Liu, Zeyu Feng, Yuan Gao, Zichao Yang. Xi- aodan Liang, Junwei Bao, Xiaodong He, Shuguang Cui, Zhen Li, and Zhiting Hu. 2022. Composable text controls in latent space with odes. arXiv preprint arXiv:2208.006 .0.338
Kevin Yang and Dan Klein. 2021. FUDGE: controlled text generation with future discriminators. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL- HLT 2021, Online, June 6-11, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Kexin Yang, Dayiheng Liu, Wenqiang Lei, Baosong Yang, Mingfeng Xue, Boxing Chen, and Jun Xie. 2022. Tailor: A prompt-based approach to attribute- based controlled text generation. CoRR.
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Dian Yu, Zhou Yu, and Kenji Sagae. 2021. Attribute alignment: Controlling text generation from pre- trained language models. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2021.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies.
Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2022. A survey of controllable text generation using transformer- based pre-trained language models. CoRR, 0bs/2201.05333
Fatemehsadat Mireshghallah, Kartik Goyal, and Taylor Kerg-Kirkpatrick. 2022. Mix and match: Learning- free controllable text generationusing energy lan- guage models. Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers).
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28.
Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hin- rich Schütze. 2020. Masking as an efficient alterna- tive to finetuning for pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Weili Nie Arash Vahdat, and Anima Anandkumar. 2021. Controllable and compositional generation with latent-space energy-based models. In Advances in Neural Information Processing Systems 34: An- nual Conference on Neural Information Processing Systems 2021, NewrlPS 2021, December 14, 2021, virtual.
Jing Qian, Li Dong, Yelong Shen, Furu Wei, and Weizhu Chen. 2022. Controllable natural language genera- tion with contrastive prefixes. In Findings of the As- sociation for Computational Linguistics: ACL 2022.

---

A Detailed Results of Multi-aspect Control
Negative
Basiiiess
We exhibit the detailed results of eight combi- nations (two sentiment attributes four topic attributes) on multi-aspect control in Table 5. We compare with three types of baselines: (I) optimization-basee methods, PPLM and Mix&Match. (2) prefix-based based method Contrastive, and (3) methods that manipulate the latent space. for example, LatentOps and the Distribution method. For automatic evaluation metrics, we fine- tune two RoBERTa-Large (Liu et al., 2019) dis- criminators to assess the attribute accuracy scores for both sentiment and topic aspects simultancously. Perplexity (PPL) is employed to gauge the linguis- tic quality of the generated sentences. Additionally, we compute the Distinctness score appraise the textual diversity, reporting Distinet-1 and Distinct- 2 in our paper. We also report the standard de- viations, which stand for the stability of models among different runs.
Sormeer Cartor
Toyk Center
Figure 4: Projection of part of estimated attribute space with t-SNE.
performance concerning attribute alignment, lin- guistic quality, and diversity. We also evaluate the inference speed for sentence generation, with the results displayed in Table 1. The experimental findings indicate that MacLaSa maintains a high inference speed as well.
We observe that PPLM demonstrates strong con- trollability in specific combinations, such the Positive-Sci/fech pairing. However, the perfor- mance of each combination varies significantly, re- sulting in subpar average results. This phenomenon also exists for DEXPERTS and MaRCo. While Mix&Mater and the Contrastive method excel at attribute alignment, their linguistic quality leaves much to be desired. We postulate that this is due to Mithh employing sampling scheme for high-dimensional text space sampling, which is hindered by the discrete nature of text space and prevents smooth text generation. The Contrastive method posits that contrasting rela- tionships between individual attributes within each aspect aid in training attribute controllers, but neglects the differences between aspects, compro- mising overall textual quality. Regarding the two latent space manipulation methods, LatentOps hibits moderate performance in both attribute rel- evance and textual quality, while the Distribution method generates fluent sentences with the desired attributes but lacks diversity.
B Distribution of Attribute Space
In Figure 4, employ the t-SNE technique project hidden representations from four attributes into 2D for visualization: positive, negative, busi- ness, and sci./tech. This offers insight into a por- tion of the estimated latent space. We observe that, on one hand, the two sentiment attributes are distinctly separated due to the classification loss Lc. which also applies to the topic aspects. Con- versely, the distribution centers of the two aspects are situated closely together, pect discrepancy loss penalty LD- Overall, the observed attribute space distribution aligns with our expectations.
Eestropolis-Hsstinge
result of the as-
Our method attains remarkable average ac- curacy of 59.18% across the eight combinations, boasting a 11.62% improvement compared to the most powerful baseline and showcasing the excep- tional controllability of our approach. Additionally, our technique excels both linguistic quality and textual diversity. MacLaSa delivers well-rounded

---

Methods
Combination
Carrectness (%)
Text Quality
Diversity
Topic Acc.
PPL 4
Postive World
20.36 1: 1.69
25.47
1 1.70
0.23
0.54
Postiiv-Sports
16.53 + 1.13
25.78
+ 1,30
0.23
0.63
Posirive-Rusiee Russiess
1.26
Positive Sci Gi/Tech
61.73 1: 0.66
25.06
1.53
0.24
0.56
Negaivve World
Segative-Sports
2:27 + 0.57
25.96
1,54
0.23
0.63
Negarive-Rusiness
1.78 1.26
26.11
0.23
0.64
Negatve Sci/Tech
13:29 1: 1.82
24.40
1.11
0.24
0.56
Averge
18.14 0:45
25.59
1,00
0.23
0.64
Positive-Wor World
34,22 4.24
37.36
3.46
0.24
0.72
Positive: Sports
8.40 + 2.66
37.36 + 3,46
0.24
0.72
Posirive- Rusiness
+:
Posilive Sci Sei/Tech TTcch
45.02 1: 4.31
37.36 3.46
0.24
0.72
DEXPERTS
Neginive-World
2.68
Negative- Sports
8.17 2:27
40.03 4: 2.35
0.21
0.68
Negaäive-Busizes Business
10.98 +: 1.50
40.03 2.35
0.21
0.68
Nogative-Scl
63.64
8.73
Average
23.93 1: 1.I1
38,70 2.51
0.23
0.70
PositiveWoold
36,22
17.13
0.18
Positive Sports
37.11 + 25.23
18.16 + 1.47
0.17
0.55
Budlive-Business
Positive-Sel /Tech /Tech
50.00 5.21
17.1 + 1.39
0.18
0.57
MaRCo
Negative World
8.22 4.91
18.79 1.88
0.19
0.59
Segotive-Sports
10.89 0.38
19.94
2.85
0.17
0.57
Negative Businnss
22.89 20.07
20.51 + 2.45
0.19
0.59
Nogarive-Soi./Be
5.39
0.18
Average
27.81 1 1.94
18.87 1 1.85
0.18
0.58
World
Positive-Sports
$8.89 + 5.06
66.58 2.52
0.35
0.84
Seirive-Rusines
39.78 +: 1.66
65.89
0.35
0.84
Positive Sci Tech
65.33 1: 2.49
69.07 2.17
0.36
0.84
Mix&Match
Negative World
41.55 1: 1.66
69.49 1,14
0.35
0.84
Seporive-Sports
47.33 + 8.13
72.72 1.33
0.36
0.84
Negarive-Ruun Rusiness
31.56 + 5.15
71.61 + 3.87
0.35
0.84
Negaive Seciive--cc//eeee Sec/Tech
Avenage
30.17 + 2.07
68.72 + 0.97
0.36
0.84
Positive-Sports
70.31 + 5.55
$2.36 8.74
0.21
0.70
Posirive: Rusiness
53,16 + 5.00
56.13 4: 14.35
0.22
0.72
Podiive Sci/ /Tch
$1.96 1: 3.09
45.03 12.27
0.23
0.71
Contrastive
Nogastve-World
40.94 + 4.26
51.27 + 15.52
0.22
0.70
Negative Sports
Negative-Business
48.84 + 6.95
61.91 1 15.14
0.20
0.70
Nogative-Soe /Tech
50.40
0.23
Awenge
53.02 + 1.52
$2.56
11.97
0.22
0.71
Rositie-World
3.34
Positive Sports
63.47 11.01
28.01 4: 1.80
0.16
0.55
Positive-Busines Business
61.73 1: 9.36
25.73
1:33::10 1,84
0.14
0.52
Positive-Sel /Toch
22.07
LatentÖps
Negatve World
34,62 1: 1.59
24.98 1 1.56
0.16
0.55
Nogokive-Sports
Negative-Busince
25.74 + 2.41
27.30 2.11
0.15
0.54
Nogative-See /Tech
31.56 2.53
0.16
Average
44.41 3.72
26.11
0.16
0.35
Postiive World
37.42 1. 4.38
13.34 0.13
0.09
0.30
Positive-Soots
71.60
14:67 0.53
0.00
0.29
Positive Rusiness
72,80 + 6.45
11.33 1.00
0.07
0.28
Positive-Sel
Distribution
Negatve World
46.80 10.89
11.89 1:9911100 1.12
0.07
0.28
Negunive-Sports
Negarive Rusinesss
26.09 1: 5.60
11.03 0.11
0.07
0.25
Negaäiv-SSch Scitee--Sc//ehtt
34,86 1: 6.25
12.25 1 0.93
0.08
0,27
Average
49.79 +
12.48 0.52
0.08
Weillve-World
59.47 + 6.66
26.26 2: 0.20
0.19
0.65
Posttive-Spots
87.93 + 4.20
28.69 1.78
0.16
0.57
Positive: Rusiness
82,87 + 3.27
27.67 + 1.55
0.15
0.57
Maclaaa
Nogative-Worl
56.54 + 1.47
26.28 + 1.26
0.16
0.59
Negative Sports
38,00 1: 2.67
32.23 0.20
0.17
0.61
Negutive- Busiinss
31.40
4.07
29.06
1.12
0.15
0.59
Negarive-Sel /Tech
44.74 + 0.34
31.95 + 0.48
0.17
0.62
Avenage
59.18 0.81
28.19 1.36
0.16
0.60
Table 5: Detailed results of each combination multi-aspect control.


Reviews:
[{"review_id": "fvyMu6HvXe", "reviewer": "Reviewer_2HjF", "summary": "", "questions": "1) Did you compare with DExperts (Liu et al. 2021) at all? There are also followup works from that group including Quark (Lu et al. 2022) and MaRCo (Hallinan et al. 2022) that are relevant baselines.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and valuable comments of our paper. We appreciate your positive feedback on our strong experimental results and wide potential applications in the context of multi-aspect controllable text generation. \bWe understand your concerns and would like to address them as follows:\n\n**Q1**: Adding baselines (DExperts, Quark, and MaRCo, et al.)\n\n**A1**:  We appreciate your suggestions to include DExperts (Liu et al., 2021), Quark (Lu et al., 2022), and MaRCo (Hallinan et al., 2022) as additional baselines in our work. However, we would like to clarify that these papers mentioned above primarily focus on text detoxification experiments under the single-aspect control setting, which significantly differs from our objective of multi-aspect controllable text generation.\n\nHowever, to address your concern, we conduct additional experiments by adapting DExperts to the multi-aspect setting, as they had conducted sentiment control experiments (section 4 in their paper) that similar with our approach. We cannot compare our approach with Quark and MaRCo because they primarily focus on addressing the task of text detoxification under the single-aspect setting. Therefore, they are not applicable to our task of multi-aspect controllable generation.\n\nOur reproduced results of the DExperts method can be found in the new table below. It can be seen that DExperts performs poorly in the multi-aspect controllable generation setting. We speculate that this may be due to its neglect of the domain discrepancies among various aspects, leading to a significant performance deficiency. In contrast, our proposed method tackles this issue by mapping sentences from different aspects into a joint latent space and incorporating specialized loss functions to minimize the distances between different attribute representations. This enables us to effectively improve both control effectiveness and text quality in the multi-aspect controllable text generation task.\n\nWe hope this clarifies the distinction between our work and the mentioned papers. We believe that our contribution lies in the development of a novel approach tailored for multi-aspect control, which is a more complex task compared to single-aspect control. We would be happy to provide further details and address any additional concerns you may have.\n\n| Models         | Accuracy $\\uparrow$ | PPL $\\downarrow$ | Dist-1 $\\uparrow$ | Dist-2 $\\uparrow$ | Efficiency $\\downarrow$ |\n|:-------------- |:-------------------:|:----------------:|:-----------------:|:-----------------:|:-----------------------:|\n| DExperts       | 24.95               | 36.92            | 0.10              | 0.51              | 0.64                    |\n| MacLaSa (ours) | 59.18               | 28.19            | 0.16              | 0.60              | 0.10                    |\n\n\n\n\n**Q2**: More analysis of the latent space?\n\n**A2**: Thank you for your suggestions regarding the analysis of the latent space in our work. \n\n**Interpolation**: We conduct additional experiments to make it clearer. Specifically, we perform interpolation experiments in the latent space by sampling representations from different aspects and then interpolating between them to generate new latent representations, which are subsequently decoded into actual sentences. The results of interpolation in the learned latent space are listed in the table below.\n\n(1) The latent space we learned is a compact and well-behaved space with local continuity. We observed that the sentences generated from different interpolation combinations remained fluent, as indicated by their low Perplexity (PPL) scores. This suggests that the latent space captures meaningful representations that preserve the language coherence and quality of the generated sentences, reflecting the inherent structure and distribution of the data.\n\n(2) While linear interpolation in the latent space may seem intuitive for combining different attribute representations, it does not directly yield multi-aspect representations. This is because the latent space representations sampled from different attributes primarily retain the characteristics of single attributes. Therefore, a simple linear combination of these representations does not guarantee the generation of meaningful multi-aspect outputs. This highlight the need for specialized modules or techniques to facilitate the search or sampling of suitable representations that capture the desired multi-aspect representations, which is also supported by [1].\n\nTo overcome this limitation, we leverage the composability of EBMs and formulate the joint distribution for the latent representations and target attribute by incorporating any constraint into the energy function $E(\\cdot)$. Then we utilize a fast ODE-based sampler to draw samples from the EBMs and input them into the VAE decoder to generate the desired multi-aspect sequences.\n\n**Distance**: Regarding the distances in the latent space and their significance, in our study, we employed Euclidean distances to measure the distances between the distribution centers of different aspect. These distances are used to reduce the differences between different aspects, and we do not aim for their absolute values. Instead, we utilize a loss function to minimize these distances.\n\nIt is important to note that these distances do not necessarily directly reflect semantic or grammatical differences. Rather, they serve as a metric tool to aid in finding appropriate representations in the latent space for achieving multi-attribute control. By optimizing the loss function, we gradually decrease the distances between different aspects, leading to improved multi-aspect control.\n\nWe believe that our approach provides a valuable contribution to the field by explicitly addressing the challenges of multi-aspect controllable text generation and achieving superior performance compared to existing methods. We are confident that our experimental results and the thorough analysis in our paper demonstrate the effectiveness and advantages of our proposed method.\n\nWe appreciate your interest in the latent space analysis and would be happy to provide any additional details or clarify any further questions you may have.\n\n| Interpolation | Accuracy $\\uparrow$ | PPL $\\downarrow$ | Dist-1 $\\uparrow$ | Dist-2 $\\uparrow$ |\n|:-------------:|:-------------------:|:----------------:|:-----------------:|:-----------------:|\n| 0.2           | 12.68               | 32.60             | 0.21              | 0.68              |\n| 0.4           | 12.72               | 32.47            | 0.21              | 0.68              |\n| 0.5           | 12.82               | 32.53            | 0.21              | 0.68              |\n| 0.6           | 13.97               | 31.74            | 0.21              | 0.67              |\n| 0.8           | 13.80               | 32.56            | 0.21              | 0.68              |\n\nNote: Interpolation means the Interpolation weight (ranging from 0 to 1) for the sentiment aspect and topic aspect:\n\n$$\nz_{\\text{interpolation}} = z_{\\text{senti}} * weight + z_{\\text{topic}} * (1 - weight)\n$$\n\n\n\n**Q3**: Extending to decoder-only LLMs?\n\n**A3**: We want to clarify that our method is extendable to decoder-only models. In fact, our model operates within the framework of a Variational Autoencoder (VAE), which consists of an encoder and a decoder. The role of the decoder in this setup is to recover real sentences from the latent variables obtained through the encoding phrase. Therefore, the specific implementation of the decoder can utilize any pre-trained decoder-only language model. In our experiments, we chose to use the GPT-2 medium model for the decoder implementation to maintain a fair comparison with previous works. We believe this addresses your concern regarding the applicability of our method to decoder-only models like LLMs.\n\n\n\n**Q4**:  Adding baselines (DExperts)\n\n**A4**: We have addressed this concern in A1 above.\n\n\n**Q5**: Adding relevant work.\n\n**A5**: Thank you for your kind reminder of these relevant works. We agree with your point about the significance of single-aspect controllable text generation. We will include and discuss these relevant works in the revised version of our paper. We appreciate your guidance in helping us improve the completeness of our paper.\n\nIn conclusion, we greatly appreciate your recognition of our strong performance and potential applications. We've added sufficient experiments according to your constructive reviews. We hope it will address your concerns and you can reconsider your assessment about the overall quality of our paper. We believe that our work offers an effective and efficient solution to the problem of multi-aspect controllable text generation, making a meaningful contribution to the community.\n\nThank you once again for your recognition and contributions to our work.\n\n[1] Yuxuan Gu, Xiaocheng Feng, Sicheng Ma, Lingyuan Zhang, Heng Gong, and Bing Qin. 2022c. A distributional lens for multi-aspect controllable text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Multi-aspect control is an important problem. Ideally we want knobs for all these things and we want them to do this efficiently.\n\n2) The performance numbers show gains over the included baselines.\n\n3) The method from my reading of it, seems useful in any VAE-LM context.", "reasons_to_reject": "1) There are a few baselines missing including DExperts (Liu et al 2021), Quark (Lu et al. 2022) and MaRCo (Hallinan et al. 2022) that could probably be integrated. These are much stronger numbers so I expect these methods to be much much stronger but potentially less efficient.\n\n2) It'd be interesting to see more of an analysis of the latent space. There's a lot here that is missing. Some guiding points: Can you do interpolations in the latent space? Can you do vector arithmetic or other sort of geometric things? Do distances in the latent space make sense or reflect anything? What kinds of distances?\n\n3) This seems hard to extend to decoder-only models like our current LLMs. This is a very minor point.\n\n\nThank you for the strong and careful rebuttal. It addressed many of these concerns:\n- The missing baseline of DExperts has been added and shown to be poor (I imagine the comparison isn't super fair to their method, but DExperts is very specifically just good in the single controllable attribute setting). It's possible that you could mess with the baseline a little and tune it to be better, but for the sake of a rebuttal this is good. Retraining or reformulating Quark and MaRCo would be nice to see in a super fair comparison, but likely would take a couple of weeks to perhaps do well, so its beyond the scope of something me as a reviewer could reliably request in this format.\n\n- The analysis of the latent space seems to indicate, perhaps further, that the method is doing what its supposed to and that the space is at least somewhat interpretable.\n\nAfter the rebuttal, I raised my soundness score to a 3. I think there's definitely more that could be done, but I don't have any strong objections to it being accepted, so my excitement level remains a 3, but a stronger/higher 3 than it was before the rebuttal.", "questions_to_authors": "1) Did you compare with DExperts (Liu et al. 2021) at all? There are also followup works from that group including Quark (Lu et al. 2022) and MaRCo (Hallinan et al. 2022) that are relevant baselines.", "paper_topic_and_main_contributions": "This paper tackles the problem of doing multi-aspect controllable text generation, i.e. you have multiple attributes you want knobs to be able to turn when you're generating from a model. The authors introduce an approach MacLaSa that does this by estimating a latent space for multiple aspects together and then samples using a fast sampler. They evaluate performance on attribute relevance and textual quality.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is written well and the figures are nice."}, "extra_text": {"Missing_References": "There are some relevant work about mapping sequences into latent space, doing unsupervised sentiment control, doing controllable text generation in general that are missing.\n\n1) Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts (Hallinan et al. 2022)\n\n2) Extracting Latent Steering Vectors from Pretrained Language Models (Subramani et al. 2022)\n\n3) Educating Text Autoencoders: Latent Representation Guidance via Denoising (Shen et al. 2020)"}}, {"review_id": "L81izxQQAw", "reviewer": "Reviewer_V3ko", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition and valuable feedback. We appreciate your positive comments regarding our novel method, superior efficiency, comprehensive comparison and clear writing. We also understand your concerns and would like to address them as follows:\n\n**Q1**:  Experimental comparison with optimization-based methods, such as FGIM\n\n**A1**: We couldn't make a comparison with FGIM due to the fact that FGIM is primarily concerned with multi-aspect text style transfer, which is a different kind of generation task with our multi-aspect controllable text generation:\n\n- Multi-aspect text style transfer: transfer a source sentence with multiple attributes $a$ to another sentence with target multiple attributes $a'$.\n\n- Multi-aspect controllable text generation: generating a sentence with target multiple attributes $a'$ given a short prompt (e.g. \"Once upon a time\", not a complete sentence).\n\nMoreover, we would like to emphasize that our research focuses on a realistic scenario where sentences with multiple attribute labels are extremely scarce. In such a setting, the FGIM method is not applicable due to its reliance on input sentences with explicit attribute labels. The scarcity of sentences with multiple attribute labels is a common challenge in real-world applications, and our goal is to address this challenge by mapping sentences from different aspects into a joint latent space and incorporating specialized loss functions to minimize the distances between different attribute representations.\n\nTherefore, the methods like FGIM are not directly applicable to our task of multi-attribute control. This is the reason why we did not compare our work with FGIM or similar approaches in our previous work.\n\nWe appreciate your understanding of the distinction between the text style transfer task and the limitations of applying FGIM to our specific problem. We will consider adding a brief discussion in the revised manuscript to address this point and further clarify the differences between our work and style transfer methods like FGIM.\n\n**Q2**: Robustness analysis.\n\n**A2**:  This point has been addressed in our original paper (see Table 1 on page 6). We have conducted experiments under different random seeds and reported the standard deviations of multiple results, which stand for the robustness of models under different runs. As can be seen, our method produces very stable results across different random seed settings, demonstrating the robustness and stability of our method.\n\n| Models               | Accuracy $\\pm$ std |\n|:-------------------- |:------------------:|\n| PPLM                 | 18.14 $\\pm$ 0.45   |\n| Mix&Match            | 50.17 $\\pm$ 2.07   |\n| Contrastive Prefixes | 53.02 $\\pm$ 1.52   |\n| LatentOps            | 44.41 $\\pm$ 5.72   |\n| Distribution         | 49.79 $\\pm$ 1.99   |\n| MacLaSa(Ours)        | 59.18 $\\pm$ 0.81   |\n\n**Q3**: Time efficiency comparison\n\n**A3**: We would like to kindly draw your attention to the table 1 on page 6 in our original paper, where we have compared the execution efficiency of different methods in our original paper, which shows the average time to generate a complete sentence (in seconds).  Our method operates in a low-dimensional latent space and utilizes a fast sampler, which enables us to achieve higher efficiency compared with prior works.\n\n| Models               | Efficiency(s) |\n| -------------------- |:-------------:|\n| PPLM                 | 40.56         |\n| Mix&Match            | 164.60        |\n| Contrastive Prefixes | 0.59          |\n| LatentOps            | 0.10          |\n| Distribution         | 0.04          |\n| MacLaSa(Ours)        | 0.10          |\n\n**Q4**: Adding reference.\n\n**A4**: We appreciate your valuable suggestion. Since FGIM is primarily focused on the task of text style transfer, which is different from the objective of our study. We have discussed the detailed differences between our method with FGIM in A1 above. We will include the reference and the above discussion in our revised paper.\n\nIn summary, we greatly appreciate your recognition of our strong performance and comprehensive comparison with prior research.  We have explained the differences between our approach and methods like FGIM, highlighting the unique nature of our controlled text generation task. We also draw your attention to the experimental results that you may have overlooked. We hope that this clarification addresses your concerns. If there are any further concerns, please let us know and we would be happy to address these concerns during the discussion phase.\n\nThank you again for your time and effort. We look forward to your re-assessment of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is mostly clearly written and has provided clear implementation instructions to reproduce results except for the human evaluation section.\n\nThe proposed ODE-based method demonstrates superior efficiency when compared with optimization and prefix-based methods.\n\nThe paper is well backed by experimental evidence and provides a  relatively comprehensive comparison with prior research. It represents a thorough and complete piece of work, showcasing the authors' meticulous evaluation of their approach.", "reasons_to_reject": "The paper lacks sufficient experimental comparison and references concerning the optimization-based method, such as FGIM by Wang et al. , 2019. \n\nAdditionally, it is missing robustness analysis, considering the model's combination with ELBO loss.\n\nLack the time efficiency comparision with the optimization and prefix-based methods.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a multi-aspect controllable text generation method that attempts eliminate the domain discrepancies by means of VAE and ODE latent samplers. The methods mentioned in the paper are a novel combination of well-known prior techniques. It is clearly stated in the paper how it differs from prior contributions and the work is well cited. The tasks used in the paper are well defined in the prior work.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Controllable Unsupervised Text Attribute Transfer via Editing Entangled Latent Representation, NeurIPS 2019."}, "extra_text": {}}, {"review_id": "8woqnossBs", "reviewer": "Reviewer_YvKi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition and valuable suggestions. We appreciate your positive feedback on the elegant method, superior performance and good writing. We understand you concerns and would like to address it as follows:\n\n**Q1**: Difference between our method and prior research.\n\n**A1**: We understand you concern and we would like to clarify that: our model is specifically designed for multi-aspect control, which distinguishes it from LatentOps's single-aspect control.\n\n(1) We introduce suitable and usable solutions to the core challenges of multi-aspect control. To inject control signal during generation, we formulate the energy-based models in the low-dimensional continuous latent space estimated by a VAE, and plug in arbitrary classifiers to sample hidden representations with desired attributes based on our needs. To mitigate mutual interference between different controllers, we introduce the aspect discrepancy loss to penalize the distance between distribution centers of each two aspects. We also utilize an ODE-based sampler to facilitate fast sampling and ensure high efficiency.\n\n(2) The combination of these components is also non-trivial. To connect the VAE and ODEs, we fit a simple GAN on the latent space and sample from it. These samples are then fed into ODEs to obtain representations that bear multiple attributes. These representations are subsequently fed to the VAE decoder to produce target text sequences.\n\n(3) Through extensive experiments, we demonstrate that our method outperforms several strong baselines in terms of attribute alignment while keeping high efficiency and comparable linguistic quality.\n\nWe believe that our approach provides a valuable contribution to the field by explicitly addressing the complexities of multi-aspect controllable text generation and achieving superior performance compared to existing methods. We are confident that our experimental results and the thorough analysis in our paper demonstrate the effectiveness and advantages of our proposed method.\n\nThank your again for your feedback. We hope that our clarification addresses your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method looks elegant and reasonable. It efficiently resolves the existing problem in multi-aspect controllable text generation. \n2. The performance of MacLaSa is better than strong baselines.\n3. The paper is well-writing.", "reasons_to_reject": "1. The idea and overall framework of MacLaSa seems like a combination of LatentOPS and Distrbution. The sampling process of MacLaSa is very similar to LatentOPS. What's wore, Eq 5, Eq 7, and Eq 10 are all very similar to LatentOPS.\nThe difference between LatentOPS and Distribution, I think, is that the latent space of MacLaSa is more compact due to the usage of aspect discrepancy loss and classification Loss, however, which is proposed in LatentOPS. Therefore, I would like the author to discuss the difference between MacLaSa with LatentOPS and Distribution in detail. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes MacLaSa, an efficient framework for controllable text generation. MacLaSa first builds a latent space by VAE with classification and discrepancy regulation. Then, it builds an EBM on the latent space and samples the target latent vector with ODE. The experimental results show that MacLaSa performs well on multi-aspect controlling text generation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have reached on a consensus that the paper is well-written and acknowledged the effectiveness of the proposed method by looking at the gains compared to the baselines. The reviewers initially pointed out some missing baselines and comparison studies as well as some missing analysis of robustness, latent space characteristics, and efficiency. The authors were able to provide more in-depth studies and data points during the discussion period which mostly addressed the major concerns from the reviewers.

As a result, all the reviewers agree that the paper is good or strong at soundness (3, 4, 3). In terms of excitement, all the reviewers have reached on a consensus that it's ambivalent (3).
Decision: Accept-Findings