Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation
Authors: Florian Mai
James Henderson
Venue: ICLR 2022 Submitted (2022)
Keywords: autoencoders
latent space learning
variable-size
natural language processing

Abstract: Text autoencoders are often used for unsupervised conditional text generation by applying mappings in the latent space to change attributes to the desired values. Recently, Mai et al. (2020) proposed $\operatorname{Emb2Emb}$, a method to $\textit{learn}$ these mappings in the embedding space of an autoencoder. However, their method is restricted to autoencoders with a single-vector embedding, which limits how much information can be retained. We address this issue by extending their method to $\textit{Bag-of-Vectors Autoencoders}$ (BoV-AEs), which encode the text into a variable-size bag of vectors that grows with the size of the text, as in attention-based models. This allows to encode and reconstruct much longer texts than standard autoencoders. Analogous to conventional autoencoders, we propose regularization techniques that facilitate learning meaningful operations in the latent space. Finally, we adapt $\operatorname{Emb2Emb}$ for a training scheme that learns to map an input bag to an output bag, including a novel loss function and neural architecture. Our experimental evaluations on unsupervised sentiment transfer and sentence summarization show that our method performs substantially better than a standard autoencoder.

Under review as a conference paper at ICLR 2022
BAG-OF-VECTON AUTOENCODERS FOR UNSUPERVISED CONDITIONAL TEXT GENERATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Text autoencoders are often used for unsupervised conditional text generation by applying mappings in the latent space to change attributes to the desired values. Recently, Mai al. (2020) proposed Emb2Emb, a method learn these map- pings in the embedding space of an autoencoder. However, their method is re- stricted to autoencoders with a single-vector embedding. which limits how much information can be retained. We address this issue by extending their method to Bag-of Vectors Autoencoders (BoV-AEs), which encode the text into a variable- size bag of vectors that grows with the size of the text, as in attention -based mod- els. This allows to encode and reconstruct much longer texts than standard au- toencoders. Analogous conventional autoencoders, we propose regularization techniques that facilitate learning meaningful operations in the latent space. Fi- nally, we adapt Emb2Emb for training scheme that learns to map input bag an output bag. including a novel loss function and neural architecture. Our empir- ical evaluations on unsupervised sentiment transfer and sentence summarization show that our method performs substantially better than standard autoencoder.
INTRODUCTION
In conditional text generation, we would like to produce an output text given an input text. Hence, parallel input-output pairs required train a good supervised machine learning model on this
type of task. Large-scale pretraining (Peters et al. 2018 Devlin et al. 2019 Lewis et al.) 2020) can alleviate the necessity for training examples some extent, but even this requires a substantial number of annotations (Yogatama et al. 2019). This is an expensive process and can introduce un- wanted artifacts itself, which are henceforth learned by the model (Gururangan et al., 2018). For
these reasons, there is substantial interest in unsupervised solutions. Text autoencoders (AEs) don't require labeled data for training, and are therefore a popular model for unsupervised approaches to many tasks, such machine translation (Artetxe et al. 2018), sentence compression (Févry & Phang 2018) and sentiment transfer (Shen et al. al.] 2017). The classical text AE (Bowman et al. 2016) embeds the input text into single fxxed-siee vector via the encoder, and then tries to reconstruct the input text from the single vector via the decoder. Single-vector embeddings are very useful, because they allow to perform conditional text generation through simple mappings in the embedding space, e.g. by adding a constant offset vector to change attributes such as sentiment (Shen et al. 2020). Re- cently, Mai al. (2020) proposed Emb2Emb, method that can learn these mappings directly the embedding space of any pretrained single-vector AE. This is a powerful framework, because the AE can then be pretrained on virtually infinite amounts of unlabeled data before applying to any down- stream application. This concept, transfer learning. is arguably one of the most important drivers of progress machine learning the recent decade: These so-called Foundation Models (Bommasani et al. 2021) have revolutionized natural language understanding (e.g. BERT (Devlin et al.) (219)) and computer vision (e.g. DALL-E (Ramesh et al., (02T)), among others. Since Fmb2Fmm was designed to work with any pretrained AE, it was an important step towards their scalability.
However, as Bommassai et al. (202)) point out, another crucial model property is expressivity, the ability to represent the data distribution it is trained on. In this regard, single-vector representations are fundamentally limited; they act as a bottleneck. causing the model to increasingly struggle to encode longer text (Bahdanau et al. 2015). In this paper, extent Emb2Emb from single-vector bottleneck AEs to Bag-of- Vector Autoencoders (BoV-AEs). which encode text into a variable-size representation where the number of vectors grows with the length of the text. This gives Bo

---

Under review as a conference paper at ICLR 2022
the food was terrible
the food was terrible
Text Autoencader Protraining
erIC
dec
Task Training
((3y)
+
Inforence
AnC
By - dec
Figure 2: High-level view of the Emb2Emb framework. Text Auloencoder Pretraining: An autoencoder is trained on an unlaheled corpus, i.c., the encoder enc transforms an input text into a continuous embedding Za, which is in turn used by the decoder dee to predict reconstruction á of the input sen- tence. Task Training: The encoder is frozen (grey), and mapping & is trained (green) on input embeddings zg output predictions 2, such that it minimize some loss L(2y). In- ference: To obtain textual predictions ÿ. the encoder is composed with T and the decoder.
dec
dec
the food was amazing
the food was amazing
Figure 1: Left: the standard setup, the representation consists of single vector, re- quiring a simple vector-to-vector mapping to do operations in the vector space. Right: In BoV-AE, the representation consists of variable-size bag of vectors, requiring a more complex mapping from one bag to another bag.
AEs the same kind of representations as attention-based models. But this added expressivity comes with additional challenges, as illustrated in Figure [ In the single-vector case, operation & in the vector space consists of a simple vector-to-vector mapping. But with BoV-AEs, & needs to map bag of vectors onto another bag of vectors, which requires more complicated operations. In this paper, we demonstrate how such a mapping can be learned in the context of the Emb2Emb framework by making the following novel contributions: (i) We propose regularization scheme for B-V-AEs, (ii) a neural mapping architecture O for Emb2Emb, and (iii) a suitable training loss.
Empirically. we show on two unsupervised conditional text generation tasks, sentiment trans- fer (Shen et al. (017) and sentence summarization (Rush et al.) 2015), that BoV-AEs perform substantially better than standard AEs if the text is too long
captured by one vector alone. Our ablation studies confirm that our technical contributions are crucial for this success.
In the following section, we review the Emb2Emb framework, before we introduce BoV-AE (Sec- tion 3) and its integration within Emb2Emb (Section
BACKGROUND: Emb2Emb
Fmbeddingg-t- to-beeddiig (Emb2Emb) was introduced by Mai et al. (020)) as both a supervised and unsupervised framework for conditional text generation. The core idea disentangle the transition from the discrete text space to continuous latent space from the specific task, allowing for scale pretraining with unlabeled data.
The workflow of the framework is depicted in Figure First, a text AE A dec a enc is trained to map an input sentence from the discrete text space X an embedding space via the encoder enc X + 2, and back to A' via a decoder dec 2 - X', such that A(x) = typically trained via negative log-likelihood, Cree = NLL(A(x), x².. contrast to other methods, A can in principle be any AE, opening the possibility for large-scale AE pretraining with unlabeled data. Second, task- specific training is frozen, and performed only in the embedding space Z the AE. To this end, the encoder mapping layer + Z Z is introduced, which is trained transform the embedding of the input z, into the embedding of the predicted output ž. The concrete loss ((2,) depends the type of task. In the supervised case, the true output is also encoded into space
2

---

Under review as a conference paper at ICLR 2022
2, and the distance between the true embedding and the predieted embedding is minimized. In the unsupervised case, the loss needs to be defined for the specific task at hand. For example, for sentiment transfer, where the goal is to transform a negative review into positive review while retaining as much of the input as possible, Mai et al. (2020) compose the loss as a combination of two loss terms] ((2 Ssim (Zz, 2,y) + Asty ( (,y) Laty encourages By to be classified positive review according to a separately trained sentiment classifier. Leim encourages the output to be close to the input in embedding space, e-g. via euclidean distance. Xaty is a hyperparameter that controls the importance of changing the sentiment of the predicted output.
A main question in Emb2Emb is how to choose the embedding space 2. Mai al. (2020) use a sin- gle continuous vector to encode all the information of the input, i.e. Z = R. This choice simplifies the mapping & to an MLP and the training loss to vector space distances, which is relatively easy to train. On the other hand, it limits the model in fundamental ways: The representation is fixed-sized, i.e. the representation cannot grow in size. Sequence-to-sequene neck struggle to encode long text sequences (Bahdanau et al. 2015), which is key reason why attention-based models are now standard practice in peneence-to-equeenn models. Hence, it would be desirable to adapt Emb2Emb in such a way that 2 contains variable-sizel sized embeddings instead.
models with a fixed-size bottle-
3 BAG-OF-VECTORS AUTOENCODER
We propose Sogg-o--Vetorrs Autoencoders (BoV-AEs) to be used with Emb2Emb. Following the naming convention by Henderson (2020), , we refer to a bag of vectors as a (multi)-set of vectors that (i can grow arbitrarily large, and (i) where the elements are not ordered (a basic property of sets). A type of BoV representation that is used very commonly is found in Transformer (Vaswani et al.) 2017) encoder-decoder models, where there is one vector to represent each token of the input text, and the order of the vectors does not matter when the decoder accesses the output of the encoder. In this work, we also rely on Transformer models as the backbone of our encoders and decoders. However, in principle, any encoder and decoder can be used, as long as the encoder produces a bag as output and the decoder takes a bag as input. Formally, Z (R ++ so the encoder produces a bag-of- vectors X [Zi,.... Zn == enc(z), where n is the number of vectors in the input bag.
3.1 REGULARIZATION
The fact that we use a BoV-ba AE presents a major challenge: AEs have to be regularized to prevent them from learning a simple identity mapping where the input is merely copied to the output, which does not result in a meaningful embedding space. In fixed-size embeddings, this is for example achieved through coderr-ooplletnnsss (choosing latent dimension that is smaller than the input dimension) or through injection of noise, either at the input or in the embedding space. While there exists lot of research on regularizing fixed-sized AEs, it is not clear how achieve the same goal in a B-E- Here, regularizing the capacity of each vector is not enough. As long as each vector can store (constant) positive amount of information, a bag of unlimited size can still store infinite information. However, it is not clear to what extent the size of the bag needs to be restricted. By default, a standard Transformer model produces as many vectors as there are input tokens, but this is likely too many, as it makes copying from the input to the output trivial. Hence, we want the encoder to output fewer vectors. In the following we explain how this is achieved in BoV-AEs.
LODrop Ideally, we want the model to decide for itself on per-example basis which vectors it needs to retain for reconstruction. To this end, we adopt LODrop, a differentiable approximation to LO regularization, which was originally developed by Zhang et al. (2021) for the purpose of speeding up a model through sparsification. The model computes scalar gates di = ((Z)) E [o, 1] (which can be exactly zero one) for each encoder output. After the gates are computed, we multiply them with their corresponding vector. Vectors whose gates are near zero (i.e., smaller than some > O) are removed from the bag entirely. An additional loss term. (X) = ALO Σ, 9i encourages the model to close as many gates as possible, where the hyperparameter dn controls the sparsity rate implicitly. However, in initial experiments, we found \ difficult to tune, as it is very sensitive with respect to other hyperparameters. We instead employ a modified loss that seeks to explicitly match
'Their total loss includes an adversarial component that encourages the outputs of the mapping to stay on the latent space manifold. We leave adaptation of this component to the BoV scenario for future work.
3

---

Under review as a conference paper at ICLR 2022
certain target ratio of open gates. Similar to the free-bits objective that is used prevent the posterior collapse problem in VAEs (Kingma et al. 2016), the objective becomes
CL. (X) = ΛLo max(r, : 2. g:).
(I)
By setting λzo to large enough value (empirically, A = 10), we find that this objective reaches the target ratio reliably for different while at the same time reducing the reconstruction loss. This allows to compare different strengths of regularization while reducing the tuning effort substantially.
4 EMB2EMB WITH BOV-AES
In the following we describe how to adapt the Emb2Emb model to BoV-AEs, i.e., how to generate an output bag X = [ž1...., 2n given an input bag X through the mapping & (X), and how to choose the loss function C(X, X). For example, in the case of style transfer, we want X to be similar to X.
4.1 MAPPING +
In contrast to Mai et al. (020), who use single-vector embedding and hence + can be as simple as an MLP, in our work, + must be capable of producing a bag vectors. The straight-forwer forward choice for & is Transformer decoder that uses cross-altention on the input BoV, and generates vectors autoregressively one at a time, formally = Transformer(2.: 21,, 2,,... 2 x), 6 2 where 2, is the difficulty of the task, this mapping may
the embedding of some starting symbol. Depending be sufficiently powerful, in particular for short texts (i.e. bags with few vectors). For longer texts, however, learning map from one bag to another is difficult, and may hence require the integration of inductive biases.
Based on the assumption that the output should be close to the input in embedding space, Mai al. (2020) propose OffsseNNet for the single vector case, which computes an offset vector to be added to the input. With similar motivation, we propose variant of pointer-generator networks (See al., 2017). which allows the model to choose between copying g an input vector and generating new one. Instead of just copying, however, our model (Transformer++) allows compute an offset vector to be added to the copied vector, analogous to Mai et al. (2020). Formally, at each timestep t,
2: = (1 - Pgen] (eoopy + Zoffiee) + PgenZi
(2)
where z; = Transformer(2.,... 1, X). Intuitively, by controlling Pgen E (0, 1), the model
makes the (soft) decision either copy vector from the input and add offset, or to generate completely new vector. Here, Pgen is a function of z and the starting symbol which we treat as context vector, Pyen a (W [z,i z(]). Similarly, Zofiset is a one-layer MLP with [z|s Zcopyl as input. Zcopy is determined through an attention function:
Zeopy
K
W X, a, = =oftmmaxx(2 K);,
(X)+ := 2;
(3)
where W cpy is learnable weight matrix. We refer to this model as Transformer++.
4.2 GENERATING VARIABLE SIZED BAGS
The output bag is generated in an autoregressive manner. In the unsupervised case, it is not al- ways clear how many vectors the bag should contain. However, due to the unsupervised nature, all information needed for computing the (task-dependent) training loss L(X, X) are also available at inference time. In this case, we can first generate some fixed maximum number N of vectors autoregressively, and then determine the optimal bag by computing the minimal (inference-time) loss value, X = min ((X11,X) This can be valuable for tasks where we do not have a good t=I...,
prior on the size of the target bag. During training, we minimize the loss locally at every step. But we don't necessarily care about the loss at very small or big bags, so we might want to weight the steps as Lioial X) = Σ w X). Here, w e R. could be any weighting, but it is more beneficial for training to only backpropagate from bag sizes that we expect be close to the optimal
4

---

Under review as a conference paper at ICLR 2022
output bag size. For instance, in style transfer, the output typically has about the same length the input. Hence, for an input size of length n, a useful weighting could be
- k
Wi 0
otherwise
in which k denotes the size of window around the input bag size.
4.3 ALIGNING Two BAGS OF VECTORS
As described in Section unsupervised sentiment transfer involves two loss terms, Lity and Lsim- In order to adapt Lsty from the single vector case the BoV case, we can simply switch from an MLP classifier to a Trransformer-basee classifier. For Leims however, we need to switch to a loss function that is defined on sets. While there well-known losses for the single-vector case, in NLP set-level loss functions are not well-studied.
Here, propose novel variant of the Hausdorff distance. This distance is commonly used in vision applications: as performance evaluation metric in e.g. medical image segmentation (Taha & Aydin
Hanbury [015
2020), or in vision systems as way compare images (Huttenlocher 1992] Takáes 1998: Lin et al. 2003 Lu et al. 2001). More recently, variants (different from ours) of the Hausdorff distance have also been used loss functions to train neural networks (Fan et al.
et al.
2017 Ribera et al.] 22199 Zhao et al. 2021). In NLP, its use is very rare (Nutanong et al., Chen 2019; Kuo et al., 2020). To the best of our knowledge, our paper is the first to present novel, fully differentiable variant of the Hausdorff distance as loss for language learning.
2016
The Hausdorff distance is method for aligning two sets. Given two sets X and X, their Hausdorff distance H is defined as
H X) =
2 align(X x) + align(X,X)) X), align(X, X) = max min aeX yE
(5)
Intuitively, two sets close if each point in either set has counterpart in the other that is close to it according to some distance metric d. We choose d to be the euclidean distance, but in principle any differentiable distance metric could be used (e.g. cosine distance). However, the vanilla Hausdorff distance is very prone to outliers, and therefore often reduced to the average Hausdorff distance (Dubuisson & Jain 1994). where
align(X, x)
Σ min d(x, y). ZEX ve
The average Hausdorff function is step-wise smooth and differentiable. Empirically, however, we find step-wise smoothness be insufficient for the best training outcome. Therefore, we propose fully differentiable version of the Hausdorff distance by replacing the min operation with softmin like follows:
(-d(m)))) Σ e(-d(z,0')) d(x,y) V'eX
align X, x)
ΣΣ xEX xEX,EX
This variant is reminiscent of the attention mechanism Bahdanau al. (2015) in the that weighted average is computed, which has been very successful at smoothly approximating discrete decisions. e.g., read and write operations in the Differentiable Neural Computer (Graves al. 2016) among many others.
5 EXPERIMENTS
Our experiments are designed to test the following two hypotheses. H1: If the input text is too long to be encoded into fixed-size single vector representation. BoV-AE-based Emb2Emb provides substantial advantage over the fixed-sized model. H2: Our technical contributions, namely LODrop regularization, the training loss, and the mapping architecture, are necessary for BoV-AE's success.
We evaluate our model on two unsupervised conditional text generation tasks: In Section [.I] we show that H1 holds even when the single-vector dimensionality is large (d=512). To this end,
5

---

Under review as a conference paper at ICLR 2022
create new sentiment transfer dataset, Yelp-Reviews, whose inputs are relatively long. However, training on this dataset is computationally very demanding Therefore, we turn to shorter text datasets to test hypothesis H2. Concretely, we test on sentiment transfer dataset of short texts (Section 5.2). and a sentence summarization dataset of medium length texts (Section 5.3)
Evaluation metries: In sentiment transfer, the goal is rewrite negative review positive view while keeping as much of the content as possible. Hence, two metrics are important, sentiment transfer ability and content retention. Following common practice (Hu et al., 2017 (Shen al. 2017]
Lample et al. 2019), we measure the former with a separately trained style classifier based on Dis- tiBERT (Sanh al. 2019). and content retention in terms of self-BLEB (Papineni et al. 2002) be- tween the imput and the predicted output. Sometimes, comparing models is easiest via a single value. Therefore, others have argued to aggregate content retention and transfer accuracy metrics (Xu et al.]
2018 Krishna et al. 2020). We follow the argumentation of Krishha et (2020) that this aggrega-
tion should be per sentence and compute a single score Ei= ACC(g) BLEU(Ú. 2)) where is the imput sentence, ÿ is the predicted sentence, and M is the number of data points. For readability, multiply all metrics by 100 before reporting.
As is standard practice in summarization, we evaluate performance on this task with ROUGE-L. (Lin 2004). Note, however, that ROUGE scores be misleading if summaries are longer than the gold summary (Napoles et al. 2011). For this reason, we also report the average output length.
5.1 YELP-REVIEWS
Our hypothesis is that AEs with single vector bottleneck are unable to reliably compress the text when it is too long. Here, we test if this holds true even for a large single-vectes vector model with d-512. To this end, we create the dataset Yelp-Reviews, which consists of strongly positive and strongly negative English restaurant reviews on Yelp (see appendix [A.3. for a detailed description). This dataset is very similar to Yelp-Sentences introduced by [Shen et al. (2017). However, while Yelp- Sentences consists of single sentences of about 10 words on average, Yelp- Reviewsiewws consists of entire reviews of 52 words average. To assess the reconstruction ability of BoV-AEs and fixed-sized AEs, we train 5 models that only differ in the target ratio. Models named L0-r denote LODrop-based BoV-AE models with target ratio
The single vector fixed-size AE is obtained by averaging the representations at the last layer of the encoder. For style transfer, we train a Transformer+ mapping using the loss described in Section 2 To obtain results varying transfer levels, we train multiple times with varying Aety. resulting in multiple points for each model in Figure 3 and
Further details can be found in the appendix A.3.2
The results (full graph shown in Figure 7 in the appendix) indicate that even large single vector models (d=512) are unable compress the text well: the NLL loss on the validation set of the fixed-size model is m33.9. L0-0.05 is only slightly better than the fixed-size model, whereas L0-0.1 already reaches substantially lower reconstruction loss (==2.1). We evaluated the down- stream sentiment transfer performance of Transformer++ with L0-0.1 and and the fixed-size model, respectively. Figure shows a scatter plot of the results, where results that are further to the top-right corner are better. We see that a comparable transfer level, the Bo is substantially better at retain- ing the input content. This supports hypothesis H1 that variable-size BoV models are particularly beneficial in cases where the text length is too long to be encoded in single-vector.
Results:
5.2
In order answer research question H2, perform large set of controlled experiments over our model's components. Due to the high computational demand, we turn to the popular Yelp- Sentences sentiment transfer dataset by [Shen et al. (0017). Texts in this dataset are s 10 words on average. As these sentences are much easier to reconstruct, we set the embedding size to d-32 so that the condition for hypothesis H1 is still valid. Here, we again train BoV-AEs for target rates 0.2, 0.4, 0.6, 0.8 and then evaluate their reconstruction and style transfer ability in the same fashion as for Yelp-Reviews. Finally, we investigate the impact of the window size and differentiable Hausdorff loss. Experimental details are given in appendix A.4.2 For completeness, we provide an
Pretraining a model of this size until convergence took more than a month on single 24GB GPU. We restrict our analysis 10-0.. because this dataset have is computationally demanding.

---

Under review as a conference paper at ICLR 2022
100
Table I: Results on Gigaword sentence summarization. Scores represent ROUGE- L. with average output words in parenthe- ses. T and T++ denote Transformer and Transformer respectively.
fixed LO-0.1
80
accuracy
40
Model
T
T++
fixed
13.1 (78.3)
13.2
20
L0-0.2
(2382)
19.8 (23.2)
18.3 (I0.7)
LO-0.4
8.0
16.4 (12.5)
10
20
30
40
50
LO-0.6
6.6 (83.5)
14.7 (51.1)
self-BLEU
LO-0.8
9.3 (5.1)
13.2 (48.6)
Figure 3: Style transfer on Yelp-Reviews
Validation reconstruction loss
Validation rec reconstruction loss
5
fixed
fixed
80
LO-0.2
LO-0.2
LD-0.4
L0-0.4
L0-0.6
0.6
accuracy
60
LO
w
LO-0.8
20
250 500 k 750 1M 1.25 M 1.5 step
20
self-BLEU
self BLEU
Figure 4: Reconstruction loss the valida- tion set for different AEs. fixed: The bag consists of single vector obtained by aver- aging the embeddings at the last layer of the Transformer encoder. LO-r: BoV-AEs with LODrop target ratio r.
Figure 5: Style transfer performance on Yelp-Sentences of BoV models compared to fixed-size AE for varying Xsty. As both content retention and style transfer are im- portant for style transfer, the further graph is to the top right, the better the model.
analysis of the computational complexity of BoV-AE in Appendix B.2.2 and a qualitative analysis in Appendix BB2.3
Reconstruction ability: Figure 4 shows the reconstruction loss on the validation set for the fixed- size model compared to BoV models. The fixed-size AE does not reach satisfactory reconstruction ability, converging at an NLL loss value of about 3. In contrast, BoV models are able to outperform the fixed-size model considerably. As expected, higher target ratios lead to better reconstruction, because the model can more vectors store the information. Models with higher target ratio also reach their optimal loss value more quickly. While I.0-.66 approaches the best reconstruction value (sv1.0) eventually, the model needs more than I million training steps reach it. In contrast, L0-0.8 needs less than 100k steps to converge, which could indicate that L0-0.8 learns to copy rather then compress the input, resulting in a bad latent space. L0-0.4 yields higher loss, but still drastically better than the fixed size model. L0-0.2 is not enough to outperform the fixed-size model. Overall, these results show we have the right settings for evaluating HI and H2, as 10 words is too long to be encoded well into single vector of d-32, whereas a BoV-AE with a high enough target ratio r can fit well.
Style transfer ability: Results are shown in Figure Up to r=0.6, they correspond well to the reconstruction ability, in that BoV models with higher target ratios yield higher self-BI.EU scores at comparable transfer abilities, outperforming the fixed-size model (H1). However, at r=0.8, the performance suddenly deteriorates at medium to high transfer levels. This supports the hypothesis that L0-0.8 lacks smoothness in the embedding space due to insufficient regularization, which in turn complicates downstream training. This is the first piece of evidence that LODrop is necessary for the success of our model (H2).
Window size: The window size l determines which bag sizes around the input bag size we back- propagate from. Here, investigates its influence the model's performance. Since the Xsty

---

Under review as a conference paper at ICLR 2022
hyperparameter is very sensitive to other model hyperparameters, we train with varying Xaty for each fixed window size and report the best style transfer score for each window size. Further exper- imental details and full results can be found in appendix A..4.3] Our results indicate that increasing the window size from zero (score 22.7) is beneficial up to some point (k=5, score 32.6), whereas increasing by too much (k=20, score 15.6) is detrimental to model performance even compared to a window size of zero. We hypothesize that backpropagating bags that are either very small or very large is detrimental because it forces the model to adjust its parameters optimize unrealistic bags, taking away capacity for fitting realistic bags.
Differentiable Hausdorff: In Section 4.3) we argue that the min operation should be replaced by softmin in order to facilitate backpropagation. Here, we test if the differentiable version is really necessary, that is, we compare Eq. 6 to Eq. Like before, we train the two variants with different Asty and select the best style transfer score on the validation set. Further experimental details can be found in appendix [A.4.3] The difference is substantial: Average Hausdorff reaches 14.6, whereas differentiable Hausdorff reaches 24.2. We hypothesize that this discrepancy is due to the difficult nature of the style transfer problem, which requires carefully balancing the two objectives, content retention (via Hausdorff) and style transfer (via the classifier). This is easier when the objective functions are smooth, which is the key advantage of differentiable Hausdorff.
5.3 SENTENCE SUMMARIZATION
In sentence summarization Rush et al. 2015), the goal is to capture the essence of a sentence in fewer words. We evaluate on the Gigaword corpus (Graff et al. 2003) similar to Rush et al. (2015). This corpus consists of more than 8.5 million training samples, but we use random subset of 500k to limit the computational cost. Inputs are average 27 words long, which is medium length compared to the other two datasets in this study. We use moderately sized vectors of d=128 and again train different B-V- with target ratios 0.2, 0.4, 0.6, 0.8. When applying the model to the sentence summarization downstream task, we train using the loss term C(2,) = Leim 2,) + Aien (2,)) This loss term is conceptually similar the loss term used for style transfer, except that Lien denotes the prediction of a model trained to predict the length of the input text from the text's latent representation (the shorter the better). We train with varying values of Aven 0.2, 0.5, 1, 2, 5, 10 and select the best model (ROUGE-L) on the development set. Intuitively, this model learns to retain as much from the input as possible while minimizing the output length. Note that this model of summarization could certainly be improved further, e.g. by accounting for relevancy and informativeness of the output (Peyrard 2019). However, our goal is not to create the best task-specific model possible, so these considerations are out of scope for this paper.
The input texts in this task are relatively long. Due to the higher number of vectors in a BoV, it may be difficult learn the mapping, especially for large target ratios r. We experiment with Transformer++ to observe to what extent this can facilitate learning.
Results: Despite the moderately large vector dimensionality, the single-vector bottleneck model achieves only considerably lower reconstruction performance than the BoV models (for details, please refer Figure in in the appendíx). Again, larger target rates r lead to faster convergence, and all BoV models converge to approximately the same validation loss value (0.9). The only exception higher loss value (1.25), but is still vastly stronger than the fixed
L0-0.2, which converges size model (3.01).
However, as shown in Table [ L0-0.2 performs the best on the downstream task, outperforming the single-vector model by more than 5 ROUGE-L points while simultaneously requiring much fewer output words. BoV models with higher target ratios than r=0.2 perform worse. Moreover, the Transformer++ architecture tends to improve results, particularly with target rates > 0.2. The ROUGE-L score itself does not improve for 0=2.2, but note that this comes at the expense of more than doubling the output length. Also note that L0-0.6 and L0-0.8 only obtain relatively high scores because they produce long outputs that even exceed the length of the input. In fact, for r = 0.6, 0.8 no value of Ate produces outputs that are reasonably good (> 10 ROUGE-L) and short (< 20 BLEU) at the same time.
The above results confirm both our hypotheses: First (H1), is beneficial to use BoV model over vengor model to reduce the compression issues induced by the fixed-size bottleneck. Secondly (H2), when using BoV model, it is imperative to regularize the number of vectors in the bag as
8

---

Under review as a conference paper at ICLR 2022
way of smoothing the embedding space, making it easier to learn the mapping for unsupervised text generation tasks. Moreover, if the number of vectors in the bag is large, our Transformer+ architecture can substantially facilitate learning the mapping.
RELATED WORK
Manipulations in latent space: Besides Emb2Emb, latent space manipulations for textual style transfer are performed either via gradient descent (Wang et al.) 2019 Liu et al. (020) or by adding constant style vectors to the input (Shen et al.] 2020 Montero et al. 2021). In computer vision, discovering latent space manipulations for image style transfer has recently become a topic of in- creased interest, in both supervised (Jahanian al. 2020 Zhuang et al. 2021) and unsupervised ways (Härkönen et al.) 2020, Voynov & Babenko 2020). While these vision methods are similar to Emb2Emb conceptually, they differ from our work in important ways. First, they focus on the latent space of GANs (Goodfellow et al. 2014), which work well for image generation but are known to struggle with text (Caccia et al., 2020p. Secondly, images typically have a fixed size, and conse- quently their latent representations consist of single vectors. Our work focuses on data of variable size, which may have important insights for modalities other than text, c.g. videos and speech.
Unsupervised conditional text generation: Modern unsupervised conditional text generation ap- proaches are based on either (a) language models (LMs) or (b) autoencoders (AEs), (a) One type of attributes during pretraining (Keskar al. 2019), which puts
LM approach explicitly conditions restrictions on the training data that can be used for training. Another type adapts pretrained LMs for conditional text generation by learning modifications in the embedding space (Dathathri al., 2020). These approaches work well because LMS are pretrained with very large amounts of data and compute power, which results in exceptional generative ability (Radford et al. iet al., (2020) that even enables impressive zero-shot style transfer results (Reif et al. ever, in contrast AEs, LMs are not designed to have latent space that facilitates learning in it. We therefore argue that AE approaches could perform even better than LMs if they were given equal resources. This motivates our research. (b) A very common approach
2019 Brown 2021
How-
AE-based unsupervised conditional text generation is to learn shared latent space for input and output corpora that is ag- nostic to the attribute of interest (e.g., sentiment transfer (Shen et al. 2017), style transfer (Lample) et al. 2019), summarization (Liu et al. 2019), machine translation (Artetxe et al. 2018)). However, in these approaches, the decoder is explicitly conditioned on the desired attribute that must be avail- able for all data points, complicating pretraining on unlabeled data. To overcome this, Mai et al.) (2020) recently proposed Emb2Emb, which disentangles AE pretraining from learning to change the attributes via simple mapping. Our paper makes important contribution by improving the expressivity of Emb2Emb through variable-size representations.
7 CONCLUSION
Our paper addresses a fundamental research question from representation learning: How do we learn text representations in such a way that NLP tasks, specifically conditional text generation, can be learned in the latent space? Previous work (Emb2Emb) uses single-vector bottleneck autoencoders, which we argued to be fundamentally limited in how much text they can capture. We presented Bag- of- Vectors Autoencoders whose latent representation grows in size with the input as in attention- based models. We proposed LODrop regularization, Transformer+ and differentiable Hausdorff to facilitate training in its embedding space. Controlled experiments revealed that Bo V-AEs perform substantially better when the text is too long to encoded into a single vector, even of size d = 512.
Our study is fundamental in nature; we do not focus on any particular task. Instead, we systemati- cally demonstrate the benefit of Emb2Emb with variable-size representations rather than fixed-sized representations via controlled experiments. However, our model is in principle fit for the future. Mai et al.) (2000) showed that the Emb2Emb framework benefits immensely from unlabeled data. As such, given enough compute and data for large-scale pretraining, Bag-of- Vectors Autoencoders could have the potential to become a Foundation Model (Bommasani et al.) (021) like BERT and GPT-3. Our study paves the way for the application of BoV-AEs for unsupervised tasks by demon- strating how to learn in their latent space.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Applications The focus of our study is not any particular application, but concerns fundamental questions in unsupervised conditional text generation in general. Unsupervised applications are useful in scenarios where few annotations exists, which is particularly common in understudied low-resource languages (e.g. unsupervised neural machine translation (Kuwanto et al. (02TT). Of course, oftentimes unsupervised solutions perform worse than supervised ones, requiring extra care during deployment to avoid harm from potential mistakes.
Despite the fundamental nature of our study, we test our model on two concrete problems, a) text style transfer and b) sentence summarization. a) Style transfer has applications that are beneficial to society, such as expressing "complicated" text in simpler terms (text simplification) or avoiding potentially offensive language (detoxification), both of which are particularly beneficial for tradition- ally underprivileged groups such as non-native English speakers. However, the same technology can also be used maliciously by simply inverting the style transfer direction. In this paper, we decided to study sentiment transfer of restaurant reviews as style transfer task. The reasons are primarily practical; deriving both from the Yelp dataset, we can study the effectiveness of our model on two datasets (sentences and full reviews) that are very similar in content but considerably different in length. On one hand, this allows us to demonstrate the effectiveness of our model in a realistic, but computationally demanding setting. On the other hand, we can perform ablations in a less expen- sive setting. Apart from serving as a test bed for scientific research, sentiment transfer itself has no obvious real-world application. With enough imagination bad actor hacks into the database of a review platform like Yelp to e.g. manipulate the content of existing reviews. However, we rate this highly unrealistic due high opportunity cost, as it is much easier to generate fake reviews with large language models rather than hack into a system and alter existing reviews.
can construe scenario where
b) Summarization systems can be very valuable for society by enabling people to process informa- tion faster. But this depends on the system's output to be mostly factual, which neural summarization systems struggle with (Maynez et al. 2020). Unfaithful outputs may convey misinformation, which can potentially harm users.
Deployment While we argue above that sentiment transfer has no useful real-world application, the model can still deployed for demonstration purposes, or be trained and deployed for other tasks, e.g., sentence simplification. However, we urge not to deploy the models developed in this pa- per directly without adaptation for several reasons. i) The absolute performance is suboptimal (c.g., no large-scale pretraining) and hence makes many mistakes that a real-world application should avoid to prevent harm. ii) The model can occasionally produce toxie output. Of course, the ex- tent which this happens strongly depends on the training data. E.g.. Yelp restaurant reviews can sometimes contain vulgar language. Any real-world application should hence consider pre- and post-filtering methods. iii) The model might be biased towards certain populations, the extent of which is not the subject of this study. For example, the sentiment transfer models would likely work better for fast food restaurants than restaurants of African cuisine, because the former more com- mon in the mostly US-centric data that the model is trained on. A real-world application needs to consider the requirements of the target audience.
Similarly, we argue that the sentence summarization model studied in this paper needs further im- provements before deployment, some of which we mentioned in the main paper. Large-scale scage pre- training could also help to mitigate hallucinated facts (Maynez et al. 2020).
Dataset The Yelp-Reviews dataset direct derivative of the Yelp Open Datase Their license agreement states that any derivative remains the property of Yelp, hence we can not directly release the dataset. However, academic researchers can easily obtain their own license for non-commercial use and recreate the dataset used in this study via the script we provide in the supplementary material. No further data collection was conducted.
www. www.yyllp.ccm////
10

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We took several precautions to ensure that our work is reproducible.
Datasets Our study is based Yelp-Sentences style transfer. For these two datasets, we provide scripts that preprocess them as in our study. For Yelp-Reviews dataset, we provide detailed description in appendix [A.3.] More- over, we provide a script that allows to construct the dataset as a derivative from Yelp data. In order to get access to Yelp data, practitioners have to obtain license from Yelp that is free of charge. The data may only be used for non-commercial or academic purposes, but this suffices to reproduce our study. The Gigaword corpus is commonly used, and can be downloaded from the Linguistic Dataset Consortium at https: catallg 1dc. upenn LiddduLDC22122) For downloading, membership is mandatory, or otherwise fees apply. However, this commonplace in NLP research institutes.
existing datasets, Gigaword sentence summarization, and
Code We provide anonymized code to reproduce all our experiment in the supplementary materi- als. Upon acceptance, the code will be made available to the public.
Experiments We provide details on each experiment's setup in the appendix. However, it's im- practical to report all details that may impact the outcome. Therefore, for each experiment we additionally provide csv file in the supplementary material. The file contains information on all training parameters, model hyperparameters and results. In combination with the code, this allows to reconstruct almost the exact experimental setup used in our study apart from parameters that are beyond our control, such as the computation environment.
REFERENCES
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In International Conference on Learning Representations, 2018.
Orhun Utku Aydin, Abdel Aziz Taha, Adam Hilbert, Ahmed A Khalil, Ivana Galinovic, Jochen B Fiebach, Dietmar Frey, and Vince Istvan Madai. On the usage of average hausdorff distance for segmentation performance assessment: Hidden bias when used for ranking. arXiv preprint MxXiv:2009.00215. 2020.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2015.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv: 10t::200800525555 2021.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal J6zefowicz, and Samy Bengio. Generating sentences from a continuous space. In CoNLL, pp. 10-21, ACL, 2016.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language gans falling short. In International Conference on Learning Representations, 2020.
Xilun Chen. Learning Deep Representations for Low-Resource Cross-Lingual Natural Language Processing. PhD thesis, Cornell University, 2019.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosin- ski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference Learning Representations, 2020.
11

---

Under review as a conference paper at ICLR 2022
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLI (1), pp. 4171-4186. As- sociation for Computational Linguistics, 2019.
Marie-Pierre Dubuisson and Anil K. Jain. A modified hausdorff distance for object matching. In ICPR (1), pp. 566-568. IEEE, 1994.
Haoqiang Fan, Hao Su, and Leonidas J. Guibas. A point set generation network for 3d object reconstruction from single image. In CVPR, pp. 2463-2471. IEEE Computer Society, 2017.
Thibault Févry and Jason Phang. Unsupervised sentence compression using denoising auto- encoders. In CoNLI, pp. 413-422. Association for Computational Linguistics, 2018.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp. 2672-2680, 2014.
David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword. Linguistic Data Con- sortium, Philadelphia, 4(1):34, 2003.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwinska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538 (7626):471-77,, 2016.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In NAACL-HLL (2), pp. 107-112. Association for Computational Linguistics, 2018
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable GAN controls. In NeurIPS, 2020.
James Henderson. The unstoppable rise of computational linguistics in deep learning. In ACL, pp- Association for Computational Linguistics, 2020.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. Toward trolled generation of text. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 1587-1596. PMLR, 2017.
Daniel P. Huttenlocher, William Rucklidge, and Gregory A. Klanderman. Comparing images using the hausdorff distance under translation. In CVPR, pp. 654-656. IEEE, 1992.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the "steerability" of generative adversarial networks. In International Conference on Learning Representations, 2020.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint (iXiv:1909.05858, 2019.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representutions, 2015.
Diederik P. Kingma, Tim Salimans, Rafal Józefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improving variational autoencoders with inverse autoregressive flow. In NIPS, pp. 4736-4744, 2016.
Kalpesh Krishna, John Wieting, and Mohit lyyer. Reformulating unsupervised style transfer as paraphrase generation. EMNLP (1), pp. 737-762. Association for Computational Linguistics, 2020.
Yen-Ling Kuo, Boris Katz, and Andrei Barbu. Compositional networks enable systematic general- ization for grounded language understanding. arXiv preprint arXiv:2008.0222 02742, 2020.
12

---

Under review as a conference paper at ICLR 2022
Garry Kuwanto, Afra Feyza Akyürek, Isidora Chara Toumi, Siyang Li, and Derry Wijaya. Low- resource machine translation for low-resource languages: Leveraging comparable data, code- switching and compute resources. arXiv preprint arXiv:2103.1327., 2021.
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc' Aurelio Ranzato, and Y-Lan Boureau. Multiple-attribute text rewriting. In International Conference on Learning Representations, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to- -orenceet--seueecee pre- training for natural language generation, translation, and comprehension. In ACL, pp. 7871-7880. Association for Computational Linguisties, 2020.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74- 2004.
Kwan-Ho Lin, Kin-Man Lam, and Wan-Chi Siu. Spatially wigen-weighted hausdorff distances for human face recognition. Pattern Recognit. 36(8):1827-1834, 2003.
Dayiheng Liu, Jie Fu, Yidan Zhang, Chris Pal, and Jiancheng Lv. Revision in continuous space: Unsupervised text style transfer without adversarial learning. In AAAI, pp. 8376-8383. AAAI Press, 2020.
Peter Liu, Yu-An Chung, and Jie Ren. Summae: Zero-shot abstractive text summarization using length-agnostic auto-encoders. arXiv preprint arXiv:1910.00998, 2019.
Yue Lu, Chew Lim Tan, Weihua Huang, and Liying Fan. An approach word image matching based on weighted hausforff distance. In ICDAR, pp. 921-925, IEEE Computer Society, 2001.
Florian Mai, Nikolaos Pappas, Ivan Montero, Noah A. Smith, and James Henderson. Plug and play autoencoders for conditional text generation. EMNLP (1), pp. 6076-6092. Association for Computational Linguistics, 2020.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. On faithfulness and fac- tuality in abstractive summarization. ACL, pp. 1906-1919. Association for Computational Linguistics, 2020.
Ivan Montero, Nikolaos Pappas, and Noah A Smith. Sentence bottleneck autoencoders from trans- former language models. arXiv preprint arXiv:2 2021.
Courtney Napoles, Benjamin Van Durme, and Chris Callison-Burch Evaluating sentence com- pression: Pitfalls and suggested remedies. In Monolingual ACL, pp. 91-97. Association for Computational Linguistics, 2011.
Sarana Nutamong, Chenyun Yu, Raheem Sarwar, Peter Xu, and Dickson Chow. A scalable frame- work for stylometric analysis query processing. In ICDM, pp. 1125-1130. IEEE Computer Soci- ety, 2016.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In ACL, pp. 311-318. ACL, 2002.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-IILT, pp. 2227-2237. Association for Computational Linguistics, 2018
Maxime Peyrard. A simple thcoretical model of importance for summarization. In ACL (1), pp. 1059-1073. Association for Computational Linguistics, 2019.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 8821 8821-8831. PMLR, 2021.
13

---

Under review as a conference paper at ICLR 2022
Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burnn and Jason Wei. A recipe for arbitrary text style transfer with large language models. arXiv preprint arXiv:2109.03910, 2021.
Javier Ribera, David Guera, Yuhao Chen, and Edward Delp. Locating objects without bounding boxes. In CVPR, pp. 6479-6489. Computer Vision Foundation / IEEE, 2019.
Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sen- tence summarization. In EMNLP, pp. 379-389. The Association for Computational Linguistics, 2015.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistiBBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 2019.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with bointer-generator networks. In ACL (1), pp. 1073-1083. Association for Computational Linguis- tics, 2017.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. Style transfer from non-parallel text by cross-alignment. In NIPS, pp. 6830-6841, 2017.
Tianxiao Shen, Jonas Mueller, Regina Barzilay, and Tommi S. Jaakkola. Educating text autoen- coders: Latent representation guidance via denoising. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 9719-8729. PMLR, 2020.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1): 1929-1958, 2014.
Abdel Aziz Taha and Allan Hanbury. Metrics for evaluating 3d medical image segmentation: anal- ysis, selection, and tool. BMC Medical Imaging, 15:29, 2015.
Barnahás Takács. Comparing face images using the modified hausdorff distance. Pattern Recognit., 31(12):18331181, 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, pp. 5998-599800 2017.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the GAN latent space. In ICML, volume 119 of Proceedings of Machine Learning Research, pp- 9786-9796. PMLR, 2020.
Ke Wang, Hang Hua, and Xiaojun Wan. Controllable unsupervised text attribute transfer via editing entangled latent representation. In NewrlPS, pp. 11034-11044 2019.
Jingjing Xu, Xu Sun, Qi Zeng, Xiaodong Zhang. Xuancheng Ren, Houfeng Wang, and Wenjie Li. Unpaired sentiment-to-sentimeen translation: A cycled reinforcement learning approach. In ACL (1), pp. 979-988. Association for Computational Linguistics, 2018.
Dani Yogatama, Cyprien de Masson d' Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. Learning and evalu- ating general linguistic intelligence. arXiv preprint arXiv;I901. 11373, 2019.
Biao Zhang, Ivan Titov, and Rico Sennrich. On sparsifying encoder outputs in sequence-to-sequeeee models. In ACI/IJCNLP (Findings), volume ACI ACL/UCNLP 2021 of Findings of ACI, pp. 2888- 2900. Association for Computational Linguistics, 2021.
Jianan Zhao, Fengliang Qi, Guangyu Ren, and Lin Xu. Phd learning: Learning with pompeiu- hausdorff distances for video-basee vehicle re-identification. In CVPR, pp. 2225-2235. Computer Vision Foundation / JEEE, 2021.
Peiye Zhuang, Oluwasanmi O Koyejo, and Alex Schwing. Enjoy your editing: Controllable {gan]s for image editing via latent space navigation. In International Conference Learning Represen- tations, 2021.
14

---

Under review as a conference paper at ICLR 2022
Appendix
A EXPERIMENTAL DETAILS
Here, we describe the experimental setup used in our experiments. We try to be exhaustive, but the exact training configurations and code will also be given downloadable source code for reference.
Table Basic statistics for each dataset used in this study. Average number of words refers input texts and output texts, respectively.
Dataset Yelp-Sentences
avg. #words
#inputs
#outputs
9.778.5
177k
267k
Gigaword
27.2 8.2
500k
500k
Yelp-Reviews
56.1/48.7
500
500k
A.1 AUTOENCODER PRETRAINING
All autoencoders consist of standard Transformer encoders and decoders Vaswani et al. 2017). with 3 encoder and decoder layers, respectively. The Transformers have 2 heads and the dimensionality is set to the same the latent vectors (Yelp-Reviews: 512, Yelp-Sentences: 32, Gigaword: 128).
In case of the fixed sized model, the representations at the last layer are averaged. Otherwise we perform LODrop described in Section 3 We set AL. 10 for all BoV models and only vary the target ratio. All models are trained with a dropout (Srivastava et al., 2014) probability of 0.1 and denoising objective, i.e tokens have chance of 10% to be dropped from the sentence. We train the model with the Adam optimizer (Kingma & Ba 2015 with an initial learning rate of Ir 0.00005 (Yelp-Reviews and Gigaword) or Ir 0.0001 (eelp-Sentences) and batch size of 64. We experimented with other learning rates (0.00005, 0.0005) for the fixed-size model on Yelp- Reviews, but the results did not improve. Models are trained for million steps on Gigaword and Kelp-Reviews and for 1.5 million steps on Yelp- Selp-Sentences. We check the validation set performance every 20,000 steps and select the best model according to validation reconstruction performance.
A.2 DOWNSTREAM TASK TRAINING
After the autoencoder pretraining, we train downstream by freezing the parameters of the encoder and decoder. The dimensionality of the one-layer mapping &b (a Transformer decoder with 4 heads) set to the same as the latent representation (Yelp-Reviews: 512, Yelp-Sentences: 32, Gigaword: 128). We set the maximum number of output vectors to N 250 on Yelp-Reviews and Gigaword, and N = 30 on Yelp-Sentences. The batch size 64 for Yelp-Sentences and Gigaword and 16 on Yelp-Revie Reviews We train for 10 epochs on Yelp- felp-Sentences and Gigaword, and for 3 epochs Yelp-Reviews. The validation performance is evaluated after each epoch.
Losses: In all tasks we have two loss components. For Lzim. we use differentiable Hausdorff unless specified otherwise (in the ablation). Lsty and Lien depend on classifiers / regressors, which we train separately after the autoencoder pretraining are then averaged and plugged into a one-layer MLP whose hidden size is half of the input size and uses the lanh actívation function. These classifiers trained via Adam (Or 0.0001) for 10 epochs and we evaluate the validation set performance after each. The total loss depends on a window size as described in Equation 4 For performance reasons (multiple computations of the loss), we set k = 0 unless specified differently.
one-layer Transformer encoder. The embeddings
A.3 YEVP-REVIEWS
A.3.1 DATASET
The dataset obtained from https:://ww caappp..cm//aaeetteetee in May 2021. Our goal is to obtain texts long enough such they cannot be reconstructed by a reasonably sized autoencoder with single-vector bottleneck. We find that be the case when limiting ourselves reviews of
15

---

Under review as a conference paper at ICLR 2022
maximum 100 words. We apply this limit due to the computational complexity of Transformers on long texts. Otherwise, we stick with similar filtering criteria as [Shen et al. (2017): We only consider restaurant businesses. We consider reviews with or 2 stars negative, and reviews with 5 stars as positive. We don't consider reviews with or 4 stars to avoid including neutral reviews. We subsample 400,000 positive and negative reviews for training, respectively, and use 50,000 for validation and test set each.
In order demonstrate the usefulness of our model on long texts, we d Our goal is to obtain texts long enough such they cannot be reconstructed by a reason- ably sized autoencoder with a single-vector bottleneck. We find that be the case when limiting ourselves to reviews of maximum 100 words Otherwise, we stick with similar filtering criteria as stars
the original Yelp
Shen et al. 2017): We only consider restaurant businesses. We consider reviews with as negative, and reviews with 5 stars positive. We don't consider reviews with 3 or 4 stars to avoid including neutral reviews. We subsample 400,000 positive and negative reviews for training, respectively, and use 50,000 for validation and test set each.
A.3.2 DOWNSTREAM TRAINING
For both the fixed-size model and the BoV model (L0-0.1), we choose the best learning rate among Ir 0.0001 and Ir 0.0005 the validation set and report test set results. We train with Luty E (0.1, 0.5, 1, 2, 5, 10), resulting in the scatter plot in Figure 3
A.4 YELP-SENTENCES
A.4.1 DATASET
Seip-Sentences consists of the sentiment transfer dataset created by Shen et al. 2017), who made their data available at https://githcb. com/shentt ianeetiaaiia///// We use their data as is without further preprocessing. presents some basic statistics about this dataset.
language
A.4.2 DOWNSTREAM TRAINING
We train BoV models with Asty 11, 10, 20, 50, 100). To make that results are not due to insufficient tuning, for the fixed-sized model, we use the following larger range: (0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 10, 20, 50, 100). All configurations trained with by 0.0005. These results produce the seatter plot in Figure
A.4.3 ABLATIONS
For the ablations on differentiable Hausdorff distance and the window size, we use the L0-0.6 model. For each option, we train with Lsty E {0.1, 0.2, 0.5, 2, 5, 10, 20, 40, 60, 80, 100) and report the best value terms of style transfer score on the validation set.
A.5 SENTENCE SUMMARIZATION
A.5.1 DATASET
The dataset is based on the Gigaword corpus (Graff et al. 2003). We largely follow the pre- processing in Rush al. (015 which obtained from the paper's GitHub repository at NAMAS. Different from them, we convert all inputs and outputs to lower case and use smaller split (1 million examples). We provide the scripts for constructing the dataset from a copy of the Gigaword corpus (which can be obtained from the Linguistic Dataset Consortium) together with the rest of our code.
bttps://githhb.cco com/ facebookarchive
'The dataset was obtained from https: /ww.///wwwooo yew wttp/////a/ceeeeeo in May 2021. We apply this limit due the computational complexity of Transformers long
16

---

Under review as a conference paper at ICLR 2022
0.325
Style transfer score
0.225
0,200
0.175
0.150
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Window size
Figure 6: Style transfer score depending on the window size.
A.5.2 DOWNSTREAM TRAINING
We train all models with ir 0.00005. For each target ratio and each of Transformer and Transformer+ we select the best Aten E (0.1...2, 0.5, 1, 2, 5, 10) terms of ROUGE-L on the validation set and report test results in Table
B ADDITIONAL RESULTS
B. YELP-REVIEWS
In Figure 7 we plot the reconstruction ability of the fixed-size model compared to the BoV-AEs on the validation set.
large dimensionality (d = 512), the single-vector model achieves substantially lower reconstruction ability than BoV-AE. With respect the target sparsity rate, we find that 0.1 is enough to reach dramatically better results than the fixed-size model, whereas r = 0.05 only reaches slightly better results after two million training steps. However, the plot shows clearly that L0-0.05 has not converged, suggesting that L0-0.05 could reach much better performance if trained for even longer.
Again, despite
B.2 YELP- SELP-SENTENCES
B.2.1 WINDOW SIZE
In Figure 6, plot the style transfer score function of the window size. As already described Section 5.2 the performance improves almost monotonically up to a certain point, after which performance decreases again. This demonstrates the importance of choosing the right window size.
B.2.2 COMPUTATION TIME
Our experiments have shown that bag-of-vector representations are more powerful than single-vector representations. However, the increased capacity of BoV-AE comes at the expense of higher com- putation time. The size of the latent representation impacts the computation time in two places: Dur- ing cross attention in the decoder and when computing the mapping. Asymptotically, the decoder's cross-attention mechanism computes O(n |sl) dot- doodpuru where n is the number vectors in the latent representation and |s] is the length of the text sequence When computing the mapping.
17

---

Under review as a conference paper at ICLR 2022
Table 3: Asymptotic computation time in the Emb2Emb framework as function of the latent
representation size and the length of the input text |s], depending on the type of autoencoder.
AE type
Cross-Attention Decoding
Mapping
general fixed BoV-AE
On |
On
O([s]
0(1)
Table 4: The number of seconds it takes to process 5% of the validation set (1264 samples) with batch size 1. Lower is better.
Model
Encoding
Mapping
Decoding
fixed
4.8
2.4
51.7
L0-0.4
7.2
12.6
50.1
L0-0.8
7.3
20.6
50.3
both at training and inference time, we produce fixed number of vectors autoregressively, but in most applications, N can reasonably be bound by linear function of n (e.g., 2n in style transfer or 0.5n summarization). The mapping essentially a Transformer decoder, so both the cross attention and self attention parts compute O(n²) dot-products. Given that = 1 for single-vector AEs and n = ([ss)) for B-V-AEs with LODrop, we obtain the asymptotic complexities as shown in Table 3
To assess the empirical impact, we measure the wallclock time Emb2Emb's "Inference" stage (ef. Figure We take separate measurements for encoding, mapping, and decoding, respectively. Since decoding speed depends on the quality of generation (e.g.. when the end-of-sequence symbol generated late due repetitions), we do the following to enable fairer comparisons. We enforce the same fixed number of decoding steps (10) in all models. The mapping is set to produce as many output vectors input vectors. We use batch size of 1, but note that the results would largely extend to larger batch sizes when binned batching is used. The results are shown in Table 4
Both the encoding and the mapping stages of Emb2Emb are more expensive in BoV models than in the fixed-size model. The difference in the encoding stage can be explained by the overhead through the LODrop layer, which includes identifying near-zero gates and discarding their respective vec- tors. The difference in the mapping grows with higher LODrop target ratios. This is expected since number of autoregressive steps decreases with the target ratio. Finally, we do not observe any meaningful speed differences between the models at decoding time. This is somewhat surprising, but could be explained by two factors. First, the self-attention part of the decoder already has a com- plexity of ()s)),, which probably dominates the total computation time. Secondly, the computation of the dot-product is casy to parallelize. In summary, we find that BoV models are slower overall, especially in the mapping. However, since our LODrop implementation prunes near-zero vectors, lower target rates mitigated the additional computation overhead. This is especially evident when comparing training speeds. While L0-0.8 processes 15 sentences per second, L0-0.4 processes can process 21 (fixed-size: 42).
B.2.3 QUALITATIVE ANALYSIS
Standard autoencoders suffer from poor performance with Emb2Emb if the text is too long to be encoded into a single vector. BoV-AEs were designed to alleviate this issue. Here, we conduct qualitative analysis of 10 randomly selected model outputs on Yelp-Sentences. For comparability, we select models with similar levels of style transfer accuracy, namely the fixed size model with performance of 59% accuracy and 17 points self-BLEU accuracy and 38 points Belf-BLEU, We randomly sample 10 examples and show them Table 5 By design of the Yelp-Sentences dataset (Shen et al. 2017). the input sentences from negative reviews, whose sentiment are supposed to be changed to positive. Note that due to how the dataset constructed, some of the input sentences already positive (#7) or just neutral (#2).
L0-0.4 with performance of 55%
sentences drawn
We observe several trends:
18

---

Under review as a conference paper at ICLR 2022
# Input sentence
Output of fixed-size model
Output of L0-0.4
but there here here and will enjoy it. 2 then why did n't they put some in ?
generally speaking it was
generally remain was worth and always happy
nothing worth coming back
to.
then she you ta are the in the ?
then why n they put some delicious
3
horrible experience! !
horrible!
horrible experience
was shame because we
was fun, there and we have been to
really nice shame because
were really looking forward to dining there. suffice stay, this is not great place to stay.
we were really looking for- ward forward and fantastic ! suffice is not stay this is great place and always great
5
suffice not stay this place is a stay.
6
the chicken was weird. my mom ordered the mar- garita panini which was
the chicken was weird. my my margarita was or- dered which was very good
the chicken was weird my mom ordered the mar- garita panini which was pretty good.
7
pretty good.
8
i i'm not willing to take the chance..
will definitely recommend your time or you. had this place at the food, it 's super.
not willing to take the great
9
i would say for the price point that it uninspired
would say for the price point that it was delicious.
10 the only pool complaint i have was from the last day of our stay.
the waitress was the the the the time here a last time
the only pool complaint i have was from the day was wonderful
Table 5: 10 randomly sampled examples from Yelp-Sentences.
1. The fixed-sized model has a difficult time retaining the aspect discussed in the input sen- tence (#10: staff instead of location, #9: food instead of price). whereas the BoV-AE stays topic. This is likely consequence of the fixed-sized model's inability to encode the input well into single vector.
2. The outputs of the fixed-sized models are often completely unusable (#1, #2) or nonsensical (45, #9, #10), whereas the outputs of the BoV-AE are at least intelligible.
3. In absolute terms, the outputs of neither model are reliably grammatical or able to flip the sentiment. This is understandable since large pretrained language model used. We would like to stress that therefore our models should not be used in production as they are (cf. Ethics Statement). Large-scale pretraining needed produce coherent outputs (Brown et al. (020). which then produces impressive outputs on style transfer (Reil) et al. 2021). As we argue Section | and our paper contributes the foundation for large scale pretraining of autoencoder models to be used in Emb2Emb.
B.3 SENTENCE SUMMARIZATION
Figure shows the development of the reconstruction loss on the validation set over the course of million training steps. Despite moderately sized dimensionality (d 128), the single-vecte bot- tleneck model achieves only considerably lower reconstruction performance than the BoV models. Moreover, we can see again that larger target rates lead to faster convergence, and all BoV models converge to approximately the same validation loss value.
19

---

Under review as a conference paper at ICLR 2022
Validation reconstruction loss
Validation reconstruction loss
run
fixed
LO-0.05
LO-0.1
LO-0.2
4
L0-0.4
3
2
500 K
M steD
1.5 M
2
Figure 7: Reconstruction loss on the validation set of Yelp-Reviews for different autoencoders. fixed: The bag consists of single vector obtained by averaging the embeddings at the last layer of the Trans- former encoder. L0-r: BoV-AE with LODrop target ratio
Validation reconstruction loss
Validation reconstruction loss
run
fixed
LO-0.2
LO-0.4
LO-0.6
L0-0.8
2-
0
500 K
1 M step
1.5 M
2 M
Figure 8: Reconstruction loss on the validation set of Gigaword for different autoencoders. fixed: The bag consists of single vector obtained by averaging the embeddings at the last layer of the Trans- former encoder. L0-r: BoV-AE with LODrop target ratio
20


Reviews:
[{"review_id": "nbPhohIpyM9", "reviewer": "Reviewer_jHbb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Here are some suggestions and questions:\n- I think the authors should describe more about the autoencoder. For example, is the input of the encoder is a bag of words or sequential tokens (a bag of words + positional encoding)? If it's the former case, why ignore orders? The order should provide lots of information about semantics. If it's the latter case, it is weird to view the embedded as a bag of vectors since the order indeed matters a lot. \n- If the authors treat the latent representations as a bag of vectors (order doesn't matter), why does the transform function Phi uses autoregressive decoding to produce z_y? What is the order for autoregressive decoding? Is it the same order as the order of the original tokens? Also, if the order doesn't matter a lot, why not use Trasnforemer to generate all z_y at the same time?\n- It would be great if the author can provide some other baselines, such as Mai et al. (2020), although they are using LSTM.\n- Any ablation studies for the pointer-generator technique?\n- From my perspective, if only replacing the autoencoder with a stronger one, the novelty is not enough. The important part should be handling the transform between multiple vectors instead of a single vector. However, there is a lot of unclear parts for handling the transform between multiple vectors when I read the paper.", "summary_of_the_paper": "This paper extends previous work, Emb2Emb, by replacing the LSTM autoencoder with a Transformer autoencoder. The authors propose some techniques to handle the difficulty of this replacement. They also conduct some experiments to support the claim. ", "summary_of_the_review": "I think the novelty is not enough and some details should be described more clearly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kEeu0YOZ9bA", "reviewer": "Reviewer_hEPH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS\n\n- The paper considers the general problem of learning an encoder that encodes the input to a variable-length sequence. This can be potentially useful in settings beyond unsupervised conditional text generation. \n\n- The proposed approach (e.g., using Hausdorff for alignment) is sensible and effective.\n\n- BoV-AEs clearly outperform Emb2Emb on a variety of tasks and metrics. \n\n\nWEAKNESSES\n\n- The experiments do not involve any other baseline. The authors say upfront that the goal of the experiments is to confirm that the multi-vector extension improves over Emb2Emb and that their proposed techniques are necessary, but a total lack of comparisons with existing works in this area makes it difficult for the reader to contextualize the contribution of the work with respect to methods other than the authors' own. \n\n- The writing is a bit unclear, though it can be followed with some efforts. For instance, the authors write \"X = {z_1 ... z_n} := enc(x)\" in the first paragraph of Section 3 without defining \"n\". It took me a while to guess that this is the input bag size that the user provides. \n\n- I'm not an expert on this topic, but it seems unlikely that this is the only work that considers multi-vector autoencoding. See [1] for example. But there is none discussed. \n\n[1] SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression (Baziotis et al., 2019)", "summary_of_the_paper": "The paper extends the unsupervised conditional text generation framework of Emb2Emb (Mai et al., 2020) from a single encoding to variable-length encodings. The new model is called a bag-of-vectors autoencoders (BoV-AE). To train BoV-AEs, the paper develops a regularization technique called L0Drop, an extension of OffsetNet in Emb2Emb for learning the mapping Phi, and an alignment loss based on the Hausdorff distance. Experiments show that BoV-AEs outperforms Emb2Emb on sentiment transfer and summarization in the reconstruction loss, Rouge/Bleu, and accuracy.", "summary_of_the_review": "The paper presents a reasonable multi-vector extension of Emb2Emb (Mai et al., 2020), but is a bit lacking in comparison with other existing works. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Y449MSM1dw", "reviewer": "Reviewer_kHZy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- This is a technically sound submission which appears to be novel.\n- The idea is relatively simple and intuitively appealing. It makes sense that the representational capacity of a fixed-sized embedding may be limited, and therefore utilizing variable-sized embeddings is a natural extension.\n    - Figures 1 and 2 are clear in describing the proposed method at a high level.\n- The empirical results on style transfer and summarization appear to be strong (although as mentioned below, Figures 3 and 5 are unclear).\n\nWeaknesses:\n\n- The writing is unclear in a number of places.\n    - Figure 2\n        - Caption mistakenly says \\hat{\\mathbf{z}}_x instead of \\hat{\\mathbf{z}}_y in part called \"Task Training\".\n    - Equation (2)\n        - X should be replaced by \\mathbb{X}\n    - Section 4.1 paragraph 3\n        - It is slightly confusing to say that \\mathbf{z}_t' is the output of the Transformer. Maybe it would be clearer to actually use the notation \\mathbf{z}_t' in Equation (2) and then define it immediately thereafter?\n    - Section 4.2 paragraph 2\n        - It is not especially clear what is meant by \"we minimize the loss locally at every step\". Presumably, as described in Section 2, this is task dependent and perhaps something like the loss described in Section 2 paragraph 2, but it would be useful to the reader if this were described somewhat more explicitly.\n    - Section 5.1 paragraph 2\n        - Should the Figure being referred to be 7 and not 8?\n    - Section 5.3 paragraph 1\n        - It would be useful to add a short description (or citation) to explain what a length regressor is.\n    - Figures 3 and 5\n        - It is not clear what the different points for either fixed or each value of L0 represent?\n        - Based on the score shown in Section 5 paragraph 3, why isn't there only one point for each model?\n    - Some of the important equations would be better presented as blocks rather than inline, e.g. the loss in Section 2 paragraph 2, the score in Section 5 paragraph 3, etc.\n- L0Drop\n    - Suppose one of the gates g_i = 0, which then means \\mathbf{z}_i = \\mathbf{0}. In Figure 2 top, is the Transformer decoder still able to attend to this \\mathbf{z}_i? And in Equations (2) and (3), is this \\mathbf{z}_i a component of the computations (in particular, will it still have a non-zero contribution to the softmax both in the Transformer in Equation (2) and in the \\alpha_i term in Equation (3))?\n        - It could be argued that if so, the encoder is not outputting a bag of vectors whose size is smaller than the number of tokens, because a 0-vector is different from there not being a vector at all.\n        - Ideally, this \\mathbf{z}_i would be completely masked out during the decoding stage rather than being set to 0.\n- Experiments\n    - It would add a lot of value to the paper to see examples of the model outputs on sentiment transfer and sentence summarization.\n    - A discussion on the computational efficiency compared to the baseline would also be very useful.", "summary_of_the_paper": "This paper presents an autoencoding model which is trained for conditional text generation tasks in an unsupervised manner. Rather than using a fixed-sized bottleneck layer in the autoencoder, the authors present a method for learning a variable-sized set of representations (referred to as a bag of vectors). This bag of vectors can then be transformed (in embedding space) in order to generate a modified sequence of text based on the original input sequence.", "summary_of_the_review": "Overall this is a good paper which, with some adjustments to the writing, would be of good value to the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3YoFKPkGs4h", "reviewer": "Reviewer_74E4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I'm a bit confused how the mapping $\\Phi$ is learned. As you generate vectors autoregressively, it's actually not a mapping between two bags of vectors, i.e., the probability of generating $(z_1,z_2)$ can be different from the probability of generating $(z_2,z_1)$. I also think there are many problems with the notations. In the text between Eq. (2) and Eq. (3), you said \"$p_{gen}$ is a function of the output $z_t'$ of Transformer...\". What is $z_t'$? Is it $\\hat z_t$? But $\\hat z_t$ is determined after $p_{gen}$. I couldn't follow this part. In Sec 4.2, what is $\\mathbb Y$ in $\\mathcal L(\\mathbb X, \\mathbb Y)$? I don't see it defined anywhere. Also, you've defined $\\mathbb X:=\\text{enc}(x)$ in the beginning of Sec. 3, but now you seem to use it as the output of $\\Phi$.\n\nThe experimental results lag far behind previous work. On Yelp-Sentences, He et al. (2020) achieved accuracy 87.90 and self-BLEU 48.38, while in Fig. 5 the proposed model has self-BLEU only 20 for the same level of accuracy. On sentence summarization, the proposed model has ROUGE-L only 18.3, while Rush et al. reached 28.34 in 2015, not to mention the results in recent years. Although it's not completely comparable as you use less training data and no parallel sentences, it is unclear whether the proposed model is usable with such poor results. There are no sentence examples provided, nor is human evaluation conducted.\n\n[1] He et al. (2020). A Probabilistic Formulation of Unsupervised Text Style Transfer.\n", "summary_of_the_paper": "This paper proposes a text autoencoder with a bag-of-vectors embedding, which is more capable of encoding longer text than a single-vector embedding. To train such a model, the authors first encode the input sequence into a vector sequence of equal length, and then add a gate to each vector and encourage the model to keep a small number of open gates.\n\nThe proposed BoV-AE can be used in (non-parallel) seq2seq tasks by learning a mapping within the embedding space, similar to the Emb2Emb method proposed by Mai et al. The difference is that the mapping is now between two bags of vectors instead of between two vectors. The authors use a Transformer with a copy mechanism and an offset vector to produce a bag of vectors. To align two bags of vectors, the authors adjust the average Hausdorff distance to a soft version to make training smoother.\n\nExperiments are conducted on a sentiment transfer task and a sentence summarization task. BoW-AE shows better performance compared to AE with a single-vector embedding.\n\n", "summary_of_the_review": "I recommend rejecting this paper because of its unclear method description, poor results, and insufficient evaluations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses unsupervised conditional text generation extending emb2emb (Mai et al, 2020) with bag-of-vectors antoencoders.

Reviewers shared several concerns about the clarity of this paper and empirical results.
Decision: Reject