Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a text autoencoder with a bag-of-vectors embedding, which is more capable of encoding longer text than a single-vector embedding. To train such a model, the authors first encode the input sequence into a vector sequence of equal length, and then add a gate to each vector and encourage the model to keep a small number of open gates.

The proposed BoV-AE can be used in (non-parallel) seq2seq tasks by learning a mapping within the embedding space, similar to the Emb2Emb method proposed by Mai et al. The difference is that the mapping is now between two bags of vectors instead of between two vectors. The authors use a Transformer with a copy mechanism and an offset vector to produce a bag of vectors. To align two bags of vectors, the authors adjust the average Hausdorff distance to a soft version to make training smoother.

Experiments are conducted on a sentiment transfer task and a sentence summarization task. BoW-AE shows better performance compared to AE with a single-vector embedding.


main_review: I'm a bit confused how the mapping $\Phi$ is learned. As you generate vectors autoregressively, it's actually not a mapping between two bags of vectors, i.e., the probability of generating $(z_1,z_2)$ can be different from the probability of generating $(z_2,z_1)$. I also think there are many problems with the notations. In the text between Eq. (2) and Eq. (3), you said "$p_{gen}$ is a function of the output $z_t'$ of Transformer...". What is $z_t'$? Is it $\hat z_t$? But $\hat z_t$ is determined after $p_{gen}$. I couldn't follow this part. In Sec 4.2, what is $\mathbb Y$ in $\mathcal L(\mathbb X, \mathbb Y)$? I don't see it defined anywhere. Also, you've defined $\mathbb X:=\text{enc}(x)$ in the beginning of Sec. 3, but now you seem to use it as the output of $\Phi$.

The experimental results lag far behind previous work. On Yelp-Sentences, He et al. (2020) achieved accuracy 87.90 and self-BLEU 48.38, while in Fig. 5 the proposed model has self-BLEU only 20 for the same level of accuracy. On sentence summarization, the proposed model has ROUGE-L only 18.3, while Rush et al. reached 28.34 in 2015, not to mention the results in recent years. Although it's not completely comparable as you use less training data and no parallel sentences, it is unclear whether the proposed model is usable with such poor results. There are no sentence examples provided, nor is human evaluation conducted.

[1] He et al. (2020). A Probabilistic Formulation of Unsupervised Text Style Transfer.

summary_of_the_review: I recommend rejecting this paper because of its unclear method description, poor results, and insufficient evaluations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper extends previous work, Emb2Emb, by replacing the LSTM autoencoder with a Transformer autoencoder. The authors propose some techniques to handle the difficulty of this replacement. They also conduct some experiments to support the claim. 
main_review: Here are some suggestions and questions:
- I think the authors should describe more about the autoencoder. For example, is the input of the encoder is a bag of words or sequential tokens (a bag of words + positional encoding)? If it's the former case, why ignore orders? The order should provide lots of information about semantics. If it's the latter case, it is weird to view the embedded as a bag of vectors since the order indeed matters a lot. 
- If the authors treat the latent representations as a bag of vectors (order doesn't matter), why does the transform function Phi uses autoregressive decoding to produce z_y? What is the order for autoregressive decoding? Is it the same order as the order of the original tokens? Also, if the order doesn't matter a lot, why not use Trasnforemer to generate all z_y at the same time?
- It would be great if the author can provide some other baselines, such as Mai et al. (2020), although they are using LSTM.
- Any ablation studies for the pointer-generator technique?
- From my perspective, if only replacing the autoencoder with a stronger one, the novelty is not enough. The important part should be handling the transform between multiple vectors instead of a single vector. However, there is a lot of unclear parts for handling the transform between multiple vectors when I read the paper.
summary_of_the_review: I think the novelty is not enough and some details should be described more clearly.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper extends the unsupervised conditional text generation framework of Emb2Emb (Mai et al., 2020) from a single encoding to variable-length encodings. The new model is called a bag-of-vectors autoencoders (BoV-AE). To train BoV-AEs, the paper develops a regularization technique called L0Drop, an extension of OffsetNet in Emb2Emb for learning the mapping Phi, and an alignment loss based on the Hausdorff distance. Experiments show that BoV-AEs outperforms Emb2Emb on sentiment transfer and summarization in the reconstruction loss, Rouge/Bleu, and accuracy.
main_review: STRENGTHS

- The paper considers the general problem of learning an encoder that encodes the input to a variable-length sequence. This can be potentially useful in settings beyond unsupervised conditional text generation. 

- The proposed approach (e.g., using Hausdorff for alignment) is sensible and effective.

- BoV-AEs clearly outperform Emb2Emb on a variety of tasks and metrics. 


WEAKNESSES

- The experiments do not involve any other baseline. The authors say upfront that the goal of the experiments is to confirm that the multi-vector extension improves over Emb2Emb and that their proposed techniques are necessary, but a total lack of comparisons with existing works in this area makes it difficult for the reader to contextualize the contribution of the work with respect to methods other than the authors' own. 

- The writing is a bit unclear, though it can be followed with some efforts. For instance, the authors write "X = {z_1 ... z_n} := enc(x)" in the first paragraph of Section 3 without defining "n". It took me a while to guess that this is the input bag size that the user provides. 

- I'm not an expert on this topic, but it seems unlikely that this is the only work that considers multi-vector autoencoding. See [1] for example. But there is none discussed. 

[1] SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression (Baziotis et al., 2019)
summary_of_the_review: The paper presents a reasonable multi-vector extension of Emb2Emb (Mai et al., 2020), but is a bit lacking in comparison with other existing works.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an autoencoding model which is trained for conditional text generation tasks in an unsupervised manner. Rather than using a fixed-sized bottleneck layer in the autoencoder, the authors present a method for learning a variable-sized set of representations (referred to as a bag of vectors). This bag of vectors can then be transformed (in embedding space) in order to generate a modified sequence of text based on the original input sequence.
main_review: Strengths:

- This is a technically sound submission which appears to be novel.
- The idea is relatively simple and intuitively appealing. It makes sense that the representational capacity of a fixed-sized embedding may be limited, and therefore utilizing variable-sized embeddings is a natural extension.
    - Figures 1 and 2 are clear in describing the proposed method at a high level.
- The empirical results on style transfer and summarization appear to be strong (although as mentioned below, Figures 3 and 5 are unclear).

Weaknesses:

- The writing is unclear in a number of places.
    - Figure 2
        - Caption mistakenly says \hat{\mathbf{z}}_x instead of \hat{\mathbf{z}}_y in part called "Task Training".
    - Equation (2)
        - X should be replaced by \mathbb{X}
    - Section 4.1 paragraph 3
        - It is slightly confusing to say that \mathbf{z}_t' is the output of the Transformer. Maybe it would be clearer to actually use the notation \mathbf{z}_t' in Equation (2) and then define it immediately thereafter?
    - Section 4.2 paragraph 2
        - It is not especially clear what is meant by "we minimize the loss locally at every step". Presumably, as described in Section 2, this is task dependent and perhaps something like the loss described in Section 2 paragraph 2, but it would be useful to the reader if this were described somewhat more explicitly.
    - Section 5.1 paragraph 2
        - Should the Figure being referred to be 7 and not 8?
    - Section 5.3 paragraph 1
        - It would be useful to add a short description (or citation) to explain what a length regressor is.
    - Figures 3 and 5
        - It is not clear what the different points for either fixed or each value of L0 represent?
        - Based on the score shown in Section 5 paragraph 3, why isn't there only one point for each model?
    - Some of the important equations would be better presented as blocks rather than inline, e.g. the loss in Section 2 paragraph 2, the score in Section 5 paragraph 3, etc.
- L0Drop
    - Suppose one of the gates g_i = 0, which then means \mathbf{z}_i = \mathbf{0}. In Figure 2 top, is the Transformer decoder still able to attend to this \mathbf{z}_i? And in Equations (2) and (3), is this \mathbf{z}_i a component of the computations (in particular, will it still have a non-zero contribution to the softmax both in the Transformer in Equation (2) and in the \alpha_i term in Equation (3))?
        - It could be argued that if so, the encoder is not outputting a bag of vectors whose size is smaller than the number of tokens, because a 0-vector is different from there not being a vector at all.
        - Ideally, this \mathbf{z}_i would be completely masked out during the decoding stage rather than being set to 0.
- Experiments
    - It would add a lot of value to the paper to see examples of the model outputs on sentiment transfer and sentence summarization.
    - A discussion on the computational efficiency compared to the baseline would also be very useful.
summary_of_the_review: Overall this is a good paper which, with some adjustments to the writing, would be of good value to the community.
review_title: null

Meta-review: The paper addresses unsupervised conditional text generation extending emb2emb (Mai et al, 2020) with bag-of-vectors antoencoders.

Reviewers shared several concerns about the clarity of this paper and empirical results.
Decision: Reject