Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces an entropic coding method that is able to compress a sentence into a latent space and perform text generation through a uniform sampling from the implicit latent space. Experiments are conducted on a toy dataset and a narrow domain text dataset to show the effectiveness of the proposed method w.r.t. AE, VAE, and Transformer models.

main_review: 1. The writing of this paper has much room for improvement. For example, the description of the method part in Sec 3.1 is super unclear. It's hard to fully understand the method without referring to the pseudo-code. Some of the key operations (e.g., "find function") are moved to the appendix. And the notations in the pseudo-code are also confusing, e.g., a word token is sometimes used with or without a subscript -- Inconsistent notations of "(s_j)_{j<i}" and "s > s'" seem to denote previous words before the i-th word.
2. Conceptually it's still not clear why splitting the latent space using the LM (trained on biased text data) could be beneficial to the generalization for unbiased text. Even though the coding method could efficiently utilize its latent space to fill sentences in all its d dimensions, wouldn't it also cover the bias from the LM trained on biased text data?
3. I don't see most of the evaluation metrics used in this paper in the literature. I am a bit skeptical whether these metrics are convincing enough to evaluate these encoding methods on more realistic, complex datasets in addition to the toy & narrow-domain datasets used in this paper.
4. The other models in comparison seem to be artificially tuned in some ways which lead to pretty low performances. For example, using only 2-layers of encoder & decoder or a small dimension for Transformer, sampling the latent space of Transformer. It's not clear how the proposed method compared with stronger models. 
summary_of_the_review: I would suggest a rejection of this paper given the current draft.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposed a method to encode/decode a sentence (or say a sequence of tokens) into/from a volume instead of just a point(vector) in d-dimensional. The paper shows some good attributes of this method via some experiments. 
main_review: 1. I feel the paper is poorly-organized. I repeat like 5-6 times to get what it's trying to do in introduction. This could immediately be a reason to reject the paper as I think it's not gonna benefit audience in ICLR.

2. After digesting it, I think I like to problem this paper discussed, and it's very pertinent to the representation learning. However, I am not sure if I got 100% understanding of the content as it's again poorly presented. Leaving all the materials in an algorithm table without much explanations make it hard to understand. I think a running example should be provided to make reader easier to understand what's going on. More importantly, authors seem to assume readers know what AC is but honestly I don't see the necessity. 

3. From the start to end I am not sure why Volume Code is good. This isn't well motivated. And I honestly don't know how the metric is picked. I don't see these properties are necessarily good. If my VAE or other models are learning a representation toward a specific task then why it's necessary to follow these designs?

4. I feel all the comparison models can be used as language models. So my understanding is that Ariel can be used to generate code provided an underlying LM. Why not train VAE or other stuff in LM task and then apply Ariel to see the results? 

5. Following 4, for this type of paper, an observation is provided and usually it should also propose a way to augment the existing methods to match the proposed metric. So it reads to me that authors should propose a way to show that representation learned by VAE can be improved by adding some components. And in my point 4, I do feel treating it as a LM can be tested. 


summary_of_the_review: Overall, I think it's not a presented in a straightforward way and it's lack of many details for readers to fully digest it. I feel the topic is interesting but in practice I don't know why it's important and author didn't motivate this. I will be considering to raise the score if the introduction is re-written in a motivating form.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to use an autoregressive language model's hidden state to index multidimensional arithmetic coding representations of vocabularies and sentences. This way the latent space's geometry is directly connected to the words chosen at the cost of sensible inductive bias between similar sentences. Evaluations of unconditional samples on toy data and small-scale real data are made. While interpolations are presented, the proposed model and the autoencoders it compares to are only ever evaluated for fluency and grammaticality, but not for semantic relatedness.
main_review: My first concern is that the proposed method essentially is arithmetic coding in multiple dimensions---and the mental abstraction I have in mind is that it is closer to the ZIP file format than an autoencoder, focused very much on perfectly reproducing training data and essentially overfitting it.

My next biggest concern is that I don't think I quite understand the problem this paper is trying to solve in terms of downstream use. For example, an autoencoder's encoder is useful because it can be a way to unsupervisedly obtain continuous representations for discrete string data---those representations could've been tested for their informativeness, e.g., by trying to predict sentence properties or cluster from them. Likewise, an AE's decoder is useful for use as a pretrained generator from latent codes: given a generation task with little data, it may be easier to learn a model that learns to generate latent codes (which are then spelled out by the AE decoder) than one that tries to generate text end-to-end. These kinds of ideas and considerations are missing for me in this document that seems very focused on showing the overfitting and recoverability capabilities of the proposed AriEL. (None of these experiments should be hard to conduct as AriEL through returning centers can be made to "look like" an ordinary autoencoder, I just have a feeling that given its focus on overfitting the training data, the interpolations, while maintaining grammaticality, will not be semantically useful, leading to low performance in these downstream tasks---but I would love to be proven wrong!)
In the same vein, I do not understand why the particular real-world dataset was chosen as opposed to any other collection of strings or datasets that are used by and compared to on previous work.

If that wasn't clear enough, here's a different way to phrase the issue: the evaluations in this paper as far as I can see are only concerned with *unconditional* sample quality, i.e., what do samples from randomly drawn points look like. That is interesting to look at to get a sense of the space, yes, but is ultimately of no relevance to any practical application---because if unconditional samples were truly what we wanted, we would almost certainly be better off just sampling from the base LM that AriEL uses to begin with! Or does this base LM perform worse? If you do want to claim AriELs use in unconditional sampling then that really is the dead simple baseline to outperform in your metrics like validity.

Next, I must admit I do not understand the "biasing" fully. Does it categorically rule out some combinations while leaving others untouched (as Section 3.3 paragraph 3 seems to imply, essentially leading to all strings having probability 0 or p where p is the same for all sentences that have nonzero probability) or is it a weighting device that essentially turns the CFG into a PCFG in which not all strings are equally likely and no string that is possible under the CFG ends up with probability 0 (which is what Figure 1 seems to show)?

Finally, I have doubts about the inductive bias of using dimensions in this round-robin per-word fashion and would like to see whether effects related to that periodicity and subword tokenization appear. One possible experiment to run here to gauge the problem might be to look at where errors occur in data or whether any other patterns arise that pertain to position (early/late/periodical for low dimensionalities)

Other issues or questions:
- the abstract says "We compare to" which sounds like AriEL is an existing technique and not a new contribution of this paper
- Section 1 line 6 says "such type"---what type?
- Section 1 paragraph 3 says "objective benchmark"---what is objective about it and compared to what "subjective" other benchmarks?
- Section 2 line 4 calls volume codes distributed representations but defines the latter as points in R^d rather than values, leading to a contradiction
- Section 2 paragraph 2's citations strike me as old; I am not intimately familiar with advances on these methods since 2017, but would be surprised if there weren't any
- Are the letters in A|B|AA|... in Figure 1 nonterminals or terminals? I suppose they are terminals for the explanation/picture to make sense, but then convention would have them be lower-case instead of upper-case :)
- The last paragraph of section 3.1 mentions RNN-based langauge models, but AriEL should work with any *softmax-based* autoregressive LM as I understand it, including Transformer-based models like the GPT family
- I also admittedly do not understand why there is no linear dependence on vocabulary size in AriEL---lines 9 and 10 in both sides of Algorithm 1 iterate over the vocabulary in a linear way, do they not? (Some of that could be asymptotically recovered with clever sampling, but I think this setting does not lend itself to such asymptotic optimizations, though that might still be good to mention.)
- Unless I missed it, Algorithm 1 is also never referenced in the main text.
- Line 1 on page 4 spells "Transformer" as "Tranformer"...
- ...and it is unclear to me how it is a fixed-length representation unless you cheat by establishing a maximum length and truncating/padding (which you later say you do---this might be good to state upfront to avoid confusion)
- instead of $d_{ff}$ try $d_{\mathit{ff}}$ or something similar to get proper spacing between letters in subscripts
- Testing only 2 or 20 layers is a *huge* difference and even a priori I would expect that both the most commonly chosen and the best-working number of layers is somewhere in the middle
- Section 3.3 paragraph 4 describes "unbiased" sentences as those that go *against* the bias defined (whatever that actually is, see the major confusion above)---shouldn't we then call them something like "opposingly-biased"?
- Generation/Deocding Quality is missing a metric that counts how many of the 10k are grammatical. Yes, if they are all the same sentence and that sentence happens to be correct that measure would give a 100% that looks more impressive than it is, but with that caveat I would rather *know* this number than now know it.
- Relatedly, it took me embarassingly long to understand why validity is the "most important of the metrics" as section 4.1 puts it---may benefit from spelling out that this is indeed the principal metric right where it is defined :)
- I unfortunately also have some quesy feelings about the uniqueness in metrics like validity. For example, if I draw 5 samples, chances are they will all be unique so I can get a high ratio score. Great! But if I draw 5 million samples, it is very unlikely that I will get out 5 million unique sentences, so I will get a low score here. Of course, as will other models, so comparisons are fair, but the fact that as the number of samples tends to infinity the scores tend to 0 strikes me as a sign that the metric definitions aren't fully thought through. If a specific suggestion helps: define validity not as a ratio but as a count of unique sentences and let the number of samples tend to infinity.
- Prediction Quality metrics are ordered in that PAB < BA < GA... is that correct? Might be useful to specify that way :)
- Section 3.4.2 talks about "interpretability," a very loaded technical term---do you mean something like fluency or legibility? Similarly to the choice of "biased" poorly chosen terminology may hurt this paper more than the actual contribution deserves.
- Section 3.4.3: 15 pairs feels like a low number, do estimates truly converge with this few samples? Put differently, is Figure 2 as smooth as it looks? Putting in the actual points may help show just how cleanly sampled that space is.
- Speaking of Figure 2, first, what is the y axis? Validity as defined in the ratio way (see above for my concerns on that) or something new? And what is this lower bound 0.746 that is "related to the language complexity"? Can you spell out that currently somewhat nebulous relationship?
- Section 4.2 paragraph 2: can you quantify "almost all"?
summary_of_the_review: The paper suffers from a lack of problem that is demonstrably made headway on, i.e., where a proposed method improves over sensible baselines on sensible metrics, largely because data, baseline, and metrics are chosen and justified poorly in my opinion.

That is why I currently cannot see what anyone could get out of this paper as it is (other than remembering that arithmetic coding and k-d trees exist and possibly being tempted to think about volumes in latent spaces and the relation between overfitting and representational capacity a bit more, no thanks to the paper which does not ever talk about these issues itself).

Finally, I should disclose: I have reviewed this paper before (but do not know its authors or anything beyond that) and some more fundamental concerns are unchanged from my last review, though the paper has improved noticeably in my mind (thank you for these improvements, it really has gotten better). My review is thus focused on what I believe is possible to still improve about this paper rather than trying to reiterate previous concerns---that is why I have little to say about the evaluations performed themselves.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes a novel methodology of volume coding for encoding and decoding sentences. The algorithm is based on arithmetic coding. The qualitative result reveals that the generations from AriEL to be more valid comparing to other models.





main_review: Strengths:
- A novel approach for latent modelling
- Strong results in terms of validity

- Weakness
In Table 1, the validity percentage of a Transformer with 512 latent dimensionality is only 17.2%. This low score strongly contrasts with our knowledge that a well-trained Transformer language model is very strong at producing valid sentences. One hypothesis is that the amount of training data is not sufficient for Transformer. If this is the case, the proposed method may lose its edge when the training data is abundantly available.

In additional to this point, if the main reason for the Transformer to fail in validity is because the vanilla Transformer was not trained on sampled latent space. We shall at least inject random noise during training to make the Transformer robust to sampling.
summary_of_the_review: I have reviewed this paper last year, I can see the authors have reorganized the content significantly. I will give a weak acceptance this time and willing to discuss among reviewers.  ​
review_title: null

Meta-review: The paper proposes an entropic coding approach for sentence embedding. 

Reviewers have spent good efforts in reviewing. They generally feel the problem is important/interesting, but also found it difficult to understand the paper. Thus, the authors are encouraged to thoroughly revise the paper according to the reviews provided, and another round of review is needed to better determine the merits of this paper.
Decision: Reject