Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new approach for dequantization ie embedding discrete data in a continuous space using variational inference and truncated flows called TRUFL. Unlike previous approaches, TRUFL aloows the dequantization layer to have a learnable truncated support. The authors perform several experiments to demonstrate the advantages of the proposed method to Categorical Normalizing Flows (CatNF) and Argmax Flows.
main_review: **Pros:** The paper is very well written and the method is explained clearly, concisely and in appropriate detail. The use of examples, figures and algorithm boxes at appropriate places makes the paper easy to follow and understand. Furthermore, the problem considered in the paper is a valid and interesting problem for modelling discrete data using NFs. Multiple papers have come out in the past year or so addressing this problem, each with their own sets of solutions (and restrictions) and the present paper proposes a new method that alleviates some of the drawbacks of CatNFs and Argmax flows. The experimental analysis by the authors consider a diverse set of problems from graph coloring, molecular generation, and language modelling with reasonable performances. 

**Cons:** I found the empirical results to be mixed for TRUFL as compared to Argmax flows and CatNF. While, this in itself not a huge issue, I am interested to understand what might explain this considering that the method basically tries to address the drawbacks of both these approaches. I also wonder if it might be possible to use synthetic examples to bring out these advantages for TRUFL.
summary_of_the_review: The paper is well written and explored. The empirical analysis is reasonable. Overall, I found the paper quite interesting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a Flow model called TRUFL designed for discrete data. The main selling point of the proposed model is that it can handle discrete data better than other dequantization schemes. The authors target two key difficulties of this dequantization problem, which are (i) making the dequantization lossless, and (ii) allowing the dequantizer to be learned easily. Building on the Categorical Normalizing Flows, the authors propose to truncate the dequantized latent space to make the latent space of different input categories less correlated. To compute the probabilities w.r.t. the truncated latent space, the authors use rejection sampling to approximate these probabilities. 
main_review: Learning how to handle discrete data in continuous deep neural networks is an important task, and contributions in this direction are very useful. However, I am a bit concerned with the authors' claim that TRUFL better achieves the two goals mentioned in their introduction: (i) easily learnable, and (ii) lossless. First, it is unclear whether TRUFL is more "lossless" than CatNF in practice. While TRUFL uses truncation to zero out the probability mass beyond a bounded range, there could still be major overlap between the latent space of different categories. And compared to that, truncating the low-probability regions might be less effective in making the dequantization lossless. To make the dequantizations more lossless, another simple choice could be to make distributions on z farther away from each other (Figure 1 CatNF, making the three gaussian farther away from each other).

In fact, I can imagine in certain cases we don't need lossless dequantization, as some variation of the data might be irrelevant to a task.

While the proposed TRUFL model may still perform lossy dequantization, according to the claim in the introduction, the benefit of TRUFL could be it's easily learnable. However, due to the similarity with CatNF and the additional need for rejection sampling, it is not very clear to me why TRUFL can be learned easily. To summarize, it is unclear to me why truncating the latent space could simplify the learning problem itself.

Since the proposed approach uses rejection sampling, it would be nice to also compare the inference time of different models, just to see how much the sampling procedure influences the model's overall efficiency.
summary_of_the_review: I tend to vote for rejection because it is unclear to me why truncating the latent distribution could necessarily improve the dequantization quality. Although the authors provide two explanations in the introduction, they are not very well justified. Apart from the truncation technique, the proposed model doesn't seem to differ too much from existing approaches.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a dequantization scheme for categorical data, where unlike with ordinal data, element-wise uniform noise cannot be used. The authors propose to encode categorical data into an interval centre and a deviation, and a (pre)dequantized value is obtained by sampling uniformly in the implied real interval. A normalizing flow is then applied to obtain the dequantized variable. The proposed method can thus in principle learn non-overlapping one-dimensional supports for the dequantized variables, enabling lossless dequantization.
main_review: The paper is mostly well written and easy to follow, although I have some reservations about the motivation:

1. I don't think the authors properly motivate the use of the normalizing flow to begin with. Is it really better to have the flow as compared to increasing capacity for m and s? I think this is a very natural question, which is neither posed nor answered in the paper; and that needs answering to better motivate the paper.

2. Part of the motivation is enabling the dequantized variable's dimension to not depend (logarithmically) on the number of categories. It is actually not clear to me that this dependence is particularly harmful. For example, one might expect that categories which are semantically close to each other have corresponding nearby dequantized intervals. However, it might not be possible to achieve this: consider a case with 3 categories, A, B, and C which are though of "semantically equidistant". There is no way to have 3 real intervals satisfying this requirement, and I suspect this might make learning the functions $m$ and $s$ harder. In contrast, if in the same example a 2-dimensional dequantized space was used, one could actually obtain the desired symmetry. I think this comment bears some philosophical resemblance to kernel methods, where the intuition is that simple classifiers might do better on high-dimensional representations. In other words, the fact that one can obtain a one-dimensional dequantization does not automatically imply that one should, and I believe the authors should make a stronger point for this choice.

3. Finally, while dequantization is a sensible approach, I think that it should be at least empirically verified that dequantizing and using a continuous model actually outperforms using a flexible discrete model to begin with. In fairness to the authors though, I think this is more of an issue I have with dequantization itself rather than the paper being reviewed.

As for novelty, the method is novel as far as I am aware.

Finally, I did not find the experiments particularly convincing: while the proposed method does seem to be slightly better than the baselines at the evaluated tasks, the improvements seems small and do not in my view compensate for the issues I have with the motivation. Additionally, only averaged results over runs are reported, and given how close the numbers are along with the lack of error bars, it is hard to assess whether the improvements are actually significant.

Minor remarks:

-In algorithm 1, $\hat{z}_t$ is used in the denominator for the change-of-volume term, but $\hat{z}_t$ is not defined.

-The notation in equation 7 should be changed to either $\tilde{f}(z;m,s)$ or $\tilde{f}(z|m,s)$ to more clearly differentiate between the different types of inputs.

-Table 2 should be referenced in the text in section 5.2.

==========================================================================================================

UPDATE AFTER REBUTTAL

==========================================================================================================

I have increased my score after seeing the authors' updates regarding motivation and the promised ablations regarding point 3 above.
summary_of_the_review: While this paper proposes a simple idea for dequantization, I am not convinced the proposed method is needed in the first place, and would need to see much stronger empirical evidence to be convinced otherwise.
review_title: null

Meta-review: The paper proposes a variational dequantization method for categorical data, based on flows with learned truncated support. The problem has been studied before, but the paper makes it clear how the proposed method differs from existing ones. The method is empirically evaluated on a large variety of diverse tasks.

The reviews were initially borderline. In general, the reviewers did not identify major quality of technical issues with the paper, and appreciated the clarity of writing. On the other hand, the reviewers were not fully convinced by the motivation or the empirical performance of the proposed method. After discussion with the authors, some concerns were allayed (especially regarding motivation) and all three reviewers decided to recommend weak acceptance.

Seeing as there are no major technical or quality issues with the paper, and the paper is clearly written and well executed, I'm leaning towards recommending acceptance, although some doubts remain about the significance of the contribution.
Decision: Accept (Poster)