Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The aim of this work is to derive deep generative models that are able to incorporate relational constraints. The proposed model learns relational constraints from training data by solving constrained optimization problems and further generates examples given relational constraints. Empirical evaluations on music and poetry generation tasks are presented.
main_review: [Strengths]

The paper is overall well-written and the method is clearly explained. The literature review is thorough. The idea of formulating the constraint synthesizing as a constrained optimization problem is novel to me.

[Weakness]
- It is unclear to me how the prototype subcomponents are chosen. One concern is that when the prototype subcomponents are similar to each other, the resulting model might generate similar examples and does not generalize well. I guess the ideal scenario is to choose the prototype subcomponents to be as diverse as possible. But it is unclear how this is guaranteed from what is presented.

- The other concern I have is on the efficiency of the synthesizing relational constraint process. The number of variables involved in the constrained optimization problem in Section 3.2 is quadratic in m, i.e., the number of subcomponents in example x. Further, the number of calls to solve this optimization problem is the size of training data. I wonder how efficient is this process.

- Since this work follows the framework from [1], I wonder whether the generation processes of samples given relational constraints as described in Section 4.2 are original contributions and how are they compared with the generation process in [1].


[1] Halley Young, Osbert Bastani, and Mayur Naik. Learning neurosymbolic generative models via program synthesis. 2019.

summary_of_the_review: The generative model for sequential data with relational constraints proposed in this work is interesting to me. Still, I have several concerns as mentioned in the main review.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a generative model for sequential data guided by relational constraint. It uses a specific type of high level structure represented by prototype alignment and a VAE framework to formulate the training. The paper experiments the proposed model on music generation and poem generation tasks, showing some substantial improvements given the proposed metrics.
main_review: I think the constrained/guided generation problem is an important one. And the method proposed in this work is interesting. Although it's relatively weak on the neural-symbolic part (since it only touches on a very special type of constraints, and relations are deterministically optimized and this process does not participate in gradient-based learning, which is less exciting to me), but the construction of the model is reasonable and results are promising. I think the amount contribution is moderate. And besides the evaluation/analysis part is unconvincing, I don't see any other reason to reject.

1. The abstract mentioned that many recent works perform well on local coherence at the cost of global coherence. But I don't see this problem in the baseline experiment results though. Basically, while the method is interesting, I don't see what problem it is actually solving.

2. The process of training is confusing to me. It seems that Fig1 bottom shows the inference step of VAE starting with the sampling of latent variables. But I am not sure what is the input at training time. Is it c_x -> z -> c given z -> x given c ??? The paper uses (z given c) and (z given c_x) at different places. I find it confusing.

3. For the poem generation task, only A1 was used. This is less exciting since the constrained sampling could end up pretty slow and bad. The paper mentioned in the appendix B that typical causal language model doesn't fit for generating words with proper rhythm at the end of each sentence. While this makes sense, but there are apparently some heuristics that are easy to come up with to walk around that. Furthermore, for A3, while sequential generation via combinatorial optimization can be intractable, there are span-based generation approaches that can mimic that. I feel there are many options not explored in this task. A refuse of using A2 and A3 sounds like a drawback.

4. I am not sure what story the loss based evaluations can tell. On one hand, the GCN discriminator metric is subject to artifacts in training data and thus potentially can be broken. On the other hand, the FD metric, as also said in this paper, is embedding based, thus does not evaluate rhyme and meter. Then why using them at all in Table 2?
summary_of_the_review: I think the major contribution of this paper is the framework. While some evaluation metrics are questionable, but I can see the coverage is better than the last year's submission, and there are improvements over different metrics. Besides, evaluating generated sequence is a challenging task and is out of the scope of this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a hierarchical approach for generating sequences with global structures (e.g., music and poems with repeating elements). The proposed method first draws a template with a few slots already filled and certain global relational constraints across the slots, and then fills the remaining slots with actual content. The global constraints are found out by solving a combinatorial optimization problem: for each training sequence, they select some parts of the sequence as prototypes and relate each of the other parts to one of the prototypes by minimizing the overall dissimilarity among associated parts. Music and poems generated this way turn out to have a higher global coherence, lyricism, and rhyme, compared to what are directly drawn from an autoregressive model. 

It is a novel algorithm, and achieves compelling empirical results in the domains of music and poetry generation. 

main_review: The strengths of the paper is its novelty and empirical results; its weakness is its clarity and related work discussion. 

The overall framework is well-motivated for certain generation tasks such as music and poetry generation, and learning relational constraints from data is an interesting idea. 
It does perform as expected in real-world datasets. 

However, the paper sweeps under the rug some key foundations of the proposed approach. 
The most important is how to construct the function $f()$. According to appendices, it seems that $f(w,w',r)$ is assigned to 1 or 0 by checking whether $w$ and $w'$ follow a pre-defined relation $r$ and each possible relation $r$ can be automatically verified since they are superficial (e.g., same rhyme or same pitch class). This seems like the crucial premise that the proposed method could work, and it should be clearly stated in the main paper. Moreover, it is worth to discuss the limitations of the proposed method: e.g., if a relation $r$ is more conceptual (e.g., $w$ and $w'$ are talking about the same topic), then would the method still work or what else is needed to make it work? 

It also suffers some inconsistency between the method overview and detailed exposition. The subcomponents of sequence $x$ are defined on the utterance level (e.g., lines of poems) but the content generation is elaborated on the word level, with the same index symbol (i.e., $j$) as the lines, which really confused me. I believe this inconsistency is merely a presentation issue but the authors should make it clearer for the readers to follow the details. 

There is other recent work using relational constraints to configure neural sequential models. E.g., Li et al. uses temporal logic to constrain neural point processes: https://proceedings.mlr.press/v119/li20p.html; Mei et al. uses Datalog programs to compile human knowledge into the architecture of a neural sequence model: https://arxiv.org/abs/2006.16723. Relations with these papers should be discussed: e.g., they all condition sequence modeling on a set of global constraints; constraints of these papers are human-specified while those of this submission are drawn from a learned distribution (though its learning depends on human-specified relational features). 


summary_of_the_review: Based on the strengths and weaknesses explained above, I believe that this paper is at the borderline; I slightly tend to reject it since the clarity issues that I raise are important. However, if the authors could well address those concerns and we have high confidence in its camera-ready version, then I would consider elevating my rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a neurosymbolic generative model for sequence data. The model extends the one proposed by Young et al. (2019) in which a neurosymbolic generative model is initially proposed for images. Experiments are conducted on two datasets, one for music generation and the other for poem generation. Evaluation results indicate that the proposed model can effectively represent high-level structures in sequence generation.

Contributions:
1. a sequence generation model in which structure information is taken into account in a principled way.
2. effectiveness of the approach on music generation and poem generation.
main_review: The authors follow the existing work (Young et al. (2019)) and extend it to handle sequence data. My understanding is that the model can take structure information into account while generating sequences, or more precisely, explicitly incorporate repetitive patterns in sequences into generation. I judge the work from the following perspectives:

1. Technical Novelty: the basic idea comes from the work of Young et al. with some adaptations to sequence data. I would rather view the work as an extension of the approach in Young et al. (2019) to sequence data. There are some new things for the adaptations (e.g., learning the constraints from data), but overall, the key building blocks (e.g., VAE, controllable text generation) have been proposed before and widely used in a lot of work. 

2. Empirical Novelty: as far as I know, there has been a lot of work on music generation and poem generation. Particularly, my experience is that it is not that difficult to make a generated poem follow rhyme and meter constraints as long as we have a relatively good language model, since such characteristics are very strong in poetry. Therefore, I would like to see more applications, or probably more important applications of the proposed approach, for example, can we apply it to dialogue generation or summarization? To do that, you may have to go beyond the repetitive patterns, but such applications, in my opinion, could generate more impact. 

3. Soundness: the authors pay a lot of effort to formalizing the constraints and describing the learning details of the constraints. However, important details of how to build the generative model are missing in the main part. Even in the appendix, some details are still missing, for example, how to build the GCN to learn \mu(c) and \Sigma(c) in B.2?  In Experiments, I see some metrics for structure consistency, but very few metrics for quality of the generated content in addition to NLL. How can we guarantee the quality of the generated content? If it is hard to find other metrics, at least human studies can also be done in music generation. In 5.2, the authors claim that they have to choose BERT because they need to do backward sampling. I do not think this is a strong reason, as BERT is basically not a generative model. I suggest the authors to think about how to adapt their approach to generative language models, or choose other PLMs as the base. 

4. Writing: there is a little mess in the organization. For example, Introduction and Related work are combined, and the explanation of the example in Figure 1 above the Related work just breaks the coherence of the Section.  Some descriptions about the related work are placed in appendix, and more seriously, some important details, such as the details of Section 4.2 are also placed in appendix. I think most of these content should be involved in the main part, even though there is page limit. 
summary_of_the_review: 1. Technical innovation is not enough to meet the bar of the conference. 
2. Empirical studies do not fully convince me the effectiveness of the approach. 
3. There are some issues on organization of the content. 
4. I cannot think about broader applications of the proposed approach in sequence generation.
review_title: null

Meta-review: This work proposes an interesting approach for learning the relational constraints of a dataset and then generating according to those constraints. Learning the constraints via a constrained optimization problem is an interesting contribution. The application of constrained generation is also interesting and can be applied to several domains though only music and poetry is examined in this work. However, the music evaluation is unconvincing and the paper lacks clarity in the description of the approach such as building the GCN. Music evaluation could be improved with human evaluation since loss metrics don't paint a full picture. Finally, a more diverse set of experiments and datasets (rather than just one poetry collection and one folk song corpus) and more analysis on the learnt constraints could give a more complete story for this approach's effectiveness on sequence data.
Decision: Reject