Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work focuses on improving the long-range entity coherence and consistency when generating long narrative stories. The authors proposed two metrics to measure the entity coherence and consistency in terms of entity usage in generated narratives. They also propose to augment a pre-trained LM with a dynamic entity memory and cross-attention to improve the entity usage.
main_review: Strengths: 
(1) The work is to address a challenging and practical problem in generating long narratives. 
(2) The authors proposed two intuitive metrics to measure the entity coherence and consistency based on their mentions and context, which are beneficial to the following research. 
(3) The proposed entity memory augmented model is reasonable and improved entity coherence/consistency in some of the experiments. The authors also provided a detailed analysis of the models in various settings.

Weakness:
(1) The two proposed metrics about entity coherence and consistency are intuitive but not rigorous. The entity coherence is based on the occurrence of the entity mentions across different sections. However, the coreference is accurate especially when the context is wide, and even an entity is mentioned across a wide context, there are a lot of other factors to consider to quantify the coherence, e.g., the intermediate mentions used in-between the first and the last mention. Similarly, for consistency, the authors considered all verbs and adjectives as the attributes of the entity. What's the reason? and how can it measure the inconsistency in the example of Figure1 (b)?
(2) The model design is reasonable, but it depends on several key hyperparameters, e.g., the size of the entity memory or the context memory, the length of each narrative section. They may have much impact on the final performance and on different datasets, they should have different choices. Also, it seems the entity set is given to initialize the memory, then what if the model generates new entities?
(3) The results didn't consistently demonstrate the improvement of the proposed approach, e.g., in Table 2, on the WikiPlots dataset, the VanillaLM is better than other models for most of the metrics; and on the WritingPrompts dataset, the VanillaLM seems to generate more mentions in average.
(4) The analysis section is very difficult to follow. Some analysis is just based on wordy descriptions without details or examples. For example, in the automatic evaluation section, it's not clear how the uncertainty is computed, how to interpret the Figures, especially Figure 4, what does the control in section 5.2 mean? and what are the detailed criteria for human rating (1-4)?
summary_of_the_review: Overall, this work proposed some interesting ideas to analyze and improve the entity usage in long narrative generation task. However, the metrics need to be carefully designed, and the analysis needs to be improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Considering little analysis that has been done on analyzing the ability of pre-trained language models (LMs) to maintain entity coherence and consistency, the authors take narrative generation as a case study and analyse the long-range entity coherence and consistency in generated stories. They firstly quantify the limitations of current LMs by proposing a set of automatic metrics for measuring model performance in terms of entity usage, and propose an augmented pre-trained LM with a dynamic entity memory in an end-to-end manner by using an auxiliary entity-related loss for guiding the reads and writes to the memory, which increases both entity  coherence and consistency in the generated stories.
main_review: Strengths:
1. The authors address important problems of LMs on entity coherence and consistency when generating long stories.

2. The work proposes new automatic metrics for analyzing narrative generation where some show high correlation with human evaluation.

3. The work proposes an improved pre-trained LMs for narrative generation by adding an entity memory which shows competitive results compared with the Vanilla LM.

4. Extensive results discussion, evaluation, and ablation analysis are presented to clearly show the performance of some of the proposed techniques.

Weaknesses:
1. The codes are not made publicly available for reproducibility.

2. The proposed entity memory-based pre-trained LM is only compared with Vanilla LM. It would be more interesting to show how the proposed model performs when compared with other state-of-the-art LMs such as BERT and the main prior related works of Clark et al. (2018) and Ji et al. (2017) that have been discussed in the introduction section.

3. There is a lack of experimental support of why the proposed entity consistency metric is  needed since it gives lower correlation with human ratings in Table 4. It would be better to add more discussion on why it can still be considered to be adapted.


4. Minor weakness:
Figure 1(a) is not easy to understand. The <sep> tokens used do not clearly show where the entity starts and ends. It is suggested that you prefer using <sep> and </sep> pair, where the former shows the beginning of the entity and the latter shows the end.
summary_of_the_review: Overall, the paper is well written and organized. The studied problem is very important which makes the paper to have many audiences at ICLR and the proposed techniques can benefit the field however after performing few more experiments to support them (see Weaknesses section Point 2 and 3).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the main contributions are 1)analyze the entity coherence of  pretrained LMs by leveraging some existing metrics and also four newly proposed metrics 2) propose a new generative model to have a higher quality generated narratives from entity coherence and consistency perspectives. Authors develop a model by adding a dynamic entity memory to the existing pretrained LMs. The cross attention between input text and entity memory alongside the self-attention coming from pretrained-LM helps to not forget the previous entities and their attributes during generating narratives.
main_review: Pros:
The motivation of the work which is relevant to a well-known issue in existing long text generative models has been written clearly and figures are very helpful in this regard. Author try to contribute in two main problems first proposing automatic metrics to measure and analyze the coherence of generated texts by existing LMs and second proposing a model to improve the coherence of generated texts by augmenting an entity memory. Different experiments and ablation studies show the effects of each component in the model from different perspectives.

Cons:
There are some not clear sections that need to be explained more clearly. Here are a set of questions and comments that addressing them can be beneficial in better understanding the paper.

Main question is regarding different forms of the same entity and how they are handled in the model. Assume we have one entity but in two different forms such as mom and mother throughout the plots, according to what has been explained in the paper model considers them as two separate entities. If this is true, what is the impact of language different forms of words on the model's performance?

For automatic evaluation of coherence of texts the C metric only considers the first and last happening location of the entities in the text, while this is very abstract level and can not be a good and fair representative of the coherence in the text level. There are some recent works that try to assess the coherence of the generated narratives by using the plots and manipulating them (Ghazarian et al. 2021). In order to show the positive impact of proposed evaluation metric it is recommended to compare with more recent works.


In the method section, since the main idea of the work is about entity memory, presenting some instances for plots/entities/attributes  can help author to better follow the steps and the proposed model.

According to table 1, if the model can have access to the whole context then there is not that much difference between the proposed model and the vanilla LMs? Therefore if we use LMs that can process long texts such as longformer then we won't have benefits coming from the entity coherence proposed by this model?

For human evaluation, how do you ensure the quality of the annotations are good? what are the agreements between human judgments? how many annotations have been collected for each narrative?





summary_of_the_review: In summary the idea of using entity memory and computing cross attention between input and this memory to incorporate information from entities and their attributes looks promising. However, the metrics showing the coherence of the text is only limited to the location of the entities while for the text the overall coherence of the text should also be considered. Some sections are opaque but I think if authors can address those the paper can be effective for the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors of this paper propose to tackle the problem of coherency and the consistent use of entities for the task of narrative generation. To achieve this, the authors propose an approach of augmenting pretrained language models with entity memory and cross attention blocks that places an emphasis on the entities in the narrative. The proposed approach is evaluated on the WritingPrompts and the WikiPlots datasets. From an evaluation perspective, the authors propose two new evaluation metrics for measuring entity coherence and consistency. To measure entity coherence, the authors propose dividing the narrative into L equal sections and computing the maximum interval span of each entity as the difference between first and last mention respectively. Similarly for entity consistency, the authors propose measuring entity consistency through the concept of measuring attribute consistency within the narrative.

Contributions:
1. A new approach of augmenting language models with entity memory that helps in entity consistency and coherence across narrative generation tasks
2. Two new metrics to evaluate consistency and coherency in the generated outputs.

main_review: Strengths and Weakness:
1. The proposed approach for tackling entity consistency and coherency through the use of memory structures that augment language modeling is novel and interesting. Further, the dynamic nature of the proposed entity memory network allows multiple attributes to be associated with an entity.
2. One of the contributions of this work is the proposed evaluation metric for entity coherence and consistency. The metric for entity coherence based on calculating C_i as the difference in the indices of first and last mentions does not truly capture coherence. Further, the example provided in the paper Fig 1b seems to be an issue with local coherence instead of long-range entity coherence. 
3. With regards to entity consistency, the authors propose quantifying it through the use of attributes associated with each entity is interesting. However, this approach does not take into consideration multiple entities in a sentence and attribution association in those cases.
4. Evaluation is done on the standard datasets of writing prompts and wikiplots. However, it would have been better if the authors had used more baselines to compare their proposed approaches instead of a simple LM for both datasets.  From the automated metric perspective, the proposed addition of entity network seems to be performing worse than a vanilla LM on the wikiplots datasets on the metrics as shown in Table 1. This trend of vanilla LM outperforming the proposed approach is evident in the proposed automated metrics in Table 2.
5. The proposed automated metrics when compared to the human evaluation show a low to moderate correlation to human judgments. The correlation for consistency is pretty low which raises questions about the proposed metric and its effectiveness to capture entity consistency.

Questions:
1. With regards to entity consistency, how were the attributes assigned if there are multiple entities mentioned in the sentence with attributes?
2.  Why does the attribute consistency on the wikiplots dataset using a simple vanilla LM outperform the proposed entity memory network augmented LM?
3. why is the scale for coherence different from the other metrics?
4. How many human annotators evaluated the generated outputs? What is the IAA?
summary_of_the_review: This paper tackles the problem of entity coherence and consistency in narrative generation and tackles the problem through the use of an entity memory network that augments the pretrained language model. This proposed approach is interesting and demonstrates the effectiveness of the proposed approach on two datasets. The clear gains of the proposed approach can only be seen on one of the datasets compared to a simple baseline. Further, the authors also propose using two new metrics for automated evaluation. These metrics show a low to moderate correlation to human judgments.
review_title: null

Meta-review: This work analyzes the ability of pre-trained language models to maintain entity coherence and consistency in long narrative generation. Along with new automatic metrics for analyzing narrative generation, it proposes a memory-augmented model that allows tracking entities to improve narrative generation.  Although all the reviewers appreciated the importance of the problem, the novelty of the proposed approach, as well as empirical improvements in a subset of experiments, they also acknowledge several major weaknesses including the lack of rigor in defining the method, the lack of clarity in writing (especially in the experiments section), insufficiently strong baselines, and an issue of reproducibility since the code cannot be released. These concerns were in part addressed during rebuttal, but not enough to accept the paper.
Decision: Reject