Show, Write, and Retrieve: Entity-aware Article Generation and Retrieval
Authors: Zhongping Zhang
Yiwen Gu
Bryan A. Plummer
Venue: EMNLP 2023 Findings (2023)
Keywords: article generation
article retrieval
named entity recognition

Abstract: Article comprehension is an important challenge in natural language processing with many applications such as article generation or image-to-article retrieval. Prior work typically encodes all tokens in articles uniformly using pretrained language models. However, in many applications, such as understanding news stories, these articles are based on real-world events and may reference many named entities that are difficult to accurately recognize and predict by language models. To address this challenge, we propose an ENtity-aware article GeneratIoN and rEtrieval (ENGINE) framework, to explicitly incorporate named entities into language models. ENGINE has two main components: a named-entity extraction module to extract named entities from both metadata and embedded images associated with articles, and an entity-aware mechanism that enhances the model's ability to recognize and predict entity names. We conducted experiments on three public datasets: GoodNews, VisualNews, and WikiText, where our results demonstrate that our model can boost both article generation and article retrieval performance, with a 4-5 perplexity improvement in article generation and a 3-4% boost in recall@1 in article retrieval. We release our implementation at [this http URL](https://github.com/Zhongping-Zhang/ENGINE).

Show, Write, and Retrieve: Entity-aware Article Generation and Retrieval
Zhongping Zhang
Yiwen Gu
Bryan A. Plummer
Boston University (zpzhang, yiweng, bplum)@bu. edu
Abstract
Language Modeling
Earity-aware Language Modeling
Thd Angillas Jale wi Eo Vaddig Profesos Lchool feanardis Donntr soccentrmun
Article comprehension is an important chal- lenge in natural language processing with many applications such as article generation or image- to-article retrieval. Prior work typically en- codes all tokens in articles uniformly using pre- trained language models. However, in many ap- plications, such as understanding news stories, these articles are based on real-world events and may reference many named entities that are difficult to accurately recognize and predict by language models. To address this challenge, we propose an ENtity-aware article GeneratloN and rEtrieval (ENgINE) framework, to explic- itly incorporate named entities into language models. ENGINE has main components: a named-entity extraction module to extract named entities from both metadata and em- bedded images associated with articles, and an entity-aware mechanism that enhances the model's ability to recognize and predict entity names. We conducted experiments three public datasets: GoodNews, VisualNews, and WikiTexx, where our results demonstrate that our model can boost both article generation and article retrieval performance, with a 4-5 per- plexity improvement in article generation and 3-4% boost in recall @1 1 in article retrieval. We release our implementation at this http URL.
Noteppmms
Buby Aupdie the Usitd Naiin takes on now seko unsuaity prefoues
enay lea
jois
Theacres Usaed
Scheol toe Wonee, Neace and Stcarty ane of See victing protosere noe appllaan prapon aocering expemett relened ce Mtaday by the catrenaty. one brozh ront sitais there ccavatin anist leckeee and lacrg put werkoogg "vaking porkowe
ratupaagnoo warl Londe moce
"egg 4i
joke Par [[FRSSN] de end covileenee beok-llnnt Nasses (DRG of spary uim on now wdo ceaumry molfiuer Mr Jike CamerooieG Mhunes Feur Secuity slking partsoo is now eurser prggeet that damm raking eapkien
soondag sunmarr ideued Koodgy bythoueeeadg
Figure I: We propose entity-aware language modeling for article generation and retrieval. Prior work (Brown et al., 2020; Zellers et al., 2020; Ouyang et al., 2022), shown in (A), typically comprehends articles by uni- formly encoding all text tokens. However, these ap- proaches face challenges in accurately recognizing and predicting named entities. In our paper, shown in (B), we propose a method to extract named entities from embedded images and explicitly model the named enti- ties in articles, boosting the performance of both article generation and article retrieval.
Pile (Gao et al., 2020) LAION400M (Schuh- mann al., 2021)). These models typically uni- formly encode all text tokens in the articles includ- ing named entities (Radford et al., 2018; Brown et al., 2020; Zellers et al., 2020). In other words, named entities like organizations, places, and dates are modeled together with other text, as illustrated in Figure ((A). However, it poses challenge for these models to accurately recognize and predict named entities, as they can be unique to specific ar- ticles. E.g., in Figure 1(A), entities like "Ms. Jolie" and "Jolie Pitt" may only appear articles related to the celebrity "Angelina Jolie."
1 Introduction
Comprehending articles enables a wide range of ap- plications such as story generation (Fan et al., 2018; Peng et al., 2018), image-to-text retrieval (Tan et al., 2022), automated journalism (Leppänen et al., 2017; Brown et al., 2020), defending against misinformation (Zellers et al., 2020; Tan et al., 2020), and writing Wiki articles (Banerjee and Mi- tra, 2016; Stephen et al., 2017), among others. In- spired by the impressive capability of large lan- guage models, recent work (e.g., Radford et al. (2019); Brown et al. (2020); Wang and Komat- suzaki (2021)) generate or retrieve articles by train- ing language models on massive datasets (e.g., The
Directly extracting named entities from user- provided prompts straightforward entity-aware approach used by methods addressing news image captioning (Biten et al., 2019; Tran et al., 2020; Liu et al., 2021). However, these methods do not

---

generalize well to article generation and retrieval since they rely on substantial contextual informa- tion (via the articles) as well as a direct indication of entities that may appear in predicted captions. For example, in Figure! (A), the article mentions that Angelina Jolie Pitt is an Oscar winner, but this information is not present in the other metadata like the image captions. Thus, a language model generating an article must infer this named entity information rather than directly extracting it from the metadata. Even a list of named entities were provided, an article generation model must deter- mine where and when to use each of them. In contrast, in news image captioning, entities used in the caption almost always appear in the body of the article (Liu et al., 2021; Tan et al., 2020), and the image itself will directly inform what named entities should be used for its caption. Thus, as we will show, adapting entity-aware mechanisms from related work (e.g.. (Liu et al., 2021; Dong et al., 2021)) I)) results in poor performance in our task.
GINE, for article generation and retrieval. Com- pared to existing language models (Brown et al., 2020; Zellers et al., 2020; Radford et al., 2021; Sun et al., 2023), our entity-aware mechanism en- hances the recognition and prediction of named entities by jointly modeling entity names and their entity categories, boosting the performance of article generation and retrieval.
We introduce named-entity extraction method recognize named entities from embedded im- ages in articles, eliminating the overhead in man- ually creating a list of named entities that will appear in the articles.
Experiments on GoodNews (Biten et al., 2019) and Vesual (Liu et al., 2021) show perplex- ity gain of 4-5 points for article generation and a Recall @1 boost of 3-4% for article retrieval. We also show that ENGINE generalizes via zero-shot transfer Wiki (Stephen al., 2017).
We perform comprehensive experiments on hu- man evaluation and machine discrimination, val- idating that ENGINE produces more realistic ar- ticles compared to prior work (Radford et al., 2019; Zellers et al., 2020). This suggests that our model can potentially contribute additional train- ing data for the development of more powerful machine-generated text detectors.
To address the aforementioned issues, we pro- pose an ENtity-aware article GeneratIoN and rEtrieval (ENGINE) framework to explicitly incor- porate and model named entities in articles. EN- GINE mainly consists two modules: a named-entity extraction module and an entity-awar module. In the named-entity extraction module, we show that providing a list of named entities for article gen- eration improves performance. However, creating such lists does require a small overhead cost. Thus, we also demonstrate we can improve performance without manual input.
Related Work
Article Generation in recent work uses large-scale pretrained transformer models that can be sepa- rated into unconditional text generation (Radford et al., 2018, 2019) and conditional text genera- tion (Brown al., 2020; Zellers al., 2020). Gen- erating articles via unconditional samples has been found to be less effective, since the models may interpret the first sentence of articles as a tweet and start posting responses (Brown et al., 2020). To enable controllable generation, GPT3 (Brown et al., 2020) conditions article generation on titles and the initial sentences of articles. Grover (Zellers et al., 2020) decomposes news articles into distinct parts and conditions generation on metadata like the author or organization. In this paper, we further explore the impact of named entities and embed- ded article images. Specifically, ENGINE produces articles conditioned on both metadata and embed- ded article images, with dedicated focus on the explicit extraction and modeling of named entities. Article Retrieval is commonly accomplished by image-text matching frameworks. Early work on
As shown in Figure I (B), we observe that im- ages associated with an article often contain in- formation about the article's events. Therefore, we explore leveraging large vision-language mod- els to extract named entities from embedded im- ages. Specifically, we employ CLIP (Radford et al., 2021) to automatically select a set of likely named entities from embedded images. In the entity-aware module, we introduce special tokens after each en- tity name to indicate its entity category. In this case, ENGINE models the named entity and its en- tity category jointly. An additional benefit brought by our antity-aware mechanism is the named-entity recognition (NER) ability, i.e., our model not only recognizes and predicts the entity names but also predicts their entity categories simultaneoushe
In summary, the contributions of this paper are:
We propose an entity-aware language model, EN-

---

(A) Named-entity Extraction
(B) Entify-awar mechanism
Domain: nytimes.com
Extracted Named Entities
is revolution that started March 5, 2012 D s. when he signed a contruct extensson will keep gold throuph 2018 (0477]) It did not matter that the Pirates were coming their 90 < CARDINAL] -lass season. McCutchen sow bright future where others lest
Date: 03-24-2014
Title: Socking
Named Pinitty Rocognition (NER)
Andrew McCurchen PERSON]>
Sponts New Face
the Pirnsburg Pirates <OPE>
<JORDINALE straight
Troy Polamalu <ORG>
The World Series +[EVENT)
Now the Pirates heading the playoff again led by intense charismatic and talented player who has become biggest star in city where Sidney Crosby -IPERSOM Ben Roethllisheger PERSON) and Troy Palamahi 
 work Jeter, though has five < AARD World Series <VVENT> rings tucked away somewhere - while McCutchen's <|PERSONIS fingers are currently bure save wedding band
Yankees +[RG|>
Visual NER ICLIP, EVA-CLIP, etc.)
March 5, 2012 <DATE>
why could handftll of players move the sport forward, rather than just llter became Jeter October (2477]]
maybe
Prompts
Extracted
Language Model
Targer Articles
(Metadata)
Named Entities
Figure 2: ENGINE overview. Our model mainly consists of two modules: (A) Named-entity Extraction: We extract named entities from both metadata and embedded images in articles. See Section 3.2 for detailed discussion; (B) Entity-aware mechanism: Each named entity is associated by its corresponding entity category. ENGINE models the entity name and category jointly to avoid inconsistency between named entities images and article text. See Section 3.1 for detailed discussion.
image-text matching has primarily focused on de- veloping bespoke models (Wang et al., 2017; Gu et al., 2018; Nguyen and Okatani, 2018; Nam et al., 2017) with various retrieval loss functions, such as triplet loss (Schroff et al., 2015) or proxy anchor loss (Kim et al., 2020). However, these models are often domain-specifer and limited in the expres- siveness of the text. recent work, large vision- language models (Radford et al., 2021; Sun al., 2023; Li et al., 2022) have addressed these limita- tions by contrastive image-language pretraining on massive datasets (e.g.. LAION-400M (Schuhmann et al., 2021)). These large vision-language mod- els have demonstrated exceptional performance in zero-shot image-text matching. Therefore, in our experiments, we employ the pretrained vision- language model EVA-CLIP (Sun al., 2023) to extract visual and text representations from articles, and compute the cosine similarity between these representations to obtain the retrieval predictions.
Tran et al., 2020; Liu al., 2021). In this paper, we effectively reverse the inputs and outputs of these approaches, i.e, we generate an article based on im- ages and captions rather than generating captions based on images and articles. As discussed in the Introduction, this shift breaks the assumptions used by entity-aware mechanisms in image captioning. causing them not generalize well in our article generation and retrieval task.
ENGINE: ENtity-aware article GeneratIoN and rEtrieval
Given user-provided prompts and embedded article images, our task aims to more accurately repre- sent articles by recognizing and predicting named entities. Thus, we incorporate candidate named entities as an additional control to our language model. We propose entity- mechanism to jointly model entity names and their corresponding entity categories in Section 3.1. To enrich the en- tity information available to language models, we introduce our named-entity extraction approach Section 3.2. Finally, we introduce the learning strat- egy of ENGINE in Section 3.3. Figure 2 provides an overview of our method.
Entity-aware mechanisms have been applied in closely related tasks, such as news image caption- ing, which aims to caption images based on arti- cles and images. Ramisa et al. (2017) proposed an end-to-end framework that takes the concatenation of article and image features as input and outputs captions by an LSTM decoder. However, this ap- proach often fails to predict named entities that were not seen during training. Thus, more recent work boosts performance by extracting entity rep- resentations from user-pu user-provin providedd articles and inserts them into generated templates (Biten et al., 2019;
3.1 Entity-aware Mechanism
As discussed in the Introduction, accurately rec- ognizing and predicting named entities can help avoid inconsistencies between the associated im- ages and the textual content of an article. For exam- ple, in NBA news, an entity- model should be

---

able to predict "Curry" given the preceding word "Stephen," whereas traditional language models might struggle in this regard. Existing methods typ- ically model named entities uniformly with other text, making the leverage of named entities less effective. To help our language model be aware of named entities, we insert the entity category pre- dicted by SpaCy (Honnibal and Montani, 2017) after each entity name. We special tokens as the indicator of these entity types. Then the entity name and its corresponding category are modeled jointly by ENGINE. We visualize our cntity-aware mechamism in Figure 2 (B).
3.3 Learning Strategy
Language Modeling. Given a set of docu- ments (21,#2,... Φnf each with variable length sequences of symbols "t, the statisti- cal language model of a text document z can be represented by the probability of next symbol given all the previous ones (Bengio al., 2003):
p(x) II p(s
(1)
where each symbol s; is processed uniformly and the document æ is viewed as an unstructured text field (also referred as body field later). Language models based only on Eq. 1 produce articles via unconditional samples. Thus, these models are not suitable for controllable generation (Hu et al., 2017). Instead, the language model can be formu- lated by the joint distribution of separate fields de- composed from the article x (Zellers et al., 2020):
3.2 Named-entity Extraction
Our entity-aware mechanism in Section 3.1 en- hances language model's ability to recognize and predict named entities. However, we find that the entity information extracted from the metadata may not be sufficient for article generation. Thus, as shown in Figure 2 (A), we explore named entity extraction methods from various sources within ar- ticles, such as embedded images. Below we discuss two ways to create the named entity list.
p(a)=
p( meta
body
(2)
where meta is a data-dependent term consisting of a set of subfields. For instance, meta includes date, title, summarr in GoodNews (Biten et al., 2019) and domain, date, topic, title in Visual- News (Liu et al., 2021). Thus, we model x by:
Oracle named-entities. This approach assumes that we are provided with all the named entities that would appear in articles, e.g., named entities pro- vided by a user. To simulate user-provided named entities, extract named entities from news arti- cles using SpaCy (Honnibal and Montani, 2017). This list then provided as input to our model.
p(x) = p( body meta
)p(
meta
Based on Eq. 3, we further introduce special tokens <start->>> and <end-r>> to indicate the boundaries of field r. The content of target field is sam- pled from the model starting with <start-7>> and ending with <end-r>, Given the named entities extracted by our method (from Section 3.2), Eq. 3 is re-formulated as:
CLIP-based NER. Existing Named Entity Recog- nition (NER) methods (Yadav and Bethard, 2018; Li et al., 2020) primarily distinguish named enti- ties within text documents and are not designed for NER involving images (referred to as ViSual-NER in Figure 2(A)). However, we note that CLIP (Rad- ford et al., 2021) was trained on 400 million image- text pairs collected from the internet, many of which likely contain named entities. Thus, we use CLIP build an open-ended Visual-NER frame- work. First, we construct candidate list of named entities for each image by extracting entities from the articles in the dataset using SpaCy (Honnibal and Montani, 2017). Subsequently, we use CLIP to predict the similarity between the article images and the candidate entities. The top A entities are then provided input to our model.
p(x) = p( body
meta
entity
)p( meta
entity
To sample from Eq. 4, we define canonical order? among the fields (or subfields) of articles F (fi f2 < ... < f F)) and model the articles left-to-right in the order using Eq. sft 8 :.... $1 Architecture. Following (Zellers et al., 2020), EN- GINE uses the GPT2 architecture (Radford et al.,
We define canonical order in Goodnews (Biten et al., 2019) as: domain, date, named-entity, title, caption, summary. body; and Visualnews (liuu et al., 2021) as: domain, date, topic, namod-entity, title, caption, body.
We set & to 10 in this paper.

---

2019) for article generation. We experiment with three model sizes: (1) ENGINE-Base has 12 layers and 124 million parameters, on par with GPT2- 124M and GROVER-Base (2) ENGiNE-Medium has 24 layers and 355 million parameters, on par with GPT2-355M and GRovER-Large; (3) EN- GINE-XL has 48 layers and 1.5 billion parameters. on par with GPT2-1.5B and GROVER-Mega. In addition, we also show ENGINE generalizes across architectures by evaluating on LLAMA (Touvron et al., 2023). For article retrieval, we implement our method on pretrained vision-language model EVA-CLIP (Sun et al., 2023).
Implementation Details We primarily imple- mented our models using Pytorch (Paszke et al., 2019) and Transformer (Wolf et al., 2020) libraries. The maximum sequence length of language mod- els is set to 1024. For ENGINE- Base and ENGINE- Medium, we used a batch size of 8 and maximum learning rate of 1 10-4, For ENGINE-XL, we used batch size of 4 to fit into GPU memory, Cor- respondingly, the maximum learning rate is set 20.5 10-4. We finetuned our models for around epochs with 0.06 epoch for linear warm-up on both datasets. We parallelized ENGINE-XL on 4 NVIDIA RTX-A6000s and ENNINE-Mediug on 2 NVIDIA RTX-A6000s. ENGINE-XL on Visual-
4 Experiments
News requires the longest training time- approxi- mately two weeks on our system.
4.1 Datasets and Experiment Settings
Datasets. We evaluate ENGINE on three public datasets: GoodNews (Biten et al., 2019), Visual- News (Liu et al., 2021), and WikiText (Stephen et al., 2017). GoodNews provides the URLs from New York Times from 2010 2018. After filter- ing out broken links or non-English articles, we downloaded 307,286 news articles. Following the split ratios of Biten et al. (2019), we randomly split 15,365 articles for validation, 30,728 articles for testing, and used the rest for training. VisualNews contains news articles from four news sources: Guardian, BBC, USA Today, and Washington Post. We obtain 582,194,, 194 news articles in total after we re- moved broken links and articles without metadata. Similarly, we get 491,796 training set, 28,932 val- idation set, and a 57,889 test set. WikiText contains 600/60/60 Wikipedia articles in train/test/validation sets, respectively, We performed zero-shot article generation on the test set of WikiText.
4.2 Article Generation
Perplexity. Table 1 presents sizes, architectures. and perplexity results of different models on Good- News (Biten et al., 2019) and VisualNews (Liu et al., 2021). We see that ENGINE variants of all model sizes significantly outperform the base- lines. On the base size, ENGINE-Base(NE) im- proves PPL over the original GPT2-124M model by factor of 2 (23.6 27.5 -> 13.1). We draw three major conclusions from Table 1. First, the data distribution still plays an important role. Finetuned GPT2s improve PPL over the original GPT2s. The improvements become less obvious with a greater model size (VisualNews: 27.5 of base size; 15.7-12.4 of XL size). Second, EN- GINE noticeably improves the performance over finetuned GPTs (4-5 perplexity points both datasets), which demonstrates the effectiveness of our approach. Third, our contributions are architec- ture agnostic. E.g., ENGINE-LLLAMA outperforms LLAMA-7B with an approximately 2-point per- plexity improvement on both datasets.
Metrics. Following Zellers et al. (2020), adopt Perplexity (PPL) to evaluate models on article generation. Perplexity is defined the exponenti- ated average negative log-likelihood of a sequence. Given Eq. 1, the perplexity of & is calculated by:
Parameter Efficiency. Table 1 shows that EN- GINE can achieve a comparable performance with alternative models using much fewer parameters. For example, ENGINE-Base(NE) with only 124M parameters, outperforms the GPT-NEO-2.77 and achieves comparable performance with finetuned GPT2- 1.5B (12.0 vs. 12.6 PPL on GoodNews, 13.1 Vs. 12.4 PPL on VisualNews). ENGINE-Medium (NE) model, with 355M parameters, outperforms all the GPT-Series baselines including GPT-J-6B. Figure 3 plots the perplexity as a function of the number of parameters on news datasets, demon-
m
PPL(x) = exp
log ((:((1)...................._)
85
(5)
where 81, 8j are ground truth tokens in and p(-) is the probability predicted by the model. We evaluate models using @ecall@k K (R@1, R@5, R@ 10) for article retrieval.
DDuring inference, we get rid of entity categories from our generated articles to make fair comparisons to other buselines.

---

GoodNews
Visual
Model Name
Parrams
7aayyess
mmodel
Teads
PPL
PPL
GPT2-124M (Radford et al., 2019)
124M
12
768
12
23.6
27.5
GROVER- Base (Zellers et al., 2020)
124M
12
768
12
23.8
21.9
GPT-Neo- 125M (Gao et al., 2020)
125M
12
768
12
27.1
29.3
GPT2-1 24M (Finetuned)
124M
12
768
12
17.3
18.3
ENGINE-Basc (ClipNE)
124M
12
768
12
14.8
16.1
ENGINE-Base (NE)
124M
12
768
12
12.0
13.1
GPT2-355M (Radford et al., 2019)
355M
24
1024
16
17.8
20.1
GRovER-Large (Zellers et al., 2020)
355M
24
1024
16
18.5
16.4
GPT-Neo-1.3B (Gao al., 2020)
1.3B
24
2048
16
15.3
15.9
GPT2-355M (Finetuned)
355M
24
1024
16
13.5
14.0
ENGINE-Nediim(CliNNE)
355M
24
1024
16
11.6
12.5
ENGINF-Medium(NE)
355M
24
1024
16
9.5
10.2
GPT2-1.5B (Radford et al., 2019)
1.5B
48
1600
25
13.9
15.7
GRovER-Mega (Zellers et al., 2020)
1.5B
48
1600
25
14.5
12.6
GPT-Neo-2.7B (Gao et al., 2020)
2.7B
32
2560
20
13.5
14.0
GPT-J-6B (Wang and Komatsuzaki, 2021)
6B
28
4096
16
11.3
11.6
GPT2-1.5B (Finetuned)
1.5B
48
1600
25
12.6
12.4
(NCipN))X((ClipNE)
1.5B
48
1600
25
10.8
11.1
ENGINE-XL(NE)
1.5B
48
1600
25
8.7
9.0
LLAMA-7B (Touvron et al., 2023)
7B
32
4096
32
8.3
8.5
ENGINE+LLAMA (NE)
7B
32
4096
32
6.5
6.4
Table Perplexity (PPL) comparison on GoodNews and VisualNews. ClipNE denotes that we select CLIP-based named entities in named-entin field (described in Section 3.2), NE denotes that apply oracle named entities in We observe that our model consistently outperforms
named-entie field. PPL is calculated only on the article body
baselines of comparable model sizes.
strating that ENGINE gets as good or better results than prior work while also using fewer parameters. Additional Baselines adapted from close-related tasks*. In Table 2(A), we that ENGINE-1 1.5B outperforms integrating BU (Anderson al., 2018) and VisualGLM-6B (Du et al., 2022) features, demonstrating that our entity-aware mechanism is more effective than directly incorporating image features. As discussed in the Introduction, this is likely due to the loose correlation between images and their corresponding articles. Table 2(B) demon- strates that the entity-aware mechanisms proposed for other text generation tasks do generalize well to article generation, where our approach ob- tains 4-5 PPL. improvement on both datasets. We also find that InfoSurgcon struggles to generate arti- cles well, which we argue is due to its sequence-to-
sequence translation framework finding it challeng- ing to effectively leverage prior knowledge from pretrained language models.
Ablation Study. Figure shows ablations of EN- GINE-Base. observe that both the caption and named-entity fields boost performance, revealing that cues from embedded images help produce higher-quality articles. Comparing using only cap- tions (Cap) vs. combining them with our Entity- mechanism, get minimum gain of 0.6 PPL, demonstrating its effectiveness. In addition. observe that ClipNE outperforms CapNE, vali- dating that CLIP-detected named entities are more effective than those extracted from captions. Article Quality User Study. Following (Zellers et al., 2020; Kreps et al., 2020; Brown et al., 2020), ask annotators distinguish machine-generaaeed articles from human- articles. We randomly selected 50 news stories from GoodNews and Vi- sualNews test sets (100 total). Given the meta- data and news images, we generated news articles using three different language models: GROVER-
'InfoSurgeon (Fung et al., 2021), InjType (Dong et al., 2021). VNC (Liu et al., 2021) are adapted from sequence-to- sequence translation, close-ended paragraph generation, and news image captioning, respsctively. For a fair comparison, GPT2- Base is applied as the hackbone of InjType, VNC, and BU for article generation.
BaseeBase

---

21.5
OPT2
Good- Visual-
25.8
Model Name
News
News
22.3
(PT) EMDTY
perplesty
20.0
(A) BU (Anderson et al., 2018)
26.6
24.3
17.5
(isualGLMMDu et al., 2022)
11.8
12.2
15.8
ENGINE-1.5B (ours)
8.7
9.0
12.5
16.3
(B) InfoSurg (Fung et al., 2021)
41.8
42.1
0.
0.5
1.0
1.5
2.0
2.5
10
6.0
InjType (Dong al., 2021)
18.2
19.0
pprrameters
VNC (Liu al., 2021)
16.7
17.8
ENGINE-Base (ours)
12.0
13.1
DROVER
OFT-AN
25.0
DIT)
perplesity
22.5
Table 2: PPL of baselines adapted from close-related tasks. (A) leverages visual features directly extracted from images, (B) contrasts entity-aware mechanisms. Adapting entity-aware mechanisms from related tasks may result in poor performance in our task.
20.5
17.5
12.5
10.2
-100
@parameters
Teat-rnly
Cap
Figure 3: Comparison of the perplexity of different language models on GoodNews (top) and VisualNews (bottom) as function of learned parameters.
Cap EA
Cap CapkE EA
17.1
17.3
Clipht E4
Cap-NE-BA
15
15.5 15.0
Mega, GPT2-1.5B (finetuned), and ENGINF-XI. This results in total of 200 articles per dataset. We recruited 200 Qualified Amazon Mechanical Turk (AMT) workers per dataset. Each article was annotated 5 times by AMT workers, where each worker was presented with the article titles, im- ages and captions, and was asked to indicate if the article was human machine generated. If they thought the article was -eebiineggeerttedd, they were asked to indicate a reason for it following the same option format as Tan al. (2020). Table reports annotation accuracy in identifying articles from VisualNews as machine or human-generated. We see EXGINE.XXL is able to generate hard-to- detect news articles (a 2% boost over GPT2-1.5B, and 5% gain over GRovER-Mega), validating the effectiveness of our approach.
10
Vauatless
Good/ens
Figure 4: Ablation results of ENGINE- -Bas.. Text-only denotes the model focuses only on text information. which is same to finetuned GPT2 model. Cap denotes caption field, EA denotes the Entity-awarr mechanism. CapNE denotes named entities extracted from captions. Both the named-entity extraction module and our entity- aware mechanism can improve the performance.
enough similarities in the distributional features that are recognized by the machine discriminator. Zero-shot Article Generation We perform zero- shot experiments on Wikipedia articles to demon- strate ENGINE'S s ability generalize. Table 4 reports performance on WikiText. We find that our oracle named entities and entity- mech- anism still can improve the performance over sev- eral baselines, even though the data distribution between Wikipedia and news is significantly differ- ent. For example, the GPT2 models finetuned on Visualnews get worse performance than the original GPT2 models on WikiText. However, our ENGINE models get comparable or better results than the original GPT2 across different model sizes. Qualitative Results. We provide a qualitative com- parison of GoodNews articles in Figure 5. Con-
Machine Discriminator. We apply OpenAl's RoBERTa (Liu et al., 2019) detector to detect gen- erated articles. The maximum article length is cut to 512 to fit the model input size. For comparison, we use the same article set from our user study. In Table 3, we observe that the machine discriminator is much better at identifying the machine -eeerattd news. We see that articles produced by ENGINE- XL can be reliably detected by RoBERTa though it gets the lowest accuracy on human evaluation. This can be due to the fact that GRovER-Mega, GPT2- 1.5B, and ENGINE-XI all share a similar under- lying model architecture. Thus, they may contain

---

Designer Unites Basque Country Domain: wore mytimmsmcmooo
Date: 06-10-2016
Jean Louis Instakki chedking the scale model of chelt for Alki.
Human-wrine News Artida
ENtiaNi--omemeodd Nows Acticle
2007, French induarial dedgnerr Jam Lauis Trateski upened a remose studko in wood cabin in the fouchils of the Pyresers His minimalis "sclugy" hes reanlly haume in unfiiely crautive qpeanier for inducsial production on both the French and Spanids sida of the Pays Buqque
Fallowing in finnccys of mak counterpurt, the Franch- Spanids architst Iean Louis lestzoki whose archive became the sulject of lot year's Banque/ / Tranosphile Rodoo Dive ohibitton, Ander Liows, the desggar of the Pays Busque label Maison & Olijt, has taken on the reatindl new mank of directoire his right This year he announced the insugural of Dasign Angdar- Motion, the lagss desgo füir fur leand Francoghone woekd.
Inseski won plaudine for decgpiing one of woeld's bioplatic chairs made biodogrdabli plant based the Baque fumiture brand klki, is only one examplc of Mr. lestackd's determination peceuots his native region. (Mr. tranoki was bomn in France in St. Jean de Lun.) Ile has also roondly revaped the brand ideniit of Commase Busque duocllate company) and belped desiga the interfors lusurious lodge in Suubion, Biarrite.
last year's Maison Oljet
Baque
puymmrr
shall chiir, part of long cilibbrationn
"ie's the onr name you know everyhhare sii Liuss, who came age deigner in the city of Alli during the fine boom in constraction made there, the communiity's groing buiices." And it wa
selling commisionen
of
l'rance""
Going back to Reigium, where the heritage the Maison Objer founders still lie, imboting his name with its celebrrre high culture assoctations he explated, the house was fully awart of it Besque roots""
This he founded new peartiee with Baqque dusigner of Spanich exrraction, Ander Licao, with the aim of crewing a multiguppse desgn studio Inspiced by
GITT2 generatod News Astice
Gaow LK generated News Astide
Teans cludad Clais Burch and his wifl, Sumns, when they first act syss on cads other headwalter in 1991, They had no idea how they would end up there. But nuither did be; not at the time, ivn 10 year laut. Thair frisnbhiip would grow into centative
In both making cltting and pulitios Mr. Madrid strino to combinne the traditiona of dress and activim with the street's disticc urbunity She designs range of produats from T-siits and tea for milkennial, to man' a duies for familics Everyching both stand her slentity designes. TIa person designing my clothes because I'm interview at in town. TIi very and strong mos fat women."
would tefce, both of them, work play together laque region
pasionatt
mwyy
Mr. Madrid
New Yirk. themselees In 2012, with both 60, Barch deckled to marry them Thirry years lane, the breehers will go their eppraat ways: Chris, the arriste diveccee of the Coype Rose Loe boutique in the West Villige, and Suaan, founder and creative dinector of her company, Flom Floss which quectalioes in foral teerles and temporary thing, where people sending you
saldenly they were happly on thisr own, taking care
The 4o--yar-ood mocher of rwo is craating now budnros. It's siming to combine tradi tradtiional fashion and actistsn She's i hoping raise money and offer service for political activium the Besque country- and beyond.
des." 'A weddiigg
your merry way" Chrls explins
puer decades, Mr Madrid been Invelved in social politicll
you can vay. With something you wost to do full-flelgdd "We empyol or have procosal you can't even focus." The couple recendl noved to Mienl, whom Chra wanes to take nontial theme to Howall, where Snans From. adon, they disue the undercerent of thiir remance lidiung friend.
actistum. She want hunger strke after the was attcced while jul,, Hir goal to become national emblen the Bosque country, she sald, but she's not consinced this is resporabillty of the Bauque gevenmenn.
Figure 5: Qualitative comparison GoodNews. We cut the articles to fit the figure size. The entity names from image information are highlighted in light purple (PERSON tag) and light blue (ORG tag) colors. We can see that the named entities captions also appear the human-written and ENGINE Eegeeaeee articles. contrast, the GPT2-generated and
articles do not contain correct entity names corresponding to the image.
Method
Human
RoBERTa
Method
124M
355M
1.5B
GRovER-Mega 72.8%
90%
GPT2
26.1
19.1
14.8
GPT2-1.5B (Finetuned)
69.6%
84%
GPT-Neo
24.9
13.1
11.5
ENGINE-XL
67.6%
84%
GPT-J-6B
9.0
GPT2 (Finetuned)
33.8
25.2
25.9
Table 3: Detection of generated articles. Left column reports the performance of AMT workers at correctly identifying an article as human or machine generated. Right column uses OpenAl's text detector based on RoBERTa (Liu et al., 2019) to perform the same task.
ENGINE (NE)
20.7
15.4
16.3
Table 4: Zero-shot article generation. We compare the perplexity of ENGINE to GPT-2 (Radford et al., 2019), GPT-Neo (Gao et al., 2020), and GPT-J (Wang and Komatsuzaki, 2021) on WikiText (Stephen al., 2017). Both GPT2 and ENGINE are finetuned on Visual- News (Liu et al., 2021) with a maximum article length of 1024. Though the data distribution between Wikipedia and news is different, proposed entity-aware method still improves the performance over several baselines.
sistent with our annotation experiment, we com- pare the human-written article with three machine- generated articles. From the results, we can see that ENGINE-XL. XL can effectively produce articles with the named entities learned from image information. In contrast, finetuned GPT2-1.5B and GROVER- Mega failed generate correct named entities in articles. For example, both ENGINE-generraed arti- cle and the human-written article mentioned "Hean Louis Iratzoki" and "Alki", which are appeared in the caption. In contrast, articles generated by GPT2
or GROVER are discussing some other entities such as "Chris Burch" and "Ms. Madrid."
4.3 Article Retrieval
Table 5 compares the Recall K retrieval scores of ENGINE with the ofate-of-the-a baselines on the

---

Image-to-article
Article-to-mmage
GoodNews
News R@5
GoodNews
VisualNews
Method
R@1
R@5
R@ 10
@
R@ 10
R@1
R@5
R@ 10
R@1
R@5
R 10
CLIP-B/16
19.4
42.7
52.9
30.6
53.5
64.6
32.5
56.8
64.4
43.1
69.5
76.5
CLIP-1/14
32.5
56.0
65.3
47.3
72.1
78.9
46.6
68.3
75.0
55.0
77.3
84.3
BLIP
19.8
38.0
48.8
22.0
43.6
52.5
15.9
30.8
37.9
15.1
33.3
42.7
BVA01-CLIP-G/14
49.6
70.3
77.1
56.9
78.9
84.7
47.6
68.3
75.2
54.9
76.5
82.5
EVA02-CLIP-L/14
50.7
73.1
79.4
57.9
78.6
85.2
49.3
69.6
76.8
54.8
77.3
82.7
ENGINE (ours)
53.8
73.5
79.6
61.9
82.0
86.7
51.5
72.0
77.8
56.3
78.3
83.9
Table 5: Retrieval results. We compare the Recall @ of ENGINE to CLIP (Radford et al., 2021). BLIP (Li et al., 2022), and EVA-CLIP (Sun et al., 2023) on the article-to- article-to-image and image-to-ane to-aticll retrieval tasks on GoodNews and VisualNews. For articles with multiple embedded images, we only the first embedded image. See Section 4.3 for discussion.
test splits of GoodNews and ViwualNews. Follow- ing Tan et al. (2022), we randomly select 1500 article-image pairs from each dataset for evalu- ation. In the image-to-article retrieval task, we observe that compare to EVA02-CLIP-L/14 (Sun et al., 2023), ENGINE boost the Recall @1 scores from 50.7 to 53.8 on GoodNews and from 57.9 to 61.9 to VisualNews. For trricee-to-image retrieval, ENGINE achieves the highest performance with @ecall@ scores of 51.5 and 56.3 on GoodNews and VisualNews, respectively. The retrieval results validate the importance of named entities within ar- ticles and the effectiivness of our proposed method,
ent views of the same prompt, which can provide additional information for training an even more powerful machine generated text detector. We note that our contributions are largely architecture ag- nostic, so they could also be used in RNN-based generators, which may provide a larger distribu- tion shift in the generated articles that may fool a discriminator trained only on Transformer outputs.
6 Conclusion
In this paper, we proposed ENGINE, an entity- aware article generation and retrieval method that explicitly incorporates and models named entities in language models. Concretely, ENGINE extracts named entities from both metadata and embedded images in articles, providing source of entity information. In addition, intro- duce an entity-aware mechanism to help ENGINE recognize and predict named entities more effec- tively and accurately. ENGINE outperforms current popular language models in quantitative and quali- tative experiments on GoodNews, VisualNews, and Wiki'Text. For example, ENGINE outperforms GPT- J by roughly 2.5 perplexity points using only quarter parameters of GPT-J and boost the perfor- mance EVA02-CLIP by 3-4 Recall accuracy in article retrieval experiments. The noticeable im- provements demonstrate that ENGINE can generate and retrieve articles more accurately and efficiently by effectively leveraging named entities.
5 Discussion
In our paper, we mainly investigate modeling machine-generated articles, which can be used directly for generation while also can provid- ing strong language features to support applica- tions like article retrieval. However, actors can also use the same technology to generate arti- cles for misinformation by modifying informa- tion of specific fields to realize two potential pur- poses: monetization (ad revenue through clicks) propaganda (communicating targeted informa- tion) (Zellers et al., 2020). Thus, the develop- ment of a better article generator can not only help humans write high-quality articles but also potentially help train a more powerful discrimi- nator. Table 3 reports the performance of using human judgements or OpenAl's RoBERTa-based machine generated text detector (Liu et al., 2019). When comparing the results of the RoBERTa de- tector for GPT2-1.5B and ENGINE-XL, we find that only 25% of the articles that were predicted as human written came from the same generation prompts. Thus, the two methods can provide differ-
comprehensive
A Acknowleggements This material is based upon work supported, in part, by DARPA under agree- ment number HR001 12020054. Any opinions, findings, and conclusions recommendations ex- pressed in this material are those of the author(s) and do not necessarily reflect the views of the sup- porting agencies.

---

Limitations
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic lan- guage model. The journal of machine learning re- search, 3:1137-1155.
We discuss limitations and potential improvements of our work in this section. First, though our method can effectively predict the correct entity names in articles, their corresponding entity cate- gories might be mistakenly predicted. For example, in Figure 5, the brand name "Alki" is recognized city name by ENGINE. Therefore, more ac- curate entity-aware mechanism could be developed in future work. Second, the image information can be further explored. In this paper, we mainly in- vestigate the captions and named entities of news images, However, other information such as the locations of images within articles may also prove useful for article generation. In addition, cur- rent methods detect named entities from images considering each entity independently using text- image matching framework. However, since the relationships between entities also affect the proba- bility that entities appear in images, the incorpora- tion of entity relationships can also be considered to further improve the entity detection module.
Ali Furkan Biten, Lluis Gomez, Marçal Rusinol, and Dimosthenis Karatzas. 2019. Good news, everyone! context driven entity-aware captioning for news im- ages. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12466 12475.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in newral information processing systems, 33:1877-1901.
Xiangyu Dong Wenhao Yu, Chenguang Zhu, and Meng Jiang. 2021. Injecting entity types into entity- -uiidd text generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
Zhengxiao Du, Yujic Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregres- sive blank infilling. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics, pages 320-335.
Ethics Statement
ENGINE is a model for article generation and re- trieval. It can either help automated journalism or defending against machine- seeeartinneeettion articles. However, there is no perfect system which can generate 100% accurate articles. Therefore, it is critical for practitioners to check the fact mentioned in articles and avoid the misinformation brought by failure generation cases. Additionally, someone could use our approach to generate misinformation. However, ENGINE applies the network structure that is same GPT2, which means the discrimi- nators trained for GPT2 articles (e.g., RoBERTa detector (Liu et al., 2019)) are also effective to dis- criminate ENGINE-gennrrted articles. Our paper helps to highlight the need for building tools like the RoBERTa detector.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018 Hicrarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 889-898.
Yi Fung. Christopher Thomas, Revanth Gangi Reddy, Sandeep Polisetty. Heng Ji, Shih-Fu Chang. Kath- leen McKeown, Mohit Bansal, and Avirup Sil. 2021. Infosurgeon: Cross-media fine- aen-igrieed information consistency checking for fake news detection. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1683-1698.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing. Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite. Noa Nabeshima, al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101 200100027.
Jiuxiang Gu, Jianfei Cai, Shafiq R Joty, Li Niu, and Gang Wang- 2018 Look, imagine and match: Im- proving textual- erative models. In Proceedings of the IEEE ference on computer vision and pattern recognition. pages 7181-7189.
References
cross- modal retrieval with gen-
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Tatsunori B Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1689-1701.
Siddhartha Banerjee and Prasenjit Mitra. 2016. Wiki- write: Generating wikipedia articles automatically. In IJCAI, pages 2740-2746.

---

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv: 1904.99099959
on computer vision and pattern recognition, pages 6087-6096.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedhack. Advances in Neural Information Processing Systems, 35:27730-27744-
Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embed- dings, convolutional neural networks and incremental parsing. To appear, 7(1):41 1-220.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric Xing. 2017. Toward con- trolled generation of text. In International Con- ference on Machine Learning, pages 1587-1596. PMLR.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high- sesh-performance deep learning library. Advances in neural information processing systems, 32:8026- 8037.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. 2020. Proxy anchor loss for deep metric learn- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3238-3247.
Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. 2018. Towards controllable story generation. In Proceedings of the First Workshop on Storytelling, pages 43 49.
Sarah Kreps. R Miles McCain, and Miles Brundage. 2020. All the news that's fit to fabricate: Ai- generated text as a tool of media misinformation. Journal of Experimental Political Science, pages 1- 14.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence machine learning, pages
Leo Leppänen, Myriam Munezero, Mark Granroth- Wilding, and Hannu Toivonen. 2017. Data-driven news generation for automated journalism. In Pro- ceedings of the 10th international conference on nat- ural language generation, pages 188-197.
PMLR.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- standing by generative pre-training.
Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020. A survey on deep learning for named entity recognition. IEFF Transactions on Knowledge and Data Engineering.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188):9
Junnan Li, Dongxu Li. Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre- training for unified vision-languad understanding and generation. International Conference on Ma- chine Learning, pages 12888-122 12900. PMLR.
Arnau Ramisa, Fei Yan, Francese Moreno-Noguer, and Krystian Mikolajezyk. 2017. Breakingnews: Article annotation by image and text processing. IEEE trans- actions on pattern analysis and machine intelligence. 10(5)
Fuxiao Liu, Yinghan Wang. Tianlu Wang, and Vicente Ordonez. 2021. Visualnews: Benchmark and chal- lenges in entity-aware image captioning. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6761-6771.
Florian Schroff. Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEF conference on computer vision and pattern recognition, pages 115-823.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 19Xiv
Christoph Schuhmann, Richard Veneu, Romain Beau- mont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komat- suzaki. 2021. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114
Hyeonseob Nam, Jung- Woo Ha, and Jeonghee Kim. 2017. Dual attention networks for multimodal rea- soning and matching. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 299-307.
Merity Stephen, Xiong Caiming, Bradbury James, and Richard Socher. 2017, Pointer sentinel mixture mod- els. Proceedings of ICLR.
Duy-Kien Nguyen and Takayuki Okatani. 2018 Im- proved fusion of visual and language representations by dense symmetric co-attention for visual question answering. In Proceedings of the IEEE conference
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. Eva-clip: Improved train- ing techniques for clip at scale, arXiv preprint arXiv:2303. 15389.

---

Reuben Tan, Bryan A. Plummer, and Kate Saenko. 2020. Detecting cross cross-modee modal inconsistency to de- fend against neural fake news. In Empirical Methods Natural Language Processing (EMNLP).
A.2 Article Quality Annotation
Table reports human accuracy identifying arti- cles as machine or human -uman--enerated to supplement the main paper. We see that ENGINE-XL is able to generate more realistic-looking news articles on Goodnews, consistent with the conclusion in main paper.
Reuben Tan, Bryan A Plummer, Kate Saenko, JP Lewis, Avneesh Sud, and Thomas Leung. 2022. Newssto- ries: Musttrtiing articles with visual summaries. In European Conference on Computer Vision, pages 661. Springer.
Human-based detector
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv preprint 23x::330..13771
76.4% GPT2-1.5B (Finetuned) 73.6%
GROVER-Meze -Mea
ENGINE-XL
70.4%
Table 6: The performance of AMT workers at correctly identifying an article as human or machine generated on GoodNews.
Alasdair Tran, Alexander Mathews, and Lexing Xie. 2020. Transform and tell: Entity-aware news image captioning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition, pages 13035-13045.
A.3 Ablation Study on Top-k Named Entities
We provide the ablation study on top-k named enti- ties in Table 7. We see that the model achieves the best performance when & is set to 15. When k is greater than 10, the improvement limited.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: Billion Parameter Autoregressive Lan- guage Model. https:/ //gi thub. hotubbb.om/kinoflllz mesh-transffrmer--3
top-k named entities PPL
5
10
15
20
Bokun Wang, Yang Yang. Xing Xu, Alan Hanjalic, and Heng Tao Shen. 2017. Adversarial cross- modal re- trieval. In Proceedings of the 25th ACM international conference on Multimedia, pages 154-162.
15.5
14.8
14.5
14.6
Table 7: Ablation study on the number of named entities detected by CLIP (GoodNews).
Thomas Wolf, Julien Chaumond, Lysandre Debut, Vic- tor Sanh, Clement Delangue, Anthony Moi, Pier- ric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: Sfate-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations, pages 38 45.
A.4 Recall of CLIP-detected Named Entities
We use Oracle NE as ground truth labels to evalu- ate the retrieval results by CLIP model. Table 8, we observe that approximately 30% named entities from Oracle NE have been retrieved by CLIP. In contrast, recall of Cap on GoodNews and Visual- News are 22.57% and 7.41% % respectively. The gap between GoodNews and VisualNews is likely be- cause captions in GoodNews often much longer than captions in VisualNews. The retrieval results validate that ClipNE contains more relevant named entities compared to named entities extracted solely based on captions.
Vikas Yadav and Steven Bethard. 2018. survey cent advances in named entity recognition from deep learning models. In Proceedings of the 27th Inter- national Conference on Computational I Linguistics, pages 2145-2158.
Rowan Zellers, Ari Holtzman. Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2020. Defending against neural fake news. Neurips.
A Additional Experiment Results
Named Entities
GoodNews
VisualNews
A.1 Article Generation
Cap ClipNE
22.57
7.41
29.84
31.19
Additional qualitative results on article generation are provided in Figure 6, 8. 9, 10, 11, 12 as the supplement of our main paper, demonstrating that ENGINE is able to generate articles given different news sources.
Table 8: Recall of Cap, ClipNE on GoodNews and VisualNews. Cap represents named entities that appear in captions, ClipNE represent named entities detected by CLIP model.

---

A.5 Ablation Study on the Canonical Order The ablation study of varying inference order is shown in Table 9. From the Table, we see that canonical orders which are not consistent with the training order result in greater PPL of the language model. The model achieves the best performance when the inference order is aligned with the train- ing order.
(nucleus sampling) method (Holtzman al., 2019) as our decoding strategy.
A.8 Article Quality Annotation Templates
Following (Tan et al., 2020), annotators are asked indicate a reason for whether the articles are human- written or manihle-manipulatte. We provide a view of the AMT worker interface in Figure 13.
canonical order
GoodNews PPL
18.2
title-date tile-datt-domain domain-summary summary
19.2
dottmaary--dtt--mai-------- demmiin-ate-tille--iine summary
20.5
17.3
Table 9: Ablation study on the canonical order during inference (GoodNews).
A.6 Ablation Study on Article Retrieval
We ablate the two modules of method ticle retrieval in Table 10. Text-only denotes that we directly use the original articles as the input. NE denotes named entity extraction, and EA de- notes the entity-aware mechanism. From the table, we observe that both named entity extraction and the entity-awar mechanism can boost the article retrieval performance.
Image-to-article R@I
Article-to-image R@5
Method
R@5 R@ 10
@@1
@@ 10
Text-only
57.9
78.6
85.2
54.8
77.3
82.7
NE
60.0
80.3
85.9
56.1
78.2
83.7
NE-EA
61.9
82.0
86.7
56.3
78.3
83.9
Table 10: Ablation study on article retrieval (Visual- News). Both named entity extraction and the entity- aware mechanism can improve article retrieval per- formance VisualNews.
A.7 Decoding Strategy for Article Generation
likelihood-maximiin decoding strategies like greedy search or beam search work well in close- ended generation such as image captions, machine translation, or summarization. However, these methods suffer from the repetitive text problem in open-ended generations like dialog story gen- eration (Hashimoto al., 2019; Holtzman al., 2019). Sampling methods (Fan al., 2018; Holtz- man et al., 2019) are therefore proposed to intro- duce more randomness and surprise to text gener- ation. our work, we adopt the top-p sampling

---

Title: Daniel Agger leaves Liverpool return to former club Brondby Domain: www.thqqurddaan.com Date: 08-30-2014
Daniel Agger has rejoined Brondhy for undisclosed fee.
I Human-wrirten News Article
ENGiNE-eenerated News Article
Liverpool have announced the sale of the defender Daniel Agger to the Danish club Brondby for an undisclosed fee. The ye-yearoold Denmark international joined the Anfield club from Brondby in 2006 and went on to make more than 200 appearances for the club but had fällen down the pecking order under Brendan Rodgers.
The Liverpool defender Daniel Agger has left Anfield to return his previous club Brondby, The Danish international midfielder was reported to have agreed personal terms with the club. The 29 year--or left Liverpool in 2006 following their releation from the Premier League and hus featured for the national wam in Denmark's recent scasons.
"Liverpool have been such a big part my life and my family's s lives for so long, that leaving is extremely difficult," Agger said. "The opportunity arose for me to return home to Brondby and this stage my career like right decision for wouldn't leave here go anywhere else and that has been scasons I have turned down many offers move other Premier League and Europcan clubs. 'I would like thank the Liverpool supporters for the incredible backing they have given me in my time here and the warmth and generosity of spirit displayed my family. "This has been home and somewhere we have been very happy and settled. However, more recently have started to feel the effects of playing regularly in league that is as physically demanding as the Premier League is, and I would not want to stay if I did not think that I was able to meet that challenge week in, week our.'
The move to Brondby was said to be cash plus deal. We're very pleased Daniel Agger has chosen to continue his development Brondby, read statement on club website. He's young man who shown great potential the Premier Leguu, and we look forward working with him again." Brondby, who Danish Superliga, luve squad of more than 200 players. The Liverpool manager Brendan Rodgers said As footballer you dream of playing in the Premier League and I'm very pleased that Daniel Agger has chosen come back Brondby. He's a good defender and I'm sure he will work very hard prove the club that he is good player. We like to think of him important player for his nationality. But he hus great character. You might think he's bit of a character but you 'll see he's as strong character as any of the other players here. Agger said: 'I am eateemely proud to play for Brondby. I will always consider it as one of the happiest days of my career so far, It has been a great and very memorable time, and the supporters have been fantastic. My futuure lies with the club, and hopefully, it will be here long time. also want to thank the Liverpool fans for their support. Without them I wouldn't have been able to carry on fighting for this shirt, so thank you!"
proven by my
very
Figure 6: Example of human-written and ENGINE- news articles from Guardian.

---

Title: State of Britain's is wildlife 'increasingly fragile'
Domain: www.bbc.com
Date: 12-08-2015
Dark green fritillary butterfly
Human-written News Article
ENGINE-generated News Artick
incressingly fraglle stare, with animals carrying our vical johs for farmers being lost more rapidly than others, say scientiors that pollinate crops or fight pests are at risk disapeeringg putting food production in jeopardy, according the team. The he research beings together millions of wildlife records spanning 40 years It suggests conservation efforts should focus on cereain areas, they add. Tom Oliver of the University of Reading, biggest
Brizain's wildlife
The of Britain's wildlife has become incressingly fragile due loss of habitat and changes in the climate, a study has suggested. Mass bird and burterfly losses have increased along with climate change, scientists found, which means many species are not as well equipped to cope. And while changes in food distribution, such floods, may cause short- problems, they can make long term problems worse. Drought and flood events could put millions of birds and insects at risk, the scientists concluded. Monitoring of more than 4,000 specits of birds, butterflies and moths across Fngland, Wales and Scotland for 40 years conchuded that the UK's biodiversity was still very good - only smaller than it was in 1975.But the biological communities in which they live were now less diverse than when these earlier surveys took place, which meant their ability to recover was less certain, they added. "We have lost several species of already endangered species, eexpliined lead researcher Dr Tom Oliver, naturalist University Reading. 'And we are moving into many of these habitats that are becoming drier and thus less suitable for them. "The results of the study were far more alarming" than expected. 'The situation looks pretty dire," said Dr James Bullock from the Centre for Ecology and Hydrology at Oxfordshire, who was not involved with the research. He told BBC News that Britain's wildlife was already extremely stressed. Our natural cnvironment is suffering from climate change, which is melting the ice sheets and expanding the occans, and from human- generated pollution, polluted environment in the future. And the species we are losing are particularly imporcant because when you lose specles, you do so in numbers, they cesse to exist in numbers. 'The loss a single species Britain alone has the potential affect millions of other organisms." "The impact of the findings is likely to be felt most in many communities across the British Isles, "We have lost a large number of species, Dr Oliver added. It's going to have big impact on the ecology of Beitain." * He explained that the risk of long term damage lightly We are more concerned ahout short term impacts. "The researcbers believe that much work needs to be done to understand how, in time, this situation might change. Research in Nature Communications suggests that climate change means there has been sharp decline in the number of burterfly specles sported in northern England down by a third berween 1970 and 2009.Species such as the large blue have struggled for many generations the cold weather has made it too cold to overwinter in the UK Follow Helen on Twitter.
led the research, said was comprehensive report ever assembled for any country in the world. "yy standardising records from an amyy of amareur blologists across the country, we have amassed an impressive array of data, giving us most complete picture yet of the state of Britain's wildlife," he told BBC News. "The picture that emerges is of an increasingly fragile system. particulariy in species that do vital jobs for humans. "Unless efforts are made reverse some of these declines, we face a future where will be less confident that effectively grow our BBenefis from nature.
most
The researchers looked records of the changing fortunes of more than 4,000 types of plant and animal living in England, Wales and Scotland between 1970 and Specis such as bees, dragonflies, grasshoppers, ladybirds, plants mosses were put into groups based the role they play nature. Groups providing pollination and pest control benefits had undergone declines while those involved in functions such as decay or mopping up carbon emissions were more stable.Plants and animals regarded of cultural importance humans, such birds, butterflies and hedgehogs. also fared badly.Prof James Ballock, of the Centre for Ecology and Hydrology in Oxfordshire, a co co-researcher on the study, said conservation efforts should strong evidence of declining "Conservtttoooooiooonnn actions, such wildlife friendly farming, can avoid the loss of biodiversity and the resulting crosion of the pollination, pest control and other benefits derive from nature," he sald.And Dr Oliver said there number of messures that individuals could take to help to improve the future for wildlife in the Beitish Thee include: Making gardens wildlife friendly Purchasing food grown a sustainable way to wildlife Helping collect data wildlife. The study published in the journal Nature Communications.
we going to have much more
on the areas
there was
unlikely be taken
Figure 7: Example of human--witter and -eeneraee news articles from BBC.

---

Title: Robberies plague Tenleytown area over the past few weeks Domain: www.washingtenppst.com Date: 05-23-2016
Video shows suspects robbing attacking Georgetown Day student
News Article
ENGINE-generate News Article
student at Georgetown Day School was robbed last month in the same area of the District where a student at another school was stabbed during a robbery attempt, and near where two other robberies also occurred last month.The student at Georgetown Day was robbed about 1:30 p.m. April in the 4200 block of Davenport Street near the private school. Three robbers approached, and began beating the victim, police said, although they did not identify him student.They went through his pants pocket and took his glasses, wallet and phone before flecing foot, police A spokeswoman for Georgetown Day issued a statement saying that student there was robbed along 42nd Street in Tenleytown on April 27, and the school provided immediate medical and security support. statement said the school has held an assembly devoted to neighborhood safety training and ir has adjusted the patrols of school security officers. hat robbery occurred two days after an April 25 incident which authorities said student at Woodrow Wilson High School was stabbed in leg during an attempted robbery near the In the Wilson incident, the robber was described as man his 30s, No description was provided of the robbers Davenport Street NW. Police released video Monday of three people described "persons of interest. "The video showed the victim and the robbers approaching each other a sidewalk, with school buses lined up a curh, Suddenly the student is knocked down and one or two of the robbers appear to go through his pockets.But But the video does not appear to make possible a detailed description of the robbers,The stabbing comes nearly week after two robberies, one of them involving a student vietim, were reported on streets near Wilson and near Georgetown Day. Both occurred within about 15 minutes of each other April 19. In each instance, three people, one of whom who was armed with knife, approached female victims and demanded their cellphones and debit cards. juvenile male has been arrested in those cases, police said.
A group that robbed Georgetown Day School of valuables including cash last month and a purse while it was locked up in dormitory has also robbed spate of Georgetown Day students, police said. The initial thefts occurred April 27 and April 27. Two men entered the school, stealing the wallets and purses of two female students, police suid. A total of four were relcased from custody. The school said the total The last robbery occurred about 1:30 p.m. April 27 near the store 4200 Davenport Street. Three male suspects fled the school on foot, and local resident engaged in an effort to help the robbers, police said. It rook about 15 minutes before the local man was able to locate them. The three defendants - one in his 20s, one in his late 20s and one in his late teens - fled along Davenport Street NW and into a residential neighhorhood, police said, That resident was identified as friend of the robbers assisted in helping apprehend them. The robbery at Georgetown Day came two days after a robbery at Woodrow Wilson High School Tenleytown. That April robbery took place near 42nd Street and Davenport Street NW, mujor thoroughfare in Tenleytown. According police, their 20s robbed woman in her 30s. The robbers fled with purse containing cash that in pink leather wallet. On Monday, the Wilson robbery went unsolved for nearly week. Staff at the school said no incidents had been reported, "We do have a staff member watching the campus. You go through the normal procedures of setting up alarms, said Tenkeytown police spokeswoman. We were n't able to get that information from the yet. The person you are talking to is someone within security. You have to keep an eye on him. 'Correction This story has been updated to reflect that the Davenport Street robbery occurred April 19, not April
$
in
Figure 8: Example of human- and ENGINE--enerated news articles from Washington Post.

---

Title: Courts again rule against streaming upstart Aerco Domain: www.ustodday.cm Date: 08-21-2014
In this photo illustration Acrcocom Web service that provides television shows online is shown on an iPhone 4$ April 22 2014 in New York City
Human-wrirten News Article
ENGiNE-generated News Article
federal court will not hear Aereo's argument that the streaming media company should allowed to operate similarly to cable company:The Second Circuit Court Appeals ruled Thursday that Acreo will have take its argument the district court, according court documents first eeported by The Washington Post.Ths decision continuses broadeasters' winning streak against the upstart service. Launched two years ago, Acreo delivered over-the air channels via online streaming to computers and tablets for about $8 monch and had brought the service to 11 chties.The Supreme Court ruled two months ago that Acreo was violating copyright law by not paying the networks for transmitting their signals.After that, the streaming media company voluntarily shut down and told the U.S. District Court that wanted to be cable TV system and apply for compulsory license stream live TV broddasts.uubeeuenny the Copyright Office's general counsel said that the office did nor think that Aereo would qualify, bur fell short of denying Acreo's filing because the issue remuined alive in the courts. colrts.Follow Mike Snider on Twitter: @ MikeSnider
The high court delivered yet another blow to Acreo a puir unanimous court opinions Thursday declaring the popular streaming service illegally alters contents of customens' cable subscriptions and guilty of violating copyright law. The Second Circuic Court of Appeals, which first ruled against Aereo years ago, urged the justices to reject its claim that it simply device for consumers access live TV without paying for it. Acreo lets consumers use digital antenna to receive live broadcasts online and streams those broadcasts as they are being transaittedd over the Internet, just like cable Tr.Thouggh those transmissions the consumer also becomes a subseriber to the over the air service. But even they keep Aereo's service on, only portion of the amount puid for cable service remains. For example, nearly all the TV shows they request, regardless of what shows they watch, For about $8 each the consumer loses the signals they might already have
16.95 per month Aereo customers receive
treated the equivalent
the Internet. Thus the company is violating copyright law by redistributing the service users already pay for withour paying for the service they are receiving from cable operators. "The Court finds that Acreo's business model violates this section of the Copyright Act, 11 judges wrote in the majority opinion. The Washington Post, which first reported on the decision, called it a blow to digital media and to consumers. ""This unanimous decision means lot, Acrco chief executive Mike Snider said in statement. Consumers longer have to pay to receive the traditional way they watch TV. They now have the option pay only service that delivers those channels, all for free. "Aerco has petitioned the court change the district court's ruling the full appeals court, which is likely to do so. The Supreme Court upheld the same ruling from two months ago. Acreo said it will appeal to the U.S. District Court for the District of Columbia. Experts tell the Post that its previous case shows that the Copyright Office's determination to give consumers access to streaming services without paying retailed companies is likely to be reversed by the appeals court. Follow @jswartz on Twitter
Figure 9: Example of human-written and ENGINE-generated news articles from USA Today.

---

Tide: James Holland, Trailblazing Cancer Researcher, Dies at 92 Domain: www.ny com 03-27-2018
Dr. Holland with his wife, Dr. Jimmie Holland, who pionecred the field paycho-oncology. While she treated the emotional distress cancer patients, focused on discovering universal remedics. She died December.
Dr. James Holland in undated phorograph. In the 1950s, he and his collesgues were considered research cowboys by some colleagues, he said.
Human-written News Article
ENGiNE-generated News Article
James E. Holland, an innovative oncologist whose scientific insights made him, as he once said, Google for cancer research, died on Thursday at home in Scarsdale, N.Y. He was 92.
James F. Holland, a founding father chemotherapy who helped pioneer a lifesaving drug treatment for podiatric leukemia patients, died Thursday N.Y. 92,
Scarsdale,
The cause widespread mycloid leukemia subtype, of the blood cells bone marrow that in fewer than in three patients.
The respicatory failure, his son Steven said.
Dr. Holland and his medical collaborators in the 1950s were scorned by some colleagues as research "cowbops" as he pur it, because they audaciously tackled what was considered an incurable disease simultaneously, instead of sequentially.
Mr. Holland was one of the few oncologists to propose, in the 1950s, the chemical treatment known as stereosporin hence the name. was also among the first suggest though not actually carry out became known as precancer therapy turning over nonmedical patients their regular doctors, while supplementing their therapies with medications to kill therapy. it scemed some, would have rendered it toxic.
to treat it with several chemicals
cells. The mere suggestion of precaneer
When their research began, fewer than in three children who were found to have acute lymphoblastic leukemia lived as long as year. Today, as many as nine in 10 survive the disease, in which the bone marrow makes too many immature white blood cells.
Ar the time, though, Mr. Holland thought it should not. He and his colleagues had been working for long as a year on drugs to prevent acute lymphoblastic leukemia in patients with advanced discase, in which cells on the surface of the blood can proliferate out of control being overexposed
Another clinical that Holland directed 1970s, contain acute mycloid leukemia, somewhat similar malfunction in blood cell production, resulted what became the standard tecatment.
chemotherapy. Doctors usmally this only after the patient's health deteriorated. Today, many hospitals will not perform precancer therapy, either.
But in the 1970s, he and his colleagues cowboys by their colleagues. Known for treating as many as nine of 10 cancer cases, they had been derided even by their own specialty, nuclear medicine, for treating these patients when thee science does not justify it, Dr. Steven Holland, his brother and his only sibling, said.
regarded research
Figure 10: Example of human-written and ENGINE-generated news articles from New York Times.

---

Robert Boulter
Human-written Wikipedia
ENGINE-gencrated Wikipedia
Robert Boulter an English film, television and theatre actor: He had a guest e-cc starring role on the television scries The Bill in 2000. This was followed by starring role the play Herons written by Simon Stephens, which was performed in 2001 at the Royal Court Theatre. He had a guest role in the television series Judge Deed in 2002. In 2004 Boulter landed a role as 1" "Caig'' 1 in the episode V 'Ieddy's s Story 1" of the television series The Long Firm: he starred alongside actors Mark Strong and Derek Jacobi, He was
Robert Boulter is an English film, relevision and theatre actor, He had guest g-@ starring role on the television series The Bill in 2000. This followed by starring roke the play Herons. He has also starred on TV teacher and actor. In 2004 Boulter starred in the stage play Waste in Plymouth. He made his film debut at the Festival theatre in London in the 2005 John Sabatalain adaption of the John Grant- written novel, Intimute Behaviour. In 2006 he won Samuel Bockett award for Best Performance his play Teddy's Story. In 2007 Boulter played the lead role in the adaptation of Patrick Burden's 's book, East of liden. In 2008 he was involved with Dominic who were writer of another film, the comedy thriller, Derek Jacobl And The Mercenaries, and the TV adaptation of the Hor Fuzz movies, In May 2008 be made his film acting debut in the film Fever. Since then be has been sought out for appeating films and TV series including Mercury Fur, which he appeared Hammersmich. where he played Ben Whishaw and London's Bush Theatre, where he appeared alongside Shane Zaza in Man of Economic Mass. In November 2008 he returned to the stage, in West End production of This Side Paradise and the West End production Uncil Goes. He is currently appearing the Shakespeare theatre Fulham with revival of Gods and Heroes. Robert Houlver biography Being called a fusion of two performers has never been embarrasing in Boulter's theatre career. In taking the beat, gruff character Toddy that readers may instantly recognise Harry Kent he defied expectations it that had seen him move towards cheerfulness, his name becoming subely attached to his image. He has also made the perfect transition from film to stage, propelling himself between brutal, seriously unpleasant characters screen and harsh, adoring protagonists with dignity and aplomb in real life. Boulter lives with his family in Burn, Devon His son Fraser Ayres is member of the team behind the websire jeffersonsworld.comm been character
in the 2005 theatre productions of the Philip Ridley play Mercury Fur, which was performed at the Drum Theatre in Plymouth and the Menier Chocolate Factory in London. He was directed by John Tiffany and starred alongside Ben Whishaw, Shane Zaza, Harry Kent, Fraser Ayres, Sophie Scanton and Dominic Hall. lakn In 2006, Boulter starred alongside Whishaw in the play Citizenship written by Mark Ravenhill. He appeared on a 2006 episode of the television series, Doctors, followed by role in the 2007 theatre production of How to Curse directed by Josie Rourke. How to Curse performed Bush Theatre in the London Borough of Hammersmith and Fulham. Boulter starred in two films 2008, Daylight Robbery by filmmaker Leonti, and Donkey Punch directed by Olly Blackbur,, May 2008, Boulter made guest appearance of the relevision series Waking the Dead, followed by appearance on the television series Survivors in November 2008. He had a recurring role ten episodes of the television series Casualty in 2010, as Kleron Fletcher V. Boulter starred the 2011 film Mercenaries directed by Paris Leonti. innn In 2000 Boulter had guest g--0 starring role on the relevision series The Bill; he portrayed 1 Scott Parry in the episode, 1 In Safe Hands V. Boulter starred as V Scott 1" the play Herons written by Simon Stephens, which was performed 2001 at the Royal Court Theatre. A review of Boulter's performance in The Independent on Sunday described him as V* horribly menacing 1" in the role, and he received critical reviews in The Herald, and Evening Standard. He appeared the relevision series Judge John Deed in 2002 1" Addem Armitage V in the episade Political Espediency V., and bad role different character's V" Toby Steele 1" on The Bill, soin He had recurring role in 2003 on two episodes The Bill, as character's \ Connor Price V.. Tn 2004 Boulter landed role as Craig 1' television series The Long Firm ; he starred alongside actors Mark Strong and Derek Jacobi. Boulter starred as \" Darren V, in the 2005 theatre productions of the Philip Ridley play Mercury Fur.
Television Play for the film version of
and John Deed and Craig Mark Strong,
John Tiffany: ;
two @ part episode
unexpected and welcome return from genre
and his daughter Sophie Stanton in the theatre, Boulter starred alongside Whishaw in the 2010 film Donkey Punch. The other main roke in the film were Mark Ravenhill, who plays a former officer in an augmented forces unit, and Josie Rourke
the play the lead character, \ demon of myth is C, who lives the aretic wasteland and is transformed when he savagely hunted by fave eating gnactus. Boulter played the first half of the story, or lion, while the other balf was The Soldier, a mercenary. V" the curse of mythology V..
cpisode Y Teddy's s Story 1" of the
Figure 11: Example of human- written and ENGINE- generated Wikipedia articles (from WikiText dataset (Stephen et al., 2017)). We generate articles conditioned on both the title Robert Boulter, and the first 50 tokens highlighted by light yellow. The articles are cut to fit the figure size.

---

1933 Treasure Coast hurricanc
Human-writen Wikipedia
ENGINE-generated Wikipedia
The 1933 Treasure Coust hurricane was the second g--29 most intense tropical cyclone to strike the United States during the active 1933 Atlantic hurricane
The 1933 Treasure Coust hurricane was the second 8-@ most intense tropical cyclone to strike United State during the active 1933 Atlantic hurricane season. The eleventh tropical storm, firth hurricane. A Category á cyclone with winds of 100 mph generated winds of 161 km per hour and water temperature of -49 degrees. The storm killed people, flonded an area of 848 sq km, and caused damage. Although it missed the Florida ceast, the system swept the rest of the United Stares less than 24 hours. The hear from the system caused widespread fires and continued to bring heavy rain and wind throughout the south and central regions. south Florida alone, the fires killed two and destroyed 98 homes in the Lesser Antilles and southwest Florida. The Turks and Caicus islands were particularly badly hit because of storm surge that produced high seas and flouding, especially along the coast from Tampa, Georgia, to Puerto Rico. The Turks and Caicos islands experienced over 125 mph winds, winds of 125 & mile per hour, and only cm to inches of rain the 30 minute interval starting around 1400 UTC. The Turks and Calcos islands also experienced an over 140 mph wind shear at 5875 kp m. south Georgia, home to the Simpson and Penally counties, the of the damage felt in the Savannah area, where 800 + homes were damaged destroyed and well over 1000 had to be demolished. The most intense tropical storm to strike the United States during the 1953 Atlantic burrieane seison occurred when Tropical Depression formed over the Bahamas during the winter of 1913. The storm resulted in catastrophic loss of life and destruction. As of 24 km west of the Bahamas, the lowest presure was 3,21 mbs and the storm had maximum sustained winds of 200 km/h, with gusts to over 200 km/h. In south Florida, the intense storm produced winds of 100 mph over south western southwest Florida and south western coast generally berween the edges of the Low Range and the extreme southeast of the storms, In western Cuba, the storms produced winds up to what the Weather Bureau described as about 65 mph \" some spots, all tropical storms Category
The eleventh tropical storm, fifth burricane. and the third major hurricane of the season, it formed east 0-@ northeast of the Leeward Islands on Auguse 31. The tropical storm moved rapidly west @-@ northwestward, steadily intensifylng to a hurricane. It acquired peak winds of 140 miles per hour ( 225 km /h ) and passed over portions of the Bahamas on Seprember 3, including Eleuthera and Harbour Island, causing severe damage to crops. buildings, and infrastructure. Winds over 100 mph 161 km / } affected many islands its path, especially those that encountered center, and many wharves were ruined. inin Subuquently, weakened and made landfall at Jupiter, Florida, early on September e with winds of 125 mph ( 201 km / hb) ). The hurricane moved across the state, passing near Tamps before moving into Georgia discipating. In Florida, the strong winds of the cyclone blew buildings off their foundations, and numerous trees were prostrated in citrus groves. The Tressure Coust reglon received the most extensive destriction, and Stuart, Jupiter, and Fort Pierce were heavily damaged. Inland, the cyclone weskened rapidly bur produced prodigions amounts of rain, causing a dam collapse near Tampa. The storm caused 3 million in damage ( 1933 USD) ) after damaging or destroying 6 eve 848 homes, lakn Unusually, the storm hit Florida less than 24 hours before another major hurricane bearing
million
a-W mile e-0 @ per a- hour 201 km/ h winds struck South Texa; never have major cyclones hit the United States in such close sucvession lnin The origins of the hurricane were from tropical ware that possibly spawned a tropical depression on August 27, although there was minimal data over the next few days it tracked to the west 3 northwest. On August 31, nearby ship reported gale force winds, which indicated that a tropical storm had developed to the east @-@ northesst of the Lesser Antilles, Based on continuity, it estimated the storm attained hurricane status later that day, Moving quickly to the west g-g northwest, the storm passed north of the Lesser Antilles and Puetto Rico. Early on September 2, a ship called the Gulfwing reported burometric pressure of 978 mbar 28 e-e 88 inHg ), which confirmed that the storm attained hurricane status, After passing north the Turks and Caicos island, the hurricane struck Eleuthera and Harbour Island in the Bahamas on September 3, the latter at 1100 UTC, A station on the latter island reported pressure of 27 aue 20 inHig 945 mb) ) during the 30 minute passage the eye.
and respectively, were downgraded to tropical storm or depression. Texas though was bit by Hurricanc Norbert in carly October 1913. At the time, the Weather Bureau reported that the storm produzed winds 800 900 km/h, and a storm tide of m. The storm passed over northeast Texas between midnight and dawn, and was the first mujor tropical cyclone impact the United States during the winter of 1913. Norbert resulted in 848 lives being lost and 98 m in damages, amongst other catastrophic damage, as well numerous fatalities.
Figure 12: Example of human- and ENGeNN-generated Wikipedia articles about 1993 Treasure Coast hurricane.

---

greate
Instructions: News Articles Evaluation
Hello, trying determine susceptible people machine generated
study, you asked discrininate between human
news articles.
You will be ansigned 5 article samples. Each article sample contains the headline, article body and imago/caption pairs. The images and captions are located after the articee body where each caption situated below its corresponding image. articia.
You should spend
than 1.5 minutes
article. Please note that we re-rendered all the article samples. As result, the primitive display NOT indicator whether the article humann or machine generated.
Instructions
Shartoutt
noux anticle human machine genwnive?) Pleane chooee the option think Iable artiii.
Select an option
Human- Generate
Duterte, Philippine President, Boasts He Killed Someone as a Teenager
Maachne-GGneratee because
more images is not the article body
Machine-Generalad because
captions very relevant to the article body
MANILA
The deach ot Chiet Justice Sereno Alon Casoelan IIt, shot his mont gate when he was 13, was the most recent in what mounting tall police killnne the Htllopimess.
Machine-Geneen Sechine-Geeerated one more images not very relevant to the caption
But Duterte is beginning to openyy talk about his personal tronsgressions, something that rarely come up before.
Since at
adminuaration June, maunifynn closa. al journallata acalleitc eve yore
Machine-Generated Generated because the image/caption pairs are not relevent to the article body
Machine-
jourclistss
Machine Machine-Generatee Generated because somethiin about the article appears
tn videow brosdced on lelevision and manitcal, contennishly intimidtring stare toward the caman The gong lender Lech Chartes Letfo convicted drug dealer, was staobed to ceath by Mi Duterte on the yoounds of a pro provicil court in 2016, the next year. Mr. Duterte best a man to death with a weodan rod crowded Haniaa public market during an ant drua operation.
Dutente tas projected
Machinn-Generated the article relavent to the article
Machine. Generated because really
Submit
Figure 13: The interface used by AMT workers in article quality annotation experiment.


Reviews:
[{"review_id": "ctclwXV4Hu", "reviewer": "Reviewer_5dtH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments, we appreciate their time and will use their suggestions to improve our paper. \n\n> Human conducted studies do not correlate well with the use-case of this paper. \n> Not clear about the human study for this use-case on asking humans to distinguish machine-generated articles from human-written articles. It makes sense in the case of fake news. It will be good to add more motivation.\n\nIn the article generation task, the ability of humans to distinguish between the two sources is an important measurement, since generative language models are trained to replicate the distribution of human-written content. This is consistent with the arguments made in prior work, such as GPT-3 [1] and GROVER [2], who performed similar experiments to evaluate the quality of generated articles (*e.g.*, Table 7.3 in GPT-3 [1], Figure 4 in GROVER [2]).  \n\n[1] Brown, Tom, et al. \"Language models are few-shot learners.\" NeurIPS (2020).  \n[2] Zellers, Rowan, et al. \"Defending against neural fake news.\" NeurIPS (2019).  \n\n> Unclear about the precision of named entities acquired from the articles as the dataset lacks ground truth information.\n> Since an external tool (SpaCy) is used for NER, how do we know the precision and accuracy of annotation? Also, the dataset itself does not have any ground truth to support it.\n\nWe agree that spaCy [3] is not a perfect predictor of named entities from articles. However, by leveraging the detected named entities, we still observe a perplexity improvement of our language models. The improvement further validates that our proposed model is effective, even in condition that the named entities are not the ground truth labels and may contain noise. In summary, using detected named entities with noise, our method already outperforms the baselines. Our experiments with oracle named entities also show that we can expect greater improvements as NER methods become more accurate or the list of entities is provided as input.\n\n[3] Honnibal, M., Montani, I., Van Landeghem, S., & Boyd, A. (2020). spaCy: Industrial-strength Natural Language Processing in Python.\n\n> Assumption of CLIP trained with a large dataset contains named entities. However, this has not been quantified.\n\nThe training data of CLIP [4] is a dataset of 400 million image-text pairs collected from the Internet. The text data in the Internet inherently contains a great number of named entities. Since OpenAI does not release their training data of CLIP, we can check another image-text pairs dataset which are also collected from the Internet, LAION-400M [5]. Upon simply reviewing the HuggingFace tutorial of LAION-400M (search for laion-400M on the HuggingFace datasets page), we see that the captions of LAION-400M contain a lot of named entities. \n\nTo validate this assumption quantitatively, we provide retrieval experiments in Appendix A.4, Table 8.  We use Oracle NE as ground truth labels to evaluate the retrieval results by CLIP model. From the Table, we observe that approximately 30% named entities from Oracle NE have been retrieved by CLIP. Compared to named entities that appear in captions, ClipNE improves the recall by 7 points on GoodNews and 24 points on VisualNews. Below we recreate Table 8 related to this experiment for your convenience.\n\n**Recall of Cap, ClipNE on GoodNews and VisualNews.**\n|  Named Entities  | GoodNews | VisualNews |\n|----------- |:--------:| :--------:|\n| Cap |  22.57 | 7.41 | \n| ClipNE | 29.84 | 31.19 |\n\n[4] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. PMLR, 2021.  \n[5] Schuhmann, Christoph, et al. \"Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.\" arXiv (2021).\n\n> Do not understand this representation in Abstract: \"4-5 perplexity gain\". Can it be more explicit like a single number?\n\nThe improvement varies with the scale of our model. For example, ENGINE-Base improves around 5.3 perplexity points compared to finetuned GPT-124M, while ENGINE-XL improves around 3.9 perplexity points compared to finetuned GPT-1.5B. Therefore, this defines a range of a 4-5 perplexity improvement in the Abstract. We shall clarify this in the camera ready.\n\n> For the image, can't we get the meta-data information from the news articles?\n\nIn the article generation task, the objective of language models is to produce articles based on human-written prompts. These articles are primarily considered as the model's output. Thus, we do not assume that the news articles are available as part of the input and, therefore, the meta-data *cannot* be obtained from the news articles during inference. \n\n\n> Using Visual NER can be noisy with CLIP, as it is only passed with Visual Information and not multimodal.\n\nWe would like to begin by noting that our primary contribution is introducing a new entity-aware mechanism that takes advantage of NERs more effectively than the entity-aware mechanisms proposed by prior work (reported in Table 2b of our paper).\n\nHowever, when demonstrating what situations are useful, we did evaluate the effectiveness of using automatically extracted NERs, such as the Visual NERs.  Nearly all detector outputs can introduce some noise, as can many datasets due to label noise. However, the key question is whether these provide a benefit that is greater than that of the noise they introduce.  As demonstrated in our experiments, using Visual NER boosts performance, which means that they are effective enough to be beneficial. Specifically, our ablation study in Figure 3 shows that the CLIP-detected named entities improve our model by around 1 perplexity points on both GoodNews and VisualNews datasets (GoodNews:15.9 vs. 14.8, VisualNews: 17.1 vs. 16.1).\n\nFinally, since we use an off-the-shelf image-text model, this suggests that as these models (or others) become more accurate at identifying NERs, our proposed entity aware mechanism will simply become more effective in these settings.\n\n\n> In section-3.2, it is not clear how CLIP predicts the similarity between the article images and the candidate entities. As, named entities are just phrases with maximum length of 2 or 3 words. However, clip training was performed on image-text pairs that are sentences.\n\nIn the CLIP paper, the authors performed zero-shot experiments on ImageNet using sentences like \"a photo of <object category>,\" demonstrating one way to adapt CLIP to a classification task with a short (often single word) label. We also note that in the official code repository of CLIP, the authors also use short phrases such as \"a diagram\", \"a dog“, and \"a cat\" as the language examples.  This suggests that CLIP should work well for very short phrases as well as longer prompts created using templates.  For our task we found these different methods of using CLIP for short phrases made negligible differences in practice.  Specifically, we followed the same experimental setup in Appendix A.4, Table 8, where we evaluate how many of the oracle named entities are extracted either using CLIP or from the image captions.  We found that over 500 randomly selected samples, the recall of using the template “a photo of <named entity>” was 30.254, whereas using the named entity alone obtained a recall of 30.253, demonstrating that either approach works equally well (in our case, we used the named entity directly).\n\n> Does changing the canonical order impact the results?\n\nWe observe that our model performs at its best when the inference order aligns with the order used during training. For example, during training, we set the order as \"domain-date-title-summary\". The ablation study of varying inference order is shown in the following table.\n\n**Ablation study on the canonical order during inference (GoodNews).**\n|   canonical order   | GoodNews PPL |\n|----------- |:--------:|\n| date-domain-title-summary | 18.2 |\n| title-date-domain-summary | 19.2 |\n| summary-date-domain-title | 20.5 |\n| domain-date-title-summary | **17.3** |\n\nFrom the Table, we see that canonical orders which are not consistent with the training order result in greater PPL of the language model. The model achieves the best performance when the inference order is aligned with the training order.\n\n\n> Ablation study lack a study on how jointly modeling entity names and their categories is boosting the performance of article generation and retrieval. Aforementioned statement was stated as the benefits of proposed approach.\n\nWe performed ablation studies on jointly modeling entity names and their categories in Figure 3 (*Cap* vs. *Cap-EA*). We see that our entity-aware mechanism improves the perplexity by approximately 0.6 points (16.6 vs. 15.9 on GoodNews and 17.7 vs. 17.1 on VisualNews). To supplement this comparison, we also finetune GPT-2 with our entity-aware mechanism (*Text-EA*). We find that compared to *Text-only*, *Text-EA* achieves 0.4 and 0.5 perplexity improvements on GoodNews and VisualNews, respectively. The additional comparison further validates the effectiveness of our proposed entity–aware mechanism.\n\n\n> Implementation details for reproducibility:\n\nWe implemented our models mainly based on Pytorch [3] and Transformer [4] libraries. \nThe maximum sequence length of language models is set to 1024. For ENGINE-Base and ENGINE-Medium, we used a batch size of 8 and a maximum learning rate of $10^{-4}$. For ENGINE-XL, we used a batch size of 4 to fit the GPU memory. Correspondingly, the maximum learning rate is set to $2^{0.5}\\times10^{-4}$. We trained our models around 3 epochs with 0.06 epoch for linear warm-up on both datasets. We parallelized ENGINE-XL on 4 NVIDIA RTX-A6000s and ENGINE-Medium on 2 NVIDIA RTX-A6000s. **We will also release our code to ensure reproducibility in the camera ready**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Adds the entity category name beside the recognized named entity for the proposed approach.\n2. Extensive comparison of results with other language models and closely related tasks.", "reasons_to_reject": "1. Human conducted study do not correlate well with the use-case of this paper.\n2. Unclear about the precision of named entities acquired from the articles as the dataset lack ground truth information.\n3. Assumption of CLIP trained with a large dataset contains named entities. However, this has not be quantified.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, authors propose a an entity-aware article generation and retrieval approach (ENGINE) for incorporation of named entities into language models. Experiments are conducted on three different datasets mainly GoodNews, VisualNews and a zero-shot article generation on WikiText showcase the effectiveness of the proposed approach against several other strong language model baselines on metrics such as Perplexity and Recall@K.", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Do not understand this representation in Abstract: \"4-5 perplexity gain\". Can it be more explicit like a single number.\n\n2. For the image, can't we get the meta-data information from the news articles?\n\n3. Using Visual NER can be noisy with CLIP, as it is only passed with Visual Information and not multimodal.\n\n4. Since an external tool (SpaCy) is used for NER, how do we know the precision and accuracy of annotation? Also, the dataset itself do not have any ground truth to support it.\n\n5. In section-3.2, it is not clear how CLIP predicts the similarity between the article images and the candidate entities. As, named entities are just phrases with maximum length of 2 or 3 words. However, clip training was performed on image-text pairs that are sentences.\n\n6. Does changing the canonical order impact the results?\n\n7. Ablation study lack a study on how jointly modeling entity names and their categories is boosting the performance of article generation and retrieval. Aforementioned statement was stated as the benefits of proposed approach.\n\n8. Not clear about the human study for this use-case on asking humans to distinguish machine-generated articles from human-written articles. It make sense in case of fake news. It will be good to add more motivation.\n"}}, {"review_id": "1AnNQmLm29", "reviewer": "Reviewer_U6FG", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments, we appreciate their time and will use their suggestions to improve our paper. Since the reviewer lowered the scores before we post our responses, we sincerely hope that the following responses can address the concerns. \n\n> The authors propose the CLIP-based Visual NER framework to extract the entities. And I want to know whether the parameters of the framework contain the CLIP-based Visual NER module. Because the framework also needs to utilize Visual NER module to extract the entities during the inference time.\n\nAs our goal is to compare language models, we did not consider the effect of modules not related to language generation as they can simply be swapped out with varying methods or entirely excluded, as we do when using user provided named entities.  For example, we provide an example of user provided named entities in our oracle entities, whereas Visual NER compares those extracted using CLIP, but this could also come from another means.  However, as the main point of comparison is the model performing language generation, this is where we make our primary comparison.  This also follows the views of how to count model parameters in prior work. For example, VisualGLM-6B [1] introduces BLIP-2 [2] as the image encoder to enable image comprehension, and they did not include the parameters of BLIP-2 when naming their model.  However, even if we include the parameters of the CLIP model we used, which is 151M, the total parameters of ENGINE and CLIP (1.5B+151M) is still much smaller than GPT-J (6B) and GPT-Neo (2.7B), **demonstrating that including CLIP’s parameters in the total parameter count of our model would not change any of our conclusions.**\n\n[1] Du, Zhengxiao, et al. \"GLM: General Language Model Pretraining with Autoregressive Blank Infilling\", ACL (2022).  \n[2] Li, Junnan, et al. \"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.\" arXiv (2023).\n\n\n>Please keep the formula reference format consistent, such as “Eq.” and “Equation”.\n\nWe thank the reviewer for this feedback. We shall use a more consistent formula reference format in our camera ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors propose to extract the named entities from both metadata and embedded images associated with articles, and utilize an entity-aware mechanism that enhances the model’s ability to recognize and predict entity types. The experimental results demonstrate the usefulness of the proposed framework.", "reasons_to_reject": "The authors propose the CLIP-based Visual NER framework to extract the entities. And I want to know whether the parameters of the framework contain the CLIP-based Visual NER module. Because the framework also needs to utilize Visual NER module to extract the entities during the inference time.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an entity-aware article generation and retrieval approach to explicitly incorporate named entities into language models. Because the applications of real-world events may reference many named entities that are hard to be recognized by language models. The experiments on three public datasets demonstrate the effectiveness of the proposed model.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please keep the formula reference format consistent, such as “Eq.” and “Equation”."}, "extra_text": {}}, {"review_id": "i57xYWfPCc", "reviewer": "Reviewer_KMxz", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments, we appreciate their time and will use their suggestions to improve our paper. \n\n> In CLIP-based NER, the authors select top k entities directly to represent the image, which remains to be explored. The balance between the accuracy and randomness of the entity should be studied more deeply. \n\nWe provide the experiments of different $k$ values in Appendix A.3, Table 7. We see that when $k$ is greater than 10, the improvement of the language model is limited. Therefore, we set the value of $k$ to 10 to balance the accuracy and randomness of the named entities. Below we recreate Table 7 related to this experiment for your convenience.\n\n**Ablation study on the number of named entities detected by CLIP (GoodNews).**\n|   top-k entities   | 5 | 10 | 15 | 20 |\n|----------- |:--------:| :--------:| :--------:| :--------:|\n| PPL |  15.5  | 14.8 | 14.5 | 14.6 |\n\n\n> The training process of ENGINE isn’t clearly illustrated. Please add loss function, lear ing rate and other training strategies to explain how the model is finetuned. \n\nWe implemented our models mainly based on Pytorch [3] and Transformer [4] libraries. \nThe maximum sequence length of language models is set to 1024. For ENGINE-Base and ENGINE-Medium, we used a batch size of 8 and a maximum learning rate of $10^{-4}$. For ENGINE-XL, we used a batch size of 4 to fit the GPU memory. Correspondingly, the maximum learning rate is set to $2^{0.5}\\times10^{-4}$. We trained our models around 3 epochs with 0.06 epoch for linear warm-up on both datasets. We parallelized ENGINE-XL on 4 NVIDIA RTX-A6000s and ENGINE-Medium on 2 NVIDIA RTX-A6000s. Consistent to other GPT models, we apply negative log-likelihood as our loss function. **We will also release our code to ensure reproducibility in the camera ready.**\n\n\n> The ablation study of ENGINE is incomplete. It is suggested that the authors should add Cap-CapNE-ClipNE-EA、Cap-CapNE-ClipNE-NE-EA according to the overview model of ENGINE. Also, the authors should give more detailed analysis to the ablation results. \n\nFollowing the reviewer's suggestions, we provide additional comparisons here as the supplement of our ablation study in Figure 3.\n\n**Ablation study on VisualNews and GoodNews.**\n|   Method      | VisualNews PPL | GoodNews PPL |\n|-------------------------------|:----------:|-----------:|\n| Text-only | 18.3 | 17.3 |\n| Cap | 17.7 | 16.6 |\n| Cap-EA | 17.1 | 15.9 |\n| Cap-CapNE-EA | 17.1 | 16.0 |\n| Cap-ClipNE-EA | 16.1 | 14.8 |\n| **Cap-CapNE-ClipNE-EA** | 16.0 | 14.8 | \n| **Cap-CapNE-ClipNE-NE-EA** | 13.2 | 12.3 |\n| Cap-NE-EA | 13.1 | 12.0 | \n\nWe bolded the newly added ablation experiments to better differentiate them from existing ones. From the table, we observe that combining entities using CapNE and ClipNE (*Cap-CapNE-ClipNE-EA*) did not provide an improvement over *Cap-ClipNE-EA*. This is because CapNE is extracted from *Cap*. When *Cap* is available as input to language models, *CapNE* does not introduce any new information to language models. Similarly, we also observe that *Cap-CapNE-ClipNE-NE-EA* does not outperform *Cap-NE-EA* for the same reason. \n\n> The experiments on article retrieval are inadequate. The authors can add ablation study of ENGINE on article retrieval.\n\nWe ablate the two modules of our method in the following Table.\n\n**Ablation study on Article Retrieval. From left to right (R@1-R@10): Image-to-article on GoodNews, Image-to-article on VisualNews, Article-to-image on GoodNews, Article-to-image on VisualNews**\n\n|   Method      | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 |\n|-------------------------------|:----------:|-----------:|:----------:|-----------:|:----------:|-----------:|:----------:|-----------:|:----------:|-----------:|:----------:|-----------:|\n| Text-only | 50.7 | 73.1 | 79.4 | 57.9 | 78.6 | 85.2 | 49.3 | 69.6 | 76.8 | 54.8 | 77.3 | 82.7 |\n| NE | **54.1** | **74.6** | **80.8** | 60.0 | 80.3 | 85.9 | 51.3 | **72.1** | 77.4 | 56.1 | 78.2 | 83.7 |\n| NE-EA | 53.8 | 73.5 | 79.6 | **61.9** | **82.0** | **86.7** | **51.5** | 72.0 | **77.8** | **56.3** | **78.3** | **83.9** |\n\n\n\nText-only denotes that we directly use the original articles as the input. NE denotes named entity extraction, and EA denotes the entity-aware mechanism. From the table, we observe that both named entity extraction and the entity-aware mechanism can boost the article retrieval performance. In most experiment settings, we find performance gains introduced by the entity-aware mechanism. In the image-to-article experiments on GoodNews, we did not observe improvements brought by the entity-aware mechanism, but we argue this is simply due to the noise introduced by entity category recognition on GoodNews, which may affect the performance of this experiment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper explicitly incorporates and models named entities in articles to boost the performance of article generation and retrieval, which is reasonable and effective in both theory and practice. The authors have also carried on various experiments to demonstrate that ENGINE can produce high-quality and hard-to-detect articles, which can potentially contribute additional training data for the development of more powerful machine-generated text detectors.", "reasons_to_reject": "1.In CLIP-based NER, the authors select top k entities directly to represent the image, which remains to be explored. The balance between the accuracy and randomness of the entity should be studied more deeply.\n2.The training process of ENGINE isn’t clearly illustrated. Please add loss function, lear ing rate and other training strategies to explain how the model is finetuned.\n3.The ablation study of ENGINE is incomplete. It is suggested that the authors should add Cap-CapNE-ClipNE-EA、Cap-CapNE-ClipNE-NE-EA according to the overview model of ENGINE. Also, the authors should give more detailed analysis to the ablation results.\n4.The experiments on article retrieval are inadequate. The authors can add ablation study of ENGINE on article retrieval.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies article comprehension task with applications including article generation and image-to-article retrieval, and proposes an ENtity-aware article GeneratIoN and rEtrieval approach, ENGINE , to explicitly incorporate and model named entities in articles. Experiments on GoodNews and VisualNews demonstrate that ENGINE can boost the performance of article generation and retrieval. The authors also perform comprehensive experiments on human evaluation and machine discrimination, validating that ENGINE produces more realistic articles compared to prior work.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Proposes an approach for article generation that is enhanced with two key components: a NER module and an entity-aware component. Experiments on 3 public datasets show significant gain on perplexity and recall@k. 

PLUS: 

- extensive comparison with LMs on closely related tasks. 
- ENGINE can potentially contribute valuable training data for machine generated text detectors. 

MINUS:  

- human study does not correlate with the main use case. 
- precision of NEs obtained from articles? 
- Missing details about loss function, learning rate and training strategies for ENGINE. 
- incomplete ablation study.
Decision: Accept-Findings