DIVHSK: Diverse Headline Generation using Self-Attention based Keyword Selection
Year: 2023

Abstract:
Diverse headline generation is an NLP task where given a news article, the goal is to generate multiple headlines that are true to the content of the article, but are different among themselves. This task aims to exhibit and exploit semantically similar one-to-many relationships between a source news article and multiple target headlines. Towards this, we propose a novel model called DIVHSK. It has two components: KEYSELECT for selecting the important keywords, and SEQGEN, for finally generating the multiple diverse headlines. In KEYSELECT, we cluster the self-attention heads of the last layer of the pre-trained encoder and select the mostattentive theme and general keywords from the source article. Then, cluster-specific keyword sets guide the SEQGEN, a pre-trained encoderdecoder model, to generate diverse yet semantically similar headlines. The proposed model consistently outperformed existing literature and our strong baselines and emerged as a stateof-the-art model. Additionally, We have also created a high-quality multi-reference headline dataset from news articles1.

heading: 1 Introduction
text: Generating diverse and semantically similar multiple outputs in natural language generation (NLG) is an important and challenging task (Tevet and Berant, 2021). The traditional single headline generation task is formulated as a sequence-to-sequence learning problem and has been extensively studied for more than a decade now (Banko et al., 2000; Zajic et al., 2002; Dorr et al., 2003; Lopyrev, 2015; Takase et al., 2016; Gavrilov et al., 2019). Recently, researchers are also interested towards diverse output sequence generation tasks. This falls into the one-to-many generation category and is being studied for multiple tasks such as paraphrase generation (Yu et al., 2021; Gupta et al., 2018), machine
1Our code and dataset are available at https://github. com/kaushal0494/DivHSK
translation (Shen et al., 2019), question generation (Shen et al., 2022) and summarization (Cho et al., 2019). In this work, we consider the problem of generating diverse headlines given a single news article. Diverse headlines present the theme of the article in semantically related yet lexically different short sentences, which may attract different sets of audiences and increase the consumption of the news.
The existing approaches for diverse sequence generation mostly diversify the decoding steps through alternative search algorithms (Vijayakumar et al., 2018; Fan et al., 2018) or mixture decoder approaches (Shen et al., 2019; Maurya and Desarkar, 2020) where different decoders generate difference output sequences. Recently, Cho et al. (2019) proposed a two-stage modeling involving a diversification stage to extract diversifying attributes and a generation stage to guide the encoder-decoder model for diverse generations. The diversifying attributes are keywords extracted from the input text with the expectation-maximization algorithm. They consider text summarization and questiongeneration tasks. In similar lines, Yu et al. (2022) leverage external knowledge graph, i.e., ConceptNet (Speer et al., 2017) to extract diverse yet relevant keywords at diversification stage and generate diverse common sense reasoning texts. These models are not directly applicable for diverse headline generation tasks because the headlines are mostly oriented toward a single common theme (event, person, etc.) in a short sentence, and these models distract the semantics of generated headlines. Our empirical experiments (Section-5) validate this point. Liu et al. (2020) used manually extracted keywords with a multi-source transformer for diverse headline generation. The model is not scalable to other datasets/tasks because keyword extraction requires a human annotator. Unlike these, we used an automated self-attention-based approach to obtain the most attentive keywords from the article
1879
automatically. To overcome the limitations of the existing models, we propose DIVHSK, a simple yet effective model for diverse headline generation using a selfattention-based keyword selection. The model has two modules/components: (a) KEYSELECT - a pretrained encoder model to extract diversifying attributes i.e. theme and general keywords from input news article and (b) SEQGEN - a regular pre-trained encoder-decoder architecture guided by diversifying attributes for generating multiple diverse yet semantically similar headlines.
Overall, our main contributions are as follows: (1) We propose a novel model DIVHSK- Diverse Headline Generation using Self Attention based Keyword Selection to generate diverse yet semantically similar headlines. (2) We release a high quality MRHEAD: Multi-Reference Headline Dataset for diverse headline generation task. (3) The performance of the proposed model is compared with several strong baselines using both automated and human evaluation metrics.

heading: 2 Problem Formulation
text: Given a news article, the goal is to generate semantically similar, grammatically coherent, fluent and diverse headlines. Formally, given a news article x, the goal is to model the conditional distribution for k target outputs p(yk|x) with valid mappings x → y1, . . . , x → yk where {y1, y2, . . . , yk} should be diverse. Here we consider k = 3, i.e., the task is to generate three diverse headlines.

heading: 3 Methodology
text: The proposed DIVHSK model has two components (1) pre-trained encoder, i.e., KEYSELECT and (2) regular pre-trained encoder-decoder, i.e., SEQGEN. As per Liu et al. (2020), multiple headlines should convey the common theme, differing on a lexical level and the headline tokens should be uniformly distributed across the source article. Towards these goals, in KEYSELECT, we first cluster the encoders’ last-layer self-attention heads to find the most attentive keywords for each cluster from the input news article. We observe that: (a) all the clusters have a few most-attentive common keywords called as theme and (b) cluster-specific most attentive keywords called as general (i.e., non-theme) keywords. We combine theme with cluster-specific general keywords to create diversifying attributes. For each of the k clusters, there is a corresponding diversify-
ing attribute. Table-4, in Appendix, presents a few sample themes and general keywords.
The input news article, theme, and general keywords (from diversifying attributes) are concatenated with [SEP] tokens to create modified input for the SEQGEN module. In this way, different cluster leads to generate diverse headlines. The theme and general keywords in the cluster lead to semantically similar and theme-oriented headlines. For pre-trained encoder and pre-trained encoderdecoder models, we use the ’encoder of T5-base’ (Raffel et al., 2020) and T5-base checkpoints, respectively. See Figure 1 for an overview of the proposed model. More details about each component are given below:

heading: 3.1 KEYSELECT: Keyword Selection Module

heading: 3.1.1 Self-Attention Heads Clustering
text: We take a pre-trained encoder model with l selfattention heads h1, h2, . . . , hl from the last layer. Each self-attention head hi usually focuses on different parts of the inputs text (Peng et al., 2020). We group these heads into k clusters C = {c1, c2, . . . , ck}; so each cluster has g = lk heads. Here we cluster the heads in a sequential manner. Next, we identify the m most-attentive keywords (not BPE) from each head. As one keyword may get high attention values from multiple heads, it may result in overlap among the keyword sets obtained from each head. Consequently, we get a maximum of g ∗m keywords from each cluster. Stop-words/function-words are not considered in keyword sets.
We have clustered the multiple heads of multihead attention of the last-hidden layer in a sequential manner. The adoption of this approach can be justified from two perspectives. Firstly, during the pre-training phase of a language model, the weights of each head within the multi-head attention mechanism are initialized with random values. Over the course of pre-training, these weights undergo the process of learning to acquire diverse values. The different heads aim to focus on different parts of the input and provide a diverse view, which is suitable for diverse keyword selection. Secondly, the proposed model is trained end-to-end, and the weights of the KEYSELECT module are consistently updated rather than being fixed. Moreover, the target headlines associated with different heads (clusters) are different. Therefore, during back-propagation, the different heads learn to focus on the keywords
relevant to their respective target reference headlines. Based on these points, we conclude that clustering heads in any order does not have a significant impact, and we choose a simple sequential manner for the clustering of the attention heads.

heading: 3.1.2 Creating Diversifying Attributes
text: Suppose the total number of keywords to guide the SEQGEN module is n. We keep r keywords as theme keywords and the remaining n − r as general keywords. The r keywords are the mostattentive common keywords across all c clusters. The rest of the n − r keywords are the mostattentive non-overlapping keywords specific to individual clusters ci. These n keywords form the diversifying attributes Kguideci for cluster ci. r is a hyper-parameter and its value can be determined empirically. In case r common keywords can not be found2, then we can take the available r′ common keywords that can be found, and the remaining n− r′ keywords can be taken from the individual clusters. See Algorithm-B in Appendix for more details.

heading: 3.2 SEQGEN: Pre-trained Seq2Seq Module
text: The diversifying attributes Kguideci are concatenated with the source article x as:
theme-keywords [SEP] general-keywords [SEP] article
to form the extended article xeci . Each cluster corresponds to specific attributes, resulting in different extended articles. We fine-tune a pre-trained encoder-decoder model with an extended article and a corresponding headline. Additionally, we employed word-mover distance (WMD; Kusner et al. (2015)) between predicted (hp) and reference (hr) headlines token ids, as
2We have not encountered any scenario where the theme keywords are not present in one or more clusters.
an additional component in the loss function to control the diversity with λ. Finally, the KEYSELECT and SEQGEN modules are trained in end-to-end manner to minimize the loss L as:
L = c∑
i=1
(1−λ)(−logPθ(yi|xei ))+λ(WMD(hpi, hri)) (1)

heading: 4 Experimental Setup

heading: 4.1 Dataset
text: One of the essential elements of the proposed work is the inclusion of multiple reference headlines for each news article. Specifically, each example in the dataset will consist of a quadruple in the following format: <article, headline-1, headline-2, headline-3>. However, the proposed approach can be easily extended to a single reference setup. Towards this, we have created a dataset that we refer to as MRHEAD: Multi-Reference Headline.
•DataSet Collection: To create the dataset, first, we scrape news articles and their headlines from Inshorts (https://www.inshorts.com/) news website and add them to a seed set. Articles under
‘All News’ category, i.e., politics, sports, technology, etc. were considered. Next, we identify news articles from other public news websites that are semantically similar to the articles in the seed set, and also note their headlines against the corresponding article in the seed set. To find semantically similar news articles we use sentence-BERT (Reimers and Gurevych, 2019) and cosine-similarity scores. Then, human annotators verify the dataset content and remove the poor-quality headlines. Following this process, we obtained 3012 articles each with at least three parallel headlines. We split the data into training, validation, and test splits of sizes 2330, 100, and 582 respectively. Dataset creation, human verification, and other statistics are reported in Appendix-A.

heading: 4.2 Baselines
text: We have meticulously chosen six baseline models for our experimentation and analysis. Our extensive observations have revealed that single-output generation models, such as textsummarization/headline generation models, do not perform well in multi-output generation settings. The primary issue with such multiple generated outputs is their lack of lexical diversity. Therefore, we have selected three literature baselines: Mixture-Decoder (MixD; Shen et al. (2019)), Mixture Content Selector (MixCS; Cho et al. (2019)), and Knowledge Graph Experts (MoKGE; Yu et al. (2022)). Additionally, we have designed three robust baselines based on diverse search algorithms and with modified loss functions: T5+DSA (diverse search algorithm), T5+WMD (Kusner et al., 2015), and T5+Avg-Loss. More details about these baselines are provided in Appendix-C.

heading: 4.3 Evaluation Metrics
text: We use four automated evaluation metrics that rely on a lexical and semantic match in a one-to-many evaluation setup, as, for a given generation there are three reference headlines. We consider BLEU4 (BLEU; Papineni et al. (2002)) and ROUGE-L (Lin, 2004) metrics as lexical-match metrics, and BERTScore (Zhang et al., 2020) and BARTScore (Yuan et al., 2021) as semantic match based metrics. To measure the diversity among the generated headlines, we use Pairwise-BLEU (self/P-BLEU; Ott et al. (2018)) metric similar to Shen et al. (2019). As stated by Shen et al. (2019), there is always a trade-off between performance and diversity, i.e., if the generated headlines are correct but similar, then the performance (BLEU and ROUGE-L scores) will be high due to large lexical overlap but the diversity will be low (high P-BLEU) and vice-versa. Towards this concern, we consider the harmonic mean (HMean) between (1 − PBLEU) and BLEU as a combined evaluation metric. For more certainty about model performance, we also conducted the human evaluation with four metrics, i.e., Fluency (Flu), Relatedness (Rel), Correctness (Corr) and Diversity similar to (Cho et al., 2019). To manage the load on evaluators, we selected three baseline models for human evaluation. Two of the models were the best-performing (according to HMean) competitor models from literature (MixCS and MoKGE), and the other one was T5-Avg-Loss, the best-performing baseline model designed by us.
We randomly selected 50 generated headlines from the baselines and the proposed DIVHSK model as a human evaluation sample. Further, we employ two sets of annotators for human evaluation to avoid any biased evaluation. For diversity we asked an absolute evaluation score on a scale of 1 (lowest) to 5 (highest) and for other metrics a comparative evaluation. See more details about human evaluation guidelines in Appendix-D.

heading: 5 Results and Discussions

heading: 5.1 Diversity vs. Accuracy Trade-off
text: Table-1 displays the automated evaluation scores obtained for various baselines and the proposed DIVHSK models. The mixture decoder model, which employs multiple decoders, achieves the highest BLEU and ROUGE-L scores. However, the high P-BLEU score for this model indicates low diversity in the generated headlines, defeating the purpose of having multiple decoders. Similar observations are noted for the T5+DSA model. Additionally, the high scores obtained for BERTScore and BARTScore metrics suggest that the DIVHSK model exhibits superior semantic similarity with the reference headlines. This is one of the key constraints that ensure the generated outputs are semantically coherent. The ideal model should obtain reasonable BLEU and ROUGE-L scores, high BERTScore and BARTScore (high semantic similarity), low P-BLEU (high diversity), and high HMean scores. The proposed DIVHSK model satisfies these ideal conditions and emerges as a state-of-the-art model. The necessary ablation experimental results are added in Table-5.

heading: 5.2 Comparison with State-of-the-Art
text: We have compared the performances of DIVHSK with MixD, MixCS, and MoKGE, which are stateof-the-art literature models. Although these models perform well for other tasks, they exhibit poor performance for the diverse headline generation task. As discussed in Section 1, recent models like MoKGE perform poorly for diverse headline generation tasks due to the inclusion of tokens/keywords from the knowledge graph that may not align with the headline’s theme and distract the learning process. Overall, it is evident from the performances of MixCS and MoKGE that existing text summarization models do not perform well for headline generation tasks. This could be due to the fact that summaries are generally long, while headlines are
Model Headline 1(⇑) Headline 2(⇑) Headline 3(⇑) P-BLEU(⇓)BLEU R-L BES BAS HMean BLEU R-L BES BAS HMean BLEU R-L BES BAS HMean
short and more focused. The models fail to adapt to these settings.

heading: 5.3 Human Evaluation Results
text: For more reliable evaluation, we also conducted human evaluation and results are reported in Tables 2 and 3. For Fluency, Relatedness and Correctness metrics, the DIVHSK model most of the time either wins or ends up with tie versus all considered baselines. Similar trends are observed across both the annotator sets. The human evaluation scores correlate well with automated evaluation scores. The average absolute diversity scores are reported in Table-3 and it is found that generated text are more diverse for proposed DIVHSK model. Considering decent automated and human evaluation scores, we conclude that our model performs reasonably well and outperforms the other methods consistently.

heading: 5.4 Effect of n and r Parameters
text: In Figure 2, we investigate the effect of varying the values of n (the total number of selected keywords) and r (the number of theme keywords) on the performance of the DIVHSK model. As n and r increase, we observe a decrease in the P-BLEU scores, indicating an increase in diversity (headlines are lexically diverse). However, the BLEU
and ROUGE-L scores also decrease due to high diversity as these metrics are based on lexical matching. Therefore, the optimal values of n and r are important to maintain the diversity and performance trade-off.

heading: 6 Conclusion
text: In this work, We present a novel task and dataset for diverse headline generation. We also propose a strong neural architecture for the task. The model, referred to as DIVHSK, uses self-attentionbased clustering to create diversifying attributes that guide the pre-trained encoder-decoder model to generate diverse headlines. We empirically demonstrate that the DIVHSK consistently outperforms all baseline models on both automated and human evaluation metrics, while maintaining diversity as a key criterion.
Limitations
• We are unable to test the proposed model’s performance on other datasets due to the unavailability of public multi-reference headline generation datasets.
• Our dataset is created over a period of 6 months and contains around 3000 examples. Although there are several commonly used benchmark datasets with a similar number of examples: e.g., R4C reading comprehension dataset (6.4K examples) (Inoue et al.,
2020), FIRE-LID (3357 examples), IIITHNER (3084 examples) datasets in GLUECoS benchmark (Khanuja et al., 2020), WNLI (634 examples), RTE (2500 examples) and MRPC (3700 examples) datasets in GLUE benchmark (Wang et al., 2018), NOPE Corpus (around 2.7K examples) (Parrish et al., 2021), we believe that it will be better to have a larger dataset for this challenging task. We plan to create a larger version of the dataset in future work.

heading: B KEYSELECT Module
text: Algorithm 1 Keyword Selection Algorithm Require: l self-attention heads h1, h2, . . . , hl Require: c clusters c1, c2, . . . , cc Require: m, n, r: Keyword-Selection hyper-parameters 1: Initialize g = l
c 2: for i ∈ {0, c− 1} do 3: Assign g heads (hig+1-h(i+1)g)) to the cluster ci 4: Initialize set wi← ∅ to store the keywords of ci 5: for each hj in ci do 6: Select top m attentive words from hj and update
the set wi 7: end for 8: # ci will contain at most g ∗m keywords 9: end for
10: for i ∈ {0, c− 1} do 11: Select r (or r′) theme keywords from overlapping
keywords across c clusters based on attention scores 12: Select n− r (or n− r′) general keywords from non-
overlapping keywords specific to the cluster ci based on attention scores 13: Cluster ci have corresponding diverse keywords set Kguidei of size n 14: end for 15: Use Kguide consists list of selected keywords for c clus-
ters in SeqGen module

heading: C Baselines
text: We compare the proposed model performance with three literature and three other strong baselines. Details of the baselines are mentioned below:
1. Mixture-decoder: In the mixture decoder (Shen et al., 2019) approach, three different decoders are used to generate the diverse headlines. Each decoder is trained with a different headline and we take the average crossentropy loss for the particular news article.
S.No. Example-1 Example-2
News Article Days after a 20-year-old B.Com student was found unconscious with her hands and legs tied up on the outskirts of Andhra Pradesh, Vizianagaram town police said the incident was staged.The woman left her hostel to meet a male friend. After her brother inquired about her at the hostel she staged the incident to convince her family she was kidnapped. A video showing a Chandigarh female traffic police constable holding her baby in her arms while on duty has gone viral on social media. The constable Priyanka was reportedly pulled up for not reporting to work at 8 am following which she took her baby to work. The clip was captured near the roundabout of Chandigarh’s Sector 15:23 on Friday.
Headline 1 20-yr-old Andhra woman found with hands, legstied staged ’kidnap’: Police Chandigarh traffic constable reports for duty with baby in arms; video goes viral Headline 2 Andhra woman found ‘unconscious’ had staged‘kidnap’ say police Video of Chandigarh cop holding baby while on duty goes viral: The Tribune India
Headline 3 Kidnapping victim found tied up in backseat afterpolice stop wrong way driver in Olympia
Video of a Woman Traffic Constable Holding Baby on Duty Goes Viral Netizens Demand Free Daycare for Cops

heading: D Human Evaluation Setup
text: We conducted a human evaluation with four metrics i.e., Fluency (Flu), Relatedness (Rel), Correctness (Corr), and Diversity. Fluency measures how fluent and grammatical the generated text is. Relatedness indicates how much the generated outputs are in the context with input(s), Correctness measures semantics and meaningfulness. Finally, Diversity measures how diverse the generated headlines are.
A human evaluation task was conducted to compare the results of our proposed model with baselines. The evaluations were carried out by 20 human evaluators, each of whom held at least a Masters’s degree and possessed a good knowledge of the English language. We selected 50 input news articles randomly from the dataset and generated three headlines for each article using the selected models. For each input, we randomly selected the kth generated headline (k ∼ 1, 2, 3) from the models (both baselines and proposed). For example, if k = 2, we selected the second generated headline from the proposed model as well as from all the other baselines. This process was repeated for all 50 input news articles. For the first task, the dataset consists of 3-tuples containing the news article, headline from the proposed model, and headline from the baseline model. The annotators were asked to provide relative scores based on fluency, relatedness, and correctness between the two headlines. They were given three options (0, 1, 2), where 1 indicated that headline-1 was better, 2 indicated that headline-2 was better, and 0 indicated a tie. The annotators were not informed about the baseline and proposed model results.
The second task aims to ensure the diversity of generated headlines. Similar to the first task, we selected 50 samples from the proposed model and other baselines for the same news articles. The dataset consists of a news article and three headlines. The annotators were asked to provide diversity scores ranging from 1 to 5, where 1 indicated headlines with the least diversity or unacceptable quality and 5 indicated diverse headlines along with good quality.
E Implementation Detail
In our proposed model, we utilized pre-trained weights of the T5-base encoder for the pre-trained encoder used in the KEYSELECT module during training. The model was trained for 20 epochs, and the best checkpoint was selected based on the
validation loss. We used l = 12 self-attention heads from the pre-trained encoder of the KeySelect module. As we aimed to generate three diverse headlines, we set c = 3, which implies g = 4. The optimal values for our best-performing model were m = 10, n = 3, r = 2, and λ = 0.5. The total number of parameters was 3 × 108. We utilized the Adam optimizer technique with a learning rate of 1e− 4. During the test phase, we used the combination of Top-K and Top-p sampling decoding strategies, where K = 50 and p = 0.95. The batch size was 32. We implemented all the models using PyTorch (Hugging-face). Model training was performed on a V100, 32GB single GPU.

heading: F Ablation Study
text: We conducted an ablation study to analyze the effect of different model components on the performance of our proposed model. The experimental results are presented in Table-5. First, we added a plug-and-play module called WordNet (Fellbaum, 1998) to our model, which is used to obtain related keywords from the input text. Specifically, if n keywords are extracted from the input text in a cluster ci, then the final set of keywords after using the WordNet module would be at least 2n keywords for that particular cluster ci. However, in this experiment, we observed a significant drop in quality across all generated headlines. Next, we experimented with removing the Word Mover Distance component from the loss function and observed a drop in performance in terms of BLEU and PBLEU scores compared to our proposed DIVHSK model. We also experimented with different values of the hyperparameter λ used in the loss function and found that our proposed model outperforms all other variations of the model. Overall, the ablation study demonstrates the importance of the different model components in achieving the best performance for headline generation.

heading: G Model Generated Headlines
text: In this section, we present the results generated by our proposed model, along with the results of baseline models. The generated headlines, along with input news and reference headline, are tabulated in Tables 6 and 7.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: After the conclusion as stated in the call for the main conference paper.
7 A2. Did you discuss any potential risks of your work? We use clean dataset after human verification and validation.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: section 1 and 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? I have used all the publicly available artifacts which don’t have any research restrictions.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? section 1 and 4

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? section 4 and appendix A

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? section 1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. section 4 and appendix A
C 3 Did you run computational experiments? Section 4 and Appendix E

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix E
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section 4 and appendix E
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Due to the large set of experiments and computationally constrained

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix E
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4 and A
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? The dataset is created by two of the co-authors and they were well aware of the risk and other details. They have considered the expected policies.
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? The dataset is created by two of the co-authors.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? The dataset comprises of data points (news articles) available in the public domain. The urls are part of the dataset.

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: The created data source is reviewed by the multiple stockholders
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? The dataset is created by two of the co-authors.

heading: Future Work
text: We plan to create a larger version of the dataset in future work