Identifying Informational Sources in News Articles
Authors: Alexander Spangher
Nanyun Peng
Emilio Ferrara
Jonathan May
Venue: EMNLP 2023 Main (2023)
Keywords: computational journalism
source prediction
document-level modeling
TL;DR: We collect training data and train models to accurately identify a range of sources informing news writing. We show that news articles use sources in typical patterns that can be predicted.

Abstract: News articles are driven by the informational sources journalists use in reporting. Modeling when, how and why sources get used together in stories can help us better understand the information we consume and even help journalists with the task of producing it.
In this work, we take steps toward this goal by constructing the largest and widest-ranging annotated dataset, to date, of informational sources used in news writing. We first show that our dataset can be used to train high-performing models for information detection and source attribution. Then, we introduce a novel task, source prediction, to study the compositionality of sources in news articles -- i.e. how they are chosen to complement each other. We show good modeling performance on this task, indicating that there is a pattern to the way different sources are used \textit{together} in news storytelling. This insight opens the door for a focus on sources in narrative science (i.e. planning-based language generation) and computational journalism (i.e. a source-recommendation system to aid journalists writing stories). All data and model code can be found at https://github.com/alex2awesome/source-exploration.

Identifying Informational Sources in News Articles
Alexander Spangher*h, Nanyun Peng", Jonathan Mayab and Emilio Ferrara
"Thomas Lord Department of Computer Science, University of Southern California "Information Sciences Institute, University of Southern California UUniversity of California, Los Angeles
Abstract
News Article, A
Prime Minister Laurent Lamothe an- nounced his resignation. from Statement
News articles driven by the informational sources journalists use in reporting. Modeling when, how and why sources get used together in stories help better understand the information we consume and even help jour- nalists with the task of producing it. this work, we take steps toward this goal by con- structing the largest and widest-ranging anno- tated dataset, to date, of informational sources used in news writing. We first show that our dataset can used to train high-performing models for information detection and source attribution. Then, we introduce a novel task, source prediction, to study the composition- ality of sources in news articles i.c. how they are chosen to complement each other. We show good modeling performance on this task, indicating that there is pattern to the way dif- ferent sources are used together in news story- telling. This insight opens the door for focus on sources in narrative science (i.e. planning- based language generation) and computational journalism (i.e. source-reoommendatiinn sys- tem to aid journalists writing stories).'
The announcement followed corruption
commission's 's report.
< from Report
"There was no partisan intereference" said
the commission.
+- -from Quote
However, curfews were imposed cities in anticipation of protests.
<- from Order
It remains to be seen whether the opposi- tion will coalesce around new candidate.
Table 1: Different informational sources used to com- pose single news article. Source attributions shown in bold. Some sources may be implicit (e.g. 4th sent.) or too ambiguous (last sent.). Information types used by journalists shown on the right. Our central ques- tion: does this article need another source?
identifying attributions via quotes, resulting in high-precision, low recall techniques (Padó et al., 2019; Vaucher et al., 2021).
In the first part of this paper we address source attribution. We define the concept of "source" broadly to capture different iiftormationngatheeee techniques used by journalists, introducing 16 cat- egories of sourcing (some shown in Tables 1, 2, and 3). We apply this schema construct the largest source nource-attribution dataset, to our knowl- edge, with 28,000 source-atributions in 1,304 news articles. Then, we train high nigh-performing models, achieving an overall attribution accu- racy of 83% by fine-tuning GPT3. We test nu- merous baselines and show that previous lexi- cal approaches (Muzny al., 2017), bootstrap- ping (Pavllo et al., 2018), and distant supervision
1 Introduction
Journalism informs our worldviews; news itself is informed by the sources reporters use. Identifying sources of information in a news article is relevant to many tasks NLP: misinformation detection (Hardalov et al., 2022), argumentation (Eger et al., 2017) and news discourse (Choubey et al., 2020). Attributing information to sources is challeng- ing: as shown in Table 1, while some attributions are identified via lexical cues (e.g. "said"), others are deeply implicit (e.g. one would have to know that ordering "curfew" creates a public record that can be retrieved/verified). Previous modeling work, we show, has focused on the "easy" cases:
BB quote, we mean information derived from a person or a document verbatim or paraphrased. Sourced infor- mation is broader and includes actions by the journalist to uncover information: first- prst-person observations, analyses or experiments.
All data and model code can he found at ht tps: / / thub. con/a a wxx2awesone some/ source--reeeeeeeee cocceeeexllee exploration

---

(Vaucher et al., 2021) fail.
A roadmap the rest of the paper is as follows. In the next part, Section 2, we address our ap- proach to source attribution: we define more pre- cisely the ways in which a sentence is attributable to a source, the way we built our dataset, and our results. In Section 3, we discuss how we study compositionality in news writing by constructing prediction problem. We close by discussing the implications and outlining future work.
In the second part of this paper, with source- attribution models hand, we turn to funda- mental question in news writing: when and why are sources used together in an article? Sources tend to be used in canonical ways: an article covering local crime, for instance, will likely in- clude quotes from both victim and police offi- cer (Van Krieken, 2022; Spangher and Choudhary, 2022), and an article covering political debate will include voices from multiple political parties (Hu et al., 2022). However, until now, the tools have not existed to study the compositionality of sources in news, i.e., why a set of sources was se- lected during the article's generative process.
Source Attribution
2.1 Problem Definition
We model news article as set of sentences, S {81, and set of informational sources Q (11---96- We define attribution function that maps each sentence to a subset of sources:"
To establish compositionality, we must show that a certain set of sources is needed in an arti- cle. We introduce new task, source prediction: does this document need another source? We im- plement this task in two settings: (1) ablating news documents where all sentences attributable to a source are removed or (2) leveraging a dataset with news edit history (Spangher et al., 2022) where up- dates add sources. We show that large language models achieve up 81.5% accuracy in some set- tings, indicating degree of predictability. In do- ing so, we pave the way for downstream applica- tions based on assumptions of compositionality in news, like source-based generative planning (Yao et al., 2019) and source recommendation engines (Spangher et al.. 2021; Caswell, 2019).
a(s) S Q for
A sentence is attributable to a source if there is an explicit or implicit indication that the facts in it came from that source. A sentence is not at- tributable to any source if the sentence does not convey concrete facts (i.e. it conveys journalist- provided analysis, speculation, or context), or if it cannot be determined where the facts originated. Sources are people or organizations and are usu- ally explicitly mentioned. They may be named entities (e.g. "Laurent Lamothe," in Table 1), or canonical indicators (e..g "commission," "author- ities") and they are not pronouns. In some cases, sentence's source is not mentioned in the arti- cle but can still be determined if (1) the informa- tion can only have come from a small number of commonly-ue
In sum, our contributions are fold:
1. We introduce the first comprehensive corpus of news article source attribution, covering 1,304 news articles across an expansive set of 16 information channels.
4 or (2) the information is based on an eye-witness account by the journalist. See Table 2 for examples of these latter two cate- gories. In the first two rows, we give examples of sourced information that knowledgeable journal- ist could look up quickly. The third shows scene that could only have been either directly ob- served, either in- in-person or via recording, and thus must sourced directly to the journalist.
2. We build state-of-the-art source attribution models, showing that they can be used to achieve 83% accuracy. Additionally, we val- idate that this task is challenging and re- quires in-depth comprehension by showing that fine-tuning GPT3 outperforms zero- and few-shot prompting, echoing results from (Ziems et al., 2023).
We formulate this attribution task with two pri- mary principles in mind: we wish to attribute as many sentences as possible to informational
3. We open the door to further work in document-level planning and source- recommendation by showing news articles use sources in compositional way. We show that models can predict when an article is missing a source 81.5 81.5% the time.
Most sentences are attributed to only one source the article, some are attributed several.
'Examples in this category include "the stock market." "legislativ//xeeutivee records," "court filings." Trained jour- nalists can tell with relative accuracy where this information came from.

---

Example sentences from different articles where sources are implicit
Tourist visits have declined, and the Hong Kong stock market has been falling for the past few weeks, but protesters called for more action.
Published Work. Price Signal, Statement
Mr. Trump was handed defeats in Ponnsylvania, Arizona and Michigan, where state judge in Detroit rejected an unusual Republican attempt to...
Lawsuit
Mr. Bannon, former chief strategist for President Trump, was warmly applauded when he addressed the party congress of the anti-immigrant National Front...
Direct Observation
Table 2: Examples of sentences with sourced information that is non-obvious and not based on lexical cues. In the first two rows, we show sentences where sourcing is implicit but where a trained journalist can deduce the source. In the last row, we show sourced sentence where the descriptive information could only have come from a direct observation by the journalist. Bold names are the source attribution, when it exists. In cases, not shown, where it does not exist, we label "passive voice". Underline indicates the specific information that was sourced. Colored annotations the right are high-level information channels that could, in future work, be mined for source recommendations.
Information Channel
Num. Sentences
notator is trained journalist with over 4 years of experience working in a major newsroom, and the other is an undergraduate assistant. The senior annotator checks and mentors the junior annota- tor until they have high agreement rate. Then,
No Quote
23614
Direct Quote
7928
Indirect Quote
6564
Background/Naarative
3818
they collectively annotate 1,304 articles including 3280
Statement/Public Speech
50 articles jointly. From these 50, we calculate an
Published Work/Press Report
2730
agreement rate of more than = .85 for source detection,
Email/Social Media Post
1352
.82 for attribution. Categories shown in Table are developed early in the anno- tation process and expanded until a reasonable set captures all further observations. Categories are refined and adjusted following conversations with experienced journalists and journalism professors. For full list of categories, see appendix
Proposal/Order/LLaw
896
Court Proceeding
540
Direct Observation
302
Other
610
Table 3: Number of sentences in our corpus, according to the information channel by which the journalist ob- served the information.
2.3 Source Attribution Modeling
We split Source Attribution into two steps: detec- tion (is the sentence attributable?) and retrieval (what is that attribution?) because, in early trials, we find that using different models for each step is more effective than modeling both jointly.
sources used, and we wish to identify when the same source informed multiple sentences." We al- low for expansive set of information channels be considered (see Table for some of the top channels) and design set of 16 canonical infor- mation channels that journalists rely on."
Prior work in Source Attribution primarily used hand-crafted rules (Peperkamp and Berendt, 2018), bootstrapping (Pavllo et al., 2018) and distance-supervision (Vaucher et al., 2021) ap- proaches (see Section 4). Although such work has shown impressive performance on curated datasets, they typically define source's informa- tional contribution rather narrowly (i.c. only direct or indirect quotes, compared with the 16 channels listed in Table 3). So, we test several variations of
2.2 Corpus Creation and Annotation
We select 1,304 articles from the NewsEdits cor- pus (Spangher et al., 2022) and deduplicate across versions. In order to annotate sentences with their attributions, we recruit two annotators. One an-
'For example, Table two sentences are attributable to commission, despite information coming from two separate channels (a published document and statement). "These 16 categories are formulated both in conversation with jourmalists and after extensive annotation and schema expansion.
We find .45 agreement for quote-type categories Note: we do not perform modeling on these categories in the present work, but use them for illustration and evaluation.

---

Direct Quote
Indirect Quote
Statement/ Speech
Email/ Social
Published Work
Other
Micro
Avg.
Detection fl score
Rules I
64.7
69.3
81.2
76.2
72.7
37.4
59.1
Rules 2
71.3
79.8
89.8
82.1
79.2
32.5
68.8
Quootstrap
85.0
81.3
51.3
58.6
33.1
3.0
33.4
Sentence Full-Doe
91.0
98.7
94.1
92.7
85.4
61.4
87.1
92.0
98.7
96.4
89.8
86.4
65.1
88.2
Rules
47.8
48.4
43.0
51.7
37.8
30.2
46.4
+coref
57.3
54.5
49.8
49.4
38.3
34.9
52.8
Rules 2
20.7
22.5
30.3
21.3
27.4
30.2
22.5
+conef
31,6
42.0
56.1
30.3
32.3
30.2
36.6
labeled sourced sents
QuoteBank
9.9
16.0
16.4
17.7
4,3
0.5
5.5
Accuracy gold-
SeqL.aell
37.2
43.4
40.0
31.2
32.3
17.7
Retrieval
SpanDetect
61.1
59.5
67.6
44.4
51.6
36.5
+conef'
51.2
56.8
60.6
79.0
54.6
42.6
53.6
GPT3 ft, Babbage
80.9
86.9
85.0
71.9
57.9
38.3
78.9
+conef
78.7
82.5
76.3
56.1
54.4
31.2
73.2
GPT3 ft, Curie
94.0
95.5
91.1
91.0
81.6
$7.3
91.4
GPT3 0-shot, DaVinci
70.9
58.8
72.5
43.1
54.6
47.6
58.5
+coref
66.9
57.6
61.9
20.2
42.6
51.4
55.4
GPT3 few-shot, DaVinci
56.5
70.1
52.3
49.4
82.8
61.6
+coref
70.0
55.6
72.7
50.5
48.8
60.7
58.6
GPT3 ft, Babbage
79.5
82.9
82.9
73.4
60.5
$3.0
70.9
Both Ace all sents
+Nones
82,4
84,8
85.9
73.4
61.0
64.5
73.1
GPT3 ft, Curie
90.4
90.7
89.9
91.1
78.0
68.9
80.0
+Nones
92.3
92.9
92.9
91.0
78.2
68.3
83.0
Table Detection, or correctly identifying source sentences, Retrieval correctly attributing sentences sources, are two steps in Source Aftribution. Both refers to the end-to-end process: first identifying that sentence is informed by source and then identifying that source. +coref refers to performing coreference resolution before- hand, and universally hurts the model. +None refers Retrieval models trained to assign "None" to sentences without sources, possibly climinating false positives introduced by Detection. Takeaway: We can attribute sources with accuracy > 80.
methods introduced in prior work on our dataset to confirm that these categories are not implicitly at- tributed. For detection, binary classification task, F1-score is used. For retrieval, we accuracy, or precision 1.
is an nsubj dependency to speaking verb gover- nor. nsubj is grammatical part-of-speech, and governor is a higher node in syntactic parse tree. Quootstrap: Pavllo al. (2018) created boot- strapping algorithm discover lexical patterns in- dicative of sourcing. Contrasting with previous baselines, which hand-crafted lexical rules, boot- strapping allowed researchers to learn large num- bers of highly specific patterns. Although the small size of our dataset compared with theirs pre- vents us from extracting novel lexical patterns tai- lored to us, use set of 1,000 lexical patterns R1 and
Baseline Methods
Rules (R1): Co-Occurrence: We identify sen- tences where source entity candidate co-occurs with a speaking verb. For detection, any sen- tence that contains such co-occurence is con- sidered a detected sentence. For attribution, we consider the identity of the source entity. We use list of 538 speaking verbs from Peperkamp and Berendt (2018) along with ones identified during annotation. We extract PERSON Named Entities and noun-phrase signifiers using a lexicon (n=300) (e.g. "authorities". "white house official") ex- tracted from Newell et al. (2018)'s dataset.
provided by the authors Similary R2, for detection, we consider all sentences that match these 1,000 lexical rules to be "detected" sentences. For attribution, we examine the enti- ties these rules extract.
QuoteBank: In Vaucher et al. (2021), au- thors train BERT-based -xtray-xttaactio model distamtly-supervised data scored from (Pavllo
Rules (R2): Governance: Expanding on RI, we parse syntactic dependencies sentences (Nivre, 2010) to introduce additional heuristics. Specifically, we identify sentences where the name
htttts/://gtcuub.co//e////// k/b ctt.cc////////////iiicccaaiaaa http://oootttttttoooo htrp:///rrrreeooro////////ooo terns.

---

al., 2018). This method is less lexically focused, and thus more generalizable, than previous meth- ods. They use their model to score and release a large corpus of documents. We examine this cor- pus and select articles that are both in their cor- pus and in our annotation set, finding 139 articles, and limit our evaluation to these articles. For de- tection, we examine all sentences with an attribu- tion, and for attribution, match the name of that source with our gold-labels.
Gold (Train)
Gold (Test)
Silver
# does sent / doc doc len (chars)
1032
272
9051
30
67.5
27
3952
7885
3984
# sources / doc % sents sourced
6.8
12.1
8.2
47.7%
46.9%
57.4%
% sents, most-used source doc % sents, least-used
37.5%
28. 18.1%
31.8%
5.9%
2.4%
6.7%
source / doc
source entropy
1.6
2.1
1.8
Detection Methods
# sources added per n/a version
n/a
+2
Sentence: We adapt binary sentence classifier where each token in each sentence embedded using the BigBird-bar base transformer architec- ture (Zaheer et al., 2020). Tokens are combined via self attention to yield a sentence embedding and again to yield a document embedding. Thus, each sentence is independent of the others.
document sent. likely 96th 92th to be sourced
Oth
Table 5: Corpus-level statistics for our training, test, and silver-stamdard datasets. Shown are averages across the entire corpus. Documents in the test set are longer than the training, but the model seems to generalize well to the silver-standard corpus, as statis- ties match. "% sents, top source" and "% sents, bot source" refer to the % of sourced sentences attributed to most- and least-used sources in a story. sources added / version" shows the number of sources added to articles each news update; it is calculated using the NewsEdits corpus (Spangher al., 2022). "sentence most likely be sourced" refers to the sentence with the highest likelihood of being a sourced sentence, as a percentile of doc. length
Full-Doc Doll- We use similar architecture to the Sentence approach, but instead of embedding to- kens in each sentence separately, we embed kens the whole document, then split into sen- tences and combine using self-attention. Thus, the sentences are not embedded independently and are allowed to share information.
Retrieval Methods
Sequence Labeling: predicts whether each to- ken in document is source-token or not. We pass each document through BigBird base to token-
Because our prompt-query as contains an entire article/source pair, we have limited additional buden:budger; so, for our few-shot setting, we give examples of sentence/source pairs where the source is mentioned in the sentence.
obtain token embeddings and then level classifier. We experiment with inducing curriculum by training on shorter-documents first, and freezing layers 0-4 of the architecture.
For +coref variations, we evaluate approaches on articles after resolving all coreferences using Ling (Otmazgin et al., 2022). For +Nones variations, we additionally train our models to de- tect when sentences do not contain sources. We this further corrective to eliminate false positives introduced during detection.
Span Detection: predicts start and stop to- kens of the sentence's source. We use BigBird-base and separate start/stop-token classifiers (Devlin et al., 2018). We experiment with inducing decaying reward around start/stop positions reward near-misses, and expand the objective induce source salience in Kirstain et al. (2021), but find no improvement.
2.4 Source Attribution Results
Generation: We formulate retrieval as open- ended generation and fine-tune GPT3 models to generate source-names. We use with the following prompt: "<article>To which source can we attribute the sentence <sentence>>? ?". We need include the whole article in order to capture cases where a source is mentioned in another sentence. We experiment with fine-tuning Babbage and Curie models, and testing zero- and few-shot for Dinci models.
As shown in Table 4, we find that the GPT3 Curie source-retrieval model paired with the Full-Doc detection module in a pipeline performed best, achieving an attribution accuracy of 83%. In the +one setting, both GPT3 Babbage and Curie identify false positives introduced by the detection stage and outperform their counterparts. Overall, we find that resolving coreference does not im- prove performance, despite similarities between

---

the tasks.
are used unevenly. The most-used source in each article contributes 35% of sourced sentences, whereas the least-used source contributes ~ 5%. This shows a hierarchy between major and minor sources used in reporting and suggests future work analyzing the differences between these sources.
The poor performance of both rules-based ap- proaches and QuoteBank, which also uses heuris- tics, indicates that simple lexical cues are in- sufficient. Although QuoteBank authors reported it outprrformed similar baselines as we tested (Vaucher et al., 2021), observe low perfor- mance from Quotebank (Vaucher al., 2021), even in categories it trained to detect.
Insight #2: Sources begin and end documents, and are added while reporting Next we exam- ine when sources are used in the reporting pro- cess. We find that articles early in their publi- cation cycle tend to have fewer sources, and add on average two sources per subsequent version. This indicates an avenue of future work: under- standing which kinds of sources get added in later versions can help recommend sources as the journalist is writing. Finally, we also find, in terms of narrative structure, that journalists tend to lead their stories with sourced information: the most likely position for a source the first sen- tence, the lcast likely position is the second. The second-most likely position is the end of the doc- ument. 12 A caveat to Table 5: many gold-labeled documents were parsed so the first sentence got split over several sentences, which is why ob- serve the last sentences having highest sourcing.
GPT3 Da zero-shot and few-shot greatly underperform fine-tuned models in almost all cat- egories (except "Other"). Further, we see very lit- te improvement in the use of a few-shot setup vs. zero-shot. This might be because the examples we give GPT3 are sentence/source pairs, which do not correctly mimic our document-level source- attribution task. We face shortcomings due to the document-level nature of our task: the token- budget required to ask a document-level question severely limits our ability to do effective few-shot document-level prompting. Approaches that con- dense prompts (Mu al., 2023) might be helpful explore in future work.
2.5 Insights from Source Analysis
Having built an attribution pipeline that performs reasonably well, we run our best-performing at- tribution model across 9051 unlabeled documents from NewsEdits and extract all sources. In this section, we explain derive insights into how sources are used in news articles. For statistics guiding these insights, see in Table 5, which shows statistics calculated both our annotated dataset ("Gold Train" and "Gold Test" columns) and the 9051 documents we just described ("Silver" col- umn). We ask two primary questions: how much an article is sourced? and when are sources used in the reporting and writing process?
Source Compositionality
3.1 Problem Definition
Our formulation in Section 2 for quotation attribu- tion aims to identify the set of sources a journalist used in their reporting. Can reason about why certain groups of sources were chosen in tandem? Can we determine if an article missing sources? We create two probes for this question:
1. Ablation: Given article (8, Q) with attri- bution a(s)v's S, choose one source q E Q. To generate positive examples, remove all sentences where q E a(s). To generate neg- ative, remove an equal number of sentences where a(s) = O (i.e. no source).
Insight #1: ~ 50% of sentences are sourced, and sources are used unevenly. Most articles, we find, attribute roughly half the information in their sentences to sources. This indicates that the percentage of sources used is fairly consistent be- tween longer and shorter documents. So, as a doc- ument grows, it adds roughly an equal amount of sourced and unsourced content (e.g. explanations, analysis, predictions)." 11 We also find that sources
2. NewsEdits:
Sample article-versions from NewsEdits, corpus of news articles with their updates across time (Spangher et al.,
1he sources might be used for different purposes: Spangher et al. (2023) performed an analysis on news ar- ticles' narrative structure, and found that sentences convey- ing Main Idea lead the article while sentences conveying Evaluations or Predictions. "E.g.
10 "Quotebank's 's algorithm condenses input data to BERT span-classifier by (1) looking double-quotes (2) identify- ing candidate speakers through a lookup table.
sents- ( BAGHDAD', (--1, Yest Westerday, the American mi littryy sa 1d See appendix, Figure 4.
ll For more details, see the appendix.

---

Other News
Disaster
Elections
Labor
Safety
FastText
66.1
65.8
69.8
68.8
68.0
+Source-Atribution
66.0
64.5
69.8
68.2
68.0
Top Ablated
BigBird
74.2
68.4
78.3
74.0
78.1
+Source-Atribution
73.9
69.7
74.9
73.4
73.4
GPT3 It, Babbage
78.3
75.5
81.5
72.7
80.0
+Source-Atribution
74.9
69.5
78.0
70.9
65.1
FastText
57.6
63.2
60.8
61.0
63,3
+Sourre-Attribation
57.8
63.2
61.1
62.3
64.1
Second Source
BigBird
63,8
61.8
63.1
64.3
61.7
+Source-Atrribttion
65.1
69.7
65.7
64.9
62.5
GPT3 ft, Babbage
67.1
67.9
72.9
58.8
65.6
+Source-Atributtin
65.4
65.1
68.0
65.9
66.7
Fas(Text
54.5
60.5
57.1
57.8
56.2
+Source-AAtributionn
54.8
59.2
57.6
56.5
56.2
Any Source
BigBird
57.5
53.9
55.5
55.8
57.8
+Source-Atribuuiin
59.4
55.3
60.6
60..4
56.2
GPT3 ft, Babbage
55.0
53.9
63.6
63.4
49.0
+Source- AttiiciAttiitttion
59.0
56.1
61.3
39.3
51.7
FastText
58.1
48.9
62.1
58.6
48.8
-Source-Atributon
56.8
55.8
61.9
61.2
49.6
Newy Edits
BigBird
63.5
63.9
64.5
64.8
-Sturrc-Attribttioon
69.4
65.3
62.6
60.4
GPT3 It, Babbage
65.0
63.9
64.6
62.4
+Source-Atribbttin
64.0
56.1
61.3
39.3
51.7
Table 6: Results for Source Prediction, broken into four canonical news topics and 'other.' "Top Ablated" is our prediction task run on articles ablated by removing the source that has the most sentences, "Second Source" is where source contributing more than 10% of sentences is removed, and "Any Source" is where any source is randomly removed. The NewsEdits task to predict whether the article at time will be added sources time add sourcing information, derived in Section to the input
++ 1. In the SSturcc-Attribttin experiments, (see Section 3.2). Takeaway: On all of these tasks, our models were able significantly outperform random (50% acc.). In general, our expectations are confirmed that: (a) harder tasks yield lower-accuracy results and (b) more powerful models improve performance. This indicates that there is pattern how sources used in news writing.
2022). Identify articles at time where the update + either adds a source or not.
one "other" topic based on articles the New York Times Annotated Corpus (Sandhaus, 2008) with keyword sets corresponding to each topic.
Each probe tests source usage in different ways. Ablation assumes that the composition of sources in an article is cohesively balanced, and induces reasoning about this balance. NewsEdits relaxes this assumption and probes if this composition might change, either due to the article's complete- ness, changing world events that necessitate new sources, some other factor.
We take 9051 silver-standard docu- ments (the same ones explored in Section 2.5) and design three variations of this task. As shown in Table 5, articles tend use sources lopsid- edly: one source is usually primary. Thus, we design Easy (Top Source, in Table 1), Medium (Secondary) and Hard (Any Source) variations of our task. For Easy, we choose the source with the most sentences attributed to it. For Medium, we randomly choose among the top 3 sources. And for Hard, randomly choose any of the sources. Then, we create y example by removing all sentences attributed to the chosen source, and we create a = example from the same document by removing an equal number of sentences that are
Ablation
3.2 Dataset Construction and Modeling
We use our Source Attribution methods discussed in Section 2 to create large standard datasets in the following manner for our two primary ex- perimental variants: Ablation and NewsEdits. To interpret results each variant better, we train classifier to categorize articles into four topics plus
14 "Spangher et al. (2022) found that many news updates were factual and tied to event changes, indicating breaking news cycle.
Theeee four have been identified as especially socially valuable topics, or "beats," due to their impact on government responsiveness (Hamilton, 201 2)

---

not attributed to any sources.
Overall, we find that our experiments are sta- tistically significant from random (50% accuracy) with t-test p < .01, potentially allowing us to re- ject the null hypothesis that positive documents are indistinguishable from negative in both settings. Statistical significance does not preclude con- founding, and both the Ablation and the NewsEdits setups contain possible confounders. In the Ab- lation set up, might be inadvertently learning stylistic differences rather than source-based dif- ferences. To reduce this risk, we investigate sev- eral factors. First, we consider whether lexical confounders, such as speaking verbs, might be ar- tificially removed in the ablated documents. We use lexicons defined in our rules-based methods to measure the number of speaking verbs in our dataset. We find mean of
NewsEdits We sample an additional 40, 000 ar- ticles from the NewsEdits corpora and perform at- tribution on them. We sample versions pairs that have roughly the same number of added, deleted and edited sentences in between versions in order reduce possible confounders, Spangher et al. (2022) showed that these edit-operations were pre- dictable. We identify aiticle-version pairs where or more sources were added between version and t ++ and label these as y = 1, and 0 or sources added as = 0.
Modeling We use three models: (1) FastText (Joulin et al., 2016) for sentence classification, (2) A BigBird- model: we use BigBird with self-attention for document classification, simi- lar to Spangher et al. (2022). 16 Finally, (3) we fine-tune GPT3 Babbage to perform prompt- completion for binary classification. For each model, we test two setups. First, we train on the vanilla text of the document. Then, in the -Source-Atributtion variants, we train by append- ing each sentence's source attribution to the end of 17 The source annotations are obtained from our attribution pipeline.
[34, 32] speak- ing verbs per document in y = [0. 1] classes in the Top case, n [35, 34] in the Medium, and n = 37] in Hard. None of these differences are statistically significant. We also do not find statistically significant differences between counts of named entities or source signifiers (defined in Section 4). Finally, we create secondary test sets where y = is non-ablated documents. This changes the nature of the stylistic differences be- tween y 1 and y = O while not affecting sourc- ing differences We rerun trials in the Top group- ing, as this would show us the greatest confound- ing effect, and find that the accuracy of our classi- fiers differs by within -/+3 points.
3.3 Results and Discussion
The results in Table show that we are broadly able to predict when major sources (Top, Sec- ondary) are removed from articles, indicating that there is indeed compositionality, or intention, in the way sources are chosen to appear together in news articles. The primary source (Top)'s 's absence is the easiest to detect, indicating that many sto- ries revolve around single source that adds cru- cial information. Secondary sources (Second) are still predictable, showing that they serve an impor- tant role. Minor sources (Any)'s absence are the hardest to predict and the least crucial to story. Finally, source-addition across versions (NewsEd- its) is the hardest detect, indicating that versions contain balanced compositions.
In the NewsEdits setup, we have taken care balance our dataset along axes where prior work have found predictability. For instance, Spangher et al. (2022) found that an edit-operations could be predicted. So, we balance for length, version number and edit operations.
Having attempted to address confounding in various ways in both experiments, we take them together to indicate that, despite each probing dif- ferent questions around sourcing, there are pat- terns to the way sources are during the journalistic reporting process. To illustrate, we find in Table 6 that Election coverage is the most easily pre- dictable across all tasks. This might be because of efforts to include both left-wing and right-wing voices. It also might be because the cast of charac-
"Concretely, obtain token embeddings of the entire document, which we combine for each sentence using self- attention. We contextualize each sentence embedding ing a shallow transformer architecture. We finally combine these sentence embeddings using another atte-tttention layer to obtain document embedding for classification. We uti- lize curriculum learning based on document length, a linear loss-decay schedule.
19we do not want to train on such datasets, because there are statistically significant length differences and other stylis- tie concerns ablated and abnatte articles. "F.g. Whether sentence would be added in subsequent version.
"Like so: <sent: 1x.
SOURCE: <source 1>.
<sent 2> SOURCE: <source 2>... <sent n> SOURCE: ksource n>.

---

ters (e.g. campaign strategists, volunteers, voters) stays relatively consistent across stories.
Persona Modeling A second area that our work draws inspiration from is the study of narra- tive characters and how they are used in fic- tion. Work by Bamman et al. (2013) and Card et al. (2016) used custom topic models to model characters by latent "personas" generated from latent document-level extended this topic-modeling approach to news sources (Spangher et al., 2021). We see poten- tial for future work merging this with our dataset and framework, using methods like discrete vari- ational autoencoders, which have been applied to dloument-plaaning (Ji and Huang, 2021).
Two additional findings are that (1) the tasks we expect are harder do yield lower accuracies and, (2) larger GPT3-based language models generally perform better. Although not especially surpris- ing, it further confirms our intuitions about what these tasks are probing. We were surprised to find that, in general, adding additional information in both stages of this project, whether coreference in the attribution stage or source information in the prediction stage, did not improve the mod- els' performance. (In contrast, adding source in- formation to smaller language model, Bi BigBire rd, helped with harder tasks like the Medium, Hard and NewsEdits). We had hypothesized that the signal introduced by this labeling would not harm the GPT3- based models, but this was not the case. It could be that the larger models are already in- corporating notion of coreference and attribu- tion, and adding this information changed English grammar in a way that harmed performance.
Isstributions.Eariiie work
Downstream Applications Diversity An in- teresting downstream applications of our work is to improve analysis of diversity in sourcing. Source-diversity has been studied in news arti- cles (Peperkamp and Berendt, 2018; Masini et al.. 2018; Berendt al., 2021; 1; Amsalem al., 2020), where authors have constructed ontologies to fur- ther explore the role of sources from different backgrounds. Opinion Mining Another strain fo- cuses on characterizing voices in text by opin- ion (O'Keefe et al., 2013). Such work has been applied in computational platforms for journalists (Radford et al., 2015) and in fake news detection (Conforti et al., 2018).
4 Related Work
Quote Attribution Prior work in quote attri- bution has also been aimed at identifying which sources contributed information in news articles. Early work explored rules-based methods (Elson and McKeown, 2010; Oeeeefe et al., 2012) and statistical classifiers (Pareti et al., 2013) to at- tribute sources to quotes. More recent work has extended these ideas by using bootstrapping to discover new patterns, Quootstrap (Pavllo et al., 2018) and training BERT-based models on pertur- bations on these patterns, or QuoteBERT (Vaucher et al., 2021). One upside of Quootstrap and QuoieBERT is that they might adapt better to new domains by learning and generalizing from new patterns. However, the method by which patterns are learned, finding quotes that repeat across out- lets, might bias this method towards discovering quotes by oft-quoted figures. These quotes, in turn, may be contextualized differently than other quotes, introducing fundamental biases in which sources get discovered. We urge more consider- ation of these potential biases, not only for per- formance considerations but faimess. Overall, our work differs from previous work in this field because we defined information more broadly. Prior work is quote-focused, whereas we include a larger set of information channels (Table 3).
Conclusions
We have offered a more expansive definition of sourcing in journalism and introduced the largest attribution dataset capturing this notion. We have developed strong models to identify and attribute information in news articles. We have used these attribution models to create large silver standard dataset that we used probe whether source in- clusion in news writing follows predictable pat- terns. Overall, we intend this work to serve as a starting point for future inquíries into the nature of source inclusion in news articles. We hope to im- prove various downstream tasks in NLP and, ulti- mately, take steps towards building a source rec- ommendation engine that can help journalists in the task of reporting.
6 Acknowledgments
Alexander Spangher would like to thank Bloomberg News for a generous year fel- lowship that has funded this work and his other work in the areas of computational journalism, computational law, nuclear fusion, and linguistics.

---

7 Limitations
box sense for giving strong baselines for predictive problems and drawing scientific conclusions about hypotheses.
A central limitation to our work is that the datasets we used to train our models are all in English. As mentioned previously, we used English language sources from Spangher et al. (2022)'s NewsEd- its dataset, which consists of sources such as ny- times.com, bbc,com, washingtonpost.com. etc.
8.2 Licensing
The dataset we used, NewsEdits (Spangher et al., 2022), is released academically. Authors claim that they received pernission from the publishers to release their dataset, and it was published as a dataset resource in NAACL 2023, We have had lawyers at a major media company ascertain that this dataset was low risk for copyright infringe- ment.
Thus, must view work in source extrac- tion and prediction with the important caveat that non- non-Western news outlets may not follow the same source-usage patterns and discourse structures in writing their news articles as outlets from other re- gions. We might face extraction biases we were to attempt to do such work in other languages, such as only extracting sources that present in pat- terns similar those observed in Western sources, which should be considered faimess issue.
8.3 Computational Resources
The experiments our paper required computa- tional resources. We used 40GB NVIDIA V100 GPUs, Google Cloud Platform storage and CPU capabilities. We designed all our models to run on 1 GPU, so they did not need to utilize model or data- aata-parallelism. However, we still need to rec- ognize that not all researchers have access to this type of equipment.
8 Ethics Statement
8.1 Risks
Since we constructed our datasets on well-trusted news outlets, we assumed that every informational sentence factual, to the best of the journal- ist's ability, and honestly constructed. We have no guarantees that such an attribution system would work in a setting where a journalist was acting ad- versarially.
We used Huggingface Bigbird rd-based mod- els for our predictive tasks, and will release the code of all the custom architectures that we con- structed. Our models do not exceed 300 million parameters.
8.4 Annotators
There is risk that, if such a work were used in larger news domain, it could fall prey to at- tributing misinformation or disinformation. Thus, any downstream tasks that might seek gather sourced sentences might be poisoned by such a dataset. This risk acute in the news domain, where fake news outlets peddle false stories that attempt to look true (Boyd et al., 2018: Spangher et al., 2020). We have not experimented how our classifiers would function in such domain. There to identify mis-
We recruited annotators from our educational in- stitutions. They consented to the experiment in ex- change for mentoring and acknowledgement in the final paper. One is an undergraduate student, and the other is a former journalist. Both annotators are male. Both identify as cis-gender. The annota- tion conducted for this work was deemed exempt from review by our Institutional Review Board.
is work using discourse-structt information (Abbas, 2022; Sitaula et al., 2020), and this could be useful in source-attribttion pipeline to mitigate such risks.
References
Ali Haif Abbas. 2022. Politicizing the pandemic: A schemata analysis of covid-19 news in two selected newspapers. International Journal for the Semi- otics of Law-Revue internationale de Sémiotique ju- ridique, 35(3):883-902.
We used OpenAl Finetuning to train the GPT3 variants. We recognize that OpenAl is not trans- parent about its training process, and this might reduce the reproducibility of our process. We also recognize that OpenAl owns the models fine- tuned, and thus we cannot release them publicly. Both of these thrusts are anti-science and anti- openness and we disagree with them on principle. However, their models are still useful in a black-
Eran Amsalem, Yair Fogel-Dror, Shaul R Shenhav, and Tamir Sheafer. 2020. Fine-grained analysis of di- versity levels in the news. Communication Methods and Measures, 1444):266-284.
David Bamman, Brendan O'Connor, and Noah A Smith. 2013. Learning latent personas of film char- acters. In Proceedings of the 51st Annual Meeting of

---

the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 352-361.
Tiancheng Hu, Manoel Horta Ribeiro, Robert West, and Andreas Spitz. 2022. Quotatives indicate de- cline in objectivity in us political news. arXiv preprint arXiv:2210. 15476.
Bettina Berendt, Özgür Karadeniz, Stefan Mertens, and Leen d' d'Haenens. 2021. Fairness beyond "equal": The diversity searcher as a tool to detect and enhance the representation of socio-political actors in news media. In Companion Proceedings of the Web Con- ference 2021. WWW 221, page 202-212, New York, NY, USA. Association for Computing Machinery.
Haozhe Ji and Minlie Huang. 2021. Discodvt: Generating long text with discourse-aware dis- crete variational transformer. arXiv preprint arXiv:2110.05990
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint 1rXiv:660.00733
Ryan L. Boyd, Alexander Spangher, Adam Fourney, Besmira Nushi, Gireeja Ranade, James Pennebaker, and Eric Horvitz. 2018. Characterizing the internet research ageney's 's social media operations during the 2016 us presidential election using linguistic analy- ses.
Yuval Kirstain, Ori Ram, and Omer Levy. 2021. Coref- erence resolution without span representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing ( (Volume 2: Short Papers), pages 14 14-19.
Dallas Card, Justin Gross, Amber Boydstun, and Noah A. Smith. 2016. Analyzing framing through the casts of characters in the news. In P Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 1410-1420, Austin, Texas. Association for Computational Linguistics.
Andrea Masini, Peter Van Aelst, Thomas Zerback, Carsten Reinemann, Paolo Mancini, Marco Maz- zoni, Marco Damiani, and Sharon Coen, 2018. Mea- suring and explaining the diversity of voices and viewpoints the news: comparative study on the determinants of content diversity of immigration news. Journalism Studies, 19(5522324-2333
David Caswell. 2019. the semantic units of news. Digital Journalism, 134)1113.
Structured journalism and
Prafulla Kumar Choubey, Aaron Lee, Ruihong Huang, and Lu Wang. 2020. Discourse as function of event: Profiling discourse structure in news articles around the main event. In Proceedingy of the 58th Annual Meeting of the Association for Computa- tional Linguistics.
Jesse Mu, Xiang Lisa Li, and Noah Goodman. 2023. Learning to compress prompts with gist tokens. arXiv preprint (Xiv:2334.0846)
Grace Muzny, Michael Fang. Angel Chang, and Dan Jurafsky. 2017. A two-stage sieve approach for quote attribution. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Pa- pers, pages 460-470.
Costanza Conforti, Mohammad Taher Pilehvar, and Nigel Collier. 2018 Towards automatic fake news detection: cross-level stance detection in news arti- cles. In Proceedings of the First Workshop Fact Extraction and VERificcttion (FEVER), pages 40-49.
Edward Newell, Drew Margolin, and Derek Ruths. 2018. An attribution relations corpus for political news. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre- Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.0450
Steffen Eger, Johannes Daxenberger, and Iryna Gurevych. 2017. Neural end-to-end learning for computational argumentation mining. arXiv preprint arXiv:1704.06111
Joakim Nivre. 2010. Dependency parsing. Language and Linguistics Compass, 4(3):338-12.
Shon Otmazgin, Arie Cattan, and Yoav Goldberg. 2022. Lingmess: Linguistically informed multi expert scorers for coreference resolution. arXiv preprint arXiv:2205. /2644.
David K Elson and Kathleen R McKeown. 2010. Auto- matic attribution of quoted speech in literary narra- tive. In Twenty-fourth AAAI conference artificial intelligence.
Tim O'Keefe, James R Curran, Peter Ashwell, and Irena Koprinska. 2013. An annotated corpus of quoted opinions in news articles. In Proceedings of the 51st Anmual Meeting the Association for Com- putatiooal Linguistics (Volume 2: Short Papers), pages 516-520.
James T Hamilton. 201 All the news that's fit to sell. In All the News That's Fit to Sell. Princeton Univer- sity Press.
Momchil Hardalov, Arnav Arora, Preslav Nakov, and Isabelle Augenstein. 2022. A survey stance detection for mis-and disinformation identification. arXiv preprini arXiv:2103.00242
Tim O'Keefe, Silvia Pareti, James R Curran, Irena Koprinska, and Matthew Honnibal. 2012. A se- quence labelling approach to quote attribution. In

---

Proceedings of the 2012 Joint Conference on Empir- ieal Methods in Natural Language Processing and Computational Natural Language Learning, pages 790-799.
Alexander Spangher, Gireeja Ranade, Besmira Nushi, Adam Fourney, and Eric Horvitz. 2020. Char- acterizing search-engine traffic to internet research agency web properties. In Proceedings of The Web Conference 2020, pages 2253-2263.
Sebastian Padó, André Blessing, Nico Blokker, Ere- nay Dayami, Sebastian Haunss, and Jonas Kuhn. 2019. Who sides with whom? towards computa- tional construction of discourse networks for polit- ical debates. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 2841-2847.
Alexander Spangher, Xiang Ren, Jonathan May, and Nanyun Peng. 2022. Newsedits: A news article re- vision dataset and a novel document-level reasoning challenge. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 127-157.
Silvia Pareti, Tim O'keefe, Ioannis Konstas, James Curran, and Trena Koprinska. 2013. Automatically detecting and attributing indirect quotations. In Pro- ceedings of the 2013 Conference Empirical Meth- ods in Natural Language Processing, pages 989- 999.
Kobie Van Krieken. 2022. Story character, news source or vox pop? representations and roles of citizen perspectives in crime news narratives. Journalism, 1975-199999
Timote Vaucher, Andreas Spitz, Michele Catasta, and Robert West. 2021. Quotebank: a corpus of quota- tions from a decade of news. In Proceedings of the 14th ACM International Conference Web Search and Data Mining, pages 328-336.
Dario Pavilo. Tiziano Piccardi, and Robert West. 2018. Quootstrap: Scalable unsupervised extraction of quotation-speaker pairs from large news corpora via bootstrapping. In Twelfth International AAAI Con- ference Web and Social Media.
Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan- and-write: Towards better automatic storytelling. Proceedings of the AAAI Conference on Artificial In- telligence, volume 33, pages 7378-7385.
Jeroen Peperkamp and Bettina Berendt. 2018. Diver- sity checker: Toward recommendations for improv- ing journalism with respect to diversity. In Adjunct Publication of the 26th Conference User Model- ing. Adaptation and Personalization, pages 41.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, al. 2020. Big hird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33: 33:17283-17297
Will Radford, Daniel Tse, Joel Nothman, Ben Hachey, George Wright, James R Curran, Will Cannings, Tim O'Keefe, Mathew Honnibal, David Vadas, al. 2015. The computable news project: research in the newsroom. In Proceedings of the 24th In- ternational Conference on World Wide Web, pages 903-908. ACM.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang. and Diyi Yang. 2023. Can large lan- guage models transform computational social sci- ence? arXiv preprint arXiv:2305.03514.
Evan Sandhaus. 2008, The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752.
A Exploratory Data Analysis
Niraj Sitaula, Chilukuri K Mohan, Jennifer Grygiel, Xinyi Zhou, and Reza Zafarani. 2020. Credibility- based fake news detection. In Disinformation, Mis- information, and Fake News in Social Media, pages 163-182. Springer.
We show more data analysis around source usage in news articles. Figure 1 shows the distribution the amount of sentences in each article that attributable sources. Although most articles have around 50% of their sentences as source sen- tences, a small number of articles source < 10% of their sentences (a manual analysis shows that that these are mainly opinion pieces) or >> 90% of their sentences (a manual analysis shows that these are mainly short, marpaargga breaking news ex- cerpts).
Alexander Spangher and Divya Choudhary. 2022. If bleeds, leads: A computational approach to covering crime in los angeles. arXiv preprint arXiv:2206.07115.
Alexander Spangher. Xinyu Hua. Yao Ming, and Nanyun Peng. 2023. Sequentially controlled text generation. arXiv preprint arXiv:2301.0222
Figure 2 shows how articles grow over time, through versions. We find that, on average, two sources are added per version. This is surpris- ingly linear, with earlier versions containing the least number of sources.
Alexander Spangher, Nanyun Peng, Jonathan May, and Emilio Ferrara. 2021. "Don't t quote me on that":
Finding mixtures sources news articles. arXiv preprint 2XX:::10040996..

---

100
Gold (Train)
sents
80
Gold (Test)
60
Silver
1000
2000
3000
4000
25
50
100
Doc. Jen.th (chars)
Mean sents are sourced
Figure 3: Do longer stories contain more sourced in- formation? We show the percentage of sentences in an article that are based on sourced information based on the length of the document. Shorter stories are almost entirely composed of sourced sentences.
Figure 1: How much of document is sourced? We
show % of sourced sentences in documents.
Num. Sources
20
0.625
15
pisent sourced)
0.600
10
0.575
0.550
0
2
8 10
Article Version #
0
10
20
30
40
Sentence index document
Figure 2: Do more sources get added to article over time? We show the number of sources in an article as gets republished, based NewsEdits (Spangher al., 2022) and find that as news unfolds, sources get added.
Figure Where the article are sources used? We show the likelihood of sentence is sourced based its position within an article. Sources are most likely to be used in the first sentence and least likely be used in the second.
In Figure 3, we show that the percent of sourc- ing is consistent the longer document gets. This means that when more sentences are added to the document, the journalists adds consistent amount of sourced and non-sourced sentences. The only exception is when articles are very short. Manual inspection reveals that these are usually breaking news paragraphs that are entirely com- posed of a reference to press release, a statement or quote.
quote. Does not have to contain any external source of information.
Narrative: A sentence giving narrative infor- mation about the source's role in events that does not contain quote. Does not have to contain any external source of information.
(a) For "Background" and "Narrative," these usually don't explicitly reference external sources of information. It's typically implied that the journalist learned this information by talking to the sources, but does not have to be the case.
In Figure 4, show the likelihood of source being present in each sentence- contence-position of our doc- ument. This indicates where in the document sources are used. The likeliest spot for a source is the first sentence, and the least likely is the sec- ond sentence. As can seen, the likelihood of a source increases further throughout the document.
4. Direct Observation: sentence where it's clear that the journalist is either (1) literally witnessing the events (2) conducting their own analysis, investigation or experiment, i.e. the journalist is their own source of ob- servation.
Annotation Definitions
1. Quote: A statement or a paraphrase of a statement given by the source to the reporter in an interview.
2. Background: sentence giving non-event information the source (i.c. descriptions, role, status, etc.), that does not contain a
(a) The difference between "Narrative" and "Direct Observation" can hazy. Un- less it very clear that the journalist

---

literally observing events unfold, do NOT use "Direct Observation."
result in, policy changes (i.e. executive order, legislative text, etc.).
(b) When "Direct Observation" is selected, the source head is "journalist," source
16. Declined Comment: A special category of quote where the source does not comment. Also includes when a source "could be reached for comment."
type "Named Individual," affiliation "Media," role "informational" and status "current," UNLESS the jour- nalist abundantly defines themselves as otherwise (i.e. "In my years diplo- mat.... "I am professor...".
5. Public Speech: Remarks made by the source in public setting open crowd.
6. Communication: Remarks made by the source in private setting closed, lect group. Can be interpreted broadly in- clude written communications.
7. Published Work:
Work written by the source, usually distributed via academic jour- nals or government publications.
8. Statement:
A prepared quote given by a source. Usually distributed in press confer- ence or in writing.
9. Lawsuit: Any information given during the course of court proceeding including claims, defense, rulings or other court-related procedures.
10. Price Signal: Any information about a com- pany's stock price, the price of goods, etc. that was obtained through analyzing market data.
11. Vote/Poll:
Information given about voting decisions, whether as a result of an actual vote or electoral or opinion poll.
12. Document:
A more generic category of in- formation distributed via writing (Published Work is subset of this class).
13. Press Report: Information obtained from media source, whether it's news article, television report radio report.
14. Social Media Post:
Information posted on social media platform (i.e. Twitter, Face- book, blog comments, etc.).
15. Proposal/Order/Lav: Information codified in text by officials resulting, or intending to


Reviews:
[{"review_id": "57hM9SWrfy", "reviewer": "Reviewer_5ZFc", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your thoughtful review! We ask you to please reconsider this one point:\n\n> No direct comparison to existing benchmarks of existing similar tasks (e.g literature on tasks such as cite-worthiness detection, quote attribution, source prediction). \n\nWe do try to compare against extensive existing methods for quote attribution. R1 and R2 are derived from [1]'s work. Quootstrap and Quotebank are directly derived from [2]s and [3]s work. We include description in 181-195 explicitly referencing these. \n\n[1] Jeroen Peperkamp and Bettina Berendt. 2018. Diver- sity checker: Toward recommendations for improv- ing journalism with respect to diversity. In Adjunct Publication of the 26th Conference on User Model- ing, Adaptation and Personalization, pages 35–41.\n[2] Dario Pavllo, Tiziano Piccardi, and Robert West. 2018. Quootstrap: Scalable unsupervised extraction of quotation-speaker pairs from large news corpora via bootstrapping. In Twelfth International AAAI Con- ference on Web and Social Media.\n[3] Timote Vaucher, Andreas Spitz, Michele Catasta, and Robert West. 2021. Quotebank: a corpus of quotations from a decade of news. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 328–336.\n\nIt's true that we did not compare to the training and test sets that these authors used in their works, but there isn't really a single established benchmark for Quote Attribution as a task. Even when follow-up work comes from the same lab, as in [2], [3]. authors typically stated and redid the benchmark.\n\nWe believe that the benchmarks created by prior authors are limited, for reasons we discuss in the Introduction and Related Works. But to be explicit, we focused on Informational Sources that are introduced via several lexical techniques, not just direct quotes, as is typically introduced in prior work ( we discuss in line 509-511 how prior work even fails for Direct Quotations). \n\nThus, our techniques really shine in new areas of source-usage that aren't typically covered by prior benchmarks. We can do a better job explaining this in the camera ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- First of its kind corpus of news article source attributions, covering 1,304 news articles across various informational categories (e.g press report, social media post)\n- The motivation for this research area is strong (e.g developing tools to aid journalists review/write articles in a source-based manner). The source attribution corpus is a first step in that direction. ", "reasons_to_reject": " - No direct comparison to existing benchmarks of existing similar tasks (e.g literature on tasks such as cite-worthiness detection, quote attribution, source prediction). It is not obvious, for any of the two tasks the paper formulates, to understand how the new methods/models contributed by this paper rank in comparison to existing work in the research area.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is about trying to identify the set of sources journalists use when reporting by way of a function that maps news article candidate sentences to corresponding sources. This task is done in two-steps: first detecting which sentences are “attributable” (i.e linkable to sources), then retrieving the attributed source. After identifying which sources each news article sentence is attributed to, authors further developed a task which aims at predicting which sources news articles might be missing. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DhymDWYgNZ", "reviewer": "Reviewer_d5Gh", "summary": "", "questions": "1) The First contribution (Lines 86-89) mentions 13 informational categories, however, 16 different categories are listed in the abstract. Is that a typo?\n2) Do you think there is a substantial difference between the articles from different news topics shown in Table 6? If so, then I would recommend moving the paragraph describing the categorization using distant supervision (starting at Line 399) to the beginning of Subsection §3.2, as it currently is under modelling, so I expected it to be a new model.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper can introduce new applications of NLP in the field of journalism.\n* Source attribution allows for a deeper analysis of the characteristics of news articles.\n* The authors attempted to provide possible explanations for the results, especially for the second experiment in §3, which I find to be useful.", "reasons_to_reject": "* The paper could benefit from another round of editing. While I do not think it is a major weakness, I believe it will make the paper easier to follow, and understand for the reader.\n* I think the description of some models is unclear, which might make them harder to replicate. For example, the baseline R1 (Lines 182-191) mentions the terms source entity candidate and speaking verbs. It was not until the second reading of the paragraph that I thought I had an adequate understanding of what they mean.", "questions_to_authors": "1) The First contribution (Lines 86-89) mentions 13 informational categories, however, 16 different categories are listed in the abstract. Is that a typo?\n2) Do you think there is a substantial difference between the articles from different news topics shown in Table 6? If so, then I would recommend moving the paragraph describing the categorization using distant supervision (starting at Line 399) to the beginning of Subsection §3.2, as it currently is under modelling, so I expected it to be a new model.", "paper_topic_and_main_contributions": "The paper focuses on the nature of news articles. More specifically, it introduces 16 different sources that are used by journalists in writing articles. The authors employ two human annotators to identify the sources used for each sentence of 1304 different news articles. Moreover, the annotators mark the tokens that refer to the source to which each sentence can be attributed. The paper then compares baseline and more advanced models to determine whether a sentence is sourced from an informational channel or not, in addition to finding the source of the information. These models are then applied to a larger set of articles, demonstrating how the models can be used to uncover information about the number of sources per article, and the composition of sources in an article.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* The Limitations section should be a separate section and not a subsection for the “Ethics Statement” section.\n* I got a bit confused on what you mean by “informational source” (Title), “informational category” (Line 89) and “information channel” (Table 3). While I guess the last two refer to the same concept, it would be easier for the reader if you use the same term for it.\n* Typos and grammatical errors:\n  * Line 230: We prompts\n  * Line 506-507: Such works focuses\n  * Line 518-519: Both our work probes how characters in text relate.\n  * Line 539: W"}}, {"review_id": "Z2hfBT2dfg", "reviewer": "Reviewer_6Nn9", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your thoughtful review and kind comments!!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A new task is introduced together with an annotated and silver dataset. \nSeveral interesting insights when it comes to news and how they are structured/sourced. \nAblation studies, in-depth analysis of results, and insights provide added value to the community. ", "reasons_to_reject": "None.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper focuses on the task of source attribution, information detection, and the novel task of source prediction. To study the compositionality of sources in news articles, a dataset is constructed and shared with news articles annotated with their source and corresponding information type. After training/evaluating several models on source detection/retrieval and thorough analysis and insights, the focus shifts to Source Compositionality. Meaning, “Can we determine if an article is missing sources?”. After a series of ablation studies, the result is, that it can be done though difficult.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 6: In “News Edits/BigBird/Safety” the wrong value is highlighted\n539: “W” at the end of the paragraph. \n894-897: “information the source” is wrong. “information of the source”? \n901: Latex Error around “not” "}}]

Author rebuttal:

Meta-review: The paper studies how to identify and analyze the sources that journalists use in their news articles. The authors defined 16 types of sources and annotated 1304 news articles with the source type and the parts of the article referring to the source. The paper then develops and evaluates models to detect whether a sentence can be attributed to a source or not, and to retrieve the source. The paper also applies the model to a larger corpus of news articles to reveal insights about different reference patterns.

Overall, the paper introduces a new task and contributes an important dataset for further research. All three reviewers appreciated the novelty and the potential impact of the work. However, they also provide some suggestions to further improve the readability of the paper which I would encourage the authors to consider while preparing the future version of the paper.
Decision: Accept-Main