Tracking the Newsworthiness of Public Documents (2024)

Abstract: Journalists regularly make decisions on whether or not to report stories, based on “news values” (Gatlung and Ruge, 1965). In this work, we wish to explicitly model these decisions to explore when and why certain stories get press attention. This is challenging because very few labelled links between source documents and news articles exist and language use between corpora is very different. We address this problem by implementing a novel probabilistic relational modeling framework, which we show is a low-annotation linking methodology that outperforms other, more state-of-the-art retrievalbased baselines. Next, we define a new task: newsworthiness prediction, to predict if a policy item will get covered. We focus on news coverage of local public policy in the San Francisco Bay Area by the San Francisco Chronicle. We gather 15k policies discussed across 10 years of public policy meetings, and transcribe over 3,200 hours of public discussion. In general, we find limited impact of public discussion on newsworthiness prediction accuracy, suggesting that some of the most important stories barely get discussed in public. Finally, we show that newsworthiness predictions can be a useful assistive tool for journalists seeking to keep abreast of local government. We perform human evaluation with expert journalists and show our systems identify policies they consider newsworthy with 68% F1 and our coverage recommendations are helpful with an 84% win-rate against baseline.1

heading: 1 Introduction
text: Despite much qualitative analysis of newsworthiness (Gatlung and Ruge, 1965; Kaniss, 1991) very little quantitative work has attempted to analyze: (1) what stories get covered, (2) why have they been
∗Corresponding Author: spangher@usc.edu 1We release all code and data to our work here: https:
//github.com/alex2awesome/newsworthiness-public
covered, and (3) what impact does the coverage has? Not only could such work increase our understanding of coverage patterns and informational salience perceptions (Hamilton and Fallot, 1974), but it could lead to assistive tools to surface leads for a journalist to pursue (Cohen et al., 2011). To that end, we propose a new task, newsworthiness prediction, to predict whether a story should get covered, according to previous coverage patterns.
Determining that a story, in this work taken to be a local government policy item2 was covered in media, as shown in Figure 1, is a challenging task. Unlike related tasks, like citation prediction (Shibata et al., 2012) or cross document event-coreference (Bagga and Baldwin, 1999), determining policy coverage requires us to establish links between documents in two different linguistic domains, with no pre-existing labels. Our first challenge, in this work, is to establish when a news article references a specific local policy document, i.e. to link them. We show that breaking this problem down into a chain of decisions, each conditional on the pre-
2i.e. A motion of gov.: a proposal, bill, settlement, etc..
14150
vious3, an application of probabilistic relational modeling (PRM) (Getoor et al., 2002), helps us outperform other retrieval-based baselines.
Next, having established links, we seek to predict if a new policy will get covered. We fine-tune language models based on large silver-linked corpora that we identify using the PRM. We find that although F1 is low for coverage prediction, our models are helpful to journalists, beating baseline 84% of the time and surfacing relevant items.
Finally, we ask whether policy text alone is enough to predict coverage. We study recordings of public meetings where policy proposals are addressed. We find that policy items that get covered in news media get discussed slightly longer in meetings and have more members of the public addressing them during public comment periods. However, we find that incorporating these discussions into our predictive models barely yields any performance improvement, indicating that most newsworthy characteristics might not get discussed.
In sum our contributions are:
1. We collect a large multimodal dataset of 13,000 SFBOS policy proposals spanning 10 years, 20,000 SFChron news articles and 3,200 hours of SFBOS meeting video (which we transcribe and diarize), in Sections 2.1, 3.1 in order to study newsworthiness in the local context of one city.
2. We link these corpora with a novel application of probabilistic relational modeling, outperforming modern baselines (Section 2.2). We find that between 2-6% of SFBOS policies get covered in SFChron (Section 2.4).
3. We establish a novel task, newsworthiness prediction, and use it to analyze what makes policy and public discussion newsworthy, finding that newsworthiness is predictable and has a strong non-temporal element (Section 3.3). We show journalists find our rankings helpful in surfacing newsworthy leads.
This work, to our knowledge, is the first to operationalize newsworthiness prediction by focusing on trying to predict historical coverage patterns. As such, we start small in order to prove that this direction is viable, focusing on one locality, San Fran-
3Shown in Figure 2, i.e. “article covers local politics” → “article covers city council meetings” → “covers past meeting” → “covers this past meeting”
cisco Board of Supervisors (SFBOS) and it’s coverage in the the San Francisco Chronicle (SFChron). This means that our analysis is necessarily limited to “newsworthiness” as defined by the SFChron, on SFBOS policy text. However, we intend in future work to expand to different localities.
Press coverage of local policy can be crucial for the health of a community: it can increase civic engagement (Smith, 1987), reduce government malfeasance (Bruns and Himmler, 2011) and engender greater productivity in society (Snyder Jr and Strömberg, 2010). Yet, many newsrooms face severe economic challenges preventing them from providing robust coverage (Fisher and Park, 2022). We hope that our work can open the door to assitive tools to increase journalists’ capabilities to cover their communities.

heading: 2 Policy Item ↔ Article Linking
text: Our goal in this section is to determine which articles cover which policies. We seek to model the likelihood a link l exists between an article, a, and a specific policy item, p, or P (l|a, p).
We apply the probabilistic relational model (PRM) framework (Getoor et al., 2002) to solve this problem. In PRM, we learn conditional attributes h1, ...ht of either the article, policy, or both and marginalize over them:
P (l|a, p) = ∑
h1
. . . ∑
ht
p(l|a, p, h1, . . . , ht) . . . p(h1|a, p)
(1)
Where, as shown in Figure 2, h2 might be “covers SFBOS”, and h3 might be “covers SFBOS votes/policy.”4 (Note that the model p(hi|a, p) = p(hi|a) if the attribute hi is only dependent on the article, a.) Not all politics articles are about SFBOS, and not all SFBOS articles cover policy. Such variety confounds unsupervised models, but is solvable when broken into easier-to-supervise subproblems. This is not dissimilar to Chain-of-Thought (CoT) (Wei et al., 2022), where language models decompose complex reasoning tasks.

heading: 2.1 Corpora: SFBOS Policy-Proposals and SFChron Articles
text: We focus on a specific local government, SFBOS, and a specific newspaper, SFChron, that has a ro-
4Because no natural linking information exists (i.e. hyperlinks in the article body), we typically model l∗ on the text of the article and/or policy proposal.
bust local news section. We start by gathering HTML of all SFChron articles published between 2013–2023 and via the Common Crawl5. We parse article text6 and deduplicate based on text, and ultimately are left with a set of 202,644 SFChron articles7. We also scrape the public meeting calendar on the SFBOS website8 to collect all SFBOS meetings between 2013-20239 and then collect the proposal text for 13,089 SFBOS policy proposals10 that were discussed a total of 27,371 times in 410 public meetings. Each policy is, on average, discussed in 3 separate SFBOS meetings.

heading: 2.2 Devising Relational Chains
text: We manually identify a sequence of hidden attributes, hi, to learn (shown in Figure 2) and handcraft models to learn each one. Each hi is chosen after conducting error analyses to determine which areas the previous learned attributes, h<i, fell short. We hire two journalists11 to annotate data for each hidden attribute, hi, and calculate that their interannotator agreement on these tasks is κ > .8.
1. h1: “a covers SFBOS”. We use the keyword t =“Board of Supervisors” to identify candidates. Then we delete t from these candidates,
5We search for all URLs matching wildcard pattern https://www.sfchronicle.com/*
6Using https://github.com/codelucas/newspaper. 7We release the full list of URLs in our experiment, as well
as scripts to replicate our collection process. 8https://sfgov.legistar.com/Calendar.aspx 9Example meeting: https://sfgov.legistar. com/MeetingDetail.aspx?ID=1108038&GUID= 8B3A2668-90A9-43E9-A694-8747176617F4
10Example of a policy proposal: https://sfgov. legistar.com/LegislationDetail.aspx?ID=6251774& GUID=420031B2-94DE-440F-AB74-25FF091F2D61
11Both journalists were U.S. citizens, but neither was a San Francisco resident. We adjust pay to be roughly $20 USD an hour.
sample negatives and bootstrap a classifier to identify more candidates. Our annotators label 100 of these and we train a classifier p(h1|a).
2. h2: “a covers votes/policy”. From h1 articles, our journalists label an additional 100 articles on whether they mention votes and policy. We train a classifier p(h2|h1, a).
3. h3: “a covers recent policy from SFBOS”. We use GPT3.5 with a 10-shot prompt to determine whether a mentions votes occurring less then a month prior to publication. We also ask GPT3.5 to confirm the government body is SFBOS (e.g. not “Oakland City Council”). We use logits for “yes”/“no” as p(h3|h2, h1, a).
4. l: “a covers policy p.” We match articles to city-council meeting minutes using cosine similarity over the vector space.
All hidden attributes, hi are binary variables, taking values “yes” or “no”. We learn them by training TF-IDF (Ramos et al., 2003) and Logistic Regression classifiers. Each hidden attribute, we find, can be learned with F1 > .8 and less than 100 annotations. For more details about learning hi, see Appendix A.
To learn the final linking model, p(l|a, p), we test different representations for articles and policies: TF-IDF, SBERT with the all-MiniLM-L6-v2 model (Reimers and Gurevych, 2019) and OpenAI’s text-embedding-ada-002 embeddings. (We note that SBERT is run on the first 256 tokens of the article and policy text). Finally, we wish to choose a threshold, λ, for Equation 1, above which items will be considered a match. To help us choose λ and to evaluate our method, our annotators manually identify 100 true pairs, which we split 50/50 into Sgold,train and Sgold,test.

heading: 2.3 Linking Results
text: Our attribute-based model, as shown in Table 1, helps us retrieve (a, p) ∈ Sgold with 68% f1. We show via an ablation experiment that each attribute hi is important for our final prediction: Table 1 shows how F1 drops from 68% to 16% when we remove hi-conditioning steps.
Surprisingly, using PRM with TF-IDF outperforms different embedding methods like SBERT (Reimers and Gurevych, 2019) and OpenAI embeddings (Ryan Greene, 2022). We suspect that specific technical phrases are important for this task, which unsupervised embeddings might ignore; training a supervised retrieval architecture like Dense Passage Retrieval (DPR) might help represent these phrases in the embeddings, but as reported by Karpukhin et al. (2020) requires 100- 1000 times more data than we have collected. Our PRM approach also outperforms retrieval-specific methods like BM25 (Robertson et al., 2009). Overall, these results indicate that attribute-specificity of PRM is crucial12. We note that our PRM approach can seen as a supervised variation of CoT reasoning (Wei et al., 2022) (albeit with a wide beam). As language models become cheaper and more scalable, more directly applying CoT-style approaches to either identify hidden attributes to train auxiliary classifiers, or directly link articles and policies, could be a viable approach.
Despite our positive results, we acknowledge that our approach is limited in several ways. First, as mentioned above, our identification of hidden attributes was based on manual error analysis and, ultimately may not a scale to new domains. Secondly, another limitation we face is that if there is no lexical overlap between a and p, we would not
12To implement BM25, we index a and use p as a search query. We use the retriv Github package: https://github. com/AmenRa/retriv.
discover a link even if there were one. Also, we might be more exposed to this risk than the results show: in constructing Sgold, our annotators might have also faced a similar bias depending on the retrieval mechanisms (e.g. search) they used. A more comprehensive evaluation set would be generated by journalists as they are working on stories. We discuss further limitations in Section 5.

heading: 2.4 Linking Analysis
text: We scale our models across our entire corpus of SFChron and SFBOS articles from 2013-2023. Examining these links gives us insights into the amount of coverage devoted to public policy.
Roughly 7.8% of SFBOS policy proposals get covered, or 1,105 out of 13,089 policies. These policies are covered by a total of 1,828 news articles. Although each policy is covered on average 1.8 times, the distribution is right-skewed and the median coverage is one time per policy. See Appendix B, Figure 6 for more details. The policies that are covered many times are a mixture of staffing (e.g. “Nomination of a Successor Mayor”), transportation bills (e.g. “Unauthorized scooter violations”) and emergency ordinances (e.g. “COVID19 Safe Shelter Operations”.) Again, see Appendix B, Table 10.
Coverage of policies is constant across time. As shown in Figure 3, between 1–3 policies are covered per meeting, out of between 50–60 presented. This equates to between 2%–6% of proposals being covered consistently throughout our 10 years window. Coverage is relatively constant throughout the observation period, removing newspaper decline(Mathews, 2022) as a possible confounder to newsworthiness decisions13. We observe a brief
13Due to an ongoing economic crisis in journalism, many newspapers are shrinking, leading to less coverage and, possi-
spike in % of p covered in 2020–2021. Closer examination reveals that the number of (a, p) pairs stay constant, however, the number of p proposed drops significantly, from an average of 64 policies per meeting prior to 2021, to 52 policies per meeting in the first 9 months of 2021. This is likely the result of COVID-related shutdowns. Conversations with SFChron journalists confirm this.

heading: 3 Newsworthiness Prediction
text: Next, we wish to ask why certain policy proposals are covered. To address this, we establish a new task, newsworthiness prediction: predict, given a policy item p, if an article will write about it. We use our linked dataset {(a, p)}, in Section 2, and treat this problem as a prediction problem where:
Y (p) = { 1, if p ∈ {(a, p)} 0, otherwise
(2)
Our goal is twofold: (1) Learning a good model can show us which features of policy-items lead to coverage. (2) Performing this task well at inference time takes us steps closer to building tools that will be useful for surfacing potential stories.
Previously, newsworthiness has been addressed as a feature-detection problem, as in (Diakopoulos et al., 2010), where engineered-features measured specific criteria14. Journalists examined combinations of features to find newsworthy items but could miss items if their newsworthiness did not fit the measurements. Because we formulate our task as a prediction task, backed by a dataset, we can also expose new and possibly unexpected features. However, a prediction-based approach is limited in its own ways. We assume that past coverage
bly, changes in what is considered “newsworthy”. 14E.g. “statistically anomalous” (Zhao et al., 2014), “sentiment=happy”)
patterns predict future patterns, and that journalists generally agree. We will explore these assumptions in Section 3.3, and we will also see notable cases where these assumptions do limit us.

heading: 3.1 Newsworthiness Training Corpus
text: We extract features from the linked (a, p) pairs derived in the first section to construct our training corpus. As shown in Figure 1, in the news article, there are remarks: “After 14 months of delay”, “widespread debate” that seem to indicate that there aspects of this policy that are not solely related to its topic that made it newsworthy.
To capture some of these features, we include SFBOS meetings where these policies are discussed. We download audio for all meetings in our corpus15 and we use the WhisperX package (Bain et al., 2023) to transcribe and perform speaker-diarization. See Appendix C for more about aligning transcripts. We associate each (a, p) with a specific meeting if: (1) p is discussed in the meeting and (2) a was published within a month of the meeting occurring.
Finally, in every SFBOS meeting, there is a special time for members of the public to speak, called “Public Comment”. Since good newswriting is emotional (Uribe and Gunter, 2007), we hypothesize that “Public Comment” might offer an additional lens on a policy’s newsworthiness. We determine which speakers are members of the public using diarization to identify speakers that only spoke during “Public Comment”16. Then, we calculate the lexical overlap between their speech and the policy
15Example: https://sanfrancisco.granicus.com/ player/clip/43908.
16We infer the sections of the transcript like “Public Comment” using time-stamped agendas, see Appendix C for more detail.
text. For more details about “Public Comment” and other meeting sections, please see Appendix E. We show all of the features that we use for newsworthiness prediction in Table 2.

heading: 3.2 Newsworthiness Descriptive Analysis
text: Before showing results from the predictive modeling, we show descriptive results. Our main takeaway from this section is that policy text, meeting text and public speakers each are conveying different newsworthiness information. We point these out because we will show in the next section, despite clear differences observed in the features that we gathered, not all are semantically useful.
Policy Text, Meeting Speech and Public Comment all cover different newsworthy topics. We see a clear pattern in the kinds of words and topics used in newsworthy policies, meeting speech and public commenters. Table 3 shows the top most likely words in each aforementioned text category, calculated as ∆p(w) = p(w|Y (p) = 1) − p(w|Y (p) = 0). In the written policy text, we observe topic-specific words like “housing”, “covid” and “cannabis” more in newsworthy policies. Topics that were more likely to receive coverage, shown in Table 8, include “Hearings” and “Environment”. However, meeting speech for newsworthy policies (which is primarily speech of the SFBOS Supervisors and staff) is directed at deliberation, like “think” and “know”. Finally, during public comment, we see topic-specific speech, but related to a different set of concerns, like “solar”, “caltrain”, “hotels”. We hypothesize that these are each different aspects of newsworthiness that are being conveyed.
Newsworthy Policies are addressed for longer at meetings, by more people. Policies that end up getting covered in SFChron are also discussed at greater length than policies that are not: this includes (1) more words spoken (Figure 4), (2) more minutes spent discussing (7.7 minutes vs. 2.1), and
(3) more speakers spent addressing it (4 speakers vs. 2.2. This number includes members of the public and council members.) 17.
The number of public commenters we are able to associate with specific policies, on the other hand, is a relatively small number. We are only able to establish an expected n = .06 speaker per newsworthy policy and n = .04 speaker per nonnewsworthy policy. This amounts to 768 speakers associated, overall, with 13,089 policies. Thus, we hypothesize that public comment will not impact our modeling performance, despite observations in Figure 3 that public commenters tend to speak to different topics. We acknowledge this as yet another limitation of our work and dataset. We hope that future work can either (1) establish better methodologies to associate more public commenters with policies (2) collect larger public meeting datasets or (3) incorporate other channels (e.g. social media).

heading: 3.3 Results and Insights
text: In order to jointly model numerical and textual features, we choose to format our features jointly as a prompt. The structure of our full prompt is shown in Table 4, and it includes all features listed in Table 2. We limit the size of the prompt by providing only the first 50 words of the text fields (besides “proposal text”). We do not notice any impact of this truncation in early experimentation. We use this prompt to fine-tune the GPT3-Babbage model, shown to be a robust classifier (Spangher et al., 2023b), outperforming architectures designed for text classification (Spangher et al., 2021a). It could be that the length time spoken is a more important variable than the time spoken itself.
Policy text is the most predictive newsworthiness attribute, followed by meeting discussion and then public comment. In our first set of experiments, we ablate the prompt to explore which components of the policy are the most important for assessing newsworthiness. We perform a temporallybased train/test split hinging on 2021/1/1. We balance our training set, with ntrain = 641/627 (Y (p) = 1/0), and leave our test set unbalanced, with ntest = 180/2310. We perform a time-based
17Journalists gave us initial feedback, saying that city councils sometimes shove important policies into sections of the meeting like “Consent Calendar” and “Roll Call”, which are typically not addressed for a long period of time. This implies either that these cases are truly a minority, or that not enough attention is being paid to these sections of the meeting.
split rather than a randomized split because our goal is ultimately to build a model that can predict future newsworthy items.
We find that the full prompt performs the best across all metrics we considered, but only marginally. As expected, ablating “Public Comment” from the prompt barely impacts performance, while ablating all “meeting info.” impacts a little more. Removing “policy text” information, thus forcing the model to just rely on meeting text alone impacts performance dramatically. GPT3, unsurprisingly, outperforms a very simple classifier, TFIDF+Logistic Regression (LR in Table 5),
but not by much, indicating that there might be simple textual cues that we are learning.
GPT4 might be capturing national newsworthiness trends. Vanilla GPT4 outperformed our expectation. We had hypothesized that many of SFChron’s newsworthiness judgements on SFBOS were local. GPT4 underperforms most other classifiers, but not by much. Manual analysis we perform finds that many errors were GPT4 failing to identify locally newsworthy items (e.g. “local scooter ban”, local street renaming) and that many correct
predictions were made on nationally newsworthy trends (i.e. “COVID-19 responses”). There are two likely conclusions: (1) SFChron has major overlaps for newsworthiness judgements with national newspapers, and (2) general newsworthy language and framing is also used for local newsworthiness.
Newsworthiness judgements are surprisingly consistent across time, with one major exception. Table 3 and Table 8 show that words related to specific events (e.g. those related to “COVID19”) are reflected in the perceived newsworthiness of policy: is the model fitting to a specific event (e.g. “COVID-19”) that happens to be newsworthy in our training and test data, or is it learning either (1) larger event-types (e.g. pandemics more generally, like “ebola”, are recurrent and newsworthy) or (2) newsworthy language patterns and other non-semantic attributes (e.g. framing)?
To test this question, we retrain our model and
increasingly restrict the date cutoffs of our training set to ask whether a model would correctly predict the newsworthiness of policies pertaining to specific events (e.g. “COVID-19”) if the likelihood of them being in the dataset were to decrease. We show in Table 6 that, except for a dropoff after excluding data from 2021, our performance does not significantly change.
To test whether this is the result of GPT3’s pretraining, we test and are able to replicate these findings with baseline Logistic Regression models. An error analysis shows that “COVID-19”related news was the least likely to be predicted correctly, and is the main contributor to this performance decrease, whereas there are numerous other specific events that emerge (e.g. environmental, transportation-related, fire-arms related events.) that our models predict correctly. We take this as evidence that major anomalous events, like COVID19 specifically, do become newsworthy and are unpredictable given our current approach. This highlights an important limitation of our approach, as mentioned in Section 3. These need to be taken into account if these tools are deployed: they must be used along with other models better tuned to these blind spots.
Human journalists find our newsworthiness judgements predictable and helpful. Finally, we recruit two expert journalists18 and conduct human experiments with two aims: (1) is our “newsworthiness” definition repeatable and (2) are our models helpful? For the first, we test how well humans able to identify newsworthy SFBOS policies. We construct a dataset by taking newsworthy policies from SFBOS meetings in our test set and a sampling nonnewsworthy policies in a 1-to-2 ratio of Y (p) = 1, 0. As shown in Table 7, our best models achieve 58.9 F1-score on this dataset, and humans score almost equivalently. It’s tempting to think our models have reached a ceiling; however, the journalists are not San Francisco-based, and are thus untrained, compared to our models.
To test how useful these models can be, we surface 10 policies from each meeting and ask journalists to (a) indicate which policies they might write about and (b) guess whether the list was a newsworthiness list or a random sample (they were told that it was a secondary method, not random). We found, for (a), that journalists preferred our lists to random 84% of the time, and for (b) were able
18Combined have > 40 years of newsroom experience
to guess which list was generated via our method 74% of the time.

heading: 4 Related Works
text: Sociology of News Production Newsworthiness is a well-studied concept in communication and journalism studies, starting most famously with Gatlung and Ruge (1965)’s identification of “news values” like timeliness, eliteness and proximity in international reporting. Kaniss (1991) followed up with work focused on local news values, identifying downtown proximity, economic boosterism and symbolification as key local news values. Each of these works fit into a broader discipline of qualitatively studying newsroom practice, but are typically done via field studies and resulted in descriptive analyses, which could not be operationalized as predictive algorithms.
Local Policy and News Coverage Analysis Prediction We are not the first to gather and analyze local policy discussions at scale (Sorens et al., 2008; Brown et al., 2021; Maxfield Brown and Weber, 2022; Barari and Simko, 2023). Nor are we the first to study broad coverage patterns in local journalism (Hamilton, 2016; Baekgaard et al., 2014; Garz and Sörensen, 2017). Hamilton (2016), notably, studied effects of different coverage patterns in journalism on local government. Besides qualitative work (Felt, 2015), or work focused on social media (Graham et al., 2015), we believe we are one of the first to link news coverage to specific policy.
Link prediction is a well studied field (Kumar et al., 2020). PRMs were introduced (Getoor et al., 2002; Taskar et al., 2003) as a way of modeling attributes, but often suffered from high computational complexities. Our approach (a) uses a relatively small dataset and (b) uses entirely supervised models to ultimately make PRMs tractable here.
Newsworthiness prediction has been approached in different ways. (Spangher et al., 2021b) and (Nishal and Diakopoulos, 2022) sought to learn distant signals for document newsworthiness: either by classifying article layout in newspapers or by collecting attributes from crowd-workers, like “surprising”, “impactful”. Our work more directly addresses the question “will this be written about?” and allows us to study it in a data-driven manner.
This task is also called lead generation (Cohen et al., 2011). Of existing approaches to lead-
generation, one is given by Diakopoulos et al. (2010), where a piece of content’s relevance to a given topic, its uniqueness, and its sentiment is quantified. Then, these metrics surface tweets related to presidential speeches. Such metrics-based systems can be interpretable, but can also miss newsworthy items that are not ranked highly by such metrics. Our work might benefit from including these metrics, and our dataset might learn to rank them well among our other features.

heading: 5 Discussion and Conclusion
text: In summary, we established links between a large corpus of news articles and local policy proposals we did so using a classical method, probabilistic relational modeling, that outperformed retrievalbased methods and embedding-based methods with only a small amount of annotated data. We used the assessed newsworthiness of prior articles to build models to predict the newsworthiness of articles. We found that the performance of our models did not degrade over time, and we found that expert journalists agreed with our newsworthiness assessments and found our tools helpful.
Our work faces many limitations and risks, which we discuss in Sections 2.3 and 3. Notably, we assumed that historical coverage patterns are a reasonable starting point for modeling future newsworthiness predictions. While we found that this yielded useful models, there might be cases where news values evolve and prior decisionmaking is morally and ethically unacceptable, for example with crime (Oliver, 2003) or suicide coverage (Niederkrotenthaler et al., 2020). Our work would serve enforce such historical patterns. Also, it might miss major, atemporal results, like COVID19. Both of these represent considerable risks, and indicate that human involvement remains crucial in any kind of newsworthiness prediction system (a point made by researchers studying other realworld predictive systems (Hong, 2023)).
Despite these risks and limitations, we see this work as presenting a crucial starting point for a larger research direction in newsworthiness prediction. By establishing “newsworthiness” as a welldefined predictive task, we hope to have opened the door to future work applying these concepts. We intend in upcoming work to explore ways to introduce control and explainability into the newsworthiness prediction pipeline that we have outlined here.

heading: 6 Ethics Statement

heading: 6.1 Limitations
text: We discuss a number of methodological limitations throughout our work, namely: (1) our assumptions as to linking articles give us an overreliance on lexical overlap, which is a bias our annotators might also share based on how they chose to retrieve (article, policy) pairs. (2) Past newsworthiness might not always generalize, and might degrade more over time. There are other limitations that exist, though. The datasets we used are all in English, and local to one geography, thus are possibly not representative.
We must view our work in newsworthiness prediction with the important caveat that non-Western news outlets may not follow the patterns. We might face fundamental differences in prediction ability or problem framing if we attempt to do such work in other languages.

heading: 6.2 Risks
text: Since we constructed our datasets using welltrusted news outlets and public meetings, we assumed that every informational sentence was factual, to the best of the journalist’s ability, and honestly constructed. We have no guarantees that such a newsworthiness system would work in a setting where the journalist is acting adversarially.
There is a risk that, if such a work were used in a larger news domain, it could fall prey to learning newsworthiness of misinformation or disinformation. This risk is acute in the news domain, where fake news outlets peddle false stories that attempt to look true (Boyd et al., 2018; Spangher et al., 2020, 2018). We have not experimented how our classifiers would function in such a domain.
We used OpenAI Finetuning to train the GPT3 variants. We recognize that OpenAI is not transparent about its training process, and this might reduce the reproducibility of our process. We also recognize that it owns the models, and thus we cannot release them publicly. Both of these thrusts are anti-science and anti-openness and we disagree with them on principle. However, their models are still useful in a black-box sense for giving strong baselines for predictive problems and drawing scientific conclusions about hypotheses. By the camera ready, we will work to reduce these anti-science thrusts by experimenting with and releasing open sourced LMs. We experimented with them using DeepSpeed to run the GPT-Neo 6.7 model and the
GPT Juno model on a V100 GPU. However, due to time constraints we were not able to get them working in time for submission. However, based on available evidence,19 we expect them to work at a similar capacity and will report results on them separately when we do.

heading: 6.3 Licensing
text: The San Francisco Board of Supervisors dataset we used is released without any restrictions. We have had independent lawyers at a major media company ascertain that this dataset was low risk for copyright infringement. We do not release the San Francisco Chronicle dataset that we gathered, but we do release relevant URLs, which are public domain, and scripts for accessing the Common Crawl.

heading: 6.4 Computational Resources
text: The experiments in our paper required minimal computational resources. We used a laptop computer to run baseline logistic regression and TF-IDF matching experiments. We used OpenAI’s finetuning and prompting architecture to train GPT3 models.

heading: 6.5 Annotators
text: We recruited annotators from two major newspapers that partnered with our institution during this work. They consented to the experiment and were paid at above $20 an hour. Both spent more than 5 years at their organizations. Neither organization is in the same locality as the San Francisco Chronicle. Both annotators are male. Both identify as cis-gender. Both are over 30 years old. This work passed a university Institutional Review Board.

heading: B Descriptive Statistics
text: In this appendix, we start by giving some more detailed statistics of our newsworthiness analysis. Then, we will discuss some data-processing challenges that we faced and overcame. We will discuss how we aligned transcripts with segments of the meetings, and we will discuss in more detail how we found and joined public commenters.
B.1 More Link Analysis In Figure 5, we show the number of times a policy is presented at each meeting, and find a median of 3 times. This aligns with our understanding of how policy progresses from SFBOS; it is introduced, it must be discussed and then it might pass. In cases where a policy is only discussed 1-2 times, it’s more likely that it did not pass.
We also examine the amount of coverage given to policy items. As shown in Figure 6, most policy items are covered between 0-1 times. However, some policy items are covered many, many times. Table 10 shows the bills that have the most coverage. We see a combination of “COVID”-related bills, “nominations” and “transportation”-related bills. While these bills do not materially affect our newsworthiness considerations, since they are more anomalies, they do provide us an opportunity to observe how coverage unfolds over time. In the future, such work could be combined with (Spangher et al., 2022), (Spangher et al., 2023b) and (Spangher et al., 2023a) to provide more of a step-by-step analysis of how coverage of especially newsworthy policies unfolds and grows over time.
B.2 More newsworthiness Analysis
We show in this section additional results from our newsworthiness modeling. In Table 8, we
performed Latent Dirichlet Allocation (Blei et al., 2003) with the number of topics set to k = 8, as we saw a replication of topics after that. We then assigned each policy item to it’s most highlyweighted topic, and counted the number of newsworthy and non-newsworthy policy items associated with each topic. We rank-order topics by the top most-newsworthy topics and the least most newsworthy topics, and shows the top 3 and bottom 3, color-coding them by their newsworthiness ranking.
As can be seen, topics like “COVID-19” and “Environment” are more present in newsworthy items, compared with “City Lawsuits”. We assign titles to the topics based on a manual assessment.

heading: C Aligning meeting transcripts with video
text: We collect 3, 200 hours of video data for SFBOS meetings from their hosted service20 In this section, we describe how we parse the sections of the video that correspond to the policy-items.
Figure 7 shows an example landing page for one SFBOS meeting, held on March 21, 2023. As seen on the left-hand side, the video is shown. On the right-hand side, a nested, hyperlinked agenda document is shown. Capital-letter headers are canonical meeting sections, and are relatively constant across meetings.
Some of the lines are shaded in blue, meaning that they are hyperlinked to a timestamp in the video. In the agenda, any line that starts with a 6-digit code refers to discussions around a policyitem that the SFBOS wishes to pass. Some of
20Called Granicus, which is a service provider used for many local governments.
the policy-item lines have hyperlinks pertaining to them while others do not.
We manually examined agendas. Many links were missing simply because several policies were discussed together in the agenda. However, others seemed to be missing randomly, leading us to believe that the agenda hyperlinks were incompletely linked.
We wish to reconstruct as completely as we can the time-stamped agenda so that we could get an accurate segmentation for the meeting, so we aim to fill in the missing agenda items. We explore a very simple hypothesis: we assume that meetings were highly organized, and there were consistent phrases used to transition to different agenda items.
So, we seek to classify transitionary phrases. We train a classifier that takes as input a list of diarized transcriptions, t, which each have their own start and end times (ts, te) annotated from the transcription process, and predicts:
Y (t) = { 1, if ∃ hyperlink ∈ (ts, te) 0, otherwise
(3)
We recognize this is noisy, as Y (t) can = 0 if both: (A) a segment is not a transition or (B) it is simply missing a transition. However, we train a simple classifier using bag-of-words representations for diarized segments and logistic regression, and we achieve f1-score=.86 on held-out data. We analyze the outputs of the classifier and see that it discovers relevant transitional speech, see Table 11 for examples.
Having labeled our transcripts with each diarized segment’s likelihood of being transitional speech, we then iterate through each transcript and peg each unlabeled block to either: (a) the next most likely transitional segment or (b) the previous item’s segment, if no segment exists above .8 likelihood. In this way, we allow multiple agenda items to be discussed in the same short segment.
This is an exceedingly simple approach that does not consider semantic similarities between the meeting agenda as an approach like, say, dynamic time warping () might. We are confident that our approach could be improved, and maybe in future work improvements could result in additional signal being observed.

heading: D Additional Joining Information
text: When we extract agendas in the SFBOS video viewer, as shown on the righthand side of Figure 7, we find that we are able to retrieve a total of only 10,877 out of 13,089 policies which were listed on the SFBOS legislative calendar website as being discussed during meetings. This is strictly a subset. All policies gathered from video viewer agendas are listed in the SFBOS legislative calendar website.
It’s likely that this discrepancy results from policies that were introduced but did not make it past preliminary stages of investigation. For instance, here is an example of a proposal that was listed in the legislative calendar website: https://sfgov.legistar.com/ LegislationDetail.aspx?ID=2070276&GUID= D31163A0-D5F8-41E7-AB90-4DECAF9E6693 as having been presented during a SFBOS meeting on 11/18/2014. However, the actions
associated with that item, as told in the website, are: “RECEIVED AND ASSIGNED”, “REFERRED TO DEPARTMENT”, “TRANSFERRED”, and “FILED PURSUANT TO RULE 3.41”. Here is the video page of the 11/18/2014 meeting: https://sanfrancisco. granicus.com/player/clip/21460. As can be seen, policy number 141197 is not listed in the agenda. That is different from, say, this proposal: https://sfgov.legistar.com/ LegislationDetail.aspx?ID=6122328&GUID= B5231DEE-0596-463F-8934-A84468D131ED, which was “CONTINUED” and “HEARD AND FILED”, with meeting details associated with each one.
So, a logical explanation is these policies that were never brought to discussion during meetings, thus they do not appear in meeting agendas. However, we cannot discount the possibility that errors were made in creating the agendas. In this case,
we were not able to track policies that were genuinely discussed. Nevertheless, we will refer to these policies as “unpassed-policies”.
This affects 164 unpassed-policies that we have identified as being covered by SFChron, out of a total of 1,015 policies, or 16% of newsworthy policies. These unpassed-policies were covered 298 times, out of a total of 2001 articles. We give a sampling of these missing newsworthy unpassedpolicies in Table 14. While it’s entirely likely that the fact that these policies were not discussed lead to them being newsworthy, we do not consider such a distinction in our modeling. We leave this to future work.

heading: E Additional Meeting Exploration
text: Having parsed each agenda and time-pegged each line-item in the agenda, we are able to roll up the time spent in each section-header. Table 13 shows the length of time spent in each section.
As can be seen, the “PUBLIC COMMENT” section occupies a major part of meetings, in terms of the amount of time spent in each meeting, and yet very few policy-proposals are explicitly discussed during this period. We hypothesize that public comment is a potentially newsworthy period in the meeting, where members of the public are able to raise the emotional tenor of a piece of policy (which makes for good news-writing (Uribe and Gunter,
2007)). So, we attempt to join publicly-made comments to entire text of the policy discussion.
As discussed in the main body, we defined “public commenters” as members of the public who only speak during the “PUBLIC COMMENT” section of the meeting. Given timestamps for this section, and speaker diarization, we are able to filter out all speakers besides those that speak during public comment. Next, we use word-overlap between the speaker’s speech and the policy text to determine whether the speaker is addressing a particular topic. For the sake of brevity, we assume that each speaker only addresses one comment.
We show in Table ?? some examples of public commenters. As can be seen, they address policy with a personal tenor. However, there are also comments that are rather noisy (e.g. meandering, not on topic, not very focused.) We feel that more work is needed to make the public comment section a usable part of this analysis.

heading: Future Work
text: We leave this to future work