Large Scale Narrative Messaging around Climate Change: A Cross-Cultural Comparison (2024)

Abstract: In this study, we explore the use of Large Language Models (LLMs) such as GPT-4 to extract and analyze the latent narrative messaging in climate change-related news articles from North American and Chinese media. By defining “narrative messaging” as the intrinsic moral or lesson of a story, we apply our model to a dataset of approximately 15,000 news articles in English and Mandarin, categorized by climate-related topics and ideological groupings. Our findings reveal distinct differences in the narrative values emphasized by different cultural and ideological contexts, with North American sources often focusing on individualistic and crisis-driven themes, while Chinese sources emphasize developmental and cooperative narratives. This work demonstrates the potential of LLMs in understanding and influencing climate communication, offering new insights into the collective belief systems that shape public discourse on climate change across different cultures.

heading: 1 Introduction
text: Understanding the stories we tell is a key priority for those engaged in climate discourse. Stories serve as a fundamental method through which people exchange information and forge shared understandings about cause and effect in our world, essentially explaining “why things occur” (Todorov, 1981; Herman, 2009). Studies indicate that storytelling can be an effective means of overcoming resistance to new ideas and changing people’s intentions to act (Shen et al., 2015; Braddock and Dillard, 2016; Ratcliff and Sun, 2020). Consequently, there is significant interest among climate advocates in leveraging the power and influence of narrative to shift public perspectives (Fløttum and Gjerstad, 2017).
In this paper, we propose a method for surfacing the latent narrative messaging of news stories related to climate change. Narrative messaging
refers to an overarching, higher-level message that a given story conveys to its readers, one that may be more or less explicit in the body of the story. Narrative messaging is thus akin to broader narratological concepts such as “schemas” (Brewer and Lichtenstein, 1980; Russell and Van Den Broek, 1992), “archetypes” (Campbell, 2008; Frye, 2020), “frames” (Entman, 1993), and “meta-narratives” (White, 2014). Despite addressing narratives at varying levels of abstraction, these models converge on a fundamental premise: stories inherently share common elements, and their selection is orchestrated by higher-level schemas or messages that shape the narrative’s construction and interpretation.
For our purposes here, we define narrative messaging as consisting of a story’s “moral” or “lesson,” i.e. an intrinsic message that readers are intended to take away that transcends the specific details of the story. For example, in a fable such as “The Lion and the Mouse,” the moral / message of the story is “a kindness is never wasted.” In a news article about climate change that focuses on policy disputes, the moral / message might be “Political compromise is important for finding solutions.” While we typically think of story morals as reserved only for short didactic fiction (such as fables), narratologists have long argued that all narratives have implicit value-driven schemas that govern how they are told (Booth, 1998).
As a narrative message, a story moral focuses on the values and intentions of the storyteller. The moral or lesson of a story is in some sense an answer to the question, “Why is this person telling me this story?” Rather than focus on the specific content of the story, attention to story morals focuses on a more general lesson to be learned (“economic interests and energy security concerns often hinder global consensus on phasing out fossil fuels”). Surfacing such latent narrative values at large scale can facilitate the process of un-
143
derstanding collective beliefs around particular societal concerns, their differences across cultures, and any meaningful changes over time.
In our work, we leverage the affordances of large language models (LLMs) like GPT-4 to extract a series of narrative features associated with a given news story up to and including the story’s central “moral” or “lesson” (Fig. 1). Such features include the identification of the main agent of the story, the central topic of the story, any antagonist or negative agent, the story’s overall valence, along with a free-form moral that is also rendered as a single keyword or phrase.
LLMs present a promising avenue for automating the labeling of story morals. Despite ongoing challenges with hallucinations in LLMs (Xu et al., 2024), their ability to infer underlying meanings―akin to deriving morals from narratives that are often implicit rather than explicitly stated―
aligns well with their capabilities. Furthermore, the ubiquity of narratives and narrative-like moral statements on the internet suggests that these concepts are likely well-represented in LLM training datasets. However, this also necessitates a critical evaluation of potential cultural biases embedded within these models.
We proceed first with a review of prior work related to our topic, particularly with respect to prior work on climate communication. We then introduce and validate our method using a combination of automated metrics and human annotation. Finally, we apply our method to the study of a collection of ca. 15,000 news articles written in English and Mandarin that are subsetted by different climate-related topics and different ideological groupings (state / offshore, left / right, see Table 4). We explore techniques of aggregating our story morals to identify salient differences in the larger narrative messaging surrounding issues related to climate change across North American and Chinese-language media.

heading: 2 Prior Work
text: In the field of “environmental communication,” the notion of “story morals,” as investigated in this study, closely aligns with the concept of “framing,” which has been extensively examined in prior research. Framing, sometimes also known as schemas, involves highlighting certain aspects of perceived reality in communicative texts to promote specific problem definitions, causal interpretations, moral evaluations, and recommended solutions (Entman, 1993). Framing’s significance is particularly evident in discussions on environmental issues, where climate change represents a “super wicked problem” characterized by the urgent need for action, yet hindered by delayed impacts and insufficient institutional efforts (Levin et al., 2009; Lazarus, 2009; Rodrigo-Alsina, 2019). Consequently, effective framing is essential to bridge the gap between awareness and action in environmental protection (Pan and Kosicki, 1993; Lakoff, 2010; Bushell et al., 2017; Fløttum and Gjerstad, 2017).
A substantial body of literature has explored various frames in climate discourse, such as “social progress,” “scientific uncertainty,” and “conflict” (Nisbet, 2009; Tong, 2014; Bolsen and Shapiro, 2018), covering different periods and geographic regions (Anderson, 2009). Some studies compare
climate issue framing in various nations (Brossard et al., 2004; Boykoff, 2007; Xie, 2015); while others perform temporal analyses correlating media coverage with significant climatic and political events, such as the COP and Kyoto Protocol (McComas and Shanahan, 1999; Young and Dugas, 2011; Keller et al., 2020; Pan et al., 2021).
Traditionally, these studies have relied on manual coding methodologies, where coders are trained to identify specific elements in articles, such as scientific controversies, typically resulting in a dataset comprising a few hundred articles. Recently, however, automated methods like topic modeling have been adopted in climate framing research to enhance data analysis efficiency (Keller et al., 2020; Rabitz et al., 2021). Our work can be seen as a further extension of such automated approaches, but with a novel focus on the values of narrative messaging by leveraging the affordances of LLMs.
Within the NLP community, the analysis of narrative understanding has gained significant interest (Ranade et al., 2022; Clark et al., 2022). This research encompasses the aim of narrative detection and understanding, across varied contexts, such as literature, social media, and health-care related communication (Ganti et al., 2023; Antoniak et al., 2023). Recent research in this area has also begun exploring the idea of “collective narratives” which involves synthesizing smaller narrative elements (such as tweets, blog entries, or news articles) into overarching narrative frameworks (Zhao et al., 2023; Shahsavari et al., 2020). The research collaborative Climate Change AI has been an important early mover in bringing together the ML and NLP fields with climate change concerns (Rolnick et al., 2022).
Our work builds off this prior work by bringing together the approaches and theories of computational narrative understanding towards the goal of studying climate-related communication in different cultural contexts. Instead of applying manually labeled codes to smaller collections, we show how GPT-4 can help surface intrinsic narrative schemas related to the implicit values driving large volumes of news production and that those schemas align with human judgments. In addition to scaling up our understanding of climate-related communication, our approach also shifts the focus from content-related questions (i.e. “what happened”) towards more value-driven questions (i.e. “why is
this being told?”). Doing so, we argue, can help surface important insights into the collective and often latent belief systems that govern what stories get told and how.

heading: 3 Story Morals

heading: 3.1 Model
text: We define a “story moral” as a general lesson that the narrator wishes to impart to the audience about the world. While the idea of the “moral” is often associated with a particular ancient narrative tradition,1 all stories are theoretically governed by a higher-order message that the storyteller wishes to convey, consciously or unconsciously, to guide or reinforce the audience around some belief or a goal. Such messaging is an implicit component of the narrative “schema” that shapes how a story is told and what aspects of the world the narrator chooses to focus on.
While some prefer to use the concept of “framing” to capture these latent narrative schemas around media communication, we prefer the concept of the “story moral” because of the way it draws attention to the behavioral values associated with any given story. The moral of the story is something we can use to guide future actions and thus is explicitly related to behavioral effects (whether it achieves those is a different question).
In order to surface the “story moral” for a given news article, we employ the prompting sequence as described in Table 1. We first extract a summary to help the model focus on key narrative elements. We then identify principal agents, such as the protagonist and antagonist, the central topic of the story, a free-form moral and moral keywords that assume positive and negative valence.
We experiment with two prompt flow frameworks: a full-context pipeline, where all prompts are given cumulatively (including the summary and the original text) so that each prior prompt and its answer are included in the subsequent prompt. Alternatively, we experiment with a simplified framework with only the summary as the context of each question to reduce cost and compute resources. Fig. 1 illustrates an example output for a sample news story. All prompting exercises were done using GPT-4 (specifically, 0125-
1While Aesop’s Fables are the best-known genre associated with story morals in the West, similar types of tales exist in both Hindu (Panchatantra) and Buddhist (Jatakas) traditions that date back to around the fifth century BCE indicating the genre’s trans-cultural significance.
preview) through OpenAI’s API and using a temperature of zero to minimize output randomness.

heading: 3.2 Validation
text: For the purposes of validation, we use a combination of human assessment and automated metrics. In order to understand GPT’s performance across different cultural settings, we use a test dataset of 64 news articles drawn from political news spanning CNN, Al-Jazeera English and four sources of Chinese-language news (described in Table 4). The mean length of documents is 987 words with a minimum of 250 and a maximum of 2,200.
To compare to reference answers, we employed a group of undergraduate students to provide answers to the prompts in Table 1 for each passage (with the summarization question omitted). Six native English-speaking and four native Mandarinspeaking student annotators were hired. Annotators were provided with a codebook of category definitions and examples, and underwent at least one round of practice annotations to affirm consistency of interpretations to the definitions. All human responses were open-responses made in English, and were made independently of each other and from GPT-4.

heading: 3.2.1 Human Evaluation
text: For the more deterministic categories (protagonist and antagonist), we measured direct agreement between GPT and majority / any human annotations. Multiple GPT responses (equal to the number of human annotators) were collected to enhance robustness. Averaged over the sets of GPT responses, we found an average of 49% / 61% (protagonist) and 71% / 97% (antagonist) for the majority / any agreement conditions. Table 5 in the Appendix has the full details.
For each of the more open-ended categories of Moral, Positive Moral Keyword, Negative Moral Keyword, and Topic, we used the following approach involving Amazon’s Mechanical Turk platform (AMT) to determine applicability and preferences for human- vs. machine-generated answers. Crowd workers were presented with three options, one from GPT-4 and two that were randomly selected from among the human annotators. The crowd workers were tasked with choosing the “most applicable” and “least applicable” options for each category given the passage text. Crowd workers were given no explicit instructions about what constituted a good or bad option and were given the freedom to select based on their own preferences, so as to avoid any selection bias.
To ensure quality responses, we required workers to have a lifetime success rate of more than 95%, and workers had to correctly answer a passage comprehension question to be considered. To partially address new concerns among researchers of crowd workers using ChatGPT to answer the questions, all passages were provided as images.
For passages in Mandarin, English translations of the text were provided to the workers to ensure we were drawing on the same pool of annotators. While the use of translations may have modified the essence of some of the articles, manual inspection of the translations deemed them to be accurate and of high quality. Nevertheless, we acknowledge that the use of translations may introduce potential cultural disparities as the morals are not being rated by individuals from the same cultural background as those who produced the morals. Future work should seek to expand the crowdsourcing validation to more closely study the differences between cultural and linguistic groups. All the same, the results shown below show no significant differences in the overall preferences between the two languages.
In total, responses from three AMT workers were collected for each passage. For evaluating the full-context prompt framework, the full dataset was used, and for the simplified prompt frame-
work, a subset of 32 articles, with an equal split among all news sources, was used.
As seen in Table 2 for the simplified prompt framework, for each category we achieved majority agreement in 75-90% of cases. Inter-rater agreement was extremely low, however, because while two annotators may have chosen a human moral they may have chosen different ones. Nevertheless, we observe that the GPT morals and central topics were selected well above a random baseline, and the positive and negative morals were no worse than random, as indicated by a χ2 goodness of fit test. In no case did crowd-workers preferentially choose GPT answers as “least applicable” in a statistically significant way.
Table 7 in the Appendix shows comparable, albeit slightly better, results for the full-context framework. This can likely be attributed to the fact that the exclusive use of summaries in the simplified prompt framework occasionally omits elements that are pertinent to constructing a good moral. This notwithstanding however, these morals from GPT were still favored over the human morals by the AMT workers. As neither prompt workflow showed a negative preference for GPT responses, we elected to employ the simplified framework for our full analysis.

heading: 3.2.2 Automated Validation
text: For the automated validation of morals and central topics, we used a group of semantic textual similarity (STS) scores relevant to our human annotations These included ROUGE-1 and ROUGE-L (Lin, 2004), and cosine similarity using pretrained embedding models from the SentenceTransformers library (Reimers and Gurevych, 2019). ROUGEbased metrics were implemented using the HuggingFace library and all embedding models were implemented using the SentenceTransformers library (Reimers and Gurevych, 2019). For these latter models, the specific models included averaged GloVe word embeddings (6b-300d) (Pennington et al., 2014), and the stsb-mpnet-base-v2 and nlimpnet-base-v2 models (Song et al., 2020).
Table 3 shows a complimentary picture using the automated evaluation metrics to compare the human responses to those from GPT on a single example of our categories (e.g. moral) (see Table 8 for full details). All pairs of responses were compared for a given story which were then combined to create a single distribution of pairwise similarity between annotations. The human-human column indicates the median of the distribution comparing only human to human responses, the human-GPT column indicates the median in only comparing human responses to GPT responses, and the GPTGPT column compares GPT responses to other GPT responses for replicability. As noted above, the number of GPT responses was chosen to be the same as the number of human annotations.
Overall, we find that the semantic variation between human responses is higher (i.e. exhibits lower similarity) than that between GPT and human responses across all metrics suggesting that GPT is decently approximating an aggregate human point of view. As Table 8 in the Appendix
indicates, positive and negative morals are exceptions with respect to some metrics, however the differences are small even when statistically significant. We also note that GPT exhibits very high similarity scores to itself on multiple runs for the same text though these still exhibit some variation. Central Topic is an exception where GPT always repeated its answer verbatim.
Finally for valence, the average standard deviation between human-only responses was 0.68 across all passages, compared to 0.66 when introducing GPT responses, thus showing good compatibility between the responses. Details can be found in Table 6 in the Appendix.

heading: 4 Analysis

heading: 4.1 Data
text: Our dataset for this study comprises approximately 15,000 news articles sourced from Dow Jones Factiva, segmented by language into Mandarin and English (see Table 4). The articles are filtered by length to fall between 250 and 2,500 words and published during the calendar year 2023. They were selected using five key terms associated with environmental issues: “climate change” (气候变化/氣候變遷 for Taiwanese sources), “pollution” (污染), “carbon emissions” (碳排放), “renewable energy” (再生能源), and “sustainability.” For “sustainability,” we utilized the Chinese term huan bao (环保), which connotes “environment-friendly.” This term was chosen because it is prevalent in Chinese environmental discourse, whereas its direct translations occur less often in English contexts. This strategic choice of keywords ensures that our dataset robustly represents significant environmental discussions within each language’s media landscape.

heading: 4.2 Valence
text: A linear regression analysis reveals that both region (North American vs. Chinese sources) and ideology (Liberal vs. Conservative, state vs. offshore) significantly affect average valence. Specifically, being in North America is associated with a lower valence (M=2.87) compared to Chinesespeaking regions (M=3.45). Mainland Chinese sources are in turn more positive (M=3.56) than offshore sources (M=3.29) most likely due to censorship and the decline of critical voices in state media (Guo et al., 2023). While statistically significant differences exist between Conservative (M=2.93) and Liberal (M=2.83) sources, they are the most similar.

heading: 4.3 Distinctive Values
text: Next we use the Fightin’ Words model for lexical feature selection (Monroe et al., 2009) to identify salient differences in moral keywords between our cultural subsets. As can be seen in Fig. 2 (top), Chinese and North American sources exhibit different moral focuses: Chinese sources emphasize promoting “international cooperation” and “sustainable development,” while North American sources concentrate on “addressing” and “adapting” to climate change, reflecting a potentially stronger sense of crisis, but also individualism. China’s call for international cooperation corresponds to findings from previous studies that climate change has transitioned from a concern primarily addressed by developed nations to a global issue where China is actively engaged and takes a proactive stance (Pan et al., 2021). The Chinese emphasis on “promoting” solutions contrasts with a North American emphasis on reacting to and accepting consequences. The developmental framing of Chinese-language news as a whole suggests
a far more proactive stance than the North American one.
Within North America (Fig. 2 (middle)), Conservative outlets place a much stronger emphasis on markets, investments, and economic and financial issues. This reflects the 2019 Pew Research Center survey that shows Conservative Republicans being skeptical towards climate policies — a majority (62%) of this group says these policies hurt the economy (Hefferon, 2019). We also note that Conservative media view climate change through the “adapting” lens, while Liberal media emphasizes the more pro-active “addressing” lens. Conservative messaging focuses on climate change as something to be lived with and accommodated, whereas Liberal media tends to view the problem more holistically as impacting human health and the natural environment.
In Chinese-language outlets, state media centers its discourse on development, underpinned by values of sustainability and environmental consciousness. Whereas offshore (Hong Kong and Taiwan) media value “sustainability” as an end in itself, state media focus on “sustainable” (and “green”) as a modifier of development. This also aligns with findings from previous studies that climate change was no longer viewed solely as an environmental obstacle to socio-economic development in China, but rather as a manageable challenge that drives and creates opportunities for economic growth (Pan et al., 2021). In contrast, offshore Chinese media tend to focus more on the individual level, emphasizing everyday life with keywords such as “community,” “recycling,” and “local,” which can partly be attributed to the smaller size of Hong Kong and Taiwan.

heading: 4.4 GPT-Assisted Open Coding
text: One of the challenges of the Fightin’ Words approach is generalizing about larger trends to which individual morals align. To address this limitation, we engaged ChatGPT-4 in the process of open coding (Strauss and Corbin, 2004). We first gave GPT truncated lists of the most distinctive moral keywords based on the Fightin Words method and asked it to devise 5-6 categories that best represented the terms. These categories were reviewed for appropriateness by the authors, headings were adjusted for brevity, and then a list of the 200 most distinctive keywords was inputted with the request of assigning them to their respective categories (with no overlap). Not all words were assigned and not all assignations were agreed upon by the authors and so a round of manual adjustments were undertaken.
The final rubric consisted of six categories: Community & Justice, Conservation, Economy, Preparedness, Governance, and Innovation. Each
category has a unique set of keywords, for which we extracted the normalized counts and compared North American and Chinese sources as well as Conservative and Liberal ones. The results, shown in Fig. 3, align with our findings from Fightin’ Words. Specifically, Chinese media shows a significantly stronger focus on issues related to governance and community participation, while North American sources emphasize the importance of facing risks and promoting innovation.
Fig. 3b reveals distinct differences in keyword emphasis between Conservative and Liberal sources. Liberal sources have higher normalized counts for Community, Conservation, and Preparedness, suggesting a strong focus on social justice issues and environmental protection. In contrast, Conservative sources emphasize Economic and Innovation categories more, indicating a prioritization of economic growth and technological advancement.
Previous studies have found significant differences in frames employed by Conservative and Liberal media. The Wall Street Journal, for example, has been shown to use more frames emphasizing negative economic consequences, suggesting that proposed solutions are unlikely to be effective, and highlighting political conflict (Feldman et al., 2017). However, based on our keyword analysis and valence comparison, we can speculate that recent articles from Conservative media continue to prioritize the economy while addressing environmental topics, albeit with a more positive frame.

heading: 5 Conclusion
text: Our study illustrates a workflow that can be applied to understand the narrative messaging of values around climate change across different cultural contexts. We show the robust validity of large language models like GPT-4 to derive highlevel conceptual information about narratives in strongly different cultural and linguistic contexts. In particular, we surface key “values” associated with climate-related news, with Chinese media focusing on a more “developmental” approach compared to a more “adaptive” approach on the North American side.
Such workflows will be an important dimension towards scaling up our understanding of climate communication. While we focus on surfacing implicit narrative values around climate change from the bottom-up, researchers can also
use our method to test more specific hypotheses and content-related “frames” determined in advance. As we discuss in the limitations section, more work is necessary to better understand the biases or norms implicit in LLMs. Nevertheless, we believe LLMs are going to be an important tool in understanding, interpreting, and influencing climate communication moving forward.
Limitations
Understanding climate communication at a large scale poses a number of research challenges. While we look at eight different news outlets across two different national and ideological contexts, wider sampling and including more cultures will be an essential next step as we scale-up this work. Our sample is also limited by the keyword filtering such that future work might explore other ways of identifying a fuller sample of climaterelated communication.
While we observe strong levels of humanjudged validity in terms of the appropriateness of GPT-generated morals, more work can be done to understand intercultural differences surrounding the perception of narrative messaging. Additionally, it is important to note that while GPTgenerated content is marked by high levels of semantic relatedness across multiple runs of the same queries there is still some observed variability even when the temperature is set to 0 making exact replication unlikely.
Another important limitation here is the dependence on GPT-4 as the primary LLM. Future work will want to explore the behavior of other large frontier models as well as the ability to employ smaller, specialized models to avoid the large carbon footprint of the bigger models.
Finally, our results also offer numerous avenues for further exploration beyond the methods presented in this paper. Future research can employ different clustering methods, compare these methods with existing methodologies like topic modeling, and apply automated coding by LLMs to answer questions that were previously human-coded. This will enable a more direct investigation into the nuances of surrounding narrative messaging as it relates to climate change.

heading: Acknowledgements
text: We would like to thank the reviewers for their constructive feedback and suggestions. This research
was supported by the Natural Sciences and Engineering Research Council and the Social Sciences and Humanities Research Council of Canada.

heading: Future Work
text: Our sample is also limited by the keyword filtering such that future work might explore other ways of identifying a fuller sample of climaterelated communication. While we observe strong levels of humanjudged validity in terms of the appropriateness of GPT-generated morals, more work can be done to understand intercultural differences surrounding the perception of narrative messaging. Additionally, it is important to note that while GPTgenerated content is marked by high levels of semantic relatedness across multiple runs of the same queries there is still some observed variability even when the temperature is set to 0 making exact replication unlikely. Another important limitation here is the dependence on GPT-4 as the primary LLM. Future work will want to explore the behavior of other large frontier models as well as the ability to employ smaller, specialized models to avoid the large carbon footprint of the bigger models. Finally, our results also offer numerous avenues for further exploration beyond the methods presented in this paper. Future research can employ different clustering methods, compare these methods with existing methodologies like topic modeling, and apply automated coding by LLMs to answer questions that were previously human-coded. This will enable a more direct investigation into the nuances of surrounding narrative messaging as it relates to climate change