CliME: Evaluating Multimodal Climate Discourse on Social Media and the Climate Alignment Quotient (CAQ)
Abhilekh Borah [], Hasnat Md Abdullah [], Kangda Wei, Ruihong Huang
1 Manipal University Jaipur, India 2 Texas A&M University, USA
abhilekh.229301149@muj.manipal.edu, hasnat.md.abdullah@tamu.edu, kangda@tamu.edu, huangrh@cse.tamu.edu
Abstract
The rise of Large Language Models (LLMs) has raised questions about their ability to understand climate-related contexts. Though climate change dominates social media, analyzing its multimodal expressions is understudied, and current tools have failed to determine whether LLMs amplify credible solutions or spread unsubstantiated claims. To address this, we introduce CliME (Climate Change Mul timodal Evaluation), a first-of-its-kind multimodal dataset, comprising 2579 Twitter and Reddit posts. The benchmark features a diverse collection of humorous memes and skeptical posts, capturing how these formats distill complex issues into viral narratives that shape public opinion and policy discussions. To systematically evaluate LLM performance, we present the Climate Alignment Quotient (CAQ), a novel metric comprising five distinct dimensions: Articulation, Evidence, Resonance, Transition, and Specificity . Additionally, we propose three analytical lenses: Actionability, Criticality, and Justice, to guide the assessment of LLM-generated climate discourse using CAQ. Our findings, based on the CAQ metric, indicate that while most evaluated LLMs perform relatively well in Criticality and Justice, they consistently underperform on the Actionability axis. Among the models evaluated, Claude 3.7 Sonnet achieves the highest overall performance.
We publicly release our [CliME](https://huggingface.co/datasets/climedataset/CliME) dataset and [code](https://github.com/abhilekhborah/CliME) to foster further research in this domain. 1 Introduction
Climate change has emerged as one of the most pressing challenges of our time, not only in scientific and policy circles but also in the public imagination. A 2023 study found that YouTube had the most significant positive effect
∗ These authors contributed equally to this work. Description: The image and caption highlight the environmental policy of the Trump administration, specifically focusing on the withdrawal from the Paris Climate Agreement. The post emphasizes the risks associated with this decision, such as increased greenhouse gas emissions and the potential for climate-related disasters. It also highlights the Green New Deal as a potential solution, emphasizing the need for a comprehensive and immediate response to climate change.
The post encourages action and emphasizes the urgency of addressing climate issues. Figure 1: CliME sample data: Each data point includes a climate change-related image from a Reddit or Twitter post, the accompanying post text, and a generated description integrating both the image and text. on climate awareness in Latin America, followed by Instagram and Twitter, demonstrating the role of these platforms in disseminating climate-related information. Social media platforms like Twitter (now X) and Reddit have emerged as prime spaces for climate discourse, shaping public opinion, mobilizing activism, and influencing policy. Viral campaigns such as #FridaysForFuture, #ShowYourStripes and #EarthHour illustrate the power of social networks in transforming grassroots efforts into global movements. However, these online platforms propagate misinformation and polarization, vastly associated with skepticism, contrarianism, and denial. In 2023 alone, posts denying climate change on X tripled compared to previous years, highlighting the persistence of climate denial narratives online. The rise of multimodal content in this domain further complicates this landscape. A flooded city image paired with “Climate policies harm the economy” spreads faster than peer-reviewed data, exploiting
Image:
Text: You don't stand a chance
ClimateMemes
Memes4Climate ClimateEmergency ClimateCrisis
GreenNewDeal
EnvironmentalJusti
ce visual-emotional resonance. The rising hype of LLMs show their outstanding capabilities at text generation but their proficiency in grounding responses in visual-textual context in climate communication, remains unproven. Early studies reveal concerning trends: when prompted with climate-skeptical posts, models like GPT-4 often default to vague, non-committal language. For instance, a study by NewsGuard found that GPT-4 was more likely than its predecessor GPT-3.5 to generate misinformation when prompted with false narratives, including those about climate change, without sufficient disclaimers or corrections. This mirrors the \"false balance\" tactic often used to undermine scientific consensus, where both sides of an issue are presented as equally valid despite overwhelming evidence favoring one side. To address these challenges and assess whether LLMs can interpret and generate credible climate communication, we introduce CliME, a benchmark comprising 2579 pairs of posts scraped from Reddit and Twitter, with most of the data focused on memes, infographics, and skeptic content. We then generate descriptions of these images and texts using DeepSeek Janus Pro followed by human annotation. These descriptors serve as the basis for evaluating LLMs’ capacity to comprehend and address climate change, guided by our proposed three key lenses: (i) Actionability, (ii) Criticality, and (iii) Justice. To assess the LLM responses generated through these lenses, we introduce the Climate Alignment Quotient (CAQ), a novel metric that quantifies gaps across five critical axes: (i) Articulation, (ii) Evidence, (iii) Reso nance, (iv) Transition, and (v) Specificity, thereby determining the extent to which LLMs capture intrinsic climate knowledge. Figure 2 illustrates the entire workflow. In summary, our contributions are as follows: (i) A first-of-its-kind multimodal benchmark, CliME, primarily featuring climate change related memes and skeptic content from social media. (ii) Three analytical prompting paradigms: Ac tionability, Criticality, and Justice lenses, designed to investigate LLMs and assess their ability to interpret and generate credible climate discourse. (iii) The Climate Alignment Quotient (CAQ), a metric to measure the intrinsic alignment of LLMs in climate communication. 2 Related Works
Climate Communication in Social Media. Grundmann and Krishnamurthy, 2010 pioneered the analysis of climate change discourse through textual content, comparing climate-related word frequencies in news articles between Europe and the USA. With the rise of multimedia platforms like Twitter (X), Instagram, TikTok, WhatsApp, and YouTube, social media has been shown to enhance public awareness of climate issues. Abdallah and Youssef, 2023 further demonstrated a positive correlation between climate-related social media content and increased public awareness. Studies highlight the effectiveness of personalized, relatable content in engaging audiences León et al., 2022, and social media is increasingly viewed as a trusted information source Hamed Mussa, 2023. However, misinformation and echo chambers remain significant barriers to active engagement Abdallah and Youssef, 2023. Hence, the analysis of climate change discourse on social media remains a rapidly evolving field that requires further attention. LLMs in Climate Change Discourse Analy sis. Recent work has applied LLMs to analyze climate change discourse. ChatREPORT, using ChatGPT with expert-designed prompts, examined 9,781 corporate sustainability reports to evaluate climate action. Thulke et al., 2024 introduced ClimateGPT, a domain-specific LLM trained on 300 billion tokens, validated through benchmarks and human evaluation. Zhou et al., 2024 leveraged GPT-4 to uncover latent narratives in climate-related news from North American and
Chinese sources. While and adapts existing LLMs for climate discourse through prompt engineering, they assume the models’ inherent capability to address climate issues. In contrast, Joe et al., 2024 evaluated GPT-4o on climate tasks of varying expertise levels using GAMI literature, revealing limitations in handling high-expertise tasks, especially those involving stakeholder identification and nuanced analysis. There remains a gap in multi-perspective assessments of LLM responses across various climate change discourse sources, including news articles and social media. Multimodal Understanding in Climate Con text. Multimodal data is increasingly used in cli mate research for tasks like stance detection, predictive modeling, and video analysis. Social media platforms, especially Twitter (X) and Reddit, are key sources of such content, including humorous posts. Recent work has adapted vision-language models like CLIP and BLIP to climate-specific tasks, achieving superior performance in stance detection and misinformation detection. Benchmarks like MultiClimate and GreenScreen evaluate models on image–text alignment, narrative coherence, and visual rhetoric in climate-skeptic content. However, they focus mainly on YouTube data, overlooking fast-spreading formats like memes and infographics. Assessment Frameworks for Climate Commu nication. Three main frameworks have emerged for evaluating climate communication: (i) humanrated with AI support, (ii) AIbased evaluation using ChatGPT, and (iii) expert annotation with Likert scales. Bulian et al., 2023 offers a qualitative approach, assessing text style, clarity, tone, and epistemological elements like accuracy and uncertainty. Gursesli et al., 2023 provides a quantitative method focused on narrative quality, measuring coherence, inspiration, and fluency. Nguyen et al., 2024 presents a domain-specific framework using expert ratings to evaluate LLMgenerated climate advice for agriculture.
However, existing frameworks overlook climate communication via social media and humorous content. 3 CliME Dataset
To understand how Large Language Models (LLMs) engage with and comprehend climate discourse, specifically in addressing climate change, we introduce the first-of-its-kind multimodal benchmark, CliME [∗] (Climate Change Multimodal Evaluation) dataset. Comprising 2,579 data points, sourced and filtered from real-world Twitter and Reddit posts, primarily memes, skeptics, and infographics, CliME shifts the focus toward evaluating models based on their ability to generate credible, actionable, and equitable climate-related communi
∗ [https://huggingface.co/datasets/climedataset/](https://huggingface.co/datasets/climedataset/CliME) [CliME](https://huggingface.co/datasets/climedataset/CliME)
cation with multimodal (image-text) contexts. 3.1 Dataset Creation
The CliME dataset was developed by systematically scraping posts from both Twitter (now X) and Reddit to capture a diverse spectrum of climate related discourse, including both memes and skeptical content. Data collection from X was conducted using Selenium and Twikit, enabling the extraction of posts tagged with climate-related hashtags such as #climatechange, #climatememes, #globalwarming, #netzero, and #climateskeptics . Reddit data was obtained via the Yet Another Reddit Scraper (YARS) library, specifically targeting posts and comments from climate-focused subreddits such as
r/climatememes and r/climateskeptics . The initial raw dataset comprised approximately ∼ 4 K posts, which subsequently underwent a rigorous filtering pipeline to enhance relevance and quality. First, language filtering was applied using the langdetect library to exclude nonEnglish texts, ensuring linguistic consistency for downstream analysis. Next, we conduct relevance verification with manual review to eliminate offtopic content, including unrelated memes and spam. Lastly, de-duplication was performed to remove redundant entries; perceptual hashing via ImageHash was utilized to generate unique fingerprints for images, identifying and flagging near-identical visuals, such as reposted memes with minor modifications.
After completing the filtering process, the final dataset comprised 2579 entries, each meeting the criteria for linguistic consistency, relevance, uniqueness, and quality. 3.2 Descriptor Generation and Human Annotations
A critical challenge lies in how LLMs process visual content: directly feeding raw pixels to textbased LLMs risks stripping away the nuanced, context-dependent narratives that images convey. This exploits the primal impact of visuals to lend credibility to false claims. To bridge this gap, we employ DeepSeek Janus Pro, an autoregressive Vision-Language Model (VLM) that unifies multimodal understanding and generation, to process both the text and its accompanying image, generating structured, context-aware descriptions.
We configure DeepSeek with a temperature of 0.3 to balance specificity and creativity; max token length of 512. These detailed text descriptors help in as Climate Change Multimodal Evaluation (CliME) Dataset
Twitter / Reddit Posts Scraping
Description Generation
—
Cohen Kappa:
… 92%
— — — — … —
Climate Alignment Quotient (CAQ)
3 Axioms
5 Dimensions
Articulation: linguistic quality and structural integrity. Evidence: response supported by evidence. Resonance: relevance to climate discourse. Specificity: concreteness and actionable detail.
Transition: references to physical climate actions. —
—
Actionability
Criticality
Justice
Figure 2: Overview of the Climate Change Multimodal Evaluation (CliME) dataset and the Climate Alignment Quotient (CAQ) workflow. The upper section illustrates the data collection process from Twitter and Reddit posts, utilizing multimodal sources (text and images) and description generation through the Janus-Pro-7B model followed by human annotations. The lower section demonstrates the CAQ evaluation framework, integrating multimodal data and analytical lenses (Actionability, Criticality, Justice) to assess climate communication across five dimensions: Articulation, Evidence, Resonance, Transition, and Specificity. sessing LLMs to distinguish fact from manipulation and ensure transparency in how visual context shapes their outputs. Subsequently, for robust verification, we manually annotated the generated descriptors. In this process, we carefully examined whether the combined interpretation of the post’s image and text aligned semantically with the descriptor’s intended meaning, assigning a score of 1 for a match and 0 for no match. The data was evaluated by two human annotators, and we filtered the data based on their evaluations. The inter-annotator agreement is 92.0% measured by Cohen’s kappa, showing near perfect agreement. Our final dataset comprises triples of image, original text, and generated descriptor, helping in systematic evaluation of assessing LLMs to distinguish fact from manipulation and ensure transparency in how visual context shapes their outputs. 4 Climate Alignment Quotient (CAQ)
4.1 Analytical Lenses
We evaluate LLMs’ ability to interpret adversarial multimodal narratives, rebut misinformation with scientific rigor, and center justice for marginalized communities in climate descriptors through three novel analytical lenses: (i) Actionability, (ii) Criticality, and (iii) Justice. Descriptors are first passed through carefully crafted analytical prompts, which serve as interpretive lenses designed to assess how effectively LLMs engage with crucial climate discourse elements. For instance, consider the descriptor of a Twitter post as shown in Figure 1: \"The image and caption highlight the environmental policy of the Trump administration, specifically focusing on the withdrawal from the Paris Climate Agreement. The post emphasizes the risks associated with this deci sion, such as increased greenhouse gas emissions and the potential for climate-related disasters.
It also highlights the Green New Deal as a potential solution, emphasizing the need for a comprehen sive and immediate response to climate change. The post encourages action and emphasizes the ur gency of addressing climate issues.\" Our analytical lenses would evaluate this as follows:
Actionability assesses the translation of climate discourse into concrete interventions. In our ex ample, the post’s endorsement of the \"Green New Deal\" as a solution is scrutinized for actionable
components. While the proposal advocates for broad objectives like transitioning to \"renewable energy\", its practical feasibility may be rated as medium due to potential political gridlock and ambiguous funding mechanisms. The message emphasizes urgency but lacks specifics, such as identifying legislative bodies that would champion the pol icy, setting deadlines for key milestones like grid \"de-carbonization\", or outlining strategies to mobilize workforce training programs. Unaddressed gaps include potential economic strains on industries reliant on fossil fuels and the absence of con tingency plans for technological bottlenecks, revealing a disconnect between aspirational goals and practical roadmaps. Criticality examines the structural roots and foundations of climate narratives. The post links the Trump administration’s withdrawal from the \"Paris Agreement\" to increased emissions and climate related disasters. A critical analysis would question this causal oversimplification, noting that while the withdrawal weakened global cooperation, emissions were already rising due to entrenched systems like \"fossil fuel\" lobbying and inadequate clean energy incentives. Framing the Green New Deal as a singular solution might sidestep debates over its scalability, such as conflicting estimates about job creation or its silence on nuclear energy’s role. This uncritical portrayal risks reinforcing partisan divides rather than addressing systemic barriers like corporate influence on climate policy .
A justice lens would highlight absent voices, such as Indigenous groups advocating for land sovereignty in solar farm expansions, and question whether the policy redistributes power or perpetuates existing inequities. Following evaluation through these analytical lenses, the resulting outputs provide essential context and data to progress into our quantitative evaluation step: calculating the Climate Alignment Quotient (CAQ) . 4.2 CAQ
The Climate Alignment Quotient (CAQ) is a composite metric specifically designed to quantify the effectiveness and alignment of climate-related communication generated by LLMs. Integrating and systematically evaluating outputs derived from the analytical lenses (Actionability, Criticality, Justice), the CAQ assesses alignment across five critical dimensions: (i) Articulation, (ii) Evidence, (iii) Res onance, (iv) Transition, and (v) Specificity . The CAQ score is calculated as a weighted sum of the five core metrics, mathematically can be described
as:
CAQ = w 1 · Resonance + w 2 · Articulation
+ w 3 · Evidence + w 4 · Transition
+ w 5 · Specificity where w 1 + w 2 + w 3 + w 4 + w 5 aggregates to 1.
Empirically, we set w 1 = 0 . 25, w 2 = 0 . 3, w 3 = 0 . 2, w 4 = 0 . 15, and w 5 = 0 . 1. (i) Articulation: Our articulation score measures the linguistic quality and structural integrity of climate communications through a dual component analysis. The articulation score is calculated as an equally weighted combination of coherence and completeness, given by:
Articulation = 0 . 5 ·Coherence +0 . 5 ·Completeness The coherence component assesses how well sentences connect and flow together and is further broken down into:
Coherence = 0 .
6 · Syntactic + 0 . 4 · Semantic
Syntactic coherence evaluates the presence of discourse markers (coordinating conjunctions, subordinating conjunctions, and tokens with \"mark\" dependency) relative to sentence count. This provides a quantitative measure of how well the text is structurally connected through explicit linguistic devices. Semantic coherence, computed using the all-mpnet-base-v2 Sentence Transformer model, measures thematic consistency between adjacent sentences through embedding similarity, capturing the continuity of ideas throughout the text. The completeness component evaluates the grammatical integrity of text by assessing syntactic structure at the sentence level. We identify two types of valid constructions: complete statements containing both subject and predicate elements, and imperative sentences beginning with action verbs. Using SpaCy’s dependency parsing, we identify subjects through dependency relations ( \"nsubj\" or \"nsubjpass\" ) and predicates via ROOT verbs. The completeness score represents the proportion of sentences that follow either of these grammatical patterns relative to the total sentence count.
This measure effectively captures whether climate communications maintain consistent grammatical structure throughout, ensuring clarity of the conveyed information by the model. (ii) Evidence: The Evidence metric distinctly assesses the extent to which the climate communications in the LLM’s responses are substantiated by verifiable data and concrete examples. We employ the specialized NLP model \"climate-nlp/longformer-large-4096 1-detect-evidence\" to compute this score. A high Evidence score indicates that the language model’s claims are supported by detailed, traceable evidence, such as explicit data points, clear references, and documented instances of climate action demonstrating a robust and genuine commitment to addressing climate challenges. In contrast, a low score reveals that the communications lack sufficient backing, potentially suggesting superficial engagement or greenwashing. The remaining three CAQ components are derived via a suite of models from ClimateBERT
, pretrained on a text corpus comprising climate related research paper abstracts, corporate and general news and reports from companies. The measures derived from this collection are as follows:
(iii) Resonance: The Resonance metric leverages the \"climatebert/distilroberta-base-climate detector\" model to quantify how strongly content engages with climate context. This model performs binary classification, outputting a probability score that represents the likelihood of climate relevance.
Higher scores indicate content that more directly addresses climate change concepts, terminology, and themes. The detector performs well at identifying subtle climate references while filtering out environmental content unrelated to climate change. (iv) Transition: The Transition metric uses the \"climatebert/transition-physical\" model to evaluate references to physical climate transition pro
cesses. It identifies content related to tangible climate adaptation and mitigation practices, infrastructure changes, and transitions in physical systems. It detects mentions of renewable energy implementation, carbon capture technologies, climateresilient infrastructure, and other physical interventions. Higher scores indicate content that addresses concrete transitional mechanisms rather than ab stract climate concepts or general environmental
concerns. (v) Specificity: The Specificity metric utilizes the \"climatebert/distilroberta-base-climate specificity\" model to assess whether given response by the model provides specific, actionable information versus general statements or vague recommendations. Higher specificity scores indicate content containing concrete actions, measurable targets, defined timelines, or detailed examples. This dimension is crucial for distinguishing between aspirational climate rhetoric and communications that provide implementable guidance or precise information that can drive meaningful action in real life. In summary, CAQ serves as a robust metric for evaluating the effectiveness of climate communications in LLMs. Higher CAQ scores indicate communications that strongly align with climate objectives, characterized by coherent articulation, high resonance, strong evidence, concrete transitional strategies, and actionable specificity; hence, aligning communication strategies of LLMs with climate objectives. 4.3 Evaluation
We benchmarked five state-of-the-art LLMs on CliME and evaluated them using our CAQ metric. The models include GPT-4o, LLaMA 3.3 70B, Gemini 2.0 Flash, Qwen QwQ (Qwen with Question) 32B, and Claude 3.7 Sonnet. All models were configured to generate outputs at a temperature of 0.1 to ensure deterministic responses. The scores are reported in Table 1; a comprehensive analysis of the CAQ scores across the models along our proposed lenses is shown in Figure 3. We find from our experiments that resonance (R) scores are consistently high across all models, indicating a strong alignment with climate-related context in the generated descriptors. In terms of Actionability, Claude 3.7 Sonnet achieves an average CAQ of 0.73, while Gemini 2.0 Flash and LLaMA 3.3 70B record slightly lower scores of Table 1: Climate Alignment Quotient (CAQ) Comparison Across Different LLMs. Color intensity indicates performance level, with deeper colors representing higher scores. Column headers are color-coded by lens (yellow for Actionability, red for Criticality, green for Justice), while component scores use blue shades proportional to their values. Actionability Criticality Justice
CAQ
CAQ
Model
CAQ
GPT-4o 1.0 .86 .68 .19 .18 .69 1.0 .81 .85 .15 .18 .70 1.0 .81 .98 .18 .15 .73
Claude 3.7 Sonnet 1.0 .82 .95 .18 .20 .73 1.0 .84 .97 .17 .21 .74 1.0 .83 1.0 .11 .18 .73
Gemini 2.0 Flash 1.0 .74 .84 .18 .15 .68 1.0 .69 .78 .17 .20 .66 1.0 .82 .99 .14 .15 .73
LLaMA 3.3 70b 1.0 .72 .74 .21 .15 .66 1.0 .78 .86 .14 .19 .70 1.0 .84 1.0 .15 .19 .74
Qwen QwQ 32b .99 .81 .80 .20 .20 .70 0.99 .77 .86 .17 .22 .70 1.0 .77 1.0 .19 .18 .73
Legend: R = Resonance, A = Articulation, E = Evidence, T = Transition, Sp = Specificity Weights: Resonance, Articulation, Evidence, Transition, Specificity
0.68 and 0.66, respectively; GPT-4o and Qwen both score around 0.70. The Criticality lens shows Claude and LLaMA performing comparably, with Gemini 2.0 Flash trailing at 0.66. For the Justice dimension, all models converge between 0.73 and 0.74. This uniformity in Justice scores suggests that fairness and equity considerations are consistently addressed, while the variability in Actionability and Criticality highlights differences in the models’ abilities to generate concrete, actionable guidance and critically evaluative discourse. The articulation (A) measure further reveals that GPT-4o and Claude 3.7 Sonnet offer more coherent communication compared to Gemini 2.0 Flash and LLaMA 3.3 70B. Moreover, the evidence (E) scores are particularly high for Claude 3.7 Sonnet and LLaMA 3.3 70B, indicating that these models more effectively ground their outputs with verifiable data. In addition to this, transition scores are consistently low across all models, suggesting a common challenge in referencing physical climate transition processes. Overall, while all models demonstrate a balanced under standing of climate discourse with overall CAQ scores hovering between 0.70 and 0.74, Claude 3.7 Sonnet and GPT-4o tend to produce more articulate and evidence-supported discourse, whereas Gemini 2.0 Flash shows slightly lower performance overall. 4.4 Analysis
In this section, we present a comprehensive analysis of the CAQ scores across the proposed lenses of
Actionability, Criticality, and Justice . This analysis encompasses both the distribution of individual CAQ scores and the evaluation of gaps between these dimensions, providing a detailed understanding of the multimodal climate discourse generated by LLMs. Figure 3: Comparative analysis of CAQ scores across the Actionability, Criticality, and Justice lenses for five large language models: Claude 3.7 Sonnet, GPT4o, LLaMA 3.3 70B, Qwen QwQ 32B, and Gemini 2.0 Flash, on CliME. Each bar represents the mean CAQ score, and error bars indicate the standard deviation, showcasing the variability in model performance.
Claude 3.7 Sonnet is seen to generally outperform other models, across all the lenses, with scores consistently above 0.70 and relatively consistent standard deviations. 4.4.1 Distribution of CAQ Scores
In Figure 4, where the x-axis is Criticality, the y-axis is Justice, and the z-axis is Actionability, for Claude 3.7 Sonnet, we observe that most points cluster around the mid-range of each axis, indicating that the LLM-generated content tends to balance urgency (Criticality), fairness (Justice), and practicality (Actionability). When Actionability increases (both in the z-coordinate and in the color scale), there is often a slight upward score in both Criticality and Justice, suggesting that more action-oriented content tends to incorporate at least moderate levels of urgency and equity considerations.
Conversely, at lower Action ability scores (cooler colors), there’s a wider spread in Criticality and Justice, suggesting that content with fewer calls to action is perceived with more varied urgency and equity. 4.4.2 Gap Analysis
In addition to this, we perform a gap analysis to evaluate the differences between CAQ scores across our proposed axes. Effective climate communication necessitates a balanced integration of these aspects. Overemphasis on one dimension, such as Actionability, at the expense of others like Justice, can lead to skewed narratives that overlook systemic inequities or fail to motivate comprehensive action. In the case of Claude 3.7 Sonnet, one of the best scoring models on CAQ, one of the best-scoring models on CAQ, we empirically find that the average gap between dimensions varies across the 2,579 evaluated descriptors. The gap between Criticality and Justice is the smallest at 0.0313, showing these dimensions are generally well-aligned. However, the gap between Actionability and Justice tends to be larger, averaging around 0.0344, suggesting a slight under representation of justice considerations in certain outputs. Similarly, the Actionability-Criticality gap averages about 0.0324. In case of Claude 3.7 Sonnet, our dimensional frequency analysis reveals that the most common largest gap type is between Actionability and Criticality, occurring in 37.1% of descriptors.
These variations across different posts provide valuable insights for developing more holistic approaches to climate communication that effectively balance considerations across these three dimensions. 5 Conclusion
In this work, we present CliME, a novel multimodal dataset curated from social media, primarily
Figure 4: 3D scatter plot of CAQ scores for the Claude 3.7 Sonnet model on the CliME dataset. Each data point represents a description’s CAQ values along three axes: Actionability (x-axis), Criticality (y-axis), and Justice (z-axis).
The color scale (legend on the right) indicates the CAQ score of Actionability. Points near the center denote balanced discourse across all dimensions, whereas deviations along any axis suggest an overemphasis or under-representation of that particular lens. featuring memes and skeptics, and introduce the Climate Alignment Quotient (CAQ) as a comprehensive metric to evaluate climate discourse generated by LLMs. By benchmarking five state-of-theart models, we analyzed their outputs across three lenses: Actionability, Criticality, and Justice . Our experiments reveal that while all models consistently capture climate-related context, significant variations exist in generating actionable and critically evaluative messages. The gap analysis uncovers subtle imbalances among these dimensions, indicating areas for improvement in the models’ outputs. In today’s landscape, misinformation and polarized narratives on climate change contribute to social harm and undermine effective action; our CAQ framework offers a promising approach to understanding the strengths and weaknesses of LLMs. By fostering balanced climate communication, our work aims to prevent the spread of misleading information and support informed public dialogue, ultimately guiding policy-making for social good. Future efforts will focus on refining the CAQ metric and expanding the CliME dataset to include a broader spectrum of multimodal content, thereby empowering both LLMs and VLMs to contribute more effectively to a sustainable future. Limitations
Though the proposed CAQ framework provides a structured and comprehensive way to evaluate climate discourse, it currently relies on existing pre-trained models for its assessment. Since these models may not be extensively trained on the latest social media data, particularly memes and other highly contextual content, there is a risk of missing nuanced climate change signals.