WXImpactBench: A Disruptive Weather Impact Understanding Benchmark for Evaluating Large Language Models
Yongan Yu , Qingchen Hu , Xianda Du , Jiayin Wang , Fengran Mo [∗] , Renée Sieber []
1 McGill University, 2 University of Waterloo, 3 Tsinghua University, 4 University of Montreal yongan.yu@mail.mcgill.ca, fengran.mo@umontreal.ca, renee.sieber@mcgill.ca
Abstract
Climate change adaptation requires the understanding of disruptive weather impacts on society, where large language models (LLMs) might be applicable. However, their effectiveness is under-explored due to the difficulty of high-quality corpus collection and the lack of available benchmarks. The climate-related
events stored in regional newspapers record how communities adapted and recovered from disasters. However, the processing of the original corpus is non-trivial. In this study, we first develop a disruptive weather impact dataset with a four-stage well-crafted construction pipeline. Then, we propose WXI MPACT BENCH, the first benchmark for evaluating the capacity of LLMs on disruptive weather impacts. The benchmark involves two evaluation tasks, multi-label classification and rankingbased question answering. Extensive experiments on evaluating a set of LLMs provide first-hand analysis of the challenges in developing the understanding of disruptive weather impact and climate change adaptation systems. The constructed dataset and the code for the evaluation framework are available to help society protect against disaster vulnerabilities. 1 Introduction
Climate change adaptation, referring to actions that help reduce societal vulnerability to climate change, demands a sophisticated understanding of the disruptive weather impacts on society (e.g., the perspective of economy and policy).
Thus, the lack of high-quality resources constrains the development of comprehensive benchmarks for weather impacts. Li et al. introduce CLLMate, a multimodal benchmark that aligns meteorological data with textual event descriptions for weather event forecasting, though it focuses on prediction rather than historical impact understanding. Developing a benchmark for understanding weather impacts is important, although the fragmented, incomplete, or dispersed disparate sources of weather events increase the difficulty of annotation. Although previous studies attempt to develop evaluation frameworks for physics-based weather forecasting models, they focus on data-driven weather modeling rather than weather impact understanding. In this paper, we propose the first disruptive weather impact benchmark to fill in this blanks. 3 WXImpactBench: Disruptive Weather Impact Benchmark
Our WXI MPACT BENCH benchmark aims to evalu ate to what extent existing LLMs can understand disruptive weather impacts, which also shows the evolution of vulnerability and resilience strategies from society across various periods. It involves two main stages: i) dataset construction; and ii) task definition and evaluation. 3.1 Dataset Construction
The construction of the dataset aims to obtain
high-quality text samples. The pipeline overview is presented in Figure 2, which consists of four stages: data collection, post-OCR correction, topic-aware article selection, and manual label annotation. Data Collection. The data is obtained through collaboration with a proprietary archive institution covering two temporal periods. The original data stored as digitalized text is obtained through OCR, which contains substantial noise due to historical newspaper layouts, including uneven printing, varying font styles, complex multicolumn structures, and overlapping text elements (e.g., advertisements). Thus, post-OCR correction is necessary to ensure the corpus is high-quality. Our choice of newspapers as the primary data source stems from their status as the most consistent and reliable documentation of social impacts from extreme weather events. After the initial selection, we chose to examine a historical and a modern period, where the historical one was chosen as it had the greatest density of corresponding meteorological information, while the modern one is defined by the Intergovernmental Panel on Climate Change.
Post-OCR Correction. The goal of post-OCR correction is to correct errors that could significantly impact human comprehension or downstream task analysis, e.g., correct the inaccurate works split and remove unnecessary characters. For example, given the source image of a digital newspaper article, the text obtained by initial OCR and post-OCR correct is shown in Figure 3. The post-OCR correction is achieved by deploying GPT-4 O with customized prompts. The correction quality is evaluated using BLEU and ROUGE metrics, achieving high consistency scores compared to human annotations, see Appendix A.3 for detailed evaluation results. Our specific prompts for correction are provided in Appendix A.2. Topic-Aware Article Selection. After the post-OCR correction, we obtain 53,521 weather events-related articles. We aim to obtain informative samples across historical and modern periods based on weather categories. This is achieved by conducting topic modeling on the article collection, where we categorize them via Latent Dirichlet Allocation (LDA) to obtain the topic words - representing the primary weather event categories.
The details of the categories are provided in Appendix B.1. Selected articles with informative weather content are manually reviewed by three domain experts, which result in
Initial OCR-Digitized Text ’ ’, ’t 1’, ’ v MK ’ A 0. P. R. 2NGIlNE IN THE DIT0H. ’ I ’, \"Th’- snow storm, which was a LL day on Mondy hover’ ing overhead, began to set in at dusk, and it gradually increased in Severity, COntInuIng until abt six o-”clock yesterday morning. The storm was the w0rst for many wrintrs, and to find a preedent for it severity IT ”1 is said we h’ ve to go back to the eventful year ... After Post-OCR Correction AC.
P. R. ENGINE IN THE DITCH. The snow storm, which was all day on Monday hovering overhead, began to set in about dusk, and it gradually increased in severity, continuing until about six o’clock yesterday morning. The storm was the worst for many winters, and to find a precedent for it in severity it is said we have to go back to the eventful year ... Figure 3: Example of the text obtained from initial OCR and after our post-OCR correction. 350 high-quality samples. This process ensures the selected articles are topic-aware, which is highly related to specific disruptive weather events. Manual Label Annotation. Having selected article samples, the next step is to assign the annotation for each of them based on the label space. Six vulnerability-related disruptive weather impacts are defined as the labeling categories, including Infrastructural, Political, Financial, Ecological, Agricultural, and Human Health. Annotation is conducted by three domain annotators following our guidelines. According to the guidelines, the annotators should assign binary labels to indicate the presence or absence of direct descriptions of specific impacts within each article.
Unlike previous study, however, each sample might correspond to more than one impact. 3.2 Task Definition and Evaluation
After finalizing the data construction, we design the evaluation framework for our benchmark WXI M
PACT BENCH . The overview is shown in Figure 4, which contains two tasks, multi-label classification and ranking-based question-answering, to evaluate the capacity of LLMs to understand disruptive weather impacts. 3.2.1 Multi-Label Text Classification
With the annotated weather impact category for each selected article, the intuitive evaluation task is multi-label classification, which aims to test the ability of LLMs to distinguish the disruptive weather impact for each given article. The previous classification tasks in disaster-related natural language processing, Yellow texts highlight impact descriptions (e.g., damage assessments, resource needs), and Green texts refer to narrative descriptions (e.g., geographical locations) . →
2016b) usually focus on modern crisis communications with structured text. Different from them, our constructed samples require the models to understand the linguistic shifts between historical and modern texts and address inconsistent styles of narratives across various periods.
Specifically, given an article sample x t corresponding to six groundtruth impacts Y t = {y t [i] [}] i =1 [with binary labels] y t [i] [∈{] [,] [}] [, the evaluated model] [ M] [ is required to] maximize the probability of the predicated impact Y ˆ t = {y ˆ t [i] [}] i =1 [towards ground-truth. The objective] function L for the given sample x t of multi-label classification task is formulated as
→
L ( Y [ˆ] t , Y t ) = −
→
6 � y i log ˆ y i , y ˆ i = M ( x t )
i =1
→
3.2.2 Ranking-based Question Answering
Question-answering (QA) requires the LLMs to reply to the given question based on their parametric knowledge. We formulate the ranking-based
→
QA task by prompting the models to identify the likelihood of each article containing the correct answer from a candidate pool. This setting could also facilitate RAG systems development in the domain, where we left the answer span extraction/generation for future studies. To construct an evaluation protocol, the first step is to obtain suitable question pairs with each annotated samples in the multi-label classification task, since the question set is unavailable. Thus, we generate pseudo questions qt for each article ( x t , Y t ) based on its annotated category via a generative LLM G which is formulated as qt = G ( x t , Y t ) . The annotated categories Y t, which are the societal impacts brought by the disruptive weather event, will become part of the prompt to ensure the generated question targets one of the specific impact categories. As a result, we have QA pair ( qt , x t ) for each sample. The next step is to construct the candi date pool for ranking. The size of the pool X t for each question qt is 100, which contains the groundtruth x t and other 99 counterexamples X t [−] [that are] randomly sampled from the ground-truth of other questions. With the constructed QA pairs and corresponding candidate pools, the evaluated model M is required to rank the ground-truth based on the relevant scores produced by a matching function S . The task objective can be formulated as
RankingList ϕ ( qt ) ← arg max M ( qt , X t , S ) S ( qt ,x t )
where {x t , X t [−] [} ⊆X] [t] [ and the output ranking list] ϕ ( qt ) is evaluated by ranking metrics. 4 Experimental Setup
4.1 Evaluation Metrics and Settings
For multi-label classification task, we use F1-score, accuracy, and row-wise accuracy as evaluation metrics.
The evaluation via F1-score and accuracy are averaged across the six impact categories, historical and modern articles, and the effect of different context lengths. Compared to the common F1score and accuracy, the row-wise accuracy is a strict metric that requires more accurate output as the model should correctly classify all six impact labels for a given article, defined as
6 � I � y ˆ i [j] [=] [ y] i [j] �
j =1
Row-wise Acc. =
N
N �
i =1
where N is the number of samples, ˆ y i [j] [denotes] the predicted label for the j -th category in the i th sample, y i [j] [is the corresponding ground-truth] label, and I ( · ) is the indicator function. The evaluation goal is to investigate the models’ ability to distinguish various disruptive weather impacts under different settings, e.g., different periods.For the long-context impact evaluation, we create an alternate version (mixed context), whose sample length is split into segments with approximately 250 tokens from the original one (long-context version) following. Note that annotations for these smaller chunks are performed independently by the same domain experts rather than automatically inherited from the original articles. This independent annotation process naturally results in some chunks containing no weather impact labels, which serve as valuable negative examples in our evaluation. These negative samples allow us to assess models’ understanding to correctly identify passages without weather impacts, particularly reflected in our row-wise accuracy metric. Eventually, we contain 350 and 1,386 samples for the original and mixed context version datasets, respectively.
The detailed statistics of the datasets are provided in Appendix D. For the ranking-based QA task, we deploy the standard metric that emphasizes the accuracy of top positions for evaluation, including Hit@1, nDCG@5, Recall@5, and MRR. 4.2 Evaluated Models
We evaluate a set of off-the-shelf LLMs on
WXI MPACT BENCH . For the multi-label text classification task, we include seven open-source models: DEEP SEEK -V3-671B, LLAMA -3.1-8B-I NSTRUCT, Mistral-7B-Instruct, M IXTRAL -8 X 7B-I NSTRUCT, MISTRAL -24B-I NSTRUCT, GEMMA -2-9 B - IT, Q WEN 2.5-7B-I NSTRUCT, and Q WEN 2.514B-I NSTRUCT; and three closed-source models: GPT-3.5-T URBO, GPT-4, and GPT-4 O. For the ranking-based QA task, we evaluate GPT-3.5-T URBO, Q WEN 2.5-7B-I NSTRUCT, Q WEN 2.5-14B-I NSTRUCT, MISTRAL -7BI NSTRUCT, and LLAMA -3.1-8B-I NSTRUCT . The relatively smaller models ensure the latency requirements. The used models for the two tasks cover different sizes and support the input length of at least 8k tokens.
The version details are provided in Appendix E. 4.3 Implementation Details
Multi-label Classification. The multi-label classification is conducted on each evaluated LLM by the same prompt provided in Appendix C.2. Different from traditional methods that decompose multi-label text classification into multiple binary classification tasks, we simultaneously identify all relevant disruptive weather impacts for each input by calling the LLM once. The example of in-context learning in the one-shot setting is handcrafted with a complex sample detailing multiple impacts. Ranking-based Question-Answering. We employ GPT-4 O for pseudo question generation with default hyper-parameters. For ranking evaluation, we adopt the sliding window mechanism within Infrastructure Political Financial Ecological Agricultural Human Health Average Positive Cases 168 61 98 54 80 117 Zero-Shot Performance
GPT-4 O 80.94 ↑ 0.39 58.46 ↑ 3.70 65.82 ↓ 0.23 46.81 ↑ 2.51 70.33 ↑ 1.92 73.23 ↑ 1.12 65.93 ↑ 1.07 GPT-4 74.87 ↑ 2.13 49.38 ↑ 2.47 55.70 ↑ 1.18 37.84 ↑ 4.71 60.00 ↑ 3.83 62.96 ↑ 3.71 56.79 ↑ 2.12 GPT-3.5- TURBO 77.59 ↑ 3.73 41.60 ↑ 7.73 42.39 ↑ 7.16 36.52 ↓ 0.01 55.63 ↑ 4.49 47.29 ↑ 10.91 50.17 ↑ 5.12 DEEP SEEK -V3-671B 81.87 ↑ 0.80 44.44 ↑ 12.40 60.91 ↑ 3.91 36.00 ↓ 0.60 61.74 ↑ 4.34 65.20 ↑ 0.20 58.03 ↑ 3.07 MISTRAL -24B-IT 79.12 ↓ 0.17 47.18 ↑ 6.91 59.64 ↑ 2.04 44.90 ↑ 10.75 67.74 ↑ 1.07 66.88 ↑ 1.30 60.91 ↑ 1.57 M IXTRAL -8 X 7B-IT 72.31 ↑ 3.01 39.29 ↑ 6.86 57.02 ↑ 0.33 36.59 ↑ 5.23 44.44 ↑ 14.23 50.00 ↑ 9.69 49.94 ↑ 4.84 MISTRAL -7B-IT 74.27 ↑ 4.27 36.63 ↑ 6.63 45.56 ↑ 0.56 39.19 ↑ 5.19 55.30 ↑ 5.30 61.61 ↑ 11.61 52.76 ↑ 1.98 Q WEN 2.5-14B-IT 76.21 ↓ 1.45 41.45 ↑ 0.34 45.07 ↑ 5.32 41.62 ↓ 2.15 52.99 ↑ 5.01 63.08 ↓ 1.64 53.74 ↓ 1.36 Q WEN 2.5-7B-IT 70.52 ↑ 4.75 34.29 ↑ 2.82 43.43 ↑ 0.83 41.06 ↓ 6.51 40.26 ↑ 3.96 38.19 ↑ 7.34 44.63 ↑ 1.88 G EMMA -2-9 B -IT 77.42 ↑ 1.52 43.33 ↑ 2.59 54.60 ↓ 0.33 42.16 ↑ 1.73 55.60 ↑ 4.10 61.82 ↑ 0.87 55.82 ↑ 1.31 LL AM A -3.1-8B-IT 70.13 ↑ 8.82 40.47 ↑ 1.51 55.29 ↓ 2.08 33.90 ↑ 4.99 55.49 ↑ 5.05 50.68 ↑ 11.72 50.66 ↑ 4.91 Average 75.93 ↑ 2.93 43.32 ↑ 5.36 53.22 ↑ 1.34 43.33 ↑ 2.80 56.32 ↑ 4.83 58.36 ↑ 5.85 54.48 ↑ 2.38 In-Context Learning (One-shot) Performance GPT-4 O 81.25 ↑ 0.29 59.54 ↑ 2.71 63.64 ↑ 1.18 50.00 ↓ 0.02 71.43 ↑ 1.39 72.94 ↑ 2.80 66.93 ↑ 1.02 GPT-4 72.63 ↑ 3.54 40.00 ↑ 5.92 55.15 ↑ 0.55 32.38 ↑ 7.30 61.29 ↑ 2.92 60.22 ↑ 5.06 54.95 ↑ 3.25 GPT-3.5- TURBO 76.88 ↑ 2.29 38.93 ↑ 9.72 48.50 ↑ 0.95 40.00 ↑ 0.02 57.30 ↓ 0.02 60.98 ↓ 0.02 54.08 ↑ 2.26 DEEP SEEK -V3-671B 81.62 ↑ 1.32 49.48 ↑ 7.40 63.37 ↑ 2.10 43.55 ↑ 1.21 62.82 ↑ 5.04 67.78 ↑ 3.07 62.90 ↑ 2.36 MISTRAL -24B-IT 78.38 ↑ 0.58 43.48 ↑ 12.36 56.99 ↑ 1.05 35.09 ↑ 6.39 61.45 ↑ 4.50 65.28 ↓ 0.08 57.41 ↑ 2.06 M IXTRAL -8 X 7B-IT 68.31 ↑ 4.47 12.50 ↑ 24.14 42.00 ↑ 8.43 26.45 ↑ 7.74 36.80 ↑ 10.26 46.46 ↑ 14.82 40.53 ↑ 8.63 MISTRAL -7B-IT 73.31 ↑ 3.31 20.74 ↑ 6.74 45.33 ↑ 5.33 31.94 ↑ 1.94 52.77 ↑ 2.77 54.87 ↑ 4.87 47.43 ↑ 1.57 Q WEN 2.5-14B-IT 78.10 ↑ 0.05 43.36 ↑ 1.21 48.42 ↑ 6.13 43.65 ↑ 0.30 62.18 ↑ 4.49 63.60 ↑ 3.78 54.95 ↑ 1.71 Q WEN 2.5-7B-IT 71.04 ↑ 2.70 31.46 ↓ 4.19 48.80 ↑ 0.69 37.68 ↓ 0.09 47.54 ↓ 14.21 45.85 ↑ 8.51 47.40 ↑ 0.75 G EMMA -2-9 B -IT 74.24 ↑ 1.54 31.79 ↑ 7.36 51.76 ↑ 0.91 34.52 ↑ 0.39 48.13 ↑ 7.87 63.76 ↑ 0.57 48.20 ↑ 1.63 LL AM A -3.1-8B-IT 71.88 ↑ 4.73 34.92 ↑ 8.01 49.50 ↑ 3.95 40.30 ↑ 1.08 52.69 ↑ 3.91 54.85 ↑ 5.48 51.33 ↑ 4.45 Average 74.27 ↑ 2.48 35.07 ↑ 8.02 52.38 ↑ 2.51 37.32 ↑ 2.60 55.31 ↑ 2.44 58.95 ↑ 4.51 56.63 ↑ 2.70
Table 1: F1-scores results of zero-shot and one-shot evaluation categorized on six impacts and two context length settings. The number in parentheses refers to the improvement with ↑ or degradation with ↓ of the evaluation on the mixed-context dataset compared to the original dataset. The number in the “Positive Cases” row denotes the positive label in each impact categorization corresponding to two context-length versions. Bold and underline indicate the best and the second-best performance. LLM-based ranker implementation following the state-of-the-art study to reduce the potential negative effect of noisy long contexts. Specifically, each article in the candidate pool was segmented into three chunks, and then the initial ranking was performed independently within each chunk. The used prompts for both stages are provided in Appendix C.3. To ensure stable results, following previous studies, all LLMs were evaluated with the temperature set to 0, and the reported performance is the average value of running the experiments three times. 5 Experiments
5.1 Results of Multi-label Classification
Table 1 and Table 2 show the performance of the evaluated LLMs on WXI MPACT BENCH for the settings of categorized by six societal impacts with different context lengths, overall row-wise evaluation, and divided into two periods, respectively. The additional experimental results are provided in Appendix G. We have the observations as follows. LLMs struggle to understand disruptive
Model Zero-Shot One-Shot
GPT-4 O 32.28 ↑ 0.29 31.99 ↓ 0.85 GPT-4 22.19 ↑ 0.38 20.46 ↑ 0.11 GPT-3.5- TURBO 21.61 ↓ 0.18 12.39 ↑ 6.18 DEEP SEEK -V3-671B 34.01 ↓ 1.72 31.99 ↓ 0.28 MISTRAL -24B-IT 19.88 ↓ 1.02 25.65 ↓ 1.08 M IXTRAL -8 X 7B-IT 25.07 ↓ 0.50 19.88 ↑ 0.12 MISTRAL -7B-IT 4.90 ↓ 0.04 8.93 ↓ 3.50 Q WEN 2.5-14B-IT 19.02 ↓ 2.45 18.16 ↓ 0.73 Q WEN 2.5-7B-IT 27.38 ↓ 6.52 25.65 ↓ 1.36 G EMMA -2-9B-IT 15.56 ↑ 0.44 9.51 ↓ 1.51 LLAMA -3.1-8B-IT 12.68 ↑ 2.18 15.56 ↓ 1.85 Average 21.96 ↓ 0.57 20.80 ↓ 0.10
Table 2: Row-wise accuracy performance across different models and prompting strategies. The used notions are the same as Table 1.
weather impacts. Table 1 shows that the F1-score for multi-label classification remains consistently low across models, especially among the political and ecological categories. The financial, agricultural, and human health impacts categories perform slightly better but still exhibit suboptimal results at 55% . The low performance might be attributed to the challenges in these categories with abstract and context-dependent narratives. Different from the infrastructure category (with the high est performance), which describes the impacts of weather events explicitly, e.g., “bridges destroyed” and “power outages”, the descriptions in other categories are usually more abstract. For example, the financial damage could be embedded within discussions of damaged infrastructure or agricultural setbacks, which makes it more difficult for models to distinguish them as standalone impacts.
Table 2 shows row-wise performance, in which the model must identify the given sample correctly for each involved category, the performance of classification drops dramatically due to the more precise requirement. Thus, a sophisticated model is expected to understand the complex societal effects of historical narratives via reasoning. Long-context LLMs not always be strong on long-context de-noising. The results in Table 1 show that, when the original long-context is segmented into smaller chunks, the classification accuracy increases in most cases. These improvements suggest that smaller chunks help models focus on relevant information and thus minimizing distraction from extraneous content. Even the used mod els are claimed with long-context capacity, more precise split that reduces potential noise is still effective for context de-noising, which is consistent with previous studies. However, we also find that this trend is not observed with the row-wise accuracy evaluation. This is due to the evaluation bias, where the F1-score measures precision and recall per category, and benefits from partial correctness. Row-wise accuracy requires an exact match across all labels. The small chunks might be helpful to improve the classification of one of the categories but not enough to correct all labels. Thus, the helpfulness of long-context de-noising via small chunks is not obvious in overall performance. In-context Learning offers limited improve ment. The in-context learning is achieved by providing one demonstration as the one-shot example for model decision. Compared zero-shot and one-shot performance in Table 1, we find that providing a single example in the prompt offers limited benefits and might decrease the performance in some cases. Such a phenomenon implies that the LLMs lack sufficient knowledge to disambiguate disruptive weather impacts even with enhanced examples for knowledge arousing. Historical Modern
Cases 200 150 GPT-4 O 70.19 ↑ 1.50 68.59 ↑ 0.59 GPT-4 65.54 ↑ 0.66 53.81 ↑ 4.50 GPT-3.5- TURBO 57.27 ↑ 4.78 52.21 ↑ 5.92 DEEP SEEK -V3-671B 65.42 ↑ 3.91 64.74 ↑ 2.05 MISTRAL -24B-IT 62.30 ↑ 4.13 63.33 ↓ 1.68 M IXTRAL -8 X 7B-IT 58.31 ↑ 4.28 50.11 ↑ 6.14 MISTRAL -7B-IT 55.27 ↑ 1.50 49.22 ↑ 2.78 Q WEN 2.5-14B-IT 57.75 ↑ 2.30 57.75 ↓ 5.48 Q WEN 2.5-7B-IT 54.29 ↑ 0.66 40.85 ↑ 4.14 G EMMA -2-9B-IT 60.41 ↑ 0.90 52.80 ↑ 1.97 LLAMA -3.1-8B-IT 55.30 ↑ 5.08 50.67 ↑ 4.98 Average 60.19 ↑ 2.70 54.92 ↑ 2.36
Table 3: F1-score performance across historical and modern impact categories in zero-shot setting. The used notations are the same as Table 1.
Model Hit@1 nDCG@5 Recall@5 MRR
GPT-3.5-T URBO 62.09 67.31 71.04 66.90
MISTRAL -7B 6.21 15.86 25.16 14.82
QW EN -2.5-14B 82.09 86.34 85.48 89.55 QW EN -2.5-7B 42.69 61.80 75.52 58.04
LLAMA -3.1-8B 64.18 70.85 75.82 69.90
Result with Bias for Reference
GPT-4 O 91.94 95.54 97.91 94.88
Table 4: The performance of ranking-based QA tasks across different models. These results indicate that our WXI MPACT BENCH
is challenging for LLMs to understand disruptive weather impact. Historical narratives are easier for LLMs to
understand. The evaluation of different narratives in terms of historical and modern articles is presented in Table 3. Surprisingly, the evaluated models perform better on the articles recorded in the historical period. The reason might be the structured and formal narrative style used to report disruptive weather events in historical periods, which more explicitly highlights cause-and-effect relationships. The observation is revealed by the earlier studies, where the historical narratives emphasize empirical observations over interpretations, offering a more immediate and naturalistic account of events. Though the modern text might dominate within the pre-trained corpus, the language patterns used in historical narrative styles are easier for language models to identify, and thus perform better on classifying disruptive weather impacts. Scaling law might hold for disruptive weather impact understanding. As illustrated in Table 1 and Table 2, larger models usually perform better than smaller ones, which is consistent with the scaling law for LLMs. For example, the largest DEEP SEEK -V3-671B obtains the best results and MISTRAL -24B-IT outperforms its 7B version. Although the model size is unavailable in closed-source models, the open-source models with the feasibility of manipulation can be viable alternatives to adaptively work for domain requirements. With proper optimization, the second-best DEEP SEEK -V3-671B for understanding disruptive weather impact might offer performance close to or on par with GPT-4 O . 5.2 Results of Ranking-based QA
The performance of each evaluated model for ranking-based QA is reported in Table 4. We find that QW EN -2.5-14B-IT achieves the best performance in this task. LLAMA -3.1-8B is slightly better than GPT-3.5-T URBO and QW EN -2.5-7BIT, while the MISTRAL -7B-IT cannot address the tasks related to disruptive weather context. Notice that the ranking results would contain bias when the evaluated model is used for question generation. This is a common phenomenon and needs to be avoided in benchmarking. The practical open-retrieval setting, i.e., identifying the relevant articles from a huge database, is left for future studies, which could further facilitate knowledge enhancement in understanding disruptive weather impacts. 6 Conclusion
In this study, we propose a disruptive weather impact understanding benchmark, WXI MPACT BENCH, to address the lack of datasets and evaluation frameworks in climate change adaptation. We first process the corpus from newspaper articles and provide comprehensive instruction for impact annotation with each processed article. Then, we design multi-label classification and ranking-based QA tasks to evaluate the LLMs in understanding various defined disruptive weather impacts. Extensive experiments on WXI MPACT BENCH reveal that the existing LLMs struggle with understanding disruptive weather impacts across different style narratives and context settings. Our WXI MPACT BENCH enables the community to evaluate the developed systems on disruptive weather impact understanding, which sup ports the society to learn from and prepare for the impacts of climate change. Limitation
Although WXI MPACT BENCH provides valuable insights (e.g., exhibit the strengths and weaknesses of various society impact understanding) about evaluating LLMs on disruptive weather, it may have potential biases in underrepresented historical events and linguistic variations. Future work could expand the range of evaluated models, strategies, and designed tasks to further strengthen the evaluations. Ethical Considerations
Our primary data source is a corpus of historical digitized newspapers, obtained through collaboration with an official organization, which should be anonymous at this moment. This organization preserves the copyright of the newspaper articles and has been granted permission to publish this subset of articles for benchmark build-up to facilitate the research community. Thus, the data is publicly available and thus no potential privacy or content safety concerns. Additionally, topic-aware article selection is conducted by researchers specializing in historical climate analysis to ensure the dataset is not biased on specific time and location. This research contributes to the broader societal goal of understanding historical disruptive weather impacts to help society defend its vulnerabilities from disasters. The interpretation of weather-related disruptions in historical newspapers might be influenced by demographic and contextual factors, which is similar to other text datasets generated through crowd-sourcing with inherent challenges in ensuring that dataset labels are fully representative of diverse societal perspectives.
Acknowledgment
Our primary data source is a corpus of three digitized newspapers (La Presse, La Patrie and Montreal Gazette), obtained through collaboration with the McGill University Library and Archives and the Bibliothèque nationale du Québec. We would like to thank DRAW McGill for their guidance throughout this project, especially Dr. Victoria Slonosky, whose expertise in historical meteorology was instrumental in accessing the corpus and advising on OCR correction. We are also deeply grateful for the support provided by OpenAI Grants and RBC Borealis AI.