# A Skill Assessment Framework for the Fisheries and Marine Ecosystem Model Intercomparison Project

Nina Rynne, Camilla Novaglio, Julia Blanchard, Daniele Bianchi, Villy Christensen, Marta Coll, Jerome Guiet, Jeroen Steenbeek, Andrea Bryndum-Buchholz, Tyler Eddy, et al.

► To cite this version:

Nina Rynne, Camilla Novaglio, Julia Blanchard, Daniele Bianchi, Villy Christensen, et al. .. A Skill As- sessment Framework for the Fisheries and Marine Ecosystem Model Intercomparison Project. Earth's Future, 2025, 13 (4), pp.e2024EF004868. 10.1029/2024EF004868 . hal-05478231

HAL Id: hal-05478231

https://hal.inrae.fr/hal-05478231v1

Submitted on 26 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Earth's Future

## RESEARCH ARTICLE

10.1029/2024EF004868

**Special Collection:**
Past and Future of Marine Ecosystems

**Key Points:**

* We developed a standardised skill assessment framework for an ensemble of global marine ecosystem models

* Selected models show agreement with the trajectory of fisheries catch, but exhibit biases compared to observed absolute catch values

* Our framework provides a solid basis to guide global marine ensemble model improvement and increase credibility of ensemble projections

**Supporting Information:**

Supporting Information may be found in the online version of this article.

**Correspondence to:**

N. Rynne,
nina.rynne@protonmail.com

**Citation:**

Rynne, N., Novaglio, C., Blanchard, J., Bianchi, D., Christensen, V., Coll, M., et al. (2025). A skill assessment framework for the Fisheries and Marine Ecosystem Model Intercomparison Project. *Earth's Future, 13*, e2024EF004868. https://doi.org/10.1029/2024EF004868

Received 6 MAY 2024

Accepted 28 FEB 2025

**Author Contributions:**

**Conceptualization:** Nina Rynne,
Camilla Novaglio, Julia Blanchard, Ryan
F. Heneghan

**Data curation:** Daniele Bianchi,
Villy Christensen, Marta Coll,
Jerome Guiet, Jeroen Steenbeek

**Formal analysis:** Nina Rynne,
Camilla Novaglio, Julia Blanchard

**Investigation:** Nina Rynne

© 2025. The Author(s).

This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.

<figure><img src="image_2.png" /></figure>

## A Skill Assessment Framework for the Fisheries and Marine Ecosystem Model Intercomparison Project

Nina Rynne¹,² ID, Camilla Novaglio²,³ ID, Julia Blanchard²,³ ID, Daniele Bianchi⁴ ID,
Villy Christensen⁵,⁶, Marta Coll⁶,⁷, Jerome Guiet⁴ ID, Jeroen Steenbeek⁶ ID,
Andrea Bryndum-Buchholz⁸ ID, Tyler D. Eddy⁸ ID, Cheryl Harrison⁹ ID, Olivier Maury¹⁰ ID,
Kelly Ortega-Cisneros¹¹, Colleen M. Petrik¹² ID, Derek P. Tittensor¹³, and Ryan F. Heneghan¹,¹⁴,¹⁵

¹School of Mathematical Sciences, Queensland University of Technology, Brisbane, QLD, Australia, ²Institute for Marine and Antarctic Studies, University of Tasmania, Hobart, TAS, Australia, ³Centre for Marine Socioecology, University of Tasmania, Hobart, TAS, Australia, ⁴Department of Atmospheric and Oceanic Sciences, University of California Los Angeles, Los Angeles, CA, USA, ⁵Institute for the Oceans and Fisheries, University of British Columbia, Vancouver, BC, Canada, ⁶Ecopath International Initiative, Barcelona, Spain, ⁷Institute of Marine Sciences, Barcelona, Spain, ⁸Centre for Fisheries Ecosystem Research, Fisheries & Marine Institute, Memorial University, St John's, NL, Canada, ⁹Department of Ocean and Coastal Science and Center for Computation and Technology, Louisiana State University, Baton Rouge, LA, USA, ¹⁰IRD, University Montpellier, Ifremer, CNRS, INRAE, MARBEC, Sète, France, ¹¹Department of Biological Sciences, University of Cape Town, Cape Town, South Africa, ¹²Scripps Institution of Oceanography, University of California San Diego, La Jolla, CA, USA, ¹³Department of Biology, Dalhousie University, Halifax, NS, Canada, ¹⁴School of Science, Technology and Engineering, University of the Sunshine Coast, Petrie, QLD, Australia, ¹⁵Australian Rivers Institute, School of Environment and Science, Griffith University, Nathan, QLD, Australia

## Abstract
Understanding climate change impacts on global marine ecosystems and fisheries requires complex marine ecosystem models, forced by global climate projections, that can robustly detect and project changes. The Fisheries and Marine Ecosystems Model Intercomparison Project (FishMIP) uses an ensemble modeling approach to fill this crucial gap. Yet FishMIP does not have a standardised skill assessment framework to quantify the ability of member models to reproduce past observations and to guide model improvement. In this study, we apply a comprehensive model skill assessment framework to a subset of global FishMIP models that produce historical fisheries catches. We consider a suite of metrics and assess their utility in illustrating the models' ability to reproduce observed fisheries catches. Our findings reveal improvement in model performance at both global and regional (Large Marine Ecosystem) scales from the Coupled Model Intercomparison Project Phase 5 and 6 simulation rounds. Our analysis underscores the importance of employing easily interpretable, relative skill metrics to estimate the capability of models to capture temporal variations, alongside absolute error measures to characterize shifts in the magnitude of these variations between models and across simulation rounds. The skill assessment framework developed and tested here provides a first objective assessment and a baseline of the FishMIP ensemble's skill in reproducing historical catch at the global and regional scale. This assessment can be further improved and systematically applied to test the reliability of FishMIP models across the whole model ensemble from future simulation rounds and include more variables like fish biomass or production.

## Plain Language Summary
To understand how climate change affects the world's oceans and fisheries, scientists use complex models that predict changes based on climate data. One specific initiative, the Fisheries and Marine Ecosystems Model Intercomparison Project (FishMIP), employs a variety of these models together to enhance predictions. However, FishMIP lacks a standardized method to evaluate how well these models match past data and to identify areas for improvement. In this study, we developed and applied a detailed evaluation method to some FishMIP models that predict historical fish catches. We used different measures to determine how accurately these models can replicate actual fish catches. Our results showed that the models have improved at predicting fish catches, both globally and in specific large marine regions, through two rounds of model improvements. We emphasized the value of using clear and relative measures for understanding the models' accuracy over time and specific measures to identify the differences in predictions between models and over time. This evaluation provides a baseline for understanding how well FishMIP can reproduce past fishing data and suggests ways to further refine and test these models in future studies, potentially including additional factors like fish numbers or overall productivity.
**Methodology:** Nina Rynne,
Camilla Novaglio, Julia Blanchard, Ryan
F. Heneghan

**Software:** Nina Rynne

**Supervision:** Camilla Novaglio,
Julia Blanchard, Ryan F. Heneghan

**Validation:** Nina Rynne

**Visualization:** Nina Rynne,
Camilla Novaglio, Cheryl Harrison, Ryan
F. Heneghan

**Writing – original draft:** Nina Rynne,
Camilla Novaglio, Julia Blanchard, Ryan
F. Heneghan

**Writing – review & editing:** Nina Rynne,
Camilla Novaglio, Julia Blanchard,
Daniele Bianchi, Villy Christensen,
Marta Coll, Jerome Guiet,
Jeroen Steenbeek, Andrea Bryndum-
Buchholz, Tyler D. Eddy, Cheryl Harrison,
Olivier Maury, Kelly Ortega-Cisneros,
Colleen M. Petrik, Derek P. Tittensor,
Ryan F. Heneghan

# 1. Introduction

Across the world's oceans, marine ecosystems are impacted by humans through fishing, pollution, land use change, and via the accelerating impacts of climate change and ecosystem degradation (Halpern et al., 2008; Hatton et al., 2021). Demand on marine ecosystems for food production is already outpacing human population growth (FAO, 2022), while climate change impacts are expected to perturb marine communities, from individuals to ecosystems (Fulton et al., 2019), driving changes in the availability, resilience, biomass and location of fish stocks (Blanchard et al., 2012; Booth et al., 2017; Cheung et al., 2010; Hollowed et al., 2013; Lotze et al., 2019; Tittensor et al., 2021).

With the growing scope of human impacts on life below water, a range of global Marine Ecosystem Models (MEMs) has been developed by the international research community to help understand and project future change; from simple models based on macroecological scalings to end-to-end models that explicitly represent physical, ecological, and human dynamics; spanning regional systems up to the global ocean. The Fisheries and Marine Ecosystem Model Intercomparison Project (FishMIP; Lotze et al., 2019; Tittensor et al., 2018, 2021; www.fishmip.org) was established in 2013 as part of the broader Inter-Sectoral Impact Model Intercomparison Project (ISIMIP; www.isimip.org) to capitalise on the benefits of bringing such models together into an ensemble. As of today, the FishMIP ensemble comprises no less than nine global and over 30 regional MEMs (Tittensor et al., 2021). Individual MEMs are forced by standardised inputs to investigate the influence of environmental conditions and global fishing on ocean biomass and catches while accounting for structural uncertainty across the models (Tittensor et al., 2018, 2021).

Amongst today's most relevant applications of MEMs is quantifying and projecting anthropogenic impacts on marine ecosystems with an overarching goal of informing climate change mitigation and adaptation policy, food security issues, and biodiversity policy (Novaglio et al., 2024). Yet to credibly project anthropogenic impacts on marine ecosystems, the reliability of these MEM projections must be assessed in terms of their skill in reproducing past observations. Skill assessment, broadly, involves comparing model outputs for each member of an ensemble with independent sources of observational data using statistical techniques, and comparing metrics of skill across models (Baumberger et al., 2017; Kubicek et al., 2015; Power, 1993; Stow et al., 2009). Robust skill assessment for ecological models is challenging and still lacks widespread usage. A recent review found that as few as 24% of published ecological modelling studies conducted some form of objective (i.e., metric based) skill assessment (Kubicek et al., 2015), while basic visual comparison is the most commonly used subjective skill assessment method, and is arguably the de facto community standard (Stow et al., 2009). More work is needed to standardise and accelerate the use of model skill assessments to enhance the credibility and reliability of ecosystem model projections for MEMs.

Rigorous model skill assessment needs to address the relevance of models to the scientific or societal question they are addressing (Jakeman et al., 2006; Planque et al., 2022). This includes identifying sets of relevant metrics that help quantify the realism of simulated variables or patterns of importance (Allen & Somerfield, 2009; Bennett et al., 2013; Power, 1993; Stow et al., 2009); and addressing the relevance of the models given technical limitations or the needs of end-users (Hamilton et al., 2019; Kubicek et al., 2015; Steenbeek et al., 2021).

There are four major challenges that have inhibited the widespread usage of skill assessment for ecological models including MEMs, and especially for cross-model comparison: (a) the absence of a standardised framework, leading to arbitrary and inconsistent choices of important metrics (Geary et al., 2020; Hipsey et al., 2020; Kubicek et al., 2015; Mayer & Butler, 1993; Rykiel, 1996); (b) the need for multiple relevant metrics to assess different aspects of model performance, as relying on a single metric can obscure divergent behaviour or favour models that are highly correlated with a particular set of observations by chance (Bennett et al., 2013; Eyring et al., 2019; Legates & McCabe, 1999; Mayer & Butler, 1993; Power, 1993); (c) credible replication of observations by a model in some regions or for a given time-period does not guarantee performance beyond the calibrated range (Eyring et al., 2019; Hipsey et al., 2020; Hollowed et al., 2013; Refsgaard et al., 2014; Steenbeek et al., 2021; Wagener et al., 2022); (d) the hypothetical nature of future projections makes their comparison to observations unfeasible (Baumberger et al., 2017; Hamilton et al., 2019; Hollowed et al., 2013; Refsgaard et al., 2014).

In the context of fisheries and ecosystem models, these challenges are compounded by limitations in the observational data available to validate MEMs, and the quality of available data. While global and regional catch
reconstructions exist (e.g., Pauly & Zeller, 2016; Watson & Tidd, 2018), global observations of fish biomass are lacking. Stock assessments, such as the RAM Legacy Stock Assessment Database (Ricard et al., 2012; www.ramlegacy.org) or recent regional standardised synthesis of biomass observations from trawl surveys (Maureaud et al., 2023) are filling this gap, though they remain limited in spatial coverage, and represent snapshots in time that do not capture variability at seasonal, interannual, or longer timescales.

Although individual skill assessment of FishMIP models has been performed on individual models (Barrier et al., 2023; Blanchard et al., 2012; Carozza et al., 2016, 2017; Cheung et al., 2011; Christensen et al., 2015; Christensen & Walters, 2004; Heneghan et al., 2021; Jennings & Collingridge, 2015; Maury, 2010; Maury et al., 2007; Novaglio et al., 2022; Ortega-Cisneros et al., 2017; Petrik et al., 2019; Sturludottir et al., 2018), these assessments vary from model to model and no standardised skill assessment across the FishMIP model ensemble has taken place yet.

This paper sets the foundations of a skill assessment framework for FishMIP based on the “Concept-State-Process-System” (CSPS) framework created by Hipsey et al. (2020), to build confidence that future predictions are robust and credible. We choose the CSPS framework for its thorough, multi-step process and extensive integration of skill assessment examples in aquatic ecosystem literature.

In choosing and adapting the CSPS framework we attempt to address the first three key challenges (absence of a standardised framework, the need for multiple metrics and model credibility beyond calibrated range) that have hindered the widespread use of model skill assessment in marine ecosystem modelling, and FishMIP in particular. In doing so we: (a) conduct the first standardised model skill assessment of fisheries catch predictions across a subset of FishMIP ensemble members and (b) demonstrate how the CSPS framework can be utilised as the foundation for building context-specific ecosystem skill assessment tools.

We argue that the subsequent uptake of such a framework will improve the credibility and reliability of global MEMs, strengthening their use to inform decision-making. The wider benefits of this case study include illustrating how the framework can be customised for the skill assessment needs of other model intercomparison projects, and therefore further the development of open-access, reliable skill assessment tools for other climate-impact assessment ensembles.

## 2. Materials and Methods

### 2.1. Concept-State-Process-System (CSPS) Framework Overview

The CSPS framework categorizes measurable elements of ecosystem structure and function across four levels (Hipsey et al., 2020). Level 0 (conceptual assessment) focuses on ensuring that model parameterizations, assumptions and representation of the underlying system are reasonable and derived from a credible scientific basis, and that the level of complexity in the model is appropriate for the questions being asked (Hipsey et al., 2020; Kubicek et al., 2015; Rykiel, 1996). Level 1 (state assessment) is concerned with a model's ability to reflect past observations of measured ecosystem properties. This is generally achieved using metrics that assess goodness-of-fit, which can highlight various mismatches between observations and simulations (Stow et al., 2009). Level 2 (process assessment) and Level 3 (system assessment) explore whether the model is right for the right reasons, that is, whether the model has captured the important underlying rates of change within the ecosystem, as well as spatial and temporal dynamics that emerge from the model (Hipsey et al., 2020).

Due to the complex task of developing a standardised skill assessment framework for FishMIP models (as detailed in the introduction) and the current lack of a set of quantitative measures to assess FishMIP models' ability to reproduce past trends and patterns, this paper focuses on the implementation of Level 1 model skill assessment. However, we will consider future developments, including the development of methods for Levels 2 and 3 in the context of FishMIP in the Discussion, noting that such methods are still broadly described in the literature and seldom considered when assessing marine ecosystem models.

CSPS addresses three of the four major challenges of MEMs skill assessment: it provides a standardised approach and selection of metrics that can be used across models (addressing the absence of a standardised assessment framework); multiple metrics are used to assess model performance (addressing the need for a suite of metrics to holistically assess model performance); and, finally it provides a framework to assess whether models can replicate ecosystem processes and properties, which is critical for MEMs to provide credible prediction outside
<table><caption>**Table 1**

*Skill Assessment Metrics*</caption><thead><tr><th>Name</th><th>Type</th><th>Ideal value</th><th>Usage</th><th>Notes</th></tr></thead><tbody><tr><td>Correlation (R)</td><td>Relative</td><td>1</td><td>R measures the degree to which simulated and observed catches change together in time. This metric indicates if both variables move in the same direction over time.</td><td>R is a relative (or dimensionless) statistic, meaning that correlation is not influenced by the magnitude of the underlying data, therefore values close to 1 can occur even if there is considerable difference in magnitude between the values. Additionally, correlation can be sensitive to outliers if they exist in the data. Relative statistics are comparable across different models or regions.</td></tr><tr><td>Average Error (AE)</td><td>Absolute</td><td>0</td><td>AE is the sum of the size of the discrepancies between simulated and observed catch value-pairs. It measures the aggregate bias (or under/overestimation) of simulated catches compared to observations.</td><td>A shortcoming of AE is that results close to zero can indicate either a close match or can be a result of positive and negative errors canceling out. To overcome this, other methods of calculating error can be used instead of, or alongside, AE.</td></tr><tr><td>Root Mean Squared Error (RMSE)</td><td>Absolute</td><td>0</td><td>RMSE gives the average distance between predicted and observed catches. It measures the aggregate bias of simulated catches compared to observation Centered RMSE—as reported in a Taylor diagram - is given as a RMSE relative to the standard deviation of observed catches.</td><td>RMSE accommodates for the shortcomings of AE as it considers the magnitude, but not the direction, of each discrepancy. As RMSE uses the square of each discrepancy, it is more sensitive to the influence of outliers than either AE or MAE.</td></tr><tr><td>Mean Absolute Error (MAE)</td><td>Absolute</td><td>0</td><td>MAE is the sum of the absolute size of the discrepancies between simulated and observed catches. It measures the aggregate bias of the simulated catches compared to observations.</td><td>MAE accommodates for the shortcoming of AE, by using the absolute value of the discrepancies. When absolute differences are of a similar magnitude, RMSE and MAE will be approximately equal (Mayer & Butler, 1993)</td></tr><tr><td>Reliability Index (RI)</td><td>Relative</td><td>1</td><td>RI is a measure of the average multiplicative factor by which simulated catches differ from observations. Similar to AE, RMSE and MAE, it can be used to measure the bias of the simulations, but as a relative statistic, it can be compared across different models or regions.</td><td>RI results are relative making them useful for comparing projections from different models or for different regions</td></tr><tr><td>Modeling Efficiency (MEF)</td><td>Relative</td><td>1</td><td>MEF measures the predictive ability of model simulations, relative to the average of the observations in an easily interpretable single statistic.</td><td>MEF ∈ (-∞, 1]. A negative result indicates that the observational average is a better predictor than the model projections. Results >0 indicate that the model is a better predictor than the average of the observations.</td></tr></tbody></table>

*Note.* List of skill assessment metrics, their usage, and additional notes. Adapted from Stow et al. (2009). See Table S2 in Supporting Information S1 for more information about these metrics.

their calibrated range (Hipsey et al., 2020; Kubicek et al., 2015; Steenbeek et al., 2021). The identification of emergent processes are context- and model-specific, as the important dynamics to be assessed vary depending on the purpose of the model (Novaglio et al., 2024; Petrik et al., 2022). Identifying relationships between historical simulations and forecasted ocean states through emergent constraints will help address the challenge arising from the hypothetical nature of model projections.

## 2.2. Metrics for Level 1 FishMIP Model Assessment

We adapted the CSPS framework to the skill assessment needs of FishMIP in three steps. First, we used the CSPS framework as a benchmark to categorize model skill assessment approaches proposed in other papers (see Table S1 in Supporting Information S1). Second, we used best practices and commonly agreed-upon statistical measures to recommend FishMIP-appropriate assessment measures (Table 1). Third, we applied the framework to
two structurally contrasting global FishMIP models and assessed their respective ability to reproduce historical
fisheries catches at both the global and Large Marine Ecosystem (LME) scale.

We used a synthesis of goodness-of-fit metrics from existing skill assessment literature (Table 1, S2), from which we selected a range of statistical measures for this study (Table 1). Following the advice of Legates and McCabe (1999) in using both relative and absolute error measures, for our Level 1 assessment we used a visual representation of correlation and bias, and a Taylor diagram plot of standard deviation, Pearson correlation and centered root mean squared error. Alongside this, we calculated 6 independent metrics of model skill assessment (Table 1; Allen & Somerfield, 2009; Bennett et al., 2013; Mayer & Butler, 1993; Stow et al., 2009; Taylor, 2001). These metrics measure: (a) the models' ability to replicate trends over time (Pearson correlation (R)); (b) bias between projections and observations (average error (AE), root mean squared error (RMSE), mean absolute error (MAE)); and (c) a combination of trend and bias (reliability index (RI), and modelling efficiency (MEF)). These skill metrics are detailed in Table 1 and Table S2 in Supporting Information S1. These metrics are calculated for the two FishMIP MEMs, considered here and described in the next section, and are tabulated for comparison.

**2.3. Marine Ecosystem Models**

We obtained model data from two published global MEMs that are members of the FishMIP ensemble and have
provided historical simulation outputs of fisheries catches under two FishMIP simulation protocols (using
Coupled Model Intercomparison Project (CMIP) Phase 5 and 6 Earth system model forcings, respectively); the
BiOeconomic mArine Trophic Size-spectrum model (BOATS; Carozza et al., 2016, 2017) and EcoOcean
(Christensen et al., 2015; Coll et al., 2020). These models are a subset of the 9 FishMIP global MEMs (Tittensor
et al., 2021), but they are the only two that had historical outputs of fisheries catches across the two simulation
rounds at time of publication. Nevertheless, these models capture a significant portion of the spectrum of model
complexity across the full FishMIP ensemble, from BOATS which resolves individual organisms by body size
alone, to EcoOcean which explicitly incorporates information about thousands of species.

BOATS is a size-structured model that uses broad-scale ecological relationships and individual-level metabolic
constraints to calculate the production of fish biomass. It is coupled with an economic module that determines
fishing effort and harvest based on the profitability of the exploitation of this biomass given globally homogenous
economic boundary conditions (Carozza et al., 2016, 2017). BOATS fish biomass production is driven by water
temperature (averaged over the top 75 m) and depth-integrated net primary production from Earth System Models
(ESMs) and implicitly includes all commercially fished animal biomass from 10 g to 100 kg for three fish groups
of increasing asymptotic mass (0.3, 8.5 and 100 kg, respectively). By default, in each grid cell of the simulated
domain BOATS assumes open-access fishing effort dynamics (Carozza et al., 2016; Tittensor et al., 2018),
although it can be forced by observational reconstructions of effort and other social or economic drivers (Scherrer
& Galbraith, 2020). BOATS uses three size-based groups to represent marine ecosystems. Parameters related to
trophic transfer efficiency and other ecological processes were calibrated using aggregate global catch data from
the Sea Around Us Project (SAUP) and biomass estimates from the RAM legacy database, by comparing
simulated harvests to observed peak catches across LMEs using Markov Chain Monte Carlo random sampling
(Carozza et al., 2017).

EcoOcean is a combined trophodynamic and species distribution model with a mass-balanced food web model at its core (Christensen et al., 2015; Coll et al., 2020). It uses fishing effort and gear type as forcings, and a gravity model to spatially spread effort across grid cells within LMEs based on expected profitability and fishing costs. Fish prices, used to estimate expected revenue, are model inputs while fishing costs are assumed to be proportional to the grid cell's distance from the nearest coast. Both fish prices and costs are used to calculate fishing effort. The EcoOcean realisation used in this study considers depth-resolved water temperature and depth-integrated small and large phytoplankton biomass as drivers. EcoOcean resolves 51 functional groups, including fish, sharks and rays, invertebrates and mammals, to represent the whole spectrum of marine organisms and integrates explicit information for 3,400 species of marine organisms (Christensen et al., 2015; Coll et al., 2020; Tittensor et al., 2021). EcoOcean uses spatial global fisheries catches from SAUP to evaluate model runs, parameterize landings by fleet and parameterize species-specific processes through stochastic stock reduction analysis, foraging arena theory for trophic interactions and habitat preferences based on depth distributions (Christensen et al., 2015).
Both MEMs use common simulation protocols, as defined by the Inter-Sectoral Impact Model Intercomparison Project (ISIMIP; www.isimip.org). We used output from protocols ISIMIP2b and ISIMIP3b (Blanchard et al., 2024; Friel er et al., 2017, 2024; Tittensor et al., 2018, 2021), which used climate forcings from the CMIP5 and CMIP6, respectively.

Total catch output data were provided in a standardised 1° grid cell format monthly. Historical simulations from both MEMs spanned 1971–2005 for ISIMIP2b and 1950–2014 for ISIMIP3b. All data used in this study is preserved online (available here: doi.org/10.5281/zenodo.13922550) (Rynne, 2024).

## 3. Marine Ecosystem Model Forcings and Observational Data

For both ISIMIP2b and 3b, BOATS and EcoOcean were forced with outputs from two Earth System Models (ESMs): Geophysical Fluid Dynamics Laboratories (GFDL) (version ESM2M and ESM4.1 for ISIMIP2b and ISIMIP3b, respectively; Dunne et al., 2012, 2020) and Institut Pierre-Simon Laplace (IPSL) (version CM5A-LR and CM6A-LR for ISIMIP2b and ISIMIP3b, respectively; Boucher et al., 2020; Sepulchre et al., 2020). These ESM simulations forced the FishMIP ensemble models for both ISIMIP simulation rounds (e.g., CMIP5 in Lotze et al., 2019; CMIP5 and CMIP6 in Tittensor et al., 2021).

Two global fishing catch data sets were initially considered to capture the variability and biases from different data reconstruction methodologies. The first, from Watson and Tidd (2018), covers the historical period 1869–2017 and combines official reconstructed estimates of fisheries catch data, including major discards, from the Food and Agriculture Organisation (FAO) FishStat database along with other publicly available sources (Watson, 2019; Watson & Tidd, 2018; http://dx.doi.org/10.25959/5c522cadbea37). The second, from the Sea Around Us Project (SAUP), covers the historical period 1950–2019, uses FAO-reported landings, Regional Fisheries Management Organisations (RFMOs), expert elicitation and other publicly available sources (Pauly & Zeller, 2016). However, at the global scale, the difference between these two data sets is small, relative to the divergence between observations and simulations (Figure 1). Therefore, we used only the Watson and Tidd (2018) reconstruction to calculate model performance metrics.

## 4. Results

### 4.1. Global Scale Skill Assessment

Globally averaged historical time series of simulated catches were strongly correlated with Watson and Tidd observations for both EcoOcean and BOATS (Figures 1b and 1d). CMIP5-forced correlations were lower for both models compared to CMIP6 (Tables 2 and 3), especially for BOATS-IPSL (R = 0.47 vs. R = 0.96). For CMIP6-forced simulations, correlation coefficients, R, ranged between 0.93 and 0.98, with slightly higher values for BOATS than for EcoOcean (Table 3). Furthermore, bias was substantially improved in EcoOcean CMIP6-forced simulations compared to CMIP5 (Figure 1a vs. Figure 1c). All simulations with EcoOcean generally underestimated catch, while BOATS-GFDL strongly overestimated catch across CMIP5 and 6, whereas the bias of BOATS-IPSL was reduced for both CMIP5 and CMIP6 (Figure 1).

The calculated errors (AE, MAE and RMSE) confirm large discrepancies in the magnitude of simulated and observed catches (Tables 2 and 3). In CMIP6-forced models, the negative result for AE for BOATS-IPSL reflected that simulated catches were lower than observations before ~1980 and higher after ~1990. This led to an AE result closer to zero than other MEM simulations because positive and negative measures canceled each other out (Figure 1, Table 3). CMIP6-forced models showed both improvement and deterioration in AE, MAE and RMSE compared to CMIP5-forced models (Tables 2 and 3). Root mean squared error results for all CMIP6-forced models were higher than MAE results, likely indicating the presence of large outlier values in the simulated catches (Legates & McCabe, 1999).

Reliability Index (RI) results showed that all CMIP6-forced models differed from fishing catch by between 1.32 and 1.54-fold on average compared to observations (Table 3). In contrast, for CMIP5 simulated catches, results differed by up to 4.32-fold (BOATS-GFDL) compared to the observed data (Table 2). This highlighted improvements in catch estimations between GFDL CMIP5- and 6-forced models. In particular, BOATS-GFDL CMIP6-forced runs showed the largest improvement, with an RI of 1.47 for BOATS-GFDL, compared to the CMIP5-forced result of 4.32 (Tables 2 and 3). Historical catch simulated with CMIP6-IPSL-forcing resulted in worsened RI slightly compared to CMIP5-IPSL-forced runs (Table 3).
<figure><img src="image_3.png" /><figcaption>**Figure 1.** Bias and correlation for modeled and observed global fishing catch time series. Reconstructed observations from Watson and Tidd (2018) and SAUP (Pauly & Zeller, 2016) and model projected catch depicting bias for (a) CMIP5 from 1971 to 2005; and (c) CMIP6 from 1950 to 2014. Scatterplot of Watson and Tidd (2018) reconstructed observations versus. model predicted catch depicting correlation for (b) CMIP5-forced BOATS (top) and EcoOcean (bottom); and (d) CMIP6-forced BOATS (top) and EcoOcean (bottom).</figcaption></figure>

For all CMIP5-forced MEM and ESM combinations, modelling efficiency was less than zero, meaning that the average of the observations was more skillful than the models' estimates over the historical period (Table 2). In contrast, modelling efficiency (MEF) was greater than zero for CMIP6-forced BOATS-IPSL and EcoOcean-GFDL while EcoOcean-IPSL was very close to zero (Table 3). Modeling efficiency remained negative for CMIP6-forced BOATS-GFDL, albeit greatly improved from CMIP5 (−2.28 versus −614.98) (Table 3).
<table><caption>**Table 2**
Global Forecast Skill Metrics for Fishing Catch With CMIP5 Forcing</caption><thead><tr><th rowspan="3">Skill metric</th><th colspan="4">MEM-ESM</th></tr><tr><th colspan="2">BOATS</th><th colspan="2">EcoOcean</th></tr><tr><th>GFDL-ESM2M</th><th>IPSL-CM5A-LR</th><th>GFDL-ESM2M</th><th>IPSL-CM5A-LR</th></tr></thead><tbody><tr><td><strong>R</strong></td><td>0.85</td><td>0.47</td><td>0.84</td><td>0.87</td></tr><tr><td><strong>RMSE</strong></td><td>3.81 × 10<sup>8</sup></td><td>1.76 × 10<sup>7</sup></td><td>4.69 × 10<sup>7</sup></td><td>3.25 × 10<sup>7</sup></td></tr><tr><td><strong>MAE</strong></td><td>3.78 × 10<sup>8</sup></td><td>1.42 × 10<sup>7</sup></td><td>4.48 × 10<sup>7</sup></td><td>2.97 × 10<sup>7</sup></td></tr><tr><td><strong>AE</strong></td><td>3.78 × 10<sup>8</sup></td><td>1.11 × 10<sup>7</sup></td><td>-4.48 × 10<sup>7</sup></td><td>-2.96 × 10<sup>7</sup></td></tr><tr><td><strong>RI</strong></td><td>4.32</td><td><strong>1.17</strong></td><td>1.82</td><td>1.46</td></tr><tr><td><strong>MEF</strong></td><td>-6.15 × 10<sup>2</sup></td><td>-0.31</td><td>-8.32</td><td>-3.48</td></tr></tbody></table>

Note. Skill metric performance for six skill metrics using Watson and Tidd (2018) observations: correlation (R), root mean squared error (RMSE; g/m²), mean absolute error (MAE; g/m²), average error (AE; g/m²), reliability index (RI) and modeling efficiency (MEF) for BOATS and EcoOcean models under both ESM forcings. Results in bold are close to ideal results.

Taylor diagrams for global catch from BOATS and EcoOcean summarize some of the previous observations. Both models demonstrate an improvement in correlation and bias when forced with CMIP6 data compared to CMIP5 data, however, BOATS-IPSL simulations show an increase in standard deviation from CMIP5- to CMIP6-forced runs (Figure 2).

## 4.2. Large Marine Ecosystem Scale Assessment

Correlations between simulated catches and Watson and Tidd (2018) reconstructed observed fishing catch varied across the LMEs, indicating differing levels of model performance at the regional scale. For all CMIP5-forced models, the median correlation (median across LME-levels) was near zero (Figure 3). Results from CMIP6-forced models showed improvement in the median and interquartile range of correlation results compared to CMIP5-forced models (Figure 3), indicating improved correlation at the LME scale overall.

Geographically, this improvement in correlation from CMIP5 to CMIP6 was evident for both BOATS and EcoOcean, but the degree of improvement (and where this occurs) differs between ESM-forcings (Figure 4, S3). For BOATS, there was an improvement in correlations in CMIP6 compared to CMIP5 across 51 and 52 LMEs with GFDL and IPSL forcing respectively. Similarly, for EcoOcean there was an improvement in correlations across 53 and 54 LMEs for GFDL and IPSL, respectively (Table S3 in Supporting Information S1). BOATS showed improvements in highly productive LMEs including in the Humboldt Current, Pacific Central-American

<table><caption>**Table 3**
Global Skill Metrics for Fishing Catch Time Series With CMIP6 Historic Forcing</caption><thead><tr><th rowspan="3">Skill metric</th><th colspan="4">MEM-ESM</th></tr><tr><th colspan="2">BOATS</th><th colspan="2">EcoOcean</th></tr><tr><th>GFDL-ESM4.1</th><th>IPSL-CM6A-LR</th><th>GFDL-ESM4.1</th><th>IPSL-CM6A-LR</th></tr></thead><tbody><tr><td><strong>R</strong></td><td><strong>0.98</strong></td><td><strong>0.96</strong></td><td><strong>0.93</strong></td><td><strong>0.93</strong></td></tr><tr><td><strong>RMSE</strong></td><td>5.88 × 10<sup>7</sup></td><td>2.10 × 10<sup>7</sup></td><td>2.34 × 10<sup>7</sup></td><td>3.25 × 10<sup>7</sup></td></tr><tr><td><strong>MAE</strong></td><td>4.84 × 10<sup>7</sup></td><td>1.83 × 10<sup>7</sup></td><td>2.04 × 10<sup>7</sup></td><td>2.97 × 10<sup>7</sup></td></tr><tr><td><strong>AE</strong></td><td>4.73 × 10<sup>7</sup></td><td>-1.13 × 10<sup>7</sup></td><td>-2.03 × 10<sup>7</sup></td><td>-2.97 × 10<sup>7</sup></td></tr><tr><td><strong>RI</strong></td><td><strong>1.47</strong></td><td><strong>1.54</strong></td><td><strong>1.32</strong></td><td><strong>1.49</strong></td></tr><tr><td><strong>MEF</strong></td><td>-2.28</td><td><strong>0.58</strong></td><td><strong>0.48</strong></td><td>-1.28 × 10<sup>-3</sup></td></tr></tbody></table>

Note. Skill metric performance for six skill metrics using Watson and Tidd (2018) observations: correlation (R), root mean squared error (RMSE; g/m²), mean absolute error (MAE; g/m²), average error (AE; g/m²), reliability index (RI) and modeling efficiency (MEF) for BOATS and EcoOcean models under both ESM forcings. Results in bold are close to ideal results.
<figure><img src="image_4.png" /><figcaption>**Figure 2.** Taylor Diagram for CMIP5 and CMIP6 simulations. (a) BOATS model predicted global catch (from 1971 to 2005; million tonnes per year) for CMIP6 using ESM IPSL (light red) and GFDL (light blue), and CMIP5 ESM IPSL (dark red) and GFDL (dark blue); and (b) EcoOcean model predicted global catch (from 1971 to 2005) for CMIP6 using ESM IPSL (light orange) and GFDL (light green), and CMIP5 ESM IPSL (dark orange) and GFDL (dark green). Plot shows standard deviation, correlation and centered root mean squared error for all 4 MEM-ESM combinations. Watson and Tidd (2018) observed global catch (between 1971 and 2005) in purple.</figcaption></figure>

Coast, Barents Sea, North Brazil Shelf, Patagonian Shelf and Canary Current (Figure 4; Table S3 in Supporting Information S1). In contrast, EcoOcean's largest correlation improvements were more randomly dispersed across European, east African and east South American LMEs (Figure 4; Table S4 in Supporting Information S1)
<figure><img src="image_5.png" /><figcaption>**Figure 3.** Box plot of Large Marine Ecosystem (LME) level correlation. (a) BOATS model correlation compared to reconstructed observations from Watson and Tidd (2018). CMIP5-forced ESM IPSL (light red) and GFDL (light blue), and CMIP6-forced ESM IPSL (dark red) and GFDL (dark blue); (b) EcoOcean model correlation compared to reconstructed observations from Watson and Tidd (2018). CMIP5-forced ESM IPSL (light orange) and GFDL (light green), and CMIP6-forced ESM IPSL (dark orange) and GFDL (dark green).</figcaption></figure>

including the Celtic-Biscay Shelf and Benguela Current. Negative correlations between observed and modeled catches persist across all simulations in polar regions (Figure 4, Tables S3, S4 in Supporting Information S1).

To address potential concerns about aggregation biases, such as Simpson's paradox, a leave-one-out analysis was conducted to determine if there are any individual LMEs that had an outsized impact on the calculated metrics (Figures S3–S6 in Supporting Information S1). This analysis revealed that the metrics generally did not change when individual LMEs were excluded, though with a few exceptions. When LME 13 (Humboldt Current) was excluded the modelling efficiency for BOATS forced by GFDL-ESM2M decreased more than when any other
<figure><img src="image_6.png" /><figcaption>**Figure 4.** Map of Pearson correlations across the world's LMEs. BOATS and EcoOcean predictions and reconstructed observations from Watson and Tidd (2018), using CMIP5 from 1971 to 2005 (a), (c), (e), (g) and CMIP6 from 1950 to 2014 (b), (d), (f), (h) ESM forcings. Higher resolution regional plots are provided in Figures S7 to S9 in Supporting Information S1.</figcaption></figure>

LME was excluded (Figure S3 in Supporting Information S1). At the same time, when the high seas (designated as LME 0) were excluded several metrics were influenced. Correlation for BOATS decreased from 0.84 to 0.51 using GFDL-ESM2M forcing, and from 0.47 to -0.67 for IPSL-CM5A-LR forcing when the high seas were
excluded (Figure S4 in Supporting Information S1). Root mean squared error for BOATS forced by GFDL-ESM2M improved, while RMSE for BOATS forced by IPSL and EcoOcean using all forcings worsened when excluding the high seas (Figure S5 in Supporting Information S1). Finally, when the open ocean was excluded, the RI worsened for all models and forcings, except for BOATS GFDL-ESM2M, which improved (Figure S6 in Supporting Information S1).

**5. Discussion**

Model skill assessment is essential for improving the credibility and reliability of Marine Ecosystem Model (MEM) simulations and for supporting their use as decision-making tools. Until now Fisheries and Marine Ecosystem Model Intercomparison Project (FishMIP) models have generally been assessed in isolation. Here, we adapted and implemented a standardised skill assessment framework to highlight commonalities and discrepancies between modelled and observed historical fish catch and to investigate the usefulness of a range of skill assessment metrics.

Agreement between simulated and observed catches across BOATS and EcoOcean, both in catch time-series variability and absolute catches, is generally higher for CMIP6- than CMIP5-forced models (Figure 1; Tables 1 and 2). These improvements may be due to changes in the MEMs. For example, EcoOcean has recently undergone substantial restructuring and the upgrades include an expanded food web from 1,400 to over 3,400 explicitly considered individual species, updated functional group representation, and the use of observed historical spatial ranges of species to initialize the model (Coll et al., 2020). This has resulted in an improved understanding and representation of key ecological and fishing dynamics. Between CMIP5 and CMIP6 BOATS' biological formulation and parameters were not changed. However, to improve the match with observed catches, starting effective effort (which then increases through time with improving catchability) was calibrated to align the model's aggregated catch by LMEs with observational reconstructions from Sea Around Us Project (SAUP; Pauly & Zeller, 2016).

While these modifications and—in the case of EcoOcean - a reconsideration of model assumptions, in line with Level 0 (conceptual) assessment, are likely substantial drivers of each model's better performance from CMIP5 to CMIP6, some of the improvement in simulated fishing catches would reflect changes in the two Earth System Models (ESMs) that provided input-forcing data to the FishMIP models (Figure S1 in Supporting Information S1; Séférian et al., 2020; Tittensor et al., 2021). Across CMIP5 and CMIP6, both ESMs captured observed long-term mean sea-surface temperature (a driver of both models) at the LME scale (Figures S1c and S1d in Supporting Information S1). Although both models were 0.2–1°C warmer in CMIP6 than CMIP5, they did not show much improvement in resolving net primary production (NPP; a BOATS environmental forcing) at the LME scale (Figures S1a and S1b in Supporting Information S1). However, across both models, phytoplankton carbon (an EcoOcean forcing) was lower in CMIP6 compared to CMIP5 (Figure S2 in Supporting Information S1), which may partly explain why EcoOcean catch bias improved between the two simulations. Ultimately, changes in both MEMs and updated ESM forcings likely contribute to improvements in model skill. Disentangling these drivers would be a fruitful avenue of future research to improve catch and biomass simulations from MEMs.

**5.1. Reasons for Bias in Simulated Catches**

Bias in fish catch across models are likely driven by a range of factors. For instance, lower trophic level (LTL) biomass and production from ESMs are major drivers of the projected spatial distribution of fish biomass and fisheries catches (Chassot et al., 2010; Heneghan et al., 2021; Kwiatkowski et al., 2020; Laufkötter et al., 2015; Stock et al., 2017; Tagliabue et al., 2021). Thus, discrepancies between observed and modeled LTL variables at the LME scale and across the open ocean (Figures S1a, S1b and S3–S6 in Supporting Information S1) will have an impact on MEM fish biomass and therefore catches. This may lead to an overestimation of fishing catches in the high seas and an underestimation in the near-coastal LMEs (Guiet et al., 2024). In the case of EcoOcean, the one-way forcing of phytoplankton biomass from ESM estimates potentially allows for bias in the estimation of higher trophic level (HTL) biomass, compared to what could be supported by LTL in a two-way coupling. In the case of BOATS, a single energy pathway connects NPP to the accumulation of commercially exploited fish species, while in reality surface pelagic species and bottom living species rely on different food chains, and experience different water temperatures (van Denderen et al., 2018). This may lead to an overestimation of demersal biomass in BOATS, ultimately leading to excess global catches (Guiet et al., 2024). Finally, internal climate variability
within the ESMs used here was not calibrated to observed variability. This means that seasonal and annual climate patterns affecting simulated catches from the MEMs will not match observation over these shorter time scales.

The ecological and fishing components of BOATS and EcoOcean, as with all MEMs, are highly simplified representations of the real world that also differ between models. It is important to note that catch reconstructions are also approximations of reality, subject to numerous sources of bias and error. This necessarily results in discrepancies between models, catch reconstructions and actual catches. For example, fishing in BOATS is here determined by globally homogenous and historically constant economic factors, such as constant fish price and fishing costs, and the development of fisheries in each grid cell is driven by the assumption of a historical increase at a constant rate of the technology-driven catchability of fish biomass, which turns initially unprofitable fishing grounds into profitable ones that can be exploited. This assumption is completed with the assumption of open, unregulated fishing access in the world's oceans (Carozza et al., 2016, 2017). These simplifications can capture broad trends in catches at a global level (Galbraith et al., 2017; Guiet et al., 2020), which show a steep increase until the mid-1990s and a later plateau or decline due to overexploitation of fish stocks, and in space a sequential shift of the development of fisheries from cool and productive to warm and unproductive regions (Pauly & Zeller, 2016; Watson & Tidd, 2018). However, they lack important dynamics, which may restrict or change patterns in fishing effort and therefore catches in the real world, particularly at the LME scale considered here.

Finally, other internal issues regarding the way key bio-ecological processes are parameterized can lead to divergence between observed and modelled catch rates, as well as the wide differences between BOATS and EcoOcean historical simulations. To identify specific internal drivers of bias and errors, further experimental attribution simulations would be necessary to separate the impact of individual ecological and fisheries components within each model (Steenbeek et al., 2024). Such experimental studies have already successfully identified key drivers of structural uncertainty in the FishMIP ensemble (Heneghan et al., 2021), and biogeochemical modelling community (Laufkötter et al., 2015).

## 5.2. Evaluation of Metrics for Marine Model Assessment

Our case study highlights how the use of multiple metrics is necessary to obtain a multi-faceted perspective of the credibility and reliability of model projections. While summary statistics correlation (R), reliability index (RI) and modelling efficiency (MEF) provide quick and useful information about model credibility, allow comparison between models or regions, and are generally easy to interpret, some important information about the ecosystem is necessarily lost as these metrics reduce the time-series into a single datum (Bennett et al., 2013; Stow et al., 2009). In addition, these metrics do not assess the ability of the models to capture observed geographical patterns in catches, with this shortcoming highlighting the need for spatial skill assessment tools, such as pattern correlation tools, to be developed and applied in parallel.

Providing too many metrics that measure the same aspect of model skill may instil false confidence in model assessment by unnecessarily replicating the same result (Olsen et al., 2016). Root mean squared error, mean absolute error (MAE) and average error (AE) are all slightly different ways of calculating the same measure—the magnitude of the bias between model simulations and observations. Therefore, it is not necessary to use all three bias metrics moving forward (AE, MAE, RMSE). As AE can be affected by positive and negative discrepancies canceling each other out, and as RMSE is more sensitive to outliers than MAE, we recommend that MAE be the primary metric used to measure bias. On the other hand, evaluating too few metrics, or metrics that only explore one component of model performance can also instil false confidence. For instance, in previous FishMIP syntheses, model outputs were normalized to show only relative changes (Heneghan et al., 2021; Lotze et al., 2019; Tittensor et al., 2021). Normalization made it possible to generate a more consistent picture of climate change impacts on marine animal biomass, but may have given a false impression of model agreement since it omitted information on discrepancies in absolute biomass across models. Therefore, we argue that it is essential for metrics exploring both absolute and normalized quantities, as used in this case study, to be deployed when assessing MEM performance.

Lastly, at present neither the observational data sets, nor model projections contain measures of uncertainty in their simulations of fisheries catch. Although individual models have assessed the impact of parameter uncertainty on estimates of biomass and fishing catch (Carozza et al., 2016; Galbraith et al., 2017; Jennings & Colingridge, 2015), existing FishMIP protocols focus on assessing long-term climate impacts on marine ecosystems and have only captured scenario and model uncertainty, by running an ensemble of models over a range of climate
scenarios. This means that, at present, no FishMIP ensemble data exists where models provide standardised measures of uncertainty in their catch or biomass estimates. Looking ahead, capturing observed and modeling catch uncertainty would make it possible to assess where predictions are falling within the uncertainty range of observed catch across different LMEs, which is critical to further refine ensemble projections.

### 5.3. Future Research

This paper sets out the features of the CSPS framework and conducts a Level 1 skill assessment for two models within the FishMIP ensemble, on simulated catch. We finish by discussing how this process can continue to be improved.

Although 9 global MEMs contribute to the FishMIP ensemble, fisheries catch simulations from only two FishMIP models were available for this assessment. However, we expect that other global and regional models will provide catch outputs as part of the current round of simulations (Blanchard et al., 2024; Frieler et al., 2024) and of ongoing FishMIP efforts to design and implement socioeconomic scenarios that consistently simulate fisheries catch across ecosystem models (Blanchard et al., 2024; Maury et al., 2024).

Individual FishMIP models provide a range of integrated outputs besides total catches analyzed here, including catches by functional group (i.e., demersal and pelagic) and by size class (e.g., small, medium and large) (Carozza et al., 2016; Cheung et al., 2011; Coll et al., 2020; Maury, 2010; Maury & Poggiale, 2013; Petrik et al., 2019). The analysis of these outputs can add to the model performance picture and can provide insights into modelled ecosystem structure and function. For example, the collapse of large target species and the increase of smaller species due to predation release (Blanchard et al., 2012; Christensen et al., 2014) can drive fisheries catch, but this process is hidden when considering aggregated biomass and catch outputs. Analysis of these existing outputs is an important next step for FishMIP, and forms part of Level 2 (process) and Level 3 (system) assessment in the CSPS framework (Hipsey et al., 2020). Looking ahead, assessing emergent and system-level relationships between ESM variables and MEM output, or between the internal state variables within the MEMs, also offer considerable potential for enhancing Level 2 and Level 3 assessments of MEM performance (Novaglio et al., 2024). Ultimately, an extensive Level 2 and 3 assessment of the FishMIP ensemble will require models to provide outputs that are not currently part of the CMIP and FishMIP protocols, including primary and secondary production rates, biodiversity turnover, trophic transfer rates or growth rates. Eliciting this information in future simulation protocols is therefore critical, since it will provide scope for in-depth assessment of modeled processes across the FishMIP ensemble.

The current simulation round of FishMIP is focussed on “Detection, Attribution and Evaluation” (ISIMIP3a, www.fishmip.org), and therefore aims to tackle issues such as resolution, coastal processes, and standardization of fishing inputs across models. To that end, finer scale inputs from ESMs may help the performance of MEMs at the regional scale. There also exists the opportunity to use FishMIP simulations coupled to ESM models forced by reanalysis data (Blanchard et al., 2024) which are constrained by observational products of atmospheric drivers, to calibrate MEMs, or to conduct post-hoc correction of FishMIP outputs (Gómara et al., 2021; Maury et al., 2024). Unlike fully coupled ESM historical simulations, ocean-only reanalysis-based simulations would have climate oscillations like ENSO cycles occurring at the correct times in history, and thus would hopefully produce more skillful comparisons of time series (e.g., Barrier et al., 2023).

### 6. Conclusions

Performing model skill assessment on complex end-to-end ecosystem models is an essential, yet challenging task, and there is still considerable progress to be made before model simulations replicate historical observations. Marine Ecosystem Models play an important role in developing our understanding of climate change impacts on future fisheries catches and marine ecosystems, and how that might affect global food security (Blanchard et al., 2012, 2017; Booth et al., 2017; Cheung et al., 2010; Cinner et al., 2022; Hollowed et al., 2013). Rigorous ensemble model skill assessment increases confidence in using MEM projections to inform policy, as well as identifying priority areas for future model improvement.

Overall, this case study showed that global fishery catch estimates are well correlated with observed trends over time, but both models show important scale mismatches that require further attention. This exercise provides useful information on the performance of two global models contributing to FishMIP and can be further used to
drive model development to improve the reliability of climate impact projections, as well as applied more broadly
across the whole suite of FishMIP models to enhance the utility of FishMIP as a whole. We finish with a set of
summary recommendations for how FishMIP (and other ensemble model projects) could better integrate model
ensemble Level 0–3 skill assessment for future simulation protocols.

1. Level 0: A comprehensive understanding of the underlying assumptions and parameterizations across the model ensemble is essential to understand why MEMs agree or disagree under different conditions. Future protocols targeted at disentangling sources of structural uncertainty across the FishMIP ensemble would concretely improve our understanding of why MEMs behave the way they do. This also includes simulation studies focussed on improving our understanding of the linkages and dependencies between MEMs and the ESMs that force them.

2. Level 1: FishMIP should move beyond exploring only relative change in simulated variables across the model ensemble, to assessing absolute change and variability. This will require using assessment metrics that capture model bias, such as MAE, RI, or MEF.

3. Level 2 and 3: To properly assess the processes and emergent properties of ensemble MEMs, future simulation protocols must require modellers to provide more than just aggregate biomass or catch. At the same time, data products on emergent ecosystem properties such as biomass size-spectra need to be assembled at spatial and temporal resolutions appropriate for comparison with global MEMs.

The CSPS framework provides a solid basis for standardizing skill assessment for FishMIP, to which other
metrics (e.g., size-based metrics) could be added. The hierarchical structure and focus of each level act as clear
guidelines to measure the predictive validity of MEMs. These initial results show that, although we are yet to fully
assess the current ensemble of global marine models, we have the tools and knowledge to tackle this task.

Data Availability Statement

No new data was produced as part of this manuscript. All analyses were run using the statistical software R (R Core Team, 2023) version 4.2.3 (15 March 2023; Shortstop Beagle) and the following packages: *colorspace* (Ihaka et al., 2024; Zeileis et al., 2020); *dplyr* (Wickham, François, et al., 2023); *forcats* (Wickham & RStudio, 2023); *ggnewscale* (Campitelli, 2024); *ggplot2* (Wickham, 2016; Wickham, Chang, et al., 2024); *here* (Müller, 2020); *Metrics* (Hamner et al., 2018); *openair* (Carslaw et al., 2024; Carslaw & Ropkins, 2012); *patchwork* (Pedersen, 2024); *purrr* (Wickham, Henry, et al., 2023); *qualV* (Boogaart et al., 2023; Jachner et al., 2007); *raster* (Hijmans et al., 2024); *readr* (Wickham et al., 2015); *rnaturalearth* (Massicotte et al., 2023); *scales* (Wickham, Pedersen, et al., 2023); *sf* (Pebesma et al., 2024); *stringr* (Wickham & Posit Software, 2023); *tibble* (Müller et al., 2023); *tictoc* (Izrailev, 2024); *tidyr* (Wickham, Vaughan, et al., 2024); *tidyverse* (Wickham, Vaughan, et al., 2023); *topmodel* (Buytaert, 2022). The R Markdown code and data used to report on the analyses in the paper and produce Figures 1–4 and Tables 2 and 3 are available in Rynne (2024).

Acknowledgments

MC acknowledges support from the Spanish project ProOceans (Plan Estatal de Investigación Científica, Técnica y de Innovación, 2020, PID2020-118097RB-I00) and the ‘Severo Ochoa Centre of Excellence’ accreditation (CEX2019-000928-S) to the Institute of Marine Science (ICM-CSIC). CMP acknowledges support from NOAA grants NA20OAR4310438, NA20OAR4310441, and NA20OAR4310442. KOC acknowledges support from the National Research Foundation of South Africa (Grant 136481). OM acknowledges support from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 817806. Open access publishing facilitated by Queensland University of Technology, as part of the Wiley - Queensland University of Technology agreement via the Council of Australian University Librarians.

References

Allen, J. I., & Somerfield, P. J. (2009). A multivariate approach to model skill assessment. *Journal of Marine Systems*, 76(1–2), 83–94. https://doi.org/10.1016/j.JMARSYS.2008.05.009

Barrier, N., Lengaigne, M., Rault, J., Person, R., Ethé, C., Aumont, O., & Maury, O. (2023). Mechanisms underlying the epipelagic ecosystem response to ENSO in the equatorial Pacific ocean. *Progress in Oceanography*, 213, 103002. https://doi.org/10.1016/j.pcean.2023.103002

Baumberger, C., Knutti, R., & Hirsch Hadorn, G. (2017). Building confidence in climate model projections: An analysis of inferences from fit. *Wiley Interdiscip. Rev. Clim. Change*, 8(3), e454. https://doi.org/10.1002/wcc.454

Bennett, N. D., Croke, B. F. W., Guariso, G., Guillaume, J. H. A., Hamilton, S. H., Jakeman, A. J., et al. (2013). Characterising performance of environmental models. *Environmental Modelling and Software*, 40, 1–20. https://doi.org/10.1016/j.ENVSOFT.2012.09.011

Blanchard, J. L., Jennings, S., Holmes, R., Harle, J., Merino, G., Allen, J. I., et al. (2012). Potential consequences of climate change for primary production and fish production in large marine ecosystems. *Philosophical Transactions of the Royal Society of London Series B Biological Sciences*, 367(1605), 2979–2989. https://doi.org/10.1098/RSTB.2012.0231

Blanchard, J. L., Novaglio, C., Maury, O., Harrison, C. S., Petrik, C. M., Arcos, L. D. F., et al. (2024). Detecting, attributing, and projecting global marine ecosystem and fisheries change: FishMIP 2.0. https://doi.org/10.22541/essoar.170594183.33534487/v1

Blanchard, J. L., Watson, R. A., Fulton, E. A., Cottrell, R. S., Nash, K. L., Bryndum-Buchholz, A., et al. (2017). Linked sustainability challenges and trade-offs among fisheries, aquaculture and agriculture. *Nature Ecology and Evolution*, 1(9), 1240–1249. https://doi.org/10.1038/s41559-017-0258-8

Boogaart, K. G., Rost, S., & Petzoldt, T. (2023). qualV: Qualitative validation methods (version 0.3-5). [Computer software]. https://doi.org/10.32614/CRAN/package.qualV

Booth, D. J., Poloczanska, E., Donelson, J. M., Molinos, J. G., & Burrows, M. (2017). Biodiversity and climate change in the oceans. In *Climate change impacts on fisheries and aquaculture* (pp. 63–89). John Wiley and Sons, Ltd. https://doi.org/10.1002/9781119154051.ch4
Boucher, O., Servonnat, J., Albright, A. L., Aumont, O., Balkanski, Y., Bastrikov, V., et al. (2020). Presentation and evaluation of the IPSL-CM6A-LR climate model. *Journal of Advances in Modeling Earth Systems*, 12(7), e2019MS002010. https://doi.org/10.1029/2019MS002010

Buytaert, W. (2022). topmodel: Implementation of the hydrological model TOPMODEL in R (version 0.7.4). [Computer software]. https://doi.org/10.32614/CRAN/package.topmodel

Campitelli, E. (2024). ggnewscale: Multiple fill and colour scales in "ggplot2" (version 0.5.0). [Computer software]. https://doi.org/10.32614/CRAN/package.ggnewscale

Carozza, D. A., Bianchi, D., & Galbraith, E. D. (2016). The ecological module of BOATS-1.0: A bioenergetically constrained model of marine upper trophic levels suitable for studies of fisheries and ocean biogeochemistry. *Geoscientific Model Development*, 9(4), 1545-1565. https://doi.org/10.5194/GMD-9-1545-2016

Carozza, D. A., Bianchi, D., & Galbraith, E. D. (2017). Formulation, general features and global calibration of a bioenergetically-constrained fishery model. *PLoS One*, 12(1), e0169763. https://doi.org/10.1371/journal.pone.0169763

Carslaw, D., Davison, J., & Ropkins, K. (2024). openair: Tools for the analysis of air pollution data (version 2.18-2). [Computer software]. https://doi.org/10.32614/CRAN/package.openair

Carslaw, D., & Ropkins, K. (2012). openair—An R package for air quality data analysis. *Environmental Modelling & Software*, 27-28(0), 52-61. https://doi.org/10.1016/j.envsoft.2011.09.008

Chassot, E., Bonhommeau, S., Dulvy, N. K., Mélin, F., Watson, R., Gascuel, D., & Le Pape, O. (2010). Global marine primary production constrains fisheries catches. *Ecology Letters*, 13(4), 495-505. https://doi.org/10.1111/j.1461-0248.2010.01443.x

Cheung, W. W. L., Dunne, J., Sarmiento, J. L., & Pauly, D. (2011). Integrating ecophysiology and plankton dynamics into projected maximum fisheries catch potential under climate change in the Northeast Atlantic. *ICES Journal of Marine Science*, 68(6), 1008-1018. https://doi.org/10.1093/icesjms/fsr012

Cheung, W. W. L., Lam, V. W. Y., Sarmiento, J. L., Kearney, K., Watson, R., Zeller, D., & Pauly, D. (2010). Large-scale redistribution of maximum fisheries catch potential in the global ocean under climate change. *Global Change Biology*, 16(1), 24-35. https://doi.org/10.1111/j.1365-2486.2009.01995.x

Christensen, V., Coll, M., Buszowski, J., Cheung, W. W. L., Frölicher, T., Steenbeek, J., et al. (2015). The global ocean is an ecosystem: Simulating marine life and fisheries. *Global Ecology and Biogeography*, 24(5), 507-517. https://doi.org/10.1111/geb.12281

Christensen, V., Coll, M., Piroddi, C., Steenbeek, J., Buszowski, J., & Pauly, D. (2014). A century of fish biomass decline in the ocean. *Marine Ecology Progress Series*, 512, 155-166. https://doi.org/10.3354/meps10946

Christensen, V., & Walters, C. J. (2004). Ecopath with ecosim: Methods, capabilities and limitations. *Ecological Modelling*, 172(2), 109-139. https://doi.org/10.1016/j.ecolmodel.2003.09.003

Cinner, J. E., Caldwell, I. R., Thiault, L., Ben, J., Blanchard, J. L., Coll, M., et al. (2022). Potential impacts of climate change on agriculture and fisheries production in 72 tropical coastal communities. *Nature Communications*, 13(1), 1. https://doi.org/10.1038/s41467-022-30991-4

Coll, M., Steenbeek, J., Pennino, M. G., Buszowski, J., Kaschner, K., Lotze, H. K., et al. (2020). Advancing global ecological modeling capabilities to simulate future trajectories of change in marine ecosystems. *Frontiers in Marine Science*, 7, 741. https://doi.org/10.3389/FMARS.2020.567877/BIBTEX

Dunne, J. P., Horowitz, L. W., Adcroft, A. J., Ginoux, P., Held, I. M., John, J. G., et al. (2020). The GFDL Earth system model version 4.1 (GFDL-ESM 4.1): Overall coupled model description and simulation characteristics. *Journal of Advances in Modeling Earth Systems*, 12(11), e2019MS002015. https://doi.org/10.1029/2019MS002015

Dunne, J. P., John, J. G., Adcroft, A. J., Griffies, S. M., Hallberg, R. W., Shevliakova, E., et al. (2012). GFDL's ESM2 global coupled climate-carbon Earth system models. Part I: Physical formulation and baseline simulation characteristics. *Journal of Climate*, 25(19), 6646-6665. https://doi.org/10.1175/JCLI-D-11-00560.1

Eyring, V., Cox, P. M., Flato, G. M., Gleckler, P. J., Abramowitz, G., Caldwell, P., et al. (2019). Taking climate model evaluation to the next level. *Nature Climate Change*, 9(2), 102-110. https://doi.org/10.1038/s41558-018-0355-y

FAO. (2022). The state of world fisheries and aquaculture 2022. Towards blue transformation. FAO. https://doi.org/10.4060/cc0461en

Frieler, K., Lange, S., Piontek, F., Reyer, C. P. O., Schewe, J., Warszawski, L., et al. (2017). Assessing the impacts of 1.5°C global warming – Simulation protocol of the inter-sectoral impact model intercomparison project (ISIMIP2b). *Geoscientific Model Development*, 10(12), 4321-4345. https://doi.org/10.5194/gmd-10-4321-2017

Frieler, K., Volkholz, J., Lange, S., Schewe, J., Mengel, M., del Rocío Rivas López, M., et al. (2024). Scenario setup and forcing data for impact model evaluation and impact attribution within the third round of the Inter-Sectoral Impact Model Intercomparison Project (ISIMIP3a). *Geoscientific Model Development*, 17(1), 1-51. https://doi.org/10.5194/gmd-17-1-2024

Fulton, E. A., Blanchard, J. L., Melbourne-Thomas, J., Plagányi, É. E., & Tulloch, V. J. D. (2019). Where the ecological gaps remain, a Modelers' perspective. *Frontiers in Ecology and Evolution*, 7, 424. https://doi.org/10.3389/FEVO.2019.00424/BIBTEX

Galbraith, E. D., Carozza, D. A., & Bianchi, D. (2017). A coupled human-Earth model perspective on long-term trends in the global marine fishery. *Nature Communications*, 8(1), 1-7. https://doi.org/10.1038/ncomms14884

Geary, W. L., Bode, M., Doherty, T. S., Fulton, E. A., Nimmo, D. G., Tulloch, A. I. T., et al. (2020). A guide to ecosystem models and their environmental applications. *Nature Ecology & Evolution*, 4(11), 1459-1471. https://doi.org/10.1038/s41559-020-01298-8

Gómara, I., Rodríguez-Fonseca, B., Mohino, E., Losada, T., Polo, I., & Coll, M. (2021). Skillful prediction of tropical Pacific fisheries provided by Atlantic Niños. *Environmental Research Letters*, 16(5), 054066. https://doi.org/10.1088/1748-9326/abfa4d

Guiet, J., Bianchi, D., Scherrer, K., Heneghan, R., & Galbraith, E. D. (2024). Small fish biomass limits the catch potential in the High Seas. https://doi.org/10.22541/au.170967563.32290483/v1

Guiet, J., Galbraith, E. D., Bianchi, D., & Cheung, W. W. L. (2020). Bioenergetic influence on the historical development and decline of industrial fisheries. *ICES Journal of Marine Science*, 77(5), 1854-1863. https://doi.org/10.1093/icesjms/fsaa044

Halpern, B. S., Walbridge, S., Selkoe, K. A., Kappel, C. V., Micheli, F., D'Agrosa, C., et al. (2008). A global map of human impact on marine ecosystems. *Science*, 319(5865), 948-952. https://doi.org/10.1126/science.1149345

Hamilton, S. H., Fu, B., Guillaume, J. H. A., Badham, J., Elsawah, S., Gober, P., et al. (2019). A framework for characterising and evaluating the effectiveness of environmental modelling. *Environmental Modelling and Software*, 118, 83-98. https://doi.org/10.1016/J.ENVSOF.2019.04.008

Hamner, B., Frasco, M., & LeDell, E. (2018). Metrics: Evaluation metrics for machine learning (version 0.1.4). [Computer software] https://CRAN.R-project.org/package=Metrics

Hatton, I. A., Heneghan, R. F., Bar-On, Y. M., & Galbraith, E. D. (2021). The global ocean size spectrum from bacteria to whales. *Science Advances*, 7(46), 3732. https://doi.org/10.1126/SCIAADV.ABH3732/SUPPL_FILE/SCIAADV.ABH3732_DATA_S1.ZIP

Heneghan, R. F., Galbraith, E., Blanchard, J. L., Harrison, C., Barrier, N., Bulman, C., et al. (2021). Disentangling diverse responses to climate change among global marine ecosystem models. *Progress in Oceanography*, 198, 102659. https://doi.org/10.1016/J.POCEAN.2021.102659
Hijmans, R. J., Etten, J., Sumner, M., Cheng, J., Baston, D., Bevan, A., et al. (2024). raster: Geographic data analysis and modeling (version 3.6-30). [Computer software]. https://doi.org/10.32614/CRAN/package.raster

Hipsey, M. R., Gal, G., Arhonditsis, G. B., Carey, C. C., Elliott, J. A., Frassl, M. A., et al. (2020). A system of metrics for the assessment and improvement of aquatic ecosystem models. *Environmental Modelling and Software*, **128**, 104697. https://doi.org/10.1016/J.ENVSOFT.2020.104697

Hollowed, A. B., Barange, M., Beamish, R. J., Brander, K., Cochrane, K., Drinkwater, K., et al. (2013). Projected impacts of climate change on marine fish and fisheries. *ICES Journal of Marine Science*, **70**(5), 1023–1037. https://doi.org/10.1093/ICESJMS/FST081

Ihaka, R., Murrell, P., Hornik, K., Fisher, J. C., Stauffer, R., Wilke, C. O., et al. (2024). Colorspace: A toolbox for manipulating and assessing colors and palettes (version 2.1-1). [Computer software] https://cran.r-project.org/web/packages colorspace/index.html

Izrailev, S. (2024). tictoc: Functions for timing R scripts, as well as implementations of “Stack” and “StackList” structures (version 1.2.1). [Computer software]. https://doi.org/10.32614/CRAN/package.tictoc

Jachner, S., Boogaart, G., & Petzoldt, T. (2007). Statistical methods for the qualitative assessment of dynamic models with time delay (R package qualV). *Journal of Statistical Software*, **22**(8), 1–30. https://doi.org/10.18637/jss.v022.i08

Jakeman, A. J., Letcher, R. A., & Norton, J. P. (2006). Ten iterative steps in development and evaluation of environmental models. *Environmental Modelling and Software*, **21**(5), 602–614. https://doi.org/10.1016/J.ENVSOFT.2006.01.004

Jennings, S., & Collingridge, K. (2015). Predicting consumer biomass, size-structure, production, catch potential, responses to fishing and associated uncertainties in the world's marine ecosystems. *PLoS One*, **10**(7), e0133794. https://doi.org/10.1371/journal.pone.0133794

Kubicek, A., Jopp, F., Breckling, B., Lange, C., & Reuter, H. (2015). Context-oriented model validation of individual-based models in ecology: A hierarchically structured approach to validate qualitative, compositional and quantitative characteristics. *Ecological Complexity*, **22**, 178–191. https://doi.org/10.1016/J.ECOCOM.2015.03.005

Kwiatkowski, L., Torres, O., Bopp, L., Aumont, O., Chamberlain, M., Christian, J. R., et al. (2020). Twenty-first century ocean warming, acidification, deoxygenation, and upper-ocean nutrient and primary production decline from CMIP6 model projections. *Biogeosciences*, **17**(13), 3439–3470. https://doi.org/10.5194/bg-17-3439-2020

Laufkötter, C., Vogt, M., Gruber, N., Aita-Noguchi, M., Aumont, O., Bopp, L., et al. (2015). Drivers and uncertainties of future global marine primary production in marine ecosystem models. *Biogeosciences*, **12**(23), 6955–6984. https://doi.org/10.5194/bg-12-6955-2015

Legates, D. R., & McCabe, G. J. (1999). Evaluating the use of “goodness-of-fit” Measures in hydrologic and hydroclimatic model validation. *Water Resources Research*, **35**(1), 233–241. https://doi.org/10.1029/1998WR900018

Lotze, H. K., Tittensor, D. P., Bryndum-Buchholz, A., Eddy, T. D., Cheung, W. W. L., Galbraith, E. D., et al. (2019). Global ensemble projections reveal trophic amplification of ocean biomass declines with climate change. *Proceedings of the National Academy of Sciences of the United States of America*, **116**(26), 12907–12912. https://doi.org/10.1073/PNAS.1900194116/SUPPL_FILE/PNAS.1900194116.SAPP.PDF

Massicotte, P., South, A., & Hufkens, K. (2023). rnaturalearth: World map data from natural Earth (version 1.0.1). [Computer software]. https://doi.org/10.32614/CRAN/package.rnaturalearth

Maureaud, A., Kitchel, Z., Fredston, A., Guralnick, R., Abrantes, J. P., Palomares, D., et al. (2023). *Fishglob: A collaborative infrastructure for marine science and management*. OSF. https://doi.org/10.31219/osf.io/mh46b

Maury, O. (2010). An overview of APECOSM, a spatialized mass balanced “Apex Predators ECOSystem Model” to study physiologically structured tuna population dynamics in their ecosystem. *Progress in Oceanography*, **84**(1–2), 113–117. https://doi.org/10.1016/J.POCEAN.2009.09.013

Maury, O., & Poggiale, J.-C. (2013). From individuals to populations to communities: A dynamic energy budget model of marine ecosystem size-spectrum including life history diversity. *Journal of Theoretical Biology*, **324**, 52–71. https://doi.org/10.1016/j.jtbi.2013.01.018

Maury, O., Shin, Y.-J., Faugeras, B., Ben Ari, T., & Marsac, F. (2007). Modeling environmental effects on the size-structured energy flow through marine ecosystems. Part 2: Simulations. *Progress in Oceanography*, **74**(4), 500–514. https://doi.org/10.1016/j-pocean.2007.05.001

Maury, O., Tittensor, D., Eddy, T., Allison, E., Bari, T., Barrier, N., et al. (2024). The Ocean system pathways (OSPs): A new scenario and simulation framework to investigate the future of the world fisheries. *ESS Open Archive*. https://doi.org/10.22541/essoar.171587166.60970779/v1

Mayer, D. G., & Butler, D. G. (1993). Statistical validation. *Ecological Modelling*, **68**(1–2), 21–32. https://doi.org/10.1016/0304-3800(93)90105-2

Müller, K. (2020). Here: A simpler way to find your files (version 1.0.1). [Computer software]. https://doi.org/10.32614/CRAN/package.here

Müller, K., Wickham, H., Francois, R., & Bryan, J., & RStudio. (2023). tibble: Simple data frames (version 3.2.1). [Computer software]. https://doi.org/10.32614/CRAN/package.tibble

Novaglio, C., Blanchard, J. L., Plank, M. J., van Putten, E. I., Audzijonyte, A., Porobic, J., & Fulton, E. A. (2022). Exploring trade-offs in mixed fisheries by integrating fleet dynamics into multispecies size-spectrum models. *Journal of Applied Ecology*, **59**(3), 715–728. https://doi.org/10.1111/1365-2664.14086

Novaglio, C., Bryndum-Buchholz, A., Tittensor, D. P., Eddy, T. D., Lotze, H. K., Harrison, C. S., et al. (2024). The past and future of the fisheries and marine ecosystem model intercomparison project. *Earth's Future*, **12**(9), e2023EF004398. https://doi.org/10.1029/2023EF004398

Olsen, E., Fay, G., Gaichas, S., Gamble, R., Lucey, S., & Link, J. S. (2016). Ecosystem model skill assessment. Yes we can. *PLoS One*, **11**(1), e0146467. https://doi.org/10.1371/JOURNAL.PONE.0146467

Ortega-Cisneros, K., Cochrane, K., & Fulton, E. (2017). An Atlantis model of the southern Benguela upwelling system: Validation, sensitivity analysis and insights into ecosystem functioning. *Ecological Modelling*, **355**, 49–63. https://doi.org/10.1016/j.ecolmodel.2017.04.009

Pauly, D., & Zeller, D. (2016). Catch reconstructions reveal that global marine fisheries catches are higher than reported and declining. *Nature Communications*, **7**(1), 10244. https://doi.org/10.1038/NCOMMS10244

Pebesma, E., Bivand, R., Racine, E., Sumner, M., Cook, I., Keitt, T., et al. (2024). sf: Simple features for R (version 1.0-17). [Computer software]. https://doi.org/10.32614/CRAN/package.sf

Pedersen, T. L. (2024). patchwork: The composer of plots (version 1.3.0). [Computer software]. https://doi.org/10.32614/CRAN/package.patchwork

Petrik, C. M., Luo, J. Y., Heneghan, R. F., Everett, J. D., Harrison, C. S., & Richardson, A. J. (2022). Assessment and constraint of mesozooplankton in CMIP6 Earth system models. *Global Biogeochemical Cycles*, **36**(11), e2022GB007367. https://doi.org/10.1029/2022GB007367

Petrik, C. M., Stock, C. A., Andersen, K. H., van Denderen, P. D., & Watson, J. R. (2019). Bottom-up drivers of global patterns of demersal, forage, and pelagic fishes. *Progress in Oceanography*, **176**, 102124. https://doi.org/10.1016/j.pocean.2019.102124

Planque, B., Aarflot, J. M., Buttay, L., Carroll, J., Fransner, F., Hansen, C., et al. (2022). A standard protocol for describing the evaluation of ecological models. *Ecological Modelling*, **471**, 110059. https://doi.org/10.1016/j.ecolmodel.2022.110059
Power, M. (1993). The predictive validation of ecological and environmental models. *Ecological Modelling*, **68**(1-2), 33-50. https://doi.org/10.1016/0304-3800(93)90106-3

R Core Team. (2023). R: A language and environment for statistical computing (version 4.2.3). *R Foundation for Statistical Computing*. [Computer software] Retrieved from https://www.R-project.org/

Refsgaard, J. C., Madsen, H., Andréassian, V., Arnbjerg-Nielsen, K., Davidson, T. A., Drews, M., et al. (2014). A framework for testing the ability of models to project climate change and its impacts. *Climatic Change*, **122**(1-2), 271-282. https://doi.org/10.1007/S10584-013-0990-2/FIGURES/3

Ricard, D., Minto, C., Jensen, O. P., & Baum, J. K. (2012). Examining the knowledge base and status of commercially exploited marine species with the RAM Legacy Stock Assessment Database. *Fish and Fisheries*, **13**(4), 380-398. https://doi.org/10.1111/j.1467-2979.2011.00435.x

Rykiel, E. J. (1996). Testing ecological models: The meaning of validation. *Ecological Modelling*, **90**(3), 229-244. https://doi.org/10.1016/0304-3800(95)00152-2

Rynne, N. (2024). A skill assessment framework for the fisheries and marine ecosystem model intercomparison project. *Zenodo*. [Code] [Dataset] [Collection]. https://doi.org/10.5281/zenodo.13922550

Scherrer, K., & Galbraith, E. (2020). Regulation strength and technology creep play key roles in global long-term projections of wild capture fisheries. *ICES Journal of Marine Science*, **77**(7-8), 2518-2528. https://doi.org/10.1093/icesjms/fsaa109

Séférian, R., Berthet, S., Yool, A., Palmiéri, J., Bopp, L., Tagliabue, A., et al. (2020). Tracking improvement in simulated marine biogeochemistry between CMIP5 and CMIP6. *Current Climate Change Reports*, **6**(3), 95-119. https://doi.org/10.1007/s40641-020-00160-0

Sepulchre, P., Caubel, A., Ladant, J.-B., Bopp, L., Boucher, O., Braconnot, P., et al. (2020). IPSL-CM5A2 - An Earth system model designed for multi-millennial climate simulations. *Geoscientific Model Development*, **13**(7), 3011-3053. https://doi.org/10.5194/gmd-13-3011-2020

Steenbeek, J., Buszowski, J., Chagaris, D., Christensen, V., Coll, M., Fulton, E. A., et al. (2021). Making spatial-temporal marine ecosystem modelling better - a perspective. *Environmental Modelling and Software*, **145**, 105209. https://doi.org/10.1016/j.envsoft.2021.105209

Steenbeek, J., Ortega, P., Bernardello, R., Christensen, V., Coll, M., Exarchou, E., et al. (2024). Making ecosystem modeling operational-A novel distributed execution framework to systematically explore ecological responses to divergent climate trajectories. *Earth's Future*, **12**(3), e2023EF004295. https://doi.org/10.1029/2023ef004295

Stock, C. A., John, J. G., Rykaczewski, R. R., Asch, R. G., Cheung, W. W. L., Dunne, J. P., et al. (2017). Reconciling fisheries catch and ocean productivity. *Proceedings of the National Academy of Sciences of the United States of America*, **114**(8), E1441-E1449. https://doi.org/10.1073/PNAS.1610238114

Stow, C. A., Jolliff, J., McGillicuddy, D. J., Doney, S. C., Allen, J. I., Friedrichs, M. A. M., et al. (2009). Skill assessment for coupled biological/physical models of marine systems. *Journal of Marine Systems*, **76**(1-2), 4-15. https://doi.org/10.1016/jJMARSYS.2008.03.011

Sturloudottir, E., Desjardins, C., Elvarsson, B., Fulton, E. A., Gorton, R., Logemann, K., & Stefansson, G. (2018). End-to-end model of Icelandic waters using the Atlantis framework: Exploring system dynamics and model reliability. *Fisheries Research*, **207**, 9-24. https://doi.org/10.1016/J.FISHRES.2018.05.026

Tagliabue, A., Kwiatkowski, L., Bopp, L., Butenschön, M., Cheung, W., Lengaigne, M., & Vialard, J. (2021). Persistent uncertainties in ocean net primary production climate change projections at regional scales raise challenges for assessing impacts on ecosystem services. *Frontiers in Climate*, **3**. https://doi.org/10.3389/fclim.2021.738224

Taylor, K. E. (2001). Summarizing multiple aspects of model performance in a single diagram. *Journal of Geophysical Research*, **106**(D7), 7183-7192. https://doi.org/10.1029/2000JD900719

Tittensor, D. P., Eddy, T. D., Lotze, H. K., Galbraith, E. D., Cheung, W., Barange, M., et al. (2018). A protocol for the intercomparison of marine fishery and ecosystem models: Fish-MIP v1.0. *Geoscientific Model Development*, **11**(4), 1421-1442. https://doi.org/10.5194/GMD-11-1421-2018

Tittensor, D. P., Novaglio, C., Harrison, C. S., Heneghan, R. F., Barrier, N., Bianchi, D., et al. (2021). Next-generation ensemble projections reveal higher climate risks for marine ecosystems. *Nature Climate Change*, **11**(11), 973-981. https://doi.org/10.1038/s41558-021-01173-9

van Denderen, P. D., Lindegren, M., MacKenzie, B. R., Watson, R. A., & Andersen, K. H. (2018). Global patterns in marine predatory fish. *Nature Ecology & Evolution*, **2**(1), 1-70. https://doi.org/10.1038/s41559-017-0388-z

Wagener, T., Reinecke, R., & Pianosi, F. (2022). On the evaluation of climate change impact models. *WIREs Climate Change*, **13**(3), e772. https://doi.org/10.1002/wcc.772

Watson, R. (2019). *Global fisheries landings V4.0* (version 4.0). *Institute for Marine and Antarctic Studies (IMAS), University of Tasmania (UTAS)*. [Dataset]. https://doi.org/10.25959/5C522CADBEA37

Watson, R., & Tidd, A. (2018). Mapping nearly a century and a half of global marine fishing: 1869-2015. *Marine Policy*, **93**, 171-177. https://doi.org/10.1016/j.marpol.2018.04.023

Wickham, H. (2016). *ggplot2: Elegant graphics for data analysis*. Springer-Verlag. Retrieved from https://ggplot2.tidyverse.org

Wickham, H., & RStudio. (2023). forcats: Tools for working with categorical variables (factors) (version 1.0.0). [Computer software]. https://doi.org/10.32614/CRAN/package.fcatcs

Wickham, H., Chang, W., Henry, L., Pedersen, T. L., Takahashi, K., & Posit and PBC (2024). ggplot2: Create elegant data visualisations using the grammar of graphics (version 3.5.1). [Computer software]. https://doi.org/10.32614/CRAN/package.ggplot2

Wickham, H., François, R., Henry, L., Müller, K., & Vaughan, D. (2023). Dplyr: A grammar of data manipulation (version 1.1.4). [Computer software]. https://doi.org/10.32614/CRAN/package.dplyr

Wickham, H., & Henry, L., & RStudio. (2023). purrr: Functional programming tools (version 1.0.2). [Computer software]. https://doi.org/10.32614/CRAN/package.purrr

Wickham, H., Hester, J., & Bryan, J. (2015). readr: Read Rectangular Text Data. In CRAN: Contributed Packages [Dataset]. *The R Foundation*. https://doi.org/10.32614/cran/package.readr

Wickham, H., Pedersen, T. L., & Seidel, D., & Posit, and PBC. (2023). scales: Scale functions for visualization (version 1.3.0). [Computer software]. https://doi.org/10.32614/CRAN/package.scales

Wickham, H., & Software, P. (2023). stringr: Simple, consistent wrappers for common string operations (version 1.5.1). [Computer software]. https://doi.org/10.32614/CRAN/package stringr

Wickham, H., Vaughan, D., & Girlich, M. (2024). tidyr: Tidy messy data (version 1.3.1). [Computer software]. https://doi.org/10.32614/CRAN/package.tidyr

Wickham, H., Vaughan, D., Girlich, M., & Ushey, K. (2023). tidyverse: Easily install and load the "tidyverse" (version 2.0.0). [Computer software]. https://doi.org/10.32614/CRAN/package.tidyverse

Zeileis, A., Fisher, J. C., Hornik, K., Ihaka, R., McWhite, C. D., Murrell, P., et al. (2020). Colorspace: A toolbox for manipulating and assessing colors and palettes. *Journal of Statistical Software*, **96**, 1-49. https://doi.org/10.18637/jss.v096.i01