# Lessons Learned from the LBA MIP

#### The Large-Scale Biosphere - Atmosphere Experiment in Amazonia (LBA) Model Inter-comparison Project (MIP) held a workshop on May 2-3 in Washington DC to review preliminary results. So far, the LBA MIP has 30 simulations from 21 models at eight towers in the Amazon basin (some modelers submitted multiple simulations representing different model configurations). Below I list issues raised at the LBA MIP workshop, all of which also apply to the Site MDC.

# Protocol

#### **Model Submissions Tool**: The LBA MIP will supply all model teams with a tool to check submitted model output for proper variable names, units, energy balance, water balance, sign convention, and ballpark values. The modelers would then resolve issues with units and variable names, radically reducing preparation time for the LBA MIP analysis team. They found a number of problems in both the protocol and the submissions associated with file format, variable names, units, sign convention, etc. The LBA MIP team could not resolve all such issues and had to eliminate a third of the models from the workshop analysis.

#### **Leap Year**: The protocol now addresses leap year because half of the models include leap year and half do not. The gap-filled weather data will include leap year, so the modeler must remove February 29 if the model does not include leap year. Model output should include leap year, so the modeler must add February 29 by duplicating the results from February 28.

#### **Carbon Units**: The protocol should specify whether the models should submit mass units of carbon or CO~2~.

#### **Sign Convention**: The protocol should explicitly define the sign convention for each variable.

#### **NETCDF Tools**: The LBA MIP team will provide some tools and generic subroutines to help create netcdf format submission files. They found a quarter of the modelers submitted various non-standard formats (including an excel spreadsheet). With so many participating models, this created a huge burden on the LBA MIP team. There was little sympathy for those modelers unfamiliar with netcdf.

#### **GMT vs. Local Time**: Some models submitted results in GMT rather than local time, so they added the time zone conversions to the Protocol. This is easy for the LBA MIP because all the towers are in the same time zone. All models use GMT and the Site MDC covers 9 time zones, so I think we should convert all the fluxes and driver data to GMT rather than require output in local time.

#### **Match Model Output to observations**: The protocol should have included model outputs that could be compared directly to ancillary observations made at the flux tower sites, such as biometric wood biomass, soil temperature, soil moisture, and ground respiration. Considering the late stage of the LBA MIP, they decided to add new variables to the protocol only if they really needed them for the analysis. 

#### **LAI**: Model output should include Leaf Area Index, particularly for the dynamic vegetation models.

#### **Driver Data**: Model output should include all input weather driver data, to confirm proper units.

#### **Model Characteristics Questionnaire**: The LBA MIP analysis team created a questionnaire to send to each modeling team to describe the basic model characteristics in standard terms. They found it impractical to glean this information by reading the reference papers for all 21 models. They considered this information essential in interpreting the model results. Model structural information would include the type of radiative transfer model, the biogeochemistry, the photosynthesis model, etc.

#### **Static Model Information**: Each model team should submit static (does not vary with time) information on model configuration, such as the geometry of the soil model, pool configuration and turnover time. Submitting full parameter tables was rejected because of the time involved and questionable usefulness. The exact nature of this type of information was not fully resolved at the workshop.

#### **Fair Use Policy**: At my urging, the LBA MIP team expanded their fair-use policy to include model output (the current policy only covers tower observations). 

#### **Mass and Energy Balance**: The LBA MIP is checking water and energy balance for each model as a fundamental objective. However, the requested model output did not include all terms to actually confirm balance. The problem was confounded by the fact that model structure varied so much that including all possible terms was impractical. Modelers have to ensure balance anyway, so the automated checking tool will check long-term balance of incoming and outgoing radiation or water as a "sanity check" during the units and variable name validation. 

# Preparing the Observed Fluxes

#### **Treat all Towers the Same**: The LBA MIP team found that the preparation of observed fluxes is as important as filling the driver data. Everyone agreed fluxes from all towers must be treated in the same manner.

#### **U\* Filter**: They applied a U\* filter to the fluxes, but there were concerns about the bias introduced into the observed fluxes vs. the need for even doing the U\* filter. 

#### **Energy Balance Filter**: They applied a daily average energy balance filter to remove those days where the energy closure problem was particularly bad. However, nobody knew how to actually fix the energy closure issue. 

#### **Storage Flux Correction**: The observed NEE was missing the storage flux at some towers, introducing uncertainty in the comparison with modeled fluxes. However, excluding the NEE without the storage flux correction eliminated 95% of the observations at those sites, so the data was left in.

#### **No Filled Data**: Any filled flux data was removed. Everyone agreed not to compare model output with filled fluxes, which is considered a "model-to-model" comparison.

#### **GPP and Respiration**: There was general concern about uncertainty in GPP and respiration estimated from the observed NEE. These were definitely not viewed as "observed" values since a model is required to split the NEE into GPP and respiration.

# Analysis

#### **Dedicated Manpower**: You need dedicated resources and manpower devoted to analysis. The volunteer LBA MIP team all lamented the lack of time to properly analyze the data and model output.

#### **Visualization**: Visualizing comparisons with so many models is a problem. They settled on a basic strategy to calculate simple numbers like root mean square error, correlation, and bias. Then they would move up to more advanced analysis, such as Taylor plots or EOF analysis.

#### **Model Summary**: A summary of model behavior with mean annual cycle, without direct comparison with data, is useful for identifying basic patterns and regimes of model behavior. At the eight Amazonian towers, we saw two very distinct (and unexplained) regimes of model behavior.

#### **No Long-term Sinks**: An analysis of long-term sources or sinks is not possible because of missing flux data, the U\* filter bias, and the energy closure issue. 

#### **Match Model to Observations**: Model output will be matched exactly with valid observed data. Model output without a corresponding observation will be ignored. They would apply a maximum missing data requirement when aggregating to longer time scales, but there was no solid resolution of exactly how to define such criteria.

#### **Process**: For future analysis, someone suggested a process where researchers submit their ideas to a central board, similar to what is currently done in the La Thuile synthesis. However, there was no firm resolution on this.

# Workshop

#### **More Time after Submission**: The LBA MIP team needed more time for analysis between model submissions and workshop (they had about a week).

#### **No Model Names**: Don't show model name when displaying results at the workshop; just show a number. Showing model names tends to induce arguments between modeling teams (a lesson, evidently, from the C4 MIP).

# Papers

####  They are planning for a special issue in an as-yet-determined journal with the following focused articles:

#### "big picture" paper with overview and summary of results

#### diurnal time scale: sensible, latent heat, and carbon flux

#### mean annual cycle time scale: sensible, latent heat, and carbon flux

#### interannual time scale: sensible, latent heat, and carbon flux

#### multiple papers focusing on specific issues, such as light use efficiency