dataesr/scientific-paragraphs-categorization huggingface dataset:
2010). These successes and the classification by the Group on Earth Observations (GEO) of a number of parameters discernable from space as essential climate variables prompted ESA to start their Climate Change Initiative (CCI) projects. There are currently 14 different CCI projects. For those interested in air-sea gas fluxes, arguably the most interesting of these projects is the Sea Surface Temperature CCI project, which has provided a 20-plus-year time series of global coverage multisensor sea surface skin and subskin temperature data  (Merchant et al. 2012). Unfortunately, the resources required to download and exploit these large global and decadal datasets can limit their exploitation by the scientific community. The development of cloud technologies and storage provides a solution. Cloud computing can be defined as interconnected computing resources that can be easily scaled up (grown) or down (shrunk) while maintaining its capability or function, rather like a ''cloud'' in the atmosphere.
a. The OceanFlux Greenhouse Gases project The OceanFlux Greenhouse Gases project was funded by the European Space Agency in 2011 to encourage the use of satellite Earth observation data for studying air-sea gas fluxes. To achieve this, the objectives of the project included the development and validation of novel gas flux Earth observation algorithms  (Goddijn-Murphy et al. 2013 , 2012)  and scientific analyses  (Land et al. 2013). Another objective was to provide datasets and processing tools that can be used by the scientific community. Accordingly, the gas flux data processing tools, collectively named the ''FluxEngine,'' are described in this paper. The FluxEngine allows users to configure the flux parameterization and to select their chosen input data, and then it generates the resulting monthly global flux datasets.
A plethora of climate study quality EO, in situ, and model data are available as input to the toolbox and to aid the interpretation of the resultant flux data (see Table  1 ). The outputs from the system are standard netCDF datasets that can be easily read into a number of third-party scientific software packages. The primary gas of interest for the OceanFlux Greenhouse Gases project was CO 2. Therefore, the FluxEngine has been developed to aid the study of the air-sea flux of CO 2 , although as described later in the paper, the toolbox can also be used to support the study of other gases, such as N 2 O and dimethyl sulfide (DMS).
The airside partial pressure of CO 2 , pCO 2A (matm) can be calculated using the concentration of CO 2 in dry air and air pressure using pCO 2 5 0. 001X [CO 2 ] (P 2 pH 2 O). ( 6 ) Term X [CO2] is the molar fraction of CO 2 in the dry atmosphere (expressed as the zonal mean in T09 and the FluxEngine), P is the air pressure (mb, expressed as a daily mean in the FluxEngine), and pH 2 O is the saturation vapor pressure (mb), which is defined in terms of sea surface temperature and salinity  (Weiss and Price 1980) where SST k is the sea surface temperature dataset of interest (K) and S is the salinity. For pCO 2W the Flux-Engine relies upon in situ pCO 2W measurements (e. g. , data from a buoy or ship) or an in situ-derived climatology of pCO 2W data (e. g. , T09).
The development of the FluxEngine The following sections describe the design of the FluxEngine, the methods used for calculating the air-sea fluxes, the input datasets that are available, and the methods used to verify the implementation. a. The FluxEngine design, input data, and implementation Figure  1  shows a diagram of the main parts of the FluxEngine, conceptually showing the input data and the contents of the output files. The input data are on the left of the diagram, with the calculation in the middle and an overview of the contents of each output file shown on the right. The air-sea fluxes are calculated using monthly composite data and the generation of these input data is described below. The output file is a netCDF4 Climate Format (CF) 1. 6 compliant file that contains. 20 data layers.
The data layers within each output file include the different components of the gas flux calculation, statistics of the input datasets (e. g. , variance of the wind speed), and process indicator layers to aid interpretation of the fluxes. The process indicator layers include fixed masks (e. g. , land, ocean basins, open ocean, and coastal classification), climatological data (e. g. , persistent SST fronts), and other modeling or Earth observation datasets useful for interpreting the fluxes (e. g. , chlorophyll-a concentrations and modelgenerated estimates of wave whitecapping). All output from the toolbox consists of monthly global coverage 18 3 18 spatial resolution data (360 3 180 arrays). Therefore, generation of gas fluxes for a complete year (12 months) requires 12 sets of data inputs (one set per month) and the FluxEngine then produces 12 output files.
The air-sea flux calculation utility contains internal integrity checks for all of the output data layers. These integrity checks highlight if any data layers contain data outside of a predefined expected range. If a data value for a data layer falls outside its specified valid data range, then a count is added to the corresponding data element position in one of the process indicator data layers. The inputs to the flux calculation (box 2 in Fig. 1 ) are monthly composite data. Table  1  shows all of the monthly datasets that are currently available for the FluxEngine to use as input. Where required, these monthly netCDF composite data were generated using the original daily data. For those datasets that were generated, each monthly composite file contains the mean (first-order moment), median, standard deviation, and the second-, third-, and fourth-order moments as calculated using one calendar month of data.
b. The computing platform The FluxEngine is currently installed on the Centre ERS d'Archivage et de Traitement (CERSAT) Nephelae Cloud. This Linux-based cloud-computing platform provides a petascale storage capacity and distributed processing over more than 600 computing nodes. It was specifically designed by CERSAT for massive archive processing, for applications including data mining and multidecadal Earth observation data reprocessing. As with all cloud computing, it offers facilities for simple backup and restoration, high-speed data processing (i. e. , the processing nodes are close to data, so there is no potential for input/output bottlenecks), the system can be tailored to a specific job (i. e. , there is no reliance on physical hardware as it uses virtual servers), no specific skills are required to use it, and it maximizes the use of resources through dynamic reallocation.
Data processing runs submitted by the user are scheduled and run on the cloud-computing nodes using a system developed by CERSAT called ''GoGo list. '' This is simply a wrapper enabling processing jobs to be executed and monitored on the cloud, and the use of GoGo list is completely invisible to the user. It must be noted though that the main reason for using the Nephelae Cloud here is to provide potential users with access to a large and continually growing satellite Earth observation dataset. The FluxEngine can also be installed and used on a desktop or laptop computer with no loss of performance.
c. Configurable options The main flux calculation within the FluxEngine is user configurable through the use of a plain text ASCII configuration file. Within the configuration a user can choose the input datasets, the flux calculation model [choose between Eqs. (4) and (  9 )], and the gas transfer velocity parameterization. A range of different gas transfer parameterizations are available (e. g. ,  McGillis et al. 2001; Nightingale et al. 2000; Wanninkhof 1992 ), including those based on sea state and surface roughness  (Fangohr and Woolf 2007; Goddijn-Murphy et al. 2012). Through a generic formulation (see the appendix), it is also possible for users to use their own wind speedbased parameterization. Through the configuration file, the user can choose to inject random noise (normally distributed with specified mean and standard deviation) to any of the main input datasets (SST, U 10 , Hs, pCO 2 , or fCO 2 ).
d. Additional software tools Three additional tools exist within the FluxEngine toolbox. These enable the calculation of integrated net fluxes, regridding of the output data, and cruise or buoy in situ data to be used as inputs. The integrated net flux tool can provide gross and net fluxes, mean values for each dataset within the output data, and estimates of the open ocean fluxes and the flux contribution from any missing data (such as that from coastal, shelf, and enclosed seas). The regridding tool converts the 18 3 18 data to the 58 3 48 grid used by the T09 climatological dataset. This allows users to easily compare the FluxEngine output with that of the T09 climatology. The in situ to netCDF conversion tool converts sparse in situ data into a spatially and temporally binned or gridded (18 3 18 grid) format.
Data quality and verification of the calculations The widely used T09 climatology dataset provides an ideal benchmark for verifying the operation and output of the FluxEngine, as it contains both the air-sea flux estimates and the input values used to calculate these fluxes. Therefore, the T09 air-sea flux data were first used to verify the FluxEngine integrated net flux tool, and then the main T09 input fields were used to verify the combination of the FluxEngine flux calculation and the integrated net flux tool. To verify the integrated net flux tool, the year 2000 T09 climatology data were linearly interpolated to a 18 3 18 grid and then provided as the input to the tool. The integrated net flux tool used the T09 ice normalization (see the appendix). The resultant net air-sea CO 2 flux for the open ocean region was 21.
The following is true for all monthly outputs: the pCO 2W data were identical to six decimal places; 2% of the pCO 2A data elements were. 6 0. 01% different; 2% of the a data elements were. 6 3% different; and 2% of the k data elements were. 6 2% different. These small differences within the calculations collectively results in 5%-12% (dependent upon the month; see Fig. 3 ) of the air-sea flux F data elements being. 6 5% different. The differences in these data fields between the output and that of the original linearly interpolated T09 dataset are likely to be a combination of (i) minor rounding differences in the flux calculations (e. g. , due to differences in precision between the FluxEngine calculations and those of the original T09 publication) and (ii) interpolation issues at boundaries.
a. Global analyses and the Southern Ocean A time series global analysis was performed to demonstrate the use of the FluxEngine. The linearly interpolated (to 18 3 18) T09 climatology data of X [CO2] , pCO 2w (and associated SST), and Earth observation SST (SST skin) and U 10 data, NCEP Climate Forecast System Reanalysis (CFSR) air pressure, and SSM/I percentage ice cover were used as the input to the FluxEngine (please see Table  1  for dataset specifics). When the FluxEngine is configured to use the T09 pCO 2w data and the chosen SST dataset is not from T09 (either due to studying a different year than the original T09 year 2000, or due to the selection of a different SST dataset), the pCO 2w data need to be reanalyzed to be consistent with this new SST dataset. Therefore, following previous studies  (Fangohr et al.
2008; Kettle and Merchant 2005; Land et al. 2013) , the FluxEngine reanalyses the T09 pCO 2W data to correct them to the chosen SST datasets using the relationship provided by T09: pCO 2W 5 pCO 2W expf[0:0433(SST C 2 T C )] 2 4:35 3 10 25 (SST 2 C 2 T 2 C )g, ( 8 ) where T C is the original temperature dataset (8C) and the subscript SST C denotes the sea surface temperature (8C). The FluxEngine was run for years 1995-2009 using Eq. (  5 ) for the flux calculations and the gas transfer velocity of  Nightingale et al. (2000). To generate an estimate of the uncertainties in the flux estimate due to the known uncertainties in the U 10 , pCO 2W , and SST input data, the FluxEngine time series processing was repeated with injected random noise enabled. The system assumes a 1. 5 matm yr 21 change in the atmospheric and oceanic partial pressures of CO 2 relative to the year 2000 of the original T09 data.
c. Underway in situ data To demonstrate the flexibility of the system, the airsea fluxes for a research cruise in the central equatorial Atlantic were calculated using the FluxEngine. In situ fCO 2 and associated SST data were downloaded from the community Surface Ocean CO 2 Atlas (SOCAT) website (v1. 5, tropical Atlantic group) and then preprocessed into the format required by the FluxEngine (using the in situ tool; see the appendix). The Flux-Engine was then used to calculate the air-sea CO 2 fluxes using the in situ fCO 2 data and the temporally and spatially corresponding EO data (the EO data sources are the same as used in sections 4a and 4b). Figure  8a  shows the resultant gridded fCO 2 data for all SOCAT, v1. 5, tropical Atlantic in situ fCO 2 data, and Fig. 8b  shows the gridded in situ fCO 2 data from a single cruise  (Bakker 2014)  in the tropical Atlantic.
d. Benchmarking The FluxEngine was run on a single cloud node (2. 4-GHz Intel Xeon, 3 GB of memory), and the time taken to process a single-year dataset was determined. All input datasets were quality filtered and converted to a suitable netCDF format in advance of this analysis. A 1-yr climatology of fluxes at 18 3 18 spatial resolution took 40 min (total process time) to complete. Disabling the process indicator layer output reduced this time to 25 min. Running a 5-yr time series (with the indicator layers off) on the single cloud node took 2 h [i. e. , ;25 3 5 yr (60 min) 21 total process time], whereas repeating the same 5-yr time series processing across five nodes using GoGo list took 25 min (as each year was executed on an independent node and so executed simultaneously). Calculating the integrated net fluxes from the 1-yr climatology netCDF files took 10 min (total process time).
Future developments The FluxEngine has been developed to allow the study of CO 2 and the current version uses the T09 climatology pCO 2W data for the waterside component of the calculation. To increase the versatility of the system, work is ongoing to extend the toolbox to use the community SOCAT datasets (e. g. ,  Pfeil et al. 2013; Bakker et al. 2014 ). There are many other climatically important gases, including nitrous oxide (N 2 O) and methane (CH 4 ). Partial support for these gases exists within the toolbox, as N 2 O, CH 4 , and CO 2 are all poorly soluble gases, so their gas transfer velocity parameterizations can be considered as interchangeable with that for CO 2. Therefore, the toolbox can be used to generate maps of gas transfer velocity to enable air-sea fluxes of N 2 O and CH 4 to be studied.
Similarly, k for the gas DMS can be considered to be the direct component (i. e. , nonbubble component) of a k CO 2 parameterization, and the toolbox already includes two methods for deriving this direct component. The generic nature of the gas calculation and parameterization lends itself to being extended to determine air-sea fluxes for other gases. For each additional gas, a gas transfer velocity parameterization and an in situ dataset or climatology of in-water concentrations, partial pressures or fugacity, and suitable solubility equations are required. After CO 2 the next largest gas climatology (in terms of the number of in situ data points) is the  Lana et al. (2010)  DMS climatology, and so future extensions of the FluxEngine are likely to include the addition of a DMS capability through exploiting published work (e.
g. ,  Goddijn-Murphy et al. 2012; Johnson 2010; Lana et al. 2011 ). This will involve exploiting (or linking in) the open source code of  Johnson (2010) , which can be used to calculate gas transfer velocities. The full capability to calculate air-sea fluxes for N 2 O, CH 4 , and other gases is not currently possible as the in-water data collections for these gases are still in their infancy, but efforts have begun to collate such datasets, for example, the Marine Methane and Nitrous Oxide (MEMENTO) database  (Bange 2006). The toolbox currently uses climatological salinity data from the T09 climatology. Recent advancements in satellite Earth observation have seen the launch of two sensors that can measure surface salinity from space. These are NASA's Aquarius and ESA's Soil Moisture Ocean Salinity (SMOS) missions, and future work will enable SMOS salinity data to be used within the toolbox.
Conclusions A flexible air-sea CO 2 data processing toolbox called the FluxEngine has been developed and presented. The flux calculation itself is user configurable, and the outputs have been extensively evaluated and compared with reference datasets. No specialist knowledge is required to use the toolbox, and it is based on standard software tools and packages that require no licenses. It is currently installed and running on the Nephelae Cloud at the Insitut Francais de Recherche pour l Exploitation de la Mer (Ifremer), where. 8 TB of climate-quality data can be used as input to the flux calculations. The use of cloud-computing approaches means that the data processing is scalable, and this feature is completely transparent to the user. Here we have used the toolbox to estimate the 15-yr-average net air-sea flux of CO 2 for the global oceans (including shelf seas and coastal zones), the four main oceanic basins, and the European shelf seas.
We have shown how subtle differences in the definitions of the European shelf seas can cause differences of. 10% in the calculated annual net fluxes. Similarly differences in the Southern Ocean definition can have an impact on the calculated air-sea flux. We therefore urge the scientific community to use a common set of oceanic region definitions, to allow the outputs from differing studies to be easily compared and contrasted. The FluxEngine provides a mechanism for this and its open source nature allows the scientific community to freely exploit the toolbox. It is hoped that the FluxEngine will help to improve the transparency and traceability of results from air-sea gas flux studies. The FluxEngine was originally developed for the ESA OceanFlux Greenhouse Gases project, and it is currently being used to produce air-sea gas flux climatologies and to study method and data uncertainties.
Users can access the version of the toolbox installed on the Nephalae Cloud on the OceanFlux Greenhouse Gases project website (http://www. oceanflux-ghg. org); alternatively, the open-source software is available in GitHub (https://github. com/oceanflux-ghg/FluxEngine). Table  A1  lists the software utilities that are available in the FluxEngine toolbox. Specific details on some of the tools are given in the following sections.
1) OFLUXGHG-FLUX-BUDGETS. PY The utility calculates integrated net fluxes (F IN ) over a given region from the monthly mean flux and ice cover data as follows. The input netCDF files are assumed to contain monthly mean net flux F, ice cover, gas transfer velocity, in-water CO 2 concentration, and interfacial CO 2 concentration. A high-spatial-resolution land mask is required and a region definition file if nonglobal regions are to be analyzed. The four main oceanic regions (Atlantic, Pacific, Indian, and Southern Oceans), as defined by the IHO, are provided within the monthly netCDF output (within the process indicator data layers), so these can be used as the region definitions for this utility if desired. A high-spatial-resolution land mask is also provided within the toolbox and its use is described below.
The method for the integrated net flux tool is now described. At each pixel or data element, the integrated net flux is initially calculated from F and the pixel's total area, which is calculated assuming the earth to be an oblate spheroid. Next, we need to account for the effect of ice, and two methods of ice normalization are available. The first is from T09, which specifies that if ice cover within a data element is ,10%, then it has a negligible effect on the integrated net flux and so the ice cover value is assumed to be 0%. Data elements with ice cover. 90% are set to 90% to account for leads, polynyas, etc. The net flux for each data element where the ice cover is. 0% is then reduced linearly by the percentage of ice cover value. The second ice normalization method  (Loose et al. 2009 ) posits that the integrated net flux from partially ice-covered regions is greater than would be expected from proportionality with ice-free fraction.
Here, the integrated net flux is proportional to (ice-free fraction) 0. 4. This implies that the integrated net flux from a pixel with 90% ice cover (the maximum assumed by T09) is actually responsible for 40% of the icefree integrated net flux. This results in a quadrupling of the estimated net flux from regions of fast ice over that assumed by the T09 method. Users are able to choose which method they prefer to use. The data are at a relatively coarse spatial resolution of 18 3 18 (which at the equator is ;111 km 3 ;111 km). Therefore, to calculate the contribution of each data element to the regional or global integrated net flux, we need to know the proportion of ocean (whether ice covered or not) actually contained within each data element. For most elements this will be 1 (open ocean or sea ice) or 0 (land), but it may be intermediate in data elements that cover region boundaries (e.
This is added to the integrated net flux to give an estimate of the total regional integrated net flux. Note that the flux from individual regions may not exactly sum to the global region flux using this method, since in one case the regional mean flux is used to estimate the missing flux, while in the other case the global mean flux is used. The net flux tool also provides gross fluxes and the average values for all spatially varying variables within the netCDF files. The gas transfer velocity k, in-water CO 2 concentration ([CO 2AQW ]), and interfacial CO 2 concentration ([CO 2AQ0 ]) are used to calculate the upward and downward integrated gross fluxes. The upward gross flux, F UG , is defined as k [CO 2AQW ] and the downward gross flux, F DG , is defined as k[CO 2AQ0 ]. Missing data are treated in the same way for these calculations as for integrated net flux.
2) RESAMPLE_NETCDF. PY This tool calculates the mean for each 58 3 48 grid cell using the corresponding 18 3 18 grid cells in the input data. No correction due to variations in area (between the 58 3 48 and the 18 3 18 grids) due to interactions with land is considered and any missing (masked) 18 3 18 data are not used in the calculation. If all data in the 18 3 18 grid cells are masked (missing values), then the output 58 3 48 grid cell is also masked. The tool can output the regridded fields as netCDF or as a single ASCII commaseparated variable (CSV) file. The output file will simply reflect the contents of the input netCDF with each data array (2D dataset) replaced by its 58 3 48 equivalent, while the optional CSV file contains columns corresponding to latitude, longitude, and all 2D datasets found in the input file.
3) TXT2NCDF. PY This utility assumes that the input data are in CSV format, with headings that include ''latitude,'' ''longitude,'' and optionally ''date'' or ''date/time'' on the first line, and corresponding values on the subsequent lines. Latitude and longitude must be in decimal degrees, while date or date/time must follow the format DD/MM/ (YY)YY [hh:mm(:ss)], where parentheses indicate options and DD is the two-digit day number within the month, MM is the two-digit month number (starting at 01) and (YY)YY is the two-or four-digit year. The resulting output data are in netCDF format, with latitude and longitude limits copied from a reference global netCDF file that is passed as one of the inputs. The output netCDF includes a dataset ''count'' containing the number of observations found in each grid cell.