dataesr/scientific-paragraphs-categorization huggingface dataset:
Second, during the merging, the du-plicates between datasets have to be identified and removed. Third, the metadata should be propagated throughout the process and made available in the final merged product. Ideally, the compiled dataset would be made available as a simple text table, to facilitate ease of access and manipulation. In this work such unification of multiple datasets is presented. This was done for the validation of the ocean-colour products from the ESA Ocean Colour Climate Change Initiative (OC-CCI), but with the intent to serve the broad user community as well. A merged dataset is not without drawbacks: it is likely to be large and so not always easy to manipulate; because the merging is done on pre-existing, processed databases, one does not have full command of the whole processing chain; and the dataset would be a compilation of observations collected by several investigators using different instruments, sampling methods and protocols, which might even-tually have been modified by the processing routines used by the repositories or archives.
Nevertheless, to minimise these potential drawbacks, we have, for the most part, incorporated only datasets that have emerged from the long-term efforts of the ocean-colour and biological oceanographic communities to provide scientists with high-quality in situ data and implemented additional quality checks on the data to enhance confidence in the quality of the merged product. In Sect. 2 the methodologies used to harmonise and integrate all data, as well as a description of individual datasets acquired, are provided. In Sect. 3 the geographic distribution and other characteristics of the final merged dataset are shown. Section 4 provides an overview of the data.
The chlorophyll data were compiled from eight data sources: BOUSSOLE, SeaBASS, NOMAD, MERMAID, AMT, ICES, HOT and GeP&CO. One requirement for chla_fluor measurements was that they were made using in vitro methods (i. e. based on extractions of chlorophyll a). Although this severely decreased the number of observations, since in situ fluorometry (e. g. fluorometers mounted on CTDs) is widely available in oceanographic databases, it was decided to exclude such data because of potential problems with the calibration of in situ fluorometers. The variable chla_hplc was calculated by summing all reported chlorophyll a derivatives, including divinyl chlorophyll a, epimers, allomers and chlorophyllide a. The two chlorophyll variables are retained separately in the database to facilitate their use.
With regard to the inherent optical properties (aph, adg, bbp), if not already calculated and provided in the contributed datasets, they were computed from related variables that were available: particle absorption (ap), detrital absorption (ad), coloured dissolved organic matter (CDOM) absorption (ag) and total backscattering (bb). The following equations were used adg = ad + ag, ap = aph + ad and bb = bbp + bbw. For the latter equation, the variable bbw was computed using bbw = bw / 2, where bw is the scattering coefficient of seawater derived from  Zhang et al. (2009). The diffuse attenuation coefficient for downward irradiance (kd) did not require any conversion and was compiled as originally acquired. Observations of inherent optical properties (surface values) and diffuse attenuation coefficient for downward irradiance, were acquired from three data sources particularly designed for ocean-colour validation (SeaBASS, NOMAD, MERMAID) and were thus already subject to the processing routines of these datasets.
The merged dataset was compiled from 10 sets of in situ data, which were obtained individually either from archives that incorporate data from multiple contributors (SeaBASS, NOMAD, MERMAID and ICES) or from particular measurement programs or projects (MOBY, BOUS-SOLE, AERONET-OC, HOT, GeP&CO, AMT) and were subsequently homogenised and merged. Data contributors are listed in Table  2. There were methodological differences between datasets. Therefore, after acquisition, and prior to any merging, each set of data was preprocessed for quality control and conversion to a common format. During this process, data were discarded if they had (1) unrealistic or missing date, time and geographic coordinate fields; (2) poor quality (e. g. original flags) or a method of observation that did not meet the criteria for the dataset (e.
g. in situ fluorescence for chlorophyll concentration); and (3) spuriously high or low data. For the latter, the following limits were imposed: for chla_fluor and chla_hplc [0. 001-100] mg m -3 ; for rrs [0-0. 15] sr -1 ; for aph, adg and bbp [0. 0001-10] m -1 ; for kd [aw(λ)-10] m -1 , where aw is the pure water absorption coefficients derived from  Pope and Fry (1997). Also during this stage, three metadata strings were attributed to each observation: dataset, subdataset and pi. The dataset contains the name of the original set of data, and can only be one of the following: "aoc", "boussole", "mermaid", "moby", "nomad", "seabass", "hot", "ices", "amt" or "gepco". The subdataset starts with the dataset identifier and is followed by additional information about the data, in the format <dataset>_<cruise/station/site>) (e.
For the latter case, a search window of 5 min in time and 200 m in distance was given, to account for station drift. A small number of observations that were identified as replicates had different subdataset identifiers (i. e. a different cruise name). These observations were considered suspicious if the values were different and were discarded. If the values were the same, one of the observations was retained. This possibly originated from the same group of data being contributed to an archive by two different principal investigators. Once each set of data was homogenised, all data were integrated into a unique table. This final merging focused on the removal of duplicates between the sets of data. Although some duplicates are known (e. g. MOBY, BOUS-SOLE, AERONET-OC and NOMAD data are found in SeaBASS and MERMAID sets of data), others are unknown (e.
g. how much of GeP&CO, ICES, AMT, HOT is within NOMAD, SeaBASS and MERMAID). Therefore, duplicates were identified using the metadata (dataset and subdataset) when possible and temporal-spatial matches as an additional precaution. For temporal-spatial matches, several thresholds were used, but typically 5 min and 200 m were taken to be enough to identify most duplicated data, which reflected small differences in time, latitude and longitude, between the different sets of data. Larger thresholds were used in some cases as a cautionary procedure. This was the case when searching for NOMAD data in other datasets because NOMAD includes a few cases where merging of radiometric and pigment data was done with large spatialtemporal thresholds  (Werdell and Bailey, 2005). With regard to all data, if duplicates were found, data from the NOMAD dataset were selected first, followed by data from individual projects (MOBY, BOUSSOLE, AERONET-OC, AMT, HOT and GeP&CO) and finally for the remaining datasets (SeaBASS, MERMAID and ICES).
This procedure was chosen to preserve the NOMAD dataset as a whole, since it is widely used in ocean-colour validation. After all data were free of duplicates, they were merged consecutively by variable in the final table. During this process, we also searched for rows (stations) that were separated from each other by time differences less than 5 min and horizontal spatial differences of less than 200 m. When such rows were found, the observations in those rows were merged into a single row. The compiled merged data were compared with the original sets to certify that no errors occurred during the merging. As a final step, a water-column (station) depth was recorded for each observation, which was the closest water-column depth from the ETOPO1 global relief model (National Geophys-ical Data Center ETOPO1;  Amante and Eakins, 2009).
For observations where the closest water depth was above sea level (e. g. data collected very near the coast), it was given the value of zero. Data processing thus included two major steps: preprocessing and merging. The first step was related to each set of contributing datasets in particular and aimed to identify problems and convert the data of interest to a standard format. The second step dealt with the integration of all data into one unique file and included the elimination of duplicated data between the individual sets of data acquired. In the next subsections a brief overview of each original set of data is provided.
Marine Optical Buoy (MOBY) The Marine Optical Buoy (MOBY) is a fixed mooring system operated by the National Oceanic and Atmospheric Administration (NOAA) that provides a continuous time series of water-leaving radiance and surface irradiance in the visible region of the spectra from 1997 onwards. The site is located a few kilometres west of the Hawaiian island of Lāna'i where the water depth is about 1200 m. Since its deployment, MOBY measurements have been the primary basis for the on-orbit vicarious calibrations of the SeaWiFS and MODIS ocean-colour sensors. A full description of the MOBY system and processing is provided in  Clark et al. (2003). Data are freely available for scientific use at the MOBY Gold directory. The products of interest are the Scientific Time Series files, which refer to MOBY data averaged over sensorspecific wavelengths and particular hours of the day (around 20:00-23:00 UTC).
For this work, the satellite band-average products for SeaWiFS, MODIS AQUA and MERIS were compiled from the January 2005 reprocessing for the early data and from the latest reprocessing for data after 2011. The "inband" average subproduct was used, and to maintain the highest quality, only data determined from the upper two arms (Lw1) and flagged "good" quality were acquired. Data from the MOBY203 deployment were discarded due to the absence of surface irradiance data. The compiled variable was the remote-sensing reflectance, rrs, which was computed from the original water-leaving radiance (Lw) and surface irradiance (Es). The water-leaving radiances were corrected for the bidirectional nature of the light field  (Morel and Gentili, 1996; Morel et al. , 2002)  using the same look-up table and method as that used in the SeaWiFS Data Analysis System (SeaDAS) processing code.
As mentioned before, the MOBY data compiled in this work are sensor-specific. Therefore, attention is necessary to use the correct MOBY data when validating a particular sensor. The way MOBY data are stored in the final merged table is consistent with the original wavelengths; however, these wavelengths can differ from what is sometimes expected to be the central wavelength of a given band and sensor. Irrespective of the wavelength where MOBY data are stored in the final table, for validation of bands 1-6 of SeaWiFS, MOBY data stored in the final merged table at 412, 443, 490, 510, 555 and 670 nm, respectively, should be used. For validation of bands 1-6 of MODIS AQUA, MOBY data stored in the final merged table  at 416, 442, 489, 530, 547  and 665 nm,  respectively, should be used. Finally, for validation of bands 1-7 of MERIS, MOBY data stored in the final merged table at  410.
The BOUS-SOLE mooring is in the western Mediterranean Sea at a water depth of 2400 m. All pigment  (2001-2012) and radiometric (2003-2012)  data were provided by the principal investigator. The compiled variables were rrs and chla_hplc. Observations of the diffuse attenuation coefficient (kd) were not included in the present compilation, as they were under internal quality revision at the time of data acquisition. Remote-sensing reflectance was computed from the original "fully normalised" water-leaving radiance (nLw_ex), which is the "normalised" water-leaving radiance (nLw previously described), with a correction for the bidirectional nature of the light field  (Morel and Gentili, 1996; Morel et al. , 2002). The solar irradiance (Fo) was computed from two available variables in the original set of data -nLw and rrs -using the equation Fo = nLw / rrs.
, , 2009)). All high-quality data (level 2) were acquired from the project website for 11 sites: Abu_Al_Bukhoosh (∼ 25 • N, ∼ 53 • E), COVE_SEAPRISM (∼ 36 • N, ∼ 75 • W), Gloria (∼ 44 • N, ∼ 29 • E), Gustav_Dalen_Tower (∼ 58 • N, ∼ 17 • E), Helsinki Lighthouse (∼ 59 • N, ∼ 24 • E), LISCO (∼ 40 • N, ∼ 73 • W), Lucinda (∼ 18 • S, ∼ 146 • E), MVCO (∼ 41 • N, ∼ 70 • W), Palgrunden (∼ 58 • N, ∼ 13 • E), Venice (∼ 45 • N, ∼ 12 • E) and WaveCIS_Site_CSI_6 (∼ 28 • N, ∼ 90 • W). The compiled variable was rrs. Remote-sensing reflectance was computed from the original fully normalised water-leaving radiance (see Sect. 2. 2. 2 for definition). The solar irradiance (Fo), which is not part of the AERONET-OC data, was computed from the  Thuillier (2003)  solar spectrum irradiance by averaging Fo over a wavelength-centred 10 nm window.
SeaWiFS Bio-optical Archive and Storage System (SeaBASS) The SeaWiFS Bio-optical Archive and Storage System (SeaBASS) is one of the largest archives of in situ marine bio-optical data  (Werdell et al. , 2003). It is maintained by NASA's Ocean Biology Processing Group (OBPG) and includes measurements of optical properties, phytoplankton pigment concentrations, and other related oceanographic and atmospheric data. The SeaBASS database consists of in situ data from multiple contributors, collected using a variety of measurement instruments with consistent, community-vetted protocols, from several marine platforms such as fixed buoys, hand-held radiometers and profiling instruments. Quality control of the received data includes a rigorous series of protocols that range from file format verification to the inspection of the geophysical data values  (Werdell et al.
, 2003). Radiometric data were acquired through the Validation search tool, which provided in situ data with matchups for particular ocean-colour sensors  (Bailey and Werdell, 2006). The criteria in the search query were defined to have the minimal flag conditions in the satellite data in order to retrieve a greater number of matchups and therefore in situ data. Regarding phytoplankton pigment data, they were acquired through the Pigment search tool, which provides pigment data directly from the archives. As stated in the SeaBASS website (see Pigment tab at http://seabass. gsfc. nasa. gov/seabasscgi/ search. cgi), the Pigment search tool was originally designed to return only in vitro fluorometric measurements, which is consistent with our approach, but over time chlorophyll a measurements made using other methods (e.
g. in situ fluorometry) were included in the retrieved pigment data. In the pigment data used in this work, a large number of in situ fluorometric measurements from continuous underway instruments were identified and discarded. These data were firstly identified from cruises with more than 50 observations per day and then re-checked on the SeaBASS website to confirm whether indeed they were continuous underway measurements. A total of 148 015 such measurements were identified and discarded. Given the large volume of this group of data, it is possible that some chlorophyll a observations from in situ methods may have escaped the scrutiny and made it into the final merged dataset. In the future, the SeaBASS plans to add ancillary information to the extractions, which will enable users to distinguish the different types of chlorophyll measurements.
NASA bio-Optical Marine Algorithm Data set (NOMAD) The NASA bio-Optical Marine Algorithm Data set (NO-MAD) is a publicly available dataset compiled by the NASA OBPG at the Goddard Space Flight Center. It is a highquality global dataset of coincident radiometric and phytoplankton pigment observations for use in ocean-colour algorithm development and satellite-data product-validation activities  (Werdell and Bailey, 2005). The source bio-optical data are the SeaBASS archive; therefore, many dependencies exist between these two datasets, which were addressed during the merging. The current version (Version 2. 0 ALPHA, 2008) includes data from 1991 to 2007 and an additional set of observations of inherent optical properties. The current version was used in this work, but with an additional set of columns of remote-sensing reflectance corrected for the bidirectional effects  (Morel and Gentili, 1996; Morel et al.
, 2002). This additional set of columns was provided directly by the NOMAD creators. The compiled variables were rrs, chla_hplc, chla_fluor, aph, adg, bbp and kd. Conversion was only necessary for aph, adg and bbp and followed the procedures described in Sect. 2. 1. For the calculation of bbp the variable bb was used with a smooth fitting to remove noise. A portion of the NOMAD data were optically weighted (for methods, see  Werdell and Bailey, 2005). These data are not consistent with the protocols chosen in this work, but these observations were retained since NOMAD is a widely used dataset in ocean-colour validation.
MERIS Match-up In situ Database (MERMAID) The MERIS Match-up In situ Database (MERMAID) provides in situ bio-optical data matched with concurrent and comparable MERIS Level 2 satellite ocean-colour products  (Barker, 2013a, b). The MERMAID in situ database consists of data from multiple contributors, measured using a variety of instruments and protocols, from several marine platforms such as fixed buoys, hand-held radiometers and profiling instruments. Comprehensive quality control and protocols are used by MERMAID to integrate all the data into a common and comparable format  (Barker, 2013a, b). Access to MERMAID data is limited to the MERIS Validation Team, the MERIS Quality Working Group and to the in situ data contributors. For this work, access has been granted to the MERMAID database through a signed service-level agreement.
The MERMAID data include subsets of several datasets used in this compilation (MOBY, AERONET-OC, BOUSSOLE, NOMAD). These observations were removed from the MERMAID dataset to avoid duplication (as discussed in Sect. 2. 1). The compiled variables were rrs, chla_hplc, chla_fluor, aph, adg, bbp and kd. Remote-sensing reflectance was calculated by dividing by π the original irradiance reflectance provided. Conversion was also necessary for aph, adg and bbp and followed the procedures described in Sect. 2. 1.
Hawaii Ocean Time-series (HOT) The Hawaii Ocean Time-series (HOT) programme provides repeated comprehensive observations of the hydrography, chemistry and biology of the water column at a station located 100 km north of O'ahu, Hawai'i, from October 1988 onwards  (Karl and Michaels, 1996). This site is representative of the North Pacific subtropical gyre. Cruises are made approximately once a month to the deep-water station ALOHA (A Long-Term Oligotrophic Habitat Assessment; 22 • 45 N, 158 • 00 W). Pigment data (chla_hplc and chla_fluor) were extracted directly from the project website. Radiometric measurements from the HOT project are also available, but observations of rrs and kd from the HOT project were acquired in this work as part of the SeaBASS dataset.
Geochemistry, Phytoplankton, and Color of the Ocean (GeP&CO) The Geochemistry, Phytoplankton, and Color of the Ocean (GeP&CO) is part of the French PROcessus Océaniques et Flux (PROOF) programme and aims to describe and understand the variability of phytoplankton populations and to assess its consequences for the geochemistry of the oceans  (Dandonneau and Niang, 2007). It is based on the quarterly travels of the merchant ship Contship London from France to New Caledonia. A scientific observer embarked on each travel and operated the sampling for surface water, filtration and various measurements at several hours of each day. The experiment started in October 1999 and finished in July 2002. Pigment data were extracted from the project website. The compiled variable was chla_hplc.
Atlantic Meridional Transect (AMT) The Atlantic Meridional Transect (AMT) is a multidisciplinary programme, which undertakes biological, chemical and physical oceanographic research during an annual voyage between the UK and destinations in the South Atlantic  (Robinson et al. , 2006). The programme was established in 1995 and since then has completed 23 research cruises. Pigment data between 1997 (AMT5) and 2005 (AMT17) were provided by the British Oceanographic Data Centre (BODC) following a specific request. For any interest in the original data, the BODC is the point of contact, which ensures that if there are any updates, the most recent data are supplied. The compiled variables are chla_hplc and chla_fluor.
International Council for the Exploration of the Sea (ICES) The International Council for the Exploration of the Sea (ICES) is a network of more than 4000 scientists from almost 300 institutes, with 1600 scientists participating in activities annually. The ICES Data Centre manages a number of large dataset collections related to the marine environment covering the North East Atlantic, Baltic Sea, Greenland Sea and Norwegian Sea. The majority of data originate from national institutes that are part of the ICES network of member countries. Data were provided (on 28 April 2014) from the ICES database on the marine environment (Copenhagen, Denmark) following a specific request. The ICES data were made available under the ICES data policy and if there is any conflict between this and the policy adopted by the users, then the ICES policy applies.
Results In this work several sets of bio-optical in situ data were acquired, homogenised and merged into a single Observations of remote-sensing reflectance are available at 134 unique wavelengths (i. e. columns), between 405 and 1022. 1 nm (Fig. 1 ). In total there are 44 191 observations (i. e. rows) with remote-sensing reflectance in the table. The total number of observations are partitioned per contributing datasets as follows: AERONET-OC (17 405), BOUSSOLE (17 364), MOBY (4513), NOMAD (3326), MERMAID (885) and SEABASS (698). The data from AERONET-OC, BOUSSOLE and MOBY correspond to continuous time series and hence the higher number of observations. The data distribution at 44X nm and 55X nm is provided in Fig. 2a  and b , respectively. Data were first searched at 445 and 555 nm and then with a search window of up to 8 nm to also include data at 547 nm.
Median values at 44X nm range from 0. 003 m -1 (AERONET-OC) and 0. 009 m -1 (MOBY), whereas at 55X nm the median values lie between 0. 001 m -1 (MOBY) and 0. 004 m -1 (AERONET-OC). The observations are evenly distributed on a monthly basis in the Northern Hemisphere (Fig. 3 ). In the Southern Hemisphere, where the number of stations is smaller, there is a decrease in the number of observations during the austral winter months (Fig. 3 ). For additional analysis, rrs band ratios were plotted against each other (490 : 555 vs. 412 : 443, Fig. 4 ). Most points are within the boundaries of the NOMAD dataset, but some scattered points were found. These points were retained in the table to allow further manipulation with different quality con-   and 16 ). The geographic distribution of remote-sensing reflectance observations (Fig.
5 ) shows a higher number of observations in some coastal regions, such as those of North America and Northern Europe. The central regions of the ocean show a lower number of observations, with the Atlantic Ocean having the highest density in relation to the other oceans. Best geographic coverage is provided by the NOMAD database. Data from SeaBASS are smaller in number but are still important. Data from MERMAID are mainly located along the coasts of Europe, North America and the central region of the North Atlantic Ocean. For chlorophyll a concentration, two types of observations were compiled, one measured by fluorometric or spectrophotometric methods (chla_fluor), and the other measured    Relative spectral frequency of remote-sensing reflectance in the final table, using 10 nm wide class intervals, defined as the ratio of the number of observations at a particular waveband to the total number of observations at all wavebands, multiplied by 100 to report results in percentage.