dataesr/scientific-paragraphs-categorization huggingface dataset:
Fig. 6  shows the real consumption and production data for three different households, along with their battery SoC that results from the RTC operations. It shows that the SoC is following the local demand and production. Indeed, the SoC of the household with wind generation is different than the SoC of batteries in households owning solar PV generation. We also assess the economic benefits that an ag-    gregator can expect from the proposed framework. The source of revenues for the fleet are the revenues from energy exports agreed on the wholesale energy market and the total bill reduction of the households. The bill reduction is computed as the total bill difference between the baseline scenario (Scenario 0 defined below) without distributed generation (battery and rooftop PV or wind) and the total bill after the is implemented.
In this work, such unification of multiple data sets is presented. This was done for the validation of the ESA OC-CCI ocean-colour products, but with the intent to also serve the broader user community. A merged data set is not without drawbacks: it is likely to be large (with hundreds of thousands of observations) and so not always easy to manipulate; because the merging is done on pre-existing, processed databases, it is not possible to have full control of the whole processing chain; the data set would be a collection of observations collected by several investigators using different instruments, sampling methods and protocols, which might eventually have been modified by the processing routines used by the repositories or archives. To minimise these potential drawbacks, we have, for the most part, incorporated only data sets that have emerged from the long-term efforts of the ocean-colour and biological oceanographical communities to provide scientists with high-quality in situ data, and implemented additional quality checks on the data, to enhance confidence in the quality of the merged product.
One requirement for "chla_fluor" measurements was that they were made using in vitro methods (i. e. , based on extractions of chlorophyll-a). Although this severely decreased the number of observations, since in vivo fluorometry (e. g. , fluorometers mounted on CTD's) is widely available in oceanographic The variable "chla_hplc" was calculated by summing all reported chlorophyll-a derivatives, including divinyl chlorophyll-a, epimers, allomers, and chlorophyllide-a. The two chlorophyll variables are retained separately in the database to facilitate their use. HPLC measurements could be considered of higher quality, but fluorometric measurements are more numerous. Thus, one option for users is to use "chla_fluor" only when there are no "chla_hplc" measurements available. To be consistent with satellite-derived chlorophyll values, which are derived from the light emerging from the upper layer of the ocean, all chlorophyll observations in the top 10 meters (replicates at the same depth, or measurements at multiple depths) were averaged if the coefficient of variation among observations was less than 50 %, otherwise they were discarded.
For the latter equation, the variable "bbw" was computed using "bbw = bw/2", where "bw" is the scattering coefficient of seawater derived from  Zhang et al. (2009). The diffuse attenuation coefficient for downward irradiance ("kd") did not require any conversion and was compiled as originally acquired. Observations of inherent optical properties (surface values) and diffuse attenuation coefficient for downward irradiance, were acquired in total from six data sources designed for ocean-colour validation and applications (SeaBASS, NOMAD, MERMAID, AWI, COASTCOLOUR, TPSS), thus already subject to the processing routines of these data sets. Concerning total suspended matter, these data were compiled as originally available from MERMAID and COASTCOLOUR. The merged data set was compiled from 27 sets of in situ data, which were obtained individually either from archives that incorporate data from multiple contributors (SeaBASS, NOMAD, MERMAID, ICES, ARCSSPP, BIOCHEM, BODC, COASTCOLOUR, MAREDAT, SEADATANET), or from particular contributors, measurement programs or projects (MOBY, BOUSSOLE, AERONET-OC, HOT, GeP&CO, AMT, AWI, BARENTSSEA, BATS, CALCOFI, CCELTER, CIMT, ESTOC, IMOS, PALMER, TPSS, TARA) and were subsequently, homogenized and merged.
0001-10] m -1 ; for "tsm" [0-1000] g m -3 ; for "kd"  [(aw(λ) -10] m -1 , where "aw" are the pure water absorption coefficients derived from  Pope and Fry (1997). Also, during this stage, three metadata strings were attributed to each observation: "dataset", "subdataset" and "contributor". The "dataset" contains the name of the original set of data, and can only be one of the following: "aoc", "boussole", "mermaid", "moby", "nomad", "seabass", "hot", "ices", "amt", "gepco", "arcsspp", "awi", "barentssea", "bats", "biochem", "bodc", "calcofi", "cc", "ccelter", "cimt", "estoc", "imos", "maredat", "palmer", "seadatanet", "tpss", "tara". The "subdataset" starts with the "dataset" identifier and is followed by additional information about the data, as <dataset>_<cruise/station/site>) (e. g. , "seabass_car81").
A small number of observations that were identified as replicates had a different "subdataset" identifiers (i. e. , different cruise names). These observations were considered suspicious if the values were different, and discarded. If the values were the same, one of the observations was retained. This possibly originated from the same group of data being contributed to an archive by two different data contributors. Once a set of data was homogenized, its data were integrated into a unique table. This final merging focused on the removal of duplicates between the sets of data. Although some duplicates are known (e. g. , MOBY, BOUSSOLE, AERONET-OC and NOMAD data are found in SeaBASS and MERMAID), others are unknown (e. g. , how many of GeP&CO, ICES, AMT, HOT are within NOMAD, SeaBASS and MERMAID).
Therefore, duplicates were identified using the metadata ("dataset" and "subdataset") when possible, and temporal-spatial matches, as an additional precaution. For temporal-spatial matches, several thresholds were used, but typically 5 minutes and 200 meters were taken to be sufficient to identify most duplicated data, which reflected small differences in time, latitude and longitude, between the different sets of data. Larger thresholds were used in some cases as a cautionary procedure. This was the case when searching for NOMAD data in other data sets, because NOMAD includes a few cases where merging of radiometric and pigment data was done with large spatial-temporal thresholds  (Werdell and Bailey, 2005). A large temporal threshold was also used when integrating observations from the three data sources that did not have time available  (ESTOC, MAREDAT and TPSS).
In regard to all data, if duplicates were found, data from the NOMAD data set were selected first, followed by data from individual projects or contributors (MOBY, BOUSSOLE, AERONET-OC, AMT, HOT,GeP&CO, AWI, BARENTSSEA, BATS, CALCOFI, CCELTER, CIMT, ESTOC, IMOS, PALMER, TPSS and TARA) and finally for the remaining data sets (SeaBASS, MERMAID, ICES, ARCSSPP, BIOCHEM, BODC, COASTCOLOUR, MAREDAT and SEADATANET). This procedure was chosen to preserve the NOMAD data set as a whole, since it is widely used in ocean-colour validation. It should be noted that, by this procedure, data from individual projects or contributors may be listed under NOMAD (e. g. , some PALMER data are found in NOMAD with metadata string "nomad_palmer_lter"). After giving priority to NOMAD, the priority was generally given to data from individual projects or contributors, but due to an incremental approach, where only new data are added to previous versions of the compilation, some data from individual projects or contributors (BATS, CALCOFI, CIMT, PALMER and TPSS) added in later stages, may be found under other data sources.
Marine Optical BuoY (MOBY) MOBY is a fixed mooring system operated by the National Oceanic and Atmospheric Administration (NOAA) that provides a continuous time series of water-leaving radiance and surface irradiance in the visible region of the spectra since 1997. The site is located a few kilometres west of the Hawaiian Island of Lanai where the water depth is about 1200 m. Since its deployment, MOBY measurements have been the primary basis for the on-orbit vicarious calibrations of the SeaWiFS and MODIS ocean colour sensors. A full description of the MOBY system and processing is provided in  Clark et al. (2003). Data are freely available for scientific use at the MOBY Gold directory. The products of interest are the "Scientific Time Series" files, which refer to MOBY data averaged over sensor-specific wavelengths and particular hours of the day (around 20-23 UTC).
For this work, the satellite band-average products for SeaWiFS, MODIS AQUA, MERIS, VIIRS-SNPP, VIIRS-JPSS (also known as NOAA-20 VIIRS), OLCI-S3A and OLCI-S3B were compiled from the "R2017 Reprocessing". The "inband" average subproduct was used, and to maintain the highest quality, only data determined from the upper two arms ("Lw1") and flagged "good" quality were acquired. Data from the MOBY203 deployment were discarded due to the absence of surface irradiance data. The compiled variable was the remote-sensing reflectance, "rrs", which was computed from the original water-leaving radiance ("Lw") and surface irradiance ("Es"). The water-leaving radiances were corrected for the bidirectional nature of the light field  (Morel and Gentili, 1996; Morel et al. , 2002)  using the same look-up table and method as that used in the SeaWiFS Data Analysis System (SeaDAS) processing code.
Therefore, attention is necessary to use the correct MOBY data when validating a particular sensor. The way MOBY data are stored in the final merged table is consistent with the original wavelengths; however, these wavelengths can differ from what is sometimes expected to be the central wavelength of a given band and sensor. Irrespective of the wavelength where MOBY data are stored in the final table, for validation of bands 1-6 of SeaWiFS, MOBY data stored in the final merged table at 412, 443, 490, 510, 555 and 670 nm, respectively, should be used. For validation of bands 1-7 of MODIS AQUA, MOBY data stored in the final merged table at 416, 442, 489, 530, 547, 665 and 677 nm, respectively, should be used. For validation of bands 1-10 of MERIS, MOBY data stored in the final merged table at  410.
The BOUSSOLE mooring is in the Western Mediterranean Sea at a water depth of 2400m. All pigment  (2001-2019) and radiometric (two subsets: 2003-2012 and 2015-2019)  data were provided by the Principal Investigators. The first radiometric subset was obtained from measurements made with multispectral Satlantic OCI-200 radiometers; the second radiometric subset was obtained from measurements made with hyperspectral Satlantic OCR radiometers, convolved with spectral response function of Sentinel3 OLCI-A bands. The compiled variables were "rrs" and "chla_hplc". Remote-sensing reflectance was computed from the original "fullynormalized" water-leaving radiance ("nLw_ex"), which is the "normalized" water-leaving radiance ("nLw" previously described), with a correction for the bidirectional nature of the light field  (Morel and Gentili, 1996; Morel et al.
SeaWiFS Bio-optical Archive and Storage System (SeaBASS) SeaBASS is one of the largest archives of in situ marine bio-optical data  (Werdell and Bailey, 2003)  with a long-established inventory  (Hooker et al. 1994). It is maintained by NASA's Ocean Biology Processing Group (OBPG) and includes measurements of optical properties, phytoplankton pigment concentrations, and other related oceanographic and atmospheric data. The SeaBASS database consists of in situ data from multiple contributors, collected using a variety of measurement instruments with consistent, community-vetted protocols, from several marine platforms such as fixed buoys, hand-held radiometers and profiling instruments. Quality control of the received data includes a rigorous series of protocols that range from file format verification to inspection of the geophysical data values  (Werdell and Bailey, 2003).
MERIS Match-up In situ Database (MERMAID) MERMAID provides in situ bio-optical data matched with concurrent and comparable MERIS Level 2 satellite ocean-colour products  (Barker, 2013a; Barker, 2013b). The MERMAID in situ database consists of data from multiple contributors, measured using a variety of instruments and protocols, from several marine platforms such as fixed buoys, hand-held radiometers and profiling instruments. Comprehensive quality control and protocols are used by MERMAID to integrate all the data into a common and comparable format  (Barker, 2013a; Barker, 2013b). Access to MERMAID data is limited to the MERIS Validation Team, the MERIS Quality Working Group and to the in situ data contributors. For this work, access has been granted to the MERMAID database, through a signed Service Level Agreement.
International Council for the Exploration of the Sea (ICES) ICES is a network of more than 4000 scientists from almost 300 institutes, with 1600 scientists participating in activities annually. The ICES Data Centre manages a number of large data set collections related to the marine environment covering the Northeast Atlantic, Baltic Sea, Greenland Sea and Norwegian Sea. Most of data originate from national institutes that are part of the ICES network of member countries. Data were provided (on 2014-04-28) from the ICES database on the marine environment (Copenhagen, Denmark) following a specific request. The ICES data were made available under the ICES data policy and if there is any conflict between this and the policy adopted by the users, then the ICES policy applies. The compiled variables were "chla_hplc" and "chla_fluor".
Arctic System Science Primary Production (ARCSSPP) ARCSSPP database is a synthesis of observations between 1954 and 2006, from the Arctic Ocean and northern Seas  (Matrai et al. , 2013). The observations were acquired from data repositories, publications or provided by individual investigators. The database includes quality-controlled observations of productivity and chlorophyll a, photosynthetically available radiation and hydrographic parameters. This collection of data was acquired at http://www. nodc. noaa. gov/cgibin/OAS/prd/accession/download/63065. For the present work, only observations of chlorophyll-a concentration with known time zones were used. The compiled chlorophyll observations were from discrete samples, but the exact method (either "chla_fluor" or "chla_hplc") was not available for all observations.
Thus, the ARCSSPP chlorophyll observations were marked as "chla_fluor", although some might have been from HPLC measurements, and were flagged with "1" in a column "flag_chla_method". The compiled variable was "chla_fluor". (doi. org/10. 1594/PANGAEA. 907419). The compiled variables were "chla_hplc", "rrs" and "aph".
British Oceanographic Data Centre (BODC) BODC is the designated marine science data centre for the United Kingdom. The data used in this work derive from a specific request for discrete observations of chlorophyll-a concentration since 1997. Initially, this request was used to compile AMT data (see section 2. 2. 9). The remaining data comprising observations of chlorophyll-a concentration from fluorometric and HPLC methods, mostly sampled in the North Atlantic, were analysed and added (the "dataset" string for this data source is "bodc"). Data not flagged with highest quality or without method of measurement were discarded. The compiled variables were "chla_hplc" and "chla_fluor".
California Cooperative Oceanic Fisheries Investigations (CALCOFI) CalCOFI is a partnership of the California Department of Fish & Wildlife, National Oceanic & Atmospheric Administration Fisheries Service and Scripps Institution of Oceanography. CalCOFI has conducted quarterly cruises off southern and central California since 1949. Data collected in the upper 500 meters include: temperature, salinity, oxygen, nutrients, chlorophyll, primary productivity, plankton biodiversity, and biomass. For this work, only observations of chlorophyll-a concentration derived from fluorometric methods flagged with highest quality were used. Data were acquired from the file "CalCOFI_Database_194903-201911_csv_10Jul2020. zip". The compiled variable was "chla_fluor".
CoastColour Round Robin (COASTCOLOUR) COASTCOLOUR data sets were designed to evaluate the performance of ocean colour satellite algorithms in the retrieval of water quality parameters in coastal waters  (Nechad et al. , 2015). Three types of COASTCOLOUR data sets are available: 1) a match-up data set where in-situ bio-optical observations are available simultaneously with a cloud-free MERIS product; 2) an in-situ reflectance data set where an in-situ reflectance is available simultaneously with an in-situ measurement of chlorophyll-a concentration and/or total suspended matter; and 3) a simulated data set where reflectances were generated by a radiative transfer model. This work used the match-up data set, which includes most of the in-situ measurements, and is available at https://doi. pangaea.
de/10. 1594/PANGAEA. 841950. The match-up data set provides optical, biogeochemical and physical data collections at 17 sites across the globe. From this data set, observations of reflectance, chlorophyll a, total suspended matter and IOPs were compiled. The remote sensing reflectances were corrected for the bidirectional nature of the light field  (Morel and Gentili, 1996; Morel et al. , 2002). The compiled variables were "rrs", "chla_hplc", "chla_fluor", "aph", "adg", "bbp" and "tsm".
Australia's Integrated Marine Observing System (IMOS) IMOS   (Schroeder et al. , 2016). The previous data compilations include an earlier subset of HPLC chlorophyll a concentration from the IMOS Bio-optical Database that was acquired through the SeaBASS archive. These data can be found under "dataset" string "seabass" and Lesley Clementson as data contributor. The compiled variables for IMOS were "chla_hplc" and "chla_fluor".
MARineEcosytem DATa (MAREDAT) MAREDAT database is a global assemblage of pigments measured by HPLC  (Peloquin et al. , 2013)   Total Chlorophyll a flagged with high quality were used. The time of day was unavailable and was set to 12:00:00 (UTC). These observations were flagged with "1" in column "flag_time". The compiled variable was "chla_hplc".
Results In this work several sets of bio-optical in situ data were acquired, homogenised and merged into a single unified data set. The data set comprises in situ observations between 1997 and 2021, with a global distribution, and includes the following variables: "rrs", "chla", "aph", "adg", "bbp", "kd" and "tsm". All observations were processed in such a way that they can be compared directly with satellite-derived ocean-colour data. The compiled data set corresponds to a table with a total of 151,673 rows and 3,458 columns. Each row represents a unique station in space and time, separated from the rest by at least 5 minutes and 200 meters. For each variable at a given station, three metadata strings are provided: "dataset", "subdataset" and "contributor". The columns of the table take the form described in Table  1.
The data contributors are indicated in Table  2. Regarding spectral variables, all original wavelengths were preserved, which required many unique wavelengths to be maintained in the database. No band shifting was performed (though some archived data in some data sources may have been merged with nearby wavelengths) and no minimum number of wavelengths per observation was imposed. This allowed further manipulation of the data set for different purposes. In the following paragraphs, the final group of observations is described in terms of each variable and the corresponding contributing data sets; however, it is important to note that the numbers reported here do not reflect the original numbers in each contributing data set, since observations close in time and space were averaged and quality controls were applied.
5 ) still show a higher number of observations in some coastal regions, such as those of North America and Northern Europe. Away from continental margins, the Atlantic Ocean has the highest density of observations. Best geographic coverage is provided by the NOMAD database. Data from SeaBASS is also well dispersed in space but fewer in number. Data from MERMAID are mainly located along the coasts of Europe, North America, and the central region of the North Atlantic Ocean. The observations from AERONET-OC, BOUSSOLE, COASTCOLOUR and MOBY are concentrated in specific sites around the world, while AWI data are available for the Atlantic and Arctic Oceans. TARA data are spread across several regions, with highest data density in the Mediterranean Sea. Observations of chlorophyll-a concentration were divided into those measured by fluorometric or spectrophotometric methods ("chla_fluor"), and HPLC methods ("chla_hplc").
Data distribution of "aph", "adg" and "bbp" at 44X nm and 55X nm for each data set is provided in Fig. 12  a-f. Median values of "aph", "adg" and "bbp" at 44X and 55X nm for each data set are summarized in Table  3. As a quality indicator, the following band ratios for the absorption coefficients report 5 for quality control  (IOCCG, 2006 ; see dotted vertical black lines in Fig. 13 ). These points are highlighted here for information, but retained in the database, since these were mostly from NOMAD and there was an interest to preserve this data set as a whole. Also, not discarding these data allows further manipulation with different quality control criteria. On the annual scale, the observations of the inherent optical properties continue to be strongly underrepresented in the Southern Hemisphere where there is a complete absence of data during the austral winter (Fig.
Astrid Bracher, Alfred-Wegener-Institute Helmholtz Centre for Polar and Marine Research (AWI) In this work, the AWI data source refers to the group of observations that were provided to OC-CCI project by Astrid https://doi. org/10. 5194/essd-2022started: 20 May 2022 c Author(s) 2022. CC BY 4. 0 License. Bracher. These are bio-optical observations collected during several cruises across the globe. All data were available through the PANGAEA repository. Observations of concentration of chlorophyll-a, and 1nm spectrally resolved remote sensing reflectances and algal pigment absorption coefficient were considered. The methods for these observations are described byTaylor et al. (2011 ), Liu et al. (2018)  andTilstone et al. (2020). For chlorophyll, data from the following cruises were used: ANT-XXIV/1, ANT-XXIV/.
: 20 May 2022 c Author(s) 2022. CC BY 4. 0 License. CCELTER investigates the California Current coastal pelagic ecosystem, with a focus on long-term forcing. The CCELTER data includes primary and derived measurements from both Process and CalCOFI-augmented cruises, as well other time series. CCELTER data include variables from the physical environment, biogeochemistry and biological populations/communities. For this work chlorophyll observations measured from discrete bottle samples from CCELTER Process cruises determined by extraction and bench fluorometry (doi:10. 6073/pasta/bbb278091dee3c96972087b7dee3673c).
The other chlorophyll a data set measured by HPLC and fluorometry methods, is a subset (2015-2021) of the IMOS Biooptical Database also available through the AODN portal. This database comprises of a suite of bio-optical parameters from samples collected during research voyages in Australian waters and is used by the IMOS Ocean Colour Sub-Facility to assess the accuracy of satellite ocean colour products in Australian coastal and open ocean waters.