dataesr/scientific-paragraphs-categorization huggingface dataset:
We recall that the density of a network of N nodes and E edges is defined as the ratio of the number of edges to the maximal possible number of edges that could exist between the nodes, i. e. , d = E/(N(N -1)/2). The contact matrix of densities gives for each pair of groups X and Y the ratio between the total number of links E XY between the n X individuals in X and the n Y individuals in Y , and the maximum number of such possible links (n X n Y if X = Y or n X (n X -1)/2 if X = Y ). Moreover, the sampling also maintains the temporal statistics of contacts. It is thus possible to measure this contact matrix and the temporal statistics in the incomplete data and to construct surrogate data having the same statistics as the original one. Spreading processes simulated using such surrogate data have been shown to reproduce well the outcome of simulations using the whole dataset.
This is expected to be the case in particular for data coming from diaries or surveys, and it is of interest to test the limits of the reconstruction method, as a function of the sampling properties, and possibly to understand how to overcome this obstacle. Here, we tackle this issue by considering incomplete contact data stemming from a non-uniform sampling procedure intended to mimic a survey procedure in which (i) not all individuals participate and (ii) the contacts of each respondent are reported with a probability depending on their duration. 30 We consider empirical contact datasets, resample them using this non-uniform sampling procedure and design surrogate data as described above. We show that, at low sampling, the use of such surrogate data in simulations of spreading processes is not enough to estimate the epidemic risk, even if it yields an improvement with respect to the use of the raw sampled data.
We thus consider the case in which additional information is available for one of the groups forming the population: if one group is uniformly sampled (for instance if wearable sensors are available for this group), yielding a good estimate of its density, it is possible through a simple rescaling procedure to estimate a new contact matrix for the data and to use it to construct another surrogate dataset that yields better results. By using two datasets with different structures and varying the parameters of sampling and spreading processes, we explore the efficiency and limits of both procedures.
Data and methodology We first describe the datasets and the different steps of our methodology, which are in the same spirit as Refs. 24, 29   tarting from a network of contacts between individuals in a population, we perform a specific resampling that leads to an incomplete dataset. We describe two methods to create surrogate datasets using the statistical information contained in the incomplete data. Spreading processes are then simulated on top of the incomplete and of the surrogate datasets, and their outcomes compared with the ones of simulations using the whole original contact network. The procedure is summarized in Fig. 1.
SIR simulations Figure  1. Sketch of the procedures considered in the article. We consider a dataset describing a contact network. We perform a resampling according to a certain (non-uniform) sampling method. We then measure a number of statistics of the resampled data and use these statistics to construct surrogate data. We simulate spreading processes on the original, resampled and surrogate data and compare the outcomes, as given by the fraction of large epidemics and by the whole distribution of epidemic sizes.
Data We will use two datasets describing face-to-face contacts between individuals, collected and made publicly available by the SocioPatterns collaboration (see the SocioPatterns website http://www. sociopatterns. org/). The first dataset (Thiers13) has been collected in a French high school in December 2013. The resulting contact network has N = 327 nodes representing students (divided in 9 classes corresponding to different fields of study) and E = 5818 weighted edges (see  14  for a detailed description and analysis of this dataset), for an average link density of 2E/(N(N -1)) ≈ 0. 11. The participation rate reached 86. 5% (there were overall 378 students in the 9 classes). The classes are of similar sizes (see Table  1 ), and most edges (69%, accounting for 93% of the weights, i. e. , of the total contact time between students) are found within classes (Fig.
2 ). The second dataset (InVS) has been collected in the office buildings of the Institut de Veille Sanitaire (French Institute for Public Health Surveillance) in March 2015, and describes the contacts between 217 individuals divided in 12 departments. The contact network has 4274 edges, i. e. , a density of ≈ 0. 18. The departments have very different sizes and participation rate, with an average participation rate of 60% (see Table  2 ) and, while most contact time occurs within departments (76%), the corresponding fraction of edges is only 42% (Fig. 2 ). In each dataset, an edge between two individuals corresponds to the fact that these individuals have been in contact at least once during the data collection, and the edge weight gives the total contact time between them. The contact matrices of edge densities are shown in Fig.
2  for both datasets. In the Thiers13 case, we have moreover access to a network describing friendship relations between students, obtained by a survey to which 135 of the 327 students answered. This friendship network has 413 unweighted edges. For each matrix, the entry at row X and column Y is given by the total number of links between individuals in class or department X and individuals in class or department Y , normalized by the maximum number of observable links (n X n Y or n X (n X -1)/2 if X = Y , with n X the cardinality of X). These matrices give only structural information as they do not take into account edge weights.
Sampling method We consider the sampling method put forward in,  30  called "EGOref", based on (i) a uniform sampling of nodes and (ii) a non-uniform sampling of edges, edges with larger weights being preferentially sampled. This procedure is designed to mimic a sampling of links obtained for instance through a survey on friendship relations in a population: it is inspired by the result of Ref. 14  that the longest contacts measured in the Thiers13 dataset corresponded to reported friendships, while many short contacts did not. In particular, we have shown in Ref. 30   network of the Thiers13 dataset can be reproduced if the EGOref sampling method is applied to the contact network of the same dataset, with correctly adjusted sampling parameters. More precisely, the EGOref sampling depends indeed on two parameters.
24 igure  4  displays the ratio between the density of the EGOref sampled network and the original contact network for the two datasets used here: the density of the sampled network increases with the parameter p. On the other hand, the density of the sampled network does not depend on the number of sampled nodes (Figure  S1  in the Supplementary Information). Despite the decrease in network density caused by the EGOref sampling, it is worth noting that the contact matrices of edge densities measured in the original contact network and in the EGOref sampled network remain very similar for both datasets (see Figures  S2-S4  in the Supplementary Information). These matrices give, for each pair of groups in the population (here, classes or departments), the number of links between these groups normalized by the maximum possible number of such links (obtained if each member of one group is linked to all members of the other group).
We then assume that the EGOref sampling method affects in a similar way all parts of the graph, so that a global rescaling of the density, which also rescales all the elements of the contact matrix of edge densities by the same factor, should yield values closer to the original data. In the following, we will apply each method to contact networks sampled using the EGOref method for various values of the parameters p and N. As Ref. 30  has shown the similarity between the friendship network of the Thiers13 dataset and the outcome of the EGOref sampling applied to the Thiers13 contact network, for a specific parameter value, we will also apply these methods to construct surrogate contact data using the friendship network instead of a sampled version of the original contact network.
Simulation of spreading processes Our goal is to understand if it is possible to use incomplete datasets to estimate the epidemic risk in a population by constructing surrogate data and using them in the simulations of epidemic spread. As a paradigm of epidemic process, we consider the Susceptible-Infectious-Recovered (SIR) model: in this model, nodes are initially all susceptible (S), except one in the Infectious state, chosen at random and seed of the process. Each Susceptible (S) node i can become infectious when in contact with an Infectious one j. This occurs at a rate βW i j /T where W i j is the weight of the link ij and T the total measurement time  30, 31  (i. e. , the probability for i to become infectious during a time step dt is βW i j dt/T ). Infectious nodes become Recovered (R) at rate µ and cannot be infected anymore.
The process ends when there are no Infectious nodes any more. We perform numerical simulations of this model for each dataset on the original contact network, on the sampled networks at various values of the parameters p and N, and on the surrogate datasets built using the two methods described above (note that we will equivalently write "surrogate data" or "reconstructed networks" to describe the surrogate datasets). For the Thiers13 case, we also perform simulations on the friendship networks and the corresponding surrogate data. We also vary the ratio β /µ that modulates, in each given network, the impact of the modelled disease. To quantify the epidemic risk, we measure in each simulation the epidemic size as given by the final fraction of recovered nodes. We compare the distributions of epidemic sizes, the fraction of epidemics with size larger than 20% and the average size of these epidemics (the cut-off of 20% is chosen arbitrarily to distinguish between small and large epidemics; changing the value of this threshold does not alter our results).
We finally note that we consider here static versions of the contact networks, while the original SocioPatterns data provides temporally resolved contacts. The EGOref sampling process indeed mimics a procedure yielding a static sample of the actual contact network. In the context of models of infectious diseases with realistic timescales of several days, this can represent enough information to obtain an estimate of the epidemic risk, as discussed in Ref. 31  For faster spread, one could add to the surrogate construction method an additional step of building realistic contact timelines as in Ref. 24.
Results We discuss separately the results obtained with the two datasets. The Thiers13 one is indeed much more strongly "structured" than the InVS one, in the sense that the fraction of interactions occurring within each class is very high. Moreover, all classes are of similar sizes and have similar link densities. Classes are also arranged in groups of 2 or 3 classes corresponding to the major topic of study of their students. In the InVS case, departments are of different sizes, their link densities vary more and the pattern of interactions between departments is less structured. corresponds to N = 135 and p = 30. The sampled network has a much lower density than the original contact data, smaller clustering and larger average shortest path. The surrogate network built using the first method has by construction the same density, and has an even smaller clustering, while the second method yields values much closer to the original ones.
Second case: a less structured network (InVS dataset) We now investigate the results obtained with the InVS dataset. As discussed above, the structuration in departments leads to a less structured contact matrix than in the highschool case. Table  5  and Figure  S4  of the Supplementary Information compare some characteristics of the sampled and surrogate data (with the second method of reconstruction) to the original network for p = 30 and N = 93 (i. e. , a similar fraction of the population as used in the example of the Thiers13 dataset). Here, even the second method of reconstruction leads to a network density smaller than the original one, even if much closer than for the sampled data. The contact matrices of the sampled and of the surrogate data are very similar to the one of the original data (Figure  S4  of the Supplementary Information), but, as in the previous case, the fraction of intra-department edges is larger in the surrogate data than in the original one (58% versus 42%) while the fraction of the weights these links carry is much smaller (58% versus 76% in the original contact data).
N Figures 9 and 10 compare the outcomes of SIR simulations performed on the contact network, the sampled and the reconstructed networks, for various values of the sampling and spreading parameters. As for the Thiers13 case, the simulations on the sampled networks strongly underestimate the epidemic risk, except obviously at large p and N. The use of surrogate data generally improves the estimation of the epidemic risk, except at large N for the first reconstruction method, as in this case almost no nodes have to be added so the network is almost unchanged by this method. For small and intermediate p, the second method gives better estimations of the average epidemic size while, at very large p, the effect of assigning randomly weights leads as before to a slight overestimation for reconstructed networks.
At small β /µ and p, the second method of reconstruction can even overestimate this size substantially. Figure  10  sheds some more light by displaying the whole distributions of epidemic sizes for several values of the spreading parameter β /µ and of the sampling parameter p, for a rather small N of the order of 40% of the total population. While the distributions obtained with the second reconstructed networks are generally closer to the ones obtained with the original contact data than with the EGOref sampled network (which always leads to a very strong underestimation) or the first reconstruction method, a peak at very large epidemic sizes is observed for the reconstructed data, which is not present for the original contact data. The maximal sizes of epidemics is thus overestimated. As in the Thiers13 case, this effect is likely to be due to the fact that the amount of weights on inter-departments edges is larger in the surrogate data than in the original data, allowing for a more efficient spread across the whole population.
Discussion This paper positions itself in the context of the issue of data incompleteness in contact networks. ] [31] [32]  We have here considered the case of non-uniformly sampled contact data and focused on a sampling procedure designed to mimic data resulting from surveys or diaries. 30 This sampling procedure results in both population sampling, as not all individuals in the population are respondents, and in non-uniform link sampling, to mimic the fact that longer contacts have a larger probability to be remembered or to correspond also to friendship links. We have applied this sampling procedure, called EGOref, on two datasets of contact networks in two different contexts and varied its two parameters, which determine the population participation rate and the fraction of sampled links.
The datasets concern populations structured in groups with very different mixing patterns: in a high school, the class structure strongly determines contacts, with more than 90% of the duration of contacts occurring within classes; in office buildings on the other hand, the population is divided into departments but the impact on the contacts is less strong. As expected from previous investigations, using sampled data to run simulations of spreading processes leads to a strong underestimation of the epidemic risk, as quantified by the distribution of epidemic sizes. We have therefore considered the issue of building surrogate data from the sampled data, such that simulations using the surrogate data yield a better estimation of the epidemic risk. The first method we envisioned has been shown to yield good results in the context of uniform population sampling.
Our study contributes to the discussion on the amount of details actually needed in contact data to be used in data-driven models. It is worth noting some of its limitations. We have focused here on one specific sampling model, while others might be of interest. We argue that this procedure is particularly relevant as it mimics surveys or diaries, as described in Ref. 30  The procedure also assumes a uniform node sampling, while positive or negative correlations with actual contact activity might exist. Additional investigations concerning such non-uniform population sampling would certainly be of interest, as well as studies on other datasets or on synthetic populations with tuneable characteristics. Finally, the effect of using sampled or surrogate data for data-driven simulations of other types of processes ought to be investigated.
Preliminary simulations of the Susceptible-Infectious-Susceptible model (SIS), in which individuals who recover become again susceptible and can catch again the disease, show that results similar to the SIR case are obtained (not shown): the sampling leads to a strong underestimation of the epidemic risk that is compensated only partially by the first method of reconstruction; the second method gives excellent results for the very structured dataset but still leads to an underestimation of the epidemic risk for the offices dataset. The issue of the performance of using surrogate data in more complex processes such as complex contagion remains open for future investigations.
, 2009 Kostadinov et al. , , 2016;; Brewin et al. , 2010 Brewin et al. , , 2015;; Fujiwara et al. , 2011; Li et al. , 2013; Roy et al. , 2013)  or a size factor characterizing the contribution of pico-(or micro-) phytoplankton to the phytoplankton community  (Ciotti and Bricaud, 2006; Mouw and Yoder, 2010; Bricaud et al. , 2012). Currently, only the products OC-PFT  (Hirata et al. , 2011)  and PhytoDOAS  (Bracher et al. , 2009; Sadeghi et al. , 2012a)  enable the simultaneous determination of chl-a for several PT. PhytoDOAS retrieves the imprints of absorption characteristics of specific phytoplankton groups among all other atmospheric and oceanic absorbers from top of atmosphere data of the hyperspectral satellite sensor SCIAMACHY (Scanning Imaging Absorption Spectrometers for Atmospheric Chartography).
, 2011)  and secondly, under the 2nd Satellite PFT Algorithm International Intercomparison Project: http://pft. ees. hokudai. ac. jp/satellite/index. shtml. The initiative strengthens the links between algorithm developers at a global scale which will help also to guide modelers and policy makers on the specific assumptions underlying each product: the intercomparison among most algorithms presented in Table  2  and to an ensemble mean of Earth System Models is presented in  Kostadinov et al. (2017). A user guide for application to open ocean waters on the most common algorithms  (Mouw et al. , 2017)  explains the current global PG algorithms and their associated uncertainties and also includes a discussion on the advantages and disadvantages of these algorithms. A global in situ dataset of HPLC and optical properties is being developed to further evaluate these algorithms.
g. ,  Alvain et al. , 2005; Hirata et al. , 2011; Soppa et al. , 2014; Swan et al. , 2016)  and PSC (e. g. ,  Uitz et al. , 2006; Brewin et al. , 2010)  data sets have been complied, complemented by the global pigment data set compiled under the MAREDAT project  (Peloquin et al. , 2013)  and recent submissions to public data bases: e. g. , SEABASS (http://seabass. gsfc. nasa. gov/), BODC (http://www. bodc. ac. uk), LTER Network Data Portal (https://portal. lternet. edu/nis/home. jsp), PANGAEA Data Publisher for Earth & Environmental Science (https:// www. pangaea. de). Among all available in situ PG data sets, HPLC-phytoplankton pigment data contain the largest number of observations resulting in the greatest spatial coverage with standardized quality control protocols  (Hooker et al. , 2012).
, 1990)  and decomposition  (Hoepffner and Sathyendranath, 1993)  methods are often applied separately to bio-optical datasets to explore the link between pigments and phytoplankton absorption. Reconstruction approaches conventionally apply a single pigment-specific absorption coefficient to a particular pigment or pigment type (e. g. , photosynthetic and photoprotective carotenoids), often obtained from measurements of extracted pigments in solvent. Only a handful of studies have examined the absorptive properties of pigment-protein complexes (e. g. ,  Johnsen and Sakshaug, 2007 ), yet differences in the spectral shape once pigments are embedded in proteins can be significant. Improved models on phytoplankton photoacclimation combined with new approaches in determining cell size should assist in improving our understanding of how pigment packaging influences the spectral signature of natural phytoplankton assemblages.
, 2016). Thus, PG related specific IOPs are not adequately represented in RTMs. This further limits tracing uncertainty in algorithms, pointing to the need for coincident IOP observations along with expended in situ datasets of phytoplankton composition. Other errors in algorithms are also difficult to assess, for instance the accuracy of in situ data used as input or for validation of algorithms due to mostly non-standardized acquisition (see details below), the above mentioned mismatch definition (see Gap 1), and the spatial and temporal upscaling of specific PT and PSC signatures of diverse communities. Several studies have demonstrated that adding spectrally-resolved optics to biogeochemical models improves model skill (e. g. ,  Dutkiewicz et al. , 2015)  as well as comparability to observed optical properties (e.
g. ,  Fujii et al. , 2007; Baird et al. , 2016). A minority of global numerical models resolve the bio-optical properties of different PG  (Gregg and Casey, 2007; Dutkiewicz et al. , 2015; Baird et al. , 2016). These advancements may provide a way forward to investigate the biological realism of phytoplankton biogeography using a larger range of satellite PG products. In addition, the reliance on HPLC for development and validation presents challenges to quantitatively assess the uncertainty of PG satellite products. However, inputs to HPLC PG datasets are (diagnostic) accessory pigment concentrations, which are only to a certain degree congruent with taxonomy or phytoplankton size. Size can vary considerably within certain functional or taxonomic groups, e. g. , diatoms can range from 3 to 500 µm but are characterized by the same diagnostic pigment (fucoxanthin) across this size range.
Similarly, grouping by accessory pigments can be problematic as there is substantial variability in pigment concentration as a function of physiological response to the environmental conditions and more importantly a given biomarker pigment is present in several PT (e. g. , fucoxanthin in diatoms and haptophytes). Some PT, e. g. , coccolithophores, cannot be inferred from HPLC pigments. In consideration of the expanding satellite sensor capabilities, there is a need for coordinated efforts to compile and generate comprehensive in situ datasets (not just HPLC) for assessing phytoplankton composition. There is also a need to provide best practice guidance to merge the different types of datasets (e. g. , HPLC, microscopy, flow cytometry) into an integrated product that encompasses different ways of grouping phytoplankton species.