dataesr/scientific-paragraphs-categorization huggingface dataset:
COVID-19 metrics, indicators, and maps Selecting the most appropriate statistical-spatial variables supposes at first to get insight into the French statistical system developed under the authority of Santé Publique France (France Public Health) and the French Ministry of Health and Solidarity  [26]. Although WHO data collection about COVID-19 started January 11  [27] , France begins to develop its own statistical database on March 3, e. g. , 14 days before the general lockdown is declared (March 17). The observation system is not immediately operational and gains momentum through a stepwise process. The number of core indicators is very limited and based almost exclusively on data gathered through hospital units or associated services (SOS médecins, biomedical analysis laboratories). Public communication focuses on the number of reported hospital admissions, admissions in intensive care units, and deaths, through the form of an official dashboard associating raw figures, graphs, and maps.
Core information is provided by the ARS (Regional Health Agencies) network based on data from local hospital units and health services. Public dissemination is ensured at the department and region levels, thus reflecting the administrative structure of the ARS  [28]. These are agencies created in April 2010 following an important reform of the health care system (PNR, National Reform Program, Law HPST (Hospital, Patients, Health, Territories) dated July 21, 2009). French regional divisions derive from Law NOTRe (New Territorial Organization of the Republic) of August 7, 2015, which granted enhanced competencies to regions encompassing a larger number of departments. The COVID-19 management system derives from these earlier reforms. A "vigilance map" developed by the authorities and color-coded green-orange-red defines alert levels and subsequent lockdown enforcement measures  [26]  (Fig.
Assessing the quality of official statistics These databases call for various comments about potential uses, limitations, and significance, all the more so as data collection procedures are not detailed by the authorities. This implies for us to evaluate the relevance and usefulness of statistical information ex-post: (1) In the absence of any information at a sub-departmental level, many geographical analyses are simply impossible to carry out. Unlike other countries, there is for instance no way to address urban vs. rural discrepancies as regards the virus spread in France  [29]. More refined data exist, but they are not released publicly. Public information mainly comes from the network of hospital units belonging to the official list of frontline medical centers, or from the list of biomedical city labs.
Such a limitation is even starker for intra-urban analysis. (2)  This fundamental lack of data on the issue produces a side effect. The official geography of COVID-19 is entirely driven by administrative vector features. The coronavirus seemingly shows respect for departmental and regional boundaries whose demarcations constrain dissemination patterns. The vigilance map elaborated by Santé publique France is a primary example of how the government and public administrations have the upper hand on the epidemic  [30]. The switch from one color-coded classification to the next one is most often done at a regional level, which means that the epidemic proceeds bureaucratically. (3) Surprisingly, community/city medicine is evacuated from the COVID-19 observation system. The only exception is database 2, replaced by database 3 after May 13 (Fig.
1 ). Everything seems to happen inside the walls of a limited number of hospitals. Database 4 is the only one relating sick population with hospital admissions. The consequences of this situation are manifold and profound. The coronavirus is apprehended almost exclusively through the lens of critical pathological conditions requiring technical targeted therapy that cannot be achieved outside specialized hospital units. Asymptomatic people, infected individuals on home care -be they tested or not-, and low-risk populations remain hidden in the background. (  4 ) EHPAD (residential care facilities for dependent elderly persons) and EMS (medical-social healthcare institutions) counting a large number of casualties have been left aside up to April 1 because of their specific position within the national health care system.
(5) To better understand what is at stake with these databases, it should be noted that the authorities take great care to avoid the term prevalence. Santé publique France produces two important indicators: (a) the incidence rate, otherwise called epidemic activity, is defined as the number of positive tests per 100. 000 inhabitants over one week; (b) the RT-PCR test positivity rate is defined as the number of positive tests as a percentage of the number of tests conducted over one week. Prevalence defined as the number of infected people within the general population is never calculated. Given that a very small number of tests have been conducted in France, prevalence as the only concept able to assess the global spread of the virus remains out of reach. In other words, the tested sample is so small that it should never be considered representative both in statistical and geographical terms.
We have observed repeated confusion as regards these concepts, including in international databases  [31] [32] [33]. (6) To sum up, just like in other countries where integrated care pathways have been developed, the entire observation system is based on monitoring the epidemic through the management of patients and among them the ones suffering from the worst COVID-19 forms. Figure  2  shows how much this process is narrow in scope and reductionist in nature both in statistical and spatial terms.
Addressing COVID-19 statistically Given these constraints on statistical data, our attention will focus on three databases: The system is still operational today. For a full description of the above, see the corresponding web pages of Santé publique France (see section Availability of data and materials). This flow chart should be read from top to bottom. On the left side and in green are the exits from care pathways. On the right side and in orange are to be found the visible individuals. The blue color introduces the geographical variable and also defines the interface between civil society and health care system. There is no information before testing. The authorities do not provide any further details about community medicine. The lower part of the chart depicts the procedure and arrangements for patient care.
Some hospital units involved in the COVID-19 system at different degrees remain out of sight. Despite the number of cases observed, EHPAD and EMS are marginally present These datasets were downloaded from the governmental online platform on June 30 2020. Most primary indicators show drawbacks implying different ex-post rectifications. These are conducted by Santé publique France. By definition, registered patients were tested positive, which motivated their inscription into the databases. The actual cause of death reported in database 1 is a much-disputed issue because of potentially associated comorbidities. This has given rise to the controversy about deaths "reported with COVID-19" or "caused by COVID-19". The idea is to mirror hospital metrics with the closest approximation to a prevalence indicator, e.
g. , test results. The advantage of database 2 is double: it allows focusing on community medicine, with an all-embracing dimension on civil society, and it provides the best proxy for general prevalence. Unfortunately, this database is suppressed on May 26. Timing is determined both by data availability and the chronology of events. Figure  3  provides an overview of the COVID-19 legal management process. The analysis is completed once the most important movement restraints are lifted at the end of June with schools reopening and, simultaneously, the beginning of summer holidays and people regaining mobility. All cards are then reshuffled. July 10 might be seen as a symbolic-legal date. The March 10-May 26 period (database 2) is thus crucial because it covers the worst phase of the epidemic.
At both ends of the curve daily deaths in hospitals are about one hundred. We intend to map the epidemic at a sub-national level. The scale adopted must always be as accurate as possible (department). The solution to be favored will be to map the closest approximation to a prevalence indicator (tests and positive tests), voluntarily leaving aside the management of patients within hospital units. For an enhanced clarity and comparability of results, all indicators will. Besides, we assume that composite, standardized indicators are not the most appropriate variables to consider with databases that are so degraded. For that reason, we will use raw isolated data within which outliers are readily identifiable. The idea is to avoid adding mathematical artifacts and blurring information through the development of complex indices.
This inferential statistic takes as a reference the null hypothesis (randomly distributed attributes) and computes expected and observed index values. From this, the algorithm derives a z-score and p-value for each chosen variable. If the p-value is statistically significant, the null hypothesis is rejected. A positive z-score means that the spatial distribution is more clustered than a random distribution. With a negative z-score, the distribution is more spatially dispersed than would be expected. These operations are intended to assess the overall pattern and trend of our data. (b) Thematic mapping Secondly, we produced a series of thematic maps illustrating spatial dissemination patterns for the two periods under consideration (lockdown and lockdown lifting). Processed data are those related to cumulated daily new cases, not overall daily cases.
This was a natural choice, given that each period has its own databases. To understand the problem more clearly, it should be added that test-related data (purple and orange) are globally underestimated (Fig. 5  and 6 ). The number of identified positive people is too small to be representative of the population at large and of the overall contamination rate. In France, during the period March 10-May 26, 423 000 tests were conducted, excluding the biological tests carried out in hospitals. Considering that the French population reaches 67 063 000 inhabitants, the overall testing rate over the peak crisis phase is only 6. 3‰, with wide interdepartmental variations. Caution should be exercised with these figures since false negative tests might achieve a rate of 10 to 40% depending on the adopted RT-PCR sampling method  [34].
c) Data clustering Given the complexity of analytical result findings, we decided in the final phase to draw up a synthesis of relevant information through the form of data clustering  [35]. This venture might be seen as a second assessment of the coronavirus crisis during its most critical phase. In the absence of sufficient data consistency and because of a degraded significance of results, we stated that we would not map the post-lockdown situation as described by the SI-DEP database. Based on most reliable outcomes, we selected two clustering methods: (1) Hot Spot analysis (Getis-Ord Gi*) addresses spatial variability through the calculation of z-scores and p-values for each department feature  [36]. The resulting data allow for the determination of cold and hot spots whose characteristics are determined by constraint parameters.
Most often (black), the overall number of positive tests conducted in city labs is less than the number of hospital admissions (hospitalizations > positive tests > deaths). The position becomes even more curious (red) with positive tests below the level of COVID-19 related deaths (hospitalizations > deaths > positive tests). This situation illustrates the varying reliability of the official database during the emerging phase of the statistical system dedicated to COVID-19, both through time and space. There are several possible reasons for this. (1) The departments registering a small number of COVID-19 cases took more time to implement the statistical platform; (2) Frontline hospitals were already conducting their own tests but these are not reported in the city labs database, which interferes with the results; (3) The reliability and homogeneity of RT-PCR testing procedures might explain certain local discrepancies; (4) The number of reporting public hospital or clinics and city labs has fluctuated over time exclusively on ARS databases.
Mapping dissemination during the lockdown lifting period Figure  7  illustrates the post-lockdown situation up to the beginning of the summer break. The creation of the SI-DEP database does not allow anymore differentiation between community and hospital testing centers, hence a strong correlation between (A), (B), (C), and (D). This new strategy for statistical data management necessarily strengthens the power of the hospital system. Information is globally blurred and less significant. The geographical pattern identified before seems to settle down. The number of tests increases, positive results fall as well as the overall number of hospital admissions and deaths. All indicators turn green, although various warning signs start to emerge suggesting that the epidemic continues to spread silently.
Discussion We observed significant discrepancies between the official prospects on the virus spread in France and our results. Many infected sites remained unseen despite the existence of dedicated observatories  [39]. This paradox leads us to believe that further investigations would greatly benefit from accurate ancillary databases focusing on community medicine, provided that access rights be granted in the future  [40]. Despite an impressive number of academic publications, the geography of SARS-CoV-2 is still in the early stages of research development. This work offers a distinct advantage: the replicability of the method is allowed by the availability of the algorithms used. Given the uncertain reliability of data sources, we chose not to revert to complex analysis techniques that would have "compensated" for obvious defects of the official databases.
As a result, statistical production interferes with the real geography of hot spots (border effects). This problem might be solved by an enhanced accuracy of spatial databases and/or through a more systematic use of mapping solutions based on kriging and spatial interpolation techniques. 2-Official COVID-19 databases report on the status of the health care system rather than improve our insight into the coronavirus itself, both in demographic or spatial terms. A systematic watch system is still to be invented. These ambiguities appeared clearly during data quality analysis, and we assume that this probably caused further interpretation problems, which in turn led to distorted decisions as regards the management of the epidemic by the authorities. Many narratives based on incidence or prevalence rates miss the point, avoiding to mention that a significant proportion of the population may carry the pathogen without symptoms, with or without transmission capacity, and that a good portion of infected people chose to remain away from health services.
We found that a number of classical methods in the area of epidemiology are fairly inappropriate in addressing spatial dynamics. For instance, standardization and normalization are known useful techniques widely used in health sciences. However, these might produce reverse effects when applied to datasets elaborated by territorial administrations whose purpose is precisely to publish normalized metrics -with varying degrees of success. The extraordinary complexity of living beings, societies, and environments do not easily comply with administrative requirements. Simulation is apparently a panacea for advanced epidemiological studies, and this is sustained by strong political demand, but these approaches have shown their limits in terms of predictability. This is why cluster modeling might benefit from the advances achieved by complex sciences and chaos systems, which give a prominent place to statistical/spatial outliers, self-reproductive and non-linear mechanisms.
Availability of data and materials The datasets supporting the conclusions of this article are freely available online. -Hospital database related to the COVID-19 epidemic, http://data. gouv. fr/en/ datasets/donnees-hospitalieres-relatives-a-lepidemie-de-covid-19/. -Database related to COVID-19 tests conducted in city, http://data. gouv. fr/en/ datasets/donnees-relatives-aux-tests-de-depistage-de-covid-19-realises-enlaboratoire-de-ville/. -Database related to the results of COVID-19 virological tests SI-DEP, http://data. gouv. fr/en/datasets/ donnees-relatives-aux-resultats-des-tests-virologiques-covid-19/.
Sloan Digital Sky Survey The Sloan Digital Sky Survey (SDSS) is a wide-field imaging and spectroscopic survey that covers roughly one-third of the celestial sphere. The first dataset in our work consists of stamp images of 496 524 galaxies and associated spectroscopically determined redshifts selected by  Pasquet et al. (2019)  from SDSS Data Release 12  (Alam et al. 2015). Each stamp image covers one of the five photometric passbands (ugriz) and has 64 × 64 pixels in spatial dimensions, corresponding to a pixel scale of 0. 396 arcsec. A galaxy is aligned to the center of each image using its angular coordinates. The pair of five images in the five passbands and the associated spectroscopic redshift forms a data instance. The dataset covers z < 0. 4 in spectroscopic redshift and r < 17. 8 in dereddened r-band petrosian magnitude.
Due to the good data quality and the relatively restricted parameter space (i. e. , low redshifts and high brightnesses), the SDSS dataset was the main source of data for developing and evaluating our bias correction methods. Similar to  Pasquet et al. (2019) , we randomly selected 393219 instances as a training sample and 103305 instances as a test sample. More detailed descriptions can be found in  Pasquet et al. (2019)  and  Alam et al. (2015).
2007). From these redshift surveys, we only took the most secure spectroscopic redshifts that are identified with high signal-to-noise ratios (S/Ns) or were measured with multiple spectral features (equivalent to the VIPERS or VVDS redshift flags, i. e. , 3 ≤ zFlag ≤ 4). We refer to them as high-quality (HQ) redshifts. We also included the most secure low-resolution prism redshift measurements for bright sources (zFlag = 4, I ≤ 22. 5) from the PRIsm MUlti-object Survey (PRIMUS; Data Release 1;  Coil et al. 2011; Cool et al. 2013) , and the secure grism redshift measurements from the 3D-HST survey (Data Release v4. 1. 5; Skelton et al. 2014; Momcheva et al. 2016). They are referred to as low-resolution (LR) spectroscopic redshifts. We selected 26 249 HQ and 6007 LR galaxies from the CFHTLS Deep fields and 108 510 HQ and 17 427 LR galaxies from the Wide fields as two separate datasets (denoted "CFHTLS-DEEP" and "CFHTLS-WIDE", respectively) with spectroscopic redshifts ranging up to z ∼ 4.
0 and dereddened r-band magnitudes in a range of 12. 3 < r < 27. 2. Like the SDSS dataset, the extracted stamp images are in the format of 64 × 64 × 5 pixels, corresponding to a pixel scale of 0. 187 arcsec  (described in detail in Treyer et al. , in prep. ). We randomly selected 10 000 instances from the Deep fields and 10 000 instances from the Wide fields with high-quality spectroscopic redshifts as separate test samples. They were not mixed because the Deep and Wide fields have potential systematics. The remaining 22 256 instances from the Deep fields and 115 937 instances from the Wide fields were regarded as training samples. They were used separately in training when bias correction was applied. A36, page 3 of 28 A&A 662, A36 (2022).
Although we discuss this procedure only in the setting of photometric redshift estimation, we note that it is potentially applicable to any study that involves data-driven models. As we discuss in Sect. 3. 2. 2, correcting z spec -dependent residuals requires a selection of data from the original training dataset. It is impractical to train a deep network using only a selected small subset of data. However, as illustrated by  Kang et al. (2020) , the learning procedure of a model can be split into representation learning and classification. In this idea, the whole network is first trained with all training data to obtain a representation at a chosen hidden layer, which is usually a lowdimensional embedding of information from input data (see, e. g. ,  Bengio et al. (2013)  for a review of representation learning); then the classification output can be fine-tuned or retrained with the representation fixed.
3. 2. 2. Step 2: To correct for z spec -dependent residuals induced by overpopulated classes The z spec -dependent residuals are strongly affected by overpopulated classes. We note that simply increasing the rate of occurrences of the instances from other classes in training cannot effectively damp the dominance of overpopulated classes, presumably because this does not balance the information content in different classes. Therefore, we attempted to correct the influence of overpopulated classes by constructing a nearly balanced training subset via undersampling. Specifically, in each z spec -magnitude cell (z spec , r i ) in the division of the input space, a number of instances from the training dataset are randomly selected no more than a predefined balancing threshold th; all the cells with number densities above the threshold are thus considered overpopulated and trimmed.
Training protocol Throughout this work, we train CNN models and apply our bias correction methods for each dataset (SDSS, CFHTLS-WIDE, Notes. ( * ) Reduced to 2 × 10 -5 after 60 000 iterations for the baseline and step 1. and CFHTLS-DEEP) using the training data. Unless otherwise noted, we show results only with the test data, assuming that the training and test samples both follow the same sampling process in each redshift-magnitude cell (z spec , r i ) in the input space. We only allow this assumption to be violated in the stage of representation learning where the goal is to expand the information volume learned by the model despite the presence of distinct systematics or sub-structures in the training data. However, the existence of such systematics would jeopardize the balancing of the data distribution, thus the training data must be purified for the later stage of bias correction.
For example, the CFHTLS-DEEP data generally have better S/Ns than the CFHTLS-WIDE data because their exposure time is longer. Because the model may not be well generalizable over different noise properties, we considered separating these two CFHTLS datasets for bias correction. In our work, the training sample used for bias correction and the test sample were randomly selected from the same parent distribution for each given dataset, although their overall distributions were allowed to be different if they followed the same sampling in each redshift-magnitude cell. The data coverage and the hyperparameters used in training for each dataset are detailed in Tables  1  and 2. To train our models, we adopted the mini-batch gradient descent technique with the default Adam optimizer  (Kingma & Ba 2015).