dataesr/scientific-paragraphs-categorization huggingface dataset:
Selection and Description of the Use Cases Three papers and their experiments  (Jean et al. , 2016; Suel et al. , 2019; Yeh et al. , 2020)  were selected due to (a) their focus on the use of DL on satellite imagery to estimate poverty, (b) the methods used (e. g. , common architecture such as convolutional neural network model-CNN), (c) the fact that all of them were published in high impact journals and consequently might be assumed that they had been subject to a high standard of review, and (d) because all of them have their source code published.
Use Case 1 (Jean et al. , 2016) Jean et al. (  2016 ) is one of the pioneering methods of the use of DL to predict household expenditure and asset wealth in developing countries using satellite images (daylight and nightlight). In that work, they used several villages as their smallest granularity ("clusters") from five African countries (Nigeria, Tanzania, Uganda, Malawi, and Rwanda). Due to the reduced number of labeled samples,  Jean et al. (2016)  used nightlight intensity as a data proxy ("preliminary task") to train DL models using transfer learning. The method used by  Jean et al. (2016)  consisted of five steps and our actions to reproduce them are as follows: 1. Data set. In the original paper, four datasets were identified, two for the wealth index and two for the satellite imagery.
To obtain a poverty indicator, they obtained household expenditure data from the World Bank's Living Standards Measurement Study (LSMS) survey and the Demographic and Health Surveys (DHS). For the satellite imagery (daytime and nightlight satellite images), they used Google Static Maps (GSM) and the National Oceanic and Atmospheric Administration's National Geophysical Data Center (NGDC, 2010; NOAA, 2014). 2. Pre-processing and cross validation. Jean et al. (2016)  used the k-fold cross validation method. In some experiments, k = 5 and k = 10 were used separately. They used two other approaches to train the models: In-Country and Out-of-Country (OOC) where no survey training data is available. The In-country approach showed uniformly better results, but the OOC approach showed good results, often close to the In-country approach.
The main software used was Python 2. 7 with caffe, GDAL (Geospatial Data Abstraction Library) and R 3. 2. 4. From this, a 4096-D output is generated for each image, with 4 images per location (4 different viewpoints of the same location). This 4096-D vector is applied to a custom network that processes each of the four images individually on three different layers, and then aggregates the images from the average. And finally, in the output layer, the sigmoid activation function is used to generate a probability for each of the deciles.
Evaluation. For the evaluation of the model, the Pearson's correlation coefficient, Kendall's tau coefficient, Cohen's kappa, and mean absolute error (MAE), Adjacent Accuracy (with adjacency interval in {±0 (regular accuracy), ±1, ±2}) were used. 5. Implementation and Infrastructure. Suel et al. (2019)  used TensorFlow in Python as main software, although no version information was provided. 1. Data set. In the original paper, four datasets were used, two of which were related to the wealth index (DHS and LSMS) and two of which related to the associated satellite daytime imagery (Landsat and nightlight images). The wealth index was constructed from the first principal component of the DHS responses using principal component analysis (PCA). It included the number of rooms occupied in a house, whether the house has electricity, the quality of flooring in the house, water supply and whether the house has a toilet, ownership of a telephone, radio, television, cars, and motorcycles.
Based on the geolocalization and the date of each village, satellite images were automatically downloaded from Google Earth Engine (GEE). In this study,  Yeh et al. (2020)  used Landsat and nightlight images which centered on each village. A composite of three years was chosen for the studied period  2009-2016 (2009-2011, 2012-2014, and 2015-2017). 2. Pre-processing and cross validation. To cross-validate,  Yeh et al. (2020)  splitted the data into five folds. The aim was to train each model on 3-folds, validate on a fourth, and test on a fifth. To avoid the overlap of satellite images of the villages, they used two configurations: Out-Of-Country and In-Country. For the OOC split, they assigned entire countries to a split. For the In-Country splits, they allowed different clusters within a country to be assigned to different splits.
The Quality of the Data Set A major limitation of the  Jean et al. (2016)  work, as shown in Table  1 criterion C1. 1, was that Google Static Maps imagery is constructed as a mosaic of aerial photographs and it loses the period of composition, so basically it was not possible to obtain the same images. In addition, due to policy restrictions, there is no persistent identifier (criterion C2. 1 from Table  1 ) to the raw data set in their repository (https://github. com/nealjean/ predicting-poverty/). Similarly, the first challenge in reproducing the experiments from  Suel et al. (2019) , as shown in Table 1 criterion C1. 5, is that it is currently not possible to guarantee the availability of the same images through Google Street View, as it is the policy of the service to provide only the most recent image.
This problem was also reported in the work of  Diou et al. (2018). As a result, there is no persistent link to their data set. The README file of the source code repository (https://github. com/esrasuel/measuring-inequalities-sview) lacks some information, such as only some cities cited in the experiment are detailed, making it difficult to compare the results, and the "headers" (entities) in the wealth indices data set are not compatible with the "headers" found in the raw data (linked to the government sites). The main challenge in the work of  Yeh et al. (2020)  is that the original data (DHS surveys) are not made available (criterion C1. 4 from Table  1 ). In addition, the description of some parameters (e. g. , chunk size, the number of images by file) is inadequate (criteria C1. 3 from Table  1 ).
The Description of the DL Architecture and Hyper-Parameter Optimization There was incomplete information about the full set of hyperparameters in the manuscript of  Jean et al. (2016)  or in their supplementary material (criteria C3 from Table  1 ). The code implementing the training of the models is missing, and only the final trained model is provided as a checkpoint. However, the model is downloadable by a non-persistent link which is not a FAIR procedure. In  Yeh et al. (2020) , as shown in criteria C3 from Table  1 , a full description of the DL experiment was missing and their official GitHub repository (https://github. com/sustainlab-group/africa_poverty) was not as updated as an alternative GitHub (https://github. com/chrisyeh96/africa_poverty_clean). In addition, the workflow steps are poorly described and the parameter settings were missing.
Implementation and Infrastructure The infrastructure information was not described in the  Jean et al. (2016)  article, as shown in criterion C4. 1 from Table  1 , but R and Python Caffe scripts were included in the README file in the GitHub repository (https:// github. com/nealjean/predicting-poverty). The main challenge was to reproduce the training procedures presented in their repository. However, the authors link to a third repository (https://github. com/jmather625/predicting-poverty-replication) that shows an alternative version of their work using PyTorch instead, which better reproduces the training pipeline but does not achieve exactly the same performance results. Suel et al. (2019)  referred to using Python and TensorFlow in their paper (criterion C4. 1 from Table  1 ), but the README file of their GitHub repository (https://github.
Mitigation Strategies and FAIR Advice (for User 1 & 2) We suggest a set of mitigation strategies for improving the reproducibility of the data and code associated with DL experiments and provide advice on satisfying FAIR criteria. Note. The first column lists the criteria used to assess each use case. The letters a, b, c after each criterion correspond to the papers from which the criteria were compiled. The answers "Partially" and "No" indicate that these are reproducible aspects that require attention. In this section, we have organized suggestions for the reader according to the general DL workflow (Figure  1 ) and the criteria listed in Table  1 , specifically C1, C3, C4, and C5. We found correspondence between "Quality of the data set" (Section 4. 1) with criteria C1, DL experiment (Section 4.
For a first experiment, one could prepare the same procedure presented from the original article, and test using a sample of the data set. • Check parameters of the datasets or code. Check the experiment parameters and those in the data source used. For example, check the parameters for satellite images acquisition (e. g. , sensor product, start and end date, spectral resolution, spatial resolution, etc. ), the variables and data collections used for census surveys. Also check for the temporary images that were downloaded and were excluded on the original paper. When working with an API (Application Programming Interface), such as GEE or GSM, internet latency or delay may be encountered which can create request errors and some images may be lost even with many trials. At this point, the task of reproduction evolves to a replication task.
Deep Learning Experiment A clear description of the DL experiment is commonly omitted. For example,  Renard et al. (2020)  found that fewer than 10% of the articles reviewed sufficiently described the hyperparameters and the data set to enable the work to be reproduced. The complexity of many DL experiments and repeated iterations resulting in many versions of code makes vigilance essential for accurate reporting. We advise that User 2 checks that their experiments are reported according to criteria C3 and C5 from Table  1. For User 1, here are some recommendations to overcome issues when reproducing of replicating DL experiments: • Look for workflows. For User 1, we suggest careful review of the workflow used to conduct the experiments, meanwhile for User 2, we advise using a workflow to record the process, etc.
We suggest programming a batch list advertising the already downloaded files. • Bugs. It is expected that some bugs will be found in the scripts. Perhaps some source codes were provided that do not match the version mentioned in the manuscript. We suggest looking at the import libraries and checking on blogs such as Stackoverflow (https://stackoverflow. com/) to find the best adjustments. • Versioning. Use the same versions as the original in order to avoid "deprecated" versions. For example, many researchers migrate TensorFlow code from TensorFlow 1. x to TensorFlow 2. Such differences can prevent successful reproduction, and it can be laborious to convert code. • Source code is available but there are many branches. For instance when we have two source code hosted links (e. g. , as described in Section 3.
2. 2), we suggest trying to merge them and compare each step. If the package libraries are missing, set up a computer with empty libraries and then install packages (after aligning them). • Programming language. If the source code is in a particular programming language with which you are not expert, use another trustable version. For instance, if you wish to use Python instead of R, as used in  Jean et al. (2016) , there is a Pytorch implementation https://github. com/jmather625/predicting-poverty-replication.
Advice About FAIR Principles (Particularly Relevant for User 2) The discussion in this section is based on the profiles of User 2 (Figure  1 ) and how they could employ FAIR principles when engaging in DL experiments, which is linked to criteria C2 from Table  1. The following points are called "advice" because they are the minimum criteria that a DL-based project must have. • Some FAIR principles are mandatory in data set design. If the data are not compliant, they cannot be re-used. DL studies typically use large numbers of comparable samples to train and then test models. DL experiments specifically designed to analyze very large datasets are usually based on originally structured data (e. g. , often large sets of images) and are associated with a specific set of attributes defined by the producer(s) of that data (human or machine).
It is almost impossible to achieve a perfect FAIR because of inherent variability in its quality (it could be incomplete or biased). This is not only due to the variability of the community's judgment, but also due to the complexity of the data and the impossibility to fill the data completely and homogeneously (e. g. , changing sensors, process development, artifacts, operating systems, etc. ). These matters will always introduce uncertainties in the outcome of the model between datasets, but also with the same data set but a different user/community user. We advise that User 2 checks that their experiments are reported according to FAIR principles I3 and R1. 2. Based on these arguments, we can easily imagine that increasing "FAIRness" in the DL study necessarily requires an iterative process to consolidate the results, as suggested for all other scientific studies and data  (David et al.
Discussion and Conclusion The requirement for reproducibility and replicability of experiments, let alone those in which new techniques such as DL are employed, is very recent. There are few instances where R & R has been practically tested, and more work is needed to develop best practices and make this a real possibility for the achievement of truly open, defensible science. We have presented an approach that researchers can use to ensure their project is indeed FAIR, that readers can use to check the robustness of a reported method, and that those wishing to reproduce a method can test whether there is sufficient information available to do so. To identify common problems with reproducing reported DL experiments, we analyzed the reproducibility of three case studies which used DL and remote sensing data to estimate poverty.
To do this we amalgamated three published approaches, the checklist organized by the ideal structure of a paper during submission  (Pineau, 2020b) , the DL workflow recommended during experimentation  (Renard et al. , 2020) , and compliance to the FAIR sub-principles for DL  (Hartley & Olsson, 2020). Although the three use cases were proposed for a specific task (poverty estimation), we believe that the evaluation methods could be applied to more general DL tasks, where difficulties might include (a) a lack of data set specificity (and the metadata related with it), (b) inadequate description of the DL experiment, (c) a lack of details of the implementation and the infrastructure used. We also feel that these recommendations can be extended to other domains. To publish a reproducible DL experiment, we recommend that creators (User 1) provide a downloadable version of the data set and simulation environment used, the data management procedure, data and code provenance, and data set versioning.
Suggestions for the design of FAIR computational workflows have recently been made by  Goble et al. (2020) , who propose some specific guidelines for the creation and use of DL models in science, explaining how they relate to the FAIR principles. There have been some recent efforts to establish standards for FAIR ML and DL models  (Pineau, 2020b; Walsh et al. , 2020). Although learning models typically use highly structured data that respect FAIR principles by design, they face greater variability in terms of their reproducibility and, to a lesser extent, their replicability. In addition to the inherent "variabilities" of DL  (Renard et al. , 2020)  that pose a challenge to R & R, there are other aspects of variability that should be better discussed in future work, such as the fact that scientific (meta)data are not only related to the difficulty of reaching consensus on vocabularies, but also to the fact that the variability of the description of these very large datasets is closely related, for example, to their temporal and spatial scale or to the technical variability of their acquisition.
https://www. gov. uk/government/statistics/english-indices-of-deprivation-2015, and Greater London Authority household income estimates https://data. london. gov. uk/dataset/household-income-estimates-small-areas); one data set relates to postcodes (from the Office for National Statistics and Postcode Directory for the United Kingdom), and one data set relates to street view imagery using GSV services. From government statistics, they used 12 indicators to label each image. 2. Pre-processing and cross validation. The Street View data set contains 4 images per postcode, each with a different direction, providing a 360° view of each measured point. For each indicator, they normalized these values and deciles determined, with decile 1 corresponding to the worst-off 10% of LSOAs, and decile 10 corresponding to the best 10% in London.
Acknowledgments This project was conducted as part of the  Belmont Forum   PARSEC  project, funded under the  Collaborative Research Action (CRA) on Science-Driven e-Infrastructures Innovation (SEI) , with funding from  FAPESP , the  ANR, JST  and the  NSF , with collaborators from  Australia , and support from the synthesis centre CESAB of the  French Foundation for Research on Biodiversity. In Brazil  the  PARSEC  project is also supported by the Grant  2018/24017-3 ,  São Paulo Research Foundation (FAPESP). J. M. is grateful for the support from  FAPESP  (Grant  2020/03514-9 ). R. David was supported by the  EOSC-Life European program  (grant agreement No. 824087 ).
Data Availability Statement The source code availability used for reproducibility can be found as follows: •  Jean et al. (2016) , https://github. com/nealjean/predicting-poverty/ • Suel et al. (2019) , https://github. com/esrasuel/measuring-inequalities-sview • Yeh et al. (2020) , https://github. com/sustainlab-group/africa_poverty and https://github. com/chrisyeh96/ africa_poverty_clean.
Case 2 (Suel et al. , 2019) Suel et al. (2019) used images taken at ground level instead of satellite images. They developed a DL model using Google Street View (GSV) images to extract features that could predict poverty statistics such as income, educa- tion, unemployment, housing, living environment, health, and crime. Their experiments focused on different regions of London and other cities in England. Suel et al. (2019)vious use case, the method used bySuel et al. (2019)was also aligned with Figure2flowchart, consisting of the following five steps (see below): 1. Data set. In the original paper, Suel et al. (2019) identified five datasets, three of which relate to the wealth indices, which are government statistics for the total population at a fine scale of Lower Layer Super Output Area (LSOA) from three sources (Census https://www.
ANNs are made of processing units which can discover and learn complex and meaningful patterns in a dataset through a trial-and-error procedure -in the case of supervised learning. Deep learning is a subclass of these machine learning techniques, named from the presence of many processing unitsalso called layers -in series and in parallel in the artificial neural network  [43]. In a broad sense, deep learning techniques have been exploited for many purposes such as speech recognition, language processing, autonomous-driving, medical diagnosis as well as in materials science  [44] [45] [46] [47] [48]. Convolutional neural networks (CNNs) are a type of deep learning algorithm that takes images as input and relates the features of the image to a property of interest. In materials science, they have been used for predicting the thermal, mechanical and hydraulic properties, design and topology optimization of material systems  [49] [50] [51] [52] [53] [54] [55] [56].
Elementary volume generation Training and testing the suitability of a CNN for predicting the permeability requires generating many elementary volumes (EVs) that will form a large dataset with good coverage in terms of v f , fiber radius r, fiber orientation and spatial distribution patterns. As discussed by Rimmel et al. [65] , images containing on the order of hundreds of fibers where each fiber's r is represented by approximately 10 pixels result in acceptable permeability values when the permeability of these images is estimated via solvers working directly on the images (i. e. , solvers that avoid using mesh based solvers thus bypassing a meshing procedure), as is our case. More specifically, the authors compared three cases with 262, 484 and 799 fibers and concluded that the case with lowest fiber count was more prone to probability of individual images whose permeability is estimated to be 0 due to locally blocked structures.
e. , on 2D slices) to train the CNN as well as on full EVs (which consisted of 800 slices) to validate our approach based on combining both the circuit analogy as an upscaling technique which is well suited for the considered fibrous media and CNN predictions. We opted for the Geodict software (Math2Market® GmbH) and its FlowDict module to perform our fiber scale simulations, as it can perform the simulations directly on the binarized images thus eliminating the need for mesh generation and as it can be automated via Python scripting to repeat the simulations on thousands of 2D binary images. We used an Explicit Jump-Stokes solver to solve the governing Stokes flow equations (i. e. , at negligible or zero-valued Reynolds number) as reported in earlier work for permeability characterization  [22, 65, 68]  with boundary conditions visually outlined in Fig.
CNN architecture We implemented a modified version of AlexNet  [69]  architecture that is derived from LeNet network and paved the way for the most recent architectures which are based on the similar filter features. It provides fairly good results in both complex classification and regression tasks  [70, 71]  and is available in Matlab's Deep Learning toolbox. The design of the CNN is outlined in Fig. 4a. CNN takes a matrix with a size of 400 × 400 (i. e. , an image with 400 × 400 pixels) and outputs the permeability, thus the network is set up for a regression task where the input is an image and the output is a numeric value. As mentioned before, we studied both axial and transverse permeability values and we used the same network architecture after training it with the desired data type.
In general, the predictions seem to be well-aligned with the simulation results for both directions. Even though the scatter is quite limited for the y-direction, predictions for x-direction deviate from the simulation results at low permeability values (roughly for permeability values lower than 1 10 -13 m 2 typically obtained when v f was equal to or higher than 0. 65). Coincidentally, Geodict tends to return 0 for transverse permeability of high v f images and we excluded these data points from the training and testing of the neural network. This practice obviously results in an imbalance within the dataset. The so-called class imbalance is a common issue, and refers to the high representation of one of the classes in the data and the resulting bias in the learning process in favor of the dominant class  [76, 77].