dataesr/scientific-paragraphs-categorization huggingface dataset:
Data Availability Statement The corresponding ESMValTool recipe that can be used to reproduce the figures of this paper is included in ESMValTool v2. 1  (Eyring et al. , 2020; Lauer et al. , 2020; Righi et al. , 2020). ESMValTool v2. 1 is released under the Apache License, VERSION 2. 0. The latest release of the ESMValTool is publicly available on Zenodo at https://doi. org/10. 5281/zenodo. 3401363. The source code of the ESMValCore package, which is installed as a dependency of the ESMValTool, is also publicly available on Zenodo at https://doi. org/.
To produce these more complex annotations, specific annotation tools are usually required (e. g. , CocoAnnotator, https:// annotator. justinbrooks. ca; LabelMe, http://labelme. csail. mit. edu/Release3. 0; ImageTagger, https://github. com/bit-bots/ imagetagger). Regardless of the type of phenological annotation being created, it is important to recognize that ML is a form of statistical inference that finds patterns in the training data, and predictions will therefore reflect any biases in those data. Therefore, the training data set should include a representative sample of taxon (or taxa) and phenological attributes or categories that should appear in the data set on which the trained model will be applied. A lack of visual diversity in the manual annotations will result in the same lack of diversity in the predictions.
The software frameworks most commonly used to develop CNNs are Pytorch (https://pytorch. org), TensorFlow (www. tensorflow. org), CAFFE (https://caffe. berkeleyvision. org), and MXNET (https://mxnet. incubator. apache. org). The framework that is best for a given task will depend on a variety of factors, including the availability of preexisting models or code for the targeted task, the hardware used, or simply the data scientist's skills with respect to a particular framework. Deployment. Once an ML model is trained, it can be deployed to predict phenological annotations for previously unannotated specimens. This process could be implemented in several ways, including via a web service available to other applications that returns the model output for any submitted image, a standalone program, or an end-user graphical user interface (GUI) web application.
Inference is usually a less computationally intensive task than training and therefore requires fewer resources. Models can be publicly shared through software repositories hosted on sites such as GitHub (https://github. com), GitLab (https://gitlab. com), or dedicated platforms such as Model Zoo (https://modelzoo. co). Such sharing encourages cross-model improvement based on transfer learning  (Bengio 2012)  or domain adaptation techniques  (Long et al. 2016). Testing and analysis. Once deployed, a model can be evaluated through qualitative judgments or quantitative measurements. Qualitative judgments are typically performed through a GUI, allowing the researcher to visualize and assess the automated predictions of the model on this test data set. For more robust measures of model accuracy, quantitative measurements of the quality of the predictions can be readily calculated and visualized (see  Lorieul et al.
The development of standard vocabularies, metadata protocols, and data structures within the scaffold of existing biodiversity data standards (i. e. , Darwin Core; see  Yost et al. 2018 ) would greatly advance this aim. When designing ML-based phenological studies, researchers should consider how to integrate output data and metadata with existing databases or data sets, ideally using terms and relationships from the Plant Phenology Ontology  (Stucky et al. 2018 , Brenskelle et al. 2019). A relatively small investment early in the research process can ensure that ML-generated, "extended" data (sensu  Lendemer et al. 2019 ) can be productively leveraged by other researchers and owners of the original data (i. e. , natural history collections), thereby increase the value of the specimens. This process would benefit from further discussion across biological and computer science disciplines, for example, within the framework of the Research Data Alliance (www.
Acknowledgments The authors thank the organizers of the iDigBio Phenology Deep Learning Workshop for bringing us together on this topic. We also thank our collaborators  Renato Figueiredo ,  Nicky Nicolson ,  Isaac Park ,  Deb Paul ,  Katja Seltmann ,  Annika Smith , and  Nadya Williams  for their ideas and participation in this workshop. This research was supported by iDigBio, which is funded by a grant from the  National Science Foundation's Advancing Digitization of Biodiversity Collections Program  (award no. DBI-1547229 ).
While allowing the reconstruction of high-resolution 3D scenes, SBL only provides one reflectivity value (corresponding to the wavelength of the laser source) per surface and thus does not provide spectral information about the scene of interest. When such information is also required, data fusion strategies can be adopted, as in  [10] ,  [11]  for instance. Multispectral Lidar (MSL) overcomes data registration and fusion problems, as spectral and spatial features are collected using a single imaging system  [12]. This emerging modality has already led to promising results for underwater imaging  [13] , object detection in urban area  [14]  and forest canopy monitoring  [15]. Using laser sources at different wavelengths, histograms associated with different spectral bands are usually recorded for each pixel of the scenes, using a single device.
1 ). Using a single waveform per pixel, the acquisition time is of the same order as when using SBL in the photon-starved regime. As discussed in  [21] , the distribution of the photons ToA is shifted in time Fig. 1. Examples of acquisition of four MSL waveforms (left), using one wavelength per waveform, and SW-MSL (right) capturing simultaneously information from four wavelengths. In this example, the vegetated scene is mostly green and the Lidar return associated with this wavelength range (second peak from the left) is larger than the other returns. with a wavelength-dependent temporal delay. Although these delays are fixed, the amplitude of the different peaks varies depending on the spectral signatures of the scene, which changes the waveform shape. The study of SW-MSL data reduces to estimating a 3D profile and to quantifying the proportion of detected photons associated with each wavelength.
We then assign prior distributions to the unknown model parameters. To overcome convergence issues (multimodal posterior distribution, slow convergence of the MCMC method in  [21] ) induced by the joint estimation of the depth and spectral profile, we estimate the parameters sequentially. More precisely, we use an algorithm based on Expectation-Maximization (EM) to first estimate the mixture weights in each pixel. These weights are then used to estimate spectral signature in each pixels and the range profile. The main contributions of the paper are: r A new SW-MSL observation model whose formulation is well suited for inference using EM-based algorithms. r An EM-based algorithm for analysis of SW-MSL data which extends the model proposed in  [30]  for SBL data. r In contrast to the method proposed in  [30] , we provide an additional step to estimate reflectivity parameters from the estimated ratios of informative photons detected.
r A comparison of several prior models for the unknown spectral profile in terms of reconstruction performance and associated computational complexity. This study demonstrates that the proposed strategy yields faster reconstruction than the method in  [21]. The remainder of this paper is organized as follows. Section II introduces the new SW-MSL observation model and the different prior distributions associated with the unknown model parameters are discussed in Section III. Section IV describes the estimation strategy based on EM. A comparison of the different prior models using synthetic SBL data is conducted in Section V in order to identify the most interesting options in terms of estimation performance and computational cost. This study is then extended in the remainder of Section V, which investigates real SW-MSL data analysis and comparison with the method proposed in  [21].
To control the quality of the generated data, we introduce as in  [30] , two parameters (α and γ) controlling the mean signal detection and the background levels. More precisely, datasets have been generated using Eq. (  2 ), such that the average number of signal and background photons in the nth pixel are αr n G 1 and b n = αγT , respectively. The parameter α is chosen in {25, 100} and γ takes values in {0. 01, 0. 05, 0. 1, 0. 3, 0. 5, 1, 5}/T. This leads to average counts per pixel from 10. 3 (for (α, γ) = (25, 0. 01/T )) to 542. 7 (for (α, γ) = (100, 5/T )). In the latter case, about 500 of these photons arise from the background. With this parametrization, and average signal to background ratio (SBR) SBR = 1 N N n=1 1 T b n L l=1 r n,l G l , can be expressed as SBR = 1 NT γ N n=1 L l=1 r n,l G l , and it ranges from 0.
B. Analysis of Real SW-MSL Data Here, we compare our approach to that developed in  [21] , using the real SW-MSL data acquired in that work, composed of L = 4 wavelengths (473, 532, 589 and 640 nm), whose IRFs are shown in Fig. 2  (see  [21]  for additional details about the imaging setup). While perfect ground truth reflectivity and range profiles are not available, we use as reference profiles the results obtained with the method developed in  [21] , applied to data acquired with a long illumination time and a negligible background contribution. We consider datasets with 1. 1, 5. 7, 11. 4 and 114. 3 signal photons per pixel on average. Two scenarios are considered. One set of data was acquired with negligible background illumination while a second data set was acquired with spatially varying background (see  [21] ), leading to an average SBR of 1.
VI. DISCUSSION We proposed a new method for analysis of SW-MSL data which can be applied for any number of spectral bands L. However, its computational cost grows with the number of unknown parameters. The main bottlenecks are the constraints imposed by S N L , which require a certain class of optimization schemes to be used in the maximization step. If a second-order method is used, the inversion of the Hessian matrix can also become computationally intensive for large values of L. However, we believe that practical SW-MSL systems will only consist of up to a dozen of spectral bands, for which our algorithm remains computationally attractive. Indeed, increasing the number of bands (either by increasing the spectral range or the spectral resolution with a fixed range) requires the consideration of longer histograms to ensure all the L peaks can be resolved, i.
VII. CONCLUSION In this paper, we developed a novel algorithm for spectral and depth profile estimation from SW-MSL waveforms, in the presence of non negligible background and reduced acquisition time (down to 1 signal photon per pixel, on average). The reformulation of the model into a mixture of illumination sources and ambient illumination allowed the use of an EM-based algorithm for sequential parameter estimation, allowing in turn a significant speed-up compared to the existing method for SW-MSL data analysis, without significant degradation of the reconstruction performance. We compared several reflectivity models and demonstrated the benefits of the C-Dirichlet model in terms of quality of estimation, level of user supervision and computational cost. While our new approach benefits from a significantly reduced computational cost, these new results are still incompatible with real-time requirements.
This work was supported in part by the  Royal Academy of Engineering  under the  Research Fellowship  scheme  RF201617/16/31  and by the  Engineering and Physical Sciences Research Council (EPSRC)  under Grant  EP/S000631/1 , in part by the  MOD University Defence Research Collaboration (UDRC)  in Signal Processing, and in part by the  IDEX of University Grenoble Alpes. The associate editor coordinating the review of this manuscript and approving it for.
1. Remote sensing of forested landscapes can transform the speed, scale, and cost of forest research. The delineation of individual trees in remote sensing images is an essential task in forest analysis. Here we introduce a new Python package, DeepForest, that detects individual trees in high resolution RGB imagery using deep learning. 2. While deep learning has proven highly effective in a range of computer vision tasks, it requires large amounts of training data that are typically difficult to obtain in ecological studies. DeepForest overcomes this limitation by including a model pre-trained on over 30 million algorithmically generated crowns from 22 forests and fine-tuned using 10,000 hand-labeled crowns from 6 forests. 3. The package supports the application of this general model to new data, fine tuning the model to new datasets with user labeled crowns, training new models,.
DeepForest Software DeepForest is an open source (MIT license) Python package supporting Python 3. 6 and Python 3. 7 and has been tested on Windows, macOS, and Linux operating systems. It can be installed using the Python Package Index (https://pypi. org/project/deepforest/) or using the conda package manager for Windows, Linux and OSX (https://github. com/conda-forge/deepforest-feedstock). The software is openly developed on GitHub (https://github. com/weecology/DeepForest) with automated testing and each release is archived on Zenodo.
In contrast, fine-tuning the prebuilt model with as few as 1000 hand labeled trees can provide significant improvement and be accomplished in approximately 8-10 hours  (Weinstein et al. 2020). This can be done using either image labeling tools (e. g. , RectLabel) or GIS software (e. g. , ArcGIS, QGIS) and DeepForest includes helper functions to convert common formats (XML and shapefiles) into a csv format. Annotations can be made on images of any size, but training the model requires images with fixed standard dimensions. The prebuilt model was trained on square crops of length 400px (40m at 0. 1m resolution), w f. CC-BY-ND 4. 0 International license available under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
train). Training deep learning models requires a number of parameter choices such as batch size and number of epochs. For users less familiar with training deep learning models, DeepForest comes with a standard configuration file with reasonable defaults. While some parameter exploration will always be helpful, our aim is to make these innovations available even to novice users. Optional GPU support and model customization allow more experienced users to quickly develop and test larger and more complex models. Data augmentation to randomly crop and flip training images is also supported. This strategy is often useful to reduce overfitting when training on small datasets (Zoph et al. 2019) but has not been extensively tested for tree crown delineation. For additional recommendations for optimal model training see the online documentation (https://deepforest.
Case Studies National Ecological Observatory Network To evaluate model performance across a range of forest types, we used data from the National Ecological Observatory Network to predict crowns in sites across the United States. This dataset consists of 212 images containing 5852 trees from 22 sites that is part of an upcoming tree crown benchmark data package  (Weinstein et al. 2020). Training and evaluation data are separated by at least 1 km when they occur at the same site. Evaluation data were created by viewing RGB images and manually delineating tree crown boxes for all visible trees. Annotations were cross-referenced with field collected positions of tree stems (from the NEON Vegetation Structure dataset; NEON ID: DP1. 10098. 001) within each plot when available. Following  Weinstein et al.
French Guiana Tropical Forest The DeepForest prebuilt model was trained on data from the United States that was collected using fixed-winged aircraft at 10cm resolution and provided as 1km 2 orthomosaics. Therefore, two key questions are: 1) Does this model generalize to To address these questions, we used data from a recently published competition comparing LIDAR tree segmentation algorithms using remote sensing from French Guiana (Aubry-Kentz et al. 2019). In the original competition, each team was sent unlabeled data to predict and the evaluation data was kept private. This process was repeated for this paper, with the third author (Aubry-Kientz) running evaluation scores for the DeepForest predictions made by the corresponding author  (Weinstein). Predictions were run on a Mac laptop with a 3.
Portland Street Trees DeepForest's use of widely available RGB data provides the potential for it to be used across very large spatial extents. Scaling up is challenging because algorithms need to handle large ranges of habitat types and because the resolution of the data available over large areas is typically coarser. To explore how DeepForest performs using coarser resolution data in unique habitats, we applied both the prebuilt model and a retrained model to crown delineation of street trees in an urban environment. The locations of urban trees are important for ecological, sociological and public infrastructure applications. In addition, the urban environment is very different from the natural environments on which the prebuilt model was trained. The image data from the Oregon Statewide Imagery Program is also coarser at 0.
3m spatial resolution (1ft), a resolution that is widely available as part of the National Agriculture Imagery Program (NAIP -https://www. fsa. usda. gov/programs-and-services/aerial-photography/imageryprograms/naip-imagery/). We used imagery from the Portland metro area that overlapped with the Portland Street Trees dataset (http://gis-pdx. opendata. arcgis. com/datasets/street-trees). The street trees dataset contains geospatial information for the majority of trees accessible from public roads in the metro area. Not all trees in an image are labeled, since many trees occur on private property and are not mapped. We divided the RGB imagery into geographically distinct training and test datasets and used the street trees dataset to guide hand-annotation of a small number of tree crowns (n=1033).
Annotation by hand took approximately three hours and covered a small geographic area of mixed urban development, empty lots and ballfields (Figure  6 ). The street trees data was collected prior to the RGB images and was cleaned to remove trees that had been cut down or were obvious errors (e. g. trees located in the middle of buildings). To evaluate the street tree case study, we used field collected location of the tree stems to measure tree recall and the rate of undersegmentation. Recall was defined as the proportion of street tree locations that were contained within a predicted tree bounding box. Undersegmentation rate was defined as the proportion of predicted boxes that matched more than one street tree. Minimizing undersegmentation is challenging because trees growing close together can appear to be a single tree from above and is therefore best evaluated against ground collected data.
Conclusion DeepForest provides an open source software package for: 1) delineating tree crowns in RGB imagery, 2) evaluating the performance of that crown delineation using hand. CC-BY-ND 4. 0 International license available under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprint (which this version posted July 8, 2020. ; https://doi. org/10. 1101/2020. 07. 07. 191551 doi: bioRxiv preprint 19 labeled evaluation data, and 3) training new models and fine-tuning of the included prebuilt model to support prediction tailored to specific forest types. The inclusion of a prebuilt model allows users to benefit from the strengths of deep learning without needing to deal with many of the challenges.
Given the enormous diversity of tree appearance at a global scale, defining a single unified model for tree crown delineation is challenging. To address this, DeepForest provides an explicit retraining method to improve performance for specific use cases. This allows the user to decide what level of accuracy is required for the target question, and then annotate local data and retrain the model to produce predictions with sufficient accuracy for their use case. We recommend defining a clear evaluation dataset, setting a threshold for desired performance before training, and using evaluation data that is geographically separate from the training data to ensure that the prediction threshold holds outside of the training region. The minimal spatial resolution for accurate tree prediction using this software remains unknown and may ultimately relate to the desired ecological or management question.
Analysis of the NEON data show that individual tree segmentation is achievable at 10cm. The Portland Street trees example shows that 30 cm data (which is publicly available for many states and counties) provides reasonable delineations. However, the accuracy will not be as high as with higher resolution data, and further analysis at this resolution is necessary. One meter resolution imagery is increasingly available at near continental scales (e. g. , NAIP 1m imagery which provides nearly complete coverage of the United States; https://www. fsa. usda. gov/programs-andservices/aerial-photography/imagery-programs/naip-imagery/). It is unlikely that these. CC-BY-ND 4. 0 International license available under a was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
S2: Technical background to DeepForest and advice on model development The goal of this paper is to provide a robust open-source implementation for RGB tree crown delineation. For detailed information on data generation and model testing see  Weinstein et al. (2019 Weinstein et al. ( , 2020)). Here we provide an overview with recommendations for parameters that could affect model performance. For more discussion with code examples see https://deepforest. readthedocs. io/. DeepForest is a deep learning object detection model. Deep learning uses a series of hierarchical filters to connect low-level image features, such as colors and shape, to high-level concepts such as crown contours. Deepforest uses the keras-retinanet implementation (Gaiser et al 2017) of the Retinanet one-stage object detector with a Resnet-50 classification backbone  (Ren et al.
2015). This classification backbone was pretrained on the ImageNet dataset  (He et al. , 2016). This pretraining is useful for reducing training time and minimize overfitting to smaller ecological datasets. Given an input image, the model predicts both the location of bounding boxes and object classes simultaneously. In the DeepForest implementation, the prebuilt model has only 1 "Tree" class. All instances of the target class are predicted, such that if there are 40 trees in the image, an ideal model will predict 40 bounding boxes. Deep learning models have millions of parameters and take significant computational resources. To reduce memory consumption, we cut each prediction image into 40m by 40m windows with an overlap of 5%. Using a pool of unsupervised LiDAR-based tree predictions generated using Silva et al (2016), we pretrained the network with a batch size of 20 on 2 Tesla K80 GPU for 5 epochs.
• Batch size Neural networks are often trained in batches of images, since the entire dataset is often too large to read into memory at once. The size of these batches affects both the speed of training (larger batches train faster) and the stability of training (larger batches lead to more consistent results). The default batch size of 1 is chosen because it is not possible to anticipate the available memory and should be increased whenever possible. Typically, batch sizes are evenly divisible by the size of the entire dataset.