dataesr/scientific-paragraphs-categorization huggingface dataset:
5, the query cell was assigned to that indexed cell along with its supporting metadata (cluster assignment, bulk prediction, pseudotime value). Using this cosine similarity threshold, 94% of 10x P. berghei cells were assigned to a cell in the SS2 P. berghei reference dataset. To align the IDC trajectories across the three 10x datasets, we first compiled a set of one-toone orthologs among 10 Plasmodium species (P. berghei, P. knowlesi, P. falciparum, P. malariae, P. ovale, P. vivax, P. gallinaceum, P. yoelii, P. chabaudi, P. cynomolgi) from OrthoMCL (61) (data S3). Using these orthologs, we built a scmap reference index that contained all P. berghei 10x IDC cells (gametocytes removed). We mapped both the P. falciparum and P. knowlesi data to this ortholog reference index. In addition to identifying the top nearest neighbor cell, we were able to incorporate information from the top three nearest neighbors to assign each cell based on the principal component space.
To do this, we took a mean of the first two principal components of the top three nearest neighbors. Given this coordinate assignment, we located the nearest cell on the PCA and assigned the query cell to this index cell. If all three of the nearest neighbors had a cosine similarity of 0. 3, then the query cell was given an assignment. With this lower cosine similarity threshold to account for cross-species differences, we were able to assign more than 96% of P. falciparum and 99% of P. knowlesi cells to a P. berghei index cell. Finally, to map single-cell samples from the field, we built a 1:1 ortholog index of the complete 10x P. berghei dataset, including the gametocytes that were excluded for the IDC evaluations. We used this reference because (i) it fully represents the IDC and mature gametocytes, and (ii) it originates from an in vivo system like the volunteer cells.
Because the gametocyte data were more sparse, the IDC cell assignment was based on the top nearest neighbor alone along with a cosine similarity threshold of 0. 4. Using this method, we were able to map 13 P. malariae cells and 22 P. falciparum cells, assigning each cell to a developmental time. "Clock" pseudotime-For the three 10x datasets, pseudotime around the IDC was calculated by fitting an ellipse to the data projected into the first two principal components using direct least squares (Fig. 3C  and fig. S15 ). Angles around the center of this ellipse were calculated for each cell and oriented to a starting cell, which was defined using known markers. To align the three species in pseudo-time, 10x data from P. knowlesi and P. falciparum were projected directly onto the P. berghei reference using scmap  (19)  and cells were given the pseudotime of their P.
Rationale We have assembled a Malaria Cell Atlas that presents the transcriptomic profiles of individual Plasmodium parasites across all morphological life cycle stages. The ambition of such an atlas is to (i) inform gene function and usage throughout the life cycle, (ii) understand the gene regulatory mechanisms underlying developmental transitions, (iii) discover parasite bet-hedging patterns, and (iv) provide a reference dataset that can be used to understand parasite biology by the malaria community in both lab and natural infections for multiple Plasmodium species.
Conclusion We generated transcriptomes for all life cycle stages of Plasmodium and released these via the interactive Malaria Cell Atlas website, www. sanger. ac. uk/science/tools/mca/mca/. The Malaria Cell Atlas provides new insights into gene function and parasite developmental progression. We have demonstrated that it can serve as a transcriptomic reference, facilitating the interpretation of data from multiple species and multiple technologies. The characterization of wild Plasmodium parasites with immense genetic diversity will advance the study of the pathology and transmission of malaria directly from infected carriers. We envision that the Malaria Cell Atlas will support the development of much-needed new drugs, vaccines, and transmission-blocking strategies. (A) The life cycle begins when an infected mosquito injects sporozoites into the mammalian host.
Nodes are colored according to their graph-based spectral clustering assignment, and each cluster is labeled by cluster number. (B) A heat map of mean expression for each cluster across all cells in the dataset. Cells are ordered by their developmental progression. Shown at the right are the number of genes in each cluster (N), the most significant Gene Ontology term (biological process) associated with the cluster (from data S1, Benjamini P < 0. 05), and the top cited gene in the cluster (based on PlasmoDB). In the clusters where there is a tie for.
Single-cell RNA-seq references for the Plasmodium genus. Left: Single-cell transcriptomes from across the life cycle of Plasmodium berghei were profiled (including liver, blood, and mosquito life stages). Center: Deep exploration of blood-stage parasites captured transcriptomic diversity at single-cell resolution across three different Plasmodium species by droplet sequencing. Right: Such datasets can serve as references to understand wild parasites isolated from clinical samples.
Funding The  Wellcome Sanger Institute  is funded by the  Wellcome Trust  (grant  206194/Z/17/Z ), which supports M. K. N. L. , M. H. , O. B. , M. B. , and  J. C. R. Also  supported by  MRC Career Development Award   G1100339  (M. K. N. L. ); the  Danish Institute of Advanced Study (D-IAS)  and the  University of Southern Denmark (SDU)  (K. N. );  Wellcome Trust  grant  107372 , the  UK Department for International Development (DFID) , the  Swedish International Development Cooperation Agency (Sida) , the  Swiss Agency for Development and Cooperation (SDC) , the  German Academic Exchange Service (DAAD) , and the  Kenyan government  (J. K. H. and H. B. );  NIH/NIAID  grant  R01AI137154  (J. R. and L. V. ); and  Chan Zuckerberg Initiative  grant reference  183501  (T. A. ).
, 2009). Unsupervised classification (segmentation) of HLS images is used to disaggregate the BRDF parameters to the HLS spatial resolution and to build a BRDF parameters database at HLS scale. We first test the proposed BRDF normalization for different solar zenith angles over two homogeneous sites, in particular one desert and one Peruvian Amazon forest. The proposed method reduces both the correlation with the solar zenith angle and the coefficient of variation (CV) of the reflectance time series in the red and near infrared bands to 4% in forest and keeps a low CV of 3% to 4% for the deserts. Additionally, we assess the impact of the view zenith angle (VZA) in an area of the Brazilian Amazon forest close to the equator, where impact of the angular variation is stronger because it occurs in the principal plane.
Introduction The Harmonized Landsat/Sentinel-2 (HLS) project  [1]  provides a surface reflectance product that combines observations from USGS/NASA's Landsat 8 and ESA's Sentinel-2 satellites at moderate spatial resolution (30 m). The main goal is to provide a unique dataset based on both satellites' data to improve the revisit time to 3-5 days depending on the latitude  [1]. Along with a common atmospheric correction algorithm  [2] , geometric resampling to 30 m spatial resolution and geographic registration  [1] , the product is also corrected for BRDF effects and band pass adjustment. The purpose of the band pass adjustment is to correct discrepancies between data coming from different sensors due differences their Relative Spectral Response (RSR) functions. In the case of the near infrared (NIR) spectral band, the spectral difference between Sentinel-2 and Landsat-8 is ~0.
HLS Data The harmonization process of the HLS product is based on a set of algorithms for atmospheric correction, cloud and cloud-shadow screening, and co-registration of data from the Operational Land Imager (OLI) onboard Landsat-8 and the Multi-Spectral Instrument (MSI) onboard Sentinel-2. The HLS product is also corrected for bandpass difference and BRDF effects. However, in this work, we considered uncorrected data to explore the feasibility of the proposed algorithm for BRDF normalization. We used all HLS data (version 1. 4) available from January 2013 to December 2017.
SURFRAD Data The global downwelling and upwelling solar radiations are continuously recorded by skyward facing and downward facing Eppley model 8-48 pyranometers mounted on a meteorological tower. Table  1  specifies the height of each tower above the target. TERN OzFlux is a national ecosystem research network set up to provide the Australian and global ecosystem modelling communities with nationally consistent observations of energy, carbon, and water exchange between the atmosphere and key Australian ecosystems. TERN OzFlux is part of an international network (FluxNet) of over 500 flux stations that is designed to provide continuous, long-term micrometeorological measurements to monitor the state of ecosystems globally  [17]. From the 40 sites in this network, we selected 5 sites for validation purposes according to data availability during the period analyzed (2013 to 2017).
Based on the individual BRDF unmixing of each image in the HLS database (starting from 2013 for Landsat 8, 2015 for Sentinel 2A and 2017 for Sentinel 2B), our goal is to reproduce the BRDF parameters V 0 , V 1 , R 0 , and R 1 in Equations (  4 ) and (  5 ) at HLS spatial resolution. Next, we apply a linear regression for the V and R parameters versus the NDVI to derive the V 0 , V 1 , R 0 , and R 1 parameters at HLS pixel level. Finally, with the BRDF parameters at HLS resolution, we apply Equation (  8 ) to derive the normalized surface reflectance (ρ N ) at 45 degrees of solar zenith angle and nadir observation  (Vermote et al. , 2009) : ρ N (45, 0, 0) = ρ(θ s , θ v , φ) * 1 + VF 1 (45, 0, 0) + RF 2 (45, 0, 0) 1 + VF 1 (θ s , θ v , φ) + RF 2 (θ s , θ v , φ) (8).
Spatial Evaluation of an Equatorial Region Figure  6  shows the HLS NIR surface reflectance image. The directional reflectance (Figure  6a ) shows higher values along the western part (compared to the eastern) where the VZA is larger (Figure  6c ). Note that the solar azimuth angle image shows nearly constant values of 63 degrees. This means that the western part is observed in the backscattering direction, which explains the higher values observed. The current HLS correction (Figure  6b ) reduces this angular effect. However, the west area still shows higher values. Finally, when applying the proposed normalization (Figure  6c ) this illumination effect is minimized. Figure  7  shows the transect surface reflectance versus the view zenith angle. Though there is variability through the different samples, we have added a trendline to show the angular dependency of each dataset.
The directional reflectance shows the highest values and the largest slope with a pvalue of 0. 02, and it is followed by the current HLS BRDF normalization with a p-value of 0. 08. The proposed method shows the lowest and more stable values with a slope closer to zero and a p-value of 0. 31, meaning that there is no significant relationship between the corrected reflectance and the VZA. Additionally, the similar intercepts of the linear regressions show that the three datasets converge to a similar value at zero degrees observation. The analysis of the red band and NDVI (Appendix A) show similar conclusions but the directional effect is lower. Figure  7  shows the transect surface reflectance versus the view zenith angle. Though there is variability through the different samples, we have added a trendline to show the angular dependency of each dataset.
The directional reflectance shows the highest values and the largest slope with a p-value of 0. 02, and it is followed by the current HLS BRDF normalization with a p-value of 0. 08. The proposed method shows the lowest and more stable values with a slope closer to zero and a p-value of 0. 31, meaning that there is no significant relationship between the corrected reflectance and the VZA. Additionally, the similar intercepts of the linear regressions show that the three datasets converge to a similar value at zero degrees observation. The analysis of the red band and NDVI (Appendix A) show similar conclusions but the directional effect is lower. Remote Sens. 2019, 11, x FOR PEER REVIEW 13 of 18.
Discussion and Conclusions Quality of atmospheric and BRDF correction is essential for any application of the HLS dataset. In this work, we present a new method to improve the BRDF normalization of the HLS data. It is based on the Franch  [11]  method, which was originally designed and evaluated to derive the surface albedo of Landsat TM and ETM+ data. In this manuscript we improve it, apply it to HLS data, explore its feasibility to correct the BRDF effects of the dataset, and validate it over two homogeneous sites and several land cover types of the SURFRAD and OzFlux networks. The proposed improvement is based on reproducing the BRDF parameters' NDVI dependency  [12, 13]  at moderate spatial resolution (HLS). In the operational context, this means that we create an HLS pixel-based database of the BRDF parameters (V0, V1, R0, and R1) that is regularly updated to account for any land cover change.
Discussion and Conclusions Quality of atmospheric and BRDF correction is essential for any application of the HLS dataset. In this work, we present a new method to improve the BRDF normalization of the HLS data. It is based on the Franch  [11]  method, which was originally designed and evaluated to derive the surface albedo of Landsat TM and ETM+ data. In this manuscript we improve it, apply it to HLS data, explore its feasibility to correct the BRDF effects of the dataset, and validate it over two homogeneous sites and several land cover types of the SURFRAD and OzFlux networks. The proposed improvement is based on reproducing the BRDF parameters' NDVI dependency  [12, 13]  at moderate spatial resolution (HLS). In the operational context, this means that we create an HLS pixel-based database of the BRDF parameters (V 0 , V 1 , R 0 , and R 1 ) that is regularly updated to account for any land cover change.
The Piecewise Aggregate Approximation (PAA) is widely used in time series data mining because it allows to discretize, to reduce the length of time series and it is used as a subroutine by algorithms for patterns discovery, indexing, and classification of time series. However, it requires setting one parameter: the number of segments to consider during the discretization. The optimal parameter value is highly data dependent in particular on large time series. This paper presents a heuristic for time series compression with PAA which minimizes the loss of information. The heuristic is built upon the well known metaheuristic GRASP and strengthened with an inclusion of specific global search component. An extensive experimental evaluation on several time series datasets demonstrated its efficiency and effectiveness in terms of compression ratio, compression interpretability and classification.
Introduction Time series databases are often large and several transformations have been introduced in order to represent them in a more compact way. One of these transformations is Piecewise Aggregate Approximation (PAA)  [13] , which consists in dividing a time series into several segments of fixed length and replacing the data points of each segment with their averages. Due to its simplicity and low computational time, PAA has been widely used as a basic primitive by other temporal data mining algorithms such as  [16, 17, 25] , in order • to construct symbolic representations of time series  [3, 26] ; • to construct an index for time series  [12, 14, 30]. Indeed, PAA allows queries which are shorter than length for which the index was built. This very desirable feature is impossible with Discrete Fourier Transform, Singular Value Decomposition and Discrete Wavelet Transform; • to classify time series.
Iterative deepening dynamic time warping For determining the number of segments, IDDTW only considers values that are power of 2 and for each value, computes an error distribution by comparing PDTW with the standard DTW at each level of compression. It takes as inputs: the query Q, the dataset D, the user's confidence (or tolerance for false dismissals) user conf , and the set of standard deviations StdDev obtained from the error distribution. Example: Let C and Q be two time series of the dataset D, let best so f ar be the DT W distance between two time series of the dataset. Suppose the distance D pdtw (Q, D) is 40 and the best so f ar is 30. The difference between the estimated distance and the best so f ar is 10. Using the error distribution centred around the approximation (40), we can determine the probability that the candidate could be better by examining the area beyond the location of the best so f ar (shown in solid black in Fig.
2 ): We disqualify a candidate if this probability is less than the user's specified error acceptance, the candidate is disqualified; otherwise, a finer approximation is used and the test is re-applied to the next depth. This process continues until the full DTW is performed. More precisely, IDDTW proceeds as follows: • the algorithm starts by applying the classic DTW to the first K candidates from the dataset. The results of the best matches to the query are contained in R, with |R| = K. The best so f ar is determined from argmaxR; • both the query Q and each subsequent candidate C are approximated using PAA representations with N segments to determine the corresponding PDTW; • a test is performed to determine whether the candidate C can be pruned off or not. If the result of the test is found to have a probability that it could be a better match than the current best so f ar, a higher resolution of the approximation is required.
Evaluation procedures for the compression quality Before explaining how to evaluate the quality of time series compression, we first describe the time series datasets that we considered. They are made up of time series associated with labels that identify the shape of the latter. For instance, in the ECG dataset, each time series traces the electrical activity recorded during one heartbeat. The two classes are a normal heartbeat and a Myocardial Infarction. Time series classification is a classic problem with time series which consists in guessing the label of an unlabeled time series based on its shape. The quality of a time series classification model is evaluated from its classification error ( ), or its accuracy (a = 1 -). When considering classification task, one of the best classification algorithm to use for evaluating the quality of time series alignment is one nearest neighbor (1NN).
Problem definition Let D = {d i } be a set of datasets composed of time series. We note |d i | the number of time series of the dataset d i. Let X ∈ d i be a time series of the dataset d i ; we note |X| = n the length of the time series X. For simplicity of notation we suppose that all the time series of d i have the same length. d i → 1N N DT W (d i ) (3. 2) 1N N DT W (d i ) is the classification error of one nearest neighbour with Dynamic Time Warping on the dataset d i. Definition 3. 2. 1N N P DT W : D × {1. n} → [0, 1] (d i , N ) → 1N N P DT W (d i , N ) = 1N N DT W • P AAset(d i , N ) (3. 3) 1N N P DT W (d i , N ) is the classification error of 1-NN with PDTW using N segments on d i. Our goal is to find the number of segments that allows PDTW to best align time series. PDTW gives a good alignment when its classification error with 1NN is low  [20].
At each iteration, the heuristic tests nb( √ 2n) = √ 8n number of segments. As we have k iterations the number of candidates tested is: |C| = k √ 8n. The details of the heuristic are presented in Algorithm 1. Time complexity: We use the training set to find the number of segments that should be considered with PDTW. For that purpose, we applied 1N N on the training set that costs O(|d| 2 × n 2 √ n), (3. 6) where |d| 2 comes from 1N N algorithm and n 2 √ n comes from P DT W. Lemma 3. 3. For a given dataset d i ,the quality of the alignment of our heuristic is better than that of DTW: FDTW(d i ) ≤ 1N N DT W (d i ). Proof. 1NNDTW(d i ) = 1NNPDTW(d i , n). Then, 1NNDTW(d i ) is one of the candidates considered by the heuristic FDTW. Since FDTW returns the minimal classification error from all candidates, the classification error of 1NNDTW is always greater than or equal to FDTW.
Nevertheless, a heuristic does not always give the optimal value. To ensure that it gives a result not far from the optimal value, one approach is to guarantee that the result of the heuristic always lies in an interval with respect to the optimal value  [9]. In our case, we are looking for the number of segments that allows a good alignment of time series. The alignment is good when the classification error with 1NN is minimal or when the accuracy is maximal. Let d i be a dataset: acc max(di) = 1 -min 1≤N ≤n {1N N P DT W (di, N )} is the maximal accuracy for the dataset d i , acc DT W = 1 -1N N DT W (d i ) is the accuracy obtained with d i and 1NNDTW, and acc F DT W = 1 -F DT W (d i ) is the accuracy of our heuristic. To ensure the quality of our heuristic FDTW, we hypothesized that 1NNDTW is better than Zero Rule classifier.
Zero Rule classifier is a simple classifier that predicts the majority class of test data (if nominal) or average value (if numeric). Zero Rule is often used as baseline classifier  [7]. The minimal value of the accuracy of Zero Rule is 1 c where c is the number of classes of the dataset. Proposition 3. 4. For a given dataset d i that has c i classes, c i ∈ N * , if acc DT W ≥ 1 c i then 1 c i × acc max ≤ acc F DT W ≤.
Experiment and results Throughout the experiments described in this paper, FDTW performs three iterations (k = 3) when searching for the appropriate number of segments for a dataset. To evaluate the ability of FDTW heuristic to propose a good number of segments for PAA. It has been compared to the IDDTW algorithm in terms of: • heuristic execution speed; • time series compression ratio; • classification error associated with the number of segments found by the heuristic.
Datasets The experiments have been performed first on 45 datasets and then on 84 datasets of UCR time series datamining archive  [5] , which provided a large collection of datasets that covers various categories of domains. Each data set is divided into a training set and a testing set. The 84 datasets possess between 2 and 60 classes, the length of time series varies from 24 to 2709, the training sets contain between 16 and 8926 time series and the testing sets contain between 20 and 8236 time series. All datasets are publicly available on the UCR time series classification page.
Compression Compression ratio: An immediate way to evaluate the quality of the segmentation is to compare the compression ratios. A segment number N 1 will be better than a segment number N 2 if it makes it possible to obtain a more compact representation with PAA. The compression ratio is given by: r = n -N n where n is the length of the time series and N is the number of segments considered with PAA. The closer r is to 1 the better is the compression. The numbers of segments used here are shown in Table  1. For the considered datasets, the mean compression ratio of IDDTW (r = 0. 654) is slightly higher than that of FDTW (r = 0. 605). However, this difference is not significant. Indeed, the wilcoxon test gives us a p-value greater than 0. 1 (p > 0. 1). Therefore, we cannot reject the hypothesis that the compression ratios of IDDTW and FDTW are equal.