dataesr/scientific-paragraphs-categorization huggingface dataset:
5. The computational time per cycle for both versions is included in Table  II. At the time of writing, COMSOL were unable to explain why these models had longer run times in version 5. 5, despite backward compatibility. There is also an associated financial cost that can be a barrier to some researchers, which is where self-programmed techniques and those implemented in free software have a distinct advantage. Excellent qualitative and quantitative agreement was obtained between all models for the opencircuit equivalent instantaneous voltage and the cumulative time-averaged equivalent voltage, as well as the current density and electric field distributions within the HTS wire at key positions during the magnet transit. The average value for all the models of the DC output voltage of the HTS dynamo, determined by the cumulative time-averaged equivalent voltage over 10 cycles of PM rotation, was calculated to be -9.
Acknowledgements MA would like to acknowledge financial support from an  Engineering and Physical Sciences Research Council (EPSRC)   Early Career Fellowship   EP/P020313/1. CB, RM and MA were supported in part by  NZ Royal Society Marsden  Grant No. MFP-VUW1806. FP would like to acknowledge the support of the  Consejo Nacional de Ciencia y Tecnologia and Secretaria de Energia de Mexico , Reference No. 541016/439167. Additional data related to this publication are available at the University of Cambridge data repository (https://doi. org/10. 17863/CAM. 54005 ). ORCID iDs  Mark Ainslie   0000-0003-0466-3680 Francesco Grilli 0000-0003-0108-7235 Loïc Quéval 0000-0003-3934-4372 Enric Pardo 0000-0002-6375-4227 Fernando Perez-Mendez 0000-0001-7980-9539 Ratu Mataira 0000-0003-0892-5430 Antonio Morandi 0000-0002-1845-4006 Asef Ghabeli 0000-0001-9907-4509 Chris Bumby 0000-0001-8555-2469  Roberto Brambilla Acknowledgements MA would like to acknowledge financial support from an  Engineering and Physical Sciences Research Council (EPSRC)   Early Career Fellowship   EP/P020313/1.
Background: Many computational methods have been developed recently to analyze single-cell RNA-seq (scRNA-seq) data. Several benchmark studies have compared these methods on their ability for dimensionality reduction, clustering, or differential analysis, often relying on default parameters. Yet, given the biological diversity of scRNA-seq datasets, parameter tuning might be essential for the optimal usage of methods, and determining how to tune parameters remains an unmet need. Results: Here, we propose a benchmark to assess the performance of five methods, systematically varying their tunable parameters, for dimension reduction of scRNA-seq data, a common first step to many downstream applications such as cell type identification or trajectory inference. We run a total of 1. 5 million experiments to assess the influence of parameter changes on the performance of each method, and propose two strategies to automatically tune parameters for methods that need it.
Recently,  [15]  highlighted the relevance of these questions, showing that variation autoencoders (VAE) algorithms for dimension reduction (DR) of scRNA-seq work very well once properly tuned, but are extremely sensitive to changes in parameters and can dramatically fail if not properly tuned. Here, we propose to challenge this issue of parameter tuning focusing on methods for DR of scRNA-seq data, not only because they can be directly useful for visualization purpose, but also because DR is a common first step for most downstream applications such as cell type identification or trajectory inference  [8, 9]. We propose a new benchmark protocol for DR methods, composed of ten scRNA-seq datasets of various complexity mixing experimentally characterized populations of cells, where we measure the quality of a DR method by its ability to map cells of a given cell type near each other in the representation space.
Using this protocol, we benchmark five popular and representative DR methods, combining both PCA-based methods, a matrix factorization method, and VAE methods, systematically varying their tunable parameters. The resulting ∼ 1. 5 million experiments reveal not only the performance of DR methods using their default parameters, but also their performance if parameters are properly tuned. We find in particular that principal component analysis (PCA)-based methods like scran  [16]  and Seurat  [17]  are competitive with default parameters but do not benefit much from parameter tuning, while more complex models like ZinbWave  [18] , DCA  [19] , and scVI  [20]  can reach better performance but after parameter tuning. We propose and evaluate two strategies to tune parameters automatically, either by changing the default parameters or by optimizing a heuristic on each new dataset.
A benchmark of DR methods for scRNA-seq data A DR method takes a scRNA-seq dataset as input and maps each individual cell to a point in d-dimensional representation space, where downstream applications such as cell type prediction or lineage reconstruction are performed. In order to empirically assess the quality of DR methods and the influence of parameter tuning, we propose a benchmark protocol, summarized in Fig. 1 , where we collected ten diverse scRNA-seq datasets with Fig. 1  Overview of the benchmark protocol. We ran five representative DR methods, systematically varying their parameters on a large grid of values, on ten scRNA-seq datasets with known cell identity. We evaluate their ability to map cells of a given identity near other cells of the same identity, as measured by the silhouette score and the AMI after k-means clustering in the representation space experimentally validated cell types, and evaluate five representative DR methods, tested on a large parameter sweep, according to their ability to map cells of a given origin near to other cells of the same cell type.
The next four datasets (sc_10x to sc_celseq2_5cl) are in vivo mixtures of three or five human cell lines from  [12] , sequenced by CEL-Seq2 or 10x. Finally, the last dataset is an in silico mixture of four FACS purified mouse tissues from  [22]  produced with Smart-Seq2, where we selected tissues with no overlap in cell types. We use this benchmark to evaluate the performance of five popular computational pipelines for DR of scRNA-seq data: scran  [16] , Seurat  [17] , ZinbWave  [18] , DCA  [19] , and scVI  [20]. These pipelines are all publicly available as R or Python packages and can The first five datasets are derived from  [21]  and  [11]  and contain CD19+ B cells (B), CD14+ monocytes (Monocytes), CD4+ helper T cells (hT), CD56+ natural killer cells (NK), CD4+/CD45RO+ memory T cells (mT), CD8+/CD45RA+ naive cytotoxic T cells (cytoT), CD4+/CD45RA+/CD25-naive T cells (nT), and CD4+/CD25+ regulatory T cells (rT).
The four next are from  [12]  and contain the five following cell lines: A549, H1975, H2228, H838, and HCC827. The last one is from  [22]  process datasets containing thousands of cells in a reasonable amount of time (less than 12 h on a GPU/CPU with 10 cores). They all implement processing steps specific to scRNA-seq data together with representative DR methods including principal component analysis (PCA) for scran and Seurat, matrix factorization for ZinbWave, and (variational) autoencoders for DCA and scVI. While these pipelines also implement various downstream tasks such as cell clustering or differential expression analysis, we restrict our analysis to the DR step. To quantify the ability of a DR method to map biologically similar cells to similar locations in the representation space, we use two complementary measures: the silhouette, on the one hand, and the adjusted mutual information (AMI) when the cells are clustered with the k-means algorithm, on the other hand (see details in the "Material and methods" section).
Both measures vary between 0 for a random embedding to 1 for an embedding that perfectly preserves the cell type information. Silhouette is a measure agnostic to any particular clustering algorithm, and measures how close a cell is to other cells of the same type compared to cells of different types; for the silhouette to be large, cell types must not only be separated, but also form compact clusters far from each other. AMI, on the other hand, directly measures how well a particular clustering algorithm recovers known cell types and is therefore a good proxy for the performance of cell type identification as a downstream task of DR. Both measures are frequently used to assess the performance of cell embedding techniques  [10, 11, 18, 20, 23]. Another standard measure to assess the Fig. 2  UMAP representation of the ten scRNA-seq datasets, run after processing of the count matrices with Seurat with default parameters ability of a clustering algorithm to recover known classes is the adjusted rand index (ARI)  [10, 11, 20, 23] ; however, we found that AMI and ARI are extremely correlated (Additional file 1: Figure  S1 ) and lead to similar conclusions, so we just report results based on AMI below.
Performance of five popular DR methods with default parameters We first assess the performance of each method with its default parameters, except for the dimension of the representation space which we arbitrarily set to 10 for all methods. Indeed, the performance scores (AMI and silhouette) strongly vary with the dimension (Additional file 1: Figure  S2  and S3 ), so fixing the dimension allows to compare more fairly the different DR methods. Figure  3a  (with "default" legend) shows the performance reached by each method on each dataset, in terms of AMI (left) and silhouette (right), and Table  2  summarizes the mean performance reached by each method over the datasets. As expected, Fig. 3a  clearly shows that for all methods, the performance varies across datasets, in a rather consistent manner.
In terms of AMI, the four cell lines datasets tend to be the easiest (AMI > 0. 9 for most methods), followed by TabulaMuris and the two Zhengmix mixtures of four cell lines (AMI in the range 0. 7∼0. 9 for most methods), followed by the three Zhengmix mixtures containing the five closely related T cell populations (AMI in the range 0. 1∼ 0. 7 for most methods). The silhouette scores overall follow the same trend, although the difference between the first two groups of datasets is less pronounced. Interestingly, we see that in cases where several DR methods allow to almost perfectly  The "Default" columns correspond to the performance of each method using its default parameters, with a dimension of 10. The "Best" column corresponds to the best performance reached after varying the parameters.
The "ANOVA AMI heuristic" column corresponds to the performances of the new default parameters described in the "Influence of parameters on performance" section. The "Silhouette heuristic" column corresponds to the performance of the heuristic described in "Tuning parameters in practice" section cluster the cell types, such as sc_10x or sc_10x_5cl, the silhouette is usually far from 1 and varies across methods, illustrating the complementarity of both measures. Besides variations across datasets, we also observe variations across DR methods. As shown in Table  2 , scran has the best AMI on average (mean AMI = 0. 84), followed by Seurat, ZinbWave, and DCA (mean AMI = 0. 75∼0. 79), but this ranking is not statistically significant (p value > 0. 05 for Wilcoxon one-way test), while scVI is clearly behind (mean AMI = 0.
While these relative performances hold for a representation in dimension 10, the performance of some methods fluctuates with the dimension of the embedding space (Additional file 1: Figure  S2  and S3 ). While scran and Seurat are rather insensitive to increase in dimension after 10, DCA's mean AMI tends to increase in higher dimensions, while ZinbWave's and scVI's mean AMI decrease with the dimension, suggesting that different methods need more or less dimension to capture the same biology. This average performance of DR methods hides important variations across datasets, as visualized in Fig. 3a. For example, we see that scVI has specifically poor performance compared to other methods on the two CEL-Seq2 datasets, which may be due to a particularity of this technology or to the fact that both datasets (sc_celseq2 and sc_celseq2_5cl) have a relatively small number of cells.
The difference in AMI across methods, and the good behavior of the linear models underlying scran, Seurat and ZinbWave, is most visible on the "difficult" Zhengmix mixtures containing the five closely related T cell populations, while the difference in silhouette, and the good behavior of the nonlinear models underlying DCA and scVI, is more visible in the "easy" datasets where all methods have an AMI above 0. 7.
Performance reachable across a parameter sweep While using a computational pipeline with default parameters is often the method of choice for practitioners, there is little guarantee that default parameters are adapted to all situations. In particular, the performance of deep learning-based methods for scRNA-seq analysis was shown to be highly sensitive to choices of parameters  [15]. In order to assess the performance of each DR method if all parameters were properly tuned in a datasetspecific way, we now run each method by sweeping all tunable parameters across a large grid of values, as summarized in Table  3 , and compute the performance reached by each method after cherry picking a posteriori the best parameters. Note that the resulting performance is therefore an upper bound on the performance that each method can reached if parameters are tuned without knowing the ground truth.
In total, sweeping across the grid of parameters results in 384 different runs per dataset for scran and ZinbWave, 288 for Seurat, 40,320 for scVI, and 107,520 for DCA, hence 0. 360 for ZinbWave. Seurat and scran are the methods that benefit least from parameter tuning, suggesting that default parameters are already good choices across most datasets. Autoencoder-based DCA and scVI benefit more for parameter tuning, and outperform scran and Seurat in mean AMI after parameter tuning, confirming the importance of parameter tuning for these models  [15]. Since the number of parameters tested for these models is also two orders of magnitude larger than for Seurat, scran, and ZinbWave, the "best" performance after cherry-picking the best parameters may be over-optimistic for DCA and scVI. As for ZinbWave, a good choice of parameters leads to the best mean AMI across methods (0.
896) and the largest improvement in silhouette compared to default parameters. More precisely, for all cell line datasets and for TabulaMuris, parameter tuning allows all method to reach an almost perfect AMI, including methods like scVI that have a very poor performance with default parameters on sc_celseq2 and sc_celseq2_5cl. On the same datasets, parameter tuning brings an important improvement to the silhouette score of 0. 2 to 0. 6 to all methods. After parameter tuning, both encoder-based methods (DCA and scVI) tend to outperform ZinbWave, which tends to outperform both PCAbased methods scran and Seurat in terms of silhouette. This highlights the possibilities of nonlinear DR methods to perform DR even on simple datasets, but the need to correctly tune parameters in order to reveal their full potential.
On the immune cell datasets, we see again that tuning parameters allows to boost performance and bridge important gaps between methods in terms of silhouette and that after parameter optimization, both autoencoder-based methods slightly outperform ZinbWave, which slightly outperforms both PCA-based methods. The AMI performance of all methods is also improved by parameter optimization for all methods but scran, and we see no clear and consistent winner after parameter optimization. This suggests that simple PCA-based methods like scran, even with default parameters, are good enough to match the performance of more complex models after parameter tuning in terms of AMI; however, the better silhouette of more complex models once tuned may be an advantage for other downstream applications beyond clustering by k-means.
The benefits of parameter tuning is further illustrated in Fig. 3b , which shows a UMAP visualization of the representation space learned by the different methods with the default parameters (bottom two rows) or after parameter tuning for AMI (top two rows), for the Zheng-mix8eq dataset. For ZinbWave and scVI, which strongly benefit from parameter tuning in this case, we see that the dataset looks very different with the default parameters or after parameter optimization, the different cell types being but better separated in the later case. Regarding the absolute performance reached across the datasets, it is interesting to note that the best AMI scores for Zhengmix5eq and Zhengmix8uneq (the two hardest datasets) are only between 0. 6 and 0. 7, when they are above 0. 9 for Zhengmix4eq (the easiest one).
This suggests that if current DR methods are good at identifying sufficiently different cell types, they have difficulties to differentiate very similar cell types, a variety of T cells in our case, even after parameter optimization. To check how the AMI results are influenced by the choice of the clustering algorithm (here k-means), we also computed the AMI performance when cells are clustered by Ward's hierarchical clustering (HC) or the Louvain algorithm  [24]. As shown on Additional file 1: Figure  S4 , the performance barely changes when k-means is replaced by HC. When Louvain clustering is used, the overall performance decreases on most datasets, possibly because we used default parameters for this algorithm, while k-means and HC are run knowing in advance the correct number of clusters.
Influence of parameters on performance Having shown that parameter tuning has the potential to boost the performance of all methods for all datasets compared to using default parameters, we now investigate in more details the influence of each parameter on the performance. For that purpose, we estimate the mean contribution of each parameter value on the performance (AMI or silhouette) with a factorial analysis of variance (ANOVA, see Methods) procedure. We find that all parameters of all methods, except for "gene covariate" for ZinbWave on the silhouette, have a significant influence on both AMI and silhouette (ANOVA p value < 0. 05) and summarize in Additional file 1: Table  S1  and S2 the potential effect of tuning each parameter by comparing the best and worse contributions to performance among the values it can take.
As shown in Table  3  and Additional file 1: Table  S1 , the best parameter values in terms of mean contribution to the performance are not always the default parameters of each method. This suggests that the best parameter values identified by our analysis, which we refer to below as "ANOVA AMI heuristic" when we pick the parameter values that have the largest positive influence on AMI, may be interesting to use as new default parameters for each method. Note that, by construction, the ANOVA AMI heuristic parameters are the same for all datasets. To test this hypothesis, we report the performance of each DR method using the ANOVA AMI heuristic as default parameters in Additional file 1: Figure  S5  and summarize the mean performance across datasets in Table  2. We see that, on average, all methods benefit from the ANOVA AMI heuristic compared to the existing default parameters, particularly scVI, DCA, and ZinbWave in terms of AMI, and particularly ZinbWave, Seurat, and scVI in terms of silhouette.
In particular, ZinbWave outperforms all other methods, both in AMI and in silhouette, with the ANOVA AMI heuristic. Interestingly, we see in Additional file 1: Figure  S5  that for all methods, the AMI increases on almost all datasets with the ANOVA AMI heuristic. Of course, these promising results should be taken with care, given that we evaluate the performance of the ANOVA AMI heuristic on the datasets used to perform the ANOVA, but they suggest a systematic approach for method developers to set default parameters. We now investigate in more details to what extent further performance gain may result from parameter tuning on each dataset separately, as opposed to setting new default parameter values common to all datasets. For that purpose, we estimate again the contribution of each parameter in the final AMI and silhouette, allowing a different contribution in different datasets by adding interaction terms in the factorial ANOVA model between the dataset, on the one hand, and the tunable parameters, on the other hand (see Methods).
Note that we remove from this analysis a few parameters that were not tested on all datasets: ERCC for scran on 10x datasets, and sum factor normalization for scran on TabulaMuris. Additional file 1: Figure  S6 -S15 summarize the contributions of each parameter in each dataset for AMI and silhouette, as estimated from the factorial ANOVA with interactions. All interactions between dataset and parameters are significantly non zero (p value < 0. 05), except for the interaction between dataset and the "gene covariate" parameter of ZinbWave, showing that the influence of most parameters on the final performance is not the same across datasets. To assess whether dataset-specific parameter tuning is useful, we now check for each parameter whether the default value provided by the ANOVA AMI heuristic, which identifies the best values on average, is also the best or within 0.
05 of the best for both AMI and silhouette on all datasets. Based on this criterion, we find that for the AMI scran, Seurat, and ZinbWave have one parameter that benefits from dataset-dependent tuning (number of top genes for scran and Seurat; dimension of the latent space for ZinbWave), DCA has two (scale variance and dropout rate), and scVI has three (dimension of the latent space, number of hidden neurons, and number of training epochs), and for the silhouette, scran, Seurat, ZinbWave, and scVI have one parameter that benefits from dataset-dependent tuning (dimension of the latent space for scran and Seurat; number of top genes for ZinbWave; number of training epochs for scVI), and DCA has two (scale variance and dropout rate). Additional file 1: Table  S1  and S2 detail the potential gain in dataset-specific tuning for each parameter of each method.
Tuning parameters in practice Having shown that DR methods benefit from various degrees of parameter tuning, we now discuss the question of how this can be done in practice. Indeed, our strategy so far to identify the best parameters and evaluate their influence on performance is only possible when one knows the true cell type for each cell in the population, but such an oracle is usually not available in practice. In the absence of such information, one must therefore rely on quantitative heuristics or qualitative validation by domain experts, e. g. , by looking at the distribution of cells in the representation space and assessing whether it shows some promising structure such as clusters. As a first step towards an automated way to tune parameters in a dataset-specific way, we now propose a simple quantitative and objective heuristic to tune parameters, which we call the silhouette heuristic, and evaluate its performance on our benchmark.
The silhouette heuristic measures how well a distribution of cells in the representation space looks like a possible clustering of distinct cell types. Given a set of cells in a representation space, typically a dataset of cells processed by a DR method with some parameter values, the silhouette heuristic first runs a k-means clustering algorithm on the cells in the representation space, and then computes the silhouette score of the dataset with respect to the cell types assigned by the k-means clustering. In particular, if the k-means clustering identifies the true cell types, then the silhouette heuristic boils down to the silhouette score with respect to the true cell types. To tune parameters for a DR method on a dataset, we then just compute the silhouette heuristic over a grid of candidate parameter choices and select the values that maximize it.