dataesr/scientific-paragraphs-categorization huggingface dataset:
Experimental results In this experiment, we utilized TiO-band images obtained from the Big Bear Solar Observatory on multiple dates, including 2018 June 7, 2019 June 18, 2019 June 24, 2020 July 18, and 2021 April 30. Then, we used the dataset to conduct ablation experiments, comparative experiments, and robustness experiments. These three sets of experiments are presented in Sects. 4. 1 and 4. 2. All training experiments were performed on a single computer equipped with an RTX 3090 GPU and a memory capacity of 24 GB. All testing experiments were carried out on another single computer featuring a GTX 1660 GPU with a memory capacity of 6 GB.
In contrast, our proposed MBP-TransCNN A121, page 8 of 14  Notes. The first two columns indicate the dataset and its corresponding date, while the third and fourth columns show the time span and the number of frames in the dataset. The fifth column shows the average time per frame for our proposed method for each time span, and the sixth column shows the average time per frame for the method proposed by  Yang et al. (2019). For ease of comparison, we highlight the shorter runtime in bold font for each time span. method based on powerful CNNs and Transformers can effectively learn the features of MBPs, which significantly reduces the impact of brightness changes and leads to more robust results for complex MBP contours and shapes.
Comparison to the deep learning Mask RCNN method We also attempted to reproduce the deep learning Mask RCNN method of  Yang et al. (2019). The goal was to compare it to our proposed method using the same annotated MBP dataset for training. To ensure a fair comparison with the Mask RCNN method proposed by  Yang et al. (2019) , we followed their preprocessing method. Specifically, we first standardized and normalized all images in our dataset and then applied a 3 × 3 Laplacian kernel convolution filter to enhance the gradient of MBPs. Subsequently, we constructed the dataset by feeding the Laplacian images into the Mask RCNN for training. We adopted the same training parameters as specified in the paper of  Yang et al. (2019) , such as RPN anchor ratios of 1, 2, 4, 8, and 16; a batch size of ten; and a learning rate of 0.
001. In both methods, we used random cropping and rotation for the input dataset. The input image size was set to 640 × 640, and the network was trained for 100 000 iterations until it achieved stable convergence, which required approximately 5 h of training time. We refer to the network trained using the  Yang et al. (2019)  method as GBP-MRCNN. For our proposed method, we followed the same training strategy, maintaining the image size and number of iterations as mentioned above. From Fig. 9 , it can be seen that the deep learning method of  Yang et al. (2019)  achieves good results in recognizing point-like MBPs, but for MBPs with complex shapes, such as block-shapes or flower-shapes, the method fails to accurately segment the contours of the MBPs. Thanks to the powerful global feature learning ability of the Transformer, our MBP-TransCNN method shows good robustness regarding complex-shaped MBPs and is less affected by changes in brightness and shape.
Running time and five evaluation metrics In this section, we present a comparative analysis of the runtimes and five testing metrics between our proposed method and the method proposed by  Yang et al. (2019). Our method directly inputs the original image to predict the MBPs, while the method of  Yang et al. (2019)  requires preprocessing steps, such as standardization, normalization, and enhancing gradients using the Laplacian convolution kernel, before the image can be used as input in the network to make predictions. Therefore, our inference time does not include any preprocessing time, whereas the inference time of  Yang et al. (2019)  includes the time for preprocessing. For the time comparison, we used a five-day dataset with the same field of view. The results are presented in Table  3.
In this study, we proposed a deep learning method called MBP-TransCNN, which is based on CNNs and Transformers, to achieve fine structure segmentation of MBP images. By utilizing the strong feature extraction and supervised learning capabilities of CNNs and Transformers, we constructed a dataset of ten original MBP images and used elastic deformation data augmentation to expand the training set to 1000 images for training. The MBP-TransCNN algorithm was trained on a computer equipped with a RTX 3090 (24 GB RAM device), and after 100 000 iterations, it required 9 h of training time until the loss value stabilized and converged. The trained model was tested on the annotated ten-frame MBP dataset captured at different time intervals. The results, as shown in Fig. 5 , indicate a good segmentation performance, with an average precision of 0.
999, a recall of 0. 887, an F1 of 0. 920, a PA of 0. 963, and a segmentation IoU of 0. 854. The experimental results demonstrate that the MBP-TransCNN deep learning method can accurately and rapidly segment the MBP structures in GST observational data. Our method achieves stable and satisfactory segmentation results for complex and small MBPs. Therefore, our approach can automatically process a large amount of observational data and generate reproducible MBP segmentation results that can facilitate research on solar physics and the solar magnetic field. Our proposed MBP-TransCNN can be readily extended to perform pixel-level segmentation of other types of solar images, a topic that we plan to investigate in future studies.
Acknowledgements. This work was supported by  Yunnan Province Ten thousand Talents Program. This work was funded by the  National Natural Science Foundation of China  ( 11727805 ). We gratefully acknowledge the use of data from the  Goode Solar Telescope (GST)  of the  Big Bear Solar Observatory (BBSO). The BBSO operation is supported by  US NSF   AGS-1821294  grant and  New Jersey Institute of Technology. GST operation is partly supported by the  Korea Astronomy and Space Science Institute  and the  Seoul National University.
Cell-type deconvolution methods aim to infer cell-type composition and the cell abundances from bulk transcriptomic data. The proliferation of currently developed methods (>50), coupled with the inconsistent results obtained in many cases, highlights the pressing need for guidance in the selection of appropriate methods. Previous proposed tests have primarily been focused on simulated data and have seen limited application to actual datasets. The growing accessibility of systematic single-cell RNA sequencing datasets, often accompanied by bulk RNA sequencing from related or matched samples, makes it possible to benchmark the existing deconvolution methods more objectively. Here, we propose a comprehensive assessment of 29 available deconvolution methods, leveraging single-cell RNA-sequencing (scRNA-seq) data from different organs and tissues.
We offer a new comprehensive framework to evaluate deconvolution across a wide range of scenarios and we provide guidelines on the preprocessing of input matrices. We then validate deconvolution results on a gold standard bulk Polymorphonuclear Peripheral/Blood Mononuclear cell (PMN/PBMC) dataset with well known cell-type proportions. We show that single-cell regression-based deconvolution methods perform well but their performance is highly dependent on the reference selection and the tissue type. Our study also explores the significant impact of various batch effects on deconvolution, including those associated with sample, study, and technology, which have been previously overlooked. Importantly, we suggest a novel methodology for consensus prediction of cell-type proportions for cases when ground truth is not available.
The large-scale assessment of cell-type prediction methods is provided in a modularised pipeline for reproducibility (https://github. com/Functional-Genomics/CATD_snakemake). Lastly, we suggest/propose/recommend that the Critical Assessment of Transcriptomic Deconvolution (CATD) pipeline be employed for the efficient, simultaneous deconvolution of hundreds of real bulk samples, utilising various references. weenvision it to be used for speeding up the evaluation of newly published methods in the future.
Furthermore, conventional methods, such as flow cytometry and immunohistochemistry which can quantify cell abundances, rely on prior knowledge of cell markers, depend on interpretation bias and are not easily scalable  6, 7. Additionally, there are only a handful of these types of datasets available, mostly from non-solid tissues such as PBMCs  8, 9. Consequently, cell-type deconvolution methods show a great advantage, since cell proportions from hundreds of samples can be quantified computationally in minutes. Deconvolution methods aim to derive cell type compositions and cell abundances from bulk RNA-seq data using large databases such as GTEx  10  , TCGA  11  , TARGET (https://ocg. cancer. gov/programs/target) and GEO  12. Specifically, cell-type deconvolution can assist in the study interaction between cell abundances and various disease phenotypes or clinical features, coupled with expression data within large databases  13.
CIBERSORT  16  , FARDEEP  18  , NNLS (non-negative least squares) and EpiDISH  19  are some examples of commonly used bulk deconvolution methods. Contemporaneous deconvolution methods leverage available single-cell datasets to extract cell-type specific features in order to perform deconvolution. Bseq-sc  17  , DWLS  20  and MuSiC  21  and others  [22] [23] [24]  utilise single-cell data to deconvolve bulk RNA-seq samples. Unlike the supervised methods, unsupervised methods like deconf  25  and CDSeq  26  only require a bulk sample to be deconvolved and the number of components (number of cell types to be found) as an input. Finally, semi-supervised methods such as ssKL and ssFrobenius, simultaneously estimate the cell-specific matrix and cell proportions, using marker information and a block-descent method that estimates both matrices  27.
Even though deconvolution remains a very promising and cost-effective approach for estimating cell-type abundances in a tissue, there are still many challenges to overcome. A plethora of methods have been developed, yet there are no clear guidelines on method selection and a systematic large-scale benchmark across tissues is missing. There are additional components such as the pre-processing of the input matrices, the selection of reference and the batch effects between data inputs that complicate deconvolution  22. However, it remains uncertain whether the proportions of closely related cell types or cell states can be accurately resolved  26. Previous benchmarking studies have evaluated the performance of deconvolution methods on a small number of datasets and on limited organs and tissues, with a focus on data from pancreatic islets and blood tissue  [28] [29] [30].
Moreover, there is a dearth of formal guidelines for the generation of suitable simulation data to drive new method development. Μost benchmarks have focused on the technical aspects of deconvolution and lack biological interpretation of real tissue data  29. Here, we have designed a curated, large-scale, detailed assessment of 29 available deconvolution methods by utilising sc-RNA-sequencing datasets from both human and mouse tissues. We first suggest a framework for simulating realistic bulk samples from single-cell methodologies, suitable for evaluating diverse methods. Next, we provide guidelines on the use of normalisation and transformation to reduce the effect of library size and the differences in expression between bulk and single-cell references respectively. Furthermore, we study the impact of batch effects in deconvolution and provide suggestions for the optimal selection of references.
Computational framework for large-scale analysis of deconvolution (CATD) We developed a computational framework that enables us to systematically investigate various factors that influence the deconvolution process, in order to provide clear guidelines for the decomposition of bulk tissue expression data. (Fig  1 ). To benchmark the 29 available deconvolution methods, we collected scRNA-sequencing published datasets from different human and mouse organs (Table  S1 ). In our pipeline, each scRNA-seq dataset is split into a training dataset, used as a "reference input" for deconvolution, and a testing dataset that was composed of pseudo-bulk mixtures to be deconvolved (Methods). The use of pseudo-bulk mixtures allows us to obtain ground truth against which we can evaluate the deconvolution methods.
The collected scRNA-seq experiments were generated by numerous protocols including Smart-Seq2 31 , 10X  32  , CEL-seq  33  , Seq-well  34  , Drop-Seq  35  , inDrop  36  and originate from different organs including brain, pancreas, lung, blood(PBMC) and placenta. Moreover, datasets of different granularity were included in the benchmark to account for the effect of cell-type numbers in deconvolution (Table  S1 ). As a result, the generated pseudo-bulks encompass a wide range of real data scenarios. Additionally, different levels of noise were added to the pseudo-bulks to mimic realistic bulk samples. In addition, 11 different normalisation and 2 transformation methods (Methods, TableS2) were tested on both pseudo-bulk and reference matrices to select the most suitable combination of preprocessing to be performed prior to deconvolution.
After the preprocessing step, we evaluated the performance of the 29 collected deconvolution methods (Table  S3 ). To compare the performance of deconvolution methods we implemented various evaluation metrics in the CATD pipeline. Pearson correlation coefficient (r/pcor), root mean square error (RMSE) and average cosine similarity (avgcos) are the ones used across this study to evaluate the performance of the methods(Methods). We select the best performing methods taking into consideration the methods that have both low RMSE and high R values consistently across different simulation scenarios, datasets and tissues. All the steps of the benchmarking have been modularised in a snakemake pipeline for the reproducibility of the results (https://github. com/Functional-Genomics/CATD_snakemake). The pipeline also allows the evaluation of the scalability of each benchmarking module using additional metrics (Methods).
We considered two different approaches in this study. In the first approach, pseudo-bulks are generated by randomly sampling cells from three pool sizes (p1=100, p2=10K, p3=100K) from a single-cell dataset and generating 1,000 samples. In the second approach, we generate an array of pseudo-bulks by setting the following parameters: the number of cell types sampled (nCT) from single-cell, the cell proportion assignment method (variance of proportions) along with the pool size (p) and the number of samples (s) (Methods, Fig. 2a ). We generated 5 pseudo-bulk datasets based on a 10X single-cell-RNA-seq from the mouse brain  41  (Table  S1 , Supplementary Figure  1a-b ) and compared them with real bulk samples derived from the same study (Fig. 2b ). Real bulk and generated pseudo-bulk samples exhibit a negative binomial distribution, as expected, for expression data  42  however, random sampling pseudo-bulk deviates the most from real bulk, with less overdispersion observed (Fig.
The selection of pre-processing methods yields varied deconvolution results Previous studies have reported that pre-processing of input data in deconvolution impacts the deconvolution performance  22, 28  , but it is yet unclear which methods should be used for each deconvolution method selected. Standard preprocessing of bulk and single-cell data includes the transformation and normalisation of the expression matrices. Transformation methods account for the mean-variance dependencies as well as the extreme count values present in a dataset. This is a crucial pre-processing step for most downstream analyses of both single-cell and bulk RNA-seq data. For instance, both the identification of DEGs and PCA analysis requires the transformation of raw count values. The same principle applies to single-cell data for performing dimensionality reduction and identification of marker genes.
Lastly, we tested 11 different normalisation methods across 3 different brain datasets that have been produced from different single-cell technologies (Hochgerner1 and Hochegerner 3=10X, Darmanis = SS2). In all three datasets, row normalisation seems to perform worse across all the deconvolution methods (lower Pearson Correlation values), while the other normalisation methods appear to perform differently depending on the dataset. The majority of normalisation methods yield good results for the first dataset (Darmani2017). On the other hand, for the other two 10X datasets, none, global_min_max, column-zscore and mean normalisation approaches appear to yield better results. (Fig  3d-f ).
5% of the variance between the bulk and the single-cell (Supplementary Figure  4a, b ). First, we generated pseudo-bulk profiles and single-cell references by randomly splitting the same dataset in half and then performing deconvolution afterwards. This task is a very simple case of deconvolution since both the bulk and the reference contain the same gene expression information. We name this 'self-reference deconvolution'. While self-reference gives the opportunity to study deconvolution with simulations, it oversimplifies the problem. For this reason, we introduced cross-sample deconvolution to more systematically study the impact of batch effects in cell-type deconvolution. For this task, we split the same pancreatic single-cell dataset in half and we selected cells from two individuals to generate the reference while the other two were used to build the bulk profiles.
In this way, we artificially introduced batch effects in cellular decomposition. Deconvolution results from one "bulk" and one "single-cell" algorithm (FARDEEP, DWLS) show a significant drop in deconvolution performance in the cross-sample task compared to the self-reference task (Fig. 5a , Supplementary Figure  5a ). These results suggest that cell-type deconvolution is severely affected by batch effects caused by expression differences across samples. Furthermore, since the deconvolution of real bulk samples typically entails expression data from completely different studies and the use of different technologies, we next studied how cross-studies (same tissue) and cross-technology tasks can impact the performance of deconvolution algorithms. For the cross-study deconvolution, we selected two human pancreatic single-cell datasets  17, 50  (TableS1) that were generated from two different laboratories.
We observed that deconvolution across studies can impact deconvolution performance severely when compared to self-reference tasks (Figure  4b , Supplementary Figure  5b ). Next, we compared the performance of deconvolution when datasets from different technologies are involved. For this task, we focused on 3 independent dataset pairs from the placenta, brain and lung tissue (Supplementary Figure  6 , Supplementary Figure  7 ). Each dataset pair was generated from the same laboratory but with two different single-cell technologies (10X, Smart-Seq2). Given that 10X and Smart-Seq2 are the most common single-cell protocols available, this test will help reveal which references are suitable in deconvolution. Results show that self-reference deconvolution with 10X sc-RNASeq data yields better results overall (higher Pearson correlation, lower RMSE values Fig.
Selecting the best performing methods in the absence of ground truth In contrast to the simulation tasks performed above, here we decompose real bulk profiles from 9 Peripheral Blood Mononuclear Cells (PBMC)  9  samples, utilising our deconvolution pipeline to assess the results and generate an honest consensus of predictions of cell-type proportions. Apart from the publicly available RNA-seq dataset from the above study, there is also flow cytometry data available for the 9 individuals which can be used as ground truth in our deconvolution pipeline. The researchers have measured the proportions of 8 immune cell types using specific markers for each cell population (CD4+ T cells, CD8+ T cells, T-regulatory cells, B cells, NK cells, myeloid dendritic cells, monocytes and neutrophils (Supplementary figure  8a ).
For this deconvolution task, we selected a recent single-cell PBMC dataset  51  to use as a reference. We re-analysed the single-cell data and matched the cell types between the single-cell and the bulk dataset so that our reference has the same cell types that had been measured in the bulk (Supplementary Figure  8c ). We also summarised the proportions of CD4+positive and T-regulatory cells measured by flow cytometry since our single-cell did not contain T-regulatory cells (Supplementary Figure  8b ). Before we apply deconvolution on the 9 real bulk samples we first performed a self-reference deconvolution to evaluate how the reference performs in pseudo-bulk deconvolution. Eight methods yield good results with a Pearson Correlation of more than 0. 92 (Supplementary Figure  8d ). Next, we applied the 29 deconvolution methods included in the pipeline on the real bulk PMN/PBMC data and obtained the cell-type predictions.
In the self-reference task, with the single-cell PBMC dataset, we confirm that the best-performing methods based on Pearson Correlation with ground truth also cluster together after a pairwise comparison of the results and without the use of ground truth. This suggests that when ground truth is not available, pairwise comparison of the results per cell type and per sample can help us select the methods that give more consistent results. In parallel, we evaluated the scalability of each module of our pipeline(the pseudo-bulk generation method,normalisation and the deconvolution methods) and we report the time(s), memory, mean load and CPU time values for each one of them for both deconvolution tasks described above(Supplementary Figure  9a-b ).
Results from Smart-seq2 and 10X single-cell datasets that have been used in this study, show that 10X data is consistently performing better as a reference, this is likely because this technology captures more cells of a specific cell type and as a result, more accurate features are extracted from these data. It should also be noted that in tasks when the bulk and the reference data come from different sources (studies or technologies) -the accuracy score of the deconvolution drops significantly. This indicates that batch effects in the expression of genes, across datasets, affect deconvolution heavily. Methods that take into account and successfully minimise these effects (DWLS, MuSiC) seem to also perform better overall in cross-reference tasks. We apply deconvolution in real PBMC samples using a suitable matching single-cell reference and conclude that two methods, DWLS and bseqsc perform best.
All the above steps that should be performed before deconvolution have been implemented as part of the pipeline with a number of parameters that can be defined by the user. We also propose a novel approach to select suitable deconvolution methods based on pairwise comparison of deconvolution results when ground truth is not available or reliable. The CATD pipeline, developed using snakemake, enables the reproducibility of our results and can be additionally utilised by users to deconvolve new bulk samples by providing only a bulk dataset and a single-cell dataset from the same tissue. When ground truth is provided too, the pipeline will provide evaluation metrics for each method. On the other hand when ground truth is absent the pipeline will perform pairwise comparison of the results and report a heatmap with methods to be selected.